Hachi: An Image Search Engine

Summary

Hachi: An (Image) Search engine Only the dead have seen the end of war .. George Santayana For quite some time now, i have been working on and off on a fully self-hosted search engine, in hope to make it easier to search across Personal data in an end to end manner. Even as individuals, we are hoarding and generating more and more data with no end in sight. Such "personal" data is being stored from local hard-disks to corporate controlled cloud-centers which makes it distributed in nature. So for following discussion, "Personal" meaning would be flexible enough to accommodate resources on a remote server and/or on different devices, as long the user could prove authentication and/or authorization to that data. Current implementation supports only "images", but eventual goal is also to support other modalities like video, text and audio, some code would be shared, while some new code would be required to better extract Features for each modality. Such distributed nature of data and potential capabilities of current self-hosted Machine learning models to extract semantic information, only to be queried through a single interface seemed enticing enough for me start this experiment in the first place. Following post at times may seem in-coherent, as i try to articulate my thoughts on the journey of development, challenges faced and future ideas. I hope to treat this as a personal essay with multiple themes, anecdotes and even random thoughts aiming to provide a higher level view of the journey and philosophy so far in more concrete terms. Also, following post doesn't aim to cover every technical choice and implementation in finer details, such discussions would instead be part of dedicated future posts! Motivation: As Humans we tend to remember different attributes/parts of an entity/information at different times, and most of search engines' interfaces refuse to accomodate that. User generally end up with an unidirectional flow of information, with no recourse of provi...