Building a search engine from scratch, in Rust: part 1

Summary

In the previous article, I introduced what project we're going to address in the following weeks: how to build a cross-platform search engine with encryption capabilities. Today, we'll have a look at the first technical challenge: how to store things on disk.You might be thinking that we start with a simple topic, to warm up and get ready for the serious parts. That's both right and wrong at the same time. Writing on the filesystem from a mobile application or a desktop application is an easy task, but doing so in the browser while using the same interface is not as easy as it seems.And on top of that, we want our storage layer to read and write encrypted files, without compromising too much on performance.The Storage ChallengeBefore diving into search algorithms and indexing, we need to solve a fundamental challenge: how do we store data consistently across different platforms? While writing to the filesystem is straightforward on desktop and mobile platforms, browsers present unique constraints that require careful consideration.Key Challenges:Cross-platform filesystem accessAsync operations requirementPerformance considerationsEncryption integration (coming up next)Exploring Available SolutionsWhen I first approached this problem, I naturally looked at what the browser ecosystem offers for storage. The most obvious solution seemed to be LocalStorage - it's simple, widely supported, and easy to use. However, its limitations quickly became apparent: the storage size is restricted (usually to just a few megabytes), and it only supports string values, which would force us to serialize and deserialize our data constantly.Next, I considered IndexedDB. It's more powerful and supports larger datasets, which initially seemed promising. But working with IndexedDB directly is notoriously complex - its API is verbose and requires handling multiple edge cases. While there are wrapper libraries that make it more palatable, they would add unnecessary bulk to our final bundle si...