So you wanna build a local RAG?

Summary

When we launched Skald, we wanted it to not only be self-hostable, but also for one to be able to run it without sending any data to third-parties. With LLMs getting better and better, privacy-sensitive organizations shouldn't have to choose between being left behind by not accessing frontier models and doing away with their committment (or legal requirement) for data privacy. So here's what we did to support this use case and also some benchmarks comparing performance when using proprietary APIs vs self-hosted open-source tech. RAG components and their OSS alternatives A basic RAG usually has the following core components: A vector database A vector embeddings model An LLM And most times it also has these as well: A reranker Document parsing (for PDFs, PowerPoints, etc) What that means is that when you're looking to build a fully local RAG setup, you'll need to substitute whatever SaaS providers you're using for a local option for each of those components. Here's a table with some examples of what we might use in a scenario where we can use third-party Cloud services and one where we can't: ComponentProprietary OptionsOpen-Source OptionsVector DatabasePinecone, Turbopuffer, Weaviate Cloud, Qdrant CloudQdrant, Weaviate, Postgres with pgvectorVector Embeddings ProviderOpenAI, Cohere, VoyageSentence Transformers, BGE, E5LLMGPT, Claude, GeminiLlama, Mistral, GPT-OSSRerankerCohere, VoyageBGE Reranker, Sentence Transformers Cross-EncoderDocument ParsingReducto, DatalabDocling Do note that running something locally does not mean it needs to be open-source, as one could pay for a license to self-host proprietary software. But at Skald our goal was to use fully open-source tech, which is what I'll be convering here. The table above is far from covering all available options on both columns, but basically it gives you an indication of what to research into in order to pick a tool that works for you. As with anything, what works for you will greatly depend on your use case. A...