Local Chatbot RAG with FreeBSD Knowledge

Summary

Out of multiple conversations with people at BSD conferences, I noticed that many would love to see a chatbot that provides precise information on FreeBSD—for users, admins, and developers. I strongly believe that there should not be an official chat.freebsd.org. Local chatbots work well and can be tweaked to fit personal needs. This documentation is written for macOS with Apple Silicon (because of the GPU support), but should work on other OSes as well. Step 1: Install Ollama (API for Multiple LLMs) brew install ollama ollama pull gemma3:latest You can try deepseek-r1:latest or even deepseek-r1:70b on more powerful GPUs. Step 2: Install Open-WebUI for a UI and Built-in Vector Database curl -LsSf https://astral.sh/uv/install.sh | sh DATA_DIR=~/.open-webui uvx --python 3.11 open-webui@latest serve Now browse to http://localhost:5000/ Welcome to your own, local chatbot! Step 3: Feed Knowledge to the Model This is where the real work begins: You need to tell the LLM what’s right or wrong, what’s necessary, and how FreeBSD differs from Linux. 3.1 Download the FreeBSD Documentation Follow this guide for more details. Install dependencies (use apt, yum etc. according to your OS): brew install hugo ruby git bmake Update your shell configuration: echo 'export PATH="$(brew --prefix ruby)/bin:$PATH"' >> ~/.zshrc echo 'export PATH="$(brew --prefix hugo)/bin:$PATH"' >> ~/.zshrc echo 'export GEM_PATH="$(gem environment gemdir)"' >> ~/.zshrc echo 'export PATH="${GEM_PATH}/bin:$PATH"' >> ~/.zshrc source ~/.zshrc Install required gems: sudo gem install rouge asciidoctor asciidoctor-pdf asciidoctor-epub3 Clone and build the FreeBSD documentation: git clone https://git.FreeBSD.org/doc.git ~/doc cd ~/doc/documentation bmake run USE_RUBYGEMS=YES RUBY_CMD=$(brew --prefix ruby)/bin/ruby More information can be found here: https://docs.freebsd.org/en/books/fdp-primer/overview/#mac-os-installation-process 3.2 Upload Documentation to Open-WebUI Go to: http://localhost:5000/workspace/knowled...