Convenient LLMs at HomeI’ve been wanting to experiment with LLMs in my homelab, but didn’t want the overhead of a dedicated GPU machine or the slowness of CPU processing. I also wanted everything to be convenient long-term: updates needed to be automated, and if the OS dies rebuilding needed to be quick and easy, etc.Running NixOS with WSL on my gaming PC seemed like the perfect solution, but I kept running into several challenges:Concerns of my vram getting locked to LLMsWSL would shut down automatically; Microsoft doesn’t support WSL running if you aren’t actively using it.NixOS on WSL didn’t support Nvidia out of the box.I refused to manage a separate Ubuntu box that would need reconfiguring from scratch.After hacking away at it for a number of weeks, I’ve now solved the blocks:Ollama (by default) unloads models if they haven’t been used in the past 5 minutes.WSL automatically starts up, and stays running.Configured Nvidia Container Toolkit for NixOS on WSL.Ollama container configured for NixOS.NixOS handles the configuration for the whole system so rebuilding from scratch is easy.My NixOS flake is already configured for automatic updates that my WSL system can just inherit.While there is some generically useful information here, this is heavily NixOS focused. Additionally I heavily rely on Tailscale to make my own networking convenient, so there are some optional Tailscale steps included as well.Live configuration that I actively use at home:Just here for referenceForce WSL to stay runningTo start with the biggest wall with an OS agnostic fix, I was able to find this github post. Assuming you are running Ubuntu on WSL you can run1 wsl --exec dbus-launch true and this will launch wsl and keep it running. You can setup a basic task in windows Task Scheduler to automatically run this command on at startup and set it to run with the user logged out.For NixOS on WSL I found this didn’t quite work, the --exec option seeming to have issues. So I set it up like this ins...
First seen: 2025-04-11 04:47
Last seen: 2025-04-11 14:48