Experimenting with Local LLMs on macOS

Summary

So, this blog post will be about LLMs, and everyone has opinions about that. To be upfront about it, I’m a skeptic (bordering on hater), yet I like experimenting with stuff so I download and run them locally on my Mac. And I’ll teach you how to do it too, if you’d like! Some call them fancy autocomplete, some argue that they are sentient and should have rights. The truth is somewhere in between. Yes, they perform next word prediction, but it’s so complex that there’s nontrivial emergent behavior. No, they don’t have creativity or a mind. I believe one day we can create sentient machines, but not in this current iteration, maybe not before we go extinct. Now that we’re out of the science fiction territory, let’s talk about their strengths. Laurie has a great post about it, which I highly recommend, but in summary they are generally good at summarizing text, regurgitating home maintenance advice from reddit, or telling you that you have cancer. I also use them for brain-dumping. I find it hard to keep a journal, because I find it boring, but when you’re pretending to be writing to someone, it’s easier. If you have friends, that’s much better, but some topics are too personal and a friend may not be available at 4 AM. I mostly ignore its responses, because it’s for me to unload, not to listen to a machine spew slop. I suggest you do the same, because we’re anthropomorphization machines and I’d rather not experience AI psychosis. It’s better if you don’t give it a chance to convince you it’s real. I could use a system prompt so it doesn’t follow up with dumb questions (or “YoU’Re AbSoLuTeLy CoRrEcT”s), but I never bothered as I already don’t read it. Lastly, I’m interested in them because it’s tech and I like tech. I don’t believe they make you that much productive, and I never let them write for me. When I ask it something, I always fact-check, they are known to hallucinate (or bullshit, if you prefer) and I’ve experienced this too many times to trust it. Most importan...