LLMs are bullshitters. But that doesn't mean they're not useful

Summary

Note: This is a personal essay by Matt Ranger, Kagi’s head of ML In 1986, Harry Frankfurt wrote On Bullshit. He differentiates a lying from bullshitting: Lying means you have a concept of what is true, and you’re choosing to misrepresent it. Bullshitting means you’re attempting to persuade without caring for what the truth is. I’m not the first to note that LLMs are bullshitters, but I want to delve into what this means. The bearded surgeon mother Gemini 2.5 pro was Google’s strongest model until yesterday. At launch it was showered with praise to the point some questioned if humanity itself is now redundant. Let’s see how Gemini 2.5 pro fares on an easy question: This is some decent bullshit! Now, you might be tempted to dismiss this as a cute party trick. After all, modern LLMs are capable of impressive displays of intelligence, so why would we care if they get some riddles wrong? In fact, these “LLM Traps” expose a core feature of how LLMs are built and function. LLMs predict text. That’s it. Simplifying a little [^1], LLMs have always been trained in the same two steps: The model is trained to predict what comes next on massive amounts of written content. This is called a “base” model. Base models simply predict the text that is most statistically likely to be next. This is why models answer “the surgeon is the boy’s mother” in the example above – it’s the answer to a classic riddle. So it’s a highly probable prediction for a question about why a surgeon can’t operate. The base model is trained on curated sets or input:output pairs to finetune the behavior. You can see effects of finetuning if you have access to preview versions of some models. For instance, a finetuned Gemini 2.5 Pro correctly notices that this question is missing the mentioned chart: However, if you asked the same question a few months ago, when Gemini 2.5 pro had an API to the incompletely finetuned Preview model, you’d get this answer: Answering “yes” to that question is statistically most l...