LLMs are cheap

Summary

This post is making a point - generative AI is relatively cheap - that might seem so obvious it doesn't need making. I'm mostly writing it because I've repeatedly had the same discussion in the past six months where people claim the opposite. Not only is the misconception still around, but it's not even getting less frequent. This is mainly written to have a document I can point people at, the next time it repeats. It seems to be a common, if not a majority, belief that Large Language Models (in the colloquial sense of "things that are like ChatGPT") are very expensive to operate. This then leads to a ton of innumerate analyses about how AI companies must be obviously doomed, as well as a myopic view on how consumer AI businesses can/will be monetized. It's an understandable mistake, since inference was indeed very expensive at the start of the AI boom, and those costs were talked about a lot. But inference has gotten cheaper even faster than models have gotten better, and nobody has an intuition for something becoming 1000x cheaper in two years. It just doesn't happen. It doesn't help that the common pricing model ("$ per million tokens") is very hard to visualize. So let's compare LLMs to web search. I'm choosing search as the comparison since it's in the same vicinity and since it's something everyone uses and nobody pays for, not because I'm suggesting that ungrounded generative AI is a good substitute for search. (It should also go without saying that these are just my personal opinions.) What is the price of a web search? Here's the public API pricing for some companies operating their own web search infrastructure, retrieved on 2025-05-02: The Gemini API pricing lists a "Grounding with Google Search" feature at $35/1k queries. I believe that's the best number we can get for Google, they don't publish prices for a "raw" search result API. The Bing Search API is priced at $15/1k queries at the cheapest tier. Brave has a price of $5/1k searches at the cheapest t...