The Case for the Return of Fine-Tuning

Summary

Déjà Tune Most of my reading this week focused on fine-tuning, sparked by Thinking Machines Labs’ announcement of Tinker. The six-month-old, already $12B-valued startup founded by OpenAI’s former CTO Mira Murati wants to bring fine-tuning back into the spotlight with a new fine-tuning-as-a-platform initiative positioned as a foundation for research collaborations with universities. A few days later, Clément Delangue from Hugging Face posted that he sensed a paradigm shift toward self-managed, open-source, and specialized LLM deployments, even backed by dedicated hardware like NVIDIA’s DGX Spark, many conversations with founders about growing client demand, or the Personal AI Workstation, a very clever marketing stunt from a16z (I’m jealous). All of this feels like a déjà vu. For a brief moment after the first wave of large language models, fine-tuning was the hottest topic in machine learning. Then, just as quickly, it disappeared from most production systems, now accounting for less than 10% of AI inference workloads. So, how did fine-tuning get sidelined so fast, and why do we feel it could be time for a come-back? And more importantly, what could be different this time? Attention, Please Before the Transformer breakthrough, the spark that led to the LLMs we use today, NLP relied on specialized models. Early progress came from recurrent architectures like RNNs and LSTMs, which for the first time learned directly from word sequences instead of relying on hand‑crafted linguistic features. A step forward in representations, but without any learning paradigm that would define later foundation models. Each application required to start from scratch on task-specific data. In 2017, Google’s Attention Is All You Need introduced the Transformer architecture, replacing recurrence and convolution with self‑attention alone. Seven months later, ULMFiT demonstrated that a pretrained language model (at the time still based on LSTMs) could be fine‑tuned for different tasks, and h...