Why Cohere’s ex-AI research lead is betting against the scaling race

Summary

AI labs are racing to build data centers as large as Manhattan, each costing billions of dollars and consuming as much energy as a small city. The effort is driven by a deep belief in “scaling” — the idea that adding more computing power to existing AI training methods will eventually yield superintelligent systems capable of performing all kinds of tasks. But a growing chorus of AI researchers say the scaling of large language models may be reaching its limits, and that other breakthroughs may be needed to improve AI performance. That’s the bet Sara Hooker, Cohere’s former VP of AI Research and a Google Brain alumna, is taking with her new startup, Adaption Labs. She co-founded the company with fellow Cohere and Google veteran Sudip Roy, and it’s built on the idea that scaling LLMs has become an inefficient way to squeeze more performance out of AI models. Hooker, who left Cohere in August, quietly announced the startup this month to start recruiting more broadly. In an interview with TechCrunch, Hooker says Adaption Labs is building AI systems that can continuously adapt and learn from their real-world experiences, and do so extremely efficiently. She declined to share details about the methods behind this approach or whether the company relies on LLMs or another architecture. “There is a turning point now where it’s very clear that the formula of just scaling these models — scaling-pilled approaches, which are attractive but extremely boring — hasn’t produced intelligence that is able to navigate or interact with the world,” said Hooker. Adapting is the “heart of learning,” according to Hooker. For example, stub your toe when you walk past your dining room table, and you’ll learn to step more carefully around it next time. AI labs have tried to capture this idea through reinforcement learning (RL), which allows AI models to learn from their mistakes in controlled settings. However, today’s RL methods don’t help AI models in production — meaning systems already be...