Reinforcement learning, explained with a minimum of math and jargon

Summary

It’s Agent Week at Understanding AI! This week I’m going to publish a series of articles explaining the most important AI trend of 2025: agents! Today is a deep dive into reinforcement learning, the training technique that made agentic models like Claude 3.5 Sonnet and o3 possible.Today’s article is available for free, but some articles in the series—including tomorrow’s article on MCP and tool use—will be for paying subscribers only. I’m offering a 20 percent discount on annual subscriptions through the end of the week. That’s the best price I’ll offer for the rest of 2025. Please click here to subscribe.In April 2023, a few weeks after the launch of GPT-4, the Internet went wild for two new software projects with the audacious names BabyAGI and AutoGPT.“Over the past week, developers around the world have begun building ‘autonomous agents’ that work with large language models (LLMs) such as OpenAI’s GPT-4 to solve complex problems,” Mark Sullivan wrote for Fast Company. “Autonomous agents can already perform tasks as varied as conducting web research, writing code, and creating to-do lists.”BabyAGI and AutoGPT repeatedly prompted GPT-4 in an effort to elicit agent-like behavior. The first prompt would give GPT-4 a goal (like “create a 7-day meal plan for me”) and ask it to come up with a to-do list (it might generate items like “Research healthy meal plans,” “plan meals for the week,” and “write the recipes for each dinner in diet.txt”).Then these frameworks would have GPT-4 tackle one step at a time. Their creators hoped that invoking GPT-4 in a loop like this would enable it to tackle projects that required many steps.But after an initial wave of hype, it became clear that GPT-4 wasn’t up to the task. Most of the time, GPT-4 could come up with a reasonable list of tasks. And sometimes it was able to complete a few individual tasks. But the model struggled to stay focused.Sometimes GPT-4 would make a small early mistake, fail to correct it, and then get more and ...