Inner Loop Agents

Summary

Inner Loop Agents Sat April 19, 2025 What if an LLM could use tools directly? As in, what if LLMs executed tool calls without going back to the client. That’s the idea behind inner loop agents. It’s a conceptual shift. Instead of thinking of agents as being a system involving client & server, you just have a single entity, the LLM. I hope it will help clarify how o3 and o4-mini work. (note: this post isn’t as long as it looks, there’s a lot of diagrams and examples) To illustrate, regular LLMs rely on the client to parse and execute tools, like this: graph TD subgraph inn["LLM (Inner Loop)"] Tokenizer-->nn[Neural Net]-->samp[Select Next Token]-->Tokenizer end text((Input))-->Tokenizer parse--->out((Output)) samp-->parse[Parse Tool Calls]-->exec[Run Tools]-->parse parse--"toolresult"-->Tokenizer On the other hand, with inner loop agents, the LLM can parse and execute tools on it’s own, like this: graph TD subgraph inn["Inner Loop Agent"] direction TB Tokenizer nn[Neural Net] samp[Select Next Token] parse[Parse Tool Calls] exec[Run Tools] end text((Input)) --> Tokenizer Tokenizer --> nn --> samp --> parse parse --> exec -->parse parse -----> Tokenizer parse ---> out((Output)) The LLM Operating Software (Ollama, vLLM, etc) In these diagrams, the LLM is emitting text that looks like this: System: You are an agent with access to the following tools: <tool name="google_maps" description="Look up directions between two places on Google Maps"> <param name="begin" description="The starting point of the trip"/> <param name="end" description="The ending point of the trip"/> </tool> User: How do you drive from Raleigh, NC to Greene, NY? Assistant: To do this, I will use my Google Maps tool. <tool name="google_maps"> <param name="begin">Raleigh, NC</param> <param name="end">Greene, NY</param> </tool> <|eot|> The LLM only generates the text after "Assistant:" That <|eot|> is a special token that the LLM is trained to emit as a way to signal that it’s done. The software you’re usi...