Infinite Tool Use

Summary

An LLM should never output anything but tool calls and their arguments. The tools hold the specific, instantiated state of what the model is doing and its goals, while the model itself holds only the information it requires for its immediate task and some additional context, leading to specialization between the LLM and its tools. Exclusively working through tools allows models to externalize large parts of their intelligence to more efficient, domain-specific programs. Table of contents: Examples The benefits of infinite tool use are best understood via examples. Text Editing Here’s how I wrote this article so far: I had an idea and wrote it down in a few bullet points. Then, I wrote the introduction. While doing that, I jumped to the end of the article, added a few more bullet points, and edited others. I started writing this section, interrupted it by writing down an idea about the architecture of such models, then came back here; realized that I should re-write this section, started doing that, edited the introduction to fit, went back to the re-write, and here we are. I’m not even half-way done with the article and I’m sure I already forgot several steps that I took. Now contrast that with the way an LLM currently writes: It generates text forward-only. (Almost) no matter how good it is, it will make mistakes, especially in out-of-distribution (OOD) domains. Forward-only generation makes multi-resolution generation much more difficult: I as a human can create hundreds of versions of the same article; edit a sentence here and there, write down an idea as a bulletpoint, delete something dumb, turn a bulletpoint into a full section, etc.; in other words, I can interleave actions at different levels of specificity. Imagine how confusing it would be to hold all those edits in memory at once! Editing through external tools allows for explicit, selective forgetting. LLMs on the other hand either need to generate from most general to most specific in order—a very limit...