AI models need a virtual machine

Summary

Applications using AI embed the AI model in a framework that interfaces between the model and the rest of the system, providing needed services such as tool calling, context retrieval, etc. Software for early chatbots took user input, called the LLM, and returned the result to the user; essentially just a read-eval-print loop. But, as the capabilities of LLMs have evolved and extension mechanisms, such as MCP were defined, the complexities of the control software that calls the LLM have increased. AI software systems require the same qualities that an operating system provides, including security, isolation, extensibility, and portability. For example, when an AI model needs to be given a file as part of its context, access control must be established that determines if the model should be allowed to view that file. We believe it is time to consider standardizing the ways in which the AI models are embedded into software and think of that control software layer as a virtual machine, where one of the machine instructions, albeit a super-powerful one, is to call the LLM. Our approach decouples model development from integration logic, allowing any model to “plug in” to a rich software ecosystem that includes tools, security controls, memory abstractions, etc. Similar to the impact that the Java Virtual Machine had, creating a specification of a VM for the AI orchestrator could enable a “write once, run anywhere” execution environment for AI models while at the same time providing familiar constraints and governance to maintain security and privacy in existing software systems. Below we outline related work in this direction, the motivation behind it, and the key benefits of an AI Model VM. Introduction AI models are being leveraged in existing software as application copilots, embedded in IDEs, and with the rise of the MCP protocol, are increasingly able to use tools, implement agents, etc. This rapid evolution of valuable use cases brings with it a greater need to en...