Life of an inference request (vLLM V1): How LLMs are served efficiently at scale

Summary

Life of an inference request (vLLM V1): How LLMs are served efficiently at scaleJune 27, 2025 · 10 min readJunhao LiSenior Software EngineerUbicloud is an open source alternative to AWS. We offer managed cloud services that build on top of PostgreSQL, Kubernetes, vLLM, and others.‍‍vLLM is an open-source inference engine that serves large language models. We deploy multiple vLLM instances across GPUs and load open weight models like Llama 4 into them. We then load balance traffic across vLLM instances, run health checks, and do upgrades. Our customers consume our managed service by sending their prompts to our API endpoints. This endpoint also determines the vLLM instance that serves their prompt.vLLM sits at the intersection of AI and systems programming, so we thought that diving into its details might interest some of our readers. In this blog post, we describe how an inference request travels through vLLM’s OpenAI-compatible API server and core engine. We also provide key code pointers.We assume readers are already familiar with the transformer architecture and large language models. If you're not, we highly recommend this video by OpenAI co-founder Andrej Karpathy. We will focus on the new V1 architecture of vLLM and how it achieves state-of-the-art text generation performance. If you're looking for the V0 behavior or multi-modal inference, please refer to other vLLM documentation.TerminologyWe use the following terms throughout this blog. These terms also align with what is used in vLLM’s codebase and documentation:Request: An incoming chat completion message from the client, formatted in the OpenAI-compatible format.Sequence: The combined stream of prompt and response tokens associated with a request. Except in special cases, a single request typically corresponds to one sequence. Sequence is sometimes used interchangeably with Request in both the codebase and this blog post.Batching: The process of grouping multiple requests together into a single forward pa...