Infrastructure for the AI Agent Era (Part 1): Agents Are Programs
- Aug 12
- 11 min read
Hello. I'm Yongrae Cho, and I lead AI infrastructure research and development on AIEEV's engineering team. I hold a PhD in Computer Science focused on the scalability and reliability of distributed systems, and I now work on the efficiency and reliability of LLM serving on Air Cloud, along with research and development in AI agent technology.
As more services, like AI agents, call a model dozens or even hundreds of times for a single request, the inference bill grows far faster than expected. If your company has ever added an agent feature, you have probably run into this. This is exactly where AIEEV's ongoing concern starts. This article is a record of chasing that problem down to its roots, from a researcher's point of view. We are splitting it into two parts. Part 1 looks at what challenges AI agents pose for serving infrastructure, drawing on the latest research from the world's top systems conferences. Part 2 explains why this trend converges on a picture of "small models running on a very large number of consumer GPUs."
In the chatbot era, one user request meant one model call. In the agent era, a single request becomes a program that chains planning, tool calls, result interpretation, and replanning, and inside that program the model gets called dozens to hundreds of times. Actual measurements show that exploratory agents make an average of 159.7 LLM calls per program. That is roughly 24 times the average for a chatbot conversation, which is 6.66 calls [1]. Now imagine the number of agents running concurrently growing into the hundreds of thousands or millions. The volume of calls that inference infrastructure has to handle becomes an entirely different order of magnitude. This is also the backdrop behind forecasts that the agentic AI market will grow from about 5.2 billion dollars in 2024 to about 200 billion dollars in 2034 [2].
This kind of shift has already appeared once in computing history. Just as computing moved from an era where the CPU executed individual instructions one at a time to the era of the operating system, which manages bundles of instructions and their state as a unit called a process, LLM serving is now shifting its unit of management from single model calls to AI agent programs (Figure 1). That leaves us with one question. Who will handle this explosion of calls, where, with which model, and at what cost.
1. Agents Are Programs, Not Calls
Once you open up the inside of an agent, the structure is clear. When a request comes in, it is (1) broken down into subtasks with a plan, then (2) a subtask is chosen and executed by calling a tool such as web search, code execution, the file system, or an external API, then (3) the model interprets the result, and (4) the state is updated before moving to the next subtask. This loop repeats until no subtasks remain.
From the point of view of a serving system, this structure breaks almost every assumption that chatbot serving was built on. Recent papers commonly point to six characteristics.
Dependent call bundles: Requests arrive as programs. The output of one call becomes the input to the next.
Frequent tool calls: While waiting on a tool's response, the GPU sits idle, and meanwhile the KV cache (the intermediate computation results the model builds while reading the prompt) just occupies space.
Occupying resources beyond the GPU: Containers, files, ports, and browser sessions stay alive alongside the program.
State accumulation: The KV cache and long-term memory pile up with every turn, growing from gigabytes to terabytes.
Long-running execution: Instead of a response that takes seconds, jobs now run for hours, days, or weeks, and the longer they run, the more exposed they are to failure.
The cost-per-call problem: Handling all of these calls with a top-tier frontier model makes the cost unsustainable.
Systems research always starts with characterizing the workload. When the workload changes, the scheduler, the memory manager, and the failure model all get redesigned. That is exactly what is happening with agent workloads right now.
2. The Frontier of Top Systems Conferences
Before getting into the main topic, I want to address one point. Why should a company building AI cloud infrastructure keep tracking research results from the top systems conferences.
The answer lies in the lineage of the software that has become today's industry standard. vLLM, which has effectively become the standard inference engine today, started as a project built on the PagedAttention paper, presented by UC Berkeley researchers at SOSP 2023, one of the top venues in systems [3]. That single paper, which brought the operating system concepts of virtual memory and paging into KV cache management, became foundational software running on GPUs worldwide within just a few years. LMCache and CacheBlend, which I will introduce later, are following exactly the same path. In this field, the top conferences are not a showcase for finished technology. They are where the blueprint for the production stack of two to three years from now gets published first.
That is why we believe that the more a company operates infrastructure, the more it needs to put its feet in foundational research and development. If we stay content with using engines other people built, we have no choice but to trail behind every time the frontier of technology moves. Reading ahead of where the frontier is moving, and pushing that frontier ourselves where it matters. That is why we review conference programs regularly.
Seen through that lens, it becomes clear that "agent serving" has already established itself as an independent research topic in the recent programs of top systems conferences such as OSDI, SOSP, NSDI, EuroSys, and MLSys. Research is emerging that addresses each of the six characteristics outlined above. Here are the representative results.
Problem | System (venue) | Core idea | Reported effect |
Scheduling unit | Autellix (NSDI '26) [1] | Schedules by program instead of by call, preventing programs from interleaving and evicting each other's KV cache | Up to 15x throughput versus existing serving engines |
Waiting on tools | Continuum [4] | Pins the KV cache with a TTL while waiting for a tool response, eliminating recomputation | Improves average job completion time by more than 8x |
Resources beyond the GPU | ThunderAgent (ICML '26) [5] | Programs own containers, ports, and files like an OS process, and reclaim them all at once on termination | 1.5 to 3.6x throughput, 4.2x reduction in disk usage |
Cost per call | AIMS (EuroSys '26) [6] | Decomposes requests into subtasks, then uses a classifier to route each one to an SLM if it is sufficient | Offloads 45.67% of subtasks to a local SLM while maintaining accuracy |
Device utilization | TailorLLM (EuroSys '26) [7] | Specializes high-frequency tasks into an SLM on the user's device with LoRA, resolving them locally | Expands local processing while reducing transfer overhead |
Reliability | LogAct (Meta) [8] | Decomposes the agent into multiple processes with separated permissions and records them to a shared log | Recovery after failure with no rework, and full audit trails |
Each figure is a value reported by that paper in its own experimental setup. What matters more than the individual numbers is the shared direction, namely a shift in the unit of optimization. Existing serving engines such as vLLM or SGLang treat a single call as the unit of optimization. But in agents, calls depend on each other and carry state forward, so scheduling by call causes different programs to interleave on the GPU, evicting the KV cache (a cache that stores and reuses prior computation results) and forcing the same computation to repeat. This is why Autellix was able to boost throughput by up to 15x simply by changing the scheduling unit to the program.
For someone who has been researching distributed systems, this is also a familiar scene. The evolution foreshadowed by Figure 1 at the start of this article, the shift of agents into the processes of a serving system, is not a distant forecast. As the papers above show, it is already underway. Alongside this, there is also active research into the memory hierarchy, covering what to keep as an agent's memory (semantics) and where to place it (physical storage), including Mem0 [9], LMCache [10], and Mooncake. We will meet this thread again in the next section.
3. A Case Study in the Frontier: CacheBlend, Reusing Retrieved Knowledge as Computed State
Let me introduce, in a bit more depth, one example that shows just how active this field is. It is CacheBlend, which won the 2025 Best Paper Award at EuroSys, one of the top conferences in systems, from a research team centered at the University of Chicago [11].
Let me start with the background. During the stage where an LLM reads the prompt (prefill), the model produces intermediate computation results called Key and Value for every token. This is the KV cache. When the same text comes in again, this computation can be reused, cutting time to first token (TTFT) significantly. However, the existing approach, prefix caching, comes with a condition. Only a segment that is completely identical starting from the very beginning of the prompt can be reused.
The problem is that agent and RAG prompts do not look like that. Agents store memory and knowledge in a vector DB, and on every call they retrieve several relevant knowledge chunks and insert them into the prompt. Even with the same chunks A, B, and C, the combination and order differ from request to request, so at best only the first chunk has a matching prefix. Most of the KV cache that was painstakingly built gets thrown away.
That said, simply concatenating each chunk's KV cache breaks quality. The KV for each chunk is the result of computing that chunk in isolation, so it is entirely missing the computation that references other text placed before it, known as cross-attention.
CacheBlend's answer is a compromise. It reuses the KV cache of the chunks regardless of prefix, but selectively recomputes only a small number of tokens where cross-attention has a large impact. It then pipelines this recomputation to overlap with the time it takes to load the KV cache from storage into the GPU. Because the recomputation cost hides behind the loading time, latency does not increase even when the KV cache sits on large, slow, and cheaper storage. As a result, compared to full prefill, which recomputes everything, TTFT becomes 2.2 to 3.3x faster and throughput increases by 2.8 to 5x, while generation quality is maintained (per the paper's abstract) [11].
There are three reasons I chose to introduce this paper at such length.
1. Academia's center of gravity
The fact that a top systems conference awarded its best paper prize to serving research for agents and RAG is, on its own, the clearest signal that this problem now sits at the center of systems research.
2. The speed of adoption into industry
CacheBlend was merged into LMCache [10], an open-source KV cache layer led by the same research team, as a non-prefix KV reuse feature [12], and LMCache combined with vLLM is reporting up to 15x throughput improvement on multi-turn question answering and document analysis workloads [13]. Then, in the LMCache v0.5.2 release published on July 22, CacheBlend was extended to work under vLLM's hybrid KV cache manager (HMA) as well, adding dual RoPE support for sliding window attention models and global P2P token matching that finds tokens across caches scattered across multiple nodes [14]. That is the pace at which an award-winning paper from a top conference becomes, in a little over a year, effectively a default feature of the standard serving stack.
3. Its fit with consumer PC environments
It is true that CacheBlend was designed with data center serving in mind, but if you look again at its core design, it reduces recomputation to a small number of tokens and hides that recomputation time behind the time it takes to load the KV from slow storage. In other words, it promotes DRAM and SSD outside GPU memory into a proper storage tier for the KV cache [11]. This design turns out to be needed most urgently on consumer PCs. Consumer GPUs have around 8 to 24GB of VRAM, so the GPU alone struggles to hold all the knowledge and KV that an agent accumulates, but the same PC usually has tens of gigabytes of DRAM and terabytes of SSD sitting idle. If you stack the KV of the agent's frequently used knowledge chunks in DRAM and SSD, then load them when needed and recompute only a small number of tokens, even a small VRAM can handle long-context agent calls. Layer on top of this the research trend that pulls even idle CPU compute into inference, such as PowerInfer [15], and the direction becomes even clearer. A consumer GPU cloud does not mean a single GPU. It means an entire PC, a single serving node made up of GPU, DRAM, SSD, and CPU together, and that node grows stronger the more idle resources inside the PC get put to use. From this point of view, CacheBlend-style techniques become an even more urgent need in consumer environments than in data centers. Of course, consumer hardware has a different ratio of storage bandwidth to GPU compute speed than a data center, so where the actual balance point between loading and recomputation lands still needs to be verified empirically. This is also a topic we are looking into within the Air Cloud environment.
As one more point worth watching from a researcher's perspective, the direction CacheBlend opens up is a world where knowledge circulates not as text but as already computed state, the KV. The agent's memory layer (the vector DB) and the serving engine's cache layer are starting to merge into a single memory hierarchy. And in a distributed GPU cloud where the cache is scattered across many nodes, finding, moving, and blending which KV lives on which node becomes an even more interesting research topic. The emergence of features like global P2P token matching also means the community is already looking in that direction. The question we face every day while building Air Cloud sits exactly here too. In an environment where tens of thousands of consumer GPUs are scattered across nodes, it is the work of finding which node holds which KV and moving it to where it is needed.
➡️ Continued in the next part.
Agents are no longer calls. They are programs, and the world's top systems conferences are already redesigning serving systems to match this shift. The next part covers the practical answer to this problem: why SLMs and consumer GPUs come together, and the reliability requirements this infrastructure needs to meet.
References
[1] Autellix: An Efficient Serving Engine for LLM Agents as General Programs (NSDI 2026): https://arxiv.org/abs/2502.13965
[2] Small Language Models are the Future of Agentic AI (NVIDIA Research): https://arxiv.org/abs/2506.02153
[3] Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023, the starting point of vLLM): https://arxiv.org/abs/2309.06180
[4] Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live: https://arxiv.org/abs/2511.02230
[5] ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System (ICML 2026 Spotlight): https://arxiv.org/abs/2602.13692
[6] AIMS: Cost-Efficient LLM-Based Agent Deployment in Hybrid Cloud-Edge Environments (EuroSys 2026; arXiv preprint titled HERA): https://arxiv.org/abs/2504.00434
[7] TailorLLM: Collaborative End-Cloud Inference of Large and Small Language Models Based on Low-Rank Adaptation (EuroSys 2026): https://dl.acm.org/doi/10.1145/3767295.3769346
[8] LogAct: Enabling Agentic Reliability via Shared Logs (Meta): https://arxiv.org/abs/2604.07988
[9] Mem0: https://arxiv.org/abs/2504.19413
[10] LMCache (open-source KV cache layer): https://github.com/LMCache/LMCache
[11] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion (EuroSys 2025, Best Paper Award): https://arxiv.org/abs/2405.16444 · https://dl.acm.org/doi/10.1145/3689031.3696098
[12] LMCache blog's introduction to CacheBlend: https://blog.lmcache.ai/en/2025/03/31/cacheblend-best-paper-acm-eurosys25-enabling-100-kv-cache-hit-rate-in-rag/
[13] LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference (MLSys 2026 invited talk): https://mlsys.org/virtual/2026/invited-talk/3646
[14] LMCache v0.5.2 release notes (July 2026): https://github.com/LMCache/LMCache/releases/tag/v0.5.2
[15] PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (SOSP 2024): https://arxiv.org/abs/2312.12456
