The Cache Layer That Makes AI Agents 10x Faster?

🧮 115,000 Tokens, 96.9% Reusable

In a dataset of 739 anonymized Claude Code conversations, the median session started with roughly 20,000 tokens of input and grew to about 115,000. By that point, 96.9% of the context was theoretically reusable — the repository files were mostly the same, the system prompt was the same, the tool definitions were the same, most of the conversation was the same. The agent had simply appended a small amount of new information to an enormous amount of old.

This creates a strange infrastructure problem. If the server loses one particular piece of memory, the GPU may have to compute that old context all over again — not because the model forgot the conversation, but because the server forgot the computation behind it. That piece of memory is the KV cache, and LMCache is the open-source project turning it into something far bigger than an ordinary cache.

⚙️ Prefill, Decode & the Cost of KV State

To see why, separate the two phases of LLM inference that get hidden behind one loading animation. When you send a long prompt, the model doesn't start emitting tokens — first it must prefill: run the input tokens through its layers and build the internal attention state it will need later. That state holds keys and values for every previous token — the KV cache. Once it exists, decode becomes far cheaper, because each new token doesn't reconstruct the entire history from scratch.

Imagine an agent reading a repository. Turn one contains your instructions, tool definitions, ten source files, and some conversation history. The agent calls a tool; turn two contains almost all of that again, plus one new result. Then another tool call, another file, another response. Context keeps growing, but the prefix stays fixed. The LMCache authors estimate that repeatedly prefilling a 100,000-token agent context can waste roughly 95% of the GPU work associated with that prefill.

🔁 vLLM Already Prefix-Caches

There's an important twist: vLLM already knows this. vLLM v1 ships automatic prefix caching enabled by default — when a request begins with tokens the engine has seen before, it reuses the corresponding KV blocks instead of recomputing them. SGLang does the same with its RadixAttention-style prefix caching. So LMCache isn't fixing a world where nobody thought of caching. The problem is where the cache lives.

The fastest KV cache sits in GPU HBM, and that's exactly where you want it. If the state you need is already in HBM, adding another storage layer can make things slower. One AMD benchmark shows this cleanly: running MiniMax-M2.5 across two MI300X GPUs at low load, with the active KV working set comfortably inside GPU memory, vLLM's HBM prefix cache completed 52 requests while LMCache completed 25 — and average time-to-first-token was worse with LMCache. The extra layer was charging overhead without solving a problem.

🗄️ When KV Outgrows GPU Memory

Then the working set stops fitting. For that same MiniMax-M2.5 setup, the researchers estimate a 100,000-token KV cache consumes about 12 GB of HBM — and that's one conversation. Add dozens of users, multiple coding agents, repository-scale contexts, and long-running sessions, and GPU memory fills up: old cache blocks get evicted, and suddenly a 97%-reusable conversation doesn't help because the reusable computation is no longer there.

LMCache attacks this by treating KV state the way systems have treated data for decades — as a hierarchy. Hot state stays near the GPU; more KV state lives in ordinary CPU DRAM; larger pools spill into NVMe and persistent or distributed backends. The relevant comparison is not "is RAM faster than HBM?" — obviously not. The real question is: is retrieving a computation you already performed from slower memory cheaper than performing it again? For short context, usually no. For enormous context, the answer can flip dramatically.

📊 Google GKE: The Tiered-Cache Benchmark

Google tested this on GKE with eight H100 GPUs and Llama-3.3-70B-Instruct. When the entire KV working set fit in HBM, slower tiers added essentially no benefit — exactly as expected. Then they pushed the working set past GPU memory, and the ladder emerged:

Same model, same GPUs — the server simply stopped throwing away reusable work once HBM became insufficient. But the benchmark also contains a more instructive counter-result. Google pushed the hierarchy hard enough to saturate GPU and CPU memory and spill heavily into SSD — and the "longer context = faster LMCache" story broke. At 10,000 shared tokens, TTFT became 121% worse than the HBM-only baseline; at 50,000, 48% worse — even though total input throughput still improved. The system could process more work overall while making an individual user wait longer for the first token.

🔀 Multi-Process Ownership: The ~13x Result

Storage capacity alone doesn't solve ownership. Imagine eight inference workers on one machine. Worker 3 computes a huge conversation and stores its KV state; the next request from that conversation lands on worker 7. If caches are isolated per-process, worker 7 can't use what worker 3 already calculated — same machine, same model, potentially identical context, but the useful state belongs to the wrong process.

This is the most interesting change LMCache made in 2026: its multi-process architecture moves cache management out of the inference engine into a separate service. Multiple vLLM instances connect to one LMCache server, so conversation state no longer has to belong to a single worker. The benchmark is almost absurd: on Qwen3-235B-A22B (a MoE with 22B active params) across eight H100s, mean time-to-first-token went from 3.98 seconds to 0.29 seconds — a ~13× reduction.

LMCache's current documentation now recommends the multi-process architecture for new deployments: one cache service beside multiple inference engines, managing a larger pool independently of GPU memory and connecting to multiple storage tiers. A worker can disappear without destroying all the reusable computation it created.

⚖️ When LMCache Pays Off vs. When HBM Wins

There is no universal threshold. The AMD team found a crossover around 250,000–300,000 tokens of sustained working set on their MiniMax-M2.5 setup — below it, HBM prefix caching made more sense; above it, the larger DRAM tier began paying for itself. But don't turn that into a rule: it belongs to that model, those GPUs, that amount of HBM, and that memory interconnect. Change the bandwidth and the economics change.

The real formula is simpler — how expensive is recomputing the missing KV state, versus how expensive is retrieving it? LMCache itself publishes results where the answer is clearly "just use HBM." In one controlled synthetic test with a 75% cache hit rate, vLLM's HBM prefix cache processed roughly 3.6k input tokens per second while LMCache managed about 1.9k. That's not a small loss — and the reason is instructive: there was plenty of reuse but not enough GPU-memory pressure, so the CPU tier was unnecessary, and LMCache still paid for cache-key handling, lookups, transfer checks, and connector work.

🎯 Cache Misses, Hash Seeds & Stable Prefixes

There are surprisingly small ways to destroy the entire optimization — your prefix actually has to match. Change serialization, move something near the beginning of the prompt, inject changing metadata, truncate the front of a long conversation, or route a request somewhere that can't see the previous cache, and reusable context becomes a cache miss.

The AMD deployment hit a particularly nasty one: Python's hash randomization. Without a consistent PYTHONHASHSEED across processes, bit-identical prompts can generate incompatible cache keys and produce zero cache hits. Nothing about the model was wrong; nothing necessarily crashed — the system simply stopped recognizing work it had already done. Which leads to a useful zero-cost optimization before installing anything: if you expect a prefix to be cached, stop changing it for no reason. Keep stable instructions stable, serialize tool definitions consistently, and don't inject constantly-changing information near the front of giant prompts unless you actually need it there.

🧩 CacheBlend & Reuse Beyond Matching Prefixes

Traditional prefix caching only helps when reused information forms a prefix — but retrieval systems constantly rearrange documents. CacheBlend explored reusing precomputed KV state even when chunks appear in different positions, selectively recomputing only the pieces whose attention relationships actually changed. Across its tests it reduced time-to-first-token by roughly 2.2–3.3× compared with full KV recomputation.

That tells you where this space is heading: not necessarily toward LMCache winning every deployment. NVIDIA has its own KV-management work, SGLang has hierarchical caching, and other systems are attacking the same problem from different directions. The industry isn't converging on one project — it's converging on one idea: KV cache is becoming too valuable to treat as disposable GPU scratch space.

🏗️ A Memory Layer for AI Agents

Coding agents are accelerating the transition. A chatbot request might be a few thousand mostly-fresh tokens; a coding agent can spend an hour repeatedly sending the same repository, the same tool definitions, and the same growing conversation back into the model. Eventually there can be more reusable computation in a request than new computation — at which point inference becomes partly a storage problem.

Where does the model's previous work live? How quickly can another GPU retrieve it? How long should it survive? Which users are allowed to reuse it? Should it sit in HBM, DRAM, NVMe, or somewhere off-machine entirely? For years the KV cache was something an inference engine created and threw away; agent workloads are making that assumption extremely expensive. LMCache is interesting because it pushes the opposite idea far: the computation behind a conversation can survive outside the GPU, outside one worker, shared, tiered, moved, and potentially persisted.

✅ Key Takeaways

1. Agent context is ~97% reusable. 739 Claude Code sessions grew from 20k to 115k tokens, 96.9% of which was theoretically reusable — the cost is recomputation, not storage.
2. Prefill builds the KV cache; losing it re-runs attention over history. Repeatedly prefilling a 100k-token context wastes ~95% of the GPU work.
3. vLLM and SGLang already prefix-cache. LMCache's problem is cache residency — where the state lives once it no longer fits in HBM.
4. If it fits in HBM, HBM wins. AMD's low-load MiniMax-M2.5 run: 52 requests (vLLM HBM) vs 25 (LMCache).
5. Tiered caching is a data-movement trade-off. Google GKE gained −79% TTFT at 100k shared tokens, but −121% TTFT when spilling hard to SSD at 10k.
6. The ~13× unlock is ownership. Moving the cache out of the engine into a shared service took Qwen3-235B-A22B from 3.98s to 0.29s TTFT across 8× H100.
7. There's no universal crossover. AMD's 250–300k-token threshold is hardware-specific; the real formula is recompute cost vs. retrieve cost.
8. Cache keys are fragile. A mismatched PYTHONHASHSEED across processes can produce zero hits on bit-identical prompts.
9. CacheBlend extends reuse past prefixes. Selective recompute of rearranged chunks: 2.2–3.3× TTFT reduction.
10. KV cache is becoming a memory layer for agents. The next bottleneck isn't computing new tokens — it's recomputing the same ones.

🔗 Resources & Links

• 📦 LMCache on GitHub
• 📄 LMCache paper — "LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference"
• 📄 CacheBlend paper — "Fast Large Language Model Serving for RAG with Cached Knowledge Fusion"
• ☁️ Google GKE tiered KV-cache benchmark
• 🔴 AMD MI300X agentic-workload benchmarks
• ⚙️ LMCache multi-process documentation
• ▶️ Code Unpacked — the channel

Project and model names verified against primary sources: the LMCache and CacheBlend arXiv abstracts, and the Google GKE benchmark post (Llama-3.3-70B-Instruct). The caption track mangles several ("LMCH" → LMCache, "VLM" → vLLM, "SGLAN" → SGLang, "QN3235B22B" → Qwen3-235B-A22B).

🕒 Timestamp Index