When Repeated Context Becomes an Infrastructure Problem | Part 2 of 3

Part 2 of a three-part series on prompt caching, prefix caching, and shared context memory for AI inference.

Part 1 framed prompt caching as a business issue: stop paying GPUs to rebuild context they have already processed.

The post you are reading is the bridge from business impact to infrastructure design.

The important shift is that prompt caching is not just a provider feature or a checkbox in an API. In a production inference fleet, it depends on how the serving stack stores, evicts, restores, and shares KV cache state.

For one worker, the solution can be simple. Keep KV blocks in GPU memory or host memory. If the next request lands on the same process before eviction, the prefix cache hits.

For a real service, that assumption breaks down quickly.

Requests move across replicas. GPU HBM fills. Long-context sessions spill. Workers restart. Agents branch. Tenants share infrastructure but do not always share request locality. A prefix cache that only helps one process is useful, but it is not a fleet architecture.

The production shape of repeated context

Many context-heavy AI applications repeat more context than they change.

A coding agent repeats project instructions, tool schemas, repository rules, file excerpts, and prior observations. A RAG assistant repeats long document chunks while the user asks follow-up questions. A support agent repeats policy and product context. A batch evaluation job repeats the same template against many inputs.

The serving stack sees these as repeated token prefixes. When the prefix is identical, the runtime can reuse cached KV blocks instead of rerunning prefill.

Inference engines and KV-transfer projects already optimize for this pattern. For example, vLLM's automatic prefix caching stores KV cache blocks for processed requests and reuses them when a new request arrives with the same prefix. SGLang's RadixAttention uses a radix-tree style runtime cache to reuse common token prefixes. Other projects expose paths for moving KV state outside the hottest GPU memory tier.

Those names matter to infrastructure teams, but they should not be the headline for the business. The broader point is simpler: repeated context is becoming a managed system resource.

What recent production systems are showing

The shift is already visible in public engineering writeups.

Uber's Gen AI Gateway is a useful pattern because it does not assume one model or one provider. Uber describes a unified interface for external LLMs and in-house hosted open-source LLMs, with cost guardrails, attribution, policy controls, PII redaction, and production monitoring around the path. The important infrastructure lesson is that model choice, cost control, privacy, and observability are now part of the same serving architecture.

DoorDash shows the same pressure one level closer to the prompt. In its LLM Assistant platform, a unified model factory selects the model per role, supports fallback across providers, and allows per-role swaps without a code release. In SafeChat, DoorDash combines internal and external models so each message is evaluated at the right cost and level of attention. In its LLM personalization pipeline, a full memory block is larger than what any single use case needs, so DoorDash trims the block to an explicit per-use-case allowlist before the LLM call. That reduces token cost and removes irrelevant context that can hurt quality.

Both examples point in the same direction. Enterprise AI systems are not winning by sending every possible byte to the biggest model on every turn. They are becoming more deliberate about which model runs, which context enters the prompt, which context repeats, and which work can be reused.

Prompt layout becomes infrastructure design

The application still has work to do.

Prefix reuse depends on token order. Stable content should come first, and changing content should come later. If a timestamp, request ID, randomized instruction, or agent-specific preamble appears before the stable project context, it can break the prefix match.

For an executive, this sounds like prompt hygiene. For an infrastructure team, it is a scheduling and routing concern for expensive GPU work.

Good cache-friendly layouts put the durable pieces first:

• System and safety instructions.
• Tool schemas and MCP descriptions.
• Role and policy definitions.
• Long documents or examples that are reused.
• Project rules and repository summaries.

Then put the fast-changing suffix last:

• The current user question.
• The newest tool result.
• The latest test failure.
• The latest retrieved chunk that is unique to this request.

Local caching is not enough

Local prefix caches are effective when traffic is small, sticky, and fits in memory. Production workloads are rarely that clean.

Imagine a large coding-agent service. One developer asks Codex-style tooling to fix a bug. The agent reads files, runs tests, applies a patch, and loops. The stable prefix grows: project rules, tool definitions, original task, file context, and useful tool observations. At the same time, many other agents are running across the fleet.

If the next step lands on the same worker and the KV state is still in memory, the cache hits. If the request lands elsewhere, or if HBM pressure evicted the state, the model recomputes the prefix. At scale, that miss pattern becomes a cost and latency problem.

The same applies to RAG and support workloads. A 100-page policy document may be reused across thousands of requests, but if each replica keeps only its own local copy of the resulting KV state, the fleet still repeats work.

At that point, caching becomes a memory hierarchy problem.

It also becomes a control-boundary problem. Many enterprises will run a mix of frontier APIs, internally hosted open-source models, and domain-tuned models. Regulated deployments will care where data flows, where reusable context is stored, and which inference path is allowed to restore it. A local prefix cache inside one worker or one provider is useful, but it cannot be the only reuse strategy when the operating model spans private infrastructure, governed model routing, and data-residency requirements.

What the serving stack needs underneath

The serving runtime should continue to own prefix matching, scheduling, and model execution. That is what vLLM, SGLang, LMCache, and similar systems are built to do.

But once useful KV state leaves local GPU memory, the runtime needs a place to put it.

That place should be:

• Larger than HBM.
• Fast enough that restore beats recompute.
• Shared across workers.
• Reachable through the serving stack's KV transfer path.
• Observable through hit rate, restore latency, and P99 TTFT.

Shared context memory fits exactly there. It is not a replacement for prefix caching. It is the tier beneath it.

In one deployment, vLLM can identify reusable prefixes while a transfer layer moves KV blocks out of local memory and MemKV supplies the shared backend. In another, SGLang can use its runtime prefix/radix cache while HiCache tiers state through device, host, and external paths.

The names differ. The shape is the same.

• The runtime finds a reusable prefix.
• HBM keeps the hottest active KV blocks.
• Host memory handles near-term spill and staging.
• A shared context tier preserves reusable KV state across workers and nodes.
• A later cache hit restores KV state instead of triggering baseline recompute.

Why this is memory, not storage

The distinction is easy to miss because the bytes may eventually land on flash. But the design goal is not storage. The design goal is memory behavior at cluster scope.

Traditional enterprise storage is built around a full storage feature stack: object or file protocols, metadata services, access-control checks, durability machinery, data services, and compatibility with general-purpose applications. That is the right design for data, models, checkpoints, logs, and durable objects. It is usually a poor fit for the hot KV restore path unless the serving stack can bypass general-purpose storage semantics with a KV-aware transfer layer.

KV cache reuse is different. The serving stack is not asking for a file. It is asking for model state that was already computed and should be moved back into the inference path before recompute wins.

That is why MemKV is positioned as context memory rather than storage. The hot path is shaped around registered GPU memory, RDMA movement, raw NVMe access, large KV blocks, and minimal metadata work. The point is not to make a storage system faster. The point is to avoid sending repeated context back through the GPU prefill path.

That also explains why storage-style solutions must be evaluated carefully. They can provide capacity and durability, but protocol layers, metadata work, host copies, and general-purpose data services can add latency and jitter in the path that matters most. A cache hit that returns too slowly is economically the same as a miss because the GPU could have recomputed the prefix instead.

What operators should measure

Do not deploy a remote KV tier because caching sounds elegant. Deploy it where the workload proves reuse is valuable.

Measure:

• Cache hit rate by prefix or KV block.
• P99 TTFT with and without restored KV state.
• Context length distribution.
• KV cache bytes per session.
• HBM eviction rate.
• Cross-replica routing rate.
• Restore bandwidth and latency.
• The fraction of requests with stable long prefixes.
• Whether restore remains faster than baseline recompute under load.
• Whether the data path avoids storage-protocol overhead and host copies.

The strongest workloads usually show up quickly once measured: coding agents, repeated system prompts, long RAG documents, support-policy assistants, batch evaluations, and agent swarms. The weakest workloads are mostly unique short prompts.

Continue reading

Part 2 coverd the architecture. Part 1 set up the business case, Part 3 turns the architecture into an evaluation framework.

• Part 1: Prompt Caching Is an AI Margin Lever, Not a Model Trick. Why repeated prefill is an operating-margin issue rather than a model-quality issue, and what that costs at fleet scale.
• Part 3: From Cache Hits to Production SLAs. A buyer and operator scorecard for proving that shared context memory improves tail latency, concurrency, and cost per useful token.

‍