From Cache Hits to Production SLAs | Part 3 of 3

‍Part 3 of a three-part series on prompt caching, prefix caching, and shared context memory for AI inference.

Part 1 explained why prompt caching matters to the business. Part 2 explained why repeated context becomes a memory hierarchy problem once GPU memory, routing, eviction, and concurrency enter the picture.

The final post asks the operating question: how do you prove that shared context memory is improving production inference?

MinIO has already introduced MemKV publicly, so this post focuses on evaluation: what a buyer, architect, or platform team should prove in production. The product position is clear: MemKV is a purpose-built context memory store for AI inference, built to place reusable KV state in a shared tier rather than forcing GPUs to recompute it.

The outcome is not cache hits by themselves. The outcome is lower P99 TTFT, higher effective concurrency, better GPU economics, and a cleaner path to long-context AI applications.

The broader enterprise context matters here. Ramp Economics Lab now tracks AI spend intensity across subscriptions, coding agents, and token/API usage, and its recent analysis says advanced AI usage increasingly means multiple frontier models, platforms with open-source model access, and vertical AI tools. Uber's public Gen AI Gateway writeup shows the same pattern in infrastructure form: one governed path across external LLMs and in-house hosted open-source LLMs. DoorDash's Assistant writeup shows role-based model selection with provider fallback, while its LLM personalization writeup shows the prompt-side version: trim context to what the use case actually needs, because irrelevant context costs money and can reduce output quality.

Those examples make the evaluation stricter. A shared context-memory tier should not be judged only on whether it stores KV blocks. It should be judged on whether it helps the enterprise reduce repeated-token pressure, preserve control of reusable context, and improve user-visible latency under the same governance boundaries that apply to model routing and data residency.

The central evaluation point is that MemKV should be judged as memory infrastructure, not as another storage target. The question is not whether it can hold bytes. The question is whether it can return reusable KV state fast enough to prevent repeated prefill.

Start with the baseline

Before evaluating any shared context tier, capture the baseline honestly.

The baseline is not average latency. The production pain usually lives in the tail. Long prompts, evicted KV state, cross-replica routing, and bursty tenant traffic show up as P99 TTFT and SLA volatility.

A useful baseline includes:

• P50, P95, and P99 TTFT by context length.
• P95/P99 TPOT or inter-token latency and end-to-end request latency by output length.
• Input token distribution.
• Prefix reuse rate by workload.
• Cache hit rate inside the serving engine.
• HBM eviction rate.
• Cross-replica routing rate.
• GPU time spent in prefill versus decode.
• Cost per useful token or useful agent step.

Those measurements separate two very different problems. If prompts are short and mostly unique, shared context memory will not change much. If prompts are long, stable, and repeatedly evicted, the system is paying a recompute tax that can be engineered out.

Use public MemKV numbers as proof points, not assumptions

In MinIO's public Llama 3.1 70B, 64K-context benchmark at production concurrency, TTFT moved from 53 seconds without MemKV to 703 milliseconds with MemKV. The same launch post also describes the larger pattern: context is discarded when local GPU memory cannot hold it, and the next GPU starts over from the beginning.

That is the right kind of evidence, but it should not be copied blindly into a business case. Treat it as a proof point to reproduce on your workload, not as a guaranteed business-case input.

The question for an enterprise team is:

• Do our applications have long repeated prefixes?
• Are those prefixes stable enough to hit?
• Are misses caused by local memory pressure or routing?
• Does restore beat baseline recompute at our target concurrency?
• Does P99 TTFT improve in a way users and SLAs can feel?

If the answer is yes, prompt caching has moved from a prompt-engineering trick to an infrastructure investment.

Connect cache mechanics to business outcomes

The platform team may measure KV block hits, restore latency, and transfer bandwidth. The C-suite will ask what changed for the business.

Both views are valid. They just need to connect.

At the application layer, the signal is repeated context: documents, tool schemas, policy bundles, codebase summaries, and multi-turn agent history.

At the runtime layer, the signal is prefix reuse: identical token beginnings that the serving engine can map to cached KV state.

At the memory layer, the signal is avoided recompute: reusable KV state remains available after local GPU memory would otherwise evict it.

At the business layer, the signal is improved unit economics: lower P99 TTFT, more concurrent sessions per GPU fleet, fewer overprovisioned GPUs for the same SLA, and lower cost per useful output.

That is the measurement chain. If one link is missing, the deployment becomes a technology demo instead of an operating improvement.

Where MemKV fits in the evaluation

MemKV should be evaluated as a shared context-memory tier below the serving runtime. It does not replace prefix matching, scheduling, batching, or model execution. Those remain the job of the inference stack.

MemKV's role is narrower and more important: preserve reusable KV state in a fast shared tier so that later requests restore context instead of asking GPUs to rebuild it.

The public product page describes MemKV as petascale, native flash-based context memory accessed over 800 GbE RDMA. In simpler terms, MemKV is positioned as a flash-backed context-memory tier that sits below GPU memory and is reached over a high-speed RDMA network. The public press release positions it as a way to provide persistent, shared context across GPU clusters where GPU-local memory is too small and general-purpose storage is the wrong hot path.

For an evaluator, those claims translate into concrete tests:

• Can the serving stack restore KV state faster than baseline recompute?
• Does P99 TTFT remain within SLA, or degrade materially less than baseline, as concurrency increases?
• Does cache locality survive routing across workers?
• Does the shared tier avoid per-GPU cache fragmentation?
• Can GPU compute and context memory scale independently?
• Does the operational model fit the production network and security design?
• Does the path behave like memory under pressure rather than like a general-purpose storage read?

Procurement and architecture converge at that point. The purchase is not just capacity. It is an attempt to convert repeated prompt work into reusable infrastructure.

Why traditional storage is the wrong comparison

Traditional storage is excellent at the jobs it was built for: durable objects, files, checkpoints, model artifacts, logs, governance, replication, and broad data sharing. Those jobs need metadata, policy, compatibility, and durability.

KV cache restore has a different requirement. It is a latency-sensitive memory operation in the inference hot path. The serving stack already knows which KV blocks it wants. The value comes from moving those blocks back into the model path before the GPU repeats prefill.

That is why a storage-first answer can look good on a capacity slide and still fail the production test. If a design routes cache hits through object, file, SAN, or NAS semantics without a KV-aware fast path, the system can inherit protocol overhead, host copies, small-block assumptions, and jitter that were not designed for GPU context memory.

The effectiveness test is simple:

• Faster P99 TTFT, not just higher throughput.
• More useful GPU utilization, not just busy GPUs.
• Lower cost per useful token or agent step, not just cheaper bytes.
• Stable restore latency under concurrency, not just a benchmark at low load.
• Independent scaling of compute and context memory, not per-GPU cache fragmentation.

That is the architectural difference. MemKV is designed to make repeated context behave like shared memory. Traditional storage is designed to make data durable and broadly accessible. Both matter in a production AI stack, but they belong in different tiers.

What this means at cluster scale

Large multi-node GPU clusters make the same problem harder to ignore.

At that scale, the question is not whether one request can be made faster. It is whether the architecture can preserve P99 latency while schedulers route work across many GPUs, tenants burst, workers restart, and long-context agents branch across many steps.

Without shared context memory, adding GPUs can create more places to lose cache locality. The fleet gets larger, but reusable state remains trapped in local memory or disappears under eviction pressure.

With shared context memory, the objective changes. The platform can treat KV state as a schedulable resource: keep the hottest state in HBM, spill valuable state into a larger tier, and restore it when the next cache hit arrives. The same evaluation framework applies at cluster scale. The business issue is the same as in Part 1: do not buy GPUs to repeat work the system has already done. The engineering issue is the same as in Part 2: repeated context becomes a memory hierarchy problem.

The buyer checklist

A serious evaluation should end with a scorecard, not a slogan.

Ask for:

• A workload profile showing repeated-context opportunities.
• A baseline with P99 TTFT and concurrency limits.
• A MemKV run with the same traffic shape.
• Hit-rate and restore-latency evidence from the serving stack.
• GPU prefill versus decode time before and after.
• Cost per useful token or useful agent step.
• A comparison against storage-style offload that includes P99 TTFT and jitter, not only throughput or capacity.
• Failure-mode behavior when workers restart or routing changes.
• A security and data-path review for KV movement.
• Evidence that reusable context stays inside the required enterprise or regulated boundary.
• A model-routing view that shows how cache behavior changes across frontier APIs, internally hosted models, and domain-tuned models.
• Token-pressure evidence: how much repeated prefill, repeated context, and avoidable token/API spend the design removes.

The most important line item is not the biggest speedup number. It is whether the system preserves user-visible latency and useful GPU work as concurrency increases.

The series takeaway

Prompt caching is the first sign of the opportunity. Prefix caching is the runtime mechanism that finds reuse. Shared context memory is how reuse becomes fleet-level infrastructure.

Part 3 closes the series. Parts 1 and 2 build the business case and the architecture behind it.

• Part 1: Prompt Caching Is an AI Margin Lever, Not a Model Trick. Why repeated prefill is an operating-margin issue rather than a model-quality issue, and what that costs at fleet scale.
• Part 2: When Repeated Context Becomes an Infrastructure Problem. How repeated context becomes a memory hierarchy problem once GPU memory, routing, eviction, and concurrency enter the picture.