Prompt Caching: Stop Paying GPUs to Read the Same Prompt Twice

Graphic of a document with three horizontal blue lines above a large blue paragraph symbol on a light background.

Published:

August 27, 2026

Prompt caching is becoming one of the most important efficiency levers in AI inference. Here is what it is, why it matters, and how MemKV turns it from a per-process optimization into shared infrastructure.

Modern AI applications repeat themselves more than people realize.

Every request to a coding assistant carries tool schemas, system instructions, repository context, style guides, security rules, and conversation history. Every enterprise chatbot carries policy documents, product catalogs, role definitions, guardrails, and retrieval results. Every agent loop carries the same operating instructions over and over while only the latest task step changes.

That repeated context is expensive. Not because text is expensive to store, but because the GPU must transform those tokens into attention state before the model can produce the first output token. That first phase is called prefill. When the same prompt prefix appears again, recomputing it from scratch is wasted GPU work.

Prompt caching is the fix: compute the reusable prefix once, keep the resulting KV cache, and reuse it when another request starts with the same prefix.

The business value is straightforward. Lower time to first token. Lower input token cost where provider pricing exposes cached tokens. Higher useful GPU work per dollar. Better P99 latency under concurrency. The technical value is just as clear: fewer repeated prefills, less pressure on scarce HBM, and a better way to serve long-context workloads.

The catch is that prompt caching only becomes a production advantage when the cache survives routing, eviction, concurrency, and scale. That is where MemKV comes in.

None of this is a theoretical optimization. Enterprises are already learning that AI cost is no longer a simple seat-based SaaS line item. Ramp Economics Lab now tracks AI spend intensity across subscriptions, coding agents, and token/API usage, and its recent analysis says average monthly token spend rose 13x from January 2025 to January 2026. Uber has publicly described a Gen AI Gateway that governs both external LLMs and in-house hosted open-source LLMs. DoorDash has described role-based model routing with provider fallback, hybrid internal/external model use, and trimming memory blocks before LLM calls to control token cost and remove irrelevant context. The common theme is clear: production AI needs model routing, context discipline, and reuse.

The Short Version

Prompt caching stores the intermediate attention state for a prompt prefix, not the final answer.

In transformer models, the model creates key and value vectors for every token it has processed. These K and V vectors are stored in the KV cache so future tokens can attend to prior context without recomputing every previous token. During normal generation, this is already how decoding works. Prompt caching extends that idea across requests: if a new request has the same beginning as a previous request, the serving engine can reuse the KV cache for that shared prefix and skip the matching prefill work.

Think of it this way:

The first request pays the full prefill cost.
The system stores the KV cache for the stable prefix.
A later request arrives with the same prefix.
The model restores the cached KV state.
The GPU only computes the new or changed portion before decoding.

The output is still generated by the model. Prompt caching is not returning an old answer. It is skipping duplicate setup work.

What Prompt Caching Is Not

Prompt caching is often confused with other forms of caching. That confusion matters because the operational expectations are different.

It is not a response cache. A response cache stores the final answer for an identical request. That is useful for deterministic APIs and static questions, but it is risky for open-ended AI interactions. Prompt caching does not reuse the answer. It reuses the model state that came from the shared prefix.

It is not a vector database. RAG systems retrieve relevant documents and inject them into the prompt. Prompt caching can make repeated RAG contexts faster, but it does not decide which documents to retrieve.

It is not application memory. A chatbot may remember facts about a user, but prompt caching is lower level. It stores the model's attention state after a specific token prefix has been processed.

It is not magic. Cache hits require reuse. Mostly unique prompts, short prompts, or prompts that change near the beginning will not benefit much.

Why Prompt Caching Matters Now

Prompt caching has existed as a serving optimization for years, but it has become much more important because AI application patterns changed.

Long Prompts Became Normal

The early chatbot pattern was a short user question and a short answer. Modern AI applications look different.

An enterprise assistant may include a multi-page system prompt, tool definitions, compliance instructions, customer history, and retrieved documents before the user's actual question appears. A coding assistant may carry a repository map, relevant files, previous tool outputs, and patch instructions. An agent may loop through the same task plan and tool schema dozens of times.

The repeated prefix can be thousands, tens of thousands, or hundreds of thousands of tokens.

That is exactly the part of the request prompt caching can reuse.

Time to First Token Became a User Experience Metric

Users feel the first-token wait more than they feel the average throughput of the cluster. That first-token wait is usually reported as TTFT: time to first token.

Long prompts make TTFT prefill-heavy. Before the model can stream the first output token, it must process the input context. If the input context is large and repeated, paying that cost on every request is the wrong default.

The production-relevant version of this metric is P99 TTFT. Averages hide tail latency. P99 is what users feel when the system is under concurrency and the GPU queue is full of long prefills.

HBM Is Fast, Expensive, and Small

GPU HBM is the right place for active computation. It is the wrong place to keep every reusable context forever.

A 64K token prompt for a large model can create tens of GiB of KV cache per session. Multiply that by hundreds of concurrent sessions and the working set quickly exceeds what a GPU node can hold. When the cache does not fit, the system has three choices:

Keep the session pinned to a GPU and reduce scheduling flexibility.
Evict the KV cache and recompute it later.
Move the KV cache to another tier and restore it when needed.

The first option hurts utilization. The second option wastes GPU cycles. The third option is the path prompt caching needs at scale.

The Practical Shape of a Cache-Friendly Prompt

The easiest way to improve cache hit rate is to put stable content first and variable content last.

That means system instructions, tool definitions, safety policies, examples, and long documents should appear before the user's latest question. If a timestamp, request ID, random nonce, or user-specific value is inserted near the front, it can invalidate the shared prefix for every request after it.

For teams building AI applications, prompt caching turns prompt layout into an infrastructure decision. Good prompt hygiene now affects cost, latency, and GPU capacity.

Useful patterns include:

Put stable system prompts and tool schemas at the beginning.
Keep tool ordering deterministic.
Keep retrieved documents in a consistent order when they are reused.
Avoid timestamps and unique request metadata in the cacheable prefix.
Separate slow-changing policy text from fast-changing user text.
Measure cached tokens, cache hit rate, and P99 TTFT together.

That is why the major AI API providers now expose prompt or context caching as a first-class feature, and why open source serving frameworks are investing in automatic prefix caching and hierarchical KV cache systems. The industry is making the same architectural bet: repeated context should be reused, not recomputed.

Where Simple Prompt Caching Breaks Down

Prompt caching is easy to describe on one GPU. It is harder to operate across a fleet.

In a single-process setup, the engine can keep a prefix cache in GPU memory or host memory. If the next request lands on the same worker before the cache is evicted, it gets a hit. That works for small deployments and short reuse windows.

Production serving creates more pressure:

Requests are load balanced across many workers.
Prefixes are evicted from HBM when concurrency rises.
Long-context sessions produce KV caches larger than local memory budgets.
Restarted workers lose in-process cache metadata.
Multi-turn agents may need the same context on different replicas.
Disaggregated prefill and decode architectures need fast KV movement.

At that point, prompt caching stops being only a model-serving feature. It becomes a memory hierarchy problem.

If reusable KV state is valuable, the infrastructure needs a place to put it. That place must be larger than HBM, faster than ordinary storage, shared across workers, and integrated with the serving engines that already manage KV blocks.

That is the problem MemKV is built to solve.

How MemKV Enhances Prompt Caching

MemKV is a high-performance inference context memory store. It sits between local GPU memory and durable storage, giving inference systems a shared tier for reusable KV state.

The point is not to replace prompt caching in vLLM, SGLang, LMCache, or other serving layers. The point is to make those caching systems useful beyond one process, one replica, or one HBM budget.

1. MemKV Makes the Cache Shared

Local prefix caches are limited by where the request lands. If the next request lands on a different worker, the local cache may not help.

MemKV provides a shared backing tier. Multiple serving workers can point at the same MemKV cluster. A prefix stored by one worker can be retrieved by another, depending on the framework integration and cache key shape.

That changes the economics of caching. A long shared prompt is no longer only a local optimization. It becomes shared infrastructure.

2. MemKV Expands the Active Cache Beyond HBM

HBM should serve the hot, active decode path. MemKV gives the system somewhere to place reusable KV state after it leaves HBM instead of dropping it.

The benefit is largest for long-context and high-concurrency workloads. The working set is often much larger than GPU memory. Without an external tier, eviction turns into recomputation. With MemKV, evicted KV blocks can remain available for later reuse.

3. MemKV Is Built for Inference Block Movement

General-purpose storage is built for durable data services. Prompt cache state is different. It is large, hot, ephemeral, and accessed in throughput-oriented KV blocks.

MemKV's data path is designed for that pattern:

RDMA transport for the high-performance path.
A first-class TCP path for environments where RDMA is not reachable.
Extent-based storage for parallel I/O across NVMe drives.
Shared-nothing horizontal scaling across MemKV servers.
HMAC-authenticated wire messages.

The goal is simple: restoring reusable KV state must be fast enough that the GPU does less repeated prefill, not more waiting on storage.

4. MemKV Fits the Frameworks Operators Already Use

MemKV integrates with the model-serving ecosystem rather than asking operators to build a new one.

In vLLM deployments, MemKV can sit behind LMCache as a shareable storage tier. In SGLang, MemKV can back HiCache's hierarchical cache through a dynamic storage backend.

The serving framework still owns scheduling, prefix matching, and model execution. MemKV provides the fast shared context-memory tier those systems can use when local memory is not enough.

5. MemKV Turns Cache Hits Into Business Results

The benchmark that matters is not raw cache capacity. It is user-visible latency under realistic concurrency.

The public MinIO MemKV launch blog reports a 64K-context benchmark where a repeated-context workload moved from baseline recompute to MemKV-restored context:

Baseline TTFT: 53 seconds
MemKV TTFT: 703 milliseconds
Result: the first token arrives in less than a second when reusable context can be restored instead of rebuilt

Those are public launch numbers for a high-reuse case. They do not mean every prompt gets faster. They mean that when the workload has reusable context, the cost of rebuilding that context can be moved out of the user path. Production teams should still validate P99 TTFT on their own traffic.

Business Benefits

Prompt caching has a technical implementation, but the reason it matters is business impact.

Lower Latency for Repeated Workloads

The first token is where users decide whether the system feels responsive. By skipping repeated prefill, prompt caching can reduce TTFT for long shared prefixes. MemKV extends that benefit across workers and memory tiers.

More Useful Work Per GPU

A busy GPU is not always a productive GPU. If it is recomputing a prefix it already processed, the utilization graph looks healthy while the business is paying for duplicate work.

Prompt caching turns repeated context into reusable state. MemKV gives that state a shared place to live.

Better Long-Context Economics

Long context windows are becoming a product feature. Customers want assistants that can reason over codebases, contracts, policies, tickets, transcripts, and conversation history. The larger the context, the more painful repeated prefill becomes.

MemKV is most valuable where the same long context appears repeatedly: RAG over large documents, coding assistants, support agents, batch evaluation, and multi-turn workflows.

More Scheduling Flexibility

Without a shared context tier, teams often try to keep sessions on the same GPU to preserve local cache hits. That reduces the scheduler's freedom. Shared context memory lets the system recover more of the benefit without hard-pinning every session to one worker.

Lower Infrastructure Waste

Every avoided prefill is GPU time, power, and queue capacity returned to the business. For private AI and neocloud operators, that can become a margin lever: more paid tokens served per rack, per watt, and per GPU-hour.

A Simple Evaluation Checklist

Before investing in prompt caching infrastructure, measure the workload. The best candidates are obvious once the data is visible.

Ask these questions:

How much of the prompt is stable across requests?
How long is the stable prefix in tokens?
What is the cache hit rate at the prefix or block level?
What is P99 TTFT with and without cache hits?
How much HBM is consumed by KV state at target concurrency?
How often are prefixes evicted and later requested again?
Do requests route across replicas that cannot share local cache?
Does the application put variable content before stable content?

The answer should not be "cache everything." The answer should be "cache the prefixes that are expensive, reused, and likely to be evicted from local memory."

What This Means for AI Infrastructure

Prompt caching is a sign that AI inference is maturing.

Early AI infrastructure discussions focused on model weights and GPU count. That is no longer enough. Long-context inference creates large amounts of intermediate state. That state is expensive to recompute, too large to keep entirely in HBM, and too valuable to throw away whenever a request moves to a different worker.

The winning architecture will look like a memory hierarchy:

HBM for active compute.
Host memory for near-term spill and staging.
MemKV for shared active context memory.
Durable object storage for model weights, datasets, logs, and long-lived enterprise data.

None of that replaces RAG, vector databases, model serving frameworks, or object storage. It is the missing tier for reusable inference context.

Conclusion

Prompt caching is simple in concept: do not make the GPU process the same prompt prefix twice when it can reuse the KV state it already computed.

At small scale, that is a useful latency optimization. At production scale, it becomes an infrastructure requirement. Long-context applications create KV state faster than HBM can hold it. Load balancers move requests away from the workers that still have local cache. Concurrency turns occasional recompute into P99 latency.

MemKV makes prompt caching practical beyond the local worker. It gives inference systems a fast, shared, NVMe-backed context-memory tier that can be used by modern serving stacks through LMCache, SGLang HiCache, and the MemKV client path.

The result is not just faster storage. It is less repeated GPU work, lower P99 TTFT, better long-context economics, and a cleaner architecture for AI inference at scale.

As always, if you have questions, join the MinIO Slack channel or drop us a note at hello@min.io.