MinIO Doubles the Sessions Your GPUs Hold, Measured the Way You'd Actually Run It

How we built the KV cache benchmark around real session behavior, and the effective GPU utilization you recover once evicted context has somewhere to go.

If you've been looking at what to do when KV cache outgrows GPU memory, you've seen the time-to-first-token claims, anywhere from sixteen to seventy times and beyond. Anyone with a career in IT infrastructure asks two questions. Seventy times greater than what? And what were users doing while that was measured?

The answers are in the fine print. Those numbers compare a context served from storage against rebuilding it from scratch, on configurations that zero the GPU and DRAM cache tiers so every retrieval counts for storage, even ones your GPUs would have served for free. One published setup gave the GPU room for nearly nine times its largest prompt, then forbade its use. And there were no users. Requests came from a load generator on a fixed schedule, not from people who pause between turns. We hold numbers in that shape too. They measure a component, not a fleet.

Real sessions work differently. A coding agent accumulates context it can never discard, then goes quiet while someone reads the answer, opens a file, waits for a build. Under load, the engine evicts idle sessions to serve active ones, and the GPUs rebuild a context they finished minutes earlier.

So the real constraint is capacity, meaning the total amount of context you can keep on hand instead of rebuilding it. GPU memory holds only a fraction of what an agentic fleet accumulates, and both throughput and tail latency suffer once it fills. Give that context somewhere to live and both recover. What follows is how we measured it, and by how much.

We Built the Test Around the Way People Actually Work

How long a session sits idle between turns decides what a benchmark actually measures. The workload comes from the NVIDIA AIPerf Agentic Code dataset generator, seeded at 42 for a thousand sessions and 15,357 turns, and its delays arrive bimodal, mostly rapid gaps of a few seconds with a minority of human-scale pauses.

We held every gap to human pace instead. All 14,357 inter-turn delays became a pause drawn between 30 and 60 seconds, seed and transformation published so the file reproduces. Every turn beyond a session's first now models a session left idle while its user is occupied elsewhere, then picked back up.

The two distributions measure different things. A three-second gap returns to context still resident in GPU memory, so it measures a cache hit. A forty-second gap under load returns to context already evicted, so it measures a recompute. Leaving the rapid gaps in would have let a substantial share of returning turns never reach the tier.

Every turn is then classified and reported on its own. A session-start is a cold prefill no cache can accelerate. A human-return is the traffic a tier can address. Folding both into one percentile would bury the class that matters inside requests the tier could never influence.

Holding every gap at human pace makes the workload harder than the trace it came from, which is the most consequential decision in the study. Every returning turn now measures the cost a context tier exists to remove.

Everything Else Stayed the Way an Operator Would Set It

Pacing the trace only gets you halfway to emulating a real-world scenario. Everything else had to match what an operator would run, including settings that work against us.

We tested three configurations: the engine alone, the engine with a host DRAM cache and nothing behind it, and the engine with a host DRAM pool backed by MemKV. All three replay the same trace, and only the offload connector and the tier behind it differ. Each climbs the same concurrency ladder in 12-minute windows.

The engine's own cache of already-computed context stays on in all three, because no operator disables it in production. Our tier gets no credit for reuse; the GPUs already serve free, which is why the ratios here are smaller than the ones you have been reading.

Replay is closed-loop, so a session's next turn goes out only after its answer returns and its pause elapses. Storage is deliberately not node-local, with two commodity NVMe nodes serving the whole inference node across the network, the way a real fleet would share a tier.

What That Bought on the Same Hardware

Past the point where sessions outgrow GPU memory, the gap opens fast. On Llama-3.1-8B at 256 concurrent sessions, NVIDIA H200s with MemKV behind them serve 2.4× the tokens per second per GPU against the same engine running no tier at all, and complete 2.59× as many turns in the same window, with non-overlapping confidence intervals. What changed is where the GPU cycles go. Without a context tier, they are spent on prefill, the pass that rebuilds a session's whole 124k-token context before a single new word can be produced. With one, they are spent on decode, the part that actually produces output. The tiered curve keeps climbing as sessions pile on, where the baseline flattens.

Tail latency improves alongside throughput rather than at its expense. On Qwen3-32B at 128 sessions, the slowest tenth of returning turns waited 51.7 seconds for a first token without the tier and 11.5 seconds with it, a 4.5× improvement. Throughput rose 26% in that same window and completed turns rose 1.35×. Three metrics moving together in a single cell rules out the usual explanation for a latency win, which is that the system was quietly doing less work.

The pattern repeats on 8 of the 10 models measured, from an 8B dense model up to a 405B one, and reaches 196% more throughput and 3.42× the completed work on Llama-3.1-70B at 64 sessions. Each of those figures carries its sample count, a confidence interval, and a control confirming both configurations did the same amount of work, across 198 measured windows.

‍Where DRAM Runs Out, MemKV Picks Up

Any capacity planner asks the same question next. Would a large host cache alone be enough? A pinned DRAM cache with no persistent tier behind it clears the first step of eviction, improving returning-turn tail latency 2.6×. One rung higher its capacity runs out, and by the top of the ladder it has converged back toward the baseline it beat. A cache and a hierarchy are different elements.

What sits behind it has to handle traffic most storage is not built for. A single offloaded block weighs roughly half a gigabyte, which makes a context tier a bandwidth problem rather than an operations-per-second one. MemKV's speed comes from what is absent: no filesystem, no object API, no metadata service in the data path. A key resolves to a block, and the block moves as raw NVMe over RoCEv2.

The tier sustains 45.5 GB/s of stores while the busiest engine we run fed it 16.4 GB/s, three to four times the headroom vLLM can produce. KV moves straight between NVMe and GPU memory at line rate on InfiniBand and RoCEv2 alike, restoring a 120 GB context in about 1.6 seconds.

What This Means for Your Fleet

Twice the concurrent sessions on the GPUs you already own. Throughput per GPU climbs as you add sessions, reaches a peak, then falls away as more of each GPU's time goes to rebuilding context instead of serving it. A context tier moves that peak out to twice the session count, so the same hardware carries roughly twice the load before performance starts to degrade. On the Llama-3.1-70B grid the gain holds in all 25 cells, improving tail latency between 2.6× and 19.6×, and the storage tier was never the bottleneck in any of it.

MemKV runs on commodity NVMe with no filesystem in the data path, over native InfiniBand or RoCE v2, and attaches to current stable vLLM through the upstream offloading API with a pip install and no forked engine.

A dedicated tier for KV cache is becoming a standard part of the inference stack. NVIDIA STX is a modular foundation for rack-scale AI storage platforms. NVIDIA CMX, built on NVIDIA STX, establishes an Ethernet-attached flash tier dedicated to KV cache, sitting between the memory inside a GPU node and the shared storage behind it. NVIDIA DOCA Memos opens that layer to storage partners through published interfaces. MemKV is an Ethernet-attached KV cache on raw NVMe, built to the same shape.

If your GPUs are full, or trending that way as agentic sessions get longer, you are paying for the same context twice, once to build it and again every time a user comes back. MemKV keeps that context on commodity NVMe and hands it back instead of making the GPUs rebuild it. The second bill never arrives.

Read the full benchmark, including the side-by-side tables, the topology, and the reproduction appendix.

‍