The Complete GPU Storage Stack: AIStor + MemKV Architecture Guide

Every inference request builds KV cache in GPU memory, and GPU memory runs out. Under production concurrency and long context, HBM fills, context gets evicted, and the next request that needs it rebuilds it from scratch. The GPU spends cycles recomputing work it already did instead of generating tokens. The pattern is the recompute tax.

The data path decides how much of it you pay. Datasets, models, and checkpoints live in the object layer, which was not built to serve context at inference speed. Local NVMe offload is fast and strands context on the node that wrote it. The gap is a shared context tier between the persistent layer and GPU HBM.

AI data gravity is consolidating on objects, which makes the persistent object tier the place to start. For background on the recompute tax, see the MemKV launch blog at MemKV launch blog.

The Complete GPU Storage Stack

Modern AI infrastructure is not fast storage placed next to GPUs. It is a purpose-built data path where every layer matches the way GPUs actually consume data: large, throughput-oriented blocks at microsecond latency, without host CPU mediation. The architecture defines three distinct layers, so durability never trades against latency and governance never trades against speed. MinIO delivers the first two. The third is the accelerated infrastructure they run on.

Architecture at a glance:

Layer 1: AIStor, the Data Foundation

AIStor is the AI data foundation the rest of the stack rests on. It brings tables, files, and objects together in a modern lakehouse under one governance model, so analytics, AI applications, model training, and inference all work from a single source of truth. No silos, no copies, and no migrations between systems each built for one workload.

Everything with a lifecycle measured in months or years lives here: training corpora, model weights, checkpoints, embeddings, documents, media, and the structured data managed as Apache Iceberg tables through AIStor Tables. Applications reach it three ways, over S3 for objects, the Iceberg catalog interface for tables, and file access for file-based tools and users. Each is a native interface into the same namespace rather than a gateway layered on top, which is why adding a workload does not mean adding a system.

AI data gravity runs toward that foundation, and the pull keeps getting stronger. PyTorch and TensorFlow data loaders, Spark, Ray, Iceberg engines, and agent runtimes all read S3 natively, so pipelines reach the data where it already sits instead of staging a copy somewhere else. Petabytes are expensive to move once they land, which means the foundation holding the data becomes the foundation new workloads get built against. Access methods come to the data. The data does not move to the access method.

Persistence is the point, because AI is not purely ephemeral. Training data must be versioned, governed, and shared across teams, and checkpoints and model artifacts must survive failures and remain reproducible under audit. AIStor governs once, applying access control, encryption, lifecycle, and audit policy uniformly across objects, tables, and files rather than per system. It is software-defined and scales out by adding nodes, protects data with erasure coding across drives and nodes, and integrates natively with Kubernetes and the AI ecosystem, which keeps code changes and operational friction low. AIStor is a validated component in enterprise AI reference architectures rather than an afterthought.

Layer 2: MemKV, the Context Memory Tier

MemKV is MinIO's purpose-built context memory store for AI inference, built to improve both time to first token and time per output token. Conventional inference keeps KV cache in per-GPU HBM, and a single conversation or multi-step agent flow builds tens of gigabytes of it. HBM is scarce and expensive, which forces a hard tradeoff: hold context in memory and starve the model, or evict it and pay the recompute penalty on the next request. Neither is acceptable at scale.

MemKV removes the tradeoff by placing a petascale, flash-backed KV pool at the correct layer of the GPU memory hierarchy. The layer has a name, G3.5, and MemKV was built from the ground up for it rather than adapted into it. It sits between GPU HBM and the persistent data foundation instead of arriving as an appliance bolted onto infrastructure that already existed. Every inference node draws from the same pool, which turns per-GPU recomputation from a recurring cost into a cache hit.

MemKV is engineered for GPU access patterns rather than legacy ones. KV cache moves from GPU memory to NVMe over end-to-end RDMA, bypassing file systems and object protocols entirely, with no CPU in the data path and no protocol translation overhead. It operates in 2 to 16 MB blocks sized for throughput-oriented GPU access, not the 4 KB blocks that legacy workloads assume. It runs as a single ARM64-native binary in the flash tier, which keeps the data path lean.

Because the pool is flash-backed rather than HBM, GPU compute and context memory scale independently. Capacity gets added without provisioning more GPU nodes, and GPUs get added without over-provisioning memory to absorb context. Petabytes of context memory provision at commodity NVMe economics rather than at the cost of adding GPUs to mask recomputation. MemKV is not a general-purpose cache bolted onto a platform built for durable file or object data. It is a flash-native context memory tier built for one place in the stack, one access pattern, and one job.

Layer 3: The Accelerated Data Fabric

The first two layers only pay off if the path between them is fast and protected. Layer 3 is that path. It is accelerated infrastructure you deploy rather than software MinIO ships, and it does two jobs: moving transport and protocol handling off host CPUs, and encrypting and authenticating data inline in hardware. Model weights and inference context are valuable enough that protection belongs inside the path, not bolted around it.

MinIO's part is software built to run inside that path instead of on top of it. AIStor and MemKV ship as single static binaries with no external metadata databases and no background services, light enough to sit in the fabric rather than beside it. Erasure coding and checksums offload to hardware, host CPUs stay free for orchestration and inference logic, and the RDMA transport MemKV depends on stays clear. MinIO is working to extend that hardware-rooted enforcement up into the S3 object layer, where enterprise AI workloads actually run.

Data Flow: From Training Data Through Inference Context Delivery

The three layers form one pipeline. What moves through it changes as work shifts from training to serving.

• Training. Datasets stream from the data foundation to GPU nodes as objects and Iceberg tables over S3. Checkpoints, artifacts, and metrics write back, where erasure coding, versioning, and immutability policies protect lineage.
• Deployment. Model weights and tokenizer assets load from the data foundation into GPU memory at initialization.
• Inference. As requests progress and HBM fills, KV cache evicts. MemKV captures those evictions as blocks on NVMe in a shared namespace, then streams them back into GPU memory over RDMA whenever any GPU in the cluster needs that context again.

The fabric underlies all three, carrying every transfer with inline encryption and authentication and no host CPU in the hot path. The payoff lands in the last phase: the GPU decodes new tokens instead of recomputing old ones, and long-context sessions that would otherwise exhaust HBM keep serving.

Object vs. Parallel File Systems vs. Direct-to-GPU Data Paths

Selecting storage for GPU workloads is not binary. Training, evaluation, and inference exhibit different I/O sizes, locality, and latency sensitivities. The right stack often includes multiple tiers tuned to each phase.

Where Filesystem and Small-File Access Come Up

Some pipelines carry access patterns that object semantics do not cover. Each has a resolution inside the stack.

• Filesystem semantics. Training and preprocessing code written against a filesystem expects hierarchical directories, in-place byte-range writes, atomic rename, byte-range locking, and mmap. Object semantics provide none of those, and refactoring around them is expensive. AIStor native filesystem access presents those semantics against the same buckets S3 and Iceberg already address, so one deployment serves both without a second namespace or a second governance model.
• The small-file problem. Data loaders that shuffle across millions of small files each epoch are bound by metadata operations rather than bandwidth, and a stalled loader starves the GPUs behind it. The fix belongs at the data layer: sharded formats such as Parquet, TFRecord, and WebDataset convert random small reads into large sequential ones, which raises throughput whether the data sits on local disk or in the object layer, and keeps the loader ahead of demand.
• Direct-to-GPU loads. Routing training data through host DRAM costs a bounce buffer copy and CPU cycles on every read, and the page cache adds latency that is hard to predict. AIStor supports a direct RDMA path from NVMe into GPU memory for S3 reads which takes the host out of that path.
• Inference context. Neither a parallel file system nor a filesystem layered on an object store provides a shared context tier. Context spilled to local disk stays stranded on the node that wrote it, so a session that reroutes pays full prefill again. MemKV is the tier built for that job.

Running a parallel file system alongside the data foundation means two systems, two access control models, and two failure domains, and it still leaves inference context unsolved.

Matching Tiers to Workloads

Each phase of the lifecycle asks something different of the stack. The mapping is short.

• Training and analytics. Sustained throughput against large objects and columnar tables. Both read from the data foundation over S3 and Iceberg, and both scale by adding nodes rather than adding systems.
• Inference. Microsecond access to context shared across every GPU in the cluster. MemKV is the tier built for that pattern, and it is the difference between a GPU generating tokens and a GPU rebuilding work it already did.
• Filesystem-bound pipelines. The same backbone, reached through native filesystem access rather than S3. The object namespace stays the system of record, so no second system enters the picture.

Match tiers to access patterns and service level objectives rather than to vendor categories. Throughput for training. Shared context at microsecond latency for inference. One governance model across both, applied once to objects, tables, and files.

FAQ

What storage is best for GPU AI workloads?

No single tier is. Training wants sustained throughput against large objects and tables. Inference wants microsecond access to context shared across GPUs, which durable tiers were never designed to serve whatever protocol they speak. The MinIO solution covers both: AIStor as the data foundation for datasets, models, and checkpoints, and MemKV as the context memory tier, so GPUs retrieve prior work instead of recomputing it.

How does MemKV reduce inference latency?

By turning cache misses into cache hits at the cluster level. When a model evicts KV cache from GPU HBM under memory pressure or session routing, MemKV persists that context as blocks on NVMe in a shared namespace. The next request that needs it streams those blocks straight back into GPU memory over RDMA, bypassing filesystems, object protocols, and kernel stacks. The reused portion of the prefill is not recomputed, so time to first token improves in proportion to how much of the context hits.

Why is AI data gravity moving toward object?

Scale, access, and inertia compound. AI corpora, model artifacts, checkpoints, and lakehouse tables run to petabytes, and object holds petabytes economically on commodity hardware. The ecosystem standardized on S3, so frameworks, lakehouse engines, and agent runtimes read it natively with no translation layer in the path. Inertia matters most: once petabytes land, moving them costs time, money, and risk, so the tier holding the data becomes the tier new workloads get built against. Iceberg tables sit on that same namespace, and native filesystem access reaches it too, so consolidating does not mean giving up existing workflows.

Can object replace parallel file systems for AI training?

Yes, for modern pipelines that read large images, videos, or columnar tables in batches. S3-native data loaders for PyTorch, and S3 file clients for TensorFlow and JAX, paired with sharded formats such as Parquet, TFRecord, and WebDataset, read data in place and scale out under Kubernetes. Workloads with heavy small-file randomness gain more from sharding the dataset than from adding a tier, since sharding converts random small reads into large sequential ones. Where native filesystem access is a hard requirement, AIStor Files reaches the same buckets in the same deployment.

How should I size the MemKV tier?

Start from session mix and context lengths. Multiply average active sessions by average retained context size for steady-state capacity, then add headroom for bursts, retries, and compaction. Plan for roughly 2 to 3 times the steady-state working set. For bandwidth, budget read throughput to cover worst-case concurrency of context restores at your target p95 latency, then provision RDMA-capable NICs, fabric, and NVMe to sustain it with margin. Because the pool is flash-backed rather than HBM, oversizing capacity usually costs less than pushing eviction harder and risking cache-miss amplification.

Does this require direct storage-to-GPU DMA or special NICs?

Not for MemKV. Its context path is RDMA-based and already bypasses host I/O stacks, so a direct DMA path is not a prerequisite during inference. RDMA-capable NICs and fabric are, and that fabric is infrastructure you deploy rather than software MinIO ships. For training data loads, AIStor supports a direct RDMA path from NVMe into GPU memory for S3 reads.