Introducing S3 over RDMA in AIStor: Object Storage at the Speed of Memory

The most expensive computers ever built spend a surprising amount of their time waiting on storage.

For the better part of a decade, we have made object storage faster the same way: more drives, more nodes, faster NICs. And it worked, modern object stores saturate 400 GbE links without breaking a sweat. But every one of those bytes still takes the same last hop it took ten years ago: into host memory, through the kernel's TCP stack, copied by CPU cores, and only then handed to the GPU that actually asked for it. The network got two orders of magnitude faster. The last hop didn't change.

S3 over RDMA in AIStor changes it. A GET lands directly in GPU memory, the object store writes into VRAM over the RDMA fabric, and the host CPU steps out of the data path entirely. Same S3 API, same buckets, same erasure coding. What disappears is the copy.

In this blog post we are going to look at why that last hop has quietly become the tax on every AI workload, training, checkpointing, and inference cold starts, and what our measurements say happens when you remove it.

The Problem Wears Three Disguises

Ask three teams what their storage problem is and you will get three different answers. They are the same problem.

Training clusters fail, constantly, and checkpoints are the insurance. Meta reported that during a 54-day snapshot of Llama 3 405B pre-training on up to 16K H100 GPUs, they experienced 466 job interruptions, about one every three hours, with roughly 78% of the unexpected ones traced to confirmed hardware issues. And that is not a Llama-sized anomaly: in a separate study of 11 months and over 150 million A100 GPU-hours on their research clusters, Meta measured the mean-time-to-failure of a 1,024-GPU job at 7.9 hours, and projected 1.8 hours at 16,384 GPUs. At 131,072 GPUs the projection is 14 minutes.

The insurance premium is the checkpoint. Meta's reliability study notes that hourly checkpointing is typical for larger jobs on their clusters, which means an average of half an hour of lost work every time the job dies. Their own stated goal in the Llama 3 paper is exactly the one you would guess: "minimize GPU pause time during checkpointing and increase checkpoint frequency to reduce the amount of lost work after a recovery." The obstacle is also the one you would guess: checkpoint writes are "highly bursty" and "saturate the storage fabric for short durations", and that is on a storage fleet Meta describes as sustaining 2 TB/s with 7 TB/s peaks. Facebook's checkpointing team put it even more directly, as far back as 2020: "checkpoint frequency is often bottlenecked by the storage write bandwidth and capacity." ByteDance felt the same pain, their ByteCheckpoint system exists because production checkpoint stalls were bad enough to justify rebuilding the whole pipeline, reducing stalls by an average of 54×.

Inference fleets pay the same tax at startup. The ServerlessLLM work from OSDI '24 identified remote checkpoint download and loading as the dominant cost in serverless LLM inference, and earned a 10–200× latency reduction largely by attacking how weights get into the GPU. Every autoscaling event, every node replacement, every new model version, someone waits while tens of gigabytes of safetensors crawl from storage into VRAM.

And training itself pays it on every batch. Even when your data loader keeps up, the CPU cores it burns to move and copy bytes are cores your training loop wanted for augmentation, tokenization and Python.

Three disguises, one problem: the bytes are in the object store, the tensors need to be in GPU memory, and everything between them runs through the host.

The Host Steps Out

RDMA, Remote Direct Memory Access, lets a NIC write into a remote machine's memory without the remote CPU touching the transfer. GPU-Direct extends that so the memory being written is GPU memory. AIStor puts the two together behind the S3 API: the client hands the object store an RDMA descriptor for a buffer in VRAM, and the storage nodes stream the object straight into it over the RoCE fabric.

Nothing about your bucket changes. The object is still erasure-coded across the same drives; a plain HTTP client still reads it. RDMA is negotiated per request, so the same tenant serves your GPU cluster over RDMA and your dashboards over HTTP at the same time.

Here is what removing the last hop measures out to. We published every benchmark, the loaders, and the raw results in minio/rdma-training, and every RDMA figure in it is backed by a server-side RDMA counter delta, so you are never taking the client's word that RDMA actually carried the bytes.

Raw delivery into GPU memory. On a single 400 GbE rail, S3 over RDMA sustained 42.4 GB/s into VRAM using 0.95 CPU cores. A heavily tuned HTTP loader on the same fabric reached 39.4 GB/s, using 68.2 cores. Same bandwidth ceiling, 70× the CPU. That ratio is the whole story in one line: at 400 Gb and beyond, the bottleneck is not the network, it is the copy.

Training. An 8× H200 ResNet-50 run over GPU-native shards finished the same 200 batches 14% faster over RDMA, not because storage was the bottleneck (storage wait was 0.00 s on both transports) but because the HTTP data path's CPU appetite competes with the training loop itself. The tax shows up even when you never wait on storage.

Checkpointing. A 16 GiB checkpoint lands in 0.65 s over RDMA, against 12.16 s for tuned HTTP and 62.55 s for a plain torch.save. At LLM scale, 112 GiB of sharded state across 8 ranks completes in 4.33 s at 28 GB/s on 3.7 CPU cores, versus 14.42 s and 30.4 cores over HTTP. Go back to Meta's math: if the cost of a checkpoint drops by an order of magnitude, checkpointing more often stops being a trade-off, and the half hour of average lost work shrinks with it. This is precisely the knob their papers say they want to turn.

Cold starts. Llama-3.1-8B's full 16 GB of safetensors weights loaded into VRAM in 0.77 seconds, faster than reading the same weights from a warm local NVMe cache (2.29 s), and 40× faster than the download-then-load pattern (30.96 s). Qwen3-32B: 65 GB in 3.10 s versus 125 s. When the model store is faster than local disk, the case for baking weights into images or pre-staging them on every node quietly evaporates.

And this generalizes past the three headliners: KV-cache offload, embedding and feature stores, pre-decoded image shards, tokenized corpora, anything whose bytes are already tensors benefits from the same path.

Tensors In, Tensors Out

One mental model tells you where S3 over RDMA applies: the GPU must be able to consume the bytes exactly as they sit in the bucket. RDMA delivers into GPU memory, so if the payload is tensors, raw arrays, safetensors, checkpoints, tokenized text, the GPU can use them the moment they land.

It equally tells you where it does not apply, and we would rather tell you now than have you find out in production. A JPEG dataset cannot use this path: GPU decoders need the encoded bitstream in host memory, so an image pipeline built on JPEGs keeps the host in the loop no matter the transport, the fix is pre-decoded shards, not a faster wire.

Where This Goes

The pattern here is an old one. GPUs got fast enough that copying became the bottleneck, so the copy had to go. It went first inside the box, where peer devices learned to reach GPU memory without staging through host RAM. Now it goes across the fabric with S3 over RDMA. Storage is simply the next thing to stop going through the CPU.

The part we care most about: you do not need a research cluster to use it. The client side is one shared library and a small Python binding; the benchmarks in minio/rdma-training run on any AIStor tenant with an RDMA fabric, and the docs walk you from bare drives to a verified RDMA transfer, including every silent-fallback trap we hit so you don't have to. Feeding GPUs at memory speed is now within reach of any team running AIStor on a RoCE network.

In the coming posts we will go hands-on with each workload: training on GPU-native shards, checkpointing at LLM scale, and cold-starting inference straight from the object store. If you have any questions, or a fabric you are wondering about, join our Slack community and ask!

Sources

• Grattafiori et al., The Llama 3 Herd of Models, Meta, 2024, arxiv.org/abs/2407.21783. 466 interruptions in a 54-day snapshot; ~78% of unexpected interruptions hardware-related; 16K H100s; storage sustaining 2 TB/s / 7 TB/s peak; checkpoint writes "highly bursty"; goal to "minimize GPU pause time during checkpointing and increase checkpoint frequency."
• Kokolis et al., Revisiting Reliability in Large-Scale Machine Learning Research Clusters, Meta, 2024, arxiv.org/abs/2410.21680. MTTF 7.9 h at 1,024 GPUs, projected 1.8 h at 16,384 and 0.23 h at 131,072; hourly checkpointing typical for larger jobs; average half an hour of lost work.
• Eisenman et al., Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models, Facebook, NSDI 2022, arxiv.org/abs/2010.08679. "Checkpoint frequency is often bottlenecked by the storage write bandwidth and capacity."
• Wan et al., ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development, ByteDance, 2024, arxiv.org/abs/2407.20143. Runtime checkpoint stalls reduced by an average of 54.2× versus existing open-source systems.
• Fu et al., ServerlessLLM: Low-Latency Serverless Inference for Large Language Models, OSDI 2024, arxiv.org/abs/2401.14351. Checkpoint download/loading as the dominant cold-start cost; 10–200× latency reduction.
• All MinIO figures: github.com/minio/rdma-training, code, docs, and raw JSON for all 146 runs, each RDMA figure carrying a server-side counter delta.