This guide gives ML infrastructure practitioners a practical framework for checkpoint frequency, storage layout, recovery validation, and capacity/throughput sizing so distributed training runs recover fast and waste minimal GPU compute.
Distributed training jobs are among the most expensive workloads in modern computing. A single run across hundreds of GPUs can cost tens of thousands of dollars per day and when that job fails without a recent, valid checkpoint, the lost compute is irrecoverable. Checkpoint I/O is frequently the bottleneck: early PyTorch implementations took nearly 30 minutes to write a single 11B-parameter checkpoint, and recovery from corrupted or missing state can waste more time than the failure itself. This guide provides a practitioner-level framework for checkpoint storage strategy, covering frequency, layout, recovery, sizing, and the infrastructure choices that make checkpointing reliable at scale. Whether you are training a 7B-parameter model on a single node or a 700B-parameter model across a large cluster, the principles here will help you minimize wasted compute and maximize effective training time.
In this guide:
Distributed training introduces failure modes that single-node jobs rarely encounter. Hardware faults, network partitions, GPU memory errors, and preemption events can terminate a run at any point. Without a recent checkpoint that captures the full training state (i.e. model weights, optimizer state, learning rate schedules, dataloader position) the only option is to restart from scratch or from a checkpoint that may be hours old.
The cost of poor checkpoint infrastructure compounds quickly. A training job that saves every 100 batches over a 100,000-batch run produces 1,000 checkpoints. Each must be written quickly enough to avoid stalling the GPUs, stored durably enough to survive the failures it exists to protect against, and organized well enough that automated recovery can find and validate the correct state. When any of these properties break down, the effective training time ratio, the fraction of wall-clock time spent on actual forward and backward passes, drops sharply.
Parallel strategies like FSDP, DeepSpeed ZeRO, and tensor/pipeline parallelism further complicate the picture. These approaches shard model and optimizer state across workers, meaning a complete checkpoint may require coordinated writes from every machine in the job. All machines must load identical state before distributed training can resume, making both write coordination and read consistency critical.
The right checkpoint interval is a function of three variables: the cost of lost compute per unit time, the observed or expected failure rate, and the throughput of your storage system.
A useful heuristic is to checkpoint frequently enough that the expected cost of lost compute between checkpoints is less than the cost of the checkpoint itself (measured in GPU idle time during the write). For a job running on 256 GPUs at $2/GPU-hour, each hour of lost progress costs $512. If your storage system can write a full checkpoint in 3 minutes, the GPU idle cost per checkpoint is roughly $25, meaning you should checkpoint at least once per hour, and more frequently if your failure rate is high.
In practice, most production teams checkpoint every 15 to 60 minutes for large-scale runs. Training frameworks should support automatic checkpoint creation at configurable intervals, and the interval should be tunable based on observed cluster reliability.
Modern checkpoint libraries, including PyTorch's Distributed Checkpoint (DCP) and ByteCheckpoint, support asynchronous save operations that avoid blocking training tasks. Asynchronous checkpointing copies the training state to a staging buffer and writes to storage in the background, allowing the next training step to proceed immediately. This approach is essential at scale: in collaboration with PyTorch's distributed checkpointing team, IBM Research demonstrated that DCP could write a 30B-parameter checkpoint (including optimizer and dataloader state, totaling over eight times the raw model data) in just over 3 minutes, compared to nearly 30 minutes with synchronous torch.save for a smaller 11B model.
The key constraint is that your storage backend must sustain enough write throughput to complete the async flush before the next checkpoint is triggered. If writes fall behind, you either stall training or skip checkpoints, both are unacceptable outcomes.
A well-designed storage layout makes automated recovery reliable and lifecycle management straightforward.
Organize checkpoints by job, then by step or epoch. A consistent naming convention enables scripts to discover the latest checkpoint without querying metadata:
s3://checkpoints/{job-id}/step-{step:08d}/rank-{rank:04d}/model.pt
s3://checkpoints/{job-id}/step-{step:08d}/rank-{rank:04d}/optimizer.pt
s3://checkpoints/{job-id}/step-{step:08d}/metadata.json
Key principles:
Not every checkpoint needs to live forever. A practical retention policy uses two tiers:
Incremental checkpoints should be retained for approximately one week, while full checkpoints should be compressed and moved to archive storage periodically. Daily checkpointing at scale adds up quickly in storage volume, and compression combined with tiered retention is one of the more effective levers for controlling that growth over time. The specific savings will depend on your checkpoint size, frequency, and existing retention habits, so it's worth benchmarking against your own baseline rather than assuming a fixed percentage.
Use object lifecycle rules to automate expiration of rolling checkpoints and transition of milestone checkpoints to lower-cost tiers.
Recovery is the reason checkpoints exist, yet it is often the least-tested part of the training pipeline.
Production checkpointing requires automated corruption detection. Before resuming from a checkpoint, your recovery script should:
If validation fails, the system should auto-recover to the most recent valid checkpoint. Recovering from a known-good older checkpoint is always better than losing an entire run. Successful recovery rates should be tracked as a reliability metric for your checkpoint infrastructure.
Once a valid checkpoint is identified, every worker in the job must load its corresponding shard and synchronize before training resumes. The sequence is:
Recovery APIs should be simple. Users should only need to provide the states they want to restore (model, optimizer, extra states), and the checkpoint library should handle shard mapping and device placement. ByteCheckpoint, for example, aims to hide process-group and device-mesh complexity from users, making recovery code nearly identical regardless of the parallelism strategy used.
A common operational need is to resume a job on a different number of GPUs than it was checkpointed on. For example, scaling from 64 to 128 GPUs after acquiring more capacity. DCP introduced resharding support for FSDP checkpoints, though tensor and pipeline parallelism resharding requires additional tooling. DeepSpeed's Universal Checkpoint (UCP) offers a unified checkpoint format with offline resharding scripts, and Megatron's distributed checkpointing (MCP) builds on DCP's workflow while extending the available file storage formats, such as Zarr. If your training pipeline may change topology between runs, choose a checkpoint format that supports resharding natively.
Checkpoint portability, the ability to write a checkpoint on one cluster and resume from it on another, is a practical requirement for teams that train across on-premises and cloud GPU pools, or that need to migrate jobs between environments during long runs.
The S3 API has become the de facto standard interface for object storage, and using it as the checkpoint I/O layer decouples your training code from any specific storage backend. Industrial platforms already store checkpoints on a variety of backends including local disk, HDFS, and NAS; users choose storage backends based on job characteristics and preference. Any S3-compatible object store, whether a cloud provider's native service or a self-hosted deployment, provides a single, consistent interface regardless of where the storage physically resides, which means a checkpoint written on one cluster can be read directly by a job running somewhere else entirely, with no format conversion, path rewriting, or vendor-specific SDK required.
MinIO AIStor is built for this pattern. As the data and memory foundation for enterprise AI, it persists checkpoints through a native S3 interface that runs identically on bare metal, in Kubernetes, and across core, edge, and cloud, so a checkpoint written on one cluster reads directly on another with no format conversion or vendor SDK.
Checkpoints are only useful if they survive the same failures they are designed to protect against. A checkpoint stored on a single local disk is lost when that disk fails, exactly the scenario that caused the training job to fail in the first place.
Traditional RAID arrays provide some protection but introduce performance bottlenecks and rebuild times that are poorly suited to the bursty, high-throughput write patterns of checkpoint I/O. Erasure coding offers a better tradeoff: data is encoded into data and parity fragments distributed across multiple drives and nodes, so that any subset of fragments can reconstruct the original data. This provides configurable durability (commonly tolerating the loss of multiple drives or even entire nodes) without the write amplification and rebuild penalties of RAID. Most modern distributed and erasure-coded object stores implement some version of this pattern.
MinIO AIStor secures and persists checkpoints by automatically striping them across drives and nodes with inline bitrot detection, so corruption is caught at read time without a separate integrity-checking pipeline. Whichever storage system you use, automated corruption detection at write and read time is worth confirming before you rely on it for checkpoint durability.
Checkpoint storage requirements scale with model size, but the relationship is not linear, optimizer state and gradient buffers often dominate the total checkpoint size.
A single checkpoint includes model parameters, optimizer state (typically 2× the model size for Adam-family optimizers), and ancillary state (dataloader position, scheduler, RNG state). A useful rule of thumb: total checkpoint size ≈ 8× the model parameter count in bytes when using mixed-precision training with an Adam optimizer — roughly 2 bytes per parameter for fp16 weights plus 4 bytes each for the two fp32 Adam moment buffers (8 bytes), for about 10 bytes/parameter before rounding down to a conservative working estimate.
These figures assume uncompressed checkpoints. Compression typically reduces checkpoint size by 20–40%, and should be applied to milestone checkpoints destined for long-term retention.
The throughput requirement is determined by your target checkpoint write time and the checkpoint size. If you want to write a checkpoint in under 5 minutes to minimize GPU idle time:
For 70B+ models, this means your storage layer must sustain multi-GB/s aggregate write throughput across the cluster. A minimum of 25GbE networking per node is a reasonable baseline; at the 700B scale, hitting ~19 GB/s (roughly 150+ Gbps) in aggregate typically requires multiple 100GbE links spread across several storage nodes, not a single link. Plan your node count and network fabric around the aggregate figure, not a per-node one. Distributed object storage architectures generally scale throughput by adding nodes, which makes it more straightforward to match storage bandwidth to GPU cluster capacity as model scale grows.
Building a reliable checkpoint storage layer does not require exotic infrastructure, but it does require intentional design. Start with these steps:
None of this requires exotic infrastructure, it requires treating checkpoint I/O as a first-class part of your training architecture rather than an afterthought. Teams that instrument, automate, and test recovery paths before they're needed spend far less wall-clock time recovering from failures they could have anticipated.
How often should I checkpoint a distributed training job?
Checkpoint frequently enough that the expected cost of lost compute between saves exceeds the cost of the checkpoint write itself; for most large-scale runs, this means every 15 to 60 minutes. Use asynchronous checkpointing to reduce the effective cost per save and tune the interval using observed GPU cost, failure rate, and storage throughput.
What storage throughput do I need for LLM checkpointing?
Throughput depends on model size and your target write time: about 190 MB/s for 7B, ~1.9 GB/s for 70B, and ~19 GB/s for 700B to meet a 5-minute target. MinIO AIStor scales throughput by adding nodes, matching aggregate write bandwidth to your checkpoint windows as models and GPU counts grow. That is the speed, scale, and economics large-model training demands.
How do I recover a distributed training job from a checkpoint?
Identify the most recent valid checkpoint via manifest and checksum validation, broadcast the path to all workers, have each worker load its rank shard, perform a barrier synchronization, and resume from the recorded step; if validation fails, fall back to the next valid checkpoint. Automate this logic in your job launcher so recovery is reliable and requires no manual intervention.
.svg)