This guide walks ML infrastructure, platform, and DevOps practitioners through deploying MinIO AIStor on a multi-node GPU cluster, covering operator installation, object store configuration, erasure coding, and TLS setup on Kubernetes. You'll learn how to size PVCs and tune multipart transfer settings for high-throughput checkpoint writes and dataset loading, and how to configure separate storage profiles for training's sequential large-object writes versus inference's random small-object reads. The guide also covers where RDMA and KV cache fit alongside the object layer, so your GPUs spend cycles on compute rather than waiting on data.
GPU AI workloads on Kubernetes require storage that delivers sustained high throughput to GPU-attached pods without becoming the scheduling or I/O bottleneck. Two gaps show up in most existing guidance. Some guides treat storage as an external appliance bolted onto the cluster, disconnected from the way Kubernetes actually schedules and manages workloads. Others cover Kubernetes storage in general but ignore the I/O requirements of accelerator clusters, recommending generic CSI drivers that collapse under multi-gigabyte-per-second checkpoint writes.
The guide addresses both gaps. MinIO AIStor is an exascale data store for high-performance AI workloads, unifying objects, Iceberg-native tables, and agentic memory in a single namespace. What follows walks through configuring AIStor on Kubernetes, from operator deployment and object store design to erasure coding and workload-specific storage profiles, so your GPUs spend their cycles on matrix math instead of waiting on data.
The examples target a multi-node GPU cluster and are written for ML engineers, platform engineers, and DevOps practitioners. The scope is the object layer that holds datasets, checkpoints, and model weights. Where an inference deployment needs a memory tier closer to the GPU than the object layer sits, the guide says so and points to it.
MinIO AIStor is licensed under the MinIO Software License, and production deployments require an active license registered through SUBNET. AIStor Free covers single-node single-drive and single-node multi-drive patterns only; the multi-node object store in this guide requires a paid subscription.
Retrieve your license file from SUBNET before starting the installation below. You pass the decoded JWT to the Helm install command. The JWT begins with eyJ. Do not pass the encrypted license block, which begins with ZXlK; the operator rejects it. If you deploy before applying a license, recent AIStor Server releases start in offline mode with the Admin API and Console reachable and all S3 operations blocked until you register the license.
MinIO recommends Helm 3.17 or later.
The hardware targets below follow MinIO's recommended configuration for production: 8 or more persistent volumes per server, 100GbE networking, 16 or more vCPU per node, and 128GB or more of available memory per node.
MinIO AIStor deploys on Kubernetes through a first-party operator model. The operator manages its own lifecycle, provisioning, scaling, upgrades, and health checks through custom resources rather than external tooling. AIStor object stores are declarative objects that the operator reconciles continuously, so they fit the same GitOps workflows you already run for the rest of your platform. Operational simplicity is the point. One control plane, one reconciliation loop, and no external tooling to maintain alongside it.
Namespace isolation
Isolating AI storage in a dedicated namespace prevents noisy-neighbor effects and simplifies RBAC. Create the namespace before deploying the object store:
kubectl create namespace ai-storage
On GPU clusters where multiple teams share infrastructure, a namespace-level resource quota keeps any single object store from monopolizing NVMe bandwidth or memory. Size the quota to the object store you deploy into this namespace. The four-node example below requests 64 CPU and 384Gi in total, caps at 128 CPU and 512Gi, and creates 32 PVCs:
yamlapiVersion: v1
kind: ResourceQuota
metadata:
name: ai-storage-quota
namespace: ai-storage
spec:
hard:
requests.cpu: "64"
requests.memory: 384Gi
limits.cpu: "128"
limits.memory: 512Gi
persistentvolumeclaims: "32"
Operator installation
Current AIStor deployments use Helm charts rather than the legacy kubectl minio plugin workflow. Add the MinIO AIStor Helm repository:
helm repo add minio https://helm.min.io/
Install the operator chart into a dedicated aistor namespace, passing your SUBNET license JWT:
helm install aistor minio/aistor-operator \
-n aistor --create-namespace \
--set license="eyJhbGciOiJFUzM4NCIsInR..."
Verify the operator is running:
kubectl get all -n aistor
You see object-store-operator and adminjob-operator pods in a Running state.
The Helm values below target a four-node GPU cluster, with each node carrying eight NVMe drives and high-bandwidth networking. The key decisions:
Save a copy of the chart's default values as a starting point:
helm show values minio/aistor-objectstore > aistor-objectstore-values.yaml
Edit the file to reflect your deployment, and remove any default or unmodified values so the file carries only your changes. The root credentials under secrets are mandatory. Pool volumes are configured with size and storageClassName directly on the pool, and nodeSelector, tolerations, and resources are pool-level fields:
yaml
secrets:
accessKey: "access-key"
secretKey: "secret-key"
objectStore:
name: aistor-gpu
pvcProtection: true
pools:
- name: gpu-pool-0
servers: 4
volumesPerServer: 8
size: 2Ti
storageClassName: local-nvme
resources:
requests:
cpu: "16"
memory: 96Gi
limits:
cpu: "32"
memory: 128Gi
nodeSelector:
gpu-storage: "true"
tolerations:
- key: "accelerator"
operator: "Exists"
effect: "NoSchedule"
services:
minio:
serviceType: ClusterIP
Setting pvcProtection: true guards the pool's persistent volume claims against accidental deletion, which MinIO recommends for production.
For the object store's own NVMe volumes, local-nvme above stands in for any local-storage class. MinIO DirectPV is the first-party volume manager for direct-attached NVMe and integrates with the same operator model if you prefer a MinIO-native path; its storage class is directpv-min-io.
Spreading Pods Across Failure Domains
A nodeSelector tells the scheduler which nodes are eligible. It does not guarantee that the four server pods land on four different nodes, and two pods sharing a node collapses both the failure isolation erasure coding provides and the co-location argument for pinning storage to GPU nodes in the first place.
Configure spread zones to distribute pods across failure domains. Label each host as its own zone to spread a pool's pods across hosts, or label each rack as a zone to spread them across racks. The operator assigns pods to zones round robin, so a pool of four servers needs at least four zones for each pod to land in a zone of its own.
Deploy the object store into the ai-storage namespace you created above, so the resource quota applies to it:
helm install aistor-gpu minio/aistor-objectstore \
-n ai-storage \
-f aistor-objectstore-values.yaml
Monitor readiness:
kubectl get all -n ai-storage
The operator creates the object store's services from the resource name, so the object store above is reachable inside the cluster at aistor-gpu.ai-storage.svc.cluster.local, with a headless service at aistor-gpu-hl for per-pod addressing. The S3 API listens on port 9000 by default. Confirm both the service name and the port with kubectl get svc -n ai-storage before wiring clients to them.
Transport encryption
On Kubernetes clusters that have a valid TLS cluster signing certificate, AIStor generates TLS certificates automatically when you deploy or modify an object store, using the Kubernetes certificates.k8s.io API. The generated certificate carries the DNS subject alternate names for the object store's services and pods, and it is signed by the cluster CA. No additional configuration is required, because objectStore.certificates.disableAutoCert defaults to false.
Clusters without a cluster signing certificate, and deployments that need certificates from your own CA, use the manual path instead. Create Kubernetes Secrets of type kubernetes.io/tls in the object store namespace and reference them through the certificates field. The operator attaches them to the pods, and recent AIStor Server releases reload updated certificates within about a minute without restarting pods.
Because the automatic certificates are signed by the Kubernetes cluster CA rather than a public CA, clients validating them need the cluster CA bundle. In-cluster workloads find it at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, which is where the training pod below reads it from.
Installation is not finished at this point. MinIO treats network encryption and server-side encryption as part of the install, not as optional follow-on work.
Set up the client alias
Create an AIStor access key for programmatic access (through the Console or mc admin accesskey create), then point the AIStor Client at the deployment so later commands are concise. Use credentials with admin permissions for the configuration steps:
mc alias set aistor https://aistor-gpu.ai-storage.svc.cluster.local:9000 ACCESS_KEY SECRET_KEY
GPU training and inference pods consume bulk data from AIStor over the S3 API, but ancillary storage such as scratch space, local caches, and checkpoint staging still requires properly configured PVCs. Getting the storage class, access mode, and capacity right prevents pod scheduling failures and I/O stalls.
Storage class selection
Match the storage class to the workload profile:
For most AI training jobs, a combination of local-nvme for per-pod scratch and the S3 API for bulk data access works well. Avoid network-attached block storage such as generic CSI drivers over iSCSI for checkpoint writes; the latency overhead compounds across thousands of iterations.
PVC manifest for GPU training pods
The following PVC provides 500 GiB of local NVMe scratch to a training pod. Setting volumeBindingMode: WaitForFirstConsumer on the local-nvme storage class binds the PVC to the same node where the GPU pod is scheduled:
yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: training-scratch
namespace: ai-workloads
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-nvme
resources:
requests:
storage: 500Gi
The training pod reads its object store credentials from a Kubernetes Secret. Create it in the workload namespace from the AIStor access key you generated earlier:
kubectl create secret generic aistor-credentials \
-n ai-workloads \
--from-literal=accesskey=ACCESS_KEY \
--from-literal=secretkey=SECRET_KEY
Reference the PVC and the S3 endpoint in the training pod spec. The AWS_CA_BUNDLE path points at the cluster CA that signed the object store's certificate:
yaml
apiVersion: v1
kind: Pod
metadata:
name: llm-training
namespace: ai-workloads
spec:
containers:
- name: trainer
image: my-registry/llm-trainer:latest
resources:
limits:
nvidia.com/gpu: 8
env:
- name: S3_ENDPOINT
value: "https://aistor-gpu.ai-storage.svc.cluster.local:9000"
- name: S3_BUCKET
value: "training-data"
- name: AWS_CA_BUNDLE
value: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aistor-credentials
key: accesskey
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aistor-credentials
key: secretkey
volumeMounts:
- name: scratch
mountPath: /scratch
volumes:
- name: scratch
persistentVolumeClaim:
claimName: training-scratch
tolerations:
- key: "accelerator"
operator: "Exists"
effect: "NoSchedule"
Capacity planning
A practical rule of thumb is to provision scratch PVC capacity at twice the size of your largest checkpoint. A 70B-parameter model stored as bfloat16 weights is roughly 130 to 140 GiB per checkpoint. Retaining two checkpoints locally before flushing to AIStor needs at least 280 GiB of scratch, which makes a 500 GiB PVC a comfortable baseline.
The S3-native API is the primary data path for GPU workloads. How your application interacts with that API, including parallelism, part sizes, and connection management, determines whether the storage layer saturates available network bandwidth or leaves GPUs idle.
Parallelism and multipart configuration
Transfer large model weights and datasets using multipart uploads and downloads. The MinIO Python SDK (minio-py) and the AIStor Client (mc) both support configurable part sizes and thread counts.
The mc put command exposes --part-size and --parallel for tuning a single large-object upload such as a consolidated checkpoint:
mc put \
--part-size 256MiB \
--parallel 32 \
/scratch/checkpoint-epoch-42.safetensors \
aistor/checkpoints/epoch-42.safetensors
Key parameters:
To upload an entire checkpoint directory rather than a single object, use mc cp --recursive --max-workers N, which parallelizes across objects using default part sizing.
Connection pooling in application code
Training frameworks that read from S3 on each iteration benefit from persistent HTTP connection pools. In Python, size the urllib3 pool to match your parallelism, and point the pool at the cluster CA so TLS validation succeeds against the operator-generated certificate:
python
from minio import Minio
import urllib3
http_client = urllib3.PoolManager(
num_pools=16,
maxsize=32,
cert_reqs="CERT_REQUIRED",
ca_certs="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
retries=urllib3.Retry(
total=5,
backoff_factor=0.2,
status_forcelist=[500, 502, 503, 504],
),
)
client = Minio(
"aistor-gpu.ai-storage.svc.cluster.local:9000",
access_key="ACCESS_KEY",
secret_key="SECRET_KEY",
secure=True,
http_client=http_client,
)
Setting maxsize=32 keeps 32 connections warm per pool, which removes TCP and TLS handshake overhead on repeated GET calls during data loading. For frameworks like PyTorch that use DataLoader with multiple workers, each worker should instantiate its own client with its own pool to avoid contention.
Prefetch and pipeline overlap
The most effective pattern for training overlaps data prefetch with GPU compute. Structure the pipeline so the next batch is fetched from AIStor while the current batch is on the GPU:
The three-stage pipeline hides AIStor I/O latency behind compute when the storage layer sustains the required throughput.
Moving objects to the GPU over RDMA
The pipeline above stages every batch through host memory. AIStor also supports transferring objects over RDMA, which shortens that path by moving object payloads directly into GPU or host memory without the intermediate CPU copy. The S3 control path stays HTTP, the transfer is opt-in per client, and clients fall back to HTTP automatically when RDMA is unavailable, so the staged pipeline remains the correct default and RDMA is an optimization layered on top of it.
On Kubernetes, the requirements are an RDMA-capable fabric and matching NIC resources exposed to the pods. See the AIStor RDMA documentation for fabric configuration, deployment, and validation.
Default storage configurations are designed for general-purpose workloads. GPU clusters have distinct I/O characteristics, including large sequential writes, high-bandwidth reads, and bursty access patterns, that reward deliberate tuning of erasure coding, drive topology, and network interfaces.
Erasure coding configuration
AIStor uses erasure coding (EC) to provide data durability without the storage overhead of full replication. The erasure set size and parity ratio affect read throughput. For this four-node, 32-drive layout, AIStor forms two 16-drive erasure sets by default, and EC:4 is a good balance of durability and performance. EC:4 on a 16-drive set writes 12 data shards and 4 parity shards, so each erasure set tolerates up to four simultaneous drive failures. With 12 data shards per set, read parallelism is sufficient to saturate a 100 Gbps NIC for large-object reads on this configuration.
Set the parity level with the storage_class configuration key (equivalent to the MINIO_STORAGE_CLASS_STANDARD environment variable):
mc admin config set aistor storage_class standard=EC:4
Sixteen-drive erasure sets are the AIStor default and the recommended maximum for performance and network efficiency, so there is no need to set MINIO_ERASURE_SET_DRIVE_COUNT for this layout. Stripe size is immutable once the cluster is initialized and has significant impact on availability and performance. Leave it at the default unless MinIO engineering advises otherwise.
Drive topology
Co-locate AIStor drives on the same physical nodes as GPUs where possible. Co-location removes one network hop for data read into GPU memory. If dedicated storage nodes are required for density or thermal reasons, keep them on the same leaf switch as the GPU nodes.
Use local persistent volumes backed by NVMe SSDs rather than network-attached block devices. The latency profile of NVMe (sub-100 microseconds) versus network block (500 microseconds to 2 milliseconds) is the difference between keeping GPUs fed and creating I/O stalls.
Network interface selection
On nodes with multiple NICs, bind AIStor to the high-bandwidth fabric. Set the MINIO_SERVER_URL environment variable to the hostname or IP on the high-speed interface. Environment variables belong at the objectStore level, applied through a Helm upgrade:
yaml
objectStore:
env:
- name: MINIO_SERVER_URL
value: "https://10.100.0.10:9000"
On nodes that carry both a management NIC (for example 1 GbE) and a data NIC (100 or 200 GbE), this binding prevents AIStor traffic from flowing over the management plane. AIStor also supports multiple NICs for internode traffic through its server configuration for aggregate bandwidth.
Training and inference impose different I/O patterns on storage. Treating them identically wastes either throughput or IOPS. The two profiles below separate them at the bucket level, which is where the object model lets you tune each one independently.
Training profile: sequential large-object writes
During training, the dominant storage operation is checkpoint writing. A distributed job across eight GPUs produces a consolidated checkpoint every N iterations. These checkpoints are large (tens to hundreds of GiB), sequential, and written in bursts.
Tune the api settings so incomplete multipart uploads survive long runs, and confirm EC:4 as the standard parity:
mc admin config set aistor api \
stale_uploads_expiry=24h \
stale_uploads_cleanup_interval=6h
mc admin config set aistor storage_class standard=EC:4
Confirm the exact api configuration key names against your running AIStor Server version with mc admin config get aistor api before applying in production. Configuration keys can change between releases.
Create a dedicated bucket for checkpoints. New buckets have versioning disabled by default, which is what you want when checkpoints are overwritten rather than versioned:
mc mb aistor/checkpoints
Inference serving reads model weights once at startup, a large sequential read, and then performs many small, random reads for tokenizer vocabularies, configuration files, and LoRA adapters. The access pattern is high-IOPS, low-latency, and read-heavy.
AIStor's cache keeps the metadata of accessed objects in node memory on PUT and GET requests, which absorbs repeated small-object reads and evicts the least recently used entries when memory fills. The cache is off by default and is enabled at server startup through the MINIO_CACHE_ENABLE environment variable. Add it to the object store's env and apply with a Helm upgrade:
yaml
objectStore:
env:
- name: MINIO_CACHE_ENABLE
value: "on"
For inference, create a separate bucket with versioning enabled, so model versions are immutable and auditable:
mc mb aistor/models
mc version enable aistor/models
Where KV cache goes
KV cache is not one of the small objects above. Every request an inference server handles produces key-value state for the tokens it has already processed, and when that state falls out of GPU memory those tokens are recomputed from scratch. The state is transient, derived, and recomputable if lost, so it needs none of the durability the object layer is built to provide, and it needs retrieval faster than the S3 path is designed to deliver. Configuring the object layer for it is the wrong fix.
MinIO MemKV holds that layer. MemKV is a petascale context memory store for AI inference that keeps KV cache blocks in a flash-native tier shared across inference nodes, so a block recalled from MemKV is a prefill recomputation that did not occur. It sits below the object layer and alongside the configuration in this guide rather than replacing any of it.
A shared filesystem, whether POSIX or NFS, is typically tuned for one blended workload profile across a single namespace. That works when training and inference share the same mount, but it makes it hard to independently tune write throughput for checkpoints and read IOPS for inference serving.
AIStor's object model configures erasure coding, lifecycle policies, and access patterns per bucket, so training and inference can be optimized independently within the same cluster. On Kubernetes, those settings map to separate buckets, bucket policies, and client configurations that you manage declaratively alongside the rest of your platform, through the same operator and GitOps workflows you already use.
The same argument extends past objects. AIStor holds objects, Iceberg-native tables, and agentic memory in one namespace under one operator, so a governed table or an agent's accumulated memory lands where the training data already sits rather than in a separate platform with its own control plane. AIStor Memory, the agentic memory service, is in Tech Preview.
How do I configure MinIO AIStor for Kubernetes GPU workloads?
Install the MinIO AIStor operator with Helm (helm install aistor minio/aistor-operator) using a valid SUBNET license JWT, then deploy an ObjectStore resource with the minio/aistor-objectstore chart into a dedicated namespace, setting the mandatory root credentials under secrets. Configure NVMe-backed PVCs for scratch space, size per-pod resource requests to MinIO's recommended node configuration of 16 or more vCPU and 128GB or more of memory, add node affinity to co-locate storage pods with GPU nodes, and configure spread zones so each server pod lands on its own node. Set the standard parity with mc admin config set aistor storage_class standard=EC:4, and bind AIStor to your high-bandwidth interface with MINIO_SERVER_URL. GPU pods then access data through the cluster-internal S3 endpoint using multipart transfers with 256 MiB parts and client-side parallelism tuned to your network capacity.
What storage class should I use for AI training on Kubernetes?
Use a local-nvme storage class with ReadWriteOnce and WaitForFirstConsumer for checkpoint and staging scratch, which guarantees node-local NVMe latency. For bulk training data, read from AIStor's S3 API directly in your data loader rather than provisioning a shared-storage PVC.
How do I enable TLS for AIStor on Kubernetes?
On clusters with a valid TLS cluster signing certificate, AIStor generates and signs certificates automatically through the Kubernetes certificates.k8s.io API when the object store is created or modified, because objectStore.certificates.disableAutoCert defaults to false. To supply your own certificates instead, create Kubernetes Secrets of type kubernetes.io/tls in the object store namespace and reference them through the certificates field, and the operator attaches them to the pods. Clients validating the automatically generated certificates need the cluster CA bundle, available in-cluster at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.
How does AIStor compare to a shared filesystem on Kubernetes?
AIStor is Kubernetes-native and managed through a first-party operator, so it integrates with namespaces and RBAC and fits GitOps workflows, while exposing the S3-native API that modern ML frameworks expect. A shared filesystem presents one namespace with one tuning profile. AIStor presents buckets, each with its own erasure coding, lifecycle policy, and access pattern, and it holds Iceberg-native tables and agentic memory in that same namespace rather than requiring a second platform for them.
What is the difference between MinIO MemKV and AIStor Memory?
MemKV and AIStor Memory hold two different kinds of memory. MemKV holds context memory, the KV cache state a GPU produces while running inference. That state is transient, derived, and recomputable if lost, and it needs microsecond retrieval, so it lives in a flash-native tier close to the GPU rather than in the object layer. AIStor Memory holds organizational memory, the durable record of what agents have learned and done across sessions, models, and runtimes. AIStor Memory is part of AIStor and is in Tech Preview. Both eliminate repeated work, at different timescales. Neither substitutes for the other.
How does AIStor deliver data to GPU memory on Kubernetes?
Data loaders read from AIStor over the S3 API using multipart transfers and per-worker connection pools, with the batch staged into pinned host memory and moved to the GPU by asynchronous DMA while the previous batch is still in the forward pass. AIStor also supports transferring objects over RDMA, which moves the object payload directly into GPU or host memory and removes the intermediate copy. The S3 control path stays HTTP and clients fall back to HTTP automatically when RDMA is unavailable.
.svg)