How to Leverage Apache Iceberg for AI Workloads

Apache Iceberg is an open table format that brings ACID transactions, schema evolution, and time travel to large analytic datasets stored in object storage. It is a structured metadata layer over raw columnar files. Multiple compute engines can read and write data concurrently without corruption.

The shift from traditional analytics to artificial intelligence changes how organizations manage unstructured data. Machine learning pipelines require the same transactional guarantees and reproducibility that Iceberg was built to provide, but at the scale of model training and inference. You cannot train reliable models on data that changes unpredictably during the read process.

This article explains what Apache Iceberg is, why AI workloads require open table formats, what changed with the Iceberg V3 spec, and how to architect your data stack to support these capabilities. You will learn how the storage layer impacts performance and how to integrate Iceberg with standard compute engines.

What is Apache Iceberg?

Apache Iceberg is a log based, snapshot driven metadata layer that sits above immutable columnar files in object storage. Think of it like a version control system for your data lake tables: it tracks every change, allows you to roll back to previous states, and ensures multiple users can read and write simultaneously without corrupting the repository.

Iceberg replaces rigid, directory based data layouts with a flexible metadata tree. This architecture enables several core capabilities:

• ACID transactions: Iceberg separates logical table state from physical files. It commits changes through atomic metadata swaps. Readers see a consistent snapshot of the data, and concurrent writers operate safely without depending on the object store to guarantee strong consistency.
• Schema evolution without rewrites: You can change table schemas without rewriting existing data files. Operations like adding, dropping, renaming, or reordering columns are applied logically at read time.
• Time travel and snapshot isolation: Every commit produces an immutable snapshot of the table state. You can query the table as of a specific snapshot ID or timestamp. This provides strict snapshot isolation for readers.
• Partition evolution and hidden partitioning: Iceberg stores partition transforms in its metadata rather than relying on explicit directory structures. You can change partition specifications over time, such as moving from daily to hourly partitions, without rewriting historical data.

Iceberg's ecosystem also includes capabilities that sit outside the core table format spec and apply across format versions. The REST Catalog API provides a vendor neutral HTTP interface for catalog operations. Iceberg Views let you version logical datasets the same way you version tables. Multi-table transactions enable atomic commits across multiple tables at once, keeping complex updates consistent. These are catalog and view level capabilities, not tied to a single table format version, and many engines already support them on tables that predate V3.

What Iceberg V3 Actually Adds

Iceberg format version 3 is the current major revision to the table spec itself. It introduces:

• Deletion vectors: binary bitmaps attached to each data file that mark deleted rows directly, replacing the positional delete files used in V2. This makes row level deletes and updates faster and avoids the delete file pileup that slows down delete heavy workloads.
• Row lineage: native _row_id and _last_updated_sequence_number fields tracked on every row, enabling change data capture without external tooling and giving you a built in record of exactly when a given row was added or last modified.
• Richer data types: a VARIANT type for semi-structured data, native geometry and geography types, and nanosecond precision timestamps.
• Default column values and multi-argument partition transforms, closing gaps with traditional database features.

For AI workloads specifically, deletion vectors and row lineage matter more than the headline naming suggests. Feature stores that need to correct or retire individual training records benefit from faster, more targeted deletes. Model registries and audit pipelines benefit from row level lineage on top of snapshot level history, since you can now trace not just which snapshot fed a training run, but which specific rows were added or changed and when.

V3 tables remain backward compatible for readers built against V2, though older engines will not apply deletion vectors or row lineage correctly, so confirm your query engines support V3 before adopting it in production.

Why AI Workloads Need Open Table Formats

Artificial intelligence workloads require data governance, reproducibility, and high throughput. An open table format delivers these guarantees without locking your infrastructure into a proprietary data warehouse. Managing metadata and catalogs adds complexity, but that overhead is necessary to maintain data integrity at scale.

• Training pipelines: Machine learning reproducibility requires versioned datasets. Point in time snapshots replace ad hoc data freezes. Data scientists can pin a model training run to a specific, immutable snapshot, ensuring the exact same data can be audited or reused months later.
• Feature stores: Feature engineering is an iterative, continuous process. Schema evolution allows you to add new features or adjust data types without rewriting massive historical datasets. Time travel ensures point in time correctness, preventing future data from leaking into historical training sets. On V3 tables, deletion vectors also make it practical to correct or purge individual feature records without a full rewrite.
• Inference and model registry patterns: Production AI requires audit trails. Snapshot history records what data fed a model during inference. Multi-table transactions allow atomic updates to model registries and metadata tables, preventing reads of partial updates. Row lineage on V3 tables extends this from "which snapshot" to "which rows," strengthening the audit trail down to the row level.

While several formats exist, they differ in architectural focus. The table below compares Apache Iceberg, Delta Lake, and Apache Hudi across capabilities relevant to AI workloads.

‍

How Iceberg Fits Into the AI/ML Data Stack

Iceberg maps onto four layers of a modern AI data architecture. Data flows from source systems into object storage, where the Iceberg table format structures it. The catalog layer tracks these tables and provides metadata pointers to query engines and machine learning frameworks.

The storage layer

The foundation of the stack is S3-compatible object storage. This layer holds the actual data in columnar formats like Apache Parquet, ORC, or Avro. It must deliver the raw aggregate throughput required to keep GPUs fed during training. Iceberg relies on this storage layer for data durability and availability.

The table format layer

This layer consists of Iceberg metadata files, manifest lists, and manifest files. It defines the table structure, partition rules, and snapshot history. By managing these files, Iceberg enforces transactional rules and schema evolution across the underlying objects without requiring a proprietary storage engine.

The catalog layer

The catalog is the entry point for any table access. It tells query engines where to find the latest metadata file for a given table. The REST Catalog API standardizes this layer, reducing dependencies on legacy systems. An embedded catalog reduces external dependencies, while a standalone catalog provides a centralized governance point.

Related: Iceberg's Catalog API: The Atomic Pointer Manager Behind Your Iceberg Tables.

The query engine layer

Iceberg supports multiple compute engines reading and writing the same tables concurrently. You can use Apache Spark for heavy ETL jobs, Trino or Dremio for interactive queries, and Databricks for machine learning. Because the catalog and table format layers are engine-agnostic, you avoid compute lock-in.

Storage Layer Requirements for Iceberg at AI Scale

The table format depends on the storage system beneath it. AI workloads push storage infrastructure to its limits. If the storage layer bottlenecks, GPU utilization drops and model training stalls.

• Throughput: Training jobs perform large sequential reads across massive Parquet datasets. Your object store must deliver high aggregate read throughput to serve concurrent requests from multiple Spark executors or training nodes. Iceberg's metadata pruning reduces unnecessary file scans, but the storage still needs to deliver the data rapidly.
• S3-compatible API: An S3-compatible API ensures integration across catalog implementations, query engines, and machine learning frameworks. It provides the HTTP semantics that cloud-native applications expect.
• On-premises and hybrid flexibility: AI data often carries privacy and data sovereignty requirements. Deploying Iceberg on-premises or in hybrid environments gives you control over latency and helps avoid high cloud egress costs for moving petabytes of training data.
• Performance at petabyte scale: Iceberg tables can scale to tens of petabytes and billions of files. The storage layer should use erasure coding and a distributed architecture to maintain performance. Erasure coding provides durability with less overhead than replication, and a distributed design avoids single-node bottlenecks.

Related: Hungry GPUs Need Fast Object Storage 

Getting Started: Key Integration Patterns

Implementing Iceberg requires connecting your compute engines to your catalog and storage layers. The following patterns show how to configure standard engines for AI workloads.

Spark + Iceberg REST Catalog

Apache Spark is the standard engine for heavy feature generation and data preparation. Configure Spark to use the Iceberg REST Catalog and an S3-compatible storage backend.

The following configuration snippet shows how to set up the Spark session properties:

‍

# Use Iceberg as Spark catalog

spark.sql.catalog.my_iceberg=org.apache.iceberg.spark.SparkCatalog

spark.sql.catalog.my_iceberg.type=rest

spark.sql.catalog.my_iceberg.uri=https://iceberg-catalog.example.com

spark.sql.catalog.my_iceberg.warehouse=s3://my-data-lake/warehouse

spark.sql.catalog.my_iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO

‍

In this configuration, the type=rest parameter directs Spark to use the REST Catalog API. The uri points to your catalog server, while the warehouse defines the root path in your S3-compatible storage. The io-impl ensures Spark uses the correct library to interact with the object store.

‍

Databricks + Iceberg

Databricks integrates with Iceberg when configured with a compatible catalog. When connected to a REST Catalog, you can treat tables as native Iceberg tables and perform schema evolution via SQL.

Because Iceberg manages the schema evolution, existing files are not rewritten. Databricks reads and writes via its catalog integration, while the underlying table format semantics come from Iceberg. This allows you to write features using Databricks and read them later with a different engine.

Trino federation over Iceberg tables stored on S3-compatible storage

Trino provides interactive query capabilities across distributed datasets. By pointing Trino to the same REST Catalog and S3-compatible storage used by Spark, you can perform multi-engine queries.

‍

Configure the Trino catalog properties to enable the Iceberg connector and point it to your REST endpoint:

connector.name=iceberg

iceberg.catalog.type=rest

iceberg.rest-catalog.uri=https://iceberg-catalog.example.com

iceberg.warehouse=s3://my-data-lake/warehouse

‍

# S3-compatible storage config

hive.s3.aws-access-key=ACCESS_KEY

hive.s3.aws-secret-key=SECRET_KEY

hive.s3.endpoint=https://s3-compatible.example.com

hive.s3.path-style-access=true

‍

With this setup, Spark can continuously update feature tables while Trino queries those same tables. Both engines respect the same snapshot isolation rules, ensuring Trino does not read partially written files.

‍

Conclusion

Apache Iceberg provides transactional guarantees, reproducibility, and multi-engine flexibility that unstructured data lakes do not provide. By separating compute from storage and standardizing metadata through the REST Catalog API, Iceberg reduces vendor lock-in and supports data federation. The V3 spec extends this further for AI workloads, with deletion vectors and row lineage adding faster, more precise row-level operations and audit trails on top of the snapshot-level guarantees Iceberg already provided.

The storage layer determines whether those guarantees hold at the scale of model training. Without high throughput, petabyte scalability, and S3 compatibility, an open table format cannot deliver its full benefits.

Ready to build an AI data infrastructure that combines open table formats with high throughput? Explore MinIO AIStor Tables to see how purpose-built object storage and a native, embedded Iceberg V3 REST Catalog provide the performance required for enterprise AI workloads.

‍