.svg)
Every Apache Iceberg lakehouse needs a catalog, and in most deployments that catalog is a separate service. Hive Metastore, AWS Glue, or Nessie sits outside the object store, adding another service boundary, another credential set, and another integration to keep consistent with the storage layer beneath it.
With AIStor Tables, we took a different approach: the Apache Iceberg REST Catalog runs embedded inside the object store itself. The catalog has its own endpoint at the /_iceberg path, separate from the S3 API, and both are served by the same AIStor deployment on the same API port. There is no external catalog service to install or operate. Table metadata and table data live on one platform, and the same AIStor credentials and access policies govern both the Iceberg REST Catalog API and the S3 API.
AIStor Tables supports the full Apache Iceberg V3 REST Catalog API, including deletion vectors, row lineage, and the Variant type for semi-structured data. It also supports Iceberg Views as a catalog capability. Query engines such as Spark, Trino, and Dremio connect to that single endpoint using standard SigV4 signing. This post explains why that embedded design changes the lakehouse architecture.
What Is Object Storage in an Apache Iceberg Data Lakehouse?
Object storage is a storage architecture that manages data as discrete objects behind an HTTP API, rather than as blocks or files in a hierarchy. In a lakehouse, it is the durable foundation everything else sits on.
Apache Iceberg is a table format. It sits on top of storage and defines how a collection of data files, commonly Parquet, behaves like a database table: which files belong to the table, what the schema is, and which snapshot represents the current state.
The catalog is the authoritative index that tells query engines where each table’s current metadata lives. It also makes changes safe: when a writer commits, the catalog atomically swaps the table’s pointer from the old metadata file to the new one, so every reader sees either the old version of the table or the new one, never a mix.
Query engines such as Spark and Trino are the top layer. They ask the catalog where a table is, then read and write the files themselves.
Without a catalog, an engine cannot find the table. Without the table format, files are just files. Without object storage, there is nothing durable underneath.
Think of it like a library: object storage is the building and shelves, Iceberg is the cataloging system that groups pages into books, the catalog is the front desk that tells every visitor exactly where to look, and the query engines are the visitors who come in to read and add books.
Here is how the four layers map onto lakehouse with an AIStor foundation:
How Data Moves Through the Lakehouse Stack
Let’s walk the data path from ingestion to query. There are two API paths: a catalog path for table operations and an S3 path for the data and metadata files themselves.
On the write side, source systems such as pipelines, streaming jobs, and batch ETL use Iceberg writer libraries. The writer first writes new Parquet data files and Iceberg metadata files to AIStor through the S3 API. It then commits the change to the embedded REST Catalog, which atomically points the table at the new metadata file. Until that commit succeeds, readers keep seeing the previous version of the table.
On the read side, query engines ask the catalog to look up a table and return the location of its current metadata. They then read the metadata and data files directly through the S3 API.
Both paths use the same AIStor credentials and access policies. Catalog requests are signed with SigV4 using the service name s3tables, while S3 requests use the standard s3 service name, so the catalog signing name is the one extra setting engines need. Engines that can’t sign catalog requests with SigV4 can use OAuth2 bearer tokens instead. The catalog can also vend short-lived S3 credentials scoped to a single table, so engines don’t need long-lived S3 keys. Credential vending is off by default.
Both paths end at the same AIStor deployment, so there is no separate metadata database to keep in sync with the storage it describes.
Data Lakehouse vs. Data Lake vs. Data Warehouse
Why the Lakehouse Model Emerged
A data lakehouse is an architecture that combines a data lake's open, low-cost object storage with the transactional guarantees, schema controls, and query performance traditionally associated with a data warehouse. Data lakes gave organizations cheap, open storage but, on their own, weak consistency and governance. Warehouses gave strong guarantees but often at higher cost and with less format openness. The lakehouse, built on a table format like Apache Iceberg over object storage, aims to deliver both properties from one copy of the data.
The table below compares the three architectures across nine dimensions, with a fifth column showing what AIStor Tables specifically contributes to each one.
The Data Lakehouse column describes what the architecture can do in general. The MinIO AIStor Advantage column shows how AIStor Tables delivers each capability from the storage system itself.
Why AIStor Tables' Embedded Iceberg REST Catalog Is Architecturally Different
Native Apache Iceberg V3 Capabilities
AIStor Tables supports Iceberg V3 tables, which bring deletion vectors, row lineage, the Variant type for semi-structured data, and native geometry, geography, and nanosecond-precision timestamp types. Each one addresses a specific pain point. Deletion vectors cut the buildup of delete files. Row lineage tracks row-level changes for incremental processing. Variant avoids storing JSON as plain strings. The geospatial types let engines skip files outside a query’s area instead of scanning the whole table. Engines write and read these features through their Iceberg libraries, so check that your engine version supports V3 before relying on them. For the mechanics of each, including how deletion vectors use Roaring bitmaps in Puffin files and how row IDs persist through compaction, see our post AIStor Tables: Native Iceberg V3 for On-Premises Object Storage.
AIStor Tables also supports Iceberg Views at the catalog level. A view stores a query definition in the catalog, so teams can define it once and share it across engines. Each view records its SQL in a specific dialect, so an engine can use a view when it has a version in that engine’s dialect or can parse the original. AIStor Tables is the first on-premises and hybrid data store to support Iceberg REST Catalog views. For comparison, Amazon S3 Tables did not offer a native Iceberg REST endpoint until March 2025, and we haven’t found documentation showing views support there today. See Iceberg Views in AIStor Tables for the REST endpoints and example payloads.
We built AIStor Tables as the first data store with the full Iceberg V3 REST Catalog API built in, including Views and multi-table transactions. The catalog runs as part of the storage engine itself, wherever AIStor runs.
Embedded Catalog vs. External Catalogs
In most lakehouses, the catalog is a separate system. A self-hosted Hive Metastore or Nessie deployment is infrastructure you run, patch, and back up, and Hive Metastore also needs its own relational database, typically MySQL or PostgreSQL. A managed service such as AWS Glue shifts that work to the provider, but it is still a separate system to configure, secure, and keep aligned with your storage.
AIStor Tables removes that layer. The catalog organizes tables into a warehouse → namespace → table hierarchy, with policy-based access control at each level. See AIStor Tables: Technical Deep Dive for the full model and example IAM policies.
Because the catalog is built into AIStor, you don’t need Hive, Glue, Nessie, or a proprietary catalog service to run Iceberg tables. Engines connect to the catalog using SigV4 signing or OAuth2 bearer tokens. The multi-engine section below covers which engines are supported.
The operational result is fewer systems to run and fewer credentials to rotate. Table metadata and data also stay in step: tables are replicated and backed up the same way as objects, so there is no separate metadata database to keep in sync with storage or restore on its own after a failure. The trade-off is scope. An embedded catalog covers the tables stored in that AIStor deployment, so a team that needs one catalog spanning several storage systems may still choose a standalone catalog. For teams whose goal is Iceberg tables on AIStor, the embedded catalog removes a whole tier of infrastructure.
Configuring Spark to Query AIStor Tables via the REST Catalog
Connecting Spark to AIStor Tables takes standard Iceberg REST catalog configuration. You point Spark’s catalog at the /_iceberg endpoint, turn on SigV4 signing with the s3tables signing name, point data I/O at the AIStor S3 API on the same host, and set the warehouse name.
The following example assumes minio is the resolvable AIStor hostname, the AIStor API is on port 9000, and a warehouse named warehouse has already been created in AIStor Tables (for example, with mc table warehouse create). It reads credentials from the AISTOR_ACCESS_KEY and AISTOR_SECRET_KEY environment variables. The example uses HTTP for simplicity, so use HTTPS in production.
spark-shell \
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1,org.apache.iceberg:iceberg-aws-bundle:1.10.1 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.defaultCatalog=aistor \
--conf spark.sql.catalog.aistor=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.aistor.type=rest \
--conf spark.sql.catalog.aistor.uri=http://minio:9000/_iceberg \
--conf spark.sql.catalog.aistor.warehouse=warehouse \
--conf spark.sql.catalog.aistor.rest.endpoint=http://minio:9000 \
--conf spark.sql.catalog.aistor.rest.sigv4-enabled=true \
--conf spark.sql.catalog.aistor.rest.signing-name=s3tables \
--conf spark.sql.catalog.aistor.rest.signing-region=local \
--conf spark.sql.catalog.aistor.rest.access-key-id=$AISTOR_ACCESS_KEY \
--conf spark.sql.catalog.aistor.rest.secret-access-key=$AISTOR_SECRET_KEY \
--conf spark.sql.catalog.aistor.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
--conf spark.sql.catalog.aistor.s3.endpoint=http://minio:9000 \
--conf spark.sql.catalog.aistor.s3.path-style-access=true \
--conf spark.sql.catalog.aistor.s3.access-key-id=$AISTOR_ACCESS_KEY \
--conf spark.sql.catalog.aistor.s3.secret-access-key=$AISTOR_SECRET_KEY
A handful of settings do the important work. The uri points Spark at the catalog endpoint at /_iceberg. The sigv4-enabled and signing-name settings sign catalog requests with the s3tables service name, which the catalog requires. The signing region is required by SigV4 but not used by AIStor, so any non-empty value such as local works. The s3.endpoint points data reads and writes at the AIStor S3 API on the same host, and s3.path-style-access makes S3FileIO address buckets correctly on an S3-compatible endpoint. The same access keys appear twice because Spark talks to two endpoints: the rest.* keys sign catalog requests, and the s3.* keys sign data requests.
Match the iceberg-spark-runtime artifact to your Spark and Scala versions. The example uses Spark 3.5 with Scala 2.12. If Spark reports that it can’t load a region when it starts, add --conf spark.sql.catalog.aistor.client.region=local or set the AWS_REGION environment variable. Like the signing region, the value isn’t used by AIStor.
Once the session starts, the AIStor catalog is queryable with ordinary Spark SQL. Run SHOW NAMESPACES to confirm the connection. In a new warehouse it returns an empty list without an error. There is no metastore URI or JDBC connection string to configure.
Multi-Engine Access: Databricks, Trino, and the Broader Ecosystem
Multi-engine access is where an embedded catalog pays off across the whole stack. Engines reach AIStor tables in two ways: through the Iceberg REST Catalog, or through table sharing.
The first path is the Iceberg REST Catalog at /_iceberg. Most engines sign catalog requests with SigV4 using the s3tables signing name, as in the Spark example above. Engines that can’t sign with SigV4 authenticate with an OAuth2 bearer token from the catalog’s token endpoint, which is enabled by default. These engines can both read and write tables.
The second path is sharing. AIStor Table Sharing natively implements the Delta Sharing protocol, which has evolved into OpenSharing, so AIStor can share Delta and Iceberg tables directly with Databricks and other OpenSharing-compatible clients. Sharing uses its own protocol and endpoints, separate from the /_iceberg catalog, and recipients get live read access to the shared tables. One copy of the data stays on-premises in AIStor, and Databricks queries it in place without export pipelines.
Most other Apache Iceberg client libraries that support REST catalogs can also connect, as long as they support SigV4 signing with a custom signing name or OAuth2 bearer tokens. Check our engine connection guide for complete, tested configurations before committing to a specific tool.
The result is one AIStor deployment, one set of credentials and access policies, and many engines working with the same governed tables. No engine owns the data, and no single engine is required to reach it.
The engine details come from MinIO’s current connection guide. That includes the Trino property change, where Trino 477 and later use iceberg.rest-catalog.security=SIGV4 and Trino 476 and earlier use iceberg.rest-catalog.sigv4-enabled=true, and the DuckDB limit, which says to keep namespaces single-level for warehouses that DuckDB queries. The OpenSharing wording follows the launch coverage, which describes OpenSharing as the successor to the Delta Sharing protocol, with MinIO among the first on-premises storage vendors in the ecosystem.
Frequently Asked Questions
What Object Storage Works Best for Apache Iceberg?
We recommend S3-compatible object storage with tight Iceberg catalog integration. Iceberg was designed for object storage: it tracks every data file in its metadata instead of relying on directory listings or file renames, which are slow or non-atomic on object stores. The S3 API is also supported by every major Iceberg engine and client library.
When evaluating options, weigh four criteria: how the catalog is deployed and operated, which engines the implementation documents as supported, how catalog and data access are authenticated, and which Iceberg spec features the implementation supports.
AIStor Tables serves the Iceberg REST Catalog from the storage system itself, at its own /_iceberg endpoint alongside the S3 API, so there is no external catalog to deploy. It supports Iceberg V3 tables and Views, documents tested configurations for engines including Spark, Trino, Dremio, PyIceberg, and DuckDB, and uses the same AIStor credentials and access policies for both the catalog and the data, with SigV4 signing or OAuth2 bearer tokens.
Can I Run a Data Lakehouse On-Premises?
Yes. An on-premises lakehouse needs S3-compatible object storage, an Iceberg catalog, and compute engines, along with the networking and identity infrastructure to connect them. Because the pattern is built on open formats and open APIs, it runs the same way in your own data center as it does in a public cloud.
We built AIStor Tables to bring native Iceberg V3 support to on-premises object storage. The catalog ships inside AIStor, so an on-premises deployment doesn’t depend on a managed cloud catalog service. Tables are also replicated and backed up the same way as objects, so your disaster recovery plan covers table metadata and data together.
Engines connect the same way on-premises as anywhere else, through the /_iceberg endpoint with SigV4 signing or OAuth2 bearer tokens, whether the cluster sits in a colocation facility or a private cloud. For hybrid setups, AIStor Table Sharing lets cloud services such as Databricks query on-premises tables in place, so the data never has to leave your environment.
The replication point is supported by the AIStor Tables product page, which lists tables backed up and replicated like objects as a core benefit.
How Do AIStor Tables Compare to External Iceberg Catalogs?
The difference is where the catalog runs. AIStor Tables builds the Iceberg REST Catalog into the storage system, while external catalogs run as separate systems outside the object store. Some external catalogs, such as Nessie, Apache Polaris, and AWS Glue, also offer an Iceberg REST endpoint, so engines connect to them in a similar way. Hive Metastore uses its own Thrift interface, so moving from Hive Metastore to a REST catalog also changes how engines are configured.
The practical difference shows up in operations. With AIStor Tables, you don’t run Hive, Glue, Nessie, or a proprietary catalog service, and you don’t maintain a separate metadata database. Table metadata and data live in one AIStor deployment, governed by the same credentials and access policies, with SigV4 signing or OAuth2 bearer tokens for the catalog. For the trade-offs, including when a standalone catalog may still make sense, see Embedded Catalog vs. External Catalog Services above.
Getting Started with AIStor Tables
An Iceberg lakehouse needs four layers: durable object storage, an open table format, a catalog every engine can reach, and the engines themselves. AIStor Tables provides the storage and the catalog in one system and supports Iceberg V3 tables, so the only other piece you bring is your choice of engines.
The fastest way to evaluate it is to read the AIStor Tables announcement, create a warehouse in AIStor Tables, and then connect Spark using the configuration above and run a query. For Trino, DuckDB, and other engines, our engine connection guide has tested configurations.
Whether you’re exploring AI-native object storage or planning your next lakehouse deployment, we’d love to help. Let’s start a conversation, or jump right in and try AIStor yourself.
.svg)
