MinIO | MinIO for Kubernetes Storage

Abstract watercolor splash with red, pink, and purple brush strokes

Customers run MinIO on Kubernetes for three reasons:

Deploy, manage and secure S3-like infrastructure where MinIO provides object storage and Kubernetes provides compute infrastructure.

Turnkey multi-cluster deployment and management of DevOps tools, providing freedom to innovate without lock-in or disruption while ensuring a consistent developer experience across locations, clouds and platforms.

Using MinIO for Kubernetes storage provides control over the software stack with flexibility to avoid cloud lock-in and provide consistent object storage across hybrid and multi-cloud.

MinIO is high-performance Kubernetes-native object storage that is compatible with the S3 API. We recommend using MinIO wherever you need complete S3 API functionality for object storage on Kubernetes. MinIO provides a single global namespace and a consistent object storage interface across multiple cloud providers, on premise and at the edge.

MinIO natively integrates with Kubernetes to streamline operations for large scale multi-tenant object storage as a service, across multiple clouds and at the edge. MinIO can be managed through multiple tools. In Kubernetes environments, AIStor Operator and kubectl plugin simplify deployment and management for DevOps and infrastructure teams.

With Kubernetes as its engine, MinIO is able to run anywhere Kubernetes does - which in the modern, cloud-native world, is essentially everywhere. See the following Kubernetes powered environments with detailed information on the integration:

Architecture

While MinIO is integrated with other Kubernetes environments, we have always supported the developer who is interested in creating customer architectures with Kubernetes. Our stock Kubernetes architecture is as follows:

MinIO provides a consistent, performant and scalable object store for any Kubernetes distribution. MinIO is Kubernetes-native by design and S3 compatible from inception. Developers can quickly deploy persistent object storage for all of their cloud native applications. The combination of MinIO and Kubernetes provides a powerful platform that allows applications to scale across any multi-cloud and hybrid cloud infrastructure and still be centrally managed and secured, avoiding public cloud lock-in.

The AIStor Operator integrates natively with Kubernetes to provide:

Storage Classes and Tiering‍

Tier across NVMe, HDD and Public Cloud Storage.

The key requirement to deploy MinIO at scale is the ability to tier across Kubernetes storage classes (NVMe, HDD, Public Cloud). This allows enterprises to manage both cost and performance.

MinIO supports automatic transition of aged objects from the fast NVMe tier to a more cost-efficient HDD tier and even cost-optimized cold Public Cloud storage tiers.

When tiering, MinIO presents a unified namespace across the tiers. Movement across the tiers is transparent to the application and is triggered by customer policies.

MinIO and Kubernetes enable hybrid and multi-cloud storage safely and securely by encrypting objects at the source - ensuring customers retain total control over the data. Kubernetes efficiently manages data across persistent block storage and cheaper object storage tiers when deployed inside the public cloud.

External Load Balancing

Load balance incoming requests with NGINX ingress controller.

All of MinIO’s communication is based on HTTPs, RESTFUL APIs and will support any standard, Kubernetes compatible ingress controller. This includes hardware based and software defined solutions. The most popular choice is NGINX.

Encryption Key Management

Manage encryption keys with HashiCorp Vault.

We recommend using HashiCorp Vault to store keys outside of the object storage system. This is a best practice for cloud native applications.

We recommend encryption be enabled by default on all buckets in production environments. MinIO uses AES-256-GCM or ChaCha20-Poly1305 encryption to protect data integrity and confidentiality with negligible performance impact.

MinIO supports all of the three server-side encryption (SSE-KMS, SSE-S3 and SSE-C) modes. SSE-S3 and SSE-KMS integrate with the KMS on the server side, whereas SSE-C uses the client supplied keys. MinIO supports setting a bucket-level default encryption key in the KMS with support for per-object encryption keys (SSE-KMS) and a broad per-deployment encryption key (SSE-S3).

MinIO relies on an external KMS to bootstrap its internal key encryption server (KES service) to enable high-performance, per object encryption. Each tenant runs its own KES server in an isolated namespace.

Identity Management

Manage identity and policy with OpenID Connect compatible Keycloak IDP.

Manage single sign-on (SSO) for Kubernetes and MinIO through a third party OpenID Connect/LDAP compatible identity provider, for example Keycloak, Okta/Auth0, Google, Facebook, ActiveDirectory and OpenLDAP. MinIO recommends OpenID Connect compatible Keycloak IDP.

Administrators can centrally manage user/application identity using an external IDP. MinIO enhances the IDP, providing AWS IAM-style users, groups, roles, policies and token service API. Enterprises gain significant architectural flexibility with an infrastructure independent and unified identity and access management (IAM) layer.

Certificate Management

Configure and manage certificates with Rancher Certificate Manager and Let's Encrypt.

TLS is used to encrypt all traffic, including internode traffic, between applications and MinIO. TLS certificates establish the identity of network-connected resources, such as a MinIO server domain, and secure network communications.

The AIStor Operator automatically configures, provisions, manages and updates certificates for MinIO tenants. The tenants are completely isolated from each other in their own Kubernetes namespace with their own certificates for improved security.

Monitoring and Alerting

Track Metrics and issue alerts using Rancher Monitoring or Grafana.

MinIO recommends using Prometheus-compatible systems for monitoring and alerting when running on Kubernetes. MinIO publishes every object storage related Prometheus metric imaginable, from bucket capacity to access metrics. Those metrics can be collected and visualized in any Prometheus-compatible tool or the MinIO Console.

External monitoring solutions scrape the MinIO Prometheus endpoint at regular intervals. MinIO recommends Grafana to monitor the Prometheus feed in MinIO. Administrators establish baselines and set alert thresholds for notifications, which can then be routed via Alertmanager to a notification platform such as PagerDuty, Freshservice or even SNMP.

Logging and Auditing

Output logs to an Elastic Stack for analysis.

Enabling MinIO auditing generates a log for every operation on the object storage cluster. In addition to the audit log, MinIO also logs console errors for operational troubleshooting purposes.

MinIO supports outputting logs to the Elastic Stack (or third parties) for analysis and alerting. To streamline operations, we recommend using the same logging and audit tool for Kubernetes and MinIO.