MinIO Blog

Built a portable Java data pipeline using MinIO's AIStor and Kafka that scales from Mac to Kubernetes. Containerized stack (Kafka, AIStor, Prometheus, Grafana) processes millions of events, preserving raw data for analysis while delivering real-time dashboard summaries.

AIStor is a foundational component for creating and executing complex data workflows. At the core of this event-driven functionality is MinIO bucket notifications using Kafka.

Build a streaming Change Data Capture (CDC) pipeline with Redpanda and MinIO into Snowflake. This solution simplifies data migration and analytics, with Redpanda offering scalability, MinIO as efficient storage, and Snowflake as a cloud-native analytics engine.

Explore the next generation of data streaming with WarpStream and MinIO! While Apache Kafka has been the standard for streaming data, it may be time to consider a simpler, more cost-effective, and cloud-native solution.

Apache Kafka and Apache Spark are two leading technologies used to build the streaming data pipelines that feed data lakes and lake houses. At a really high level, Kafka streams messages to Spark where they are transformed into a format that can be read in by applications and saved to storage.

Kafka and Spark Structured Streaming are used together to build data lakes/lake houses fed by streaming data and provide real time business insights.