Home icon

Stream data from Amazon MSK to Apache Iceberg tables in Amazon S3 and Amazon S3 Tables using Amazon Data Firehose

Big Data Blog

This AWS Big Data Blog article details how to stream data from Amazon MSK to Apache Iceberg tables in Amazon S3 and Amazon S3 Tables using Amazon Data Firehose, offering two solution approaches for efficient data pipeline management.

  • Supports real-time streaming analytics across financial services, IoT, and ecommerce
  • Uses Apache Iceberg for SQL-like data lake processing with ACID transactions
  • Enables seamless data delivery from Amazon MSK to Iceberg tables without coding
  • Provides two streaming solutions:
    1. Direct streaming to Iceberg tables in Amazon S3
    2. Streaming to S3 Tables with Lake Formation integration
  • Offers automatic data transformation, compression, and error handling

The solution simplifies big data architectures by providing a fully managed, no-code approach to streaming data pipelines with advanced query capabilities.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 6
2024
Stream real-time data into Apache Iceberg tables in Amazon S3 using Amazon Data Firehose
Oct 1
2024
Amazon Data Firehose delivers data streams into Apache Iceberg format tables in Amazon S3
Jul 30
2026
Amazon MSK Express brokers now deliver data to streaming tables for Apache Iceberg
Jul 30
2026
Deliver Apache Kafka data to streaming tables for Apache Iceberg with Amazon MSK Express brokers

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.