Home icon

Stream data from Amazon MSK to Apache Iceberg tables in Amazon S3 and Amazon S3 Tables using Amazon Data Firehose

Big Data Blog

This AWS Big Data Blog article details how to stream data from Amazon MSK to Apache Iceberg tables in Amazon S3 and Amazon S3 Tables using Amazon Data Firehose, offering two solution approaches for efficient data pipeline management.

  • Supports real-time streaming analytics across financial services, IoT, and ecommerce
  • Uses Apache Iceberg for SQL-like data lake processing with ACID transactions
  • Enables seamless data delivery from Amazon MSK to Iceberg tables without coding
  • Provides two streaming solutions:
    1. Direct streaming to Iceberg tables in Amazon S3
    2. Streaming to S3 Tables with Lake Formation integration
  • Offers automatic data transformation, compression, and error handling

The solution simplifies big data architectures by providing a fully managed, no-code approach to streaming data pipelines with advanced query capabilities.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 6
2024
Stream real-time data into Apache Iceberg tables in Amazon S3 using Amazon Data Firehose
Oct 1
2024
Amazon Data Firehose delivers data streams into Apache Iceberg format tables in Amazon S3
Mar 24
2025
Using Amazon S3 Tables with Amazon Redshift to query Apache Iceberg tables
Nov 15
2024
Amazon Data Firehose supports continuous replication of database changes to Apache Iceberg Tables in Amazon S3

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.