Home icon

Unlock real-time data insights with schema evolution using Amazon MSK Serverless, Iceberg, and AWS Glue streaming

Big Data Blog

This article discusses a comprehensive solution for real-time data insights using Amazon MSK Serverless, Apache Iceberg, and AWS Glue streaming, focusing on schema evolution in data lakes.

  • Solution enables efficient real-time data synchronization and schema modifications
  • Utilizes Amazon MSK Serverless, Debezium MySQL connector, and AWS Glue for CDC pipeline
  • Implements automatic schema evolution using Apache Iceberg table format
  • Supports adding new columns without disrupting data processing
  • Enables near real-time data transfer from RDS to Iceberg tables via Kafka

The architecture provides a flexible, scalable approach to handling streaming data with dynamic schema changes, reducing operational complexity and maintaining data pipeline continuity.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Aug 26
2025
Amazon Timestream for InfluxDB: Expanding managed open source time series databases for data-driven insights and real-time decision making
Mar 21
2024
Unlock insights on Amazon RDS for MySQL data with zero-ETL integration to Amazon Redshift
Feb 3
2026
Use Amazon MSK Connect and Iceberg Kafka Connect to build a real-time data lake
Nov 6
2024
Stream real-time data into Apache Iceberg tables in Amazon S3 using Amazon Data Firehose

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.