Home icon

Unlock real-time data insights with schema evolution using Amazon MSK Serverless, Iceberg, and AWS Glue streaming

Big Data Blog

This article discusses a comprehensive solution for real-time data insights using Amazon MSK Serverless, Apache Iceberg, and AWS Glue streaming, focusing on schema evolution in data lakes.

  • Solution enables efficient real-time data synchronization and schema modifications
  • Utilizes Amazon MSK Serverless, Debezium MySQL connector, and AWS Glue for CDC pipeline
  • Implements automatic schema evolution using Apache Iceberg table format
  • Supports adding new columns without disrupting data processing
  • Enables near real-time data transfer from RDS to Iceberg tables via Kafka

The architecture provides a flexible, scalable approach to handling streaming data with dynamic schema changes, reducing operational complexity and maintaining data pipeline continuity.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Oct 3
2025
Unlock real-time advertising insights with Amazon Marketing Stream and AWS
Oct 24
2025
Implement fine-grained access control for Iceberg tables using Amazon EMR on EKS integrated with AWS Lake Formation
Nov 18
2025
Amazon Redshift now supports Just-In-Time (JIT) ANALYZE for Apache Iceberg tables
Oct 17
2025
How TP ICAP transformed CRM data into real-time insights with Amazon Bedrock

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.