
Build a near-real-time transactional data lake using AWS DMS, Kinesis, Glue streaming ETL, and Apache Hudi to capture CDC changes from RDS and visualize data with QuickSight.


<div><p>This article demonstrates building a near-real-time transactional data lake using Apache Hudi, AWS DMS, Kinesis, Glue streaming ETL, and QuickSight visualization.</p><ul><li>AWS DMS captures CDC changes from RDS to Kinesis Data Streams</li><li>AWS Glue 4.0 streaming jobs read, enrich, and upsert data to S3 in Hudi format</li><li>Apache Hudi enables ACID transactions, time travel queries, and upsert/delete operations</li><li>CloudFormation template automates setup of RDS, DMS, Kinesis, Glue jobs, and networking</li><li>Data validated via Athena queries; visualized in QuickSight dashboards</li><li>Supports both new record ingestion and real-time updates to existing records</li></ul><p>This solution enables organizations to replicate relational database changes to S3 data lakes in near-real-time with full transactional support and analytics capabilities.</p></div>


Create an Apache Hudi-based near-real-time transactional data lake using AWS DMS, Amazon Kinesis, AWS Glue streaming ETL, and data visualization using Amazon QuickSight

Related articles