Build a streaming ETL pipeline on Amazon RDS using Amazon MSK

Database Blog

This article discusses how to build a streaming ETL pipeline on Amazon RDS using Amazon MSK (Managed Streaming for Apache Kafka). It outlines an architecture pattern for creating a real-time data pipeline using Amazon MSK to ingest and process streaming data from RDS for SQL Server to Amazon Redshift.

Specifically, the article covers:

Streaming data pipeline architecture overview
Solution overview with detailed steps
Prerequisites for deploying the solution
Deployment steps using a CloudFormation template
Configuring RDS for SQL Server for change data capture (CDC)
Setting up the Amazon MSK cluster and Debezium connector
Creating a materialized view in Amazon Redshift for real-time data ingestion
Optional: Using AWS Glue Streaming for data transformation
Clean-up steps for deleting the deployed resources

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Mar 21
2024

Build an end-to-end serverless streaming pipeline with Apache Kafka on Amazon MSK using Python

Feb 21
2024

Simplify data streaming ingestion for analytics using Amazon MSK and Amazon Redshift

Jun 20
2025

Stream data from Amazon MSK to Apache Iceberg tables in Amazon S3 and Amazon S3 Tables using Amazon Data Firehose

Jun 2
2025

Build a secure serverless streaming pipeline with Amazon MSK Serverless, Amazon EMR Serverless and IAM

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Build a streaming ETL pipeline on Amazon RDS using Amazon MSK

Related articles