Build an end-to-end serverless streaming pipeline with Apache Kafka on Amazon MSK using Python
Big Data Blog
This article explains how to build a serverless data streaming pipeline using AWS services like Amazon Managed Streaming for Apache Kafka (MSK), AWS Lambda, Amazon DynamoDB, and Amazon Kinesis Data Firehose. It outlines an example scenario of monitoring modem connectivity and processing real-time telemetry data from modems.
Specifically, the article covers:
- Creating a serverless Apache Kafka cluster on Amazon MSK
- Creating a Kafka topic and producing records using Python
- Storing events in Amazon S3 using Kinesis Data Firehose
- Storing recent events in DynamoDB using AWS Lambda
- Benefits of using a serverless architecture with automatic scalability and pay-as-you-go pricing
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Jun 2
2025
2025
Build a secure serverless streaming pipeline with Amazon MSK Serverless, Amazon EMR Serverless and IAM
Jun 26
2025
2025
Building serverless event streaming applications with Amazon MSK and AWS Lambda
Mar 27
2024
2024
How Cornerstone Built a Full-Fledged Apache Kafka Replication Using Amazon MSK Connect
Jan 16
2024
2024
Real-time serverless data ingestion from your Kafka clusters into Amazon Timestream using Kafka Connect
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.