Home icon

Build an end-to-end serverless streaming pipeline with Apache Kafka on Amazon MSK using Python

Big Data Blog

This article explains how to build a serverless data streaming pipeline using AWS services like Amazon Managed Streaming for Apache Kafka (MSK), AWS Lambda, Amazon DynamoDB, and Amazon Kinesis Data Firehose. It outlines an example scenario of monitoring modem connectivity and processing real-time telemetry data from modems.

Specifically, the article covers:

  • Creating a serverless Apache Kafka cluster on Amazon MSK
  • Creating a Kafka topic and producing records using Python
  • Storing events in Amazon S3 using Kinesis Data Firehose
  • Storing recent events in DynamoDB using AWS Lambda
  • Benefits of using a serverless architecture with automatic scalability and pay-as-you-go pricing
Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Jun 2
2025
Build a secure serverless streaming pipeline with Amazon MSK Serverless, Amazon EMR Serverless and IAM
Jun 26
2025
Building serverless event streaming applications with Amazon MSK and AWS Lambda
Nov 21
2025
Improving throughput of serverless streaming workloads for Kafka
Oct 31
2024
Build a streaming ETL pipeline on Amazon RDS using Amazon MSK

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.