Building a serverless pipeline to deliver reliable messaging
Compute Blog
This post describes an architecture for building a reliable, high-performance serverless streaming pipeline to manage large audit records between 256k and 6MB in size. Specifically, the article covers:
- Ingesting large messages with low latency using API Gateway and AWS Lambda
- Orchestrating message processing with AWS Step Functions
- Storing message data in Amazon S3 and metadata in Amazon DynamoDB
- Providing traceability through Amazon SNS notifications at different stages
- Ensuring high data durability by leveraging multiple services
- Conclusion and deployment instructions
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Mar 21
2024
2024
Build an end-to-end serverless streaming pipeline with Apache Kafka on Amazon MSK using Python
Jun 2
2025
2025
Build a secure serverless streaming pipeline with Amazon MSK Serverless, Amazon EMR Serverless and IAM
Mar 24
2026
2026
Building a Scalable Messaging API with AWS End User Messaging and SES
Apr 15
2026
2026
Modernizing life-saving workloads with AWS serverless
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.