Stream data from Amazon DocumentDB to Amazon Kinesis Data Firehose using AWS Lambda
Blog
This article explains how to stream data changes from Amazon DocumentDB to Amazon S3 using AWS Lambda and Kinesis Data Firehose (now Amazon Data Firehose).
- Lambda reads DocumentDB change stream events via event source mappings
- Lambda function processes changes and pushes data to Firehose delivery stream
- Firehose automatically archives data to S3 with automatic prefixing by date/hour
- Setup requires DocumentDB cluster, Lambda function, S3 bucket, and Firehose delivery stream
- Change streams must be enabled on DocumentDB collections to capture insert/update/delete operations
- Lambda needs VPC endpoints for DocumentDB and Secrets Manager connectivity
- Secrets Manager stores DocumentDB credentials for secure Lambda access
- UpdateLookup option captures full document state but has performance overhead
- Firehose has limits: 2,000 transactions/sec, 5,000 records/sec, 5 MB/sec per stream
- Solution enables archiving, backup, and analytics use cases with minimal code
This approach provides a serverless, low-code solution for continuously archiving DocumentDB changes to S3 for analytics, backup, or data warehouse integration.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.