Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service
Big Data Blog
This article demonstrates how to monitor MWAA-orchestrated ETL pipelines by streaming logs from multiple AWS services into Amazon OpenSearch Service and using an AI agent for natural language root cause analysis.
- Centralize logs from Amazon MWAA, AWS Glue, and Amazon EC2 into OpenSearch using CloudWatch subscription filters
- Deploy three CloudFormation stacks to provision OpenSearch domain, ETL infrastructure, and Bedrock AgentCore MCP server
- Use AWS Lambda to decompress and transform CloudWatch logs for real-time indexing into OpenSearch
- Query logs with natural language through an OpenSearch MCP server integrated with Amazon Bedrock Claude model
- Receive root cause analysis and remediation recommendations from the AI agent in a single query
- Reduce mean time to resolution by eliminating manual log group navigation and timestamp matching
This solution enables data engineers and DevOps teams to troubleshoot multi-service ETL failures faster by providing unified, AI-powered observability without infrastructure maintenance.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2024
2024
2026
2024
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.