Root Cause Analysis with Amazon Managed Service for Prometheus and AWS DevOps Agent

This article demonstrates building a proactive root cause analysis (RCA) pipeline that combines Amazon Managed Service for Prometheus machine learning-based anomaly detection with AWS DevOps Agent to automatically investigate incidents without human intervention.

• RCF anomaly detectors learn normal metric baselines and automatically detect anomalies instead of using static thresholds
• AWS DevOps Agent investigates root causes autonomously using Prometheus MCP tools to query metrics and correlate signals
• Architecture spans Amazon EKS, Managed Service for Prometheus, Lambda webhook forwarder, API Gateway, and AWS DevOps Agent
• In a 5G network degradation test scenario, the pipeline detected a 95% throughput drop, fired an alert within 25 seconds, and delivered an RCA report in under 60 seconds
• AWS DevOps Agent Skills encode investigation runbooks as structured diagnostic steps for every RCF-triggered investigation
• Prometheus MCP server runs as Lambda behind API Gateway with OAuth authentication via Amazon Cognito
• RCF detects anomaly onsets (not sustained anomalies) and scores spike then decay within 60-90 seconds as the algorithm adapts

The pipeline shifts from reactive firefighting to proactive autonomous investigation, enabling teams to reduce alert fatigue and dramatically accelerate time-to-diagnosis for performance degradations.