Amazon OpenSearch Ingestion 101: Set CloudWatch alarms for key metrics

This article provides a comprehensive guide to setting up CloudWatch alarms for monitoring Amazon OpenSearch Ingestion pipelines, covering key metrics across sources, processors, sinks, and buffers.

• Monitor source metrics like requestsTooLarge, requestsRejected, and source-specific errors for HTTP, S3, DynamoDB, DocumentDB, Kafka, and other sources
• Track processor performance using metrics like Lambda invocation failures, Grok pattern mismatches, and processing time thresholds
• Set sink alarms for bulk request errors, S3 write failures, and Dead Letter Queue issues to catch delivery problems
• Monitor buffer usage and persistent buffer lag to identify pipeline bottlenecks and latency issues
• Use DLQ examination to diagnose document errors like mapping conflicts and field limit violations
• Analyze bulkRequestLatency and processor timeElapsed metrics to identify performance bottlenecks
• Review ERROR and WARN-level logs for detailed exception information when alarms trigger

The guide emphasizes proactive monitoring through appropriate alarm thresholds and regular log review to maintain healthy OpenSearch Ingestion pipelines and optimize data flow performance.