Enhance monitoring and debugging for AWS Glue jobs using new job observability metrics, Part 2: Real-time monitoring using Grafana
Big Data Blog
The article provides a guide on how to integrate AWS Glue job observability metrics with Grafana using Amazon Managed Grafana for real-time monitoring of data pipelines built on AWS Glue.
Specifically, the article covers:
- Solution overview and architecture for integrating AWS Glue job metrics with Grafana
- Steps to implement the solution by setting up Amazon Managed Grafana workspace, configuring CloudWatch data source, and importing a pre-built Grafana dashboard
- Using the Grafana dashboard to analyze causes of job failures by viewing error metrics, resource utilization, and data throughput
- (Optional) Configuring a cross-account setup to monitor AWS Glue jobs in a different account
- Considerations when using this solution, such as Grafana's dependency on IAM Identity Center and its pricing model
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Mar 29
2024
2024
Enhance monitoring and debugging for AWS Glue jobs using new job observability metrics, Part 3: Visualization and trend analysis using Amazon QuickSight
Mar 31
2026
2026
Amazon SageMaker Unified Studio adds Observability for AWS Glue jobs via CloudWatch metrics
Feb 15
2024
2024
Achieve Complete Data Observability on AWS with TCS Approach to Observability Challenges
Jan 25
2024
2024
Building Observability on Amazon Managed Grafana for 5G O-RAN sites built on EKS-Anywhere
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.