Enhance monitoring and debugging for AWS Glue jobs using new job observability metrics, Part 2: Real-time monitoring using Grafana
Big Data Blog
The article provides a guide on how to integrate AWS Glue job observability metrics with Grafana using Amazon Managed Grafana for real-time monitoring of data pipelines built on AWS Glue.
Specifically, the article covers:
- Solution overview and architecture for integrating AWS Glue job metrics with Grafana
- Steps to implement the solution by setting up Amazon Managed Grafana workspace, configuring CloudWatch data source, and importing a pre-built Grafana dashboard
- Using the Grafana dashboard to analyze causes of job failures by viewing error metrics, resource utilization, and data throughput
- (Optional) Configuring a cross-account setup to monitor AWS Glue jobs in a different account
- Considerations when using this solution, such as Grafana's dependency on IAM Identity Center and its pricing model
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Mar 29
2024
2024
Enhance monitoring and debugging for AWS Glue jobs using new job observability metrics, Part 3: Visualization and trend analysis using Amazon QuickSight
Mar 31
2026
2026
Amazon SageMaker Unified Studio adds Observability for AWS Glue jobs via CloudWatch metrics
Jul 10
2024
2024
Centralize observability with Amazon Managed Grafana Enterprise plugins
Sep 11
2024
2024
Grafana Labs Kubernetes Monitoring now available in AWS Marketplace
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.