AWS Glue now provides enhanced observability metrics for Apache Spark jobs, which provide fine-grained insights into performance, reliability, resource utilization, error classification, and data throughput, enabling improved monitoring, debugging, and optimization of data pipelines.

<div>
<p>This article describes new enhanced CloudWatch metrics for monitoring and debugging AWS Glue Apache Spark jobs. These metrics provide insights into job health, performance, throughput, and resource utilization, enabling better visibility and control over data pipelines.</p>
<p>Specifically, the article covers:</p>
<ul>
<li>Enabling the new metrics through the AWS Glue console or APIs</li>
<li>A use case demonstrating how the metrics help optimize resource utilization and cost</li>
<li>Querying and visualizing the metrics in CloudWatch</li>
<li>Creating CloudWatch alarms based on the metrics</li>
<li>Other available enhanced metrics for error analysis, throughput, etc.</li>
<li>Using dashboards in Amazon Managed Grafana and Amazon QuickSight for enhanced observability</li>
</ul>
</div>


Enhance monitoring and debugging for AWS Glue jobs using new job observability metrics

Related articles