Introducing the Apache Spark Optimization Agent for AWS Glue ETL
News
This article introduces the Apache Spark Optimization Agent for AWS Glue ETL, which analyzes completed Spark job runs and delivers optimization recommendations within minutes.
- Analyzes job configuration, logs, and Spark telemetry to identify performance bottlenecks and cost drivers
- Provides specific configuration recommendations with evidence and reasoning for each suggestion
- Identifies issues like insufficient parallelism, skewed joins, memory pressure, idle executors, and over-provisioning
- Balances cost and performance trade-offs based on user-selected goals
- Checks best practices including AWS Glue version upgrades and AWS Graviton processor usage
- Available at no additional charge for AWS Glue 4.0 and later in all regions except GovCloud and China
- Accessible via AWS Glue console or Amazon SageMaker Unified Studio MCP Server
The agent helps optimize Spark workloads by providing data-driven recommendations that improve either cost or performance while maintaining acceptable trade-offs.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Nov 18
2026
2026
Introducing the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL
Nov 17
2026
2026
Introducing the Apache Spark Optimization Agent for Amazon EMR Serverless
Nov 10
2026
2026
Automate ETL pipeline creation with Amazon Bedrock AgentCore and AWS Glue
Nov 13
2026
2026
AWS Glue announces automatic version upgrades for Apache Spark jobs
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.