Introducing the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL
Big Data Blog
AWS announces general availability of the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL, which automatically analyzes completed Spark jobs and recommends configuration changes to reduce cost or improve performance.
- Analyzes Spark job runs to identify inefficiencies like data skew, memory pressure, shuffle issues, and suboptimal join strategies
- Provides two analysis modes: single job-run analysis and baseline comparison against known-good runs
- Accessible from EMR Studio UI, AWS Glue Studio UI, or MCP-compatible clients with no additional charge
- Recommendations labeled by type: [infra] for compute resources, [config] for Spark settings, [data] for physical data layout changes
- Tested across hundreds of workloads showing 24-31% cost reduction and 23-49% performance improvement
- Read-only agent that does not modify code or settings; users decide which recommendations to apply
The optimization agent eliminates manual Spark tuning by any engineer, replacing hours of expert analysis with automated recommendations grounded in job telemetry.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.