Home icon

Introducing the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL

Big Data Blog



AWS announces general availability of the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL, which automatically analyzes completed Spark jobs and recommends configuration changes to reduce cost or improve performance.

  • Analyzes Spark job runs to identify inefficiencies like data skew, memory pressure, shuffle issues, and suboptimal join strategies
  • Provides two analysis modes: single job-run analysis and baseline comparison against known-good runs
  • Accessible from EMR Studio UI, AWS Glue Studio UI, or MCP-compatible clients with no additional charge
  • Recommendations labeled by type: [infra] for compute resources, [config] for Spark settings, [data] for physical data layout changes
  • Tested across hundreds of workloads showing 24-31% cost reduction and 23-49% performance improvement
  • Read-only agent that does not modify code or settings; users decide which recommendations to apply

The optimization agent eliminates manual Spark tuning by any engineer, replacing hours of expert analysis with automated recommendations grounded in job telemetry.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Nov 17
2026
Introducing the Apache Spark Optimization Agent for Amazon EMR Serverless
Dec 16
2025
Introducing the Apache Spark troubleshooting agent for Amazon EMR and AWS Glue
Dec 16
2025
Introducing Apache Spark upgrade agent for Amazon EMR
Dec 2
2025
Announcing the Apache Spark upgrade agent for Amazon EMR

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.