Introducing Apache Spark troubleshooting agent for Amazon EMR on EKS

This article introduces the Apache Spark troubleshooting agent for Amazon EMR on EKS, which helps diagnose failed Spark applications using natural language prompts.

• Automatically retrieves Spark logs from S3 or CloudWatch and correlates signals across systems to identify root causes
• Provides code recommendations for application-level failures in PySpark workloads
• Accessible from Amazon EMR console or MCP-compatible AI assistants like Claude Code and Kiro CLI
• Diagnoses out-of-memory errors, data skew, configuration issues, and code-level problems
• Fully managed service with no additional cost; uses Amazon Bedrock LLM with RAG-based knowledge base
• Reduces mean-time-to-resolution from days to minutes by eliminating manual log correlation

The troubleshooting agent unifies Spark diagnostics across EMR on EKS, EC2, Serverless, and AWS Glue, reducing incident investigation time and requiring only IAM role setup.