How Slack achieved operational excellence for Spark on Amazon EMR using generative AI
Big Data Blog
This article describes how Slack built an AI-powered monitoring framework for Apache Spark on Amazon EMR to optimize performance and reduce costs.
- Custom Spark listener framework captures 40+ metrics across application, job, stage, and task levels
- Real-time metrics streamed to Kafka, processed via Spark SQL into actionable insights
- Five Pillars of Spark Monitoring framework: metadata, configuration, performance, execution, resource usage
- AI-assisted tuning via Amazon Bedrock and MCP server integration with developer tools
- Structured custom prompts prevent hallucinations and ensure deterministic recommendations
- Results: 30-50% cost reduction, 40-60% faster job completion, 90% less developer tuning time
Slack transformed Spark optimization from manual trial-and-error to automated, data-driven analysis using comprehensive metrics and generative AI recommendations.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Apr 18
2024
2024
Slack delivers native and secure generative AI powered by Amazon SageMaker JumpStart
Jan 23
2026
2026
Bridging the Knowledge Gap: Using Generative AI on AWS to Preserve Critical Expertise
Feb 4
2026
2026
Generative AI for operations leaders: How Domo and Amazon Bedrock optimize efficiency
Jan 23
2026
2026
How the Amazon.com Catalog Team built self-learning generative AI at scale with Amazon Bedrock
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.