Home icon

Build a centralized observability platform for Apache Spark on Amazon EMR on EKS using external Spark History Server

Big Data Blog

This AWS Big Data Blog article details how to build a centralized observability platform for Apache Spark on Amazon EMR on EKS using an external Spark History Server (SHS), providing a unified monitoring solution across multiple clusters.

  • Solution enables collecting Spark events from multiple EMR on EKS clusters into a central S3 bucket
  • Deploys Spark History Server on a dedicated Amazon EKS cluster
  • Secures access using AWS Load Balancer Controller, AWS Private CA, Route 53, and AWS Client VPN
  • Integrates DataFlint for enhanced performance monitoring and insights
  • Provides a single, secure interface to monitor and troubleshoot Spark applications

The solution addresses the complexity of monitoring Spark applications across diverse workloads by creating a centralized, secure, and comprehensive observability platform.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

May 1
2025
Build end-to-end Apache Spark pipelines with Amazon MWAA, Batch Processing Gateway, and Amazon EMR on EKS clusters
May 12
2026
Implement centralized observability for multi-account Amazon EKS
Jul 10
2026
Amazon EMR on EKS now supports Apache Spark troubleshooting agent
May 28
2024
Introducing Amazon EMR on EKS with Apache Flink: A scalable, reliable, and efficient data processing platform

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.