How Nielsen uses serverless concepts on Amazon EKS for big data processing with Spark workloads
Architecture Blog
Nielsen Marketing Cloud transformed their big data processing by implementing a serverless-inspired architecture on Amazon EKS for handling Spark workloads. They addressed significant challenges in scaling and performance by reimagining their data processing approach.
- Original system suffered from performance degradation and remote data shuffle issues
- Moved from large Spark clusters to multiple local mode Spark clusters on EKS
- Implemented a workflow using SQS queues, Lambda functions, and EKS pods
- Utilized Kubernetes Horizontal Pod Autoscaler to dynamically scale processing capacity
- Achieved 130% performance improvement per instance
- Reduced processing costs by 55%
The new architecture enables linear scaling, improved efficiency, and significant cost savings by breaking down large data processing jobs into smaller, independent workloads.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2025
2025
2025
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.