Home icon

How Veeva scaled big data with Amazon EMR on EKS

Big Data Blog



This article describes how Veeva Systems scaled its Compass analytics platform to handle petabyte-scale data processing by migrating from Amazon EMR on EC2 to Amazon EMR on EKS with Karpenter.

  • Replaced always-on EMR clusters with ephemeral Kubernetes jobs that scale from 40 to 520 nodes in under 6 minutes
  • Implemented Karpenter for intelligent EC2 instance provisioning across 20+ instance families, achieving ~70% Spot instance utilization
  • Reduced bronze-to-silver ETL runtime by 55% (9h 12m to 4h 7m) and eliminated $42,000 monthly idle compute costs
  • Optimized data partitioning, Delta Lake compaction, and Spark Adaptive Query Execution to reduce skewed join stages from 7 hours to 55 minutes
  • Built multi-layer observability stack with Spark History Server, Prometheus, CloudWatch, and OpenTelemetry for production SLA management
  • Enforced security through IAM Roles for Service Accounts, Kubernetes namespaces, AWS KMS encryption, and data lineage tracking

The migration demonstrates how cloud-native architectures with intelligent autoscaling enable cost-effective, high-performance analytics at petabyte scale.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Oct 23
2026
Amazon EMR on EKS reduces EMR charge by 30 percent
May 28
2024
Introducing Amazon EMR on EKS with Apache Flink: A scalable, reliable, and efficient data processing platform
Nov 16
2026
How Salesforce efficiently right-sizes Kubernetes workloads at scale on Amazon EKS
Sep 24
2026
Run interactive workloads on Amazon EMR on EKS with Spark Connect

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.