Spark on Amazon EKS networking – Part 2
Containers Blog
This article discusses Pinterest's journey in designing and implementing their networking topology for running large-scale Spark workloads on Amazon Elastic Kubernetes Service (Amazon EKS).
Specifically, the article covers:
- The design options considered for cluster networking, VPC networking, and ingress/egress strategy
- The chosen design: custom networking with unroutable CGNAT IP ranges for pods, single VPC per cell with VPC peering (evolving to Transit Gateway), and ingress routing via EKS ingress controller with NGINX service
- Cell sizing considerations like rebuild time, NAUs, IP limits, and mutating API throttling
- The collaborative design process involving multiple teams at Pinterest and AWS
- Conclusion and future plans for further scaling and adoption of IPv6
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.