Spark on Amazon EKS networking – Part 1
Containers Blog
This article discusses the design process used by Pinterest and AWS to create a networking solution for running large-scale Spark workloads on Amazon Elastic Kubernetes Service (Amazon EKS) while addressing known scale limits.
Specifically, the article covers:
- Scoping the problem: Pinterest faced challenges with IP exhaustion, API request rate limits, and network address usage when running Spark on Amazon EKS at scale.
- Design tenets: A set of goals and principles were established, such as addressing scale limits, using a modern scalable architecture, optimizing for cost, and minimizing operational overhead.
- Design process: An iterative, scenario-based approach was followed, including listing design options, creating prototypes, evaluating against scenarios and tenets, seeking stakeholder buy-in, and making a joint decision.
- Known scale limits: Key limitations like IPv4 exhaustion, AWS API quotas, network address utilization, and control plane scaling were identified.
- Conclusion: The design process was crucial for defining the problem, establishing boundaries, aligning stakeholders, and collaborating across teams.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.