Announcing Spark Connect on Amazon EMR on EKS: Interactive PySpark development, anywhere

This article announces support for Spark Connect on Amazon EMR on EKS, starting from EMR release 7.14 and emr-spark-8.1, enabling interactive PySpark development from preferred tools like VS Code, PyCharm, and Jupyter notebooks.

• Spark Connect separates client from server, allowing local development while operations run on scalable EKS clusters
• Supports interactive development, embedded Spark in applications, multi-tenant data exploration with governance, and CI/CD data-quality tests
• Client-server architecture uses secure gRPC/TLS channels with managed authentication-proxy router and Secret Agent service
• Each session uses its own IAM execution role, custom tags, and cost tracking for security and compliance
• Enables multi-cluster, multi-Region, and hybrid architectures without requiring direct network access to data stores
• Available in all AWS Regions where EMR on EKS is available, with no additional charges beyond standard EMR pricing
• Includes step-by-step setup guide covering namespace creation, security configuration, virtual cluster setup, and endpoint management

Spark Connect on EMR on EKS extends interactive and embedded workloads to existing Kubernetes clusters with built-in security, multi-tenancy, and observability.