Announcing Spark Connect on Amazon EMR on EC2: Interactive PySpark anywhere
Big Data Blog
This article announces Spark Connect support on Amazon EMR on EC2, enabling interactive PySpark development from local IDEs or SageMaker Unified Studio Data Notebooks.
- Develop and debug PySpark interactively using Visual Studio Code, PyCharm, Jupyter, or SageMaker Unified Studio while Spark runs on a dedicated cluster
- Spark Connect uses client-server architecture with gRPC/TLS to separate application code from the Spark engine
- Single cluster supports up to 1,000 concurrent sessions with isolated execution roles, tags, and lifecycles for team sharing
- Available with emr-spark-8.0 (Apache Spark 4.0.2 and later) in all AWS Regions except GovCloud and China
- Supports DataFrame and SQL APIs; RDD-based APIs not supported; authentication tokens expire after 1 hour
- No additional charge beyond Amazon EC2 instance costs for the cluster
Spark Connect eliminates version mismatches and slow deploy-check loops by running code against the cluster's Spark engine while developing locally, enabling consistent debugging and production-scale data analysis.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2026
2025
2026
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.