Introducing Apache Spark Connect support in AWS Glue interactive sessions
Big Data Blog
AWS Glue interactive sessions now support Apache Spark Connect, enabling developers to write code locally in any IDE while executing remotely on AWS Glue's managed Spark clusters.
- Spark Connect uses gRPC protocol to decouple client from server, allowing local development with remote execution
- Works with VS Code, PyCharm, SageMaker Unified Studio, and any Python environment without kernel installation
- SageMaker Unified Studio provides automatic session creation, endpoint retrieval, and token refresh
- Use sagemaker-studio SDK for programmatic access in local IDEs, CI/CD pipelines, and Python applications
- Spark Live UI includes dedicated Connect tab for monitoring active sessions and operations
- Requires AWS Glue version 5.1 and PySpark 3.5.6; supports automatic token refresh for long-running sessions
Spark Connect bridges local development and remote execution, eliminating the need to split development between local Spark and AWS Glue interactive sessions.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.