Run interactive workloads on Amazon EMR on EKS with Spark Connect

This article announces support for interactive Apache Spark sessions with Spark Connect on Amazon EMR on EKS, enabling developers to work interactively from managed notebooks and IDEs.

• Develop and debug Spark applications interactively from SageMaker Unified Studio, Jupyter, and Visual Studio Code
• Persistent Spark context spans cells and scripts, blending local Python with remote Spark operations
• Client-server architecture decouples application client from Spark driver for flexible development
• Supports ad hoc data exploration and incremental PySpark development before production deployment
• Sessions run as secured pods on virtual clusters with IAM execution roles and project/user tagging
• Available with EMR 7.14 (Spark 3.5) and emr-spark-8.1.0 (Spark 4.1) across all AWS Commercial Regions

Spark Connect on EMR on EKS enables interactive development workflows while maintaining separation between client environments and Spark infrastructure.