Home icon

Apache Spark lineage now available in Amazon SageMaker Unified Studio for IDC based domains

News

This article announces the general availability of Apache Spark Data Lineage in Amazon SageMaker Unified Studio for IDC-based domains.

  • Captures schema and transformation lineage from Spark jobs on EMR and AWS Glue
  • Supports EMR-EC2, EMR-Serverless, EMR-EKS, and AWS Glue executions
  • Visualize lineage as interactive graphs in SageMaker Unified Studio
  • Query lineage data using APIs for programmatic access
  • Compare transformations across Spark job execution history
  • Available in all existing SageMaker Unified Studio regions

In summary, Apache Spark Data Lineage helps users identify root causes of issues and understand data transformation impacts across their data pipelines.

Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Mar 17
2026
Amazon SageMaker Unified Studio supports aggregated view of data lineage
Jul 7
2026
Amazon SageMaker now supports data lineage in IAM-based domains
Feb 9
2026
Using Amazon SageMaker Unified Studio Identity center (IDC) and IAM-based domains together
Jan 26
2026
Apache Spark 4.0.1 preview now available on Amazon EMR Serverless

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.