Amazon EMR now supports native data quality checks in Apache Spark jobs
News
Amazon EMR now validates data quality natively within Apache Spark jobs, catching quality issues inline before bad data reaches downstream systems.
- Powered by AWS Glue with no additional cost; uses same declarative rules language as AWS Glue
- Supports Apache Iceberg, Apache Hudi, and Delta Lake across EC2, EKS, and Serverless modes
- 30+ rule types including dynamic rules and ML-based anomaly detection for drift tracking
- Author rules in Spark scripts or reference rules from AWS Glue Data Catalog
- Apply validation after reading data, during transformation, or after processing
- Take actions like failing pipelines, filtering records, or routing to quarantine
- Results at dataset, rule, and row level for easy issue identification
- Available on emr-spark-8.1.0 in all AWS Commercial and GovCloud regions
Amazon EMR Data Quality eliminates the need for separate validation infrastructure while providing flexible, consistent data quality management across Spark workloads.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Sep 22
2026
2026
Amazon EMR introduces Long Term Support with Apache Spark 4.1
May 27
2026
2026
Amazon EMR now supports Apache Spark 4.0.2 in general availability
Aug 8
2024
2024
Amazon EMR 7.2 now supports Apache Spark 3.5.1
Sep 29
2026
2026
Announcing Spark Connect on Amazon EMR on EKS: Interactive PySpark development, anywhere
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.