Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality

This article announces Amazon EMR Data Quality, now bundled in AWS runtime for Apache Spark (emr-spark-8.1.0+), enabling native data quality validation within Amazon EMR jobs using AWS Glue Data Quality capabilities.

• Define and run data quality rules using Data Quality Definition Language (DQDL) directly in Spark jobs across all three Amazon EMR deployment modes
• Pattern 1: Ad hoc validation—define rules inline for exploratory or one-time processing without centralized ruleset management
• Pattern 2: Write-Audit-Publish with managed rulesets—gate data promotion using centrally managed rulesets from AWS Glue Data Catalog
• DQDL supports completeness, uniqueness, value ranges, referential integrity, and ML-based anomaly detection rules
• Managed rulesets enable version control, team sharing, and reuse across multiple pipelines without modifying job code
• Results include per-rule pass/fail outcomes and overall quality scores to branch pipeline logic accordingly

Amazon EMR Data Quality eliminates the trade-off between custom Spark quality checks and routing data through AWS Glue, providing managed data quality validation natively within EMR pipelines.