Build declarative ETL pipelines with AWS Glue 6.0
Big Data Blog
This article introduces Spark Declarative Pipelines (SDP) in AWS Glue 6.0, which simplifies ETL pipeline development by automatically resolving dependencies and managing execution order without manual orchestration code.
- SDP eliminates need for separate orchestrators by running bronze-silver-gold pipelines as single declarative jobs
- Supports Python decorators (@dp.table, @dp.materialized_view, @dp.temporary_view) to define datasets with automatic dependency inference
- Provides execution modes (VALIDATE for dry-run, RUN for normal execution) and refresh scopes (selective or full-refresh options)
- Includes step-by-step walkthrough building three-layer order processing pipeline with CSV input and Data Catalog integration
- Enables SQL and Python coexistence in same pipeline with automatic cross-language dependency resolution
- AWS Glue 6.0 priced 30% lower per DPU-hour than 5.1 with Apache Spark 4.1 runtime
SDP separates the "what" (declared datasets) from the "how" (automatic flows and execution), reducing boilerplate and enabling developers to focus on business logic rather than pipeline orchestration.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2026
2026
2024
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.