Building medallion architecture with Iceberg materialized views in Amazon SageMaker

This article demonstrates how to build a medallion architecture using Apache Iceberg materialized views in Amazon SageMaker, replacing traditional ETL orchestration with declarative SQL definitions.

• Iceberg materialized views collapse ETL jobs, orchestration, and CDC logic into single SQL statements per layer
• Bronze layer captures raw data; Silver applies cleaning and transformations; Gold creates business metrics
• Incremental refresh processes only changed rows using Iceberg's row-level change tracking
• Supports nested materialized views: Gold views read from Silver views which read from Bronze tables
• Available in AWS Glue 5.1+, Amazon Athena Spark, and Amazon EMR 7.12+
• Step-by-step tutorial builds a three-layer pipeline with sample ride-sharing data in SageMaker Unified Studio
• Demonstrates INSERT, UPDATE (MERGE), and DELETE operations with incremental refresh propagation
• Minimum refresh granularity is one hour; cascading refresh requires manual triggering between layers

Iceberg materialized views simplify medallion architecture by eliminating separate job artifacts, DAGs, and watermark tracking while maintaining incremental processing through SQL definitions.