Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue

This article demonstrates how to run DuckDB, an embedded SQL engine, inside AWS Glue 6.0 jobs to perform cost-effective ETL operations on data in Amazon S3.

• DuckDB reads Parquet files from S3 via httpfs extension and writes Iceberg tables to Amazon S3 Tables through REST catalog
• AWS Glue 6.0 runtime (Amazon Linux 2023, Python 3.13) provides modern glibc compatibility for DuckDB native extensions
• Single-node vectorized execution eliminates shuffle and inter-node I/O overhead for datasets fitting in worker memory
• Measured results show DuckDB completed same workload in ~56 seconds at $0.0103 per run vs Spark at ~117 seconds and $0.0205
• Pattern complements Spark for SQL-centric jobs; switch back to PySpark for distributed workloads without infrastructure changes
• Complete walkthrough includes eCommerce order aggregation example with Lake Formation permissions and production error handling guidance

DuckDB on AWS Glue 6.0 provides a lightweight, cost-effective alternative for SQL-based transformations, enabling faster iteration and lower operational costs for scheduled ETL jobs.