Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake
Big Data Blog
This article demonstrates how to migrate Apache Hudi-based data lakes to Apache Iceberg using in-place migration and full table migration approaches on AWS Glue and Amazon S3.
- In-place migration keeps original Parquet files and builds Iceberg metadata on top using add_files procedure or Apache XTable
- Full table migration rewrites data into new Iceberg tables using CTAS or CREATE+INSERT statements
- Iceberg offers enhanced partition evolution, hidden partitioning, REST Catalog openness, and deep AWS service integration
- AWS managed features include Glue Iceberg table optimizer, Amazon S3 Tables, zero-ETL integration, and materialized views
- Hudi Copy-on-Write tables rewrite entire files on updates; Merge-on-Read tables append delta logs for faster writes
- Disable garbage collection during in-place migration to prevent data loss from shared files
- Validation runbook recommended comparing row counts, aggregates, and sample lookups before production cutover
The migration strategy should be chosen based on performance requirements, cost, data freshness needs, and acceptable downtime during cutover.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Apr 3
2024
2024
Use Apache Iceberg in your data lake with Amazon S3, AWS Glue, and Snowflake
Nov 2
2026
2026
Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg
Nov 26
2025
2025
Achieve 2x faster data lake query performance with Apache Iceberg on Amazon Redshift
Sep 2
2026
2026
Build a dynamic streaming data lake with Apache Iceberg and Apache Flink
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.