Use AWS Glue ETL to perform merge, partition evolution, and schema evolution on Apache Iceberg

This article covers how to use AWS Glue ETL to perform data merge, partition evolution, and schema evolution operations on Apache Iceberg tables in a transactional data lake.

Specifically, the article covers:

• Overview of the solution architecture
• Setting up the infrastructure with AWS CloudFormation
• Creating an Iceberg table using AWS Lambda and granting access using AWS Lake Formation
• Integrating Iceberg with the AWS Glue Data Catalog and Amazon S3
• Merging data from a Dropzone location into the Iceberg table using an AWS Glue ETL job
• Querying the Iceberg table using Amazon Athena
• Performing partition evolution on the Iceberg table
• Performing schema evolution on the Iceberg table
• Updating data in the Iceberg table using a positional update