Migrating data from an Amazon Aurora snapshot into Amazon Aurora DSQL

This article demonstrates how to migrate data from Amazon Aurora PostgreSQL snapshots to Aurora DSQL using AWS Glue for managed ETL operations.

• Export Aurora PostgreSQL snapshot to S3 in Parquet format using snapshot export feature
• Use AWS Glue crawlers to discover Parquet files and catalog schema in AWS Glue Data Catalog
• Create PySpark ETL job to transform data and handle data type conversions
• Convert serial integer primary keys to UUIDs to avoid hot partitions in Aurora DSQL
• Fix foreign key relationships after primary key conversion
• Write transformed data to Aurora DSQL using JDBC with IAM authentication tokens
• Set batch size to 2500 rows and isolation level to NONE due to Aurora DSQL transaction limits
• Verify migration with row counts and aggregate functions on both databases

This approach enables complex data transformations during migration that aren't possible with Aurora DSQL's native COPY command, making it suitable for large-scale migrations requiring schema changes or data type conversions.