Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

This article provides a comprehensive guide for migrating Google BigQuery workloads to an open lakehouse architecture on AWS using Apache Iceberg, emphasizing data openness, portability, and AI-readiness.

• Open lakehouse architecture decouples storage from compute, enabling multiple engines (Athena, Redshift, EMR, SageMaker) to access the same governed data without duplication
• Target architecture uses Apache Iceberg on Amazon S3 as the storage layer with medallion-pattern data refinement (Bronze/Silver/Gold layers)
• Unified governance through AWS Glue Data Catalog and Lake Formation enforces fine-grained access controls consistently across all engines
• Six-phase migration approach: assess workloads, design target architecture, convert and validate with POC, migrate data using BigQuery exports and AWS DataSync, execute cutover with parallel validation, and optimize post-migration
• BigQuery SQL conversions include STRUCT/ARRAY to Iceberg types, MERGE statements, JavaScript UDFs to Lambda functions, and scheduled queries to EventBridge or Step Functions
• Data migration uses BigQuery EXPORT DATA to Parquet, AWS DataSync for transfer from Google Cloud Storage to S3, and incremental synchronization during transition period
• Cutover strategies include re-pointing tools (dbt, Tableau, Power BI), rebuilding workloads on AWS equivalents, or relaying data back to Google Cloud temporarily
• Post-migration enables new capabilities: ML model training with SageMaker, generative AI with Bedrock, real-time analytics with Kinesis/Firehose, and cross-platform sharing via Iceberg REST catalog

The phased approach enables organizations to validate value incrementally while maintaining business continuity, with customer examples demonstrating 60% cost reductions and 75% latency improvements.