How Delhivery migrated 500 TB of data across AWS Regions using Amazon S3 Replication
Storage Blog
This post discusses Delhivery's process of migrating over 500 TB of data and 70 million objects across AWS Regions using Amazon S3 Replication and S3 Batch Operations to comply with India's new data residency regulations.
Specifically, the article covers:
- Delhivery's existing data ingestion pipeline with Apache Kafka, Amazon MSK, and Amazon S3
- Three approaches considered for the migration: duplicate connectors, cross-Region data consumption, and Amazon S3 Cross-Region Replication (CRR) with S3 Batch Replication (chosen approach)
- Planning phase involving assessing data metrics and identifying specific data for migration
- Execution phase covering prerequisites like creating buckets, IAM roles, and SNS topics, then setting up CRR and executing S3 Batch Replication
- Key learnings from the migration experience
- Conclusion highlighting the importance of careful planning, execution, and monitoring for large-scale data migrations
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
Nov 14
2024
2024
AWS Backup now supports copying Amazon S3 backups across Regions and accounts in opt-in Regions
Nov 20
2024
2024
Announcing auto migration of EC2 databases to Amazon RDS using AWS Database Migration Service
Sep 4
2024
2024
How Phreesia replicated a 30 TB SQL Server database to Amazon S3 with AWS DMS
Dec 26
2024
2024
Backing up Oracle databases to Amazon S3 at scale
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.