How London Stock Exchange Group migrated 30 PB of market data using AWS DataSync

This article details how the London Stock Exchange Group (LSEG) migrated 30 PB of ultra-high-quality global market data from Wasabi cloud storage to AWS using AWS DataSync. The key objectives were to accelerate the data transfer, emphasize efficiency, and optimize costs.

Specifically, the article covers:

• The solution architecture involving AWS services like DataSync, Step Functions, Lambda, DynamoDB, and CloudWatch for orchestration, automation, and monitoring
• Automating the migration workflow by generating DataSync tasks, scaling DataSync agents across Availability Zones, and implementing failure handling
• Considerations for scaling and optimizing DataSync performance, handling API throttling, and data validation strategies
• How LSEG achieved 80% reduction in storage costs by migrating active data to S3 Intelligent-Tiering and older data to S3 Glacier Deep Archive
• The benefits of storing data on AWS, including simplified data access for customers through AWS Data Exchange and higher availability/resiliency