Implementing AWS DataSync with hundreds of millions of objects

This article discusses techniques for using AWS DataSync to efficiently transfer and validate large datasets, even exceeding 100 million objects, in hybrid cloud environments.

Specifically, the article covers:

• Overview of AWS DataSync and the new manifest file feature for specifying files to transfer
• Solution overview involving restructuring data, event-driven approach with manifest files, and using include filters for large batches
• Detailed walkthrough of implementing the event-driven approach with manifest files using AWS services like Lambda, SQS, and S3
• Techniques for batching objects using UUID prefixes and include filters while staying within DataSync quotas
• Conclusion highlighting the benefits of using manifests and include filters for transferring and validating large datasets with DataSync