Scalable cross-cloud data migration to Amazon S3 with distributed rclone

This article presents a distributed architecture for migrating petabytes of data from multi-cloud providers to Amazon S3 using rclone, ECS, SQS, and EC2 Auto Scaling.

• Three-layer architecture: Discovery (ECS Fargate), Queueing (SQS), Execution (EC2 rclone workers)
• Automated object enumeration batches files into self-contained SQS messages for fault isolation
• EC2 Auto Scaling fleet scales 0-5 instances based on queue depth, achieving 15-120 Gbps throughput
• Supports S3-compatible providers: IBM Cloud, Google Cloud Storage, Azure Blob Storage
• Built-in retry logic and dead-letter queue handling eliminate manual intervention
• CloudWatch Logs and metrics provide comprehensive observability across all layers
• Single CloudFormation template deploys entire infrastructure with pre-configured IAM roles
• Tested migration of 2.7 PB dataset completed in two weeks for approximately $2,000 compute cost

This solution provides a resilient, provider-agnostic pattern for large-scale cross-cloud data migrations with automated scaling and built-in failure recovery.