Migrate large HPC datasets from the edge to the cloud then synchronize continuously
Storage Blog
This article describes a solution for migrating large High-Performance Computing (HPC) datasets from on-premises locations with limited bandwidth to the AWS cloud, and then continuously synchronizing the data. It covers the following key points:
Specifically, the article covers:
- Using AWS Snowball Edge Storage Optimized devices for initial bulk data transfer
- Using the snow-transfer-tool to efficiently transfer data to Snowball devices
- Using AWS DataSync for post-migration synchronization of updated data and metadata
- Mounting migrated data on Amazon FSx for Lustre for low-latency access to HPC workloads
- Detailed steps for ordering Snowball devices, transferring data, configuring DataSync, and setting up FSx for Lustre
- Cleaning up resources after migration to avoid future charges
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.