University of California Irvine backs up petabytes of research data to AWS

The University of California, Irvine (UCI) developed a comprehensive backup solution for its research data using AWS services, addressing the challenge of backing up petabytes of data across 1500 faculty research labs.

• Backed up over 5 PB of data and 250 million files using rclone for data movement
• Implemented a custom Python wrapper to manage backup jobs and AWS service integration
• Used multiple AWS services including S3, IAM, CloudWatch, and SNS for monitoring and management
• Created a tiered backup approach with daily incremental and weekly deep sync backups
• Developed custom CloudWatch dashboards for cost tracking and per-server insights

The solution provides a scalable, cost-effective approach to managing large-scale research data backups with robust security and performance optimization.