How Autodesk migrated 2.3 billion documents to Amazon OpenSearch Service using Migration Assistant and intelligent routing

This article describes how Autodesk migrated 2.3 billion documents to Amazon OpenSearch Service using Migration Assistant and intelligent routing to overcome single-domain scalability limits.

• Re-architected from single-index Elasticsearch 7.1.1 domain to four multi-index OpenSearch 2.19 domains with 24 total indices
• Implemented stratified bin-packing algorithm to distribute 4.3 million projects evenly across domains, achieving 49% utilization with 2x growth buffer
• Used Amazon DynamoDB routing table with 4.3 million project-to-domain mappings for intelligent query routing to relevant shards only
• Achieved 228 GB/hr migration throughput through multiple PoC iterations, completing 2.3 billion document migration in 6 hours with zero downtime
• Reduced average shard size from 200 GB to 12.34 GB, improved P99 query latency by 72% (17s to 5s), and decreased storage by 37% (83.3 TB to 52 TB)
• Routing queries now scan ~12 of 180 shards per index, reducing search load by 93% compared to broadcast queries

The multi-domain architecture with intelligent routing enables horizontal scalability, reduces blast radius, and improves query performance while maintaining production stability.