How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI
Machine Learning Blog
This article describes how Salesforce implemented cross-Region inference routing in Amazon SageMaker AI to solve capacity and availability challenges when serving models globally across 30+ Regions.
- Inference router enables single invocation path across SageMaker endpoints in multiple Regions and AWS accounts
- Router prefers local Region, shifts traffic to healthy remote destinations when local endpoint shows capacity constraints or errors
- Data residency requirements pinned to in-Region endpoints; cross-Region traffic capped to protect local capacity
- Each response includes InvokedRoutedResourceArn and RoutingReason fields for observability and troubleshooting
- Salesforce integrated router into Internal Gateway, replacing custom health checks and failover logic
- Failover test showed traffic shifted to remote Region within 2 minutes of errors, with under 100ms cross-Region latency overhead
Cross-Region inference routing enables Salesforce to add capacity securely while maintaining data residency commitments and eliminating per-application routing logic.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2025
2026
2025
2026
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.