Home icon

How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI

Machine Learning Blog



This article describes how Salesforce implemented cross-Region inference routing in Amazon SageMaker AI to solve capacity and availability challenges when serving models globally across 30+ Regions.

  • Inference router enables single invocation path across SageMaker endpoints in multiple Regions and AWS accounts
  • Router prefers local Region, shifts traffic to healthy remote destinations when local endpoint shows capacity constraints or errors
  • Data residency requirements pinned to in-Region endpoints; cross-Region traffic capped to protect local capacity
  • Each response includes InvokedRoutedResourceArn and RoutingReason fields for observability and troubleshooting
  • Salesforce integrated router into Internal Gateway, replacing custom health checks and failover logic
  • Failover test showed traffic shifted to remote Region within 2 minutes of errors, with under 100ms cross-Region latency overhead

Cross-Region inference routing enables Salesforce to add capacity securely while maintaining data residency commitments and eliminating per-application routing logic.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Apr 17
2025
How Salesforce achieves high-performance model deployment with Amazon SageMaker AI
Nov 17
2026
Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints
Aug 15
2025
Optimizing Salesforce’s model endpoints with Amazon SageMaker AI inference components
Nov 17
2026
Amazon SageMaker now supports AI data preparation capabilities to build high-quality datasets for model development

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.