Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters
Machine Learning Blog
This article introduces Amazon SageMaker HyperPod Global Inference Gateway, a managed service that routes inference requests across multiple HyperPod clusters and AWS Regions through a single endpoint.
- Routes requests across clusters in one or multiple Regions based on KV cache headroom, queue depth, Region proximity, and error rate
- Automatically fails over traffic when a cluster fails without requiring client changes
- Eliminates need to manage multiple endpoints per model per cluster or build custom load balancers and cross-Region networking
- Authenticates all requests with SigV4 and IAM, with traffic carried over AWS PrivateLink
- Supports OpenAI-compatible requests and custom identity tokens via JWT
- Available in 13 AWS Regions including US, Europe, and Asia Pacific
The Global Inference Gateway maximizes GPU utilization across distributed clusters by intelligently routing requests and handling failover automatically, reducing operational complexity for teams serving large language models at scale.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2026
2026
2026
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.