Home icon

Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters

Machine Learning Blog



This article introduces Amazon SageMaker HyperPod Global Inference Gateway, a managed service that routes inference requests across multiple HyperPod clusters and AWS Regions through a single endpoint.

  • Routes requests across clusters in one or multiple Regions based on KV cache headroom, queue depth, Region proximity, and error rate
  • Automatically fails over traffic when a cluster fails without requiring client changes
  • Eliminates need to manage multiple endpoints per model per cluster or build custom load balancers and cross-Region networking
  • Authenticates all requests with SigV4 and IAM, with traffic carried over AWS PrivateLink
  • Supports OpenAI-compatible requests and custom identity tokens via JWT
  • Available in 13 AWS Regions including US, Europe, and Asia Pacific

The Global Inference Gateway maximizes GPU utilization across distributed clusters by intelligently routing requests and handling failover automatically, reducing operational complexity for teams serving large language models at scale.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Sep 18
2026
Introducing Amazon SageMaker HyperPod Inference Gateway
Nov 18
2026
Amazon SageMaker AI now supports cross-Region inference routing
Sep 24
2026
Amazon SageMaker HyperPod Inference Gateway for scalable LLM inference
Nov 19
2026
Keep inference running through regional capacity shortfalls with cross-Region inference routing in Amazon SageMaker AI

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.