Home icon

SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Machine Learning Blog



Amazon SageMaker Inference now supports GPU partitioning using NVIDIA Multi-Instance GPU (MIG) technology to divide physical GPUs into independent, hardware-isolated partitions for improved resource utilization.

  • Deploy multiple models on a single GPU with hardware-enforced isolation between partitions
  • Choose right-sized GPU fractions instead of committing to whole accelerators
  • Supports P4d (A100), P5 (H100), G7, and G7e instance families with varying partition profiles
  • Add partition profiles to endpoint configurations and specify partition types for inference components
  • Partitions provide predictable latency, fault isolation, and reliable performance planning
  • Performance testing shows mig-3g.40gb H100 partition outperforms full A100-40GB by 53-89% on throughput
  • Available in seven AWS Regions including US East, US West, Asia Pacific, and Europe

GPU partitioning enables efficient multi-model hosting with hardware isolation, eliminating the need for self-managed Kubernetes while providing cost savings and predictable performance for inference workloads.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Oct 30
2026
Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints
Mar 24
2026
Deploy SageMaker AI inference endpoints with set GPU capacity using training plans
Oct 22
2026
Announcing new ways to customize models with SageMaker AI, without managing GPUs
Dec 3
2024
Speed up your AI inference workloads with new NVIDIA-powered capabilities in Amazon SageMaker

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.