SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
Machine Learning Blog
Amazon SageMaker Inference now supports GPU partitioning using NVIDIA Multi-Instance GPU (MIG) technology to divide physical GPUs into independent, hardware-isolated partitions for improved resource utilization.
- Deploy multiple models on a single GPU with hardware-enforced isolation between partitions
- Choose right-sized GPU fractions instead of committing to whole accelerators
- Supports P4d (A100), P5 (H100), G7, and G7e instance families with varying partition profiles
- Add partition profiles to endpoint configurations and specify partition types for inference components
- Partitions provide predictable latency, fault isolation, and reliable performance planning
- Performance testing shows mig-3g.40gb H100 partition outperforms full A100-40GB by 53-89% on throughput
- Available in seven AWS Regions including US East, US West, Asia Pacific, and Europe
GPU partitioning enables efficient multi-model hosting with hardware isolation, eliminating the need for self-managed Kubernetes while providing cost savings and predictable performance for inference workloads.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
2026
2026
2026
2024
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.