Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints

This article announces GPU partitioning support in Amazon SageMaker AI inference, enabling multiple models to share a single GPU using NVIDIA Multi-Instance GPU (MIG) technology.

• Divide GPUs into up to seven isolated partitions with dedicated memory and compute resources
• Run small and mid-size models like embeddings, rerankers, and classifiers on shared accelerators
• Hardware-enforced isolation prevents busy models from consuming resources of neighboring models
• Right-size accelerators to workloads and access newer instance generations more efficiently
• Configure partitioning at endpoint creation; application code remains unchanged
• Available on P4d, P5, G7, and G7e instances across multiple AWS regions

GPU partitioning reduces infrastructure costs and improves resource utilization for customers running multiple smaller models on SageMaker AI endpoints.