Supercharge your auto scaling for generative AI inference – Introducing Container Caching in SageMaker Inference

AWS has introduced Container Caching for SageMaker Inference, a new capability designed to significantly reduce scaling times for generative AI models. This feature addresses critical challenges in deploying large language models by pre-caching container images.

• Reduces container download time during scaling events
• Provides up to 56% reduction in latency when scaling a new model copy
• Supports popular frameworks like LMI, Hugging Face TGI, PyTorch, and NVIDIA Triton
• Automatically enabled for supported SageMaker Deep Learning Containers
• Requires no additional configuration from users

In performance tests with the Llama3.1 70B model, Container Caching dramatically improved scaling times, reducing end-to-end scaling from 379 to 166 seconds, demonstrating significant efficiency gains for generative AI inference workloads.