Amazon SageMaker inference launches faster auto scaling for generative AI models

The article discusses a new capability in Amazon SageMaker inference that enables faster auto scaling for generative AI models. It outlines how the new ConcurrentRequestsPerModel and ConcurrentRequestsPerCopy CloudWatch metrics track concurrency and in-flight requests, allowing faster detection and scaling compared to the previous SageMakerVariantInvocationsPerInstance metric.

Specifically, the article covers:

• The need for rapid detection and auto scaling for generative AI models like large language models to handle fluctuating demand
• Components of the auto scaling process, including monitoring metrics, triggering auto scaling, provisioning new instances, and load balancing
• Using the new ConcurrentRequestsPerModel and ConcurrentRequestsPerCopy metrics with target tracking or step scaling policies for Application Auto Scaling
• Steps to implement the new metrics for single model endpoints and inference components
• Sample results showing up to 40% reduction in overall end-to-end scale-out time for Meta Llama models
• Conclusion encouraging the use of the new metrics for faster auto scaling of generative AI models on SageMaker