Amazon SageMaker launches faster auto-scaling for Generative AI models

This article announces a new capability from Amazon SageMaker Inference that allows faster auto-scaling for Generative AI models. It helps reduce the time it takes for models to scale automatically, enabling customers to improve the responsiveness of their applications as demand fluctuates.

Specifically, the article covers:

• Two new high-resolution CloudWatch metrics (ConcurrentRequestsPerModel and ConcurrentRequestsPerModelCopy) that track the actual concurrency or number of in-flight inference requests being processed by the model
• The ability to create auto-scaling policies using these metrics to scale models deployed on SageMaker endpoints, with new instances or model copies added in under a minute
• Availability on accelerator instance families in all AWS regions where Amazon SageMaker Inference is available, except China and the AWS GovCloud (US) Regions
• Links to the AWS ML blog and documentation for more information