Stop guessing auto scaling targets for TTS on Amazon SageMaker AI

This article explains how to replace guessed auto scaling targets for text-to-speech endpoints on Amazon SageMaker AI with values derived from actual measurements.

• Deploy Kokoro-82M TTS model as custom container on SageMaker AI real-time endpoint using AWS CDK
• Measure Q_max: highest concurrency one instance sustains while meeting SLO (50 requests for Kokoro-82M)
• Measure T_total: wall-clock time from scale-out request to recovered traffic (320 seconds including detection lag)
• Derive scaling thresholds and cooldowns from Q_max, T_total, SLO, and surge ratio using arithmetic formulas
• Use five-command toolchain: drift, qmax, ttotal, plan, validate to measure and deploy policy
• Scale on ConcurrentRequestsPerModel native metric instead of borrowed throughput metrics
• Enforce admission control at container level to prevent queueing past SLO recovery point

Measurement-driven auto scaling eliminates silent cost overruns and SLO breaches by grounding every policy parameter in real endpoint behavior rather than assumptions.