Run GPU batch inference on Amazon ECS Managed Instances with scale to zero

This article demonstrates how to run GPU batch inference on Amazon ECS Managed Instances with automatic scale-to-zero capability, eliminating idle costs for bursty asynchronous workloads.

• ECS Managed Instances handle GPU instance provisioning, AMI updates, NVIDIA driver management, and security patching automatically
• Solution uses SQS for job buffering and Application Auto Scaling to scale instances to zero when idle
• Includes CloudFormation template for deploying complete pipeline with text-to-speech inference using 1.7B parameter Qwen3-TTS model
• GPU health monitoring detects hardware failures and auto-repairs instances without operator intervention
• Baked model weights in container images eliminate runtime downloads and cold start variability
• Pay only for active inference time; scales back to zero after 5 minutes of queue inactivity
• GPU management fees reduced 35% for G-series and 60% for P-series starting July 2026

This pattern is ideal for bursty GPU inference workloads where latency tolerance is measured in minutes rather than milliseconds, eliminating the operational burden of managing GPU instances while optimizing costs.