Amazon SageMaker AI cuts generative AI inference scale-out time by up to half with automatic container image caching

Amazon SageMaker Inference now supports container image caching, enabling up to 2x faster end-to-end scaling for generative AI models during scale-out events.

• Pre-caches container images so new instances start serving traffic faster without pulling from ECR
• Eliminates cold-start latency caused by pulling large container images (10 GB or more)
• Requires no customer configuration changes; automatically caches specified image URIs
• Supports accelerator instance types, single-model endpoints, and inference component-based endpoints
• Complements existing optimizations: sub-minute concurrency metrics and instance-store caching
• Available in all AWS commercial regions where SageMaker Inference is supported

Container image caching significantly accelerates generative AI model scaling by eliminating the bottleneck of pulling large container images during scale-out events.