Reduce inference cold starts on Amazon SageMaker HyperPod with model caching

This article announces model caching for Amazon SageMaker Inference on HyperPod, which eliminates inference cold starts by pre-loading model weights and container images onto cluster nodes.

• Weights cache downloads model weights to local NVMe storage, enabling 7 GB/s read speeds instead of network downloads
• Image cache pre-pulls inference server container images to skip 5-7 minute ECR pulls
• Pods start serving traffic in seconds rather than 25-30+ minutes for large models like DeepSeek-R1
• Enable caching by adding modelCacheConfig to InferenceEndpointConfig or JumpStartModel resources
• Supports Amazon S3, FSx for Lustre, HuggingFace Hub, and SageMaker JumpStart as model sources
• Benchmarks show ~60% faster scale-out and up to 97% reduction in image pull time
• Pods prefer cached nodes but fall back to network downloads if needed; automatic cleanup on resource deletion

Model caching is now generally available across all regions where SageMaker HyperPod operates, significantly improving autoscaling response times for large language model inference.