Fast model loading for AI inference on Amazon EKS

This article explains how to dramatically reduce cold-start time for AI inference workloads on Amazon EKS by optimizing model loading and compilation caching.

• Identified bottlenecks: weights loading from S3 (35% for 64 GiB models, 92% for 203 GiB models) and torch.compile (65% for smaller models)
• Recommend 4 GiB chunk size for Run:ai Model Streamer instead of smaller chunks with higher parallelism, reducing serial round-trips
• Add aggressive timeout and retry logic to handle S3 request tail latency and stalled connections
• Cache torch.compile artifacts on NVMe persistent volumes to eliminate 53-second recompilation on subsequent pod starts
• Achieved 80% reduction in startup time: 82s→16s for 67 GiB model, 457s→32s for 203 GiB model on p5.48xlarge
• All optimizations require only environment variables and configuration changes, no code modifications needed

Configuration-only tuning plus upstream contributions to Run:ai Model Streamer enable fast model loading across EKS, EC2, ECS, and SageMaker without additional infrastructure overhead.