Introducing Fast Model Loader in SageMaker Inference: Accelerate autoscaling for your Large Language Models (LLMs) – part 1

AWS has introduced Fast Model Loader in SageMaker Inference, a new capability designed to accelerate the deployment and scaling of Large Language Models (LLMs) with significant performance improvements:

• Streams model weights directly from Amazon S3 to GPUs
• Can load large models up to 15 times faster compared to traditional methods
• Reduced scaling time by 19-22% for the Llama 3.1 70B model
• Eliminates intermediate loading steps by using Direct Memory Access (DMA)
• Supports pre-sharding of model weights into uniform 8 MB chunks

Key benefits include faster model loading, improved resource utilization, and more efficient scaling during autoscaling events. Customers can use Fast Model Loader through SageMaker Studio or the SageMaker Python SDK to optimize their LLM deployments.