How Amazon scaled Rufus by building multi-node inference using AWS Trainium chips and vLLM

This article details how Amazon scaled Rufus, its generative AI shopping assistant, using multi-node inference with AWS Trainium chips and vLLM. Key highlights include:

• Developed a leader/follower multi-node inference architecture to handle large language models
• Used hybrid parallelism strategies to maximize compute and memory utilization across nodes
• Implemented a multi-node inference unit abstraction on Amazon ECS for reliable scaling
• Utilized Elastic Fabric Adapter (EFA) for low-latency, high-bandwidth cross-node communication
• Deployed the solution across tens of thousands of Trainium chips to support Prime Day traffic

The solution enables Amazon to run larger, more capable AI models with improved performance, throughput, and customer engagement for the Rufus shopping assistant.