Scaling Rufus, the Amazon generative AI-powered conversational shopping assistant with over 80,000 AWS Inferentia and AWS Trainium chips, for Prime Day

This article describes how Amazon scaled its AI-powered shopping assistant Rufus for Prime Day using AWS Inferentia and AWS Trainium chips. It highlights the key architectural choices, optimizations, and strategies employed to serve the massive demand during Prime Day while maintaining low latency and reducing costs.

Specifically, the article covers:

• Solution overview: Rufus uses AWS Inferentia2 and Trainium chips, Amazon ECS, ALB, and NVIDIA's Triton Inference Server to power its large language model (LLM) for conversational shopping
• Optimizing inference performance: Techniques like streaming architecture, INT8 quantization, continuous batching with vLLM, and Neuron SDK optimizations were used to reduce latency and improve throughput
• Scaling up: Load balancing with least outstanding requests (LOR) routing, continuous batching, and scaling across multiple AWS Regions allowed Rufus to scale to over 80,000 Trainium and Inferentia chips while maintaining low latency
• Conclusion: Rufus leveraged AWS chips and services to reliably deploy and serve its multi-billion parameter LLM at scale for Prime Day