How Rufus doubled their inference speed and handled Prime Day traffic with AWS AI chips and parallel decoding

This article details how Rufus, Amazon's AI shopping assistant, optimized its inference performance for Prime Day 2024 using AWS AI chips and parallel decoding techniques.

• Faced challenge of handling millions of queries per minute with a 300 ms latency requirement
• Implemented parallel decoding with multiple decoding heads to predict tokens simultaneously
• Used AWS Trainium and Inferentia chips to accelerate performance
• Achieved key performance improvements:
  • Two times faster token generation
  • 50% reduction in inference costs
  • Seamless scalability during peak traffic
• Utilized Neuronx-Distributed Inference (NxDI) framework for implementation

The solution demonstrates how innovative AI chip and decoding techniques can dramatically improve large language model inference performance and efficiency.