Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips
Machine Learning Blog
This article details how Exa optimized its embedding infrastructure to achieve 50% throughput improvement on AWS Neuron hardware (Inferentia2 and Trainium2) for indexing billions of web documents.
- Implemented weight layout optimization and autocast to reduce unnecessary transposes and downcast matrix multiplications
- Developed custom fused SwiGLU MLP kernel using NKI to eliminate high-bandwidth memory stalls
- Created shape-specific attention kernel to reduce FLOPS spent on padding and improve tensor engine utilization
- Used agentic development workflow with Flyte to verify correctness, measure performance, and identify optimization opportunities
- Deployed flexible multi-region scheduler to optimize cost-performance across different instance types and hardware families
The optimization journey demonstrates how systematic profiling and custom kernel development on AWS Neuron hardware can deliver significant cost-per-document improvements at web-scale embedding workloads.
The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.
Related articles
The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.