Home icon

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

Machine Learning Blog



This article details how Exa optimized its embedding infrastructure to achieve 50% throughput improvement on AWS Neuron hardware (Inferentia2 and Trainium2) for indexing billions of web documents.

  • Implemented weight layout optimization and autocast to reduce unnecessary transposes and downcast matrix multiplications
  • Developed custom fused SwiGLU MLP kernel using NKI to eliminate high-bandwidth memory stalls
  • Created shape-specific attention kernel to reduce FLOPS spent on padding and improve tensor engine utilization
  • Used agentic development workflow with Flyte to verify correctness, measure performance, and identify optimization opportunities
  • Deployed flexible multi-region scheduler to optimize cost-performance across different instance types and hardware families

The optimization journey demonstrates how systematic profiling and custom kernel development on AWS Neuron hardware can deliver significant cost-per-document improvements at web-scale embedding workloads.



Go to article

The AWS News Feed is currently looking for gold sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.

Related articles

Oct 28
2026
High-throughput enterprise-scale document migration on AWS
Nov 13
2026
Accelerate design space exploration with AI on AWS
Nov 10
2026
46% fewer nodes, 2× faster indexing: Confluent on Amazon OpenSearch Service
Nov 9
2026
Amazon DocumentDB Serverless now provides up to 35% higher throughput

The AWS News Feed is currently looking for silver sponsors. If you want to support the AWS community and reach a large audience of AWS professionals, consider sponsoring the AWS News Feed.