How Deepgram optimized voice AI at scale with AWS Trainium

This article describes how Deepgram optimized its Nova-3 speech-to-text model on AWS Trainium2 to improve inference efficiency for batch workloads like contact-center analytics and medical transcription.

• Ported Nova-3 from PyTorch to AWS Trainium2 using TorchNeuron with no CPU fallbacks
• Replaced model layers with fused kernels from NKI Library, achieving 2.0x speedup on FFN layers
• Used Neuron Explorer profiling and NAD toolkit to identify bottlenecks and write custom kernels
• Built lightweight Rust data-parallel runtime supporting eight model replicas per device
• Achieved 11-70% better audio processing per dollar compared to previous compute options
• Configured trn2.3xlarge instance with eight model replicas running concurrently

The optimization demonstrates how AWS Trainium enables cost-effective inference scaling for voice AI models, with benefits extending to customers self-hosting on Amazon SageMaker AI.