Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2

This article demonstrates how NVIDIA CUDA Multi-Process Service (MPS) combined with NVIDIA Triton Inference Server on Amazon EC2 reduces automatic speech recognition (ASR) inference infrastructure requirements by 75 percent.

• Single ASR requests use only 15-20% of GPU compute capacity, leaving 80% idle with default time-slicing behavior
• CUDA MPS enables concurrent kernel execution across multiple processes without code changes, eliminating context-switching overhead
• Hybrid optimization stack combines ONNX Runtime with TensorRT for the encoder and PyTorch CUDA for the decoder
• Triton Inference Server handles dynamic batching for transcription and sequence batching for streaming diarization
• On g7e.4xlarge instances, achieves 92.1 requests per second per GPU at 352ms mean latency with sub-second SLA compliance
• TensorRT + ONNX + MPS configuration delivers 88% infrastructure reduction (2 GPUs vs 16) at 111.6 RPS
• Complete implementation provided in GitHub repository with Dockerfiles, Triton configs, FastAPI gateway, and benchmark scripts

The solution reduces Heidi Health's GPU infrastructure from 16 instances to 4 while maintaining strict latency requirements for processing 2.4 million clinical consultations weekly.