Evaluate your Amazon Nova Sonic voice agent at scale, no microphone required

This article introduces the Nova Sonic Test Harness, an open-source framework for automated testing and evaluation of Amazon Nova Sonic voice agents at scale without requiring microphones.

• Solves voice agent testing challenges: bidirectional streaming, non-deterministic responses, multi-turn context, and audio-text divergence
• Automates conversation testing using LLM-powered user simulators and judges instead of manual interaction
• Evaluates agent quality through rubric-based metrics: Goal Achievement, Response Accuracy, Tool Usage, Conversation Flow, System Prompt Compliance, Voice Formatting
• Detects audio hallucinations by comparing transcribed audio against text output for factual discrepancies
• Supports batch testing with parallel execution across dozens of scenarios and personas
• Offers multiple input modes: text, Amazon Polly TTS, scripted, and dataset-driven testing
• Includes pre-built scenario packs for healthcare, banking, and customer service domains
• Provides interactive dashboard, JSON/CSV results, and CI/CD-friendly verdicts for integration

The test harness enables rapid iteration of voice agent prompts and configurations while providing comprehensive quality validation before deployment, eliminating manual testing bottlenecks.