Evaluating generative AI models with Amazon Nova LLM-as-a-Judge on Amazon SageMaker AI

This article introduces Amazon Nova LLM-as-a-Judge, a new capability on Amazon SageMaker AI for evaluating generative AI models with advanced reasoning capabilities. The key highlights include:

• A novel approach to model evaluation using LLMs to compare model outputs
• Trained on diverse datasets spanning 90+ languages with rigorous bias reduction
• Provides pairwise comparisons between model iterations with statistical confidence metrics
• Achieves 45% accuracy on JudgeBench and 68% on PPE benchmarks
• Integrated with Amazon SageMaker AI for scalable, systematic model assessment

The solution enables organizations to systematically evaluate generative AI model performance beyond traditional metrics, offering a more nuanced and flexible assessment approach that closely reflects human preferences.