Evaluate models with the Amazon Nova evaluation container using Amazon SageMaker AI

This article introduces new Amazon Nova model evaluation features in Amazon SageMaker AI, including custom metrics, LLM-based judging, log probability capture, metadata analysis, and multi-node scaling.

• Custom metrics (BYOM) enable domain-specific evaluation criteria via AWS Lambda functions
• Nova LLM-as-a-Judge automates preference testing with pairwise comparisons and Bradley-Terry scoring
• Token-level log probabilities reveal model confidence for calibration and uncertainty detection
• Metadata passthrough enables stratified analysis across segments without post-processing
• Multi-node execution scales evaluations from thousands to millions of examples
• Case study demonstrates IT support ticket classification with structured JSON responses
• Failure analysis identifies low-confidence predictions and overconfident errors for improvement

The post provides step-by-step implementation guidance for preparing datasets, building custom Lambda processors, launching evaluation jobs, and analyzing results with confidence-based quality gates.