Use decision models with Amazon Bedrock AgentCore Evaluations

This article demonstrates how to integrate decision models with Amazon Bedrock AgentCore Evaluations through code-based evaluators for more reliable agent evaluation.

• Decision models return calibrated probability distributions over fixed answer sets, unlike LLM-as-a-judge evaluators that generate free-form text
• Jev (hosted) and Strands Decider 2B (open-source) support three question types: yes/no, choice, and score questions
• Code-based evaluators adapt OpenTelemetry spans from agents into decision model requests and map results back to AgentCore format
• Validation requires testing decision models against labeled data to measure agreement, calibration, and identify disagreement patterns
• Recommended approach: use decision models as fast screening gates alongside built-in LLM evaluators for deeper assessment
• Sample HR Assistant agent demonstrates end-to-end evaluation combining deterministic checks, built-in evaluators, and decision-model evaluators

Decision models provide narrow, validated checks with measurable confidence scores that complement broader LLM-based evaluations for agent quality assurance.