The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

This article provides a comprehensive playbook for evaluating AI agents on Amazon Bedrock AgentCore using built-in, DeepEval, and AutoEval evaluators to measure business outcomes and agent behaviors.

• Start with business outcomes, not metrics: define success, break it into observable behaviors, then map to evaluators at trace or session level
• AgentCore offers 15+ built-in AWS evaluators covering response quality, task completion, tool use, skills, and safety
• DeepEval and AutoEval provide managed open-source metrics for task completion, conversational quality, tool use, and specialized signals like security and humor
• Multi-dimension evaluators score multiple independent criteria from a single judge call, enabling granular failure analysis
• Three evaluation modes: on-demand for development, online for production health signals, and batch for regression testing before releases
• Example stacks provided for customer support, multi-step research, and domain-specific agents with recommended starting evaluators

Effective agent evaluation requires choosing 2-3 evaluators that best show business outcomes, then adding more when failure modes emerge, using the ListEvaluators API to discover the full catalog.