Evaluating Deep Agents using LangSmith on AWS

This article explains how to evaluate AI agents using LangSmith on AWS, combining LangChain and Anthropic best practices into a practical guide for testing deep agents throughout their lifecycle.

• Five evaluation patterns: custom test logic, single-step evals, full agent turns, multi-turn conversations, environment state checks
• Three grader types: code-based (deterministic), LLM-as-judge (flexible), human graders (calibration)
• Agent evaluations harder than LLM evals due to non-determinism, error propagation, and creative solutions
• Offline evaluations use pytest with LangSmith for pre-deployment testing and regression detection
• Online evaluations monitor production traces without reference outputs using code, LLM, and composite evaluators
• Text-to-SQL deep agent example demonstrates full lifecycle from development to production monitoring
• Amazon Nova 2 Lite provides fast, cost-effective reasoning for agentic workloads on Bedrock

The post provides practical patterns and code examples for building reliable AI agent evaluation systems that catch issues early and continuously improve agent performance in production.