Integrating subject matter experts into generative AI evaluation with the AWS Generative AI Innovation Center

This article explains how to integrate subject matter experts (SMEs) into generative AI evaluation through a four-phase flywheel approach to bridge the gap between prototype and production.

• Phase 1: Structured conversations with SMEs to translate expert intuition into concrete, repeatable evaluation criteria
• Phase 2: Build ground truth datasets using Amazon SageMaker Ground Truth and develop automated metrics with Amazon Bedrock Evaluations
• Phase 3: Shift SME role from scoring every output to validating and calibrating the automated scoring system
• Phase 4: Continuously review and update rubrics, ground truth, and metrics as product evolves
• Use real model outputs and rubrics with anchor examples to help SMEs articulate quality standards
• Measure correlation between automated and human scores; investigate divergences to improve both

The approach enables organizations to evaluate generative AI at scale with confidence while reducing SME bottlenecks through continuous feedback loops.