How Tealium built a chatbot evaluation platform with Ragas and Auto-Instruct using AWS generative AI services

This article details how Tealium built a chatbot evaluation platform using Ragas and Auto-Instruct with AWS generative AI services, specifically Amazon Bedrock. The solution focuses on improving RAG (Retrieval Augmented Generation) pipelines through comprehensive evaluation and error correction techniques.

• Developed an evaluation framework using Ragas to assess RAG system performance across metrics like faithfulness, context precision, and answer relevancy
• Implemented automatic prompt engineering using Auto-Instruct to generate and rank improved instructions
• Created a human-in-the-loop UI for subject matter experts to provide feedback on model outputs
• Utilized Amazon Bedrock's flexible foundation models for evaluation and instruction generation
• Achieved 85% context utilization, 86% faithfulness, and 76% answer relevancy in initial testing

The collaboration demonstrated Amazon Bedrock's potential in powering sophisticated generative AI solutions with improved evaluation and refinement capabilities.