From Problem to Fix in Minutes: Automating Search Relevance Tuning with the Amazon OpenSearch Service Relevance Agent

This article introduces the Amazon OpenSearch Service Search Relevance Agent, which automates the process of diagnosing and fixing search relevance issues in minutes through an AI-powered conversational interface.

• Analyzes User Behavior Insights (UBI) data to surface relevance problems like low click-through rates and zero-result queries
• Generates improved search query configurations with explanations of field weight changes and scoring adjustments
• Validates improvements using offline Search Relevance Workbench (SRW) experiments with NDCG, MAP, and Precision metrics
• Operates through OpenSearch Agent Server with three core skills: User Behavior Analysis, Hypothesis Generation, and Evaluation
• Maintains full transparency by documenting all decisions and making configurations auditable before deployment
• Enables continuous improvement cycles by automating repetitive search tuning work for engineers and non-technical stakeholders

The Search Relevance Agent transforms search quality from a one-time project into a continuous, human-in-the-loop practice accessible through the Ask AI button in OpenSearch UI.