From problem to fix in minutes: Automating search relevance tuning with the Amazon OpenSearch Search Relevance Agent

This article introduces the Amazon OpenSearch Search Relevance Agent, which automates the process of diagnosing and fixing search relevance issues through conversational AI.

• Analyzes User Behavior Insights (UBI) data to identify relevance problems like low click-through rates and zero-result queries
• Generates improved search query configurations with explanations of field weight changes and scoring adjustments
• Validates improvements using offline Search Relevance Workbench (SRW) experiments with NDCG, MAP, and Precision metrics
• Operates through OpenSearch Agent Server with three core skills: user behavior analysis, hypothesis generation, and evaluation
• Provides full transparency by documenting all decisions and making configurations auditable before deployment
• Enables continuous improvement cycles by automatically surfacing emerging relevance gaps in search performance
• Accessible via Ask AI button in OpenSearch UI search workspace for both search engineers and non-technical users

The Search Relevance Agent transforms search quality improvement from a manual, time-consuming process into an automated workflow that takes minutes, enabling teams to maintain optimal search performance as catalogs and query patterns evolve.