Measuring and improving search quality with Amazon OpenSearch Service

This article explains how to measure and improve search quality using Amazon OpenSearch Service with User Behavior Insights (UBI) and Search Relevance Workbench (SRW).

• UBI captures search behavior in two indices: ubi_queries (user queries and results) and ubi_events (user actions like clicks and impressions)
• Amazon OpenSearch Ingestion (OSI) pipelines deliver UBI data from applications to OpenSearch indices asynchronously
• Search Relevance Workbench evaluates search quality using query sets, search configurations, and judgment lists based on user behavior
• COEC (Clicks Over Expected Clicks) model corrects position bias when deriving relevance judgments from click data
• Key metrics include Coverage, Precision, MAP, and NDCG to measure ranking quality against relevance judgments
• Common tuning levers include synonyms, field weights, semantic retrieval, and reranking processors

By collecting behavioral signals and validating changes against real relevance judgments, teams can systematically improve search quality instead of relying on guesswork.