Improving search quality in Amazon OpenSearch Service with UBI and SRW
AWS details how to enhance search quality on Amazon OpenSearch Service using User Behavior Insights (UBI) and Search Relevance Workbench (SRW). UBI provides an open schema to capture user search behavior, while SRW offers a toolkit for measuring and evaluating search relevance. This framework enables teams to collect signals, generate relevance judgments, and validate changes, addressing issues like irrelevant results and position bias. This guide is the first in a two-part series, focusing on data capture and initial evaluation for developers and architects.
- →Framework for improving search quality using UBI and SRW
- →Addressing the challenge of measuring search relevance
- →Defining and collecting user behavior data with UBI schemas
- →Implementing data collection pipelines using Amazon OpenSearch Ingestion
- →Prerequisites and configuration for UBI data in OpenSearch Service
Notes (5) ›
- Framework for improving search quality using UBI and SRW
AWS introduces a repeatable framework for Amazon OpenSearch Service, combining User Behavior Insights (UBI) for capturing search behavior and Search Relevance Workbench (SRW) for measuring quality. This framework helps teams address the lack of deliberate signal collection and feedback loops in search, enabling validation of changes before deployment.
- Addressing the challenge of measuring search relevance
The post highlights common issues in search quality measurement, such as vocabulary gaps leading to zero results and position bias in click signals. Traditional server logs often miss critical user intent and interaction data, making it difficult to understand true search effectiveness.
- Defining and collecting user behavior data with UBI schemas
User Behavior Insights (UBI) defines an open schema for capturing search interactions, comprising two indices: `ubi_queries` for user queries and returned documents, and `ubi_events` for subsequent user actions like impressions and clicks. A shared `query_id` links every event back to the query, enabling comprehensive behavior tracking across sessions.
- Implementing data collection pipelines using Amazon OpenSearch Ingestion
Applications generate UBI-formatted records, which are then securely sent as SigV4-signed HTTP POST requests to Amazon OpenSearch Ingestion (OSI) pipelines for managed delivery to OpenSearch Service. OSI, powered by Data Prepper, provides HTTP source endpoints and sinks data to the corresponding UBI indices in your domain.
- Prerequisites and configuration for UBI data in OpenSearch Service
To follow the guide, users need an Amazon OpenSearch Service domain running OpenSearch 3.5 or later, permissions for OpenSearch Ingestion pipelines, and a search application instrumented to emit behavioral records. This involves creating the `ubi_queries` and `ubi_events` indices with appropriate mappings and setting up the OSI pipelines to route data.
https://aws.amazon.com/blogs/big-data/measuring-and-improving-search-quality-with-amazon-opensearch-service/
Related releases
- Amazon MSK Connect Now Supports Restarting Connectors and Tasks AWS What's New ·
- Amazon Quick integrates with AWS Agent Registry for agent and MCP server discovery AWS What's New ·
- Amazon Kinesis Data Streams Adds Support for Streaming Apache Iceberg Tables AWS Big Data Blog ·
- AWS Partner Revenue Measurement Expands Service Coverage for User Agent Strings AWS What's New ·
- Amazon WorkSpaces Applications Expands to Three New AWS Regions AWS What's New ·
- Amazon Kinesis Data Streams now delivers to general purpose S3 buckets AWS What's New ·