Databricks combines vector search and AI Classify for efficient large-scale document classification
Databricks has introduced a new method for classifying documents against taxonomies with over 100,000 labels, addressing limitations of existing approaches like regex and direct LLM calls which struggle with cost, maintenance, and context windows. The solution pairs vector search with the Databricks AI Classify function, retrieving a shortlist of candidate labels before AI Classify makes the final selection. This hybrid approach has demonstrated higher accuracy at a significantly lower cost compared to using frontier models alone across multiple benchmarks, benefiting thousands of Databricks customers dealing with use cases like biomedical entity linking and vendor normalization.
- →Databricks AI Classify function for large taxonomies
- →Challenges in large-scale document classification
- →Hybrid approach: Vector search with AI Classify
- →Vector search method for label retrieval
- →Performance and cost benefits demonstrated
Features (1) ›
- Databricks AI Classify function for large taxonomies
Databricks AI Classify is a function that takes a document and a map of labels with descriptions to return the best matching label(s), utilizing techniques to manage large documents and taxonomies effectively.
Enhancements (3) ›
- Hybrid approach: Vector search with AI Classify
A new solution combines vector search to retrieve a shortlist of candidate labels from a large taxonomy, which are then passed to the Databricks AI Classify function for final selection, improving accuracy and reducing cost.
- Vector search method for label retrieval
This method embeds labels and documents using a Qwen3-Embedding-8B model, then scores label relevance using a hybrid approach combining semantic cosine similarity and lexical BM25 scoring, merged via Reciprocal Rank Fusion.
- Performance and cost benefits demonstrated
Across benchmarks, the SQL-native vector search combined with AI Classify achieved five points higher accuracy than cost-efficient frontier models at approximately one-hundredth of the token cost, benefiting thousands of customers.
Notes (2) ›
- Challenges in large-scale document classification
Mapping text to taxonomies with over 100,000 labels, common in biomedical entity linking, vendor normalization, and company deduplication, is difficult with traditional methods like regex, trained classifiers, and direct LLM calls due to issues with cost, maintenance, and context limits.
- Evaluation of classification methods
Three methods were evaluated: vector search alone, vector search combined with AI Classify, and direct calls to frontier LLMs. The focus was on balancing quality, cost, and throughput for production workloads.
https://www.databricks.com/blog/scaling-document-classification-100k-labels
Related releases
- Databricks restricts UI disabling of partner-powered AI features Databricks Release Notes ·
- Databricks to remove partner-powered AI features setting toggle Databricks Release Notes ·
- Sharing Foreign Schemas and Tables with OpenSharing is Generally Available Databricks Release Notes ·
- Databricks OpenSharing now generally available for foreign Iceberg tables Databricks Release Notes ·
- Databricks Lakeflow Connect Unifies Marketing Data with Managed Connectors Databricks Blog ·
- Databricks and Abacus Insights Power AI for Health Plan MLR Analysis Databricks Blog ·