ReleaseBytes
databricks Databricks Blog ·

Databricks combines vector search and AI Classify for efficient large-scale document classification

blogaidatabricksengineer
feature announcement

Databricks has introduced a new method for classifying documents against taxonomies with over 100,000 labels, addressing limitations of existing approaches like regex and direct LLM calls which struggle with cost, maintenance, and context windows. The solution pairs vector search with the Databricks AI Classify function, retrieving a shortlist of candidate labels before AI Classify makes the final selection. This hybrid approach has demonstrated higher accuracy at a significantly lower cost compared to using frontier models alone across multiple benchmarks, benefiting thousands of Databricks customers dealing with use cases like biomedical entity linking and vendor normalization.

  • →Databricks AI Classify function for large taxonomies
  • →Challenges in large-scale document classification
  • →Hybrid approach: Vector search with AI Classify
  • →Vector search method for label retrieval
  • →Performance and cost benefits demonstrated
Features (1) ›
  • Databricks AI Classify function for large taxonomies

    Databricks AI Classify is a function that takes a document and a map of labels with descriptions to return the best matching label(s), utilizing techniques to manage large documents and taxonomies effectively.

Enhancements (3) ›
  • Hybrid approach: Vector search with AI Classify

    A new solution combines vector search to retrieve a shortlist of candidate labels from a large taxonomy, which are then passed to the Databricks AI Classify function for final selection, improving accuracy and reducing cost.

  • Vector search method for label retrieval

    This method embeds labels and documents using a Qwen3-Embedding-8B model, then scores label relevance using a hybrid approach combining semantic cosine similarity and lexical BM25 scoring, merged via Reciprocal Rank Fusion.

  • Performance and cost benefits demonstrated

    Across benchmarks, the SQL-native vector search combined with AI Classify achieved five points higher accuracy than cost-efficient frontier models at approximately one-hundredth of the token cost, benefiting thousands of customers.

Notes (2) ›
  • Challenges in large-scale document classification

    Mapping text to taxonomies with over 100,000 labels, common in biomedical entity linking, vendor normalization, and company deduplication, is difficult with traditional methods like regex, trained classifiers, and direct LLM calls due to issues with cost, maintenance, and context limits.

  • Evaluation of classification methods

    Three methods were evaluated: vector search alone, vector search combined with AI Classify, and direct calls to frontier LLMs. The focus was on balancing quality, cost, and throughput for production workloads.

Read the original announcement →

https://www.databricks.com/blog/scaling-document-classification-100k-labels

Related releases