Databricks Introduces Feature Views for Managed ML Feature Pipelines
Databricks has released Feature Views, a new managed framework designed to simplify the creation, serving, and governance of ML features across the entire lifecycle. This feature aims to eliminate training-serving skew and reduce operational overhead by allowing developers to define a feature once and use it for both experimentation and production inference, including real-time applications. The public preview is now available, with streaming capabilities requiring an Enterprise-tier workspace.
- →Managed framework for defining, serving, and governing ML features
- →Support for both batch and real-time feature pipelines
- →Streamlined experimentation with Genie Code and SDK
- →Integration with Unity Catalog for governance and lineage
- →Production-ready pipelines orchestrated by Databricks
Features (3) ›
- Managed framework for defining, serving, and governing ML features
Feature Views provide a unified, managed framework to define ML features once, ensuring consistency across experimentation, training, and production inference. This abstraction aims to reduce duplicated code, eliminate training-serving skew, and simplify the productionization of real-time ML.
- Support for both batch and real-time feature pipelines
The framework allows features to be sourced from both batch and streaming data, with the ability to switch between them easily. For streaming use cases, Feature Views can deliver end-to-end p99 latency of 200ms, serving fresh signals for applications like fraud detection and personalization.
- Streamlined experimentation with Genie Code and SDK
Databricks' Genie Code and Feature Engineering Client SDK facilitate rapid development. Developers can declare features locally, compute them over historical data, and assemble point-in-time-accurate training sets within a single notebook environment.
Enhancements (1) ›
- Integration with Unity Catalog for governance and lineage
Feature Views are first-class objects within Unity Catalog, enabling discoverability, access control, and comprehensive lineage tracking. This integration ensures that features are governed as data, and their dependencies are automatically recorded with MLflow for seamless model deployment and inference.
Notes (2) ›
- Production-ready pipelines orchestrated by Databricks
When features are ready for production, Databricks manages the underlying pipelines, writing to appropriate online and offline stores. This includes optimizing components like Lakehouse and Realtime Mode for high-quality data, scalability, and reliability, handling complex scenarios like long backfills and stream processing.
- Public preview available, with specific requirements for streaming
Feature Views are currently in public preview. Real-time streaming materialization requires an Enterprise-tier workspace in a region that supports Lakebase. Documentation and quickstart notebooks are available for getting started.
https://www.databricks.com/blog/introducing-feature-views