Databricks Unifies Genie Agent Governance Across Structured and Unstructured Data
Databricks details how its Genie Agents leverage existing Unity Catalog governance to securely operate across structured and unstructured data. This allows agents to reason over tables, metrics, and documents while respecting user permissions, simplifying architecture and eliminating the need for separate systems for data types. The solution prevents data loss and unauthorized access by establishing Unity Catalog as the security perimeter, rather than relying on the LLM. It utilizes features like Automatic Identity Management (AIM), object privileges, row filters, column masks, and the newly generally available Attribute-Based Access Control (ABAC) policies.
- →Attribute-Based Access Control (ABAC) policies now GA
- →Genie Agents operate under end-user credentials
- →Automatic Identity Management ensures up-to-date access
- →Comprehensive access control for structured data
- →Apply consistent governance to unstructured documents
Features (1) ›
- Attribute-Based Access Control (ABAC) policies now GA
Attribute-Based Access Control (ABAC) policies are now generally available in Unity Catalog, along with governed tags and automated data classification. ABAC allows defining policies once and applying them across hundreds of tables based on tags, simplifying the management of row and column security compared to per-table configurations.
Notes (4) ›
- Genie Agents operate under end-user credentials
Databricks Genie Agents enforce governance by running with the end user's identity, ensuring all queries and answers are filtered by existing permissions. This design positions Unity Catalog as the security perimeter, preventing manipulation or bypass at the model layer, which is crucial for defensible governance controls.
- Automatic Identity Management ensures up-to-date access
Automatic Identity Management (AIM) for Microsoft Entra ID and Okta automatically syncs users, groups, and memberships to Databricks, eliminating the need for SCIM applications. This ensures access controls for Genie Agents are always current, with just-in-time provisioning enabling immediate access and revocation based on identity provider changes.
- Comprehensive access control for structured data
Genie Agents leverage Unity Catalog's four layers of access control for structured data: object privileges, row filters, column masks, and Attribute-Based Access Control (ABAC). These mechanisms ensure granular control over data assets like tables, views, and metric views, from table-level access to masking sensitive columns and filtering rows based on user permissions.
- Apply consistent governance to unstructured documents
The same robust governance principles and Unity Catalog mechanisms used for structured data are extended to unstructured files stored in Unity Catalog Volumes. This allows a single Genie Agent to reason across both data types while maintaining consistent, permissions-aware insights based on the end user's access rights.
https://www.databricks.com/blog/how-ground-genie-agents-both-structured-data-and-documents-without-losing-governance
Related releases
- Databricks SDK for Go v0.172.0 Enhances IAMv2 and Job Cluster Management Databricks Go SDK Releases ·
- Databricks Java SDK v0.146.0 Adds IAM V2 API, Updates Job Cluster Field Databricks Java SDK Releases ·
- Databricks SDK for Python v0.128.0 Adds Account and Workspace IAM V2 API Methods Databricks Python SDK Releases ·
- Amtrak Builds Unified Data Backbone with Databricks for Rail Network Transformation Databricks Blog ·
- Databricks Re-architects Serverless Network Config Delivery for 97.5% Latency Cut Databricks Blog ·
- How a Major Freight Railroad Scaled Pipeline Creation with Databricks Genie Code Databricks Blog ·