Unity Catalog Managed Tables: Interoperability and Governance for Lakehouse
External engines like Spark, Flink, and DuckDB can now create, read, and write to Unity Catalog managed Delta tables with centralized governance. This integration leverages Predictive Optimization for improved query performance and reduced storage costs, while maintaining full interoperability. This feature is now in Public Preview, enabling enterprises to use their preferred engines on a single copy of data with enforced access policies.
- →External engine access to Unity Catalog managed tables now in Public Preview
- →Predictive Optimization improves query performance and lowers storage costs
- →Upgrade existing external tables to managed tables with no data rewrite
- →Open APIs enable broader ecosystem integration with Unity Catalog
- →Catalog commits provide a centralized source of truth for table state
Features (1) ›
- External engine access to Unity Catalog managed tables now in Public Preview
External engines, including Apache Spark, Apache Flink, and DuckDB, can now create, read, and write to Unity Catalog managed Delta tables with centralized governance and automatic optimizations. This feature is currently in Public Preview.
Enhancements (5) ›
- Predictive Optimization improves query performance and lowers storage costs
Unity Catalog managed tables utilize Predictive Optimization to automatically tune table layouts, collect query statistics, and select clustering columns. This can lead to significant storage cost savings and faster query performance without manual intervention.
- Upgrade existing external tables to managed tables with no data rewrite
Existing external tables can be upgraded in place to managed tables using the 'ALTER TABLE SET MANAGED' command. This upgrade preserves interoperability and allows existing tables to benefit from features like Predictive Optimization without requiring a data rewrite.
- Open APIs enable broader ecosystem integration with Unity Catalog
External access is built on open APIs for metadata, credential vending, and commit coordination, derived from the open-source Unity Catalog (UC OSS) project. This allows any engine to integrate and work with both Databricks UC and UC OSS deployments.
- Catalog commits provide a centralized source of truth for table state
Catalog commits make Unity Catalog the commit coordinator for UC managed tables, ensuring it is the source of truth for table state. This enables benefits such as safe external writes, multi-statement transactions, and auditing of external operations.
- Delta Kernel facilitates seamless engine integration with Delta tables
Delta Kernel provides libraries for reading, writing, and committing to Delta tables, allowing engines to integrate seamlessly with Databricks-managed catalogs or self-hosted UC OSS deployments. Engines built on Kernel inherit the latest Delta protocol features automatically.
https://www.databricks.com/blog/how-unity-catalog-managed-tables-bring-interoperability-performance-and-unified-governance