Databricks Improves Lakebase Postgres Compute Cache with Larger Shared Buffers
Databricks has enhanced its Lakebase Postgres compute-side caching to boost throughput and reduce latency for disaggregated storage. This involves disabling the Local File Cache (LFC) and configuring shared buffers to utilize up to 75% of DRAM on fixed-size computes with CU >= 80. The change eliminates double buffering and allows Postgres to make smarter eviction decisions, enabled by the adoption of huge pages to mitigate memory management overhead. Initial production results demonstrate significant performance gains, including up to 2x throughput and 5x fewer reads from storage.
- →Enable larger Postgres shared buffers for fixed-size computes
- →Implement Huge Pages for efficient memory management
- →Demonstrate significant performance improvements
Features (1) ›
- Enable larger Postgres shared buffers for fixed-size computes
Databricks has disabled the Local File Cache (LFC) and configured Postgres shared buffers to utilize up to 75% of DRAM on fixed-size Lakebase Postgres computes with CU >= 80. This change eliminates the previous ~1 GB shared buffer cap, ensuring hot pages remain in the fastest memory layer and avoiding slower LFC access.
Enhancements (1) ›
- Implement Huge Pages for efficient memory management
To manage the increased memory usage from larger shared buffers, Databricks has adopted huge pages (2 MB each) for Lakebase Postgres. This significantly reduces the size of page tables and lowers Translation Lookaside Buffer (TLB) miss rates, mitigating memory management overhead for numerous backend processes.
Notes (1) ›
- Demonstrate significant performance improvements
Production results show substantial gains from the compute cache improvements, including up to 2x throughput and 5x fewer reads from storage, 1.3x throughput in another example, and 5x lower CPU use with 2x higher throughput in a third scenario. These improvements highlight reduced latency and better resource utilization.
https://www.databricks.com/blog/improving-lakebase-postgres-compute-cache
Related releases
- Databricks Lakeflow Connect Integrates Celigo Audit Logs (Beta) Databricks Release Notes ·
- Databricks Lakeflow Connect Adds Anysphere Organization Connector (Beta) Databricks Release Notes ·
- Anaplan Connector for Databricks Lakeflow Connect Now in Beta Databricks Release Notes ·
- Databricks AI Search now shows supporting chunks for generated answers Databricks Release Notes ·
- Databricks Outlines AI Solutions for Financial Services at Sibos 2026 Databricks Blog ·
- Data Classification to Soon Scan Unity Catalog Views Databricks Release Notes ·