Databricks details autoscaling for Lakebase Postgres

Lakebase Postgres achieves autoscaling by decoupling the compute layer from the durable storage layer. This architecture allows compute nodes to start, stop, move, or change size without disrupting the underlying database, which is crucial for dynamic resource adjustment.

The autoscaling algorithm monitors the VM's one-minute CPU load average every five seconds. It aims to keep CPU load at or below 90% of capacity, increasing compute units (cpuGoalCU) when load rises and decreasing when sustained load falls.

Memory is tracked at two frequencies: overall VM metrics every five seconds and Postgres-specific memory usage every 100 milliseconds. The goal is to maintain memory use below 75% of allocated RAM to prevent out-of-memory issues and allow headroom for new allocations.

A third signal estimates the database's active working set to ensure optimal performance by fitting data closer to Postgres. This uses a modified HyperLogLog algorithm to probabilistically estimate the number of distinct pages accessed, which influences the compute-cache size (lfcGoalCU).

To account for changing workloads, the HyperLogLog implementation stores timestamps instead of simple bits, allowing for time-bound estimates of distinct pages accessed. This enables the system to estimate working sets for various windows, from one to 60 minutes, collected every 20 seconds.

The final scaling target is determined by taking the largest of the three goal compute sizes (CPU, memory, working set), constrained by user-configured minimum and maximum limits. This allows the system to scale up or down based on the most pressing resource demand.