Databricks details advanced AUTO CDC for bitemporal and partial updates

Databricks' Bitemporal AUTO CDC allows tracking business and system time independently, which is critical for compliance with regulations like SEC Rule 17a-4 and FINRA. It enables point-in-time reconstruction of records even if events arrive out-of-order, replacing hand-written MERGE logic. This feature is currently in Beta.

Bitemporal tables ensure that the exact dataset used for model training can be reconstructed months later for audits, even after Delta Lake's VACUUM operation cleans up old file versions. This is achieved by storing logical history as data rows, making it immune to storage retention windows.

AUTO CDC Partial Updates automatically handle incomplete change data capture sources that only send changed fields, preventing unintentional overwrites of existing data. This allows NULL values in updates to be interpreted as "do not update" for specified columns. Partial Updates are now Generally Available.

Databricks is expanding AUTO CDC capabilities into open-source Apache Spark 4.2 by contributing the Python API for AUTO CDC Type 1. This aims to bring standardized, out-of-order change data capture to the broader ecosystem through a series of reviewed proposals and pull requests.