ReleaseBytes
databricks Databricks Blog ·

Databricks Simplifies SQL ETL with Declarative Patterns in Lakehouse

blogdatadatabricksgaanalyst
feature

Databricks introduces new declarative ETL patterns directly into its Lakehouse SQL editor, enabling SQL analysts and analytics engineers to simplify common transformation logic. These patterns, including append-only updates, automatic Change Data Capture (CDC), and targeted batch overwrites, reduce the need for complex procedural code. Users can now define these operations directly within SQL queries, with Databricks handling underlying state tracking, scheduling, and incremental processing. This aims to streamline data warehousing workflows and improve efficiency for SQL-first practitioners.

  • →Append-Only Updates for Ingestion Workloads
  • →Declarative Change Data Capture (AUTO CDC)
  • →Targeted Batch Overwrites with REPLACE WHERE
Features (3) ›
  • Append-Only Updates for Ingestion Workloads

    SQL users can now define simple APPEND flows for incremental ingestion, automatically tracking new data from sources like cloud object storage. Databricks manages state tracking and serverless pipeline processing without manual setup or scheduling.

  • Declarative Change Data Capture (AUTO CDC)

    AUTO CDC allows SQL users to define complex CDC logic with minimal code, handling inserts, updates, and deletes, even with out-of-order data. It supports specifying keys, sequencing, delete handling, and storing results as SCD Type 1 or Type 2.

  • Targeted Batch Overwrites with REPLACE WHERE

    REPLACE WHERE flows bring a declarative pattern for targeted incremental batch recomputation for specific data subsets, such as date ranges or partitions. Leveraging Databricks' Enzyme engine, it identifies and processes only the changed data within the specified predicate, improving performance and cost efficiency.

Read the original announcement →

https://www.databricks.com/blog/modernizing-sql-etl-lakehouse-declarative-patterns

Related releases