ReleaseBytes
databricks Databricks Release Notes ·

Databricks Lakeflow Designer August Release Adds Unique Operator and Canvas Enhancements

datadatabrickspreviewengineermediadatabricks-lakeflow
feature

Databricks' Lakeflow Designer August release introduces significant updates for visual data preparation. It adds a new Unique operator for duplicate row removal and enhances existing operators like Aggregate, Select, and Python with capabilities such as dynamic column selection and streaming output. Users can now disable operators, view lineage highlighting, run all operations at once, and transform data directly within results tables. These improvements aim to streamline data transformation workflows and enhance productivity for engineers using Lakeflow Designer.

  • →New Unique operator to remove duplicate rows
  • →Python operator gains a dedicated preview mode
  • →Enable and disable individual operators or groups
  • →Visualize operator lineage on the canvas
  • →New "Run all" button for full visual data prep execution
Features (5) ›
  • New Unique operator to remove duplicate rows

    The Lakeflow Designer now includes a Unique operator that can remove duplicate rows, optionally keying on a subset of columns and ordering rows to control which duplicate is retained.

  • Python operator gains a dedicated preview mode

    The Python operator now allows different code execution during previews versus full runs by checking config["is_preview"], facilitating the skipping of side effects like writes or notifications during previewing.

  • Enable and disable individual operators or groups

    Users can now disable individual operators or entire groups to exclude them from runs, and then re-enable them as needed for flexible workflow management.

  • Visualize operator lineage on the canvas

    Selecting an operator on the canvas will now highlight its lineage, visually tracing its connections from upstream sources to downstream sinks.

  • New "Run all" button for full visual data prep execution

    A new "Run all" button has been added, allowing users to execute every operator in the visual data preparation process simultaneously.

Enhancements (3) ›
  • Aggregate operator supports COUNT DISTINCT function

    The Aggregate operator has been expanded to support a COUNT DISTINCT function, allowing users to count the unique values within a column.

  • Select operator offers dynamic column selection

    The Select operator can now dynamically choose columns based on data type, name pattern, or formula, enabling the output schema to adapt automatically to input schema changes.

  • Streaming output for the Python operator

    Output from the Python operator now streams into the output pane as the operator runs, providing results incrementally before the full run completes.

Read the original announcement →

https://docs.databricks.com/aws/en/release-notes/product/2026/august#lakeflow-designer-august-release

Related releases