Databricks Runtime 19 Enhances Data Manipulation and Performance
Databricks Runtime 19 introduces new `truncate` and `trim_array` functions for SQL and DataFrame APIs, providing more precise control over numerical and array data manipulation. Python UDF workers can now use session-scoped environment variables, improving flexibility for custom functions. Furthermore, Auto-optimized shuffle v2 is now enabled by default, aiming to boost the performance of Spark workloads that don't explicitly set shuffle partition counts. These updates benefit data engineers and developers working with Databricks Spark environments.
- →New `truncate` function for numbers
- →New `trim_array` function for array manipulation
- →Session environment variables for Python UDF workers
- →Auto-optimized shuffle v2 enabled by default
Features (3) ›
- New `truncate` function for numbers
SQL and the Scala and Python DataFrame APIs now support a `truncate` function. This function allows users to truncate a number to a specified scale without any rounding, offering precise control over numerical data.
- New `trim_array` function for array manipulation
A new `trim_array` function is available in SQL and the Scala and Python DataFrame APIs. This function removes elements from either the end of an array (for positive counts) or the beginning (for negative counts).
- Session environment variables for Python UDF workers
Users can now set session-scoped Python worker environment variables using the `spark.pythonWorkerEnv.<variable-name>` configuration prefix. These variables are applied when workers for regular scalar Python user-defined functions (UDFs) are launched.
Enhancements (1) ›
- Auto-optimized shuffle v2 enabled by default
Auto-optimized shuffle v2 is now the default when a fixed number of shuffle partitions is not explicitly set. This change primarily benefits workloads that rely on Spark's automatic shuffle partition management, potentially improving performance.
https://docs.databricks.com/aws/en/release-notes/runtime/19#databricks-runtime-19-september-21-2026
Related releases
- Databricks explains why AI analytics requires governed data for trustworthy insights Databricks Blog ·
- Data Ontology: Providing Business Context for Trustworthy AI Agents Databricks Blog ·
- Databricks SQL 2026.36 Rolling Out to Current Channel Databricks Release Notes ·
- Databricks Genie One MCP Server GA; Beta Endpoint Deprecated Databricks Release Notes ·
- Databricks SQL Version 2026.36 Now Available in Preview Channel Databricks Release Notes ·
- Databricks SQL Version 2026.32 Rolls Out to Current Channel Databricks Release Notes ·