Announcing On-Demand State Repartitioning for Apache Spark Structured Streaming on Databricks
Databricks now offers on-demand state repartitioning for Apache Spark Structured Streaming, allowing users to resize stateful query partitions without rebuilding checkpoint state. This capability addresses historical limitations where partition counts were fixed, preventing efficient scaling and tuning of long-running streams. Available in Public Preview for Databricks Runtime 18+ with the RocksDB state store, it enables operations like fraud detection to optimize resource usage and reduce costs. Users can configure the new partition count and monitor the state redistribution process via query progress metrics.
- →Overcoming Fixed Partition Count Limitations
- →Prerequisites and Configuration
- →Monitoring Repartition Operations
Notes (3) ›
- Overcoming Fixed Partition Count Limitations
Historically, the number of partitions for stateful Structured Streaming queries was locked into the checkpoint upon creation, preventing dynamic scaling as data volumes changed. The on-demand repartitioning feature physically redistributes state data to match a new partition count, safely resolving this inflexibility.
- Prerequisites and Configuration
This feature requires Databricks Runtime 18 or above and the RocksDB state store provider. Users can activate it by setting `spark.sql.streaming.stateStore.partitions` to the desired value and restarting the streaming query.
- Monitoring Repartition Operations
The state redistribution process, which has a runtime proportional to the amount of state, is visible through standard `StreamingQueryProgress` events. The `controlBatch.REPARTITION` field reports the repartition duration in milliseconds, allowing users to track the overhead of resizing.
https://www.databricks.com/blog/announcing-demand-state-repartitioning-apache-sparktm-structured-streaming-databricks
Related releases
- Databricks `ai_transcribe()` SQL Function for Audio Transcription (Beta) Databricks Release Notes ·
- Databricks Lakebase Snapshots API (Beta) for Programmatic Management Databricks Release Notes ·
- Databricks Introduces Session Restore for Serverless Jobs in Public Preview Databricks Release Notes ·
- Databricks Genie now displays visualizations directly in Slack replies Databricks Release Notes ·
- Databricks Introduces Configurable Data Retention for System Tables (Beta) Databricks Release Notes ·
- Databricks Terraform Provider v1.132.0 Adds Domain & Sandbox Management Terraform Databricks Provider Releases ·