Managed Service for Apache Spark: Shuffle Partition Config Type Change
Google Cloud has updated the Managed Service for Apache Spark, formerly Dataproc on Compute Engine, by changing the configuration type for Spark shuffle partitions from integer to string for versions 2.2.82+ and 2.3.30+. This change requires users programmatically setting this configuration to update their code to use string literals for compatibility. Users setting the configuration via command-line, properties files, or Spark SQL commands are unaffected. This adjustment impacts specific subminor cluster image versions and aims to ensure consistent configuration handling.
- →Managed Service for Apache Spark
Breaking changes (1) ›
- Managed Service for Apache Spark Managed Service for Apache Spark
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The configuration for Spark shuffle partitions ( spark.sql.shuffle.partitions ) has changed from an integer to a string type. This change impacts image versions 2.3.30 and later in version 2.3 , and 2.2.82 and later in version 2.2 . Impact: This change only affects users who are programmatically setting the configuration in code using spark.conf.set() with an integer literal. Impacted example: spark.conf.set("spark.sql.shuffle.partitions", 100) User action: Update your code to pass a string literal instead of an integer. E
Notes (1) ›
- Managed Service for Apache Spark
New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions : 2.1.114-debian11, 2.1.114-rocky8, 2.1.114-ubuntu20, 2.1.114-ubuntu20-arm 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm 2.3.30-debian12, 2.3.30-ml-ubuntu22, 2.3.30-rocky9, 2.3.30-ubuntu22, 2.3.30-ubuntu22-arm
https://docs.cloud.google.com/release-notes#May_19_2026