Managed Service for Apache Spark: Shuffle Partition Config Type Change

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The configuration for Spark shuffle partitions ( spark.sql.shuffle.partitions ) has changed from an integer to a string type. This change impacts image versions 2.3.30 and later in version 2.3 , and 2.2.82 and later in version 2.2 . Impact: This change only affects users who are programmatically setting the configuration in code using spark.conf.set() with an integer literal. Impacted example: spark.conf.set("spark.sql.shuffle.partitions", 100) User action: Update your code to pass a string literal instead of an integer. E

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions : 2.1.114-debian11, 2.1.114-rocky8, 2.1.114-ubuntu20, 2.1.114-ubuntu20-arm 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm 2.3.30-debian12, 2.3.30-ml-ubuntu22, 2.3.30-rocky9, 2.3.30-ubuntu22, 2.3.30-ubuntu22-arm