GCP Managed Service for Apache Spark Changes Default Conda Configuration
Google Cloud details breaking changes in recent Managed Service for Apache Spark image versions, primarily affecting Conda channel configurations and default aliases. These changes require users to specify exact subminor versions and manually configure channels if needed, impacting cluster creation and package installations. The updates also introduce Iceberg 1.10 support and enable Spark skewed-join optimizations by default.
- →New Image Versions Lack Preconfigured Conda Channels and Default Aliases
- →Default Conda Channel Configuration Change Impacts All Supported Image Versions
- →Iceberg 1.10 Support Added to Dataproc 2.3 Images
- →Spark Skewed-Join and Self-Join Optimizations Enabled by Default
- →Spark Scheduler Exit Timeout Configured
Breaking changes (2) ›
- New Image Versions Lack Preconfigured Conda Channels and Default Aliases
Subminor image versions 1.3.96, 1.4.81, 1.5.92, 2.0.161, and 2.3.32 do not have preconfigured Conda channels or default aliases until August 25, 2026. Users must specify exact subminor versions (e.g., 2.3.32-debian12) when creating clusters. Packages cannot be installed via Conda without manual channel configuration.
- Default Conda Channel Configuration Change Impacts All Supported Image Versions
A default change schedule will impact all supported image versions, affecting Conda channel configurations. Workloads must use new images after August 25, 2026, as prior subminor versions with preconfigured Conda channels will be disallowed. Existing clusters may need to be deleted and replaced.
Features (1) ›
- Iceberg 1.10 Support Added to Dataproc 2.3 Images
Dataproc 2.3 images now include support for Iceberg 1.10. Users can opt-in by setting the dataproc:dataproc.iceberg.version cluster property to 1.10 during cluster creation.
Enhancements (2) ›
- Spark Skewed-Join and Self-Join Optimizations Enabled by Default
New GCE image versions now have Spark skewed-join and self-join optimizations enabled by default, potentially improving query performance for specific workloads.
- Spark Scheduler Exit Timeout Configured
The spark.scheduler.listenerbus.exitTimeout property has been configured to 30s for Compute Engine deployments, affecting how the Spark scheduler handles listener bus timeouts.
https://cloud.google.com/release-notes#managed-service-for-apache-spark
Related releases
- Terraform Google Provider v7.44.0 Adds New Resources and GCP Enhancements Terraform Google Provider Releases ·
- Critical Conda Channel Bug Fixed in Managed Service for Apache Spark Google Cloud release notes ·
- Cloud SDK 579.0.0: Breaking Changes, Feature Promotions, and New Commands Google Cloud release notes ·
- Managed Service for Apache Spark drops default Conda channels Google Cloud release notes ·
- Terraform Google Provider v7.42.0: New resources, BigQuery, Colab enhancements Terraform Google Provider Releases ·
- Terraform Google Provider v7.41.0: New Resources and Enhancements Terraform Google Provider Releases ·