How Yahoo Optimizes Apache Spark Workloads with Flexible VMs on Google Cloud
This post details how Yahoo leverages flexible VMs in Google Cloud's Managed Service for Apache Spark to enhance resource optimization and reliability for its massive analytics workloads. By defining a ranked list of acceptable VM shapes, the system automatically absorbs regional capacity fluctuations, preventing provisioning delays. This approach, which requires Auto-Zone placement and careful configuration, significantly reduces cluster creation failures and ensures continuous data pipeline execution for high-scale, deadline-driven environments. Yahoo has reported an 85% reduction in provisioning failures due to regional capacity stockouts.
- →Addressing VM Capacity Constraints with Flexible Configurations
- →Configuration Guidelines for Flexible Spark Clusters
- →Operational Benefits of Flexible VMs for Massive Workloads
- →Yahoo's Impact: 85% Reduction in Provisioning Failures
Notes (4) ›
- Addressing VM Capacity Constraints with Flexible Configurations
Yahoo utilizes flexible Virtual Machines (VMs) in Managed Service for Apache Spark clusters to automatically manage resource fluctuations. This eliminates the brittleness of fixed VM configurations, which can lead to delays in cluster provisioning due to regional capacity constraints.
- Configuration Guidelines for Flexible Spark Clusters
Implementing flexible VM configurations requires enabling Auto-Zone placement to search for capacity across an entire region. Users must also maintain core and memory symmetry across VM types when using autoscaling and align component properties for consistent YARN and Spark resource allocations.
- Operational Benefits of Flexible VMs for Massive Workloads
Flexible configurations significantly improve cluster creation success by allowing Managed Spark to select from a ranked list of VM types when the preferred option is unavailable. This approach, combined with auto-zone placement, leads to better regional resource utilization and reduced provisioning friction for large-scale data environments.
- Yahoo's Impact: 85% Reduction in Provisioning Failures
By adopting flexible VMs in Managed Service for Apache Spark, Yahoo successfully reduced cluster provisioning failures caused by regional capacity stockouts by 85%. This allows their data infrastructure to automatically handle capacity constraints, ensuring continuous workload execution and preventing downstream processing delays.
https://cloud.google.com/blog/products/data-analytics/how-yahoo-optimizes-apache-spark-with-flexible-vms/
Related releases
- Google Cloud details agentic data pipelines for MLOps with the Data Agent Kit Google Cloud Blog ·
- Managed Service for Apache Spark Updates Image Versions with Iceberg 1.10 and OpenLineage 1.49 Google Cloud release notes ·
- Terraform Google Provider v7.44.0 Adds New Resources and GCP Enhancements Terraform Google Provider Releases ·
- Critical Conda Channel Bug Fixed in Managed Service for Apache Spark Google Cloud release notes ·
- Google Cloud Composer updates with new Airflow builds, package upgrades, and deprecations Google Cloud release notes ·
- Cloud SDK 579.0.0: Breaking Changes, Feature Promotions, and New Commands Google Cloud release notes ·