ReleaseBytes
gcp Google Cloud Blog ·

Google Cloud Dataflow Adds Pause/Resume and NVIDIA Blackwell GPU Support

blogaigcpgaengineermediagcp-dataflow
feature

Google Cloud Dataflow now offers generally available Pause/Resume capabilities for batch jobs, allowing users to restart failed long-running workloads or dynamically reallocate compute resources. This enhancement aims to improve compute efficiency, reduce wasted resources, and boost developer productivity for complex data pipelines. Additionally, Dataflow introduces support for NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs on G4 VMs to accelerate demanding AI inference workloads. These new features enable processing of large language models within Dataflow, optimizing costs and speeding up AI development.

  • →Pause/Resume for Dataflow Batch Jobs Reaches General Availability
  • →NVIDIA RTX PRO 6000 Blackwell GPU Support for AI Workloads
Features (2) ›
  • Pause/Resume for Dataflow Batch Jobs Reaches General Availability

    This feature enables Dataflow users to resume failed long-running batch jobs from the point of failure, preventing wasted compute resources. It also allows for manual pausing and resuming of jobs, facilitating dynamic reallocation of accelerated compute resources for higher-priority AI workloads.

  • NVIDIA RTX PRO 6000 Blackwell GPU Support for AI Workloads

    Dataflow now supports NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs on G4 VMs, providing 96GB vGPU memory and 1.6 TB/s bandwidth. This accelerates AI inference within Dataflow jobs, enabling the use of 70B+ parameter models with native ML capabilities like RunInference and GPU-enabled autoscaling.

Read the original announcement →

https://cloud.google.com/blog/products/data-analytics/new-dataflow-features-to-enable-large-scale-ai-workloads/

Related releases