Fast, Fault-Tolerant PyTorch Training on Databricks AI Runtime
Databricks details strategies for achieving fast, fault-tolerant PyTorch training on its AI Runtime, addressing challenges like frequent GPU failures and inefficient data pipelines at scale. The article emphasizes optimizing "goodput"—the productive computation time of GPUs—by implementing robust checkpointing and efficient dataloading mechanisms. It highlights the use of PyTorch's distributed checkpoint API (DCP) and asynchronous saves to reduce recovery costs and minimize idle GPU time. These best practices are crucial for engineers and data scientists building large-scale AI models to manage costs and ensure model integrity.
- →GPU failures are expected at scale in AI training
- →Optimize checkpointing for fault tolerance and recovery
- →Ensure efficient dataloading to maximize GPU utilization
- →Prevent silent model corruption with robust data pipeline checkpointing
Notes (4) ›
- GPU failures are expected at scale in AI training
Large-scale GPU training jobs face a high probability of interruptions, making rapid and automatic recovery essential for maintaining "goodput"—the proportion of time GPUs spend on productive computation—and managing total GPU spend.
- Optimize checkpointing for fault tolerance and recovery
The monolithic `torch.save` bottleneck can be overcome with PyTorch's distributed checkpoint API (DCP), which shards model state across ranks. Asynchronous saves, implemented via UCVolumeWriter on AI Runtime, further minimize idle GPU time, making frequent checkpointing affordable and significantly cutting recovery costs.
- Ensure efficient dataloading to maximize GPU utilization
Inefficient data pipelines that cannot keep pace with accelerators reduce goodput by starving GPUs. Overlapping data preparation with computation, facilitated by Databricks AI Runtime's UC volumes and UCVolumeReader, is critical to prevent idle GPU time.
- Prevent silent model corruption with robust data pipeline checkpointing
Beyond just model checkpointing, securing the data pipeline state is vital to prevent silent training-data corruption upon job resumption. Checkpointing the data pipeline alongside the model ensures full state recovery and model integrity.
https://www.databricks.com/blog/fast-fault-tolerant-pytorch-training-ai-runtime
Related releases
- Databricks Enhances Dashboards and Genie Code with New Features, Fixes Databricks Release Notes ·
- Databricks Genie One Enhancements: On-Demand Scheduled Tasks, Conversation Renaming, and Mobile Sign-in Databricks Release Notes ·
- Databricks Model Serving adds Zhipu AI GLM 5.3 Flash as a hosted model Databricks Release Notes ·
- Databricks Lakeflow Jobs Now Integrates Genie Code as a Beta Task Databricks Release Notes ·
- Databricks Introduces Private Network Gateway in Private Preview Databricks Release Notes ·
- Databricks Genie Agents Enhance Capabilities with GA Agent Mode APIs and Unity Catalog Functions Databricks Release Notes ·