ReleaseBytes
gcp Google Cloud Blog ·

GCP Introduces llm-d for Efficient RL Job Interleaving on Accelerators

blogaiinfragcpengineer
feature announcement

Google Cloud has introduced llm-d, a new project designed to improve the efficiency of reinforcement learning (RL) jobs for large language models. By enabling co-operative time-slicing, llm-d interleaves independent RL jobs onto shared hardware, significantly increasing accelerator utilization from approximately 40% to 70% without impacting model convergence. This solution addresses critical infrastructure bottlenecks in distributed RL, benefiting teams pushing the boundaries of AI research and development by reducing wasted compute and lowering total cost of ownership. The system is designed to optimize both synchronous and asynchronous RL workloads.

  • →Co-operative time-slicing for RL job interleaving with llm-d
  • →llm-d stack for improved RL infrastructure efficiency
  • →Addressing structural waste in distributed RL
  • →Time-slicing architecture layers
Features (1) ›
  • Co-operative time-slicing for RL job interleaving with llm-d

    The llm-d project introduces co-operative time-slicing to dynamically interleave independent reinforcement learning (RL) jobs onto shared physical hardware. This approach treats discrete RL steps as schedulable entities, increasing aggregate accelerator duty cycles from a ~40% baseline up to 70% without impacting model convergence or accuracy.

Enhancements (1) ›
  • llm-d stack for improved RL infrastructure efficiency

    The llm-d stack is designed to eliminate accelerator idle time for inference, agentic, and RL workloads. It includes a throughput-driven inference engine (llm-d-router), a high-velocity Agent Sandbox (recipe), and core pipeline primitives like the Weight Propagation Interface (WPI) to enhance reliability and speed in weight transfer.

Notes (2) ›
  • Addressing structural waste in distributed RL

    Distributed RL post-training often suffers from severe resource bottlenecks due to synchronous sampling and training phases running sequentially, causing trainers and samplers to sit idle. Asynchronous architectures can also experience idle gaps while waiting for trajectory batches. llm-d aims to mitigate this structural waste by optimizing resource utilization.

  • Time-slicing architecture layers

    The time-slicing system is organized into three layers: workload-scoped (user code with time-slice client library), cluster-scoped (control and orchestration plane managing job access), and node-scoped (hardware management performing checkpoint/restore swaps on accelerators).

Read the original announcement →

https://cloud.google.com/blog/products/containers-kubernetes/introducing-co-operative-time-slicing-for-rl-in-llm-d/

Related releases