ReleaseBytes
gcp Google Cloud Blog ·

GKE Inference Gateway Enables Global AI Routing with Sub-1% Overhead

blogaigcpgaarchitectgcp-gke
feature

Google Cloud has introduced a multi-cluster GKE Inference Gateway architecture designed to manage globally distributed AI inference workloads efficiently. This system allows fragmented compute capacity across multiple GKE clusters to behave as a single pool, maximizing accelerator utilization and delivering near-linear throughput scaling. Benchmarks showed the gateway adding less than 1% routing overhead while achieving a 99.9% success rate for production-level traffic across 17,000 compute nodes. It leverages memory-aware scheduling based on live application signals like KV-cache token utilization to optimize load balancing.

  • →Unified Global AI Inference Routing
  • →Memory-Aware Load Balancing for LLMs
  • →Low Routing Overhead and High Throughput
  • →Seamless Integration with Distributed LLM Engines
  • →Lessons for Distributed AI Inference Deployments
Features (2) ›
  • Unified Global AI Inference Routing

    The multi-cluster GKE Inference Gateway enables globally scattered AI compute capacity to function as a single resource pool. It focuses on global, multi-region traffic distribution and high availability for AI workloads.

  • Memory-Aware Load Balancing for LLMs

    The system utilizes an Endpoint Picker Proxy (EPP) to read KV-cache token utilization from inference engines, emitting it as a metric for the load balancer. This enables smart routing decisions based on live application signals rather than traditional round-robin, preventing accelerator underutilization due to memory pressure.

Enhancements (2) ›
  • Low Routing Overhead and High Throughput

    Benchmarking demonstrated that routing traffic through the GKE Inference Gateway added less than 1% overhead, delivering 99.5% of the throughput of a direct, local cluster call. Scaling to three clusters achieved a near-linear throughput boost while maintaining a 99.9% success rate under heavy concurrency.

  • Seamless Integration with Distributed LLM Engines

    The multi-cluster GKE Inference Gateway integrates with GKE's native Service selectors and LeaderWorkerSet. This ensures that global traffic is routed to correct regional services while respecting underlying multi-node topologies like tensor parallelism.

Notes (1) ›
  • Lessons for Distributed AI Inference Deployments

    Key takeaways include the importance of smart, real-time signal-based load balancing, recognizing memory exhaustion as a bottleneck for agentic workloads, and planning for AI-scale latency in routing layers. These insights help maximize 'intelligence per dollar' from expensive accelerator fleets.

Read the original announcement →

https://cloud.google.com/blog/products/containers-kubernetes/gpu-and-tpu-utilization-with-multi-cluster-gke-inference-gateway/

Related releases