ReleaseBytes
gcp Google Cloud Blog ·

Benchmarking Gemma 3 LLM Performance on Google Cloud TPUs for Diverse Workloads

blogaigcparchitectretailmediagcp-gke
announcement

Google Cloud details a comprehensive benchmarking exercise comparing Gemma 3 12B and 27B models on TPU v6e infrastructure. The analysis reveals that LLM inference performance varies significantly based on workload type, specifically distinguishing between decode-heavy generation and prefill-heavy classification tasks. For high-concurrency generation, the 12B model scales better, while both models show similar peak scaling for classification. Architects are advised to consider workload specifics for model selection and implement latency-based autoscaling for cost-effective deployments.

  • →Generation Workloads Hit Performance Walls with Larger Models
  • →Classification Workloads Show Parameter Size Parity
  • →Architectural and Scaling Recommendations for TPU-based LLM Inference
Notes (3) ›
  • Generation Workloads Hit Performance Walls with Larger Models

    For decode-heavy generation tasks, the Gemma 3 27B model experiences a performance plateau past 64 concurrent users on TPU v6e. In contrast, the Gemma 3 12B model demonstrates superior scaling, achieving nearly double the normalized throughput at 128 users, suggesting memory or compute limits for the larger model under high generation loads.

  • Classification Workloads Show Parameter Size Parity

    In prefill-heavy classification tasks, model parameter size has a negligible impact on performance. Both Gemma 3 12B and 27B models achieve similar peak scaling and operate efficiently within the hardware's capacity, allowing for the deployment of larger, more capable models without significant throughput penalties.

  • Architectural and Scaling Recommendations for TPU-based LLM Inference

    The analysis recommends scaling infrastructure based on End-to-End latency metrics rather than standard CPU/Memory triggers to prevent hardware saturation and request dropouts. Aggressive vLLM bucket padding optimizations are suggested to conserve memory, aligning model selection and serving configurations with specific input/output profiles for cost-effective AI inference.

Read the original announcement →

https://cloud.google.com/blog/topics/developers-practitioners/not-all-llm-workloads-are-equal-benchmarking-tpu-performance-on-classification-vs-generation/

Related releases