Recent benchmarking of Gemma 3 12B and 27B on Google Cloud TPU v6e shows that generation‑heavy requests saturate the hardware far earlier for the larger model, while classification‑heavy requests exhibit comparable scaling for both sizes. Engineers responsible for model serving, capacity planning, and reliability need to adjust model choice, autoscaling triggers, and runtime flags to avoid hidden latency spikes and request loss.
TPU inference scaling for Generation Workloads
When the workload is dominated by decoding (low input, high output), the 27B model reaches a hard ceiling: throughput plateaus at roughly a 4.12× normalized multiplier with 128 concurrent users, and the curve flattens after 64 users. By contrast, the 12B variant continues to climb, achieving about an 8.19× multiplier at the same concurrency level. The data suggest that the larger parameter count exhausts the TPU’s decode bandwidth, creating a performance wall.
Practical implication
Recommendation: For high‑concurrency generation, either select the 12B model or enforce a per‑replica concurrency limit of 64 for the 27B model. This prevents the system from hitting the saturation point that leads to latency spikes.
TPU inference scaling for Classification Workloads
Classification scenarios—characterized by large prompts (≈4,000 tokens) and minimal output (≈10 tokens)—showed near‑identical scaling for both model sizes. At 128 concurrent users, both models delivered normalized throughput in the 6.0×–6.4× range without evident TPU saturation.
Practical implication
Recommendation: Deploying the larger 27B model for classification or summarization does not incur a throughput penalty, allowing teams to benefit from its higher capacity without sacrificing latency.
Operational Adjustments
The benchmarks were run on a GKE Autopilot cluster linked to a single‑host TPU v6e node pool with a 2×2 chip topology, using the vllm-project/tpu-inference stack. Two operational observations emerged:
- Standard CPU or memory‑based autoscaling policies miss the true saturation point; scaling should be driven by end‑to‑end latency metrics.
- Enabling the
VLLM_TPU_BUCKET_PADDING_GAPflag reduces memory pressure and can smooth out latency under bursty loads.
Both points highlight that TPU‑specific metrics and configuration knobs are essential for reliable scaling.
Related CloudNinjas coverage: Google Cloud.
What This Means For Practitioners
When planning LLM inference on TPUs, separate the workload type from the model size. Use the 12B model or enforce concurrency caps for generation‑heavy services, and feel free to run the 27B model for classification‑oriented pipelines. Adjust autoscaling to watch latency rather than CPU/memory, and enable bucket‑padding to mitigate memory‑related drops. Ongoing monitoring of throughput and latency across concurrency levels will reveal when the hardware hits its decode limit, guiding future capacity decisions.
