Vera Rubin NVL72 raises inference throughput and scaling efficiency – practical implications for AI infrastructure

The NVIDIA Vera Rubin NVL72 system entered the MLPerf Inference v6.1 benchmark suite and posted up to 3.7× higher throughput than the prior‑generation GB300 NVL72 on the Qwen3‑VL model, and up to 2.5× on DeepSeek‑R1. The gains stem from a combination of newer Tensor Cores, the Transformer Engine, NVFP4 precision, a sixth‑generation NVLink‑based interconnect, and software updates in Dynamo, vLLM, and TensorRT‑LLM.

Performance and scaling improvements

In the closed division of MLPerf Inference v6.1, Vera Rubin achieved the best offline, server, and interactive scores for the two tested models. The throughput advantage translates directly into more tokens generated per rack, which can increase revenue for services that bill per token or per request. The GB300 NVL72 platform, by contrast, demonstrated 99% scaling efficiency when expanding from a single 72‑GPU rack to four racks (288 GPUs), meaning that adding GPUs yields almost proportional throughput gains.

Architectural changes that drive the numbers

Key hardware updates include:

• Sixth‑generation NVLink and NVLink Switch: advertised as delivering ten‑fold higher packet rates and three‑fold lower latency compared with commodity Ethernet, enabling the disaggregated prefill‑decode serving model to operate efficiently at rack scale.
• Enhanced Tensor Cores and Transformer Engine: accelerate both the prefill and decode phases of large language model inference.
• NVFP4 precision: reduces the memory footprint of model weights, attention matrices, and KV cache, allowing higher batch sizes without a noticeable quality loss.

On the software side, the benchmarks relied heavily on:

• vLLM with the open‑source Dynamo inference framework for Qwen3‑VL.
• TensorRT‑LLM for DeepSeek‑R1.
• Continuous optimizations that delivered up to 1.6× higher performance over the previous MLPerf version (v6.0) and further gains after the v6.1 submission.

Operational considerations

Practitioners planning to adopt Vera Rubin should evaluate the following:

• Interconnect provisioning: The NVLink‑based fabric requires compatible switches and cabling; existing Ethernet‑only designs may need redesign to exploit the latency and bandwidth benefits.
• Software stack alignment: To capture the reported throughput, deployments must run the supported versions of Dynamo, vLLM, and TensorRT‑LLM. Keeping the inference stack up‑to‑date is essential for continued performance gains.
• Scaling strategy: The near‑linear scaling observed for GB300 suggests that adding more NVL72 racks can be cost‑effective, but only if orchestration layers can distribute requests without bottlenecks.
• Memory budgeting: NVFP4 precision reduces memory usage, but engineers should validate that the precision trade‑off meets their quality requirements for each model.
• Ecosystem compatibility: Partner submissions (e.g., Nebius) indicate that the platform can be integrated into broader solution stacks, but integration testing remains necessary.

Related CloudNinjas coverage: AI engineering.

What This Means For Practitioners

Vera Rubin’s benchmark results show that a modern NVLink‑centric rack can deliver substantially higher token throughput and maintain almost linear scaling as GPUs are added. Teams should assess whether their current interconnect and software stack can support these capabilities, and plan upgrades accordingly to reduce cost per token and improve service latency. Monitoring upcoming software releases and benchmark updates (e.g., the forthcoming MLPerf Endpoints for agentic workloads) will help ensure that deployments stay aligned with the fastest performance trajectory.