vLLM’s inference engine has been updated to use a push‑based KV transfer mechanism instead of the traditional pull‑based approach. By moving the coordination and setup of the KV cache transfer to the prefill side, the decode instance can start generating tokens sooner, which directly improves latency and throughput for workloads that separate prefill and decode across different nodes.
Push‑Based KV Transfer Overview
The new design eliminates the serial round‑trip that previously required the decode node to request block identifiers, allocate destination buffers, and then issue a read operation. In push mode, the decode node registers its target blocks ahead of time, allowing the prefill node to begin the transfer as soon as its KV blocks are ready. This overlap of producer and consumer work removes a step from the critical path to the first token.
Implementation Details
Three participants are involved in the push workflow:
- Decode instance: announces the destination block IDs before the prefill computation completes.
- Prefill instance: matches each incoming request, prepares NIXL descriptors, and triggers the transfer.
- Background thread: runs on the prefill side, handling the actual NIXL write operation so the main forward‑pass loop remains uninterrupted.
The NIXL connector continues to rely on RDMA‑capable interconnects such as AWS Elastic Fabric Adapter (EFA). On supported instance types, EFA provides GPUDirect RDMA and Neuron Direct RDMA, enabling direct movement of KV blocks between GPU or Trainium memory without staging through host RAM.
Operational Impact
Benchmarks on a single‑prefill‑single‑decode (1P1D) deployment using EC2 P5en instances with EFA showed up to three‑fold reduction in time‑to‑first‑token and up to thirty percent improvement in time‑per‑output‑token for the Qwen3‑32B model. The gains are most pronounced under high request rates, where the serialized coordination of pull mode becomes a bottleneck. Because the feature was validated on both Trainium and GPU clusters, teams can expect similar benefits regardless of the accelerator, provided the underlying network supports the required RDMA paths.
Security and Reliability Considerations
Direct memory transfers over RDMA bypass host memory, which reduces the surface for accidental data exposure but also places trust in the integrity of the interconnect. Operators should ensure that EFA is configured with the appropriate security groups and that the instance types in use are listed as supporting GPUDirect or Neuron Direct RDMA. Monitoring the background thread’s health and the completion status of NIXL descriptors is advisable to detect stalled transfers that could delay token generation.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Teams deploying disaggregated vLLM should evaluate the push‑mode connector on their own workloads, confirming that their instance types support the required RDMA features. Enable the pre‑registration of decode buffers and verify that the background transfer thread is active. Track TTFT and TPOT metrics after the switch to quantify the benefit, and incorporate health checks for NIXL transfer completion into existing observability pipelines.
