Nebius reported a 66.9% reduction in the wall‑clock time needed to collect a full rollout batch for a reinforcement‑learning (RL) training step. The improvement came from re‑architecting how tasks are routed to containers, how sandboxed environments are executed, and how batches are assembled before they reach the trainer. For anyone running agentic AI workloads, the change demonstrates that the non‑GPU parts of the pipeline can be the primary limiter of overall training speed and cost.
Why rollout latency dominates RL training
In RL‑based agent training, a training step cannot finish until every trajectory in the batch is complete. Even if most attempts finish quickly, a few long‑running trajectories hold up the entire step, leaving GPUs idle while they wait for the final result. This “time to a trainable batch” metric often diverges from simple tokens‑per‑second throughput, because the latter masks the long tail of stragglers. When inference GPUs sit idle, the overall utilization drops dramatically, inflating both time‑to‑insight and cloud spend.
Concrete infrastructure changes that delivered the cut
The Nebius experiment focused on three levers:
- Routing optimisation: directing a conversation to a container that already holds the relevant cached history reduces repeated pre‑fill work.
- Sandbox execution improvements: streamlining the environment that runs the agent’s code cuts the latency of non‑GPU steps such as browser interaction or tool calls.
- Batch assembly redesign: grouping sibling attempts into a “ready bank” and only handing complete groups to the trainer removes ordering bias and lets the trainer pull work as soon as it is fully available.
These adjustments were applied to a Terminal‑Bench‑style workload and measured only the collection phase – the forward, backward, and optimizer phases of the trainer were excluded. The result was a two‑thirds reduction in the fixed‑weight collection time for a 256‑trajectory batch.
Operational and security considerations
Several practical implications follow:
- Instrument the pipeline to record time to a trainable batch alongside raw throughput. This surface metric reveals hidden bottlenecks.
- Track the outcome of each episode (model success, model failure, infrastructure failure, unknown). Excluding infrastructure‑related failures from policy‑gradient updates prevents the model from learning to exploit system glitches.
- Manage data staleness explicitly. When generation uses weights that are a few updates old, overlap between generation and training can improve utilization, but only if the divergence between the generating policy and the current policy is measured and kept within acceptable bounds.
- Scale CPU‑heavy sandbox capacity independently of GPU resources. The adage “your most expensive GPU is often waiting on your cheapest container” suggests separate autoscaling policies for each layer.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
To reap similar gains, teams should:
- Add precise timing hooks around rollout collection, grouping, and hand‑off to the trainer.
- Implement a lightweight cache‑aware router that prefers containers with relevant state already loaded.
- Design sandbox environments to be as lightweight as possible and monitor their latency separately from GPU work.
- Adopt a batch‑ready‑bank model: only release a batch to the trainer when all sibling attempts have finished, regardless of launch order.
- Log episode outcomes and filter out infrastructure‑induced failures before they influence the learning signal.
- Periodically measure policy divergence to decide how many updates old generation data can be before it should be down‑weighted or dropped.
By treating rollout infrastructure as a first‑class component, engineers can keep high‑end GPUs busy, lower overall training cost, and avoid subtle feedback loops that arise from unmeasured staleness or infrastructure failures.
