CoreWeave has added three tightly coupled services to its Forge platform: Model Distillation, RL Rollouts, and a programmable Serverless Training API. Together they close the loop between inference traffic and model updates, allowing teams to refine models without a full migration to a new generation.
What changed in the platform
Production requests are now intercepted by CoreWeave Agent Lens, which records inference inputs and outputs. The captured data is automatically relabeled, evaluated against candidate base models, and fed into a managed workflow that can run model distillation, supervised fine‑tuning (SFT), or reinforcement learning (RL). Candidates are deployed to Serverless Inference and compared head‑to‑head with the currently serving model before traffic is shifted.
At the same time, CoreWeave released a programmable training API (limited preview) that lets developers write Python loops that orchestrate learning logic while CoreWeave provisions the compute and storage. The API supports mixing RL updates with periodic SFT steps, and it integrates with the RL Rollouts service, which hot‑loads checkpoint weights into a live inference endpoint without dropping in‑flight requests.
Why engineers should care
- Cost efficiency: Distillation can produce a smaller model that mimics a larger, more expensive one, reducing inference spend.
- Faster iteration: Fresh production data can be turned into training data within the same platform, shortening the time from failure detection to model improvement.
- Operational simplicity: The serverless training API abstracts compute orchestration, letting teams focus on learning logic rather than cluster management.
- Zero‑downtime updates: RL Rollouts hot‑load new checkpoints while preserving active requests, keeping the service available during training cycles.
Architectural and operational implications
Adopting the closed‑loop workflow introduces several design considerations:
- Data pipeline: Agent Lens adds a capture layer in front of inference endpoints. Teams must ensure that captured payloads are stored securely and that any personally identifiable information is handled according to policy.
- Model versioning: Each candidate model is deployed to Serverless Inference for head‑to‑head evaluation. Practitioners need a systematic versioning strategy to track which checkpoint corresponds to which production run.
- Compute budgeting: Serverless Training spins up managed compute for each training job. Monitoring usage and setting quotas will prevent unexpected cost spikes.
- Continuous delivery: The workflow fits naturally into an MLOps CI/CD pipeline—triggered by new traffic data, running distillation or RL jobs, and promoting the winning candidate automatically.
- Security posture: Because inference traffic becomes training data, access controls around the Agent Lens storage and the training API become critical. Separate IAM policies should govern read/write access to captured logs versus model artifacts.
Sample programmable training loop
learner = await training.create_learner(model=model, lora=lora)
sampler = await serving.create_adapter(name="sampler")
optimizer = AdamConfig(learning_rate=1e-5)
for step in range(num_steps):
trajectories = await run_agent(sampler, tasks)
update = learner.batch()
update.forward_backward(build_rl_batch(trajectories), loss="importance_sampling")
update.optim_step(optimizer=optimizer)
if (step + 1) % sft_interval == 0:
update.forward_backward(next_demo_batch(), loss="cross_entropy")
update.optim_step(optimizer=optimizer)
update.publish_weights(to=sampler)
await update.result()
await (await learner.save_state("final")).result()
await learner.close()
The snippet shows how RL rollouts and periodic SFT steps can be combined while CoreWeave handles the underlying infrastructure.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Teams should evaluate whether their current model‑update process can be replaced by a closed‑loop workflow that leverages live traffic. Start by piloting Agent Lens on a low‑risk endpoint, define a head‑to‑head evaluation metric (e.g., win rate), and experiment with the programmable API in preview mode. Monitor compute costs, data‑handling policies, and checkpoint hot‑load latency to ensure the new flow meets reliability and security requirements before scaling to production workloads.


