AI agents have traditionally sent every intermediate decision to a generative model, consuming tokens and compute that never appear in the final response. Kev, an open‑source family of decision‑only models built on Qwen 3.5, replaces that pattern with a single‑pass, non‑autoregressive inference that returns probabilities for yes/no, choice, or score queries without producing text.
What is Kev and how it differs
Kev comes in three sizes—0.8 B, 4 B, and 9 B parameters—and operates in a prefill‑only mode. The model receives the full agent state, a question, and a list of candidate actions, then runs a single forward pass. A pointer head extracts a probability distribution over the candidates, eliminating the need for an autoregressive decoding loop. The API mirrors TypeSafe’s System One primitives (Noul, Choice, Score), and the response format matches the /v1/systemone endpoint, allowing existing clients to point at a local Kev server.
Operational impact
Because Kev never generates text, token consumption drops dramatically. OpenAI researchers have reported agent workloads costing $7,000 per day, a figure that can be mitigated by off‑loading bounded decisions to a decision‑only model. Kev also supports batching: multiple questions can be evaluated against the same context in one pass using a block‑causal attention mask, with the 4 B model processing three questions in 277 ms (bf16 on an M5 instance). The open‑source Apache 2.0 release includes weights, training code, and evaluation tooling, giving teams the option to run and fine‑tune the model on their own infrastructure.
Implementation considerations
While Kev’s 9 B variant achieved 83.7 % accuracy on a locked out‑of‑domain test, the source notes that probability scores may not always reflect true confidence. Temperature calibration can drift on unseen data, which is critical for agents that use probability thresholds to decide whether to act or escalate. Fine‑tuning alters inherited capabilities, with reported declines on general‑knowledge and arithmetic tasks, especially in the smaller models. Because Kev can only score supplied candidates, it cannot hallucinate new options, but it also cannot compensate for missing or malformed candidate lists. Debugging failures often points to infrastructure rather than the model itself, suggesting that monitoring and observability of the inference pipeline remain essential.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Teams should evaluate whether their agent loops contain bounded decisions that can be isolated from open‑ended generation. If so, integrating Kev can reduce token spend, lower latency, and simplify scaling by separating decision logic from large generative models. Practitioners need to benchmark Kev against their existing generation‑based approach, watch calibration drift on production data, and plan for fine‑tuning trade‑offs. Because Kev is open‑source, organizations can host it behind existing security controls, but must still treat the model serving endpoint as a potential attack surface and apply standard inference hardening practices.
