Workers AI now includes a rejectIfBusy flag that forces synchronous inference calls to fail instantly when the platform cannot allocate compute capacity. Engineers who need predictable latency or who cannot afford request queuing should enable this flag and be prepared to handle the resulting error response.
How to Enable reject busy inference
In the Workers runtime the option is supplied as the third argument to the AI.run binding. The same flag can be sent in the request body when using the native REST endpoint.
const response = await env.AI.run(
"@cf/google/gemma-4-26b-a4b-it",
{ messages: [{ role: "user", content: "Explain capacity queues." }] },
{ rejectIfBusy: true }
);
For a direct HTTP call the JSON payload includes an options object:
curl --request POST \
--url "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/run/@cf/google/gemma-4-26b-a4b-it" \
--header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
--header "Content-Type: application/json" \
--data '{
"messages": [{ "role": "user", "content": "Explain capacity queues." }],
"options": { "rejectIfBusy": true }
}'
Operational impact of rejecting busy requests
When capacity is exhausted the platform returns an error instead of placing the request in a queue. This eliminates variable queuing latency, which can be valuable for real‑time user experiences or for services that enforce strict SLAs. However, the failure path now becomes part of normal operation, so monitoring must capture the rate of rejectIfBusy rejections and alert on unexpected spikes that could indicate under‑provisioned models.
Architectural considerations
Adopting reject busy inference changes the request flow:
- Clients must be prepared to retry, fallback to a smaller model, or degrade gracefully.
- Capacity planning can be more aggressive because the system will not silently buffer excess load.
- Service composition should treat the rejection as a first‑class error, similar to rate‑limit responses, to avoid cascading failures.
Because the flag is only applicable to synchronous calls, any background or batch workloads that can tolerate waiting should continue using the default queuing behavior.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Enable rejectIfBusy wherever latency predictability outweighs the convenience of automatic queuing. Update client error handling to recognize the specific rejection response, instrument metrics around its occurrence, and adjust model capacity or fallback strategies based on observed patterns. Treat the new error as a signal for capacity planning rather than a rare exception.
