AI Gateway’s custom‑cost mechanism now recognises two additional fields – per_cache_read_token and per_cache_write_token – that can be supplied in the cf-aig-custom-cost response header. By adding these rates, operators can reflect negotiated cache‑related pricing from any provider directly in the cost calculation.
What changed in the custom‑cost model
The JSON payload that AI Gateway expects for custom cost calculations can now include the following keys:
{
"per_token_in": 0.000001,
"per_token_out": 0.000002,
"per_cache_read_token": 0.0000001,
"per_cache_write_token": 0.0000005
}
If either cache‑rate key is present, AI Gateway switches to a cache‑token pricing mode. When a cache‑rate is omitted, the system falls back to the per_token_in value for that missing dimension. Supplying neither cache key leaves the original input‑and‑output token pricing untouched.
Why the addition matters to engineers and operators
Cache operations can dominate the cost profile of AI workloads, especially when large language models repeatedly hit the same embeddings or inference results. By exposing cache‑read and cache‑write token rates, teams can now align billing with the actual economics of their cache contracts, avoiding hidden expenses and enabling more accurate budgeting.
From an operational standpoint, the change eliminates the need for manual post‑processing to separate cache costs from regular token costs. AI Gateway automatically distinguishes cache tokens that are bundled with input tokens from those reported separately, ensuring that a single token is not counted twice.
Architectural and operational considerations
- Configuration updates: Any service that emits the
cf-aig-custom-costheader must be updated to include the new cache fields when applicable. Missing fields will default as described, but explicit values give tighter cost control. - Cost monitoring: Dashboards that track AI Gateway spend should be extended to surface cache‑read and cache‑write token metrics. This helps verify that the provider’s negotiated rates are being applied correctly.
- Provider integration: Providers that embed cache tokens inside the input token count need no special handling; AI Gateway will separate them internally. Providers that emit cache tokens separately must ensure the counts are accurate to avoid double‑counting.
- Testing and validation: Before rolling out to production, run a cost‑validation suite that exercises both cache‑read and cache‑write scenarios. Confirm that the total cost matches the expected sum of token and cache rates.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Update any custom‑cost generation logic to emit per_cache_read_token and/or per_cache_write_token when you have cache pricing agreements. Verify that monitoring pipelines capture the new cost dimensions, and run a validation pass to ensure no double‑counting occurs. Doing so will give you a clearer view of AI workload spend and prevent surprise billing from cache activity.