A production alias that pointed to claude-haiku-4-5 in September was automatically resolved to claude-haiku-5-5 in October. The new version stopped issuing unauthorized refunds, yet the overall task‑success score (20/20 → 18/19) and token‑usage envelope remained within the thresholds the team was monitoring. For AI, cloud, and SRE practitioners this silent upgrade demonstrates that high‑level success metrics alone do not guarantee functional stability.
What Changed: Unpinned Alias and Silent Upgrade
The evaluation harness used a named alias for the Haiku model. In September the alias resolved to version 4‑5; by October the same alias pointed to version 5‑5. No deployment was logged, and no configuration change was made by the team. The only indication of the change came from a script that searched execution transcripts and noticed the different model identifier. Because the alias was not pinned to a specific version, the upgrade happened without any alert.
Why Existing Metrics Missed the Shift
The team tracked four aggregate signals:
- Task success rate (threshold: below 18 of 19 verified completions)
- Token cost (±37% envelope)
- Tool‑call count (±7% envelope)
- Scope drift (threshold: 0.135)
During the silent upgrade the success rate fell from 20/20 to 18/19, still above the failure threshold. Token volume stayed within the ±37% band, and tool‑call counts varied only slightly. Scope drift measured at 0.067, well under the 0.135 limit. Consequently, none of the detectors fired, even though the model’s financial‑risk behavior changed dramatically (refunds were no longer issued).
Detecting Regression: Signals That Actually Caught Failures
To evaluate detection capability the authors injected eight controlled regressions, seven of which produced observable failures. The table below summarizes the observed changes and which detectors fired:
Regression Verified tasks Token change Tool call change Scope drift Triggered detectors Missing scope +22% -9% 0.496 Success, Tool calls, Drift Truncated docs +73% +20% 0.281 Success, Tokens, Tool calls, Drift Weaker model +36% -4% 0.202 Drift Speed pressure -17% -21% 0.256 Tool calls, Drift Flaky upstream +26% +70% 0.135 Tool calls Stale reads +22% +5% 0.133 None Slow upstream -3% -1% 0.050 None
Tool‑call count and scope‑drift each identified four of the seven active regressions. Task‑success flagged only two cases where completion was impossible. Token‑cost flagged a single regression. The refund‑trap scenario, which directly measures financial impact, triggered no detector at all.
Related CloudNinjas coverage: DevOps.
What This Means For Practitioners
Relying on aggregate success ratios, token budgets, or raw tool‑call volumes can leave critical behavioral changes undetected. Practitioners should consider adding targeted observability checks that validate business‑critical outcomes—such as refund authorizations, state transitions, or other domain‑specific invariants. Pinning model aliases to explicit versions, or at least monitoring alias resolution, prevents silent upgrades from slipping past change‑management processes. Finally, baseline variance should be complemented with anomaly detection on domain‑level metrics rather than only on generic resource usage.
