Providing an AI SRE agent with pre‑aggregated cluster context dramatically speeds up fault diagnosis while keeping accuracy roughly the same. Practitioners who already use a coding assistant such as Claude Code should feed it the same context before benchmarking a new AI‑SRE offering, because the difference shows up in minutes rather than seconds during real incidents.
Experiment Overview
We injected 50 distinct faults from the SREGym benchmark into a three‑node Amazon EKS cluster. Both test arms used the same model – Claude Code on Claude Sonnet 5 via Amazon Bedrock – and the same judge model, Claude Opus 5, to score the first submitted diagnosis. One arm interacted with the cluster through a raw shell and kubectl commands; the other arm accessed a read‑only MCP server (Radar) that supplied four layers of context: current failures, a ranked list of broken objects, a change timeline, and resource relationships with filtered logs.
Impact of Context on Speed and Accuracy
Correct diagnoses rose marginally from 45 to 46 out of 50, and the mean judge score improved from 0.905 to 0.936. The real gain was in timing:
- Average time to submit a diagnosis dropped from 191 seconds to 64 seconds.
- Median time fell from 87 seconds to 44 seconds.
- 90th‑percentile latency shrank from 632 seconds to 94 seconds.
- Diagnoses within two minutes rose from 60 % to 86 %.
- Diagnoses within five minutes rose from 78 % to 90 %.
Tool calls per fault fell from 16.3 to 9.8, indicating fewer model turns and lower token consumption.
What the Context Provides
The read‑only MCP server returned data that would otherwise require multiple kubectl queries:
- Current failures – a ranked list of objects already known to be unhealthy, allowing the agent to start its investigation at the right point.
- Change timeline – recent spec changes and Helm release history, which survive beyond the default one‑hour event TTL.
- Resource relationships – owners, topology, and linked objects delivered together, avoiding per‑object lookups.
- Filtered logs – only error lines, with secrets redacted, so the agent can focus on diagnostics.
In a concrete fault where a Deployment lost the CART_ADDR environment variable, the shell‑based agent made 58 tool calls and spent 701.6 seconds, largely decoding a Helm secret to compare manifests. The context‑aware agent needed just seven calls and 34.8 seconds, because the change timeline already highlighted the missing variable.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
When evaluating AI‑SRE products, first enrich the existing coding agent with a read‑only view of the cluster that includes failures, recent changes, relationships, and filtered logs. Measure baseline speed, accuracy, and token usage before adding the new product. The data show that context alone can cut diagnosis latency by 60‑70 % and reduce API traffic, which translates into faster remediation and lower operational cost. If a vendor’s AI‑SRE claims superior performance, the comparison should be against this context‑augmented baseline rather than a raw shell interface.
