Instead of relying on one generic AI assistant to investigate alerts, raise tickets, and ship fixes, the engineers introduced a collection of narrowly scoped agents that operate inside the self‑hosted Sugabots workspace. The change reduced unnecessary wake‑ups for the on‑call engineer and gave the team a concrete audit trail for each step of an incident.
Why Separate Agent Responsibilities
Dividing work into distinct agents lets engineers review and correct only the portion they own. A triage agent decides whether an alert warrants immediate escalation, an in‑hours review, an auto‑fix, or no action. If the decision is wrong, the engineer can adjust the triage logic without touching the implementation logic. Conversely, an implementation agent receives a brief that includes the triage findings and produces a pull request. This separation makes it possible to evaluate the impact of a change to one agent in isolation, which the team found essential for building confidence in the automation.
AI Incident Triage Architecture
The core of the system is a pod – a shared environment that groups agents, tools, and state. Pods connect to external services through the Model Context Protocol (MCP), which abstracts the integration points (for example, Sentry, Linear, GitHub, and incident.io). Each agent runs its own model version and instruction set; the triage agent uses Opus 5.5 with prompts that cover evidence gathering, classification, escalation, handoff, and tool‑failure recovery. The backlog agent writes tickets to Linear, while the implementation agent creates a GitHub pull request. Because the pod’s connection and tool policies are defined centrally, all agents share the same access constraints while retaining independent behavior.
Operational Flow and Safeguards
The incident workflow proceeds as follows:
- The
Sentrytriage agent inspects a new or recurring issue, extracts impact data, checks for existing tickets, and selects one of four outcomes: urgent escalation, in‑hours review, auto‑fix, or no action. - If the outcome is an auto‑fix, the agent briefs the implementation agent, which generates a PR for engineer review.
- All follow‑up work is recorded in
Linearby the backlog agent, preserving a consistent backlog that can be prioritized during regular cycles. - When uncertainty exists about urgency, the instructions force an escalation to the on‑call engineer, ensuring no potential outage is missed.
- Noise alerts are archived but remain linked to a spike‑alert route back to
incident.io, providing a safety net if the classification later proves incorrect.
This design keeps the investigation visible as it moves between people and agents, allowing engineers to challenge assumptions, add evidence, or reassign responsibility without losing context.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Adopting a similar pattern requires attention to three practical areas:
- Responsibility granularity: Define clear boundaries for each agent’s output and ensure engineers can validate those outputs independently.
- Shared workspace configuration: Use a pod‑like construct to centralize tool connections and policy enforcement, leveraging a protocol such as MCP to avoid hard‑coding credentials in each agent.
- Escalation safeguards: Encode conservative fallback rules (e.g., always alert on‑call when urgency is ambiguous) to prevent silent failures.
By structuring AI‑driven incident handling this way, teams can iteratively improve individual agents, maintain auditability, and reduce on‑call fatigue without sacrificing reliability.
