Amazon CloudWatch now ships CloudWatch Omni, an observability layer built for AI agents that lives outside the traditional AWS Management Console. It adds IDE‑embedded trace collection, built‑in evaluation metrics, and a standalone web console, letting engineers monitor and debug agent behavior without switching contexts.
Unified Trace Capture and Built‑In Evaluators
Omni records every execution trace of an agent and attaches a set of evaluators that score correctness, coherence, retrieval quality, and tool selection. The data model supports side‑by‑side comparison of prompt revisions, creation of test datasets from live traffic, and automated regression detection across configuration changes. Because the traces are captured at the agent level, engineers can see the exact sequence of calls, tool invocations, and model responses that led to a given output.
IDE Extension and Separate Operator UI
Developers interact with Omni through a native extension for VS Code and Kiro. The extension surfaces traces in‑line with code, provides a one‑click playground, and exposes the evaluators without leaving the editor. A Cloud Login option lets the IDE forward telemetry to Amazon CloudWatch for persistent storage and team sharing; the connection is optional, so all data can remain local during early development.
Operators use a distinct web experience that is accessed via SSO and does not require the AWS console. The UI shows the same trace data that developers see, enabling a shared view of production fleets while keeping the monitoring surface lightweight.
Architectural and Operational Considerations
Omni relies on OpenTelemetry‑style instrumentation to inject trace points into the agent code. The instrumentation step is guided by the extension, reducing manual setup. Because the data path can stay local or be forwarded to CloudWatch, teams must decide where trace storage lives and what latency is acceptable for analysis.
- Model‑agnostic design: The service works with any model provider, framework, or runtime, so existing agent stacks do not need to be rewritten.
- Integration with AI code assistants: Kiro, Claude Code, and Codex can automate dev‑server configuration and dependency installation, shortening the time from install to first traced run.
- Operational workflow: The eval‑driven workflow encourages building test datasets from production traffic and running experiments before promoting changes, aligning observability with CI/CD pipelines.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Practitioners should start by installing the Omni extension in their preferred IDE and running the sample project to become familiar with trace navigation and built‑in evaluators. Evaluate whether local‑only telemetry meets your security and compliance posture, or enable Cloud Login to centralize traces for team collaboration and production monitoring. Incorporate the evaluator suite into automated test suites to catch prompt regressions early, and configure SSO access for operators to ensure a consistent view of agent health across development and production environments.
