Datadog’s Agent Observability now includes an Online Experiments capability, enabling teams to run AI agent versions against actual production traffic and directly compare their outcomes. Practitioners care because it moves validation from isolated datasets into the live environment where real‑world load, data patterns, and latency are present.
Shift from Offline Datasets to Live Traffic
The new feature replaces the previous workflow that depended solely on predefined evaluation datasets. By injecting agent versions into the production request path, engineers can observe how changes affect end‑to‑end metrics such as response time, error rates, and resource consumption.
Operational Considerations
- Experiment traffic must be routed through Datadog’s observability pipeline, which may add minimal overhead.
- Teams need to define experiment scopes to avoid unintended side effects on critical user flows.
- Metrics collected during experiments should be isolated from baseline data to preserve clear comparison.
Security and Data‑Handling Implications
Running experimental agents on production traffic introduces the need to review data exposure. Practitioners should verify that any model code handling live payloads complies with existing data‑privacy policies and that experiment logs do not leak sensitive information. Monitoring for anomalous behavior becomes part of the security posture, as unexpected agent responses could surface new attack vectors.
Related CloudNinjas coverage: DevOps.
What This Means For Practitioners
Adopt the Online Experiments feature when you need confidence that AI agent updates behave correctly under real load. Start with low‑risk traffic slices, instrument additional observability points, and establish rollback criteria. Keep an eye on Datadog’s release notes for refinements to experiment controls and any guidance on data governance.

