Recent practice shows that AI code generators have accelerated the amount of code shipped, but the downstream testing effort has not kept pace. Datadog’s acquisition of Propolis introduces agents that can explore an application, follow user flows, and produce tests from plain‑language prompts, effectively moving the bottleneck from writing code to validating it.
Why the testing bottleneck matters
When engineers can produce thousands of pull requests per month, manual test authoring becomes a limiting factor. Traditional test scripts require explicit definition of user flows, and any change to those flows forces a rewrite. The result is a growing gap between release velocity and test coverage, which can increase the risk of regressions and performance issues in production.
Datadog’s agent‑driven test generation architecture
Datadog integrates the AI‑driven capabilities of Propolis with its existing observability stack. The agents receive a natural‑language description of a desired test scenario, navigate the target application, and emit test artifacts. Simultaneously, Datadog’s logging and tracing infrastructure captures the execution context, providing the data needed to assess both functional correctness and system performance. This coupling eliminates the need for hard‑coded test scripts that are brittle when the underlying flows evolve.
Operational considerations for integrating AI‑generated tests
Adopting AI‑generated testing requires adjustments in several operational areas:
- CI/CD pipeline changes: Test generation becomes a dynamic step that may run after code checkout and before build, requiring orchestration to ensure generated tests are available for downstream stages.
- Observability data handling: The logs and traces produced by the agents must be retained and correlated with the generated test results, which may affect storage and retention policies.
- Quality gating: Since tests are produced automatically, teams should define acceptance criteria—such as minimum coverage thresholds or performance baselines—to prevent low‑quality or flaky tests from entering the pipeline.
- Security review: Any code or configuration emitted by an AI agent should be scanned for secrets or insecure patterns before execution, as the source does not guarantee compliance with security standards.
Related CloudNinjas coverage: DevOps.
What This Means For Practitioners
Engineers should start by evaluating whether their current test coverage lags behind release cadence. If so, trialing an AI‑generated testing tool that integrates with an observability platform can provide immediate insight into the feasibility of shifting test authoring to agents. Key actions include:
- Map existing test creation steps to potential AI‑driven prompts.
- Instrument CI/CD pipelines to ingest generated tests and associated observability data.
- Define measurable quality gates for automatically created tests.
- Incorporate security scanning of generated artifacts as a mandatory step.
Monitoring the impact on release lead time, test flakiness, and performance regression detection will indicate whether the new approach delivers the intended acceleration without compromising reliability.