What changed? AWS Security Agent, now integrated into AWS Continuum, was evaluated on a newly built benchmark that continuously pulls freshly disclosed CVEs and packages each as an isolated, container‑based challenge. Starting from an unauthenticated endpoint and a single hint, the agent had to discover the flaw, craft an exploit, and retrieve a hidden flag. The run covered 37 CVEs from 2025‑2026 and succeeded in extracting the flag through the intended vulnerability for 33 cases (89.2%).
New contamination‑reduced benchmark
Traditional public suites such as CVE‑Bench or XBOW reuse the same set of vulnerabilities, allowing large language models to memorize exploit steps and inflate scores. To avoid that bias, the team created a pipeline that:
- selects only very recent disclosures, limiting the chance that a model has already seen a full proof‑of‑concept;
- disables any runtime lookup of CVE identifiers, forcing the agent to rely solely on the live target;
- packages each vulnerability as a Docker‑Compose sandbox with a flag seeded at the exact sink the vulnerability is meant to expose.
Each package also includes a canonical PoC used later for an integrity audit, but the agent never sees it during execution. This design keeps the benchmark as close as possible to a real‑world, zero‑knowledge penetration test.
Performance highlights and limitations
The agent captured flags for 35 of the 37 challenges. Two of those captures were later re‑classified because the exploit path deviated from the intended sink, and the challenges were hardened accordingly. The remaining two misses occurred when the agent successfully exploited the bug but terminated the container before the flag could be read, a behavior that mirrors production guardrails that prevent destructive actions on live systems.
The target set spans eight vulnerability classes—including SQL injection, SSRF, and multi‑step XSS‑to‑admin‑cookie chains—and covers a variety of software stacks such as web frameworks, container registries, ML platforms, e‑commerce services, tRPC APIs, media platforms, and an LDAP admin console. This breadth demonstrates that the autonomous agent can navigate diverse environments without prior configuration.
Architectural and operational takeaways
From an engineering perspective, the results suggest a few concrete considerations:
- Continuous benchmark refresh: Maintaining a pipeline that ingests newly disclosed CVEs can keep automated testing tools from becoming stale against evolving threat landscapes.
- Isolation and guardrails: Running each test in a disposable Docker sandbox prevents accidental impact on production resources and provides a clean rollback point for missed or destructive runs.
- Auditability: Pairing each challenge with a canonical PoC enables post‑run verification that the exploit followed the intended path, which is essential for compliance and for measuring true automation capability.
- Coverage awareness: While the agent handled eight classes of bugs, practitioners should map their own asset inventory to those classes and identify gaps where manual testing remains necessary.
Operationally, integrating an autonomous pen‑test agent requires clear policies around when destructive actions are allowed, how results are logged, and how false positives (e.g., off‑path captures) are triaged.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Practitioners should view the 89% end‑to‑end success rate as evidence that autonomous testing can reliably surface exploitable flaws in fresh code, but they must also account for the benchmark’s contamination‑reduction methods when interpreting scores. Deploying such agents in CI/CD pipelines can provide early detection of regressions, provided that the test environment mirrors production isolation and that guardrails prevent unintended side effects. Ongoing evaluation of benchmark freshness, coverage of vulnerability classes, and audit trails will be critical to maintain confidence in automated results.