Agent evaluation on Amazon Bedrock AgentCore has been expanded to include three evaluator sources—AWS‑provided built‑ins, DeepEval‑managed, and AutoEval‑managed checks—so you can run them as part of your deployment pipeline. The change matters because it moves the focus from surface metrics like latency to the actual business outcomes an agent must deliver, while also surfacing hidden risks such as PII leakage or incomplete task execution.
From Business Outcomes to Measurable Behaviors
Instead of picking generic metric names, start with the concrete result the agent is expected to achieve. For a support bot, that might be first‑contact resolution; for a research assistant, a fully sourced answer; for a moderation tool, accurate flagging. Break that outcome into observable actions—completing the user’s goal, staying on‑topic across turns, respecting compliance boundaries, and invoking backend services correctly. Then map each action to an evaluator that can capture it at the appropriate granularity, whether per‑turn (trace‑level) or across the whole conversation (session‑level).
Evaluator Catalog in AgentCore
AgentCore offers three tiers of evaluators:
- Built‑in evaluators are authored and quality‑tested by AWS. They are referenced by IDs such as
Builtin.GoalSuccessRateandBuiltin.Helpfulness, and the service handles model selection and execution. - Third‑party managed evaluators come from the open‑source libraries DeepEval and AutoEval. When used in managed mode, AgentCore hosts the evaluation code, selects a judge model, and runs inference on its own capacity, mirroring the built‑in experience.
- Custom evaluators let you start from a built‑in or third‑party evaluator and replace the underlying model or prompt to suit a specific rubric.
All three tiers can be applied at trace or session level, letting you choose the right trade‑off between AWS‑provided quality assurance and the control you need over model behavior.
Implementation, Ops, and Security Implications
From an architecture perspective, the evaluators run as a service call within the same AWS account, so you do not need a separate evaluation platform. Incorporating them into CI/CD pipelines means you can gate promotions on business‑outcome scores rather than raw latency or error rates. Operationally, the managed third‑party evaluators reduce the overhead of maintaining separate evaluation infrastructure, but they also introduce a dependency on the hosted judge model version; monitoring for model updates becomes part of the operational checklist.
Security considerations arise because evaluators can inspect the full agent transcript, including any user‑provided data. Ensure that evaluation logs are stored with the same protection level as production logs, and be aware that a mis‑configured custom evaluator could inadvertently expose PII if it forwards raw content to an external model. Using only built‑in or managed third‑party evaluators keeps the inference path inside AWS, simplifying compliance audits.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopt the three‑step mapping from business outcome to observable behavior and then to a specific AgentCore evaluator. Integrate the chosen evaluators into your test suites to catch hidden failures early, and treat evaluation results as a gate for production releases. Keep an eye on evaluator versioning, log handling, and any custom prompt changes that could affect compliance. By doing so, you gain a measurable, repeatable way to ensure agents do what the business expects while maintaining operational and security hygiene.
