AgentCore Evaluations has been extended to accept code‑based evaluators that can invoke external decision‑model backends. By converting the standard evaluator payload into a request for a decision model and translating the returned probabilities back into an AgentCore result, teams can add fast, calibrated checks alongside the existing LLM‑as‑a‑judge evaluators.
Why Decision Models Matter
Decision models output a probability distribution over a predefined answer set instead of free‑form text. This structure provides a direct confidence metric that can be programmatically thresholded, aggregated, or used to trigger downstream actions. For questions that are narrow and repeatable—such as “Did the response stay on policy?” or “Is the answer grounded?”—a decision model delivers a deterministic label and a calibrated score, reducing the variance seen with generative LLM outputs.
How to Wire a Decision Model into AgentCore
AgentCore supports two custom evaluator styles: an LLM‑as‑a‑judge and a code‑based evaluator that runs in an AWS Lambda function. The code‑based path can call any external system, including a decision model. The integration steps are:
- Receive the
EvaluatorInputJSON that AgentCore supplies to the Lambda. - Map the relevant fields (e.g., user query, tool results, agent response) into the request format expected by the decision‑model API.
- Invoke the chosen backend—either the hosted Jev service from TypeSafe or the open‑source Strands Decider 2B you run yourself.
- Parse the returned probability distribution. Decision models support three question types:
noul: binary yes/no with a 0‑1 confidence.choice: up to 255 options, returning the top choice and full distribution.score: ordered levels (2‑10), returning a weighted score and confidence.
- Translate the model’s output into the
EvaluatorResultstructure required by AgentCore, including the label and numeric confidence.
Both Jev and Strands Decider can answer multiple questions in a single request, allowing you to use them as a screening layer that flags turns for deeper analysis by the built‑in evaluators.
Operational and Security Considerations
Running a code‑based evaluator introduces a Lambda function that must be maintained and monitored. Practitioners should consider:
- Latency: external model calls add round‑trip time; batch evaluation may tolerate higher latency than real‑time traffic.
- Version control: decision‑model APIs may evolve; keep the request mapping logic versioned alongside your agent code.
- Observability: log request payloads and model responses (sanitized) to track confidence trends and detect drift.
- Network exposure: the Lambda needs outbound connectivity to the decision‑model endpoint, which may require VPC configuration or internet egress permissions.
- Data handling: ensure any user‑generated content sent to the external model complies with your data‑privacy policies.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Decision‑model evaluators give you a deterministic, thresholdable signal for narrow policy or quality checks, complementing the richer explanations from built‑in LLM evaluators. When designing an AgentCore evaluation pipeline, consider placing decision‑model checks early to filter out obvious failures, then let the LLM‑as‑a‑judge evaluators handle the remaining, more nuanced turns. Monitor confidence distributions to fine‑tune thresholds and watch for drift that may require model retraining or rubric adjustment.
