The launch of the Google Cloud Developer plugin revealed that judging agents by their final reply misses critical execution details. Engineers must shift to trace‑based evaluation to ensure reliability and operational safety.
Grade Execution Steps, Not Summaries
Initial tests used an LLM‑as‑judge to compare the agent’s closing text against a checklist. This approach failed to see what the agent actually did. In a Cloud Run cleanup scenario the grader marked the run as a failure because the final message warned about data loss instead of confirming that the help command had been run. The raw tool‑call log showed the agent executed gcloud help run services list as the first step, satisfying the rubric. The lesson is to capture and score the raw execution trajectory—commands issued, responses received, and validation steps—rather than relying on the agent’s summary.
Preserve Turn Structure for Asynchronous Calls
Agents often dispatch multiple tool calls in parallel within a single turn. Flattening these events into a sequential list erases the concurrency context, making it appear as if instructions arrived after the calls. The plugin’s evaluation uncovered a pattern where a skill‑loading call and two documentation searches were sent together. Because the skill’s guidance arrived only after the searches started, the agent ignored the intended instruction. Evaluation pipelines must retain the original turn boundaries so engineers can spot cases where parallel dispatch precedes the injection of guidance.
Test Live Runs and Dry Runs Side‑by‑Side
Relying solely on simulated "dry" executions hides discrepancies that only appear when real APIs are invoked. By running both live and dry scenarios and comparing the resulting telemetry, engineers can surface harness quirks such as unexpected parameter handling or silent failures. This dual‑run approach provides a safety net for detecting mismatches between intended behavior and actual API interactions.
Related CloudNinjas coverage: Google Cloud.
What This Means For Practitioners
Adopting trace‑centric evaluation changes how you instrument and monitor agentic systems. Consider the following actions:
- Instrument tool‑call logging. Ensure every external invocation, response, and status is recorded in a structured format.
- Maintain turn metadata. Preserve the original turn order and parallelism when exporting telemetry for analysis.
- Run paired live and dry tests. Automate a pipeline that executes both modes and flags divergences.
- Update evaluation rubrics. Shift checklist items from “did the agent say X?” to “did the agent execute X?” and verify with logs.
By aligning evaluation practices with these principles, AI engineers, platform teams, and SREs can detect hidden failures early, improve observability, and reduce the risk of silent misbehaviour in production deployments.
