Coding assistant tool selection is no longer a theoretical concern; Armature’s recent study shows that agents such as Claude Code, Codex, and Cursor actively choose third‑party utilities (debuggers, package managers, email services, etc.) when they generate code. The behavior varies with repository language, the agent’s search strategy, and whether a simulated human orchestrator intervenes, meaning engineers must treat the assistant’s choice as part of the build and security pipeline.
How the agents decide
The experiment observed 17,000 tool‑choice sessions across 75 public GitHub repositories, with a validated subset of 5,292 sessions forming the core data set. Three agents were tasked with creating functional repositories that matched real‑world requirements, then asked to replace parts of the code and select appropriate tools.
- Cursor relied on web searches in roughly two‑thirds of its sessions.
- Codex performed a web search in 94 % of sessions, typically using site‑restricted operators.
- Claude Code used its internal priors for most decisions, searching the web only about 30 % of the time, but when it did, it examined three times as many pages as Codex.
These differing source strategies lead to divergent tool picks, even for the same functional requirement.
Contextual bias and language dependence
When the agents were asked to select an email/communication provider, the winning tool aligned with the repository’s programming language:
- TypeScript code chose Resend in 55 of 89 runs.
- Python code chose SendGrid in 22 of 24 runs.
- Go code chose Postmark in 20 of 24 runs.
- Java code chose Azure ACS in 22 of 23 runs.
This demonstrates that the same functional need can resolve to different third‑party services purely because of language‑specific patterns in the training data or the agent’s priors.
Operational and security implications
Because agents can embed third‑party SDKs, API keys, or configuration files without explicit human approval, the following considerations arise:
- Dependency management: CI pipelines must treat generated dependency lists as mutable inputs, scanning them for licensing, version, and supply‑chain risks each time an agent runs.
- Secret handling: The study used artificial API keys, but in production an agent could insert real credentials. Secrets management processes should enforce that any newly introduced keys are vetted before deployment.
- Observability: Tool selection is driven by the agent’s internal weighting of brand reputation (“gravity”). Monitoring which external services appear in generated code can surface unexpected shifts in the stack.
- Vendor strategy: Tool vendors that do not surface in the agent’s knowledge base risk being omitted from future codebases, effectively disappearing from the stack.
Related CloudNinjas coverage: AI engineering.
What This Means For Practitioners
Engineers should treat AI coding assistants as autonomous components that can influence the software supply chain. Practical steps include:
- Integrate automated SBOM generation and scanning into the post‑generation stage to catch newly added third‑party packages.
- Enforce a review gate that validates any new API keys or service endpoints before they reach production.
- Periodically audit generated code for unexpected vendor lock‑in, especially after model upgrades (e.g., moving from Codex to Astra or Fable 5.1 as Armature plans).
- Consider providing curated, agent‑friendly metadata (e.g., well‑documented SDKs) to increase the likelihood that preferred tools are selected.
By acknowledging that coding assistants now participate in tool selection, teams can adapt their architecture, CI/CD processes, and security controls to maintain visibility and control over the evolving stack.
