DoorDash introduced a multi‑agent large language model (LLM) pipeline that automatically removes stale feature flags from more than 60,000 flags spread across 623 repositories. By feeding live experiment data, requiring engineer sign‑off, and using isolated Git worktrees with parallel agents, the system can generate validated pull requests in under 14 minutes at a cost of under five dollars per cleanup.
How the System Operates
The workflow pulls real‑time experimentation metrics from the MCP service to identify flags that are no longer exercised. An LLM‑driven agent proposes a removal plan, which is then presented to an engineer for approval. Once approved, the agent creates an isolated Git worktree, modifies the flag definitions, and runs automated validation steps before opening a pull request.
Architectural and Implementation Implications
Key components include:
- Multi‑agent LLM orchestration that can scale across hundreds of repositories.
- Integration with MCP for live data, meaning the pipeline must handle streaming or batch metric ingestion.
- Use of isolated Git worktrees to keep changes sandboxed until validation passes.
- Parallel execution of agents to keep overall latency low.
Practitioners need to consider how to provision LLM resources, manage prompt consistency, and ensure that generated code respects repository policies. The reliance on automated validation also implies a need for robust test suites that can catch regressions introduced by flag removal.
Operational and Security Considerations
Automation reduces manual triage effort, but introduces new operational surfaces: monitoring LLM output quality, tracking approval audit trails, and handling failed validations. Security teams should review the generated patches for unintended side effects, especially when flags gate security‑critical code paths. Cost tracking is straightforward—DoorDash reported an average of $4.79 per cleanup, which can be used to benchmark budget impact.
Related CloudNinjas coverage: DevOps.
What This Means For Practitioners
Teams with large flag inventories can prototype a similar LLM‑driven cleanup loop, starting with a small flag sample to validate PR quality and cost. Focus on integrating live experiment data, establishing clear approval gates, and building automated validation that mirrors existing CI pipelines. Continuous monitoring of LLM accuracy and cost will determine whether the approach scales beyond pilot phases.
