Paper review · October 26, 2026

Automating Large-Scale Feature Flag Cleanup with a Multi-Agent LLM System: An Industrial Case Study at DoorDash

Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash

Agents Intermediate cs.SE

Using an Orchestrator agent and parallel removal agents to automate cleanup of stale feature flags, the system produced engineer-approved PRs for 45 of 50 flags (90%), at an average of $4.79 and 13.8 minutes each.

Problem

Feature flags are used for gradual rollouts and experiments, but once a rollout is complete they become technical debt and carry the risk that dormant code is switched back on. DoorDash has more than 56,000 flags (called DVs internally) across 623 repositories, and as of May 2026, 1,659 were stale. Manual cleanup takes 1–2 hours per flag (internal estimate). AST-based rule tools such as Piranha could not be applied in this environment because of dependency-injection wrapper patterns, non-boolean flags and changes spanning multiple files.

Method

It is a two-stage multi-agent system built with Google ADK. In stage 1, an Orchestrator based on Claude Sonnet 4.6 reads the Jira ticket, queries the experimentation platform via MCP for the actual production rollout state (rollout percentage, final value), finds code references and produces a cleanup report. After an engineer confirms the target value, in stage 2 removal agents based on Claude Opus 4.6 work in parallel, each flag in its own isolated git worktree (up to 4 per repository). A removal agent removes constants, wrappers and call sites, simplifies conditionals, fixes tests, gets builds and tests to pass, reaches at least 95% patch coverage and passes Detekt linting before creating a PR. Each agent also has a one-hour timeout.

Results

Of the 50 most recent flag cleanups, 31 (62%) were approved as-is on the first attempt, 14 (28%) were approved after minor edits, and 5 (10%) required engineer intervention. The overall cleanup success rate is 90%.
The first-pass CI rate was 88% (44/50). All 6 failures were due to insufficient patch coverage, not incorrect code changes.
Success rates by complexity were 100% (6/6) for simple, 94% (17/18) for medium and 85% (22/26) for complex cases. The authors attribute failures not to flag type or number of files but to deep call chains that pass through multiple interfaces.
Each cleanup took 13.8 minutes on average (median 12.3 minutes), with LLM API costs averaging $4.79 (median $3.49). This is much shorter than the internally estimated manual time (1–2 hours).
The authors report that no bugs or regressions were observed in any of the 50 generated PRs. Their interpretation is that the limitation lies in completeness (some dead code was missed), not correctness.

Limitations

Only 50 cases were evaluated, in one company's Kotlin repositories with a specific DV access pattern (dependency-injection wrappers). Each flag was run only once, so the effect of LLM nondeterminism was not observed. The complexity categories were assigned by the authors after the fact, and the 1–2 hours of manual cleanup is an internal estimate, not a measurement. Developer feedback was collected informally. There is no experiment directly comparing against baselines such as Piranha on the same data. Generalization to other languages or flag frameworks needs further validation.

Why it matters

It is a case of deploying coding agents on repetitive cleanup work in a real production codebase, with success rates, costs, time and failure causes published as numbers. The authors present a data source that queries actual deployment state, worktree isolation for parallel work and deterministic verification gates before PR creation as practical prerequisites. It is a direct reference for teams considering similar automation.

Key terms

Feature Flag: A mechanism for turning features on and off through runtime configuration without redeploying code. It is used for gradual rollouts, A/B tests and emergency kill switches.
MCP(Model Context Protocol): A protocol that lets LLM agents call external tools and data sources in a standard way. In this system, it was used to query the actual rollout state on the experimentation platform.
git worktree: A git feature that allows multiple working directories from one repository to be used at the same time. It was used to isolate parallel agents so they would not overwrite each other's files.
Patch Coverage: The share of lines changed in a given change that are executed by tests. This system requires at least 95% as a condition for creating a PR.

Original

Authors: Atharv Chandratre, Jai Datar · Posted 2026-10-22 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.