Google says stale TODOs derail coding agents into unrequested work: Reveneau

Why it mattersOld TODO and FIXME comments quietly raise the cost of every agent session. A one-off cleanup pass pays back in steps, tokens and wait time.

Stale comments in a codebase quietly change how an AI coding agent behaves. That is the finding of a new study by two Google engineers. It was posted on the Google Developers Blog and written up by InfoQ.

The authors are Sam Saccone and Madhuri Garikapati. Saccone is a senior staff software engineer at Google. Garikapati is a senior staff data scientist. They worked from Google's own agent traffic. The sample was millions of file-reading sessions and hundreds of millions of agent steps, across thousands of engineers.

What the numbers say

Google found that 13 percent of sessions showed the agent an inline TODO(bug/...) or FIXME(bug/...) reference. In 6 percent of those, the agent then went and worked on that bug. 83 percent of those actions were not asked for by the developer.

Of those unprompted side quests, 41 percent chased closed tickets. 32 percent chased bugs that had already been fixed. 9 percent hit tickets marked "won't fix" or obsolete. 57 percent involved open bugs that were outside the scope of the task.

A side quest was not always a change to the code. 61 percent of them were lookup-only: the agent opened a few more files to read about the bug. 39 percent led to code edits on an open bug the developer had not mentioned. Both still ran up the step and token count.

The cost of a distracted session

Google ran a multi-way fixed-effects regression on its traffic. It controlled for the developer, the codebase, the task type and the model tier. It also controlled for files read, prompt length and where the agent met the bug reference.

A distracted session took 70 percent more steps and used 77 percent more tokens. The engineer waited 105 percent longer for it. The result held at p<0.0001.

The fix

Google wrote a presubmit analyzer and a scanner that clean obsolete TODO references out of the codebase. Over six months, the tool removed thousands of them. Human reviewers accepted more than 95 percent of its changes.

The authors frame the change as a context problem. Old specs, long-resolved bugs and outdated documentation used to sit quietly in the codebase. Now the agent treats them as current context, and acts on them. "Obsolete specs, long resolved bugs, and lagging documentation no longer rest," they wrote, "but rather are unintentionally ingested as context."

A team that cleans these up is cutting the input its agents read. Each session then asks for fewer files and finishes sooner. The authors suggest a presubmit check. It flags a TODO(bug/...) whose ticket is already closed, before the comment lands on the main branch.

Source

Google Developers Blog: Agent Distraction, by Sam Saccone and Madhuri Garikapati
InfoQ coverage, by Sergio De Simone

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.