Study finds coding agents choose grep over LSP code navigation

AgentConnect engineer Pengcheng Xu has posted a study of Claude coding agents choosing between two ways to find code in a repository: plain grep and an LSP-backed semantic navigation path. The write-up was dated 2026-08-12 on the AgentConnect blog and reached the front page of Hacker News this morning, where the discussion has 56 points.

The study compared three Claude models, Opus 4.8, Sonnet 4.6 and Haiku 4.5, on Python and TypeScript repositories across several task types. Token counts were only compared on runs that both approaches finished, which the author said is meant to stop a failed early exit from looking efficient.

Where each tool won

On simple code-location tasks with both tools available, the three models picked the semantic tool only 0 to 6 percent of the time. Forcing them to use the semantic tool first cut task success from 100 percent to 89 percent in that test.

On reference-completeness tasks, where the agent had to find every caller of a function, the same models picked the LSP path 45 to 57 percent of the time on their own. Runs with the LSP reached 1.00 precision against 0.76 for grep. Recall stayed near 0.66 in both setups, so the LSP removed false matches without finding more true ones, and on the stronger models the precision gain came with higher token use rather than a saving.

The codebase mattered as much as the language. On the clean TypeScript library remeda, the LSP added no F1 and used 16 percent more tokens. On the less clean TypeScript codebase hono, it added 0.246 F1 and used 12 percent fewer tokens. On the Python requests library, LSP added 0.072 F1 and used 19 percent more tokens. The predictor was how badly grep's precision degraded on that specific codebase, which does not reliably follow from whether the language has strong static types.

The output format changed the results

The tested LSP tools originally returned only file, line and column, so the agent had to open each hit to read the code. The author changed the response to include roughly two lines of source at each reference, with the same underlying semantic backend. On a multi-file rename task with Opus 4.8, pass at 1 rose from 0.67 to 0.83, and follow-up file reads fell from 15.2 per episode to 3.2, below grep's own 4.3.

Grep still covers what semantics cannot

The study said semantic references are a subset of textual occurrences, so grep will always cover renames that also touch comments, strings and config, where find_references will not. It also said the same model can behave differently across two harnesses because training gives it concrete action patterns to reuse, and a familiar prompt-call-result loop is part of the capability the model brings. A team adding a fancier retriever to a coding-agent stack should measure the whole loop from start to finish, and should treat the format of what a tool returns as a design decision of its own.

Source

Primary source: Grep beats LSP? Why coding agents ignore your fancier tools, by Pengcheng Xu on the AgentConnect blog. Community discussion: Hacker News item 49560260.