JioHotstar tested six retrieval backends, and plain file search won

Teams building coding agents reach for a vector database as the first step. A team at JioHotstar tested whether that step helps. On most of their tasks, it did not.

InfoQ reported on 11 November on an engineering study by Senior Staff Engineer Abhishek Bansal and his team at JioHotstar. They compared six ways to help a coding agent find things in a codebase. The piece was written by Leela Kumili. The original study was posted on the JioHotstar engineering blog, which would not open for us at publishing time.

What they tested

The team used Claude Sonnet as the coding agent. They gave the agent six different retrieval backends, one at a time. The agent then worked on the same tasks under each. The six were plain filesystem tools, Sourcebot, Graphify, CocoIndex Code, Repowise, and an in-house retrieval-augmented pipeline. Filesystem tools here means the plain set an agent already uses, InfoQ reports. That means text search, reading a file, and finding a symbol.

The tasks came from two places. One was subsets of SWE-bench Verified and SWE-bench Lite, plus CodeScaleBench. The other was a custom question-answering set from JioHotstar's own codebase. The agent then faced a repository the benchmarks had never seen. The team measured task accuracy, tokens used, the number of agent turns, and tool usability.

What the numbers said

InfoQ writes that the team "did not identify a single retrieval approach that performed best across all tasks". Structural code search parses the code and lets the agent follow symbols. It beat the other tools on clearly defined engineering tasks. Semantic retrieval, where the agent searches by meaning, helped more on ambiguous requirements.

The important finding is the plain one. Adding a retrieval layer on top of filesystem tools did not consistently raise task completion. It also did not consistently lower token use. InfoQ reports the agents preferred familiar operations. That means text search, reading a file, and locating a symbol. They used these over higher-level abstractions that reshaped the codebase for them.

The limit the paper's reviewers named

InfoQ cites Tarun Bhargava from a LinkedIn comment on the study. He asked for "cost/accuracy numbers with multiple runs of the same setup". The published results come from one run per setup. A single-run benchmark on an agent that can fail unpredictably is weaker than it looks.

The team's recommended combination is plain. Keep filesystem tools as the baseline. Add structural search for concrete code tasks. Reach for semantic indexing only when a requirement is ambiguous or crosses many files. InfoQ also cites Siddharth Vaithinathan from his own LinkedIn post on the study. He argued that "repository indexing alone cannot capture why architectural decisions were made". He named three things it misses: the reason, the constraints that shaped the decision, and what earlier incidents taught the team.

Source

JioHotstar Benchmarks Knowledge Bases for AI Coding Agents by Leela Kumili, InfoQ. The original study is on the JioHotstar engineering blog at blog.hotstar.com.