issue-rag measures RAG over 95 Vite issues and audits its own judge

Most RAG demos show that retrieval helps by asserting it, then leave the measurement to the reader. The open-source project issue-rag takes the opposite approach. It is a working answer system over one project's GitHub issue tracker. Its real content is the measurement around the system. The repository was published on 1 November 2026 under MIT.

The repository has 102 stars on 7 November. The code is Python, and the default target repository is vitejs/vite. The author's own README is the source for every number below.

What was measured

The author built a 95-question eval set by hand. Each question is a real issue title. Each reference answer is the maintainer reply that closed the issue. No question was written by a model.

The corpus holds 5,473 documents from 771 issues and 2,593 comments. Closed-book answers from deepseek-chat scored correct on 8.4 percent of items. A BM25 retrieval setup reached 64.2 percent correct. The weighted accuracy went from 0.221 to 0.700 under the project's judge, deepseek-chat. Both tables are produced by a python -m issue_rag.evaluate run, and the author says the numbers are not hand-edited.

The gain is reported as an interval

The author's main claim is that no single gain number is defensible. Under the stricter deepseek-reasoner grader, the same 95 stored answers moved from 0.305 to 0.711, a 2.33 times gain. Under the holistic rubric used in an earlier draft of the README, the gain was 4.42 times. Under a blind human review of 30 answers, the gain was 5.2 times.

So the project reports the range: 2.3 times to 3.2 times under model judges, and about 5 times in the one human sample. The author notes that this means the machine-measured gain is a floor, not an inflated figure.

Four retrievers, indistinguishable

The project tests four retrieval setups: bm25, dense, hybrid, and hybrid with a cross-encoder rerank. On title queries, accuracy spans 0.632 to 0.700. The ordering flips when the grader changes. Two runs of the same grader drift by 0.005 to 0.026. The author concludes that a ranking among the four cannot be defended on this data.

A separate test rewrites the 95 titles as plain user questions. The ranking then changes: the reranker wins at 0.795 against BM25 at 0.674. The author writes that the ranking of retrievers is not a property of the retrievers alone. It depends on how close a query is to the corpus wording.

Two findings that cut against the pitch

Retrieval raises the fabrication rate from 1.1 percent to 20 percent. Answers move from honestly uncertain to confidently wrong. And the retriever that returns the best context is not the retriever with the best answers. The project measures context by whether the answering comment is in the top five results.

For a team building a retrieval feature, the pattern the project shows is the useful part. Report an interval, run the grader twice to measure its own noise, and read a sample of answers blind. A single gain number from a single judge is the number you should not quote.

Source

• zaxiu88888/issue-rag on GitHub, read 7 November 2026.