Entropic Thoughts ships a browser AI code comment classifier with 77 percent balanced accuracy on cross-validation

The blog Entropic Thoughts published a rebuilt version of its AI code comment classifier on 2026-09-09. The tool runs in the browser, takes a block of code comments, and returns a calibrated probability that the text was written by a human or generated by a language model. Everything happens client-side, so nothing typed into the page leaves the browser.

What the numbers say

The author, kqr, reports a balanced accuracy of 77 percent on cross-validation, which they describe as how often the classifier gets the human versus robot verdict right when the two classes are equally likely. The confusion matrix breaks that down: given a known-human input the classifier calls it human 73 percent of the time, and given a known-robot input it calls it robot 80 percent of the time. Precision is 75 percent, recall 80 percent, F1 is 77.

A separate manual test on a smaller set of real-world comments, rather than the training-style synthetic set, scored higher: 88 percent accuracy, 89 percent precision, 86 percent recall. The post notes that non-synthetic cases were easier to tell apart than the training data, and that the browser tool also prints a calibrated probability on every classification, so an individual verdict comes with its own confidence figure rather than just the aggregate rate. When the classifier reports 80 percent confidence or more, the false positive rate drops to 5 percent.

How the data was built

The classifier was trained on a public corpus the author estimates at about $30 to collect if built cleanly the first time. The process: pick permissively licensed or copyleft repositories, check out their latest commit from 2021, sample a few random files, keep the human comments, then strip the comments and have LLMs write new ones for the same files. Balancing the token count per file across humans and each model keeps the classifier from learning to fingerprint files instead of style. The 2021 cutoff picks a period before code assistants were common in mainstream open source.

What is different from the previous version

An earlier version of the classifier was trained on partly private data and shipped without source. This one is built on public data and a better foundation, which is why the author felt able to publish the browser tool and the article at the same time. The demo also exposes feature-level activations: click any part of the classified text and the page shows which features triggered on that portion and how each one moved the overall judgment, instead of returning a single verdict with no explanation.

A team lead running review on a batch of pull requests can paste comments into the page and get a per-block confidence figure, along with the features that pushed the number one way or the other. That is a firmer starting point for a policy conversation than an unaided reading, especially on a project where AI-assisted contributions are permitted but expected to be labelled.

Source

• Better AI code comment detector, Entropic Thoughts, 2026-09-09