A researcher says AI models have poor research judgement on his projects

Arjun Panickssery wrote this week that the models doing "autonomous R&D" still have poor judgement. His short post on LessWrong reached 161 points on Hacker News. The thread has 166 comments. The claim is one person's observation across three side projects.

Panickssery said he is monitoring three builds. The first is a new state-of-the-art engine for Scrabble. The second is a bot for Ultimate Tic-Tac-Toe. The third is a research project on philosophy for an AI Philosophy contest. The claim in the post is not a benchmark score. It is a pattern across all three.

What he says the model does wrong

Panickssery writes that in every project the model's judgement is poor in a basic way. He names four faults.

On a literature review, Panickssery writes that the model misses the papers that are obviously most useful. It cannot sort by what matters. The model also runs experiments with no information value. The training set is too small to tell the compared architectures apart, he writes.

On the philosophy project, the model built a wiki over many tokens. Panickssery writes that it kept going down directions which were trivial. On Ultimate Tic-Tac-Toe, the model found a blog post with a clever idea. It did not recognise the post as useful. The author's strong online play and the post's recency would have made its value plain, he writes.

He names the model as Astra, which is Reflection AI's model.

A second data point

Panickssery wrote that he judged an Apart hackathon the prior weekend. He reviewed eight papers. All eight scored 60 percent to 95 percent on Pangram, a tool that detects AI-written text. The papers "were often very similar to each other", he writes. He adds that they were "methodologically incomplete in a vague but similar way". He treats this as a shared pattern, not one model's quirk.

Where he says the model does help

The post names an exception. "Hill-climbing tasks with fast feedback loops" are the one shape of task that works well, Panickssery writes. He gives an example. He improved a Scrabble move-generator benchmark by 10 times, he writes. He did so by prompting Astra to "try harder". He also asked it to aim for a specific performance number and to try every method in turn.

The implication is that a model can grind on a measurable score. It struggles when the task is to pick what is worth working on.

What the comments add

The Hacker News thread has 166 comments at the time of writing. Several readers agree. They write that models surface what is new to the reader. The model does not find what is new in the field. One developer writes that models produce derivative code patterns that lack elegance. Others cite cases where models did find novelty in a tight environment with enough iterations. Each of these is one commenter's experience, not a measured result.

For a team using coding or research agents, Panickssery's post is an argument to test on the team's own work. A vendor's benchmark will not show the quiet failure modes he names. The agent produces output. The output looks reasonable. The judgement embedded in it is where the gap sits.

Source

Arjun Panickssery's comment on LessWrong. Hacker News thread with 161 points and 166 comments.