Pathway released a task generator for private ARC-AGI-1 tests

On August 11, 2026, Pathway published benchmark results for BDH-CQ, a 150-million-parameter reasoning model, and released arc-task-gen, an open source task generator that produces fresh ARC-AGI-1-style problems in the same distribution as the public evaluation set. The generator repository has 9,396 stars, from a creation date of August 4.

What the generator does

The arc-task-gen repository produces original tasks matched to the properties of ARC-AGI-1, so a private evaluation can be run on problems a model has not encountered before. Output uses the standard ARC format (a tasks.json file with train and test arrays) and is compatible with existing ARC evaluation harnesses. Pathway describes the private set as a complementary measure to the public benchmark, on the grounds that a public benchmark cannot fully isolate a model's ability to infer a rule from few examples from its potential prior familiarity with the specific tasks.

The model the eval was built to measure

Pathway says BDH-CQ scored 29.5% pass@2 on the public ARC-AGI-1 evaluation set at a computed inference cost of $0.0007 per task. The company says this is roughly 11 times cheaper per task than GPT 5.6 Luna (Low), even after OpenAI's 80% price cut of 5.6 Luna on July 30. Pathway says Luna scores 34.2% on the same benchmark. The cost figures are Pathway's, computed from measured hardware time; comparison costs are drawn from what other systems have reported to the ARC Prize Foundation's public leaderboard, as of July 2026.

Pathway describes BDH-CQ as a post-Transformer recurrent architecture in which memory is kept inside the model and updated at inference time, and queries are solved through iterative computation in a latent space rather than by generating a text-based chain of thought. The company argues this is what accounts for the cost gap, since a transformer-style reasoning system pays for extra tokens on every step.

The results were evaluated by Łukasz Kaiser, a co-author of the 2017 paper that introduced the Transformer architecture, who Pathway quotes as saying he replicated the ARC-AGI-1 numbers himself. The results were also reproduced by Richard Zhong, an NYU researcher, and by Remigiusz Kinas, a contributor to Bielik, according to Pathway.

Two useful things come from this release, whatever ends up being true about the model.

The first is the generator itself. Almost every team that runs internal evaluations against a public benchmark eventually has to deal with data contamination, since a benchmark that is easy to find with a search is also easy for a scraper to collect. A generator that produces fresh tasks in the same distribution is a way to run the same test on problems the model has provably not been trained on, and this one is open source under a permissive licence and compatible with the existing eval tooling.

The second is a reminder about how to read benchmark posts. The cost numbers Pathway reports for its own model are computed against measured hardware time. The comparison numbers are API prices reported by other systems to a public leaderboard, so the two sides of the "11 times cheaper" claim are measuring different things. The comparison is worth reading with the definitions attached.

Source

• Pathway's 150M-Parameter Model Breaks the ARC-AGI-1 Cost-Efficiency Frontier, Pathway, August 11, 2026.
• pathwaycom/arc-task-gen on GitHub. Star count read from the GitHub API on August 31, 2026.