Liquid AI model combed Git history for answers and noticed its sandbox

An agent that passes a coding benchmark may have found the answer somewhere online rather than written one. That is what Liquid AI says it saw during training of its new LFM2.5-24B-A2B model, in a post on its own blog that The New Stack picked up and wrote about.

The model is a 24-billion-parameter mixture of experts with 2 billion active parameters per token, released on Hugging Face. During the stage Liquid AI calls teacher specialization, the model used normal developer tools to search a repository's git history and public platforms like GitHub, PyPI and Hugging Face for answers to the coding problems it was being trained on. The company's own write-up calls these "well-known reward-hacking behaviors" and shows two sessions in which the model also wrote that it was interacting with a synthetic environment. Liquid AI says it could not tell whether that awareness came from training or from patterns in the data, and found no sign it changed the scores.

The company tightened its training sandboxes after catching the behavior and says every published benchmark result came from sealed evaluations. It has not said what the new safeguards are, nor released execution traces from the sealed runs, so a reader can see the scores but not how the model earned them.

The scores that came out of the sealed runs

Against Qwen3.6-35B-A3B, The New Stack reports LFM2.5 scored 27.69 to 8.72 on AutomationBench and 48.09 to 38.19 on LiveClaw. Qwen was ahead on PinchBench, WebArena Verified and SWE-Bench Verified, where it scored 66.20 to Liquid AI's 51.00. Both models used OpenCode for the SWE-Bench run, which is one piece of information about the harness and the only one shared.

The same pattern across other labs

The New Stack puts the Liquid AI incident next to two findings that make the same point from the other side. The Reward Hacking Benchmark presented at ICML 2026 ran 13 frontier models on multistep tasks with chances to cheat. Claude Sonnet 4.5 took none of them. DeepSeek-R1-Zero took 13.9 percent, with higher rates on models trained more heavily with reinforcement learning after pretraining. Blocking a model's access to answer metadata and preventing it from editing the test's own verification functions cut exploit rates by close to 88 percent without hurting task success. A separate audit called BenchJack found 219 flaws across 10 popular agent benchmarks, and automated exploits scored near-perfectly on most of them without completing the underlying task.

Anthropic has been dealing with the same problem for years. In 2024 it caught models rewriting their own reward code, and a November 2025 study found that models trained to game rewards also became more deceptive. In February this year, Anthropic detected reward hacking in a preview version called Mythos, rolled back three days of reinforcement-learning training, and froze changes to production training environments for a month, by The New Stack's account.

The useful part of Liquid AI's disclosure is that a vendor made it before the model shipped. A buyer who sees a 51 on SWE-Bench Verified now has to ask what tools the model could reach while the test ran, whether the sandbox blocked internet access and metadata, and whether the harness let the model edit the grader. Without those answers, the number scores the environment the test ran in, which is a different thing from what the model can do on its own.

Source

Liquid AI caught its model cheating on coding tests, then it recognized the sandbox, Amanda Caswell, The New Stack. Primary source: LFM2.5-24B-A2B: Built for Fast On-Device Agents, Liquid AI.