KillSwitch is an adversarial programming language and the best LLM scored 66.8

A coding agent reads a short specification, writes a short program, and the task is one almost every modern model clears. Dominik Picheta thought that said less about understanding than people were reading into it, so he wrote a programming language whose whole job is to make a language model fail at the exercise. He calls it KillSwitch. On the three tasks he ran, Claude Opus 5 scored 66.8 and no model did better.

What the language does

KillSwitch is a stack-based interpreted language with functions, variables and conditionals. The design, Picheta writes, carries five traps aimed at a model that is pattern-matching rather than reading the spec. The specification itself contains "hidden lies which humans are unlikely to read, or which they are likely to recognize as lies": a model that absorbs the whole document and quotes from it will state the wrong rule. Identifiers must respect character-counting rules the interpreter enforces at run time. The grammar includes constructs such as "hack into a computer" that trigger a safety guardrail in some models. The phrase "ignore previous instructions" has a real meaning in the language, so a model that treats it as a prompt-injection flag refuses to parse it. And scripts are mostly narrative noise with the actual instructions placed by line count, so lifting the "code" from the "text" is itself the first task.

What the benchmark asks

The run has three tasks, which Picheta describes in plain text: implement Hello World, explain what a script printing the maximum of two numbers will output, and modify an existing script that converts Celsius to Fahrenheit so it does the inverse conversion. Each task is scored on correctness, and the full chart lives on bench.killswitch-lang.org. Picheta names four results in the post: Claude Opus 5 is the highest at 66.8, Fable 5.1 and Gemini 4 Argon score low because their safety guardrails refuse to run code that mentions "hack", and MiMo-V2.6-Pro is the strongest open-weight entry. The remaining scores for other models are on the leaderboard rather than in the post, so this item does not quote them.

What the author concludes

Picheta is careful about what the 66.8 means. "So does this prove LLMs can 'understand' the code?" he asks. "I would argue that the answer to this is 'no'." The tasks are, he writes, "fairly simple", and if the next generation of models cracks them he intends to design harder ones. The write-up is one person's weekend benchmark, with no attempt to run each model many times, so a single score is an upper bound on an unlucky run as much as a measurement.

The useful read for a team that depends on a coding agent is the shape of the failures. A benchmark the agent passes by paraphrasing the problem back and writing a textbook answer tells a buyer nothing about whether the agent read the spec or recognised the shape. A benchmark that lies in the spec, that counts characters, that embeds the real logic inside narrative noise, measures a different thing: whether the agent held to the exact text in front of it, or filled gaps from memory. For an engineer running an eval over a product that depends on a model reading documentation, writing bespoke adversarial inputs for the live corpus is more work than running SWE-bench, and it answers a question that SWE-bench cannot.

Source

KillSwitch: An LLM-Adversarial Programming Language, Dominik Picheta.