Gauge is a CLI that tests whether coding agents can use your docs

An agent that writes code for a team's customers is now a user of that team's docs. If the docs are unclear, the agent uses the wrong command. It cites a price that changed last month. It quotes a feature that was removed. Gauge is a new command-line tool that runs real coding agents against a product and reports where they get stuck. A team at Y Combinator open-sourced it this week.

The Show HN post reached 40 points on Hacker News. The repository is MIT-licensed on GitHub and had 103 stars when checked. It was created two weeks earlier.

What the tool does

Gauge works in a loop its authors call hillclimbing. A team defines a task, such as "install the SDK and send a test event". Gauge spins up a Docker container and runs a coding agent inside it. Claude Code, Codex, Pi, and OpenCode are supported. It records every command, output and comment the agent produces. The team reads the transcript, sees what the agent tried, then fixes the docs or the CLI. The same task is run again. The agent's behaviour on the fix is compared with the baseline.

The authors say Gauge finds classes of problem a team rarely sees on its own. A command that asks an interactive question the agent cannot answer. A deprecated code pattern the agent reaches for first. A feature the agent claims is unsupported because an old blog post says so. Workflows that confuse the agent across several commands.

The Clerk result

In the launch post, Evan at Gauge wrote that Clerk used it to move coding agents. The agents had been using a legacy "keyless" mode. Clerk wanted them to use its new clerk init command instead. Clerk validated a pre-release change to the SDK's error messages against the agent in Gauge, then released. Agents using clerk init went up three times after release, in Clerk's own figure.

That is one case, with one company, on one change. The measurement is Clerk's own. Still, the shape of the test is the point. A team can see what the agent does on a change before the change is published. That beats reading about the result in a support ticket a month later.

How it fits in a workflow

Gauge is written in TypeScript and needs Node.js 22.13 or newer. It pulls a pinned Docker image that holds the agent harnesses. A test run on one machine behaves the same way on another. A hosted version adds managed infrastructure and automated analysis. The open-source version runs on a developer's own machine with their own agent credentials.

Hacker News commenters raised fair objections. A developer who has done this manually asked whether automated repetition wastes model tokens. Another asked how a friction point defined for one model carries to another. Both are real questions. A team adopting Gauge would want to measure its own cost. It would also want to pick tasks whose answer is worth more than the model calls it takes to find.

Source

• gauge-sh/gauge-cli on GitHub, by the Gauge team.
• Show HN discussion.