The output-trimming hook experiment: a test that prints 3,000 lines

This is the fourth experiment in Reveneau's Claude Code token benchmark. It asks whether a hook that filters test output lowers the cost of a run whose tests print a lot. Reveneau is an AI software development consultancy whose code is written by AI and whose every change must pass an eval suite (a set of automated tests) before release, so test output enters its conversations many times a day, and a filter that works would be worth having in every build. Reveneau is independent of Anthropic. The full method is on the method page.

The words used on this page

A token is a piece of text the model reads or writes. The context window is all the text the model reads in one request, and the output of every command the model runs goes into it. Claude Code sends the whole conversation again on every request; the service stores the unchanged start, and tokens stored this way are cache write tokens, while tokens read back from the store on a later request are cache read tokens [5]. CLAUDE.md is a file of instructions loaded at the start of every session; this experiment used none. The effort level is a setting for how much reasoning the model does; this experiment passed no flag for it. A hook is a shell command that Claude Code runs on its own at a fixed point in its work: Anthropic's hooks guide, read on 4 October 2026, says hooks are user-defined shell commands that Claude Code runs at specific points, which gives deterministic control, meaning the action always happens instead of depending on the model choosing to do it [3]. A non-interactive run is claude -p with a prompt: Claude Code does the task and exits. A turn is one request to the model and its reply. A median is the middle value of five sorted runs.

What was compared

A second test file was added to the task. It prints 3,000 lines, each 67 to 73 characters long, and passes. It is published as task/noisy.test.js. The published files call it the noisy test; this page calls it the test that prints 3,000 lines.

The prompt was changed so that the model would run both test files with one command and see the long output:

One test in test/ledger.test.js fails. Fix the code in src/ so that node --test test/ledger.test.js test/noisy.test.js passes (run that exact command to check; the noisy test prints a lot). Change only src/. Do not change the tests. When done, stop.

The two settings were "direct", with no hook, and "hook", with a filter attached. Both ran five times on claude-sonnet-4-6 with no CLAUDE.md, no --effort flag, Claude Code 2.1.118 on a Claude subscription, the flags on the method page, and a fresh copy of the folder for every run, on 4 October 2026.

The hook

The hook is Anthropic's own example. Its costs page, read on 4 October 2026, has a section on offloading processing to hooks, which says a hook can preprocess data before Claude sees it, and gives an illustration: instead of Claude reading a 10,000-line log file to find errors, a hook can grep for ERROR and return only matching lines, reducing context from tens of thousands of tokens to hundreds [1]. That is Anthropic's illustration, and this experiment did not test it. The same page then gives a PreToolUse hook that filters test output to show only failures: a settings entry with the matcher Bash (the rule that says which tool the hook applies to), and a shell script that reads the command, checks whether it starts with npm test, pytest or go test, and if so rewrites it to pipe through grep -A 5 -E '(FAIL|ERROR|error:)' | head -100 [1].

Anthropic's hooks reference, read the same day, explains the two fields the script returns. PreToolUse fires before a tool call executes and can control whether it proceeds [2]. permissionDecision: "allow" skips the permission prompt, and updatedInput modifies the tool's input before execution, replacing the entire input object, so the script returns the whole original input with only the command changed [2]. The reference adds that Claude Code evaluates permission rules against the input the hook returns [2]. In plain words: every time the model asks to run a shell command, the script sees the command first, and if it is a test command the script swaps in a filtered version before anything runs.

Two strings in the published filter-test-output.sh differ from Anthropic's example. The test-runner pattern is ^node --test in place of ^(npm test|pytest|go test), because the task runs its tests with node, as the comment at the top of the script says. The word list is (FAIL|ERROR|error:|not ok), with not ok added; the script records no reason for that second change. The rest is as Anthropic published it. The hook was attached with the --settings flag, which Anthropic's CLI reference describes as a settings file whose values override the same keys for that session [4]; the settings file registers the script under PreToolUse with the matcher Bash.

The result

Every figure is Reveneau's own measurement from the run of 4 October 2026, a median of five, with the dollar figure being Claude Code's own list-price figure on a subscription account. The rows are in runs.json.

Setting	Pass	Cost median (min to max)	Cache read median	Cache write median	Output median	Turns median	Seconds median
direct, no hook	5 of 5	$0.159 ($0.149 to $0.173)	183,709	25,154	660	7	16
through the trimming hook	5 of 5	$0.201 ($0.164 to $0.245)	290,134	26,301	984	10	21

All ten runs passed. The hook setting cost more: $0.042 more at the median by Reveneau's arithmetic, with a median of 10 turns against 7, 290,134 cache read tokens against 183,709, and 984 output tokens against 660. Its range, $0.164 to $0.245, was the widest of any setting in the benchmark apart from Opus.

What the transcripts show

Claude Code saves a transcript per session, and Reveneau read the transcripts of all ten runs to find out why the filtered setting cost more. Two things were found.

In every one of the five direct runs, the model itself appended | tail -20 to the test command, so only the last 20 lines of output entered the conversation and the 3,000 printed lines never did. The largest tool result in those five runs was 869 characters. The direct setting therefore had no long tool result to filter: its median of 183,709 cache read tokens and 7 turns compares with 184,772 and 7 in the no-CLAUDE.md Sonnet setting on the method page, which had no long output at all.

In the five hook runs, the filtered command produced no output when the tests passed, and the model saw only Claude Code's 31-character notice, (Bash completed with no output). The pattern matches only failure words, and Node's summary lines, # tests 5, # pass 5 and # fail 0, contain none of them. The model then ran the test command again in other forms to confirm the result: with | head -60, with | grep, with ; echo "exit: $?". Across the five hook runs there were between 2 and 8 test commands per run, against 1 in each direct run. One hook run reached 15 turns. In all five hook runs the last test command ended with ; echo and the exit code; the hook rewrote that command too, placing the filter after the echo, so the filter applied to the echo alone, the test's full output came back, and Claude Code saved it to a file and showed the model a preview of its first part.

Reveneau's reading

A filter that removes the success summary costs turns. The model needs to know whether the tests passed, and when the filter leaves it nothing to read, it asks again, and every extra turn re-sends the whole conversation and reads the growing conversation from the cache once more, which is where the extra 106,425 cache read tokens at the median come from by Reveneau's arithmetic. On this task the model's own habit of adding | tail -20 already kept the output small, so there was no large tool result for the filter to remove.

Two conclusions follow, both limited to this task. First, a filter should keep the test runner's summary line, so the model learns the result in one read. Reveneau did not run a hook that does this, so there is no measurement of one yet; it is a candidate for a later dated run. Second, before adding a filter, read a few transcripts to see what the model already does with long output, because on this task it had solved the problem itself.

Anthropic's 10,000-line log illustration describes a different situation: a file the model would read in full, with no habit of trimming it. That case was not measured here and nothing on this page says the hook would fail there. The sister page on hooks that trim output covers Anthropic's guidance on where such a hook helps, and the agent costs guide covers the other way Anthropic suggests for long output, which is to give it to a subagent, a separate conversation the main one starts for part of the work.

What this experiment cannot show

One invented task of one failing test in a 30-line project, with one added test that prints 3,000 lines, is one task. The medians are medians of five, and nothing here is a statistic about Claude Code in general or about hooks in general.

The experiment ran one filter, Anthropic's example with two strings changed, and did not run a version that keeps the summary line. It did not measure a log file the model would read in full, which is Anthropic's own example. It did not measure what a filter does when a test fails, because every run passed, although the fail case is the one where the filter's pattern would match lines and return them.

The model was claude-sonnet-4-6 as resolved by Claude Code 2.1.118 on 4 October 2026; the same version resolved opus to claude-opus-4-7 and haiku to claude-haiku-4-5-20251001, and neither was run with the hook. Another model could handle long output differently; the tail -20 habit was observed in the ten Sonnet runs of this experiment and was not looked for on another model. The dollar figures are Claude Code's own list-price estimates on a subscription account. The limits page lists everything the run cannot show.

What Reveneau recommends

Reveneau's recommendation, drawn from Anthropic's documentation and consistent with this measurement: if you add an output filter, make it keep the line that states the result, and check a transcript afterwards to confirm the model reads that line once and moves on. Anthropic's costs page says to verify a hook with /hooks and that a debug file records when a hook rewrites a command [1]; a transcript shows what the model did next. A reader can measure their own filter against no filter with the do-it-yourself page, and the token usage guide explains why each extra turn costs a full read of the conversation.