Cloudflare cuts AI code review cost 68 percent by reordering a prompt

The spend chart on an internal AI code reviewer can double in five months without the architecture changing, and almost all of it hides in the first message sent to the model.

Cloudflare engineer Ryan Skidmore reports that the company's AI code reviewer averaged $1.19 per review in April and $2.46 by late August, with no change to how the coordinator handed work to specialist reviewers. After six weeks of tuning that focused on what reached the provider and in what order, Cloudflare says the average review now costs $0.77 at high effort, down 68 percent from the two most expensive weeks before the work. The median review time fell from four minutes nine seconds to one minute 32 seconds.

Where the money was going

Right before the work started, Cloudflare's AI review spend split 23 percent output tokens, 43 percent cache writes, 34 percent cache reads. Most of each dollar was paying for context the model had already seen or was about to see again. One session made 16.6 model calls on average and over those calls wrote 72,000 tokens to the cache and read back 1.08 million, because every call sends the whole conversation again.

Anthropic's prompt cache stores a prefix at a breakpoint, charges 1.25 times the input price to write it, and 0.1 times to read, with a five-minute expiry. From 14 September Cloudflare logged the hash of each session's first system block as sent to the provider, and found a cached prefix was reaching the first request only 14 to 26 percent of the time. Every session started with the same 11,000 to 21,000 shared tokens.

The one change that moved the number

OpenCode v1 put the agent prompt, the working directory and the date in one block with a single cache breakpoint at the end, so the shared half was never stored on its own and could not be reused between repositories. On 16 September Cloudflare added a marker to the end of every agent prompt and a plugin hook that split the block in two at that marker, with the shared half ending on bytes identical in every job. To keep it that way, the list of available skills was disabled and tool descriptions were frozen between jobs.

That change alone, together with moving the internal standards reviewer's reference index into the prompt instead of reading it with three tool calls per session, took the cost per review from $2.47 to $2.02.

Fewer calls, cheaper models

Smaller changes on 21 and 23 September took another 20 cents off every review: removing tools that sub-reviewers kept being rejected from calling anyway, reading files in parallel up to 150 KB in the first response, skipping reviewers on re-reviews when nothing new had been touched, and stopping after the review was posted instead of handing the whole conversation back to the model for a summary nobody read.

Model and effort changes made up $1.06 of the $1.67 saved. In a split from 3 October, Sonnet 5.5 was 62 percent cheaper per run than Sonnet 5 for code quality, 58 percent cheaper for performance, and 47 percent cheaper for security. Both models cost the same per token, so the saving came from using fewer tokens.

Cost per review is not the only measure, so Cloudflare blind-sampled 657 findings and had two models from different providers judge each one. Important findings per review held at 0.30 against 0.28 before. Severity marked as higher than deserved dropped from 0.46 per review to 0.14, a 70 percent fall.

Source

Cutting our inference bill by 68% by being obsessive about optimization by Ryan Skidmore, Cloudflare.