CodeRabbit put Astra through its code review evaluation and found the gain sits in cross-file work

CodeRabbit published its evaluation of OpenAI's GPT-6 Astra for code review on 4 September. The company says Astra caught approximately 4 percent more labeled bugs through actionable findings than GPT-5.6 Sol, and 22 percent more than Opus 5. On the harder cross-file subset those gaps widen to 20 percent over Sol and 33 percent over Opus 5.

CodeRabbit sells code review, so it has a commercial interest in which model it runs and these are its own measurements. The company states the limits itself: the results describe one part of review performance, do not establish an overall ranking of review quality, and do not predict a team's defect rate.

What was measured

The metric is actionable bug coverage: how many labeled bugs a model catches through findings a developer can act on. CodeRabbit says the overall gain looks modest because the full evaluation includes simpler reviews where a stronger model has less room to separate itself, and calls the result early and directional.

The cross-file subset is the interesting half. CodeRabbit's reading is that Astra's advantage comes from connecting information that sits in different places, linking the intent of a change to consequences spread across a codebase. The company is careful to say the evaluation does not isolate why that improved, and that a larger context window on its own does not explain it.

The price of the extra reasoning

CodeRabbit lists Astra's standard API rates as 10 dollars per million input tokens and 50 dollars per million output tokens, checked on 4 September against published pricing, with Claude Fable 5.1 at the same base rates. Against a fixed illustrative task of 100,000 uncached input tokens and 10,000 billable output tokens, it puts Astra at 2.5 times the cost of Sol, about 4.7 times Terra and about 47 times Luna.

Those multiples describe token prices at fixed usage, not the cost of finishing a job. A model that needs fewer tokens or fewer attempts closes some of the gap, and CodeRabbit notes OpenAI reports lower estimated task costs for Astra in some of its own evaluations despite the higher token prices. That is OpenAI's claim about its own model, and it is exactly the sort of figure worth checking against your own workload.

Where this leaves the routing decision

The practical shape of the finding is that the benefit is concentrated. If the gain on straightforward reviews is a few percent and the gain on reviews with evidence scattered across files is five times that, then sending every pull request to the most expensive model pays a premium on the majority of cases to capture a benefit that only appears in some of them.

CodeRabbit's own suggestion is to run a candidate model beside the current one on the same tasks and compare answer quality, verification time and total cost, rather than deciding from a capability score or a token price alone. The harder question it does not answer is how a team identifies a hard cross-file review before the review happens, which is what any routing rule would need.

The post also describes using Astra to build a game in Godot as a second test of the same capability, reasoning about how one change affects the rest of a system. That part carries no measurement.

Source

• GPT-6 Astra in code review: Gains, privacy, and cost, CodeRabbit, 4 September 2026