Entelligence benchmarks GPT-5.6 Luna against GPT-6 Astra on code review, and reports 74% precision at 28 times lower cost

Entelligence AI has published a benchmark that runs GPT-5.6 Luna against GPT-6 Astra on 50 public pull requests, using the same prompt on the same diffs and a paired-judge protocol to decide whether each finding is a real bug. The Hacker News thread has 65 points at the time of writing. Entelligence sells code-review tooling built on OpenAI models, so the numbers are worth reading with that in mind, and the company publishes every prompt, raw output, judge verdict and bug-class label alongside the article.

The headline numbers

By Entelligence's report, GPT-5.6 Luna found 69 verified bugs across the 50 PRs. GPT-6 Astra found 92. Total cost was $0.20 for Luna and $5.66 for Astra, a 28x per-review gap and a 20x gap per verified bug. Luna wrote 2,104 output tokens per review to Astra's 688, and still came in cheaper because Luna's output price is $1.20 per million tokens against Astra's $50. Mean review time was 23 seconds for Luna and 36 for Astra.

Precision is where the story turns. Entelligence reports that 74% of Luna's findings were verified, against 96% of Astra's, so one Luna comment in four did not hold up under the judges' review.

Where Luna falls behind

The gap is not even across codebases. Entelligence reports that Luna came within two verified bugs of Astra on Sentry, Discourse and Grafana. On Keycloak, an identity and access management server, Luna found 6 verified bugs to Astra's 14, and only 50% of its Keycloak findings held up against 93% for Astra.

The bug-class split points the same way. On security bugs, Luna found 9 of 24 and Astra found 19. Two Keycloak bugs Astra caught and Luna did not, as reported: federated recovery codes were never marked as used, so a code could be redeemed more than once, and a global view permission overrode denials set on individual clients. Catching either needs a model that has worked out what the permission model does after the change.

The judges, and the disclosed limits

Findings from both models, from GPT-5.6 Sol and from the public Entelligence reviewer comments were pooled per PR and judged separately by GPT-6 Astra and GPT-5.6 Sol. A bug counts only when both judges agree, and Entelligence reports 91% agreement. Astra is one of the judges, which Entelligence flags in its own limits section as a bias it cannot remove. Every PR predates both models' training cutoffs, so the "did the model just remember the fix" test one reader asked for is not possible on this benchmark.

Running both models on every PR found 117 of 143 verified bugs (82%) for $5.86 total, versus 92 for Astra alone at $5.66. Entelligence's own read: Luna is good enough for everyday correctness bugs at that price, and it would not let Luna review authentication or permission code alone.

The number that matters to a review pipeline is the class split, not the average. A setup that routes most pull requests through a cheap model and reserves an expensive model for files that touch authentication, permissions, cryptography and payments is exactly the shape this benchmark supports, if the reader can label their own diffs that way.

Source

Primary source: GPT-5.6 Luna vs GPT-6 Astra: Is a $1.20 Model Good Enough for Code Review?, by Aditya Jha for Entelligence AI. Benchmark data: AI-Code-Review-Evals on GitHub. Hacker News discussion: thread 49703003.