Alibaba open-sources Open Code Review, a code review CLI that runs deterministic checks first and calls an LLM only for the rest

Alibaba has open-sourced Open Code Review, a command-line tool for reviewing pull requests that pairs a deterministic pipeline for file selection, bundling and rule matching with an LLM agent for dynamic analysis. InfoQ reported the release on 20 September. The project is Apache-2.0 licensed and stands at 38,109 stars and 2,715 forks on GitHub.

What is deterministic, what is not

The pipeline runs three fixed stages before any model is called: it picks which files in a pull request are worth reviewing, bundles them so the agent sees related changes together, and matches them against a built-in ruleset for common defects such as null pointer errors, thread-safety mistakes, XSS and SQL injection. Only what falls through those stages is handed to the LLM, which the README says "resolve dependencies dynamically" and produces line-level comments on the diff.

Alibaba says the tool "has served tens of thousands of developers and identified millions of code defects" inside the company over the two years before the release. The description in the repository calls it "battle-tested at Alibaba's scale."

The benchmark, as Alibaba reports it

The company built a benchmark called AACR-Bench from 50 open-source repositories and 200 real pull requests across 10 programming languages, cross-validated by more than 80 senior engineers, with 1,505 annotated ground-truth issues. The dataset is published on Hugging Face.

On that benchmark, Alibaba says Open Code Review reaches higher Precision and F1 than a general-purpose coding agent using the same underlying model, while consuming "only ~1/9 of the tokens" and completing reviews faster. The README also states that the tool "deliberately trades lower Recall for higher precision", so it finds fewer of the issues in the dataset but reports fewer false positives on the ones it does surface. InfoQ reported the comparison target as Claude Code and quoted a best-configuration recall of 20 percent on the same benchmark. Both numbers come from Alibaba's own testing; there is no independent replication.

Where it plugs in

The CLI accepts any OpenAI-compatible or Anthropic-compatible endpoint, so the choice of model is left to the caller. It integrates with GitHub, GitLab and Gerrit for the code hosting side, ships as an MCP server, and adds slash commands and skill definitions for Claude Code, Codex, Cursor, Kimi Code, OpenCode and QCA Forward. VS Code integration is listed alongside those.

What decides whether this is worth adopting

The pitch here rests on a claim about workload split: a lot of what a general coding agent spends tokens on when it reviews a diff is work a rule engine and a file selector can do for free, so keep those parts out of the model. If the one-ninth token ratio holds outside Alibaba's own benchmark, the review cost per pull request drops to a fraction of what a general agent charges for the same job on the same underlying model. That would move review from a per-merge check to something a team can afford to run on every push.

The tradeoff Alibaba states out loud is real. Lower recall means some defects a broader agent would catch will slip through, so this is a review layer that runs on every diff, then a human read for the merge itself. Two things to check against a team's own history before adopting it: whether the built-in ruleset covers the language mix the team uses, and whether the defects the tool misses on a real backlog are the ones that matter.

Source

• Primary: github.com/alibaba/open-code-review
• Reporting: InfoQ, September 20
• Benchmark dataset: AACR-Bench on Hugging Face