The eval suite as a diligence artefact: ask for the suite, the pass gate, and a failing run

The eval suite is the artefact to ask for first on any AI target, because it is the only document that can show whether the company measures its own product quality. Everything else in the data room is a description. The suite is a measurement, and you can run it.

This page assumes you know what an eval suite is. If not, eval-driven development is the guide that explains it, and evals vs tests vs code review explains how it differs from a test folder. Here the question is narrower: how do you read a suite in a diligence session, and what does each of its three parts tell you?

Why is the eval suite the artefact to ask for?

The eval suite is the artefact to ask for because a company that measures its quality can show you the measurement, and a company that asserts it cannot. Every other artefact, the deck, the demo, the architecture diagram, the customer references, is produced to be shown. The suite is produced to be run, and a suite that gates releases has a history you can read.

Reveneau builds its own software this way, with an eval suite written from the specification that every change must pass before it ships, and so its diligence checklist asks the target for the same three things it would have to show a client: the suite, the gate, and a failing run. That is why the request is specific. "Do you have evals?" gets a yes. "Show me the run that failed last month" gets a document or a silence.

Both major providers now tell their developers to build this way. Anthropic's guidance on developing test cases says to "prioritize volume over quality", meaning more cases with automated grading beat fewer cases graded by hand, to automate grading where possible, and to design cases that mirror the real task distribution including edge cases; it describes code-based, human and model-based grading as the three methods [1]. OpenAI's evaluation guide defines evals as tests of model outputs against "style and content criteria that you specify" and calls them essential when upgrading or trying new models [2]. A startup that followed either guide has a suite you can read in an hour. One that did not has told you how it ships.

What are the three things to ask for?

Ask for the suite, the pass gate, and a failing run, in that order, because each one answers a different question.

The suite answers what is measured. Ask for the cases, the graders, and the last ten runs with their scores. Count the cases; hundreds is current practice, tens is a demo. Read how they are graded: code-graded checks are the most reliable, model-graded ones are useful for judgement calls and should use a different model from the one being tested, and human-graded ones are slow and rarely re-run. Check whether the cases were derived from the specification or from the code; a case that asserts what the code already does is a regression test, useful for catching changes and silent on whether the product does what was promised.

The pass gate answers what is enforced. Ask what score a change must reach before it merges, who set it, and whether the gate is in the pipeline or in someone's head. A gate in the pipeline blocks the merge. A gate in a policy document blocks nothing. Ask what happens when a case is flaky: is it fixed, quarantined with a ticket, or deleted?

A failing run answers whether the measurement has ever mattered. Ask for the most recent run that fell below the gate, what change caused it, how it was noticed, what was done, and how long that took. A team with a real suite has a story for this and it takes two minutes to tell. A team whose suite has never failed either has a perfect product or has never used the suite as a gate, and only one of those is possible.

How do you read a suite in a session?

Read the suite in a session by running it, then reading its history, then reading its cases, in that order, so that the numbers are in hand before the interpretation starts.

1. Run it on a clean checkout of the current commit. Note the score, the run time, and whether it calls the model or stubs it. A suite that stubs the model tests the code around the model and nothing about the model's output.
2. Compare the score to the gate. If the current commit is below the gate and shipped anyway, the gate is a number on a page.
3. Read the last ten runs. Scores over time, and what changed between runs. A flat score across a quarter of active development means the suite does not touch what changed.
4. Read the failure history. Every run below the gate, and its resolution. This is the part that cannot be produced for the meeting.
5. Map the cases to the claims. Take the claim table from how to verify an AI claim with a test and, for each claim that three live inputs could not test, find the cases in the suite that cover it. An accuracy claim with no cases behind it stays untested.
6. Run your three held-out inputs through the graders. The same inputs from the live test. If the graders score them sensibly, the graders are doing work. If the graders pass anything, they are decoration.
7. Ask what is not covered, and who decided. Every suite has gaps. A team that can list its gaps is managing them. A team that says the suite is complete has not looked.

What to check in an eval suite has the engineer's checklist for step 5 through 7 in more depth, and when evals give false confidence lists the ways a green suite can be wrong.

What does a weak suite look like?

A weak suite looks healthy at a glance and fails on the history. The table gives the signs.

Sign	Strong suite	Weak suite
Case count	Hundreds, growing with each spec change	Tens, unchanged since the tool was set up
Grading	Code-graded where possible, model-graded with a different model, documented rubrics	Model grades its own output, or a person eyeballs it
Derived from	The specification and reported customer failures	The code as it stands
Pass gate	In the pipeline, blocks the merge, threshold documented	In a policy, or "we look at it"
Failure history	Several failures a quarter, each with a resolution	None, ever
Model coverage	Calls the production model at the pinned version	Stubs the model, or calls whatever is default
Held-out inputs	Graded sensibly	Passed regardless

The middle rows matter most. A suite derived from the code, graded by the model that wrote the code, with a gate nobody enforces, is three layers of the same thing checking itself. It will be green.

What if there is no suite?

If there is no suite, the company has no measurement of its own product quality, and every accuracy or reliability claim in the deck is untested by definition. That is a finding, and it has a cost attached: building a first suite from the specification, or from the code if there is no specification, is a defined piece of work that belongs in the post-close plan and in the price. Adding evals to an existing codebase describes that work from the engineering side, and the business case for evals puts a shape on what it costs and saves.

There is a version of "no suite" that is worse, which is a folder of model-written unit tests presented as one. Those tests confirm that the code does what the code does. Stack Overflow's 2025 survey found 66 percent of developers name output that is "almost right" as their main frustration with AI tools [3]; a test the model wrote against its own almost-right code will pass, and that is the case the suite exists to catch.

What the suite cannot tell you

The suite cannot tell you whether the product is secure, whether the code can be maintained, or whether the margin works. Veracode's July 2025 test of over 100 models found 45 percent of generated code samples failed security tests regardless of whether the code ran correctly [4], so a green suite says nothing about the scan; security of AI-written code in diligence covers that. Diligence on an AI-written codebase covers the rest of the engineering read, and the AI startup due diligence guide puts the suite in its place among the other artefacts.

If you are checking a vendor's suite rather than a target's, how to evaluate a vendor's eval suite is the same method with the questions turned around, and we ask people to run it on us.