What is an eval suite in AI software development?

An eval suite is a set of automated checks, written from the project specification, that a system must pass before it is released. Instead of a person reading through the code and deciding it looks right, the suite runs the code against defined scenarios and compares the result to a defined correct answer.

Why can't the same AI that wrote the code also grade it?

A 2024 research paper found that GPT-4 rates its own answers higher than an independent human evaluator would, for work of the same actual quality, a pattern called self-preference bias. If the grader shares training data, habits, or gaps in knowledge with the writer, a wrong answer that looks right to the writer can also look right to the grader.

What should I ask a vendor who says they use an eval suite?

Ask who wrote the tests and when, relative to when the code was written. Ask what fraction of the spec has a test tied to it, and whether you can see one full example: the requirement, the test, and the result. Ask what happens when a test fails, and whether you can watch a real failure get caught before it is released.

Does a high test pass rate mean the software is correct?

Not on its own. A pass rate only measures how many written tests passed, not how much of the actual requirement those tests check. A suite that covers 20 percent of the spec can still show 100 percent passing, so ask what the tests cover before trusting the number.

Should the tests be written before or after the code?

Writing the test from the specification, before or independent of the implementation, is what keeps a test from simply copying whatever the code already does. A test written by reading the finished code tends to confirm the code's own assumptions instead of checking them against the original requirement.

What is self-preference bias in AI evaluation?

Self-preference bias is a tendency for an AI model to rate its own output more favorably than an independent evaluator would, even when the output quality is the same. Researchers measured this directly in GPT-4 and linked part of the effect to the model favoring text that reads as familiar to it, not necessarily text that is correct.

Can a vendor show me a real example of the eval suite catching a bug?

Yes, and you should ask for one. A vendor confident in their process can show you a specific case, step by step: the requirement, the test written against it, a version of the code that failed that test, and the fix that made it pass. If a vendor cannot produce a concrete example, treat the "we test everything" claim as unverified.

Is a large number of tests the same as good test coverage?

No. A vendor can write hundreds of simple tests that check only what is visible (does the button exist, does the page load) without ever testing whether the actual business logic in the specification is correct. Ask what the tests check, not how many there are.

Who should be accountable if the eval suite misses a bug?

The vendor, as a company, not a single named engineer and not the AI model. An automated check can only catch a rule someone thought to write down, so gaps will happen, and the question that matters is whether the vendor owns the fix and the consequences when one does.

How is Reveneau's eval suite different from just running unit tests?

The suite is written from the specification itself, not generated by looking at the code after the fact, and grading is kept separate from the model that wrote the implementation. Reveneau treats an eval suite as the mechanism that lets a small team work as fast as AI writes code and still keep the software correct.

How to Evaluate a Vendor's Eval Suite: Reveneau

The sentence that ends every sales call now

Every AI development vendor says some version of the same line: "we test everything with an eval suite." It sounds precise. It sounds like the vendor did the hard part already, so you do not have to ask.

We say a version of this line too. An eval suite, a set of automated checks written from the project's specification, is how we decide whether a piece of AI-written code is allowed to be released. We know why the sentence is persuasive, and we know what it leaves out. "We test everything" tells you a suite exists. It does not tell you who wrote it, what it actually checks, or who is allowed to say a test passed.

A vendor who cannot answer those questions in plain terms probably does not have an eval suite worth trusting. Here is what to ask, and why each question matters.

Question one: who wrote the tests, and when

The order matters more than the fact of testing at all. A test written from the specification, before the code exists or independent of the finished code, checks the code against what was asked for. A test written by reading the finished code checks the code against itself.

That second kind of test is common, and it is close to useless. Say a developer, human or AI, writes code that rounds a price down instead of up, then writes a test by looking at what the code does and confirming it. That test will pass forever. It never asked what the specification actually required. It only asked whether the code agrees with itself.

Ask the vendor directly: "Do you write the tests from the spec, or from the code?" If the answer is vague, or if nobody on their side can describe the order those two things happen in, assume the tests check nothing useful.

Question two: what fraction of the spec has a test tied to it

A vendor can show you a dashboard with 400 passing tests and a 100 percent pass rate, and that number can still mean almost nothing. Pass rate measures how many of the written tests passed. It says nothing about how much of the actual requirement got a test written against it in the first place.

Picture a project with 50 separate requirements in its specification. If only 10 of them have a real test, the suite can pass at 100 percent while 40 requirements are released completely unchecked. The number that matters is the coverage: how many of the things you asked for have a test that would fail if that thing broke.

Ask for one concrete example, shown step by step, instead of a dashboard screenshot: pick one requirement from your own spec, and ask the vendor to show you the exact test written against it, and what a failing version of the code looks like next to a passing one. A vendor who can do this in a few minutes, on a real example, has a suite that does what they say. A vendor who cannot produce a single example on request probably does not.

Question three: does the model that wrote the code also grade its own tests

Most buyers never think to ask this one, and it is the question that matters most.

If the same AI model writes the code and then writes, or runs, the test that judges that code, you have asked a system to check its own work with no outside reference point. Researchers at SB Intuitions tested this directly: they measured whether GPT-4, used as an evaluator, rates its own answers higher than an independent human rater would, for the exact same quality of output. It does. The paper, Self-Preference Bias in LLM-as-a-Judge (Wataoka and Takahashi, 2024), found GPT-4 shows a measurable tendency to prefer its own writing, and traced part of the cause to the model favoring text that reads as familiar to it, whether or not that text is actually correct.

This is a reason to keep the writer and the grader separate. A vendor whose process keeps those two roles apart, using a different model, a different pass, or a fixed rule set that does not change to match whatever the code already does, has a real check. A vendor who lets one pass write the code and the same pass decide it is correct has built a check that can only agree with itself.

Ask the vendor plainly: "Is the model that writes the code the same one that decides whether the test passed?" If they have not thought about this question before, that is itself an answer.

Question four: what happens when a test fails

A suite is only proof of anything if failing it actually stops something. Ask what happens the moment a test fails: does the change get blocked from release, or does someone have the option to override it and release it anyway? Ask how often that override gets used, and on what kind of failure.

A vendor who treats a failing test as a firm stop, which nobody can ignore under deadline pressure, is telling you the suite can actually stop a release. If overriding a failed test is routine, the suite is only as strong as the person deciding to skip it that day, and you are back to trusting a person's judgment call, which is exactly what an automated suite exists to reduce.

Question five: who is accountable for what the suite misses

No suite catches everything. A test suite can only check for a rule someone thought to write down in the first place, which means a genuinely new kind of mistake, one nobody anticipated when the tests were written, can get past a suite that is otherwise thorough. This is true of all testing, at every vendor.

What separates a vendor worth trusting from one that is not is what happens after something gets past the suite. Ask who is responsible when a bug reaches production despite the suite: a named individual, or the vendor as a company. A single engineer taking the blame does not fix the underlying gap, because the next new kind of mistake will not look like the last one either. A vendor that treats the miss as the company's problem, updates the suite so that specific gap cannot recur, and takes responsibility for the fix until it is in production has a real process behind its marketing line.

What a real answer sounds like

You will not get a perfect answer to every question above from every vendor, and you should not expect one. What you are listening for is specificity. A vendor with a real eval suite can point to a real requirement, a real test, and a real failure that test caught, in the time it takes to have a conversation. A vendor without one will answer in the same confident, general language every time: comprehensive, thorough, we test everything.

Whatever the sales pitch says, the suite is the thing you ask to see.

Why this is worth the extra ten minutes on a call

None of these five questions are hard to ask. None of them require you to read code or understand how a model works. They require the vendor to describe their own process in plain terms, during the call, without slides.

That is the actual test. A team that built a real eval suite uses that process every day, so describing it is easy for them. A team that added the phrase "eval suite" to their pitch after noticing buyers respond to it will delay, speak in general terms, or change the subject back to speed and cost. You are trying to tell the difference between a process and a phrase, and the difference shows up fast once you ask for one concrete example instead of a summary.

In the work we do, this is also why we keep the grading step separate from the writing step, and why, if something still got past the suite, you would reach someone who speaks for the company, and that stays true when one employee leaves. A specification is the actual contract for what gets built. The suite either checks the finished work against that contract, honestly, or it checks the work against itself and calls that proof.

A test that grades itself proves nothing. A test that checks the work against what was actually asked for, written before the code and graded apart from whoever wrote it, is the only kind worth paying for.

Sources

Self-Preference Bias in LLM-as-a-Judge: Wataoka and Takahashi (2024) measured that GPT-4, used as an evaluator, rates its own answers higher than an independent human rater would for the same quality of output, and linked part of the effect to the model favoring familiar-sounding text over correct text.

How to evaluate a vendor's eval suite