How to verify an AI claim with a test instead of a demo

You verify an AI claim by turning it into a test you can watch: a specific input the team did not prepare, processed live, with the intermediate steps visible, and an output you can compare against what the claim promised. The demo you were shown in the first meeting is the team's evidence for the team's version. The test is yours.

This page is the protocol. It assumes you have already built the claim table described in what changes when the target is an AI company, so you know which sentences you are testing. It needs a screen share, an hour, and three inputs you prepared yourself.

What does a demo show?

A demo shows the product working on an input the team chose, through a prompt the team wrote, on a model the team selected, at a moment the team picked. Every one of those choices is legitimate for a sales meeting and every one of them removes information a diligence reviewer needs.

Reveneau's diligence protocol replaces the demo with a held-out test for the same reason its own eval suites hold back cases the model never sees during development: a system that only ever meets inputs it was tuned on will look better than it is, and the size of the gap is the thing being bought. AI demo vs the last mile is the longer argument about that gap.

There is a second reason the demo is weak evidence, and it is the one the enforcement cases turn on. A demo cannot show you what is behind the interface. It cannot show you whether a person completed the task in the seconds the progress bar was spinning, whether a rule-based system handled the case and the model was decoration, or whether a third party's service did the work under the company's brand. The trace can, and the protocol below asks for it.

What are the stakes of getting this wrong?

The stakes are that a false AI claim is now treated as securities fraud or consumer deception, and an investor who funded it on a demo will be asked what was tested.

The record so far is short enough to know in full. On 18 March 2024 the SEC settled with two investment advisers, Delphia and Global Predictions, over false and misleading statements about their use of AI; the penalties were $225,000 and $175,000 [1]. On 25 September 2024 the FTC announced Operation AI Comply, five actions against companies whose AI claims it said were deceptive, including DoNotPay, which settled for $193,000 over marketing an "AI lawyer" [2]. On 14 January 2025 the SEC issued a cease-and-desist order against Presto Automation, a restaurant technology company, finding that the speech recognition in every deployed unit of its AI ordering product had for a period been owned and operated by a third party, and that when Presto deployed its own technology the vast majority of orders still required human intervention [3]. On 11 April 2025 the SEC charged the former chief executive of Nate, Inc., which had raised over $42 million on the claim that its app completed purchases with AI, alleging that it relied in large part on contract employees entering orders by hand [4].

Read those four together and the pattern is the point. A third party's technology behind the brand. Humans behind the progress bar. A completion rate that was true only for the demo path. Each is invisible in a demo and visible in a trace.

The protocol, step by step

The test takes six steps. Run it in a screen share, with the team driving and you supplying the inputs.

1. Write the claim as a testable sentence. "Our AI reads a lease and extracts every obligation" becomes "Given a lease we supply, the product returns a list of obligations that a lawyer would agree is complete." If the claim cannot be rewritten this way, it is positioning, and the result is "untestable".
2. Prepare three inputs the team has never seen. One typical case, one hard case, one case that should be refused or flagged. Source them yourself: a document from a portfolio company, a public filing, a record you construct. Do not send them in advance.
3. Run them live, in order. Typical case first, so you see the product at its best. Hard case second. The refusal case last, because a product that confidently processes an input it should have rejected has told you about its error handling.
4. Ask for the trace. For each run, ask to see the prompt that was sent, the retrieved context if any, the tool calls, the model's raw response, and the post-processing. Ask which model and which version served the request. Ask how long the request took at the model, against how long the interface took.
5. Score the output against the claim. For each input, the result is one of: confirmed (the output matches what the claim promised), contradicted (it does not), or partial (it does for the typical case and fails the hard one). Write the score against the claim table line.
6. Record what you could not test. A claim about scale, a claim about accuracy across a population, a claim about a customer's result: none of these can be tested with three inputs. They go to the eval suite review, and if the suite does not cover them, they stay "untested" in the report.

An hour covers three inputs with traces. If the team needs longer, the reason is itself informative.

What does the trace tell you?

The trace tells you what did the work. That is the single fact a demo hides and the single fact the enforcement cases turn on.

Four things to look for. Latency shape: a model request returns in seconds; a human completing the task returns in minutes, and the interface will have been designed to make the wait look normal. Model identity: the trace names the model and version; if it does not, ask why, and if the answer is "we route dynamically", ask to see the routing rule. Retrieval and tools: a product that claims the model reasons over the customer's data should show the retrieval step; if the context window contains the answer already, the model is formatting, which is a different product from the one described. Post-processing: rules applied after the model response are where a demo path gets its reliability; ask how many there are and what happens on inputs they do not cover.

Anthropic's developer guidance describes three grading methods for AI outputs, code-based checks, human review and model-based grading, and recommends automating grading where possible [5]. The relevance for the test is that a team which grades its own outputs can show you the grader; ask to run your three inputs through it. OpenAI's evaluation guide describes evals as tests of model outputs against "style and content criteria that you specify" [6]. A team with evals can tell you, on the spot, whether your hard case is inside or outside the distribution the suite covers.

How do you score the result?

Score the result against the claim, one line per claim, and take the table to the investment committee.

Claim (from the deck)	Test input	Trace showed	Result
"Extracts every obligation from a lease"	Portfolio company lease, 40 pages	Model call, 11 seconds, no human step	Partial: 14 of 17 obligations found
"Handles any contract type"	Supplier agreement in German	Refused with "unsupported document"	Contradicted: English only
"Flags unusual clauses"	Lease with a manufactured penalty clause	Rule-based post-processing, no model flag	Confirmed by rule, untested for model
"99 percent accuracy"	Three inputs	Cannot be tested with three inputs	Untested: refer to eval suite

The last row is the important one. Most accuracy claims cannot be tested live, and the honest result is to say so and refer them to the eval suite as a diligence artefact, where a suite with hundreds of graded cases can support a population claim in a way three inputs never will.

What if the team declines to run the test?

If the team declines, treat every claim as untested and price the deal that way. There are reasonable objections, and each has an answer. "The inputs might contain customer data": use inputs you sourced. "The trace exposes our prompts": you are under NDA, and a prompt is text a competitor can approximate in an afternoon. "It takes engineering time": an hour. "The system is not built for ad hoc inputs": then the claim about handling any input is contradicted before the test starts.

A team that runs the test and fails the hard case has shown you something useful: a product with a known boundary and a team that knows where it is. A team that will not run the test has shown you something too. AI feature or AI wrapper is the next check once the claim is confirmed, and the full question list for the management meeting is on questions for the management meeting of an AI startup. The AI startup due diligence guide puts the claim test first for a reason: nothing else in the review matters if the product does not do the thing.