Why a good AI demo does not show the product works

In a study published in July 2025, 16 experienced software developers completed 246 tasks, some with AI tools allowed and some without. Afterwards the developers estimated that AI had cut their completion time by 20%. The measurement showed that allowing AI had increased completion time by 19% [1]. The study is small and covers one type of work, so it supports no general claim about AI and speed. Its use here is narrow: people who used an AI tool on real tasks formed an impression of how well it worked, and the measurement showed the opposite.

A demo asks you to form that impression in one meeting. A demo is one run of an AI product on an input somebody chose, and it shows that the product can produce a good answer once. Three things limit what it proves: the input was selected, the output of a language model changes between runs, and real users send inputs nobody planned for. The stronger evidence is an eval, a repeatable test made of inputs, a written expectation for each, and a rule that scores each output. This page belongs to the guide on why AI evals matter, and the definition is in what an AI eval is.

A demo is one run on an input somebody chose

Every demo input is selected. The presenter picks questions the product handles well, in the wording it handles well, and rehearses them. The demo is therefore a sample of the product's best cases.

NIST, the United States standards institute, addresses this in its profile for generative AI, published in July 2024. One of its suggested actions reads: "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments" [2]. GAI is NIST's short form for generative AI. A demo is narrow because it has few inputs, non-systematic because nobody planned which situations to cover, and anecdotal because it reports one occasion. The profile is voluntary guidance.

The same document warns about a second popular piece of evidence, an AI model passing a test designed for people. It says that testing through "standardized tests designed for humans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or reliability in those domains" [2]. A slide saying that the model passed a professional exam belongs in the same category as the demo.

OpenAI's documentation names the habit in its list of mistakes to avoid: using "it seems like it's working" as an evaluation strategy [3].

The same input gives different output on different runs

A large language model (LLM) is the type of AI model that produces text, and it can give a different output each time it receives the same input. Research posted in August 2024 by Berk Atil and 12 other authors tested this under the conditions most favourable to repeatability. They took five LLMs, configured them with the settings meant to make output the same every time, and ran eight common tasks 10 times each. They report "accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%" [4]. They add that "none of the LLMs consistently delivers repeatable accuracy across all tasks" [4]. Those are the largest differences the authors observed, and the paper is a preprint, a research paper posted before formal review.

A demo samples that variation once. If the run you watched was one of the good ones, the demo gives you no way to know.

Simple arithmetic shows how much a short demo can hide. Take an invented product that gives a correct answer 90% of the time on any single try, with each try independent of the others. The chance that five demo questions in a row all come out correct is 0.9 x 0.9 x 0.9 x 0.9 x 0.9 = 0.59049, which is 59%. A product that fails one customer in ten therefore gives a faultless five-question demo in 59 of every 100 demos. The chance that 20 questions in a row all come out correct is 0.9 to the power of 20 = 0.1216, which is 12%. These figures are an illustration.

Anthropic's engineering team gives the same type of calculation for agents, which are AI programs that take several steps on their own: "If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)³ ≈ 42%" [5]. A trial is one attempt at one task. The method for measuring this is in agent reliability across repeated runs.

Real users send inputs nobody planned for

The third limit is the difference between a meeting room and real use. NIST states it in one sentence: "Measurement gaps can arise from mismatches between laboratory and real-world settings" [2]. Customers misspell words, write in other languages, paste in long documents, ask two things at once, and ask for things the product was never meant to do.

A test published in June 2024, called τ-bench (tau-bench), measured agents under conditions closer to real use. Its authors, Shunyu Yao and three co-authors, set up conversations between a simulated user and an AI agent that had been given tools and policy rules to follow. They report that agents built on leading models of that date "succeed on <50% of the tasks", and that in the retail setting fewer than 25% of tasks were completed correctly in all 8 of 8 attempts [6]. Those models are old now, the user in the test was simulated by a language model, and the figures give no information about current products. The method still applies: the authors ran the same task repeatedly and counted how often every attempt succeeded. A business with thousands of customers needs that number, and a demo cannot produce it.

The records of what wrong answers have cost companies are in public AI failures and the tests that target each one.

What the research on failed AI projects says

A buyer may ask what share of AI pilots reach production, meaning real use by customers. This page gives no percentage. The figures that are often repeated could not be traced to a measurement we could read, and the RAND Corporation's 2024 report on the subject repeats an estimate made by other people.

RAND's own research was a set of interviews. For the report published on 13 August 2024, James Ryseff, Brandon De Bruhl and Sydne Newberry interviewed 65 data scientists and engineers, 50 from industry and 15 from universities, each of whom had built AI models for at least five years [7]. Their findings are about causes:

• "Misunderstandings and miscommunications about the intent and purpose of the project are the most common reasons for AI project failure" [7].
• "Too often, trained AI models are deployed that have been optimized for the wrong metrics or do not fit into the overall business workflow and context" [7]. A metric is the number a team chooses to measure success.
• 14 of the 50 industry interviewees said senior leaders underestimated the time needed to train a model that solved the business problem [7].
• 30 of the 50 discussed persistent problems with data quality [7].

RAND states that most interviewees were engineers, so the findings may reflect an engineer's view more than a manager's. The first two findings describe questions that a demo cannot answer. A demo shows a capability. What the project is for, and which number counts as success, are the two things an eval makes a team write down.

What a demo shows and what an eval shows

A buyer's question	What a demo shows	What an eval shows
Can the product do this task at all?	Yes, on the inputs shown	Yes, and on what share of the cases tested
How often is it wrong?	Nothing	A pass rate on a stated number of cases
Does it give the same answer next time?	Nothing, because each input ran once	The share of repeated attempts that passed
What does it do with inputs we did not plan?	Nothing, because the presenter chose the inputs	Results on cases taken from real customer messages
Which cases fail, and how?	Nothing, because failures are left out	The failed cases, readable one by one
Will it still work after a change?	Nothing	The same cases run again and compared with the last run

A demo still has a use. It shows what the product is meant to do and whether the idea is worth testing. Treat it as the first step of the evaluation and make the buying decision after the later steps.

How a pilot differs from an eval

A pilot is a period in which a limited group of real users works with the product. It is better evidence than a demo, because the inputs are real. A pilot's inputs arrive once, so after a change to the product nobody can run last month's pilot again. And a pilot is usually judged by how the users felt, which the developer study at the top of this page shows can differ from the measurement.

Save the inputs from the pilot and write the expected behaviour for each. They become the first cases of an eval suite, the full set of test cases for a product. Anthropic's engineering post says "20-50 simple tasks drawn from real failures is a great start" [5], which is that company's advice from its own practice, with no study supporting the figure. The method is in how to build an LLM eval dataset from real usage.

What to ask for after a demo

Ask the vendor, or your own team, for three things in writing.

1. The eval set. The list of inputs and the expected behaviour for each. Check who wrote the expectations and whether the inputs came from real customers. If no set exists, the product has been judged by impression alone.
2. The pass rate, with the count it was measured on. A percentage, the number of cases it was measured on, and how many times each case was run. How to read an AI eval report explains why 92% of 50 cases is weaker evidence than 92% of 1,000.
3. The failures. The cases that failed, with the actual outputs. They show what some customers will receive.

Then ask to add ten of your own inputs, taken from real customer messages, and to see the results. Ask also whether the demo inputs are in the eval set, and what happens to the product when the AI model it uses is replaced, which is covered in what happens when the AI model changes. Two articles give more detail: how to evaluate a vendor's eval suite and the work between an AI demo and production.

Where Reveneau fits

At Reveneau we treat a demo as an introduction and an eval suite as the evidence. All of Reveneau's code is written by AI, and every change must pass a large eval suite before release. We write that suite from the specification, the written description of what the software must do, before the code exists. For an AI feature, the expected behaviour for each case is written down with the client before the feature is built, and what we show a client is the suite's result: the cases, the pass rate and the failures.

The checks in that suite that need judgement are graded by Jev, TypeSafe AI's decision model, which returns a probability for a written question instead of writing text. On Reveneau's own suite the run is ten times faster than with our previous language-model grader, so the suite can run on every change.

We use AI instead of hiring more engineers, so a build takes a small team, and that saving goes into the client's price. Reveneau, as a company, takes responsibility for the whole project through production and after release. To discuss a product you are planning, see AI development at Reveneau or contact us.