What is the difference between a pilot and a proof of concept?

A proof of concept answers a narrow technical question, usually "can this be built at all," often on fake or sample data. A pilot goes further: it uses a real (or close to real) data set and a real workflow, and it is meant to answer whether the approach is worth building in full, which is a bigger question than whether it is technically possible.

Why do so many pilots fail to lead anywhere?

Most pilots are not built to produce a real answer. They run on sample data instead of the messy real kind, nobody agreed in advance what result would count as a pass or a fail, and the pilot quietly turns into free discovery work for a bigger build that was already decided. When nothing was ever defined as failure, the pilot cannot tell you anything you did not already believe.

What should be defined before a pilot starts?

Two things: a limited scope that includes the real problem, meaning real data and a real workflow rather than a clean demo version, and a written success criterion that both sides agree to before any work begins. If you cannot write down what result would make you stop, you do not have a pilot yet.

How long should a pilot run?

Long enough to meet the real problem at least once, short enough that the cost of running it stays small next to the cost of the full build it is meant to inform. There is no fixed number that fits every project; the right length is set by how long it takes the hardest part of the real workflow to show up, not by a calendar convention like "four weeks."

Should a pilot be paid or free?

A pilot that produces real, usable work and a real answer is work, and treating it as work (paid, scoped, timed) is what keeps both sides honest about the goal. A pilot offered for free, with no defined end condition, tends to slowly turn into an open-ended sales process where nobody is accountable for reaching a conclusion.

What happens if the pilot fails?

A pilot that fails on its own written criterion has done its job: it told you the approach was not worth the full build, and it did that for the cost of a pilot instead of the cost of the whole project. The pilot succeeded even though the approach did not. Treating that as a bad outcome instead of a correct one is what pushes teams toward pilots that can never fail.

How is a pilot scoped differently from the full engagement?

A pilot covers less of the product, and it must be just as honest. It should cover one real, hard part of the problem from start to finish rather than a small, easy part of everything, because an easy part can look fine without ever reaching the part that actually breaks. The full engagement then builds on what the pilot proved, rather than repeating the pilot's easy cases at a larger scale.

Who should set the success criterion, the vendor or the client?

Both, together, before the pilot starts. If only the vendor sets it, the target tends to move toward whatever the pilot happens to achieve. If only the client sets it without technical input, it can end up testing the wrong thing. A criterion agreed by both sides in writing is what makes the result mean something to either of them later.

Can a pilot use synthetic or sample data?

It can, but the moment it does, its ability to test the real problem drops. Real data carries the messiness (missing fields, unusual cases, inconsistent formats) that is often the actual hard part of a build. A pilot that never uses that messy data can look successful while testing almost nothing about whether the full build will work.

What is the biggest warning sign that a pilot is not a real test?

Nobody can answer the question "what result would make us stop here?" If the honest answer is "there isn't one, we're building this regardless," the pilot is the first phase of a decision that was already made, presented as a test to make the decision feel evidence-based.

How to structure a pilot before a full build: Reveneau

We have sat in enough pilot kickoffs to notice a pattern: the client wants to know if this will work, and the vendor wants to win the full build, and those two goals quietly pull the pilot in opposite directions. Nobody says this out loud. It shows up instead in a hundred small decisions: which data set to use, how long the pilot runs, what counts as done. Most of those decisions get made to protect the later sale, not to produce an honest answer. Here is what makes a pilot a real answer instead of a sales pitch that looks like a test: a scope that includes the actual problem, and a success criterion both sides agree to before any work starts.

A pilot that avoids the hard part cannot fail

The first thing to check in any pilot is whether it is even possible for it to go badly. If the answer is no, no matter how the work goes, then it is not a test of anything.

This happens most often through the data. A pilot run on a clean sample set, hand-picked rows with no missing fields, no weird formats, no unusual cases, will almost always look like a success. The trouble is that the messy, inconsistent, incomplete version of the same data is usually where the real difficulty is in a full build. We have seen this with document processing pilots specifically: a demo that reads twenty well-formatted PDFs cleanly tells you close to nothing about a system that will eventually face ten thousand scanned, rotated, and partially illegible ones. The pilot passed. The problem it was supposed to test was never part of it.

The fix is simple to state and uncomfortable to do: the pilot has to include the parts of the problem most likely to break it. If you are testing whether an AI system can handle customer support tickets, the pilot set needs the angry, ambiguous, multi-issue tickets in it alongside the clean single-question ones. If you are testing whether a new checkout flow reduces the number of people who leave before paying, the pilot needs to run against real traffic with real payment failures and real mobile users on slow connections, rather than a curated group of testers on new laptops. A pilot that only ever meets the easy version of the problem tests nothing, however small it is.

Without a written success criterion, every pilot succeeds

The second failure mode is more common and harder to see, because it looks reasonable at the time: nobody defines, before the pilot starts, what result would count as "this did not work."

Without that written limit, a pilot cannot fail, because there is no fixed point to fail against. Ask "did the pilot succeed?" at the end and the answer will almost always be some version of "yes, with some things to work out," because that phrase fits any outcome. A vague goal like "see how it goes" or "get a feel for the approach" produces a vague answer, and a vague answer is only a shared feeling that things are probably fine, which is exactly the feeling that gets both sides into a full build neither has actually tested.

The correction is a single written sentence, agreed before the first line of code: what specific, measurable result would make us stop using this approach? A feeling is not enough. You need a number, a rate, a time, a pass or fail against a defined case. "The system correctly classifies at least this share of real historical tickets, unassisted" is a criterion. "The system seems promising" is not. Writing the number down before you have any results is what keeps it honest, because after the results exist, there is always a reason to move the number.

This is also where the two sides need to meet and talk, instead of only exchanging documents. A client who sets the target alone, without technical input, can end up asking the pilot to prove something the technology was never going to prove in that time frame, and a vendor who sets the target alone can quietly set it at whatever the pilot was always going to reach. The number has to be agreed in a conversation between the people who understand the problem and the people who understand the build, and it has to be signed off by both before anyone starts writing code, not adjusted afterward to match whatever came out.

This is worth taking seriously because many pilots lead to nothing. MIT's Project NANDA studied more than 300 enterprise AI deployments in 2025 and found that 95 percent of generative AI pilots produced no measurable return on business metrics like revenue or cost, with only 5 percent translating into real operational impact. Separate research from IDC, done with Lenovo, found that for every 33 AI proofs of concept a company launched, only 4 reached production, so 88 percent never moved into production. Both studies point to the same main cause, and it was the pilots more than the model: pilots that were never set up to produce a real answer in the first place, whether that was unclear objectives, data that was not ready, or a criterion nobody had written down.

A pilot with no failure condition is unpaid work

Put the two problems together and you get the actual definition worth working from. A pilot that cannot touch the real problem, and cannot fail on a written criterion, is a demo with a due date, built to justify a decision that was already made.

This matters most for the side doing the building, because that structure quietly shifts the cost. A real pilot is work: it should be scoped, timed, and paid for like the work it is, because that is what keeps both sides accountable to reaching a real conclusion rather than open-ended work done to build goodwill. When a pilot is offered free with no defined end point, it tends to continue without direction. There is no moment where either side has to say "this did not meet the target," because there was never a target, only an ongoing conversation that eventually turns into a contract because stopping feels more awkward than continuing. In the work we do, we treat a pilot as a small, real engagement with its own scope and its own criterion, not as a free trial of the full build. That is also why we ask clients to define the failure condition before we agree to run one: if you cannot write down what result would make you stop, you do not have a pilot yet, you have a plan waiting for approval.

A pilot that fails against its own criterion has still done its job

The last piece is the one teams resist most: a pilot that fails against its written criterion has done exactly what it was for. You spent a limited amount of time and money to find out that an approach does not work, instead of finding that out after committing to the full build. That means the pilot worked.

See it that way and, for the first time, both sides want the same thing. The team that ran the pilot does not have to make the result look better than it was, because a negative result was always one of the acceptable outcomes, written down before anyone cared about the answer. The team receiving the pilot is not stuck arguing about whether "some things to work out" secretly means yes or secretly means no, because the criterion already says which one it is. A pilot that can fail, and sometimes does, is the only kind that can also genuinely succeed.

You can only trust a pilot's result if the pilot was able to fail. Scope it to the real problem, write down what failure looks like before you start, and accept the answer.

Sources

88% of AI pilots fail to reach production, but that's not all on IT (CIO): reports MIT NANDA's finding that 95 percent of generative AI pilots produced no measurable business return, and IDC's finding that 88 percent of AI proofs of concept never reach production.

How to structure a pilot before a full engagement