What is the difference between a pilot and a proof of concept?

A proof of concept answers a narrow technical question, usually "can this be built at all," often on fake or sample data. A pilot goes further: it uses a real (or close to real) data set and a real workflow, and it is meant to answer whether the approach is worth building out, not just whether it is technically possible.

Why do so many pilots fail to lead anywhere?

Most pilots are not built to produce a real answer. They run on sample data instead of the messy real kind, nobody agreed in advance what result would count as a pass or a fail, and the pilot quietly turns into free discovery work for a bigger build that was already decided. When nothing was ever defined as failure, the pilot cannot tell you anything you did not already believe.

What should be defined before a pilot starts?

Two things: a bounded scope that touches the real problem, meaning real data and a real workflow rather than a clean demo version, and a written success criterion that both sides agree to before any work begins. If you cannot write down what result would make you walk away, you do not have a pilot yet.

How long should a pilot run?

Long enough to hit the real problem at least once, short enough that the cost of running it stays small next to the cost of the full build it is meant to inform. There is no fixed number that fits every project; the right length is set by how long it takes the hardest part of the real workflow to show up, not by a calendar convention like "four weeks."

Should a pilot be paid or free?

A pilot that produces real, usable work and a real answer is work, and treating it as work (paid, scoped, timed) is what keeps both sides honest about the goal. A pilot offered for free, with no defined end condition, tends to drift into an open-ended sales process where nobody is accountable for reaching a conclusion.

What happens if the pilot fails?

A pilot that fails on its own written criterion has done its job: it told you the approach was not worth the full build, and it did that for the cost of a pilot instead of the cost of the whole project. The pilot succeeded even though the approach did not. Treating that as a bad outcome instead of a correct one is what pushes teams toward pilots that can never fail.

How is a pilot scoped differently from the full engagement?

A pilot is smaller in surface area but not smaller in honesty. It should touch one real, hard part of the problem end to end rather than a shallow slice of everything, because a shallow slice can look fine without ever meeting the part that actually breaks. The full engagement then builds out from what the pilot proved, rather than repeating the pilot's easy path at a larger scale.

Who should set the success criterion, the vendor or the client?

Both, together, before the pilot starts. If only the vendor sets it, the bar tends to drift toward whatever the pilot happens to achieve. If only the client sets it without technical input, it can end up testing the wrong thing. A criterion agreed by both sides in writing is what makes the result mean something to either of them later.

Can a pilot use synthetic or sample data?

It can, but the moment it does, its ability to test the real problem drops. Real data carries the messiness (missing fields, edge cases, inconsistent formats) that is often the actual hard part of a build. A pilot that never touches that mess can look successful while testing almost nothing about whether the full build will work.

What is the biggest warning sign that a pilot is not a real test?

Nobody can answer the question "what result would make us stop here?" If the honest answer is "there isn't one, we're building this regardless," the pilot is the first phase of a decision that was already made, dressed up as a test to make the decision feel evidence-based.

How to structure a pilot before a full build: Reveneau

We have sat in enough pilot kickoffs to notice a pattern: the client wants to know if this will work, and the vendor wants to win the full build, and those two goals quietly pull the pilot in opposite directions. Nobody says this out loud. It shows up instead in a hundred small decisions: which data set to use, how long the pilot runs, what counts as done. Most of those decisions get made to protect the follow-on sale, not to produce an honest answer. Here is what makes a pilot a real signal instead of a dressed-up sales pitch: a scope that touches the actual problem, and a success criterion both sides agree to before any work starts.

A pilot that avoids the hard part cannot fail

The first thing to check in any pilot is whether it is even possible for it to go badly. If the answer is no, no matter how the work goes, then it is not a test of anything.

This happens most often through the data. A pilot run on a clean sample set, hand-picked rows with no missing fields, no weird formats, no edge cases, will almost always look like a success. The trouble is that the messy, inconsistent, incomplete version of the same data is usually where the real difficulty lives in a full build. We have seen this with document processing pilots specifically: a demo that reads twenty well-formatted PDFs cleanly tells you close to nothing about a system that will eventually face ten thousand scanned, rotated, and partially illegible ones. The pilot passed. The problem it was supposed to test was never in the room.

The fix is simple to state and uncomfortable to do: the pilot has to include the parts of the problem most likely to break it. If you are testing whether an AI system can handle customer support tickets, the pilot set needs the angry, ambiguous, multi-issue tickets in it alongside the clean single-question ones. If you are testing whether a new checkout flow reduces drop-off, the pilot needs to run against real traffic with real payment failures and real mobile users on slow connections, rather than a curated group of testers on new laptops. A pilot that only ever meets the easy version of the problem stays hollow no matter how small it is.

Without a written success criterion, every pilot succeeds

The second failure mode is more common and harder to see, because it looks reasonable at the time: nobody defines, before the pilot starts, what result would count as "this did not work."

Without that line, a pilot cannot fail, because there is no fixed point to fail against. Ask "did the pilot succeed?" at the end and the answer will almost always be some version of "yes, with some things to work out," because that phrase fits any outcome. A vague goal like "see how it goes" or "get a feel for the approach" produces a vague answer, and a vague answer is not a signal. It is a shared feeling that things are probably fine, which is exactly the feeling that gets both sides into a full build neither has actually tested.

The correction is a single written sentence, agreed before the first line of code: what specific, measurable result would make us walk away from this approach? Not a vibe. A number, a rate, a time, a pass or fail against a defined case. "The system correctly classifies at least this share of real historical tickets, unassisted" is a criterion. "The system seems promising" is not. Writing the number down before you have any results is what keeps it honest, because after the results exist, there is always a reason to move the number.

This is also where the two sides need to sit in the same room, not exchange documents. A client who sets the bar alone, without technical input, can end up asking the pilot to prove something the technology was never going to prove in that time frame, and a vendor who sets the bar alone can quietly aim it at whatever the pilot was always going to hit. The number has to survive a conversation between the people who understand the problem and the people who understand the build, and it has to be signed off by both before anyone starts writing code, not adjusted afterward to match whatever came out.

This is worth taking seriously because the base rate for pilots that go nowhere is not small. MIT's Project NANDA studied more than 300 enterprise AI deployments in 2025 and found that 95 percent of generative AI pilots produced no measurable return on business metrics like revenue or cost, with only 5 percent translating into real operational impact. Separate research from IDC, done with Lenovo, found that for every 33 AI proofs of concept a company launched, only 4 reached production, an 88 percent failure to graduate. Both studies point at the same root cause: not the model, but pilots that were never set up to produce a real answer in the first place, whether that was unclear objectives, data that was not ready, or a criterion nobody had written down.

A pilot with no failure condition is unpaid work

Put the two problems together and you get the actual definition worth working from. A pilot that cannot touch the real problem, and cannot fail on a written criterion, is a demo with a due date, built to justify a decision that was already made.

This matters most for the side doing the building, because that structure quietly shifts the cost. A real pilot is work: it should be scoped, timed, and paid for like the work it is, because that is what keeps both sides accountable to reaching a real conclusion rather than an open-ended goodwill exercise. When a pilot is offered free with no defined end point, it tends to drift. There is no moment where either side has to say "this did not meet the bar," because there was never a bar, only an ongoing conversation that eventually turns into a contract because stopping feels more awkward than continuing. In the work we do, we treat a pilot as a small, real engagement with its own scope and its own criterion, not as a free trial of the full build. That is also why we push clients toward defining the failure condition before we agree to run one: if you cannot write down what result would make you walk away, you do not have a pilot yet, you have a plan that is looking for permission.

A pilot that fails on its own terms is not a loss

The last piece is the one teams resist most: a pilot that fails against its written criterion has done exactly what it was for. You spent a bounded amount of time and money to find out that an approach does not hold up, instead of finding that out after committing to the full build. That is the pilot working, not the pilot failing.

Reframe it that way and the incentives line up correctly for the first time. The team that ran the pilot is not on the hook to make the result look better than it was, because a negative result was always one of the acceptable outcomes, written down before anyone had a stake in the answer. The team receiving the pilot is not stuck arguing about whether "some things to work out" secretly means yes or secretly means no, because the criterion already says which one it is. A pilot that can fail, and sometimes does, is the only kind that can also genuinely succeed.

A pilot only earns your trust in its result if it was built to lose. Scope it to the real problem, write down what failure looks like before you start, and let the answer be the answer.

Sources

88% of AI pilots fail to reach production, but that's not all on IT (CIO): reports MIT NANDA's finding that 95 percent of generative AI pilots produced no measurable business return, and IDC's finding that 88 percent of AI proofs of concept never reach production.