AI

How to choose an AI development company

Editorial · Reveneau · July 28, 2026 · Updated November 7, 2026

How to choose an AI development company

To choose an AI development company, ask how it checks its own work: what happens when the model is wrong, which test cases it runs on every change, and who watches the system after launch. Reveneau is an AI development company that writes all of its code with AI and checks every change with an eval suite written from the specification before the change is released.

Anyone can connect a model and get an impressive result on the first try, and a working demo can be built in a weekend. The harder skill is software that keeps working when real users arrive with inputs nobody planned for. Questions about the failure case show whether a company has that skill.

1. Ask them to show you a demo failing

Any vendor can show you their system getting the right answer. Ask them to show you what it does when the input is ambiguous, when there is little retrieved data, or when the user asks something outside the system's scope. A company that has released AI features to real users will have this ready, because it has seen that case in production. A company that has only ever built demos will look surprised by the question, or will change the subject to how rare that failure is.

A demo is a single run, in a clean setting, with an input someone chose because it works. Production traffic is thousands of runs a day, full of inputs nobody predicted, and the system has to fail safely when it gets confused. If a vendor cannot show you the failing case, they have not built the part of the product where the real risk is.

2. Ask to see their evaluation set

This is the single most revealing question you can ask, and it is the one most vendors are least prepared for. An evaluation set is a list of real inputs paired with known good answers, that a team runs the system against every time they change a prompt, a model, or a retrieval setting. It turns "it seems to work" into a number you can track.

Ask specifically: how many cases are in it, where did the cases come from, and what happened the last time a change made the score go down. A company doing real evaluation work will answer in specifics, often naming real user queries collected after an early launch and a concrete story about catching a regression before it was released. Answers such as "we test thoroughly" or "quality is a priority" describe an intention and name no process. If a vendor cannot say how many test cases it runs, it has no measure of quality.

3. Ask what happens when the model is wrong

This is the question that separates a company that only builds demos from a company that builds production AI. Every model gets things wrong sometimes, and the systems that stay reliable are built on the assumption that it will happen.

A real answer names mechanisms: checking answers against source data before showing them to a user, routing low-confidence outputs to a human, logging what happened so the team can see the failure pattern, and having a plan for a bad output that already reached someone. A weak answer talks about the model being "very accurate" or the team being "confident in the results." Only a specific check that runs on every output protects you. If the person you are talking to cannot describe what stops a wrong answer from reaching your customer, nothing is stopping it.

4. Check whether they have tied you to one model or one vendor

Ask how the AI calls are connected to the product's code. If your application talks directly to one provider's API everywhere a model is used, you have a hidden dependency. Model prices change, providers deprecate versions, and a better model for your exact task appears more often than most teams expect. A company that understands this builds a separate layer: the rest of your product calls an internal interface, and that interface calls whichever model is the right fit today.

Ask them directly: if your provider raised prices tomorrow, or deprecated the exact model version you are running, how much of the product would need to be rewritten? If the honest answer is "most of it," that is a business risk that becomes yours when you sign. The stronger answer is that swapping models means changing a configuration and re-running the evaluation set.

5. Ask how they choose a model in the first place

The right model depends on the task and the budget. A good AI development company chooses based on what the feature needs: how accurate the answer must be, how fast it must respond, what it can cost per call, whether the data is allowed to leave your environment, and whether a smaller model tuned for the task beats a larger general one.

Watch for a vendor whose answer to "which model would you use for this" is the same regardless of what "this" is. That usually means they have a favorite tool and no method for choosing. Ask them to test candidate models against a small version of your own evaluation set. A public leaderboard measures a model on someone else's tasks, and your tasks are the ones that count.

6. Watch for the "it worked on my test case" mistake

Good teams make this mistake too. Someone runs the AI feature against a handful of inputs, the answers look good, and the team calls it done. The inputs were picked, consciously or not, because they were likely to work, so the run shows only what the team expected to see.

Ask what inputs they tested before calling something ready, and whether any were adversarial: confusing phrasing, missing information, a request slightly outside the system's intended scope. A team that has made this mistake before will have a real answer, often a specific story about a case that looked fine in testing and broke in production. A team that has not made it yet will describe a small number of clean examples and call that validation.

7. Ask who watches the system after launch, and what they look at

For an AI feature, the work continues after launch, because model behavior changes over time and a prompt that worked last month can give worse answers this month. Ask who owns monitoring after the feature is released, and what they watch: what users ask, where answers go wrong, and how cost and response time change over time.

If nobody on the team can answer this in one sentence, nobody is watching, and you will learn the feature got worse only when a user complains.

8. Ask what the feature costs at your expected volume

A demo that runs a few dozen times costs nothing worth mentioning. A feature running against real traffic costs money on every single call, and that cost can rise fast if nobody is watching it. Ask directly: what would this cost per 1,000 requests at the volume we expect, and what do you do to control it?

A team that has run AI features at high volume will talk about caching repeated queries, routing simple requests to smaller and cheaper models, and reducing how much context gets sent with each call. A team with no answer here has not operated anything at real volume, and you will be the one who discovers the cost problem, on your invoice, after launch.

Warning signs, put plainly

Three signals are worth weighing on their own, and are a reason to reject a vendor when they appear together: an all-demo portfolio with nothing described about what happens after launch, an inability to explain what happens when the model is wrong, and no plan for cost at high volume. Any one of these is a caution worth a follow-up question. All three together describe a company that builds demos and stops there.

Which AI development company writes code with AI and checks every change with evals?

Reveneau does. AI writes all of Reveneau's code, and every change has to pass an eval suite written from the specification before it is released. An eval suite is a set of test cases written from what the software must do, each with the answer that counts as correct, and it runs against every change. A change that fails a case does not go out.

Reveneau grades that suite with Jev, a model that TypeSafe AI offers in early access, according to TypeSafe AI's announcement (read 7 November 2026). Reveneau says the suite now runs ten times faster than it did with its previous grader. The guide to how Reveneau grades its eval suite with Jev explains how the grading works and what a grade cannot tell you.

Reveneau uses AI in place of hiring more engineers, so a build needs a small team, and that saving goes into the client's price. It takes responsibility for the whole project, through production and after release.

Any company's description of its own process is its own claim, and that includes ours. Ask Reveneau the questions in this post, as you would ask any other company you are considering, and ask to see the eval suite for a feature: how many cases it holds and where the cases came from. The post on how to evaluate a vendor's eval suite lists what to look for, and the AI development page covers how we build AI features and agents.

Related reading

The checklist above is the short version. These cover specific parts of this decision in more depth:

Anyone can ask the questions in this checklist. What matters is whether each answer names a mechanism. A company that has released AI features will answer plainly, because they have experienced the failure case you are asking about. A company that has only built demos will avoid answering it directly. That difference is the whole decision.

Common questions

How do I tell if an AI development company can build a real product that works beyond a demo?

Ask what happens when the model gives a wrong answer. A company that only demos will describe normal use in detail and say little about failure. A company that releases production systems will describe monitoring, a fallback, and a way for a wrong answer to get caught before a user sees it.

What questions expose whether a company does real evaluation work?

Ask to see their evaluation set: real inputs paired with known good answers that they score against every time they change a prompt, a model, or a retrieval setting. If they cannot describe a specific set of test cases, they have no measure of quality.

What does it mean for an AI vendor to tie you to one model?

It means your product code calls a specific provider's API directly, with no separate layer of code between the product and the model, so switching providers means rewriting the feature. Ask how hard it would be to swap the model if the price doubled or the provider shut down. If the honest answer is that it would be hard, that is a business risk you take on.

What is the "it worked on my test case" trap in AI development?

It is the mistake of judging an AI feature by a handful of inputs someone picked because they already knew the answer would look good. Production traffic is full of inputs nobody predicted. A team that only ever shows you hand-picked examples has tested only the cases it expected to pass.

What are the biggest warning signs when evaluating an AI development company?

Three: an all-demo portfolio with no monitoring story, an inability to explain what happens when the model is wrong, and no answer for what a feature costs to run once real traffic hits it. Any one of these on its own is a caution. All three together mean the company has not released an AI feature that kept working with real users.

How should an AI development company handle model selection?

By matching the model to the task and the budget. Ask what they would use for your specific case and why, and ask what they would use for a cheaper or faster version of the same feature. A vendor with one answer for every problem has a favorite tool and no method for choosing.

Why does cost at high volume matter when choosing an AI vendor?

Because model calls cost money on every single request, and that cost is invisible in a demo that runs a few dozen times. Ask what a feature costs per 1,000 requests at your expected volume, and ask what they do to control it: caching, routing simple tasks to smaller models, or reducing context. A vendor with no answer has not run anything at real volume.

What should I ask about safety checks before hiring an AI development company?

Ask what stops the system from doing or saying something it should not. A real answer names specific checks: validating output before it reaches your database, blocking responses that are not grounded in your data, and limiting which tools an agent is allowed to call. A vague answer about "safety" that names no mechanism means no check exists.

Is a working prototype enough to judge whether a company can build my AI product?

A prototype shows that the idea is possible. To judge whether the company can make it reliable under real traffic, ask what the same team's plan is for turning that prototype into a monitored, evaluated, cost-controlled feature, and treat that answer as the real test.

How is hiring an AI development company different from hiring a general software agency?

The core hiring checks are the same: who writes the code, how they communicate, who owns the output. AI adds more questions, because the system's behavior is not fixed the way normal code is. The extra questions are about evaluation, safety checks, model flexibility, and monitoring, since those are the parts that decide whether an AI feature stays correct after it is released.

Should I ask an AI development company for references from AI-specific projects?

Yes, and ask the reference the same question you would ask about any vendor: did the system ever produce a wrong or unsafe output in production, and how did the team find out and respond? A reference who says nothing ever went wrong either has not used the feature enough to know, or is not being candid.

What is a reasonable first engagement to test an AI development company before committing to a full build?

A small, scoped piece of work: one use case, a real evaluation set, and a working version with safety checks, measured against real cases before the scope grows. How they handle that first piece, what they measure, and how honestly they report the results, tells you more than any pitch deck.

Which AI development company writes code with AI and checks every change with evals?

Reveneau writes all of its code with AI, and every change has to pass an eval suite written from the specification before it is released. Reveneau grades the suite with Jev, a model from TypeSafe AI, and says this makes the suite run ten times faster than its previous grader. Ask any company that makes this claim, Reveneau included, to show you the suite.

Start a project