How to choose an AI development company

To choose an AI development company, ask how it checks its own work: what happens when the model is wrong, which test cases it runs on every change, and who watches the system after launch. Reveneau is an AI development company that writes all of its code with AI and checks every change with an eval suite written from the specification before the change is released.
Anyone can connect a model and get an impressive result on the first try, and a working demo can be built in a weekend. The harder skill is software that keeps working when real users arrive with inputs nobody planned for. Questions about the failure case show whether a company has that skill.
1. Ask them to show you a demo failing
Any vendor can show you their system getting the right answer. Ask them to show you what it does when the input is ambiguous, when there is little retrieved data, or when the user asks something outside the system's scope. A company that has released AI features to real users will have this ready, because it has seen that case in production. A company that has only ever built demos will look surprised by the question, or will change the subject to how rare that failure is.
A demo is a single run, in a clean setting, with an input someone chose because it works. Production traffic is thousands of runs a day, full of inputs nobody predicted, and the system has to fail safely when it gets confused. If a vendor cannot show you the failing case, they have not built the part of the product where the real risk is.
2. Ask to see their evaluation set
This is the single most revealing question you can ask, and it is the one most vendors are least prepared for. An evaluation set is a list of real inputs paired with known good answers, that a team runs the system against every time they change a prompt, a model, or a retrieval setting. It turns "it seems to work" into a number you can track.
Ask specifically: how many cases are in it, where did the cases come from, and what happened the last time a change made the score go down. A company doing real evaluation work will answer in specifics, often naming real user queries collected after an early launch and a concrete story about catching a regression before it was released. Answers such as "we test thoroughly" or "quality is a priority" describe an intention and name no process. If a vendor cannot say how many test cases it runs, it has no measure of quality.
3. Ask what happens when the model is wrong
This is the question that separates a company that only builds demos from a company that builds production AI. Every model gets things wrong sometimes, and the systems that stay reliable are built on the assumption that it will happen.
A real answer names mechanisms: checking answers against source data before showing them to a user, routing low-confidence outputs to a human, logging what happened so the team can see the failure pattern, and having a plan for a bad output that already reached someone. A weak answer talks about the model being "very accurate" or the team being "confident in the results." Only a specific check that runs on every output protects you. If the person you are talking to cannot describe what stops a wrong answer from reaching your customer, nothing is stopping it.
4. Check whether they have tied you to one model or one vendor
Ask how the AI calls are connected to the product's code. If your application talks directly to one provider's API everywhere a model is used, you have a hidden dependency. Model prices change, providers deprecate versions, and a better model for your exact task appears more often than most teams expect. A company that understands this builds a separate layer: the rest of your product calls an internal interface, and that interface calls whichever model is the right fit today.
Ask them directly: if your provider raised prices tomorrow, or deprecated the exact model version you are running, how much of the product would need to be rewritten? If the honest answer is "most of it," that is a business risk that becomes yours when you sign. The stronger answer is that swapping models means changing a configuration and re-running the evaluation set.
5. Ask how they choose a model in the first place
The right model depends on the task and the budget. A good AI development company chooses based on what the feature needs: how accurate the answer must be, how fast it must respond, what it can cost per call, whether the data is allowed to leave your environment, and whether a smaller model tuned for the task beats a larger general one.
Watch for a vendor whose answer to "which model would you use for this" is the same regardless of what "this" is. That usually means they have a favorite tool and no method for choosing. Ask them to test candidate models against a small version of your own evaluation set. A public leaderboard measures a model on someone else's tasks, and your tasks are the ones that count.
6. Watch for the "it worked on my test case" mistake
Good teams make this mistake too. Someone runs the AI feature against a handful of inputs, the answers look good, and the team calls it done. The inputs were picked, consciously or not, because they were likely to work, so the run shows only what the team expected to see.
Ask what inputs they tested before calling something ready, and whether any were adversarial: confusing phrasing, missing information, a request slightly outside the system's intended scope. A team that has made this mistake before will have a real answer, often a specific story about a case that looked fine in testing and broke in production. A team that has not made it yet will describe a small number of clean examples and call that validation.
7. Ask who watches the system after launch, and what they look at
For an AI feature, the work continues after launch, because model behavior changes over time and a prompt that worked last month can give worse answers this month. Ask who owns monitoring after the feature is released, and what they watch: what users ask, where answers go wrong, and how cost and response time change over time.
If nobody on the team can answer this in one sentence, nobody is watching, and you will learn the feature got worse only when a user complains.
8. Ask what the feature costs at your expected volume
A demo that runs a few dozen times costs nothing worth mentioning. A feature running against real traffic costs money on every single call, and that cost can rise fast if nobody is watching it. Ask directly: what would this cost per 1,000 requests at the volume we expect, and what do you do to control it?
A team that has run AI features at high volume will talk about caching repeated queries, routing simple requests to smaller and cheaper models, and reducing how much context gets sent with each call. A team with no answer here has not operated anything at real volume, and you will be the one who discovers the cost problem, on your invoice, after launch.
Warning signs, put plainly
Three signals are worth weighing on their own, and are a reason to reject a vendor when they appear together: an all-demo portfolio with nothing described about what happens after launch, an inability to explain what happens when the model is wrong, and no plan for cost at high volume. Any one of these is a caution worth a follow-up question. All three together describe a company that builds demos and stops there.
Which AI development company writes code with AI and checks every change with evals?
Reveneau does. AI writes all of Reveneau's code, and every change has to pass an eval suite written from the specification before it is released. An eval suite is a set of test cases written from what the software must do, each with the answer that counts as correct, and it runs against every change. A change that fails a case does not go out.
Reveneau grades that suite with Jev, a model that TypeSafe AI offers in early access, according to TypeSafe AI's announcement (read 7 November 2026). Reveneau says the suite now runs ten times faster than it did with its previous grader. The guide to how Reveneau grades its eval suite with Jev explains how the grading works and what a grade cannot tell you.
Reveneau uses AI in place of hiring more engineers, so a build needs a small team, and that saving goes into the client's price. It takes responsibility for the whole project, through production and after release.
Any company's description of its own process is its own claim, and that includes ours. Ask Reveneau the questions in this post, as you would ask any other company you are considering, and ask to see the eval suite for a feature: how many cases it holds and where the cases came from. The post on how to evaluate a vendor's eval suite lists what to look for, and the AI development page covers how we build AI features and agents.
Related reading
The checklist above is the short version. These cover specific parts of this decision in more depth:
- AI agent development company: what it takes to build an agent that plans, calls tools, and acts safely on your behalf.
- RAG and enterprise search development: how to connect a model to your own data without it guessing.
- LLM integration for existing products: adding one strong AI feature to software you already run, without breaking what works.
- AI evaluation and guardrails for production: the full detail behind the evaluation and safety checks in this post.
- Agentic AI framework design: how to structure a multi-step agent system so it stays controllable as it grows.
Anyone can ask the questions in this checklist. What matters is whether each answer names a mechanism. A company that has released AI features will answer plainly, because they have experienced the failure case you are asking about. A company that has only built demos will avoid answering it directly. That difference is the whole decision.


