How to add LLM integration to an existing product without breaking it

An AI feature added to a product that already works costs three things: the build, a fee to the model provider for every request, and the checks that keep it correct after launch. Reveneau builds features this way: AI writes all of its code, every change has to pass an eval suite written from the specification, and because it uses AI instead of hiring more engineers, that saving goes into the client's price.
A team with a working product, real users and one workflow that fits a model (a support queue to summarize, documents to classify, a form to fill in) can add one good feature without a rebuild. The feature has to be added so that the rest of the product keeps working when the feature fails. That kind of AI work reaches production, and it needs its own method, separate from agent and RAG projects.
How much does it cost to build an AI product?
The first decision that sets the cost is how the feature gets its model. There are three ways, and each one is priced differently.
| Approach | What you pay for | What to know in November 2026 |
|---|---|---|
| Call a hosted model | A fee per request, by the tokens sent and returned | Prices are on each provider's public list |
| Tune a hosted model on your own examples | The tuning work, then the requests | OpenAI closed tuning to new organisations on 7 May 2026, and the Gemini API has no model that can be tuned |
| Run your own model | Servers with graphics chips, and the people who keep them running | No price list: you measure it |
For a first feature, the hosted model is the place to start, and its price is public. On Anthropic's price list, read 17 November 2026, Claude Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens. A request that sends 2,000 tokens and gets 500 back costs $0.009. Two discounts on the same page change that figure:
- The Batch API, for work that can wait for its answer, gives "a 50 percent discount on both input and output tokens". The same request costs $0.0045. Classifying the day's support tickets is a good fit. A reply the user is waiting for is a poor one.
- Prompt caching, for requests that all start with the same long instructions. On Claude Sonnet 5.5, a cache hit is billed at 0.05 times the base input price ($0.10 per million tokens, against $2), and writing to a five-minute cache costs 1.25 times the base input price.
| Example request (2,000 tokens in, 500 out) | One request | 100,000 requests a month |
|---|---|---|
| Claude Sonnet 5.5, standard | $0.009 | $900 |
| Claude Sonnet 5.5, Batch API | $0.0045 | $450 |
Tuning is closing at two providers. OpenAI's deprecations page, read 17 November 2026, says that since 7 May 2026 new tuning jobs are "not available to organizations that have not previously run fine-tuning", and that from 6 January 2027, "Active existing customers will no longer be able to create new fine-tuning jobs". Google's Gemini API documentation, read the same day, says "we no longer have a model available which supports fine-tuning in the Gemini API or AI Studio". Our post on what a startup should build on now that OpenAI fine-tuning is closing covers the choices that remain.
The other two parts have no price list. The build is people's time, and our post on what custom software development costs shows how to work it out from team size, weeks and weekly cost. The checks after launch cost model calls as well, because a grader that scores each case is one more request; the real price of an LLM judge goes through that bill. For the prototype stage, how to get a working AI prototype in weeks shows how to measure what one real request costs before you commit to the full build.
Start with the narrowest feature you can grade
The way a first AI feature is scoped decides whether it can be released. "Add AI to our product" is a wish. Nobody can build it, and nobody can say afterward whether it worked, because there was never a definition of working.
The fix is to pick something narrow enough that you could write down, in one sentence, what a correct answer looks like. "Summarize this support ticket into three bullet points that a human agent can scan in five seconds." "Classify this incoming lead as sales-ready, needs-nurturing, or spam." "Draft a first-pass reply to this email that a human will edit before sending." Each of those has a form you can grade. You can look at ten outputs and say which are good, and that ability separates a feature you can release from an idea you can only demo.
Narrow also means you can measure improvement. If the feature is "summarize tickets," you can time how long agents spend per ticket before and after, and you have a real number. If the feature is "make support smarter," there is no before and after to compare, because the after was never defined. A project like that can run for months and end with nothing to point to, even when the engineering is good.
Narrow is also the only way to build a useful evaluation set: the cases that tell you whether a change made the feature better or worse. A vague feature has no fixed target to score against. A narrow one has a target from the start.
Isolate the AI code so a bad model call cannot cause your product to fail
Once you know what you are building, the next decision is about the structure of the code, and it matters more than which model you pick. The AI code has to sit behind a separate interface the rest of your product does not need to know about.
In practice that means two things. First, the model call sits behind a single function or service with a clear interface: give it an input, get back an output or a defined failure. Nothing else in your codebase should call the model directly, the same way nothing else should query your database without going through your data layer. This is ordinary software design, and it matters more here, because the thing behind that interface is the least predictable part your product depends on. A database query fails in a small number of well-understood ways. A model call can fail, time out, return output in the wrong format, or return a confident answer that is wrong, and your code needs a plan for each.
Second, put the feature behind a flag. Release it switched off, turn it on for a small share of traffic, watch it, and widen the rollout once you trust it. A flag also gives you an instant off switch. If the feature misbehaves at 2 a.m., you turn it off in seconds, and the product keeps running as it did before you touched it. Any team can release a risky change this way. AI features are the case where skipping it costs the most, because the failure is a wrong answer shown to a user with total confidence, and no crash report records it.
The fallback needs its own paragraph, because teams often design it last. When the model call fails, what does the user see? The right answer is whatever they saw before this feature existed: the plain text box instead of the AI-drafted reply, the unsorted list instead of the AI-ranked one. If your fallback is a loading icon that never stops or an error page, the feature can cause the rest of the product to fail, and that is the risk a team with an existing product cannot take on. The reason to add one feature to something that already works is that the rest keeps working.
Hosted API or run it yourself: the real tradeoff
For a first feature, call a hosted model API. You set up no graphics chips, keep no model files updated, run no on-call rotation for a new service, and pay in proportion to use instead of a fixed bill.
The tradeoff is real. A hosted API charges per request, and at a high enough volume those charges add up to more than the fixed cost of running your own model. It also means sending your data to a third party, which matters for some industries and some data. And you depend on someone else's uptime and someone else's price changes.
Self-hosting reverses each of those. You control where the data goes, and at a high enough volume the fixed cost of the chips can be lower than the per-request bill. You also take on the work: keeping a model server running, handling updates, planning capacity for spikes, and building the monitoring a hosted provider gives you. Revisit it once you have three real numbers: your request volume, your hosted cost per request, and the fixed cost of the alternative. Decide with those numbers, and leave out the assumption that owning the hardware makes it cheaper.
Response time is the other tradeoff, and users feel it. A database read returns in milliseconds. A model call takes seconds, and longer for a longer output. The interface has to be designed around that: a visible loading state, the answer streamed as it is written so the user sees progress, or a result presented as a draft to review. A feature that blocks the screen until the model responds will feel broken even when it works as intended.
The separate interface from the last section helps here too. If the model call sits behind it, moving from a hosted API to a self-hosted model later, or from one provider to another, is a change inside that interface. The rest of your product stays as it is.
How to know if it is working after launch
Releasing the feature starts the next part of the work: finding out whether it is good, with an answer you can show to someone else.
Build a small evaluation set before launch: real, representative inputs, each paired with the answer you would accept. A few dozen well-chosen cases, covering the normal case and the unusual cases you can already think of, give you something to score new outputs against every time you change a prompt, swap a model, or change how much context you send. Without it, you can only guess whether a change helped, and a feature run on guesses stops improving.
After launch, watch two things, because they answer two questions. First, the result the feature exists to improve: time per ticket, how often users accept the draft, whatever the original narrow goal was. That tells you whether it delivers value. Second, the running signals: how often the fallback is used, how often your safety checks reject a model answer, and how response time and cost move once real traffic arrives. That tells you whether it works properly.
Real users will send the feature inputs you did not think of. That is why evaluation continues after launch. A team that keeps scoring live cases sees a feature getting worse within a week. A team that releases and moves on hears about it from a support ticket months later, after users have learned not to trust the feature.
Where Reveneau fits
Adding one dependable AI feature to a product that already works is part of our AI development work, and it is a smaller job than a full AI product build. The scoping, the isolation behind a clear interface and the evaluation set above apply to it in the same way. AI writes the code, and every change passes an eval suite written from the specification. For the longer argument on why a working demo and a production-ready feature differ, read an AI demo is not a product.
Generated code helps on this kind of job, and it needs care here too. The integration is common work and is finished fast. The parts specific to your product (where the feature sits, what it must not touch, what happens when the model is unavailable) are the parts a model fills in with a reasonable guess if your specification says nothing about them. Those are the sentences worth writing carefully, so we write them in the first days.
Sources
- Anthropic, "Pricing", Claude Platform documentation (read 17 November 2026): https://platform.claude.com/docs/en/about-claude/pricing
- OpenAI, "Deprecations", OpenAI API documentation (read 17 November 2026): https://developers.openai.com/api/docs/deprecations
- Google, model tuning page, Gemini API and AI Studio documentation (read 17 November 2026): https://aistudio.google.com/docs/model-tuning


