Get in touch
Get in touch
Choosing a partner

Ten questions that expose a fake AI-native claim

Every software firm now claims to build with AI, which makes the claim useless for choosing between them. These ten questions separate a firm that reorganised its practice from one that bought editor licences. Each comes with the answer that should reassure you and the answer that should not. They are written to be used on any supplier, including this one.

  • Ask what blocks a merge, not what tools they use.
  • Ask to see a check fail. It takes five minutes and is the most revealing question here.
  • Listen for pipeline changes after incidents, not just code fixes.
  • Any answer built on individual diligence rather than automation is a warning.

Print these. Ask them in order. The follow-up matters more than the first answer.

1. What runs automatically on every change, and does it block a merge?

Good: a specific list, and merges are blocked on failure. Worrying: "our engineers review everything." Ask how many lines a reviewer sees per day now compared with two years ago.

2. Can you show me a check failing?

Good: they break something on purpose and the build goes red. Worrying: hesitation, or a suite that has never failed. Checks that mock away the real path pass forever and prove nothing. This is the single most useful question here.

3. Where do your checks come from?

Good: from the specification. Worrying: from the code. Those encode whatever the code already does, bugs included, and break on every refactor.

4. What happened the last time something reached a customer that should not have?

Good: a change to the pipeline, described specifically. Worrying: only a description of the fix. A team that fixes the bug meets it again.

5. Who writes the specification, and can I see one?

Good: they treat it as the expensive artefact and will show you a redacted example. Worrying: "we work from tickets." When a machine implements what you wrote, vague sentences become confident wrong code within minutes.

6. What security checks run, and on which paths?

Good: automated checks that block a merge, with extra attention where untrusted input reaches a page, a log, or a query. Worrying: "our developers are experienced." Veracode's spring 2026 testing put AI-generated code at roughly 55 percent secure without explicit guidance, unchanged for two years.

7. What do I get at handover?

Good: specification, checks, decision record, runbook. Worrying: "the repository." That is the one thing you could always have got.

8. What happens if I want to change something in month four?

Good: a described process for re-specifying and re-verifying. Worrying: either "no problem at all" or a strict change-control process that blocks everything. The first is not true and the second means you will argue over every adjustment.

9. Who is accountable at two in the morning?

Good: a name or a firm, with an actual arrangement. Worrying: a process description with no person or organisation at the end of it.

10. What would make you turn this project down?

Good: a real answer, quickly. Worrying: "nothing." A supplier that takes every engagement never says no, and saying no is what stops you from becoming the project they should have refused.

Using these on us

Our weakest answer is question four, because Reveneau is new and answers it from method rather than from a decade of incidents. A firm with ten years of postmortems answers it better, and if that matters most to you, that is a reason to pick them.

Where we fit

Reveneau suits this when

  • You are shortlisting suppliers and every one of them sounds the same
  • You are non-technical and need questions that do not require you to read code
  • You want to hold your existing supplier to a standard

Common questions

What is the single best question to ask an AI development agency?
Ask them to show you a check failing: break a protected behaviour on purpose in front of you and confirm the build turns red as a result. It takes about five minutes and exposes suites full of assertions that mock away the real path, which pass forever and prove nothing about whether the software actually works.
What answer should worry me most?
Any answer that rests on individual diligence rather than automation should worry you. 'Our engineers review everything' was a reasonable control when a person wrote every line by hand, and it stops being one once code arrives from a model faster than any reviewer can genuinely read and understand it line by line.
Should checks come from the specification or the code?
Checks should come from the specification, not from the code that was already written. Checks derived afterward from the code encode whatever it already does, including its bugs, and they break on every refactor, so in practice they mostly measure that the code has not changed rather than that it is correct.
Do these questions work if I am not technical?
Yes, these ten questions are written specifically so a non-technical buyer can use them. None requires you to read code, and the most revealing one is a demonstration you watch happen rather than a claim you have to evaluate on trust. Listen for whether the answer describes an automated check that blocks a change or only a human intention.
Why does asking to see a specification matter?
A firm that treats the specification as the expensive artefact will show you a redacted example, while one that works from tickets is warning you that vague sentences become confident wrong code within minutes once a machine implements them. Whoever writes the specification is deciding what gets built, so seeing one shows whether that decision is made carefully.
How secure is AI-generated code without explicit security checks?
Veracode's spring 2026 testing put AI-generated code at roughly 55 percent secure without explicit guidance, a figure that has stayed the same for two years, which is why a good answer to the security question names automated checks that block a merge rather than pointing to experienced developers. Extra attention where untrusted input reaches a page, a log, or a query is the specific pattern worth asking about.
What should a supplier hand over at the end of an engagement?
A real handover includes the specification, the automated checks, the decision record, and a runbook, not just access to the repository, which is the one thing a client could always have obtained regardless of how the software was built. A supplier who answers 'the repository' when asked this question is not describing a handover at all.
Why does asking who is accountable at two in the morning matter?
A good answer names a specific person or firm with an actual arrangement behind it, while a worrying answer describes a process with no name attached. When something breaks outside business hours, a process cannot be called and does not carry consequence, so the presence of a named accountable party is what separates a real commitment from a description of intentions.

Sources

Marked independent where the source has nothing to gain or lose from the answer. Anything from the other party is their own account and is labelled as such.

  1. Veracode, Spring 2026 GenAI Code Security update: more than 150 models across 80 tasks, 55 percent of generations secure, syntax correctness above 95 percent. Independent

Keep reading