Diligence on an AI-written codebase: what to ask for and what a good answer looks like

Diligence on an AI-written codebase asks for six artefacts: the specification the code was generated from, the eval suite with its pass gate and last failing run, the provenance record of which tool produced which files, a security scan from the last quarter, the inventory of models and versions in production, and a live walkthrough by the person who directs the coding agents. Each has a good answer and a weak one, and this page sets out both.

It assumes the general questions in assessing code quality are still being asked. What changes is that several of the general signals stop working, and the artefacts above take their place.

Why do the usual signals stop working?

The usual signals stop working because a model produces them at no cost. A reviewer reading a human-written repository could infer things from the shape of the history: a single author meant a bus factor of one, inconsistent style meant a team without conventions, a high test count meant somebody cared. On a model-written codebase, one person driving an agent produces the commit volume of a team, the style is consistent by default, and the tests were written by the same model that wrote the code, against the code it had just written.

Reveneau asks for the specification before the repository on an AI-written codebase, because code generated from a written spec can be checked against it and code generated from a conversation cannot. That is the first thing a good answer contains: a document, older than the code, that says what the code should do.

Two studies explain why opinion is a poor substitute for evidence here. METR's randomised trial, published 10 July 2025, had 16 experienced open-source developers complete 246 real issues on repositories they knew well; with AI tools allowed they took 19 percent longer, while forecasting a 24 percent speedup beforehand and believing afterwards that they had been 20 percent faster [1]. METR's February 2026 update, with 57 developers and over 800 tasks, reported estimates whose confidence intervals cross zero and said selection effects limit what can be concluded [2]. The point for diligence is the gap between what developers felt and what was measured. A founder's account of how the code was built is a feeling. The eval suite is a measurement.

Stack Overflow's 2025 survey of 33,662 developers gives the shape of the risk: 84 percent use or plan to use AI tools, 46 percent distrust the accuracy of the output against 33 percent who trust it, and 66 percent name solutions that are "almost right" as their main frustration, with 45 percent saying that debugging AI-generated code takes more time [3]. "Almost right" is the state a generated codebase is in by default. The artefacts below are how you find out whether this one was moved past it.

What to ask for, and what a good answer looks like

Ask for the six artefacts in writing before the management meeting. The table gives the good and the weak answer for each.

Ask for	Good answer	Weak answer
The specification	A written document, versioned, older than the code it describes, with acceptance criteria	"It is in the prompts", or a deck
The eval suite	Hundreds of cases graded by code, run on every change, with a pass gate and a history of failures	A test folder the model wrote, all green, never failed
Provenance	A record of which tool generated which files, with the tool's terms and the code-referencing setting	"Mostly Copilot and Claude Code, we think"
Security scan	A static and dependency scan from the last quarter, with the findings triaged and the fixes merged	"We will run one"
Model inventory	Every model and version in production, with the date each was pinned and the migration plan for its retirement	The model name from the deck
The walkthrough	The person who directs the agents explains one recent change end to end: spec, generation, eval run, review, merge	A demo of the coding tool

A good answer to every one of these arrives within a day, because a team that has the artefact sends the artefact. A team that has to create it will take a week and the result will be new.

How do you read the repository in a day?

Read the repository in a day by starting from the evidence that cannot be generated and working towards the code, rather than opening the code first.

1. Read the specification against the product. Does the spec describe the product you saw in the claim test? Where it does not, the code was generated to a conversation the spec never captured, and the eval suite cannot cover what the spec does not say.
2. Run the eval suite yourself. On a clean checkout, on the current commit. Note the pass rate, the run time and whether the suite touches the model at all or only the code around it. The eval suite as a diligence artefact is the full method.
3. Read the last five failing runs. What failed, who noticed, what changed, how long it took. A suite with no failures in its history has never been used as a gate.
4. Run the scanners. A static analysis tool and a dependency checker for the language. Veracode's July 2025 test of over 100 models found 45 percent of generated samples failed security tests, so treat the scan as the floor of what you expect to find [4]. Security of AI-written code in diligence covers the rest.
5. Check the provenance record against the tools' settings. GitHub's documentation describes Copilot's code-referencing feature, which compares a suggestion and its surrounding code against an index of public repositories and, when it accepts a match, logs the file URL and the licence it found or "NOASSERTION" [5]. Ask whether it was on. Code provenance and licence exposure covers the licence side.
6. Pick three recent changes and trace them. From the issue or the spec change, through the generation, the eval run, the review, and the merge. Three is enough to see whether the process in the walkthrough is the process in the history.
7. Open the code last. With the eval results, the scan and the traces in hand, read the modules that failed or were flagged. That is where a day of reading pays.

What about the people behind the agents?

The team question changes shape. On a human-written codebase the question was how many people could maintain this. On a model-written one it is how many people can direct the agents, judge their output and write the specs the evals are derived from.

The question matters more than it did, because the agent removes the friction that used to force knowledge to spread. When three engineers had to write the code, three engineers understood it. When one engineer and an agent write it, the understanding can sit with one person, and the repository will look as though a team built it.

Ask the person doing the walkthrough to explain a decision the agent got wrong and how it was caught. Ask who writes the specification and who writes the evals, and whether they are the same person; when they are, the evals test what one person imagined and nothing else. Ask what happens when that person is on holiday. Assessing the engineering team still applies, with the role titles changed.

Anthropic's guidance for developers building test cases says to prioritise volume of cases over hand-graded quality, to automate grading, and to design cases that mirror the real task distribution including edge cases [6]. A team that has read that advice has a suite with hundreds of machine-graded cases. A team that has not has a demo and a feeling.

What goes in the report?

The report on an AI-written codebase states, for each of the six artefacts, whether it existed before the request, what it showed, and what it would take to create or fix. The consequence for the deal is usually the third column: a company with a spec and a suite needs nothing; a company with neither has a quarter of work ahead before the codebase can be handed to anyone, and that quarter is a cost in the model.

If the target is preparing for this review rather than undergoing it, preparing an AI-written codebase for review is the founder's version of this list, and how to review AI-generated code is the engineer's. How many tests does AI-generated code need answers the question the founder will ask next. The rest of the method is in the AI startup due diligence guide.