Choosing a development partner for regulated work

Choosing a partner for regulated work is harder than choosing one for ordinary product work, because the thing that matters most is invisible in a portfolio. A supplier can show you a product that looks excellent and tells you nothing about whether its retention rules are enforced or its audit trail is complete.

So the assessment has to be about process, and process claims are cheap. These are the questions where a real practice and a described one give different answers.

"What grades the code?"

This is the most useful question, and it is worth asking exactly this way rather than asking whether they test.

The answer you want describes independence: the verification comes from a fresh context that did not produce the change, evaluating against criteria written separately, ideally with a different model doing the grading than did the work. The reasoning is on never let the model grade its own work, and it matters in regulated work more than anywhere.

The answer that should concern you is any version of "the agent writes the code and the tests and the pipeline passes." That is internal consistency, not verification. The tests encode the implementation's interpretation of the requirement, including whatever it misunderstood, so they pass while proving nothing about the requirement.

A supplier who has thought about this will recognise the question immediately and have a specific answer. A supplier who has not will answer a different question, usually about their test coverage percentage.

Listen also for what happens when you push past the first answer. A supplier might correctly say the grading is independent and stop there, satisfied that the word "independent" answered the question. A stronger answer explains what independence actually means in their setup: a separate run with its own context window and no access to the reasoning that produced the change, criteria pulled from a document rather than from the diff being graded, and a result that is recorded even when it fails rather than silently retried until it passes. That last detail matters more than it sounds like it should, because a grading step that quietly retries until it gets a passing result is not independent verification, it is the same process checking its own homework with extra steps.

"Were the checks written before or after the code?"

Order determines meaning. A check written after the implementation encodes what the implementation does, which is circular and in a compliance context worse than nothing, because it produces a passing result a reviewer will read as evidence.

The good answer is that checks derive from the specification before implementation exists. The useful follow-up is to ask to see it, because this claim is verifiable: in a repository, the commits show which came first. A supplier confident in the practice will be comfortable showing you. One who is not will explain why that is difficult to demonstrate.

There is a specific pattern in commit history that confirms the practice rather than merely being consistent with it. Look for a commit that adds a failing check with no corresponding implementation change, followed later by a commit that makes it pass. That sequence can only happen if the check was written first, because a check written after the fact is authored alongside the code that already satisfies it and never exists on its own in a failing state. A supplier who understands this distinction will be able to point to an example without having to search for one, because they will recognise it as the specific evidence the question is actually asking for.

"What happens when the pipeline is failing and the release is due?"

This question reveals the real process rather than the described one, and the answer is revealing regardless of what it is.

The concerning answers involve making the pipeline pass: skipping the check, adjusting the threshold, marking the test as flaky and moving on. Nobody says this in a sales conversation, so ask it as a scenario and listen for whether the answer includes any mechanism that prevents it.

Good answers include structural safeguards: changes to compliance checks are reviewed separately from the change that motivated them, the count of skipped checks is tracked and cannot rise without anyone noticing, threshold changes need separate approval, and removing a check traced to a provision requires a recorded written reason. These are the mechanics on using evals as compliance evidence, and a supplier who has them will describe them without prompting.

"How do you know an obligation is covered?"

The answer should involve a traceable mapping: an obligation list, assertions derived from it, and each check naming the provision it satisfies. Then coverage is a query.

If the answer is a coverage percentage, that is a different measurement answering a different question. Line coverage tells you what fraction of code was executed by tests. It tells you nothing about whether a regulatory obligation is enforced, and the two are frequently confused in supplier conversations because one of them is easy to produce.

"What do you do about generated code being insecure?"

A supplier working with AI-written code in a regulated context should be able to speak directly to this, because the evidence is public. Veracode's spring 2026 testing across more than 150 models found only 55 percent of generations produced secure code while syntax correctness ran above 95 percent [1].

A partner who acknowledges this and describes what they do about it is being honest with you. A partner who claims their process eliminates the issue, or who has not encountered the finding, is either not paying attention or is managing the conversation. Neither is what you want on a system holding customer records.

What to be suspicious of

Compliance claimed as an outcome. "Our platform is HIPAA compliant" is close to meaningless as stated, because compliance is a property of an organisation's whole operation, not a feature of a product. What a product can be is built to support specific obligations, described specifically.

Certification substituted for verification. A certification tells you an organisation passed an assessment at a point in time against a defined scope. It is worth having and it is not the same as the system enforcing your obligations, and a supplier who answers every technical question by referring to a certificate is avoiding the question.

Reluctance to show the repository. Most of the claims above are visible in a codebase. A partner unwilling to walk you through a representative one, under an appropriate agreement, is asking for more trust than the situation warrants.

A proposal that treats every regulated engagement the same way. Financial recordkeeping, healthcare privacy, and consumer credit obligations share a structure, which is the argument of this whole hub, but they are not identical, and a proposal that describes the same generic testing approach regardless of which one applies has probably not read your specific obligations yet. A partner who has actually scoped your situation will ask which regulator applies before pricing the work, not after.

Questions worth asking beyond the technical ones

The technical questions above tell you whether a supplier can verify their own work. A shorter set of questions tells you what happens when something is found wrong after the fact, which is a different and equally important property.

Ask who is accountable when an obligation turns out to be missed after release, and listen for whether the answer names a mechanism or a feeling. A supplier who says they will "work with you to fix it" has described an intention. A supplier who describes a specific process, how the gap is found, how it is triaged against severity, how affected records are identified and remediated, has described a mechanism, and the second is the one that will actually function under pressure.

Ask what happens to the check suite itself over the life of the engagement, since a suite that is not maintained decays the same way any other part of a codebase does. Obligations get amended, features get added that the original obligation list never anticipated, and a suite frozen at the moment of delivery slowly stops matching the system it is supposed to verify. A partner with an ongoing relationship to the codebase, rather than a one-time delivery, is in a better position to keep the two aligned.

What we would expect you to ask us

For symmetry, since this page would mean little otherwise: ask us the questions above and expect specifics. Ask what grades our code, and the answer is a fresh context with the criteria written from the specification first. Ask about the order, and the commits show it. Ask what happens when the pipeline is failing before a release, and the answer is that the check suite changes go through separate review, because the alternative is a control that the people it constrains can disable without anyone noticing.

We would also expect you to ask what we do not do. We do not determine which obligations apply to your business, and we would want counsel involved in that before we write a single assertion, because a rigorously verified implementation of the wrong requirements is an expensive way to be non-compliant.

The broader question of selecting a partner is covered in choosing a software development partner. If you want to talk about a regulated build, get in touch.