Building compliant software in insurance / Building it right
Evals as insurance compliance evidence
When an examiner asks how a company knows its security program still matches its risk assessment, or that every licensed state has a correctly configured notification rule, the usual answer is a document. A check suite derived from those same requirements and run on every change answers the same question with behavior instead of intent, which is a stronger and more durable kind of evidence.
Key takeaways
- Documents describe what a team intended to build. Checks describe what the running system actually does, continuously.
- The evidential strength comes from the run history, which cannot be produced retroactively the way a policy document can.
- Every compliance-relevant check should name the model law section or state configuration it verifies, so coverage is visible rather than assumed.
- A check that can be quietly weakened under deadline pressure is not a control, so changes to compliance checks need their own review discipline.
There is a moment in most insurance examinations where an examiner asks how the company knows a particular control has been operating correctly, and the honest answer is often a document: a policy, a control matrix, a completed questionnaire. Everyone in the room understands that the document describes what the team intended to build, and the harder question, whether the running system has actually matched that intention continuously, usually goes unanswered.
A check suite, run on every change and retained over time, answers it. That is the case this page makes.
Why behavior is stronger evidence than intent
A control exists in two states: what it was designed to do, and what the system currently does. A policy document captures the first, at the moment it was written, by the people who designed the control. Nothing about the document changes when the system it describes drifts away from it.
That drift is the ordinary condition of a growing application, not a sign of carelessness. A state gets added to the company's license list, a new integration partner starts receiving policyholder data, a reporting tool gets built to answer a one-off business question and quietly becomes a permanent fixture. None of these events update the security policy automatically, and none of them necessarily breaks anything the team would notice in ordinary use. The policy still describes the system correctly as it existed when the policy was written. It stops describing the system as it exists now, and nothing signals the gap.
A check derived from the same requirement, run on every change, closes that gap by construction. The day a new state is added to the license list without a matching notification configuration entry, the check that verifies configuration coverage either catches it and fails, or reveals that it was never built to catch that specific case, both of which are better outcomes than silence.
The history is the actual evidence
What makes this genuinely strong evidence is the record of the check having run, consistently, over time, rather than the result of any single run.
A check that passes today tells an examiner about today. A record of the same assertion passing across two years of commits, with the date it was first introduced, tells the examiner the control held throughout that period, and it has a property a document cannot match: it cannot be produced after the fact. A security policy can be rewritten the week before an exam. A two-year history of a specific check, tied to specific commits, cannot be manufactured retroactively.
This means retention of results matters as much as the checks themselves. Keep the run history for as long as the underlying obligation is relevant, which is typically much longer than a continuous integration system's default log retention. A team with excellent checks and a ninety-day results window has built most of the value and thrown away the part that would actually convince an examiner.
Trace every check to a requirement
A check suite only functions as compliance evidence if someone can map it back to the obligations it was built to satisfy. Each compliance-relevant check should carry a reference, in the code, to the specific model law section or state configuration entry it verifies.
This turns coverage into something visible rather than something inferred. Put the list of NAIC model law obligations, translated per translating NAIC model law into engineering requirements, beside the check suite, and any obligation with no check attached is immediately apparent. It also makes state amendments manageable: when a state updates its adopted version of the model law, the checks that reference that state's configuration are one search away, rather than an archaeology exercise across a check suite nobody has read closely in a year.
Guard the checks themselves
The failure mode that matters most here is ordinary, and it is worth naming plainly. When a release is blocked by a failing compliance check and a deadline is close, there are two ways to make the pipeline green: fix the underlying issue, or weaken the check. The second option is faster and, in a compliance suite, it is the one that actually removes the control while leaving the appearance of one in place.
This is a structural risk, not a story about bad intent. An engineer under deadline pressure, or an AI agent instructed to make a build pass, will both find the shortest path to a green pipeline, and weakening an assertion is often the shortest path available. The fix is structural too: require a separate, recorded review for any change to a compliance-relevant check, distinct from the change that motivated it. Fail the build if the count of skipped or disabled checks increases. Require a documented reason for removing or loosening any check traced to a specific obligation. Our piece on why you should never let the model grade its own work covers the general mechanism, and it applies directly to a compliance check suite, which is itself a control that can be silently disabled by the people it is meant to constrain.
What a check suite cannot do
It is worth being direct about the limits, because overstating this would itself be a kind of compliance risk.
A check suite cannot decide which states' adopted versions of the model law apply to a given business, cannot interpret an ambiguous provision, and cannot cover an obligation nobody translated into an assertion in the first place. Its coverage is exactly as good as the obligation list it was built from, no better. That means the sequence matters: counsel scopes which obligations apply and in which states, the team translates each into an assertion, and only then does the check suite enforce them. Skip the first step and the result is a rigorously, continuously verified implementation of the wrong requirements.
If you want to see what your current check coverage actually maps to, or want help building the suite from your obligations list, compliance gaps in insurance software covers the patterns worth checking first, and get in touch if you would rather we ran the assessment with you.
Best for
- Insurers preparing for a data security exam or market conduct review who currently have policy documents but no continuous verification
- Teams that have already translated the model law into assertions and need the discipline to keep the check suite trustworthy over time
Avoid if
- The obligation list itself has not been scoped by counsel yet, since a check suite is only as good as what it was pointed at
Verify before you commit
- Ask how long check results are retained, and whether it matches how long the underlying obligation applies
- Ask what review process exists for changes to compliance-relevant checks specifically
- Ask which model law section or state configuration entry each check traces back to
Common questions
Why is a check suite stronger compliance evidence than a policy document?
A policy document describes what a team intended to build at the time it was written, and it does not update itself when the running system drifts from that intention. A check suite describes what the system actually does on every change, so it closes the gap between intent and behavior by construction rather than by hoping nobody notices the drift.
What gives a check suite its evidential strength during an examination?
The run history, which cannot be produced after the fact. A policy can be rewritten the week before an exam, but a two-year record of a specific check passing across specific commits cannot be manufactured retroactively, which makes it far more convincing to an examiner than a document describing intent.
How long should compliance check results be kept?
For as long as the underlying obligation is relevant, which is usually much longer than a continuous integration system's default retention window. A team with strong checks and a ninety-day results log has built most of the value and discarded the part that actually demonstrates continuous compliance to an examiner.
How do you stop a compliance check from being quietly weakened under deadline pressure?
Through structural review rather than trust. Require a separate, recorded review for any change to a compliance-relevant check, distinct from the change that prompted it, fail the build if the number of skipped checks increases, and require a documented reason for loosening any check tied to a specific obligation.
Can a check suite determine which insurance regulations apply to a business?
No. A check suite can only verify obligations that have already been identified and translated into assertions. Deciding which states' adopted versions of the model law apply, and to which categories of the business, is a legal scoping question that has to happen first, with counsel, before any check is written.
What should every compliance-relevant check reference?
The specific model law section or state configuration entry it verifies, recorded in the code next to the assertion. This makes coverage visible as a comparison between the obligation list and the check suite, rather than something a reviewer has to infer from test names.
Is building a compliance check suite only useful for large insurers?
No. The evidential value comes from the run history rather than company size, and a smaller team often benefits most, since it typically has the least spare capacity to run a thorough manual review every year. A modest suite that runs on every change starts building credible history from its very first run.
What is the risk of treating a check suite as a complete compliance solution?
It creates false confidence in coverage that does not exist. A check suite's coverage is exactly as good as the list of obligations it was derived from, so if an obligation was never translated into an assertion, no amount of rigorous, continuously passing checks elsewhere will catch that gap.
Related reading
Adding eval tests was the best decision we made
We generate every line of code we ship. The single change that made that safe was writing the check before the code, and it turned out to change how we review, how we spec, and how fast we can move.
Never let the model grade its own work
When the same run writes the code and the tests, a green build proves the code agrees with itself. That is not verification, and it is the most common way an AI-built codebase becomes confidently wrong.
How many tests does AI-generated code need?
The honest answer is not a number, and coverage percentages are the wrong unit. Here is the unit we use instead, and why a suite of fifteen checks can be worth more than two thousand.