Explainability and adverse action

Consequential decisions generate questions. Someone denied housing, offered worse terms, or ranked out of consideration may ask why, and depending on the context and the decision, an explanation may be an obligation rather than a courtesy.

The engineering point of this page is narrower and applies regardless of which legal framework attaches: the explanation has to be captured at the moment the decision is made, because it cannot be reliably reconstructed afterwards.

Why reconstruction does not work

Say a decision was made in March and questioned in September. To explain it, you would need the model as it was in March, the exact inputs it received, and the reference data it drew on at that time.

In a typical production system, none of those is kept. The model has been retrained, perhaps several times. The applicant's record has been updated. The third-party data has changed, because the vendor's data changes continuously. Reference data like comparable sales has changed.

Running the current model on the current data produces an answer, and that answer is not an explanation of the March decision. It is a new decision made today, presented as history. If the two disagree, which they often do, you have created a worse problem than the one you were trying to solve.

Here is a specific version of that disagreement. An applicant denied in March disputes the decision in September, and the team, having no captured explanation, reruns the current model against the applicant's current record. The current model returns an approval, because a collection account that was outstanding in March was resolved in June and no longer appears. The team now has to decide what to tell the applicant. Saying "you would be approved today" does not explain why the March decision was a denial, and saying nothing about the discrepancy invites the reasonable question of whether the March decision was ever correct in the first place. Neither answer is available if the March decision itself, with the inputs as they stood in March, was never captured. The dispute is resolvable only by looking at what actually happened in March, and the current model cannot tell you that no matter how accurate it is today.

What to capture, at decision time

An immutable record, written when the decision is made, containing:

The inputs. Every value the model received, as it was at that moment. Not a reference to a record that will change, the actual values.

The provenance of each input. Where it came from and when it was retrieved. This is what makes a dispute possible to resolve, because when someone challenges a data item you can identify which decisions used it, as described on tenant screening and fair housing.

The model version. A precise identifier, tied to an artefact you have retained. Retaining old model versions costs storage and is what makes this whole capability real.

The output. The score, decision, or ranking, and the threshold or rule applied to reach the final outcome.

The factor attribution. Which inputs caused this particular result, computed at decision time.

That last one is the one teams skip because it costs computation on every decision. Computing it later is precisely what does not work, so the cost is the price of being able to answer the question at all.

What a real explanation contains

There is a difference between describing a model and explaining a decision.

"Our model considers payment history, income stability, and prior tenancy" describes the model. It is true of every applicant and tells this applicant nothing about their own case.

"This decision was primarily driven by two reported collection accounts from 2023, retrieved from this source on this date" explains the decision. It names specific factors, tied to specific data, that this person can check and challenge.

The second kind is what people are actually asking for, and it is only possible if the factor attribution was captured at decision time.

Two further properties matter. The explanation should be in plain language, since an explanation nobody can understand does not function as one, and HUD's screening guidance calls for plain writing for eligibility criteria [1]. And it should be actionable where possible: an explanation that tells someone what would change the outcome is far more useful than one that only states what went wrong.

Explainability as a model selection constraint

Here is where this becomes an architecture decision rather than a feature.

Some model families explain themselves naturally. A rule set states its reasons. A linear model has interpretable coefficients. A small tree can be read. Others require post-hoc attribution methods that approximate an explanation and come with real caveats about faithfulness, and those caveats are uncomfortable to discuss with someone challenging a decision about their housing.

So explainability belongs in the model selection conversation, weighted according to how consequential the decision is. For decisions affecting whether someone gets housing, our engineering view is that a modest accuracy gain does not obviously justify moving from a model that can explain itself to one that cannot, and the more complex option should have to prove it is better.

This is the same argument as on tenant screening and fair housing, and it is worth having explicitly, early, with the compliance implications being considered rather than as an afterthought once a model is already performing well in evaluation.

A practical middle ground exists for teams that have already invested in a learned model and are not prepared to replace it outright: pair the model with a simpler, explainable model trained on the same data and compare their outputs on every decision. Where the two agree, the simple model's reasoning can stand in as the explanation, since it produced the same answer through steps a person can follow. Where they disagree, that disagreement is itself informative, and it is worth routing those specific cases to human review rather than trusting either model's number unexamined. This does not solve the underlying tension, since a disagreement still leaves the question of which model was right, but it turns an invisible risk into a visible and boundable one, concentrated in a known subset of decisions rather than spread silently across all of them.

Human override needs a record too

Where a person can override a model's output, the override is itself a decision and needs the same treatment: who, when, what the model said, what they decided instead, and why.

Two reasons. First, an override without a recorded reason removes the audit trail at exactly the point where a human judgment entered, which is the point most worth being able to examine. Second, override patterns are informative: if one reviewer overrides in a consistent direction, or overrides cluster in a particular segment, that is worth knowing about both for model improvement and for its own sake.

A specific pattern worth watching for in override data: a reviewer who overrides the model favourably for some applicants far more often than others, in a way that lines up with a group difference rather than with the strength of the individual case. This can happen with no ill intent at all. A reviewer may simply find some applicants easier to picture themselves approving, for reasons that have nothing to do with the criteria the model or the policy actually specify, and every individual override can look reasonable in isolation while the pattern across many overrides tells a different story. This is exactly the kind of finding that only shows up in aggregate, which is why override records need the same kind of periodic review as the model's own outcome distribution, described on testing a model for disparate impact, rather than being treated as a step that needs no oversight of its own.

Retention

All of this needs to be kept for as long as questions can arrive, which is generally longer than a default log retention window and longer than teams expect.

The specific retention period depends on the applicable framework and is a question for counsel. The engineering advice is to decide it deliberately, apply it to the decision records and the model artefacts together, and assert it, because a decision record whose model version has been deleted is not a complete record.

Storage cost is the usual objection to retaining old model artefacts, and it is worth naming plainly rather than treating as settled. A model artefact is typically a small fraction of the storage a product already spends on logs, images, or documents, and the cost of keeping it is bounded and known in advance. The cost of not having it, when a real dispute arrives and the original decision cannot be explained, is not bounded and not known in advance. Framed that way, the retention decision is not close, even before counsel weighs in on what the applicable legal minimum happens to be.

If you want help building decision capture into a product before you need it, get in touch.