Running an AI pilot inside a bank: scope, tests, records and sign-off

The US Treasury wrote in December 2024 that it "recommends that financial firms prioritize their review of AI use cases for compliance with existing laws and regulations before deployment" and "that they periodically reevaluate compliance as needed" [12]. A pilot is the first time that review is tried on real work. An AI pilot is a limited trial of an AI system on real tasks, with set limits on its use, run before the bank decides whether to use it more widely.

This page sets out how to run one in a bank, as of 26 October 2026, using the documents supervisors have published. It is part of the guide on AI in banking. Most of these documents are guidance or voluntary frameworks. A bank should confirm with its own examiners and counsel what they expect for the use it plans.

Step 1: rate the use before anything else

Start by deciding how much harm a failure could cause. The Financial Stability Board's (FSB) sound practices of 15 October 2026 begin the AI lifecycle with "effective and systematic assessment of the materiality and risks associated with AI use cases at inception and thereafter" [7]. The revised US model risk guidance of 17 April 2026 says "Model purpose, together with model exposure, determines model materiality" [1]. Materiality means how much harm a failure could cause, to customers or to the bank.

Fed Vice Chair for Supervision Michelle W. Bowman said on 7 July 2026, giving her own views, that "actual use and materiality will help inform the type and intensity of the governance and controls that should be applied for the specific AI deployment" [8]. Canada's Guideline E-23, which takes effect on 1 May 2027, says "Institutions should deploy models only when they meaningfully contribute to decision-making, risk assessment or business purposes and deliver reliable outcomes consistent with their intended use" [9].

A tool that drafts internal meeting notes and a model that sets credit limits are both AI, and they need different pilots.

Step 2: decide which rules cover the pilot

The 2026 model risk guidance covers "traditional statistical and quantitative models and non-generative, non-agentic AI models" [1]. Generative AI and agentic AI are "not within the scope of this guidance" [1]. Generative AI writes text, images or code. Agentic AI takes actions on its own, such as calling other software. For tools outside the guidance, "a banking organization's risk management and governance practices should guide the determination of appropriate governance and controls" [1].

So a pilot of a fraud scoring model follows the model risk guidance, and a pilot of a chatbot follows the bank's own written rules. The use itself brings other rules: lending law for credit, consumer protection law for customer service, privacy rules for customer data. The pages on credit decisions, customer service and customer data cover those.

For anti-money laundering pilots, US agencies said in a joint statement of 3 December 2018 that "Pilot programs undertaken by banks, in conjunction with existing BSA/AML processes, are an important means of testing and validating the effectiveness of innovative approaches" [10]. BSA/AML means Bank Secrecy Act and anti-money laundering. The statement adds that such pilots "should not subject banks to supervisory criticism even if the pilot programs ultimately prove unsuccessful" [10]. The page AI for fraud and financial crime covers it.

Step 3: test before use

The 2026 guidance says "Validation generally occurs prior to a model's first use" [1]. Validation checks whether a model performs as expected, including its reliability and limits [1]. The guidance accepts that "certain circumstances (e.g., an urgent business need) may necessitate using the model before validation is completed" [1], and gives examples of controls for that case: "placing limits on model use or more closely monitoring its performance" [1]. A pilot with a small group of users and close review is one such limit.

On testing, the guidance says "Testing may include a range of activities, from out-of-sample and out-of-time testing" through checks on data [1]. An out-of-sample test uses data the model did not learn from. An out-of-time test uses data from a later period than the data the model learned from. It also says "The rigor of effective testing is commensurate with model complexity and materiality" [1]. And it warns: "Using a model beyond its intended purpose introduces additional uncertainty and risk" [1]. Write the pilot's purpose down, and keep the pilot to it.

Step 4: measure the results

Two measures from the 2026 guidance apply to any pilot. "Outcomes analysis compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use" [1]. And "Ongoing model monitoring involves an evaluation of the extent to which a model is performing as expected" [1], as conditions change.

The Bank of England and Financial Conduct Authority survey of 2024 shows what firms measure. The most common monitoring metrics were "accuracy, precision, recall and sensitivity (reported by 88% of firms using AI)" [6]. Accuracy is the share of all answers that are right. Precision is the share of the cases the model flagged that were truly positive. Recall, which the survey lists with sensitivity, is the share of the truly positive cases that the model found. 81% of firms using AI applied an explainability method, a way to show which inputs drove an output, and "Feature importance (72%) and Shapley additive explanations (64%) are the most popular explainability approaches" [6].

Pilot stage	What to record	Source
Rating	Purpose, exposure, materiality and the rules that apply	2026 guidance [1]; FSB [7]
Before use	Test results on data the model did not learn from, and on a later period	2026 guidance [1]
Limits	Users, tasks and volumes allowed during the pilot	2026 guidance [1]
Results	Outcomes compared with real results; accuracy, precision and recall	2026 guidance [1]; UK survey [6]
Explanations	The method used and checks that it matches the model	UK survey [6]
Sign-off	The accountable person's decision and the evidence it rests on	UK survey [6]
After the pilot	The monitoring plan for wider use	2026 guidance [1]

Generative AI: what to test

For a generative AI pilot, the National Institute of Standards and Technology (NIST) publishes two voluntary documents. The AI Risk Management Framework (AI RMF 1.0) was released on 26 January 2023 and is "intended for voluntary use" [3]. Its core has four functions, Govern, Map, Measure and Manage, and "GOVERN applies to all stages of organizations' AI risk management processes and procedures" [4]. NIST's page says the framework "is being revised as part of the White House AI Action Plan" [3].

The second document is NIST AI 600-1, the Generative AI Profile. NIST's page says: "On July 26, 2024, NIST released NIST-AI-600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" [3]. It numbers 12 risks, from confabulation to value chain and component integration [5]. It defines confabulation as "The production of confidently stated but erroneous or false content" [5]. Reveneau's view is that a pilot of a tool that writes answers should build test cases for this risk first.

In the Treasury's 2023 interviews, many financial institution representatives said they believe their existing practices align with aspects of the NIST framework, and "many also noted that it is challenging to establish practical and enterprise-wide policies and controls for emerging technologies like Generative AI" [11]. A pilot is the place to write those controls for one use, before the bank writes them for all uses.

Step 5: people, records and change

In the 2024 UK survey, "84% of firms reported having an accountable person for their AI framework" [6]. Name one for the pilot. The 2026 guidance says "internal audit's role is generally to evaluate whether the model risk management practices are rigorous and effective" [1], so internal audit checks whether the process works. It also says "Adequate documentation helps to support effective model risk management" [1].

Change needs its own record. The UK survey found that "Change management practices were cited by 87% of respondents with 71% being not AI-specific and 16% being AI-specific" [6]. For an AI system, a change of model version, instructions or data source is a change, and should mean running the tests again.

The Office of the Comptroller of the Currency (OCC) rescinded (withdrew) its examiners' booklet on model risk, the "'Model Risk Management' booklet of the Comptroller's Handbook" [2], on 17 April 2026, and the bulletin names no replacement. A bank's own records are the evidence it can show, and a bank should ask its examiners what they expect to see.

Set the pass criteria first

Reveneau's view is that a pilot should have its pass and fail criteria in writing before it starts. For a generative AI tool, those criteria are an eval: a written, repeatable set of test cases with the correct result for each, run on the system before the pilot, during it and after every change. Decide in advance what score lets the pilot widen, what score stops it, and who signs. A pilot run this way ends with a decision and the evidence behind it. The guide on LLM evals explains the method, how many test cases an LLM eval needs covers the size of the set, and using evals to decide a release covers the decision.

Where Reveneau fits

Reveneau is an AI software development consultancy with offices in San Francisco and Vancouver. Reveneau says AI writes all of its code and every change must pass an eval suite written from the specification, the written description of what the software must do. Reveneau says Jev, TypeSafe AI's decision model, grades that suite, and that the suite runs ten times faster than it did with its previous grader. It says it uses AI instead of hiring more engineers and puts that saving into clients' prices, and that it takes responsibility for the whole project through production and after release. To discuss a pilot, see Reveneau's services or contact Reveneau.