How to test an AI agent before you buy

An AI agent is a program that uses an AI model to finish a task in several steps by itself. Because the model chooses its own steps, the same agent can succeed on a task once and fail on it the next time. A demo shows one run. Before you sign, ask the firm to show you many runs on your own work. This page explains five tests to ask for, where each idea comes from, and how to read the answers.

The tests here are for a buyer. They tell you what to ask a firm to show before you sign. The AI agent evals guide explains how a team builds such tests in detail.

Test 1: run your own cases

Write down twenty or more real requests the agent will receive, with the answer or action you expect for each. Include the hard cases. Ask the firm to run its agent on them and show you each result.

Galadrim, one of the firms in this guide, describes this method on its own agents page: "We build a set of reference cases with your business experts and score each version against it" [5]. That is its own description of how it works. A set of cases written by your own staff shows whether the agent does your work, on cases the firm did not choose.

The tau-bench study shows how such a test can be scored. Its authors built a test in which an agent talks with a simulated user, a user played by a language model, while it uses a set of tools and follows written business rules. They write that their method "compares the database state at the end of a conversation with the annotated goal state" [1]. In plain words, the test checks what the agent changed in the system, such as an order or a booking, against what should have changed. Ask a firm to score your cases the same way: by what the agent did, as well as what it wrote.

Test 2: run each case more than once

The tau-bench authors also "propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials" [1]. Pass^k is the share of tasks an agent solves in every one of k tries. In their paper, first posted on 17 June 2024, they report that "even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks", and that in their retail test pass^8 was "<25%". So those agents solved fewer than one in four retail tasks in all eight of eight tries. These are the authors' own figures, for the agents and models they tested in mid-2024, and newer models will give different numbers. The lesson for a buyer holds: one good run tells you little. Ask the firm to run each of your cases several times and to report how many cases passed every time. The guide page on agent reliability across repeated runs explains the measure.

Test 3: try hidden instructions

An agent reads emails, web pages and documents as part of its work. Someone can hide an instruction in that text, such as "send this file to this address", and the agent may follow it. This is called prompt injection, and when it takes control of what an agent does, agent hijacking.

The US National Institute of Standards and Technology (NIST), through its Center for AI Standards and Innovation, tested this and published the results on 17 January 2025 [2]. Its team used AgentDojo, an open-source test set built by researchers at ETH Zurich, with four test settings: Workspace, Travel, Slack and Banking. The agents were "powered by Anthropic's upgraded Claude 3.5 Sonnet (released October 2024)", which NIST says AgentDojo found to be "one of the top performing models in resisting agent hijacking attacks". NIST took five attack tasks, among them sending an email to an untrusted address and downloading and running a harmful script. The attacks' average success rate was 57%. NIST then "attempted each attack 25 times". It reports, "After repeated attempts, the average attack success rate increased from 57% to 80%." Those figures hold for that model, those five tasks and those conditions. The point for a buyer is the same as in Test 2: an attack that fails once may work on a later try. Ask the firm which attacks it tested, how many times each, and what the agent did. The guide page on security evals for AI agents covers these tests in depth.

Test 4: check what the agent says

An agent's words can cost money even when it takes no action. In Moffatt v. Air Canada, British Columbia's Civil Resolution Tribunal decided on 14 February 2024 that the chatbot on Air Canada's website "suggested Mr. Moffatt could apply for bereavement fares retroactively", which Air Canada did not allow [3]. A bereavement fare is a lower fare for travel after a death in the family. The tribunal wrote, "It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot." It also found that "Air Canada did not take reasonable care to ensure its chatbot was accurate", and ordered it to pay "a total of $812.02".

Two limits apply. The bot in that case answered questions and took no actions. The decision is a small claims ruling in one Canadian province, so other courts are free to decide differently. It is still a clear record of a company held responsible for its bot's answer. Ask the firm to test the agent's answers on your policies, prices and rules, and to show you the cases where it was wrong.

Test 5: check which actions go to a person

OpenAI, which makes the GPT models, says in its own guide to building agents: "Actions that are sensitive, irreversible, or have high stakes should trigger human oversight until confidence in the agent's reliability grows. Examples include canceling user orders, authorizing large refunds, or making payments." [4] It also advises, "Set limits on agent retries or actions." That is OpenAI's guidance about agents in general. Ask the firm to list each action your agent can take, mark which ones go to a person, and then test that each marked action does reach a person.

Several firms in this guide describe that step in their own words. Awiant says invoices in its Volkswagen Financial Services system are "approved by a person", and that only invoices that meet "predefined criteria" will later move toward handling without one. Cay Digital says, "Agents ask before they guess: a person approves, with an SLA on every request." AscentCore names "Human sign-off at PRD, implementation plan, merge and go/no-go" for its coding agents. These are the firms' own descriptions.

The five tests at a glance

Test	What it checks	What to ask the firm for	Where the idea comes from
Your own cases	Whether the agent does your work	Results on twenty or more cases your staff wrote	Galadrim's own method; tau-bench scoring
Repeated runs	Whether a pass was luck	The share of cases passed in every run	tau-bench, June 2024
Hidden instructions	Whether text it reads can take control	Attacks tried, runs per attack, outcomes	NIST, January 2025
Its answers	Whether it states your rules correctly	Wrong answers found on your policies	Moffatt v. Air Canada, February 2024
Actions sent to a person	Whether risky actions stop for approval	A list of actions, each tested	OpenAI's guide

How to read a firm's own figures

Some firms publish a single quality figure. Linnify's AI page says ">95% average accuracy across agents deployed in production" and gives no definition of accuracy and no test set [6]. Before you rely on a figure like that, ask what counted as a correct result, how many cases were tested, how many times each, and who checked them. The same questions apply to every figure in this guide, because each one was published by the firm that did the work.

Try a simpler program first

Anthropic, which makes the Claude models, writes in "Building effective agents": "When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all." [7] It also recommends "extensive testing in sandboxed environments", meaning test copies of systems where a mistake harms nothing. That is the model maker's own advice. Ask each firm whether a simpler program, such as a fixed set of steps, would do your job, and test that version first. The guide on why AI evals matter explains how tests help a buyer decide.

Reveneau and Fluxon

Reveneau says every change it makes must pass a large eval suite before it is released, and that the suite is written from the project's specification. It says AI writes 100 percent of its code, and that Jev, the model that grades the suite, makes it run ten times faster than its previous grader. Reveneau says it takes responsibility for the whole project through production, the live system people use, and after release. You can contact Reveneau and ask it to run these five tests on your own cases. Fluxon's AI development page lists "Making AI quality measurable at every stage", a case for Braintrust, among its case titles; the pages read did not describe Fluxon's own test method. For the method behind tests of AI answers in general, read the guide to LLM evals, and for the test copies an agent needs, the page on test environments for agent evals.