ToolReplay checks AI agent tool-call transcripts for cheating, redundant calls, and permission overreach

ToolReplay is a Python command-line tool that reads recorded transcripts of an AI agent's tool calls and reports three kinds of trouble: non-determinism, where the same call returned different results in later runs, redundant calls, where the agent asked the same tool for the same thing more than once, and permission overreach, where the agent called a tool it was not allowed to call. The repository was created on 14 September and has picked up 170 stars in the hours since.

What it reads

The input is a JSON Lines file where each line records one tool call: an index, the tool name, the arguments and the recorded response. A separate JSON scope file lists which tools each agent is allowed to call. That is it, and there are no other dependencies: the README lists Python 3.11 or newer and the standard library.

Replay is deterministic on purpose. The tool never calls a live service. It reads the recorded response and checks it against the same call made earlier in the same transcript, so an audit run is reproducible, safe to run offline, and has no side effects.

Hash chain over the transcript

The audit also seals the transcript into a SHA-256 hash chain. Each record's digest folds in the previous digest along with the current record, so any later change to a single line changes every digest that follows it. The check catches an edited or reordered transcript rather than trusting the file as it stands.

Scope enforcement is strict and case-sensitive. If a scope allows fs.read and the agent called fs.write, the tool reports it as overreach and exits with a non-zero code.

What that costs to adopt

The output is deterministic and carries no timestamps, so two clean runs produce byte-identical reports that Git can diff. Three exit codes make it clean to wire into CI: 0 for clean, 1 for findings or a broken chain, 2 for a usage or parsing error. The dirty sample in the repository has 6 tool calls, the clean sample 4, and the 34-test suite runs in about 0.019 seconds.

For a team that already logs its agent runs, the cost is writing the transcript out in the format ToolReplay reads and keeping a scope file per agent. The value is that a review question that used to need someone to read a long log now returns a decision from a script, and the same script says whether the log was edited after the fact. The traction here is early, 170 stars on the day the repository was created, but the shape is a real tool that a working team can add to its pipeline this afternoon.

The licence is MIT and the code is on GitHub.

Source

Primary source: ToolReplay on GitHub.