Melvin Vivas released Local Evals, an open-source grader for JSON output and tool calls that runs on your own laptop

Melvin Vivas, who publishes as donvito on GitHub, released Local Evals on 8 September and pushed his last commit on 11 September. The repository holds an Apache-2.0 licensed Node.js 22 app that grades three kinds of model output on the user's own machine: document to JSON, text to JSON, and tool-call proposals. It reached 91 GitHub stars within a week of the first commit.

What it grades

A run in Local Evals is a dataset plus a pipeline. Document to JSON pairs a vision-capable target for OCR with a text target for extraction, both graded against a JSON Schema. Text to JSON drops the OCR stage and grades the extraction alone. Tool-call runs check whether the model proposes the expected tool with the expected arguments, and record every call rather than executing any. Each case shows the input, the model output, the score and the execution log side by side.

The extraction step can send the schema inside the prompt (prompted-json), or use strict response_format.json_schema with the provider (schema-constrained-json). If the strict call comes back with output that does not match the schema, the run falls back to prompted-json for that case and logs a warning, rather than failing the whole batch. A truncated response stops the case with a diagnostic instead of being counted as an incompatibility.

Where the models come from

Local Evals connects to a model that is already running somewhere. It does not download weights and it does not start a server. The two documented paths are an OpenAI-compatible endpoint on a base URL like http://127.0.0.1:8080/v1, which covers llama.cpp and LM Studio, and OpenRouter's catalog at https://openrouter.ai/api/v1. Each named target records which of vision, structured JSON output and tool calls the model actually supports, and the app tests connectivity from the settings page before a run starts.

Datasets, run history and API keys stay in a SQLite file under .localevals/ in the working directory. Keys are encrypted with a companion .credentials.key file that has to travel with the database. An offline npm run demo command populates a separate demo database with synthetic runs, some deliberately regressed, so the app can be tried without a provider.

Eval platforms for LLM output like Braintrust, Langfuse and PromptFoo have grown into the space that used to be occupied by ad hoc scripts, and most of them are cloud services that read the prompts and the dataset. A grader that runs on a laptop and stores everything locally is a smaller answer to the same question, and it lets a team with private inputs, such as customer receipts or internal tool schemas, run regressions on real data without moving it. The tradeoff is that operating the eval suite, and any collaboration around it, is on the team rather than on a vendor.

Source

Repository: donvito/local-evals on GitHub. Apache-2.0 license, Node.js 22 or newer.