Switch lets Jev grade evals by picking one page at a time

Most AI judges grade a long answer by reading every page at once. Onur Solmaz, an engineer at Hugging Face, released switch, a small framework that turns that into a navigation task. The judge picks one page to read. It sees that page. It picks the next page, or it stops. Only then does it answer the rubric. The repository is ten days old and has 430 stars.

Switch is a harness for decision models like TypeSafe's Jev. Solmaz says a decision model answers typed questions with probabilities. It writes no text. So it cannot generate tool calls on its own. Switch generates the tool calls in code. The model only picks one from a short list each step. Jev was the first such model. The llama.cpp server now runs open decision models through the same API.

Domains, states and the trace file

A user of switch writes a domain. The domain says what the state of a task looks like. It says what the model must answer in that state. It says how the answers lead to the next state.

One worked example is a support-ticket triage domain. The model reads one log at a time. When the evidence is enough, it picks a stop choice. A last step picks which team gets the ticket. Another example is chess. Another is WikiRace, which clicks from one Wikipedia article to another.

The main example is the rubric judge. A task output can be 200 pages of PDFs, Word files and Markdown. The rubric may need only a few of those pages. The usual judge wastes most of its context. The switch rubric judge reads parts of files one step at a time. It loads page images. It searches for terms. It stops when it can grade each rubric item. Every state, question and answer goes to a trace file. A run can be replayed step by step without calling the model again.

What Jev can and cannot do inside switch

Switch works with any decision model that the pi-ai library lists as a classifier. Jev is one of those. Solmaz says Jev is weak at chess. As a judge, he says, it is not yet strong enough to replace the best chat models.

He proposes a use for narrow jobs. A chat model can produce enough graded examples first. Then an open decision model can be fine-tuned for one domain inside a harness like switch. A one-job agent built that way could cost a hundredth of what a general chat model costs, he writes. It never needs to produce prose.

The published license is MIT. A browser viewer in the repository plays a run back step by step. A replay command can fork from any step. That shows what a different answer would have changed.

For a team already using a chat model as the judge, switch asks a simple question. If the rubric only needs a few pages, let the model pick those pages. Stop paying to read every page every time.

Source

• switch on GitHub, Onur Solmaz