AI NewsModels & agentsAnnouncement
Privatemode gets an off-the-shelf LLM to match Jev on typed decisions in one forward pass, and adds images while it is at it
Privatemode says you can turn any hosted LLM into a Jev-like typed decision model by numbering the options in the prompt, ending the prompt inside the answer, and reading the model's probabilities for the option indexes at that one position, and reports GLM-5.3-Flash matches Jev's accuracy across 29 public datasets while also answering questions on images.
Image: Privatemode
Why it mattersIf this holds up under other teams' benchmarks, the argument for buying a dedicated decision model shrinks to price and to a few 21-option-or-more use cases, and any team already paying for an LLM can add typed routing decisions without a second vendor.
A team already paying for a hosted LLM can now add typed routing decisions without buying a second model. Privatemode's Johannes Hötter and Marko Rosenmüller published a benchmark on 24 September showing GLM-5.3-Flash, driven by a prompt trick that costs one output token, matches TypeSafe's Jev decision model on accuracy across 29 public datasets. As a bonus the LLM setup answers questions about images, which Jev cannot read.
The trick, in one paragraph
Number the options in the prompt. Feed the state, the question and the option list to the model as JSON, with an integer index on every option. End the prompt with the literal string choice_index:, so the next token the model produces is an index. Ask the API for one token with a mask that allows only those index tokens, and read the log-probabilities the model assigns at that single position. Normalise them over the options and pick the highest. There is no fine-tuning. The model ships as it does.
The benchmark
Privatemode ran three systems on 29 public labelled datasets: GLM-5.3-Flash hosted on Privatemode, TypeSafe's Jev, and Laya, a 421-million-parameter local model. On the 20 text datasets both hosted systems answer, each is more accurate on 10 and within one percentage point on the remaining 8. The reported gap of 0.6 percentage points in Jev's favour is well inside chance (p = 0.64). Laya's median gap trails the other two by 13 to 15 points.
The number of options in a question matters more than the choice of system. On banking77's 77 intents and CLINC150's 151 intents, Jev hits an option-name budget of 192 tokens and cannot answer CLINC150 at all. GLM-5.3-Flash handles both.
Latency, cost and images
The cost question is not one-sided. Privatemode measured from a Frankfurt client and got 180 ms for GLM-5.3-Flash against 264 ms for Jev; from a US client the order reversed, 299 ms for Privatemode against 164 ms for Jev. On price, Jev stays cheaper: one million decisions costs about EUR 62 on GLM-5.3-Flash and about EUR 16 on Jev. Above 21 options the LLM sends more input tokens than Jev, so the gap widens on tasks with big option lists.
The image path is what the LLM adds that Jev cannot do at all. On RVL-CDIP, a set of 1,600 scanned business documents in 16 classes, GLM-5.3-Flash reaches 70.2% accuracy, and neither Jev nor Laya reads the file. An image adds about 1,350 input tokens, so a million document decisions works out to about EUR 270.
The reader who has been putting this off
A team that has been sitting on the decision to buy a dedicated model, and running its routing through a full JSON response instead, has a new answer to ask. If accuracy was the reason to buy Jev, the accuracy is now available on any hosted model that returns log-probabilities and lets you mask a single position. If speed was the reason, the numbers still favour Jev on ten-option-or-fewer routing at scale, especially from the US. And if the reason was that the decision is about a scanned invoice or a screenshot, Jev never had that.
Privatemode also released the playground, the code and the datasets, so the benchmark can be re-run by anyone with an API key to check.
Source
Primary source: Turn GLM-5.3-Flash into a Jev-like System One model, by Johannes Hötter and Marko Rosenmüller at Privatemode.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
