Ubicloud hits 100 percent tool-call accuracy with new vLLM 0.31 flag

An open-weight model that misses 2 tool calls in every 25 now has a one-flag fix. Jesse Donkervliet and Junhao Li at Ubicloud published measurements on October 20. The team ran Moonshot AI's K2VV ToolCall benchmark, a 2,000-request test. Kimi K2.6 served through vLLM 0.28 passed 92 percent of tool calls. With constrained decoding on, Kimi and DeepSeek hit 100 percent schema accuracy. vLLM 0.31 added the feature as a server-side flag. The post reached 64 points on Hacker News.

What a tool-call failure looks like

An agent sends the model a tool description as a JSON schema. The model is supposed to answer with a call that matches that schema. A call that does not match is dropped by the harness. The agent then retries, pollutes its context with error messages, and spends more tokens. Two in every 25 calls failed in Ubicloud's test of Kimi K2.6 on default vLLM 0.28. The simple approach lets the model sample tokens freely and checks the output after.

What constrained decoding does

Constrained decoding guides generation one token at a time. At each step, only tokens that match the schema are allowed. vLLM uses the XGrammar library. XGrammar precomputes token masks for each phase of a tool call: the name, the index, the arguments and the end markers. The engine applies those masks on the GPU during generation. Ubicloud reports that the runtime cost is negligible on this benchmark.

Why vLLM leaves it off by default

vLLM added constrained decoding in version 0.25. Ubicloud enabled it in a fork and saw accuracy fall back to 91 percent when it ran the benchmark again. The team traced the drop to a vLLM default change. vLLM maintainers found three problem shapes with model-wide auto-enable. The first is general model issues where unconstrained output diverges from the format. Users reported this for gpt-oss-20b, Qwen 3.6 and GLM 5.1. The second is bugs in translating a model's schema to grammar. The third is interactions with speculative decoding that can wrongly reject tokens.

vLLM 0.31 adds the --tool-strict-level flag. A team can turn constrained decoding on for a known-good model. Other models stay on the default. Ubicloud re-ran the K2VV ToolCall benchmark with the flag on. The result was 100 percent accuracy across the Kimi and DeepSeek versions it serves. The team then deployed the change to production.

What this changes for a team using open-weight models

The accuracy question is now about which models and which schemas, not whether the technique exists. A team already carrying its own patch can read the Ubicloud post as validation of the upstream flag. A team using open-weight models without a fix can expect a few percent of tool calls to fail silently. Each failure costs tokens, time and context. The fix is a flag away. The verification is still on the team.

Source

Ubicloud blog and Hacker News discussion.