Bespoke Labs releases Nimble, an open Jev alternative that hit 90.12 percent on the same 324 examples

Bespoke Labs released Bespoke Nimble on 18 September, a 9-billion-parameter open-source model that answers the same shape of typed question as TypeSafe AI's Jev: pick one option from a list, or decide true or false, and return a probability for each allowed answer. The repository holds the training data, the recipe and the code; the weights are on Hugging Face as bespokelabs/Bespoke-Nimble-9B.

One token per answer, no reasoning stream

Nimble reads a context and a flat schema. Each field is either an enum with up to 26 string choices or a boolean. Every allowed answer has a code that is one token long, and the scorer reads the model's raw scores for those tokens directly. There is no generated text to parse. Bespoke Labs describes four uses: route a request to one of several destinations, check a yes or no condition, apply a policy, and rate an outcome on an ordered scale.

Nimble picks one answer in one step, without writing any reasoning first, so a decision is one forward pass. The model accepts up to 2,048 tokens per prompt, including the schema.

The numbers Bespoke Labs published

Bespoke Labs ran seven models on the same 324 held-out examples and reported the count of matches to the reference labels: Gemma 3 270M IT at 28.70 percent, Qwen3.5-0.8B at 45.37 percent, Qwen3.5-4B at 61.42 percent, Qwen3.5-9B (the base model) at 66.36 percent, Qwen3.8-27B at 84.88 percent, Bespoke-Nimble-9B at 90.12 percent, and Jev 1.13.0 at 93.21 percent.

Bespoke Labs says the reference labels are synthetic, produced by a model check with no human review, and that the 324 examples form 162 closely related pairs from six source families, so the test is narrow. The training set has 2,676 examples across 10 categories including commerce, education, science, software, supply chain and workplace.

How it was trained

The recipe applies LoRA to Qwen3.5-9B and optimises cross-entropy over the allowed candidate scores. LoRA rank 16, learning rate 5e-5, effective batch size 8, one epoch, BF16, 2,048-token prompt limit. Tuning ran on L40S; the final fit and evaluation ran on H100. Bespoke Labs says it did not distil from Jev: the training uses hard reference labels rather than saved Jev probabilities.

The data curation method is contrastive. For each example the team writes two evidence sentences, then changes at most eight words in one to flip one fact and therefore the answer, keeping the rest of the context the same. Each pair is checked against the policy, and each sentence is removed in turn to confirm the missing fact becomes unknown. Only pairs where every check passes and the two labels differ are kept.

Where the traction sits and what to run it on

The repository was 460 stars and 35 forks two days after release, from an organisation that has published open datasets and models before. A Mac needs Apple Silicon and enough memory to hold the 18 GB weights plus overhead; Bespoke Labs recommends 64 GB rather than 24 GB. A Linux machine needs an NVIDIA GPU that supports BF16. The MLX runner on the Mac reads the prompt once and scores every field in parallel; the CUDA runner scores each field with the full prompt.

Nimble is one of several open-source models this month aiming at Jev's decision surface, and the first that carries a published comparison against Jev on the same examples with the working shown. A 3.09-point gap between an open model and a closed one is a specific number a team can weigh against the price of a paid API call.

Source

• Bespoke Nimble repository
• Bespoke-Nimble-9B on Hugging Face