NaiveAI's 2,100 tokens per second peak hides a 984 sustained rate

Gabriele Beltramo ran a benchmark against NaiveAI's new Flash coding model. His review was published on Latent Space today. The point of the review is the gap between the vendor's headline number and the sustained one. NaiveAI says its model runs at 2,100 tokens per second. Beltramo measured about 984 tokens per second during thinking. He measured about 1,816 tokens per second during answers.

What he tested

Beltramo compared NaiveAI against two other Flash-class coding models. DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash were the two. He ran 30 questions across difficulty levels. He also set three coding tasks: a solar system simulator, a 3D tree generator, and a ray tracer.

The benchmark numbers he reported

NaiveAI scored 71.9 on NL2Repo-Bench. That was the top score of the three. On DeepSWE v1.1, DeepSeek led at 77.0. NaiveAI scored 67.8. On Agent's Last Exam, DeepSeek scored 35.4. NaiveAI scored 32.4. These are Beltramo's own figures.

The speed difference was clearer. NaiveAI's median end-to-end time was 3.1 seconds. GLM 5.3 Flash took 5.3 seconds. DeepSeek V4.1 Flash took 10.1 seconds. Those are median times across the test runs.

The coding results were mixed

NaiveAI did well on the solar system task. It tied with DeepSeek on the tree generator. It came second on the ray tracer. Beltramo wrote that NaiveAI was "much more consistent" than GLM 5.3 Flash. The model's thinking mode also used about nine times more tokens than competitors on the hard problems.

His recommendation

Beltramo wrote that the model is a fit for teams that want fast iteration. He said it is less of a fit for teams that watch token costs. The main trade is that NaiveAI returns an answer faster but burns more thinking tokens on harder tasks.

Where the code is

Beltramo published the test code on GitHub at gbeltramo/llm-verbosity. Another team can run the same benchmark on its own account and compare its numbers. That is the whole point of the review. Vendor speed claims measure the peak throughput on an ideal prompt. A team running agents against real tasks lives at the sustained rate. On this model the two rates are about two to one apart.

A team shopping for a new coding assistant now has a worked example. The review shows how to check a vendor's speed claim. The test code is public and the comparison models are named. A team can swap in its own top three candidates and run the same questions on its own accounts. The gap between claimed and measured speed in this review is one data point. The method behind it is reusable.

Source

• NaiveAI Promises 2,100 Tokens per Second. How Well Does It Code?, Latent Space, Gabriele Beltramo
• gbeltramo/llm-verbosity benchmark code, GitHub