Aim a microbenchmark at 300 milliseconds, matklad argues

A microbenchmark that runs for a tenth of a second is wrong in one direction, and one that runs for ten seconds is wrong in the other. Alex Kladov, who writes as matklad, argues in a short blog post that there is a right answer in between, and that it is around 300 milliseconds.

The post is titled "Benchmark in Milliseconds" and is a short piece of advice rather than a research note. The argument is that a microbenchmark should take long enough to swamp fixed overhead and short enough that a developer wants to run it again.

Why 300 milliseconds, not three or three thousand

Below roughly ten milliseconds, the measurement is poisoned by things that are not the code under test: interpreter startup, warm-up time, timer resolution, caches that are cold on one run and warm on the next. The signal you want is the inner loop, and in a run that short the inner loop is a small part of the total.

Above one second, each iteration costs you the back-and-forth that makes performance work tractable. A developer changes an allocation strategy, runs the benchmark, and either tries another change immediately or walks away. One second is already a stretch; ten is a coffee break.

In the middle, hundreds of milliseconds give two readable properties at once. The number is in a unit people process by eye: Kladov contrasts "1.31s" with "239ms" and notes that the second one is easier to scan. And a single run is cheap enough to repeat without ceremony. He recommends running ten in a row and looking at the spread, which is a cheap way to see whether the number is stable without reaching for a statistical tool.

The quieter point about what benchmarks are for

The post makes an argument about the purpose of a microbenchmark that is worth copying. Kladov writes that the goal is to give the author "enough intuition to make a correct decision," not to publish a figure with a confidence interval. A microbenchmark is a dial a developer reads to decide whether a change was useful. It is not evidence for a paper. In that light, three hundred milliseconds, ten runs, and an eye on the spread is enough.

The practical move for a working developer: pick an input size that puts a single run near 300 milliseconds. If the benchmark finishes in two milliseconds, give it more work until it does not. If it finishes in two seconds, cut the work until it does not. Then run it ten times in a row. The spread tells you whether the result is reliable, and the small number of digits of change between runs tells you whether a code change moved the number or the noise did.

Source

Primary: Benchmark in Milliseconds, Alex Kladov on the matklad blog.