Peter Vijeh labels 4,290 Reddit comments with Gemini for $9 and fine-tunes a local model to 0.83 F1

Peter Vijeh published a write-up on 17 September of how he replaced a per-call Gemini dependency for a narrow named-entity task with a fine-tuned local model. He paid Gemini 3.1 Pro to label 4,290 Reddit comments for $9, then fine-tuned GLiNER large v2.5 on those labels and now runs the tagger on his own machine. The Hacker News thread on the post has 82 points at the time of writing.

The task is extracting knife brands, models, and steel specifications from posts on Reddit, for a site he runs. Gemini did the labelling once, through OpenRouter at temperature 0, for a per-comment cost of $0.0021. GPU time for ten training runs came to $2.50, so the whole one-off cost was under $12.

The labelling recipe

Vijeh writes that Gemini "handles character offsets poorly", so instead of asking the model to return the start and end positions of each span, the prompt asked for the exact substring. TypeScript then computed the character positions from the returned substrings. The 4,290 comments produced 3,907 entity spans across 2,250 training examples on a held-out split.

The model that replaced the API

The local model is GLiNER large v2.5, a 459M-parameter DeBERTa-v3-large encoder. Vijeh reports 0.83 F1 on a held-out 225-comment validation set, with 0.904 F1 on brand, 0.877 on product, and 0.829 on material. He tuned per-class thresholds and reached 0.911 recall on material specifically.

Training itself is small: batch size 2 with gradient accumulation 8, learning rate 1e-5, 24 minutes on a Tesla T4. That is why the compute cost is a rounding error next to the labelling cost.

The one thing that broke five runs

Five of the ten training runs produced nothing. Vijeh traces this to a tensor called words_mask, which controls how sub-tokens pool into words in GLiNER. He had filled it as a binary attention mask, one value per position, when it needed to hold incremental word indices instead. The loss went flat and no error message named the tensor. He says the tensor was the whole cost of the project: monetary spend was tiny, but the days of debugging were not.

A team looking at the same trade-off should read the failure story as carefully as the F1 numbers. The saving comes from doing the labelling once, not from waving away the engineering. On a task narrow enough that a single frontier model can label a few thousand examples for the price of lunch, the maths shifts: one payment for the labels, then no payment per request.

Source

Primary source: I had Gemini train its own replacement for $9, by Peter Vijeh. Discussion: Hacker News thread 49740330.