Hugo Vergnes trained a 3.8B language model to 0.384 on CORE for $998 in 43 hours

Hugo Vergnes published a write-up on 2026-09-04 of a from-scratch training run for a 3.8-billion-parameter decoder-only language model. The run took 43 hours on eight rented B200 GPUs, cost $998, and scored 0.384 on the CORE benchmark. The article reached the front page of Hacker News on 2026-09-10.

What the run measured

Vergnes started from Andrej Karpathy's nanochat as a reference. His own project, called little-lm, is config-driven: every run is a YAML file that names the model, dataset, optimizer, schedule and callbacks. The 3.8B model uses a Llama-style architecture with RMSNorm, RoPE, grouped-query attention with 24 query heads and 8 key-value heads, relu-squared MLPs, QK-norm, and value embeddings on every other layer that take 721M of the 3.848B total parameters.

For context, Vergnes reports GPT-2 1.5B scores 0.2565 on CORE and nanochat d32, at about 1B parameters and roughly the same $1,000 budget, scores 0.310. His 3.8B run at 1024-token context scored 0.338 for $820. Rerunning the same recipe at 2048-token context scored 0.384 for $998.

Five changes that made the difference

An earlier 858M run on FineWeb-Edu scored 60.45 percent on PIQA, worse than the 2019 GPT-2 124M at 63 percent, after six days of A100 time. Vergnes lists five changes from that post-mortem that produced the 3.8B result: a trapezoidal learning-rate schedule that holds a useful rate through most of the run instead of decaying to zero; the Muon optimizer for matrix parameters with AdamW for everything else; the ClimbMix dataset in place of FineWeb-Edu; FP8 training through torch._scaled_mm with dynamic tensorwise scaling on all three GEMMs, and padding the vocabulary from 50,257 to 50,304 tokens to unlock the fast tensor-core path; and a 1024-token training context that doubles batch size at fixed memory.

He measured 480,000 tokens per second in steady state on the 8-B200 node, 92 percent SM activity and 40 percent SM occupancy, which he reads as compute-bound and well-fed rather than starved on data. That works out to about 1,047 TFLOP/s sustained per B200, or 25 percent MFU against Blackwell's dense FP8 peak.

Where the context-length gain came from

The jump from CORE 0.338 at 1024 tokens to 0.384 at 2048 tokens was almost entirely two tasks: SQuAD went from 0.000 to 0.311, and boolq from 0.513 to 0.710. Both are 10-shot tasks in the DCLM CORE bundle with median prompts well above 1024 tokens, so at the shorter context Vergnes's harness kept the test example and cut the demonstrations. On SQuAD, scored on exact-token match, that drove the score monotonically to zero as the model got more fluent. The other 19 tasks together moved +0.008.

The write-up also documents what did not work: document-boundary masking with flex attention, Liger RMSNorm (slower than PyTorch 2.9's built-in), Liger RoPE (2.2 times faster in isolation but no measurable end-to-end change), and streaming datasets (2 to 3 percent slower than local shards and vulnerable to network hangs).

For a team choosing whether to fine-tune an open model or train one from scratch, the $998 recipe sets a floor: a solo engineer on rented GPUs can now land a 3.8B model that beats OpenAI's 2019 frontier by a wide margin, with the config, the ablations, and the failed experiments all documented. Vergnes measured a 2.59x throughput jump on the same code from a single RTX 5090 to a single B200, before accounting for the extra VRAM that lets a larger batch fit.

Source

• Hugo Vergnes, "Training a 3.8B LLM to 0.384 CORE for $998"
• Hacker News discussion