Volotat's mini-AGI trains from scratch on 8GB of VRAM and keeps 99.84 percent of its knowledge after reading only chess: Reveneau

Why it mattersA model that keeps training on the files a team reads day to day removes the separate fine-tuning step and lets the parameter count grow with free disk space rather than with GPU memory.

Alexey Borsky released mini-AGI on 19 September, a byte-level language model that trains from scratch on a single 8GB VRAM GPU and keeps learning from every stream of characters it reads. The project reached 227 points on Hacker News and 261 GitHub stars in two days, under an MIT licence.

The design starts from three constraints, stated in the README. The model must fit in 8GB with room for gradients and optimiser state. It must not forget what it already knew when it reads something new. And it must be able to read anything, so the alphabet is the 256 byte values with no tokenizer to fit.

The forgetting number

Training on a single stream one subject at a time is the classic cause of catastrophic forgetting. When the author read 524,000 characters of chess into the model at the experts' own learning rate, the other seven subjects it knew climbed from 1.12 to 3.73 nats, a loss of most of its knowledge.

Running the trunk of the network at one tenth of the experts' rate takes that number from 2.23 nats to 0.0067 nats, which the README reports as 99.84 percent of progress retained against chance. The trunk carries 97.6 percent of the squared gradient norm, and slowing it down keeps the shared parts of the model still while the experts change. The gap between 50.68 percent and 99.84 percent retention is one value in a config file.

The project's own note underneath that finding is worth quoting: the expert pool is not what prevents forgetting. Freezing the working set costs only 13.8 percent of the effect, and in that arm 93 of 136 experts received no gradient at all and the model still collapsed.

Weights that live on disk

Every expert is a file on disk holding its weights and its Adam moments. Above disk sit two caches, a RAM cache and a resident VRAM working set. Before every chunk the model scores demand from the previous chunk's routing decisions and pages the right experts onto the card. The parameter count is bounded by free disk space.

The pool grows when the model runs short of capacity, and prunes experts that nothing has asked for over a long window. New experts are built by recombining hidden units from several existing ones, because a clone of one parent is too similar to the source, and a random expert computes nothing worth routing to.

Where the model actually is

The author calls the current run a small toy-level model on the first line of the README. On the project's own eight-subject mixture it reads at 0.833 nats per character on held-out text, with 318M characters read so far and 169 experts. MambaByte-353M, the closest published comparison, read 94 times more data.

The fitted data-scaling exponent is 0.239 with R squared 0.96, between Kaplan's 0.095 and Chinchilla's 0.28. Read straight off that trend, the README projects 3 to 24 days on one laptop GPU to reach bits-per-byte figures of 1.10 down to 0.80. Weights are not published yet: the first pass over the corpus is still running.

For a working developer, the interesting property is that reading is training. Point the same reader at your own files and the model updates on them, saves nothing until you pass --save, and scores the held-out mixture before and after so you see whether reading your notes made the model worse at anything else. Fine-tuning is the same event as reading, on the kind of GPU already in a laptop.

Source

This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.