Slotstream runs a 125B model on an 8 GB Mac from SSD

Carlos Galarza published slotstream, a Swift binary that runs Qwen3.8-Flash-Next on an Apple Silicon Mac by streaming the model's experts off the SSD instead of holding all of them in RAM. Qwen3.8-Flash-Next is a 125-billion-parameter mixture-of-experts model, and its 4-bit weights are 105 GB on disk. The repository was created on 28 August and has gained 241 stars in five days.

How it works

A mixture-of-experts model only uses a handful of its experts for any given token. Slotstream reads the routed experts on demand from the SSD into a fixed cache of memory slots that are shared across layers, so the working set stays small. The alternative most local runtimes take is to memory-map the whole weight file, which asks the kernel to load close to 100 GB even when almost none of it is needed for the current token.

What it costs and what it delivers

The README publishes measured numbers on the author's hardware. On an 8 GB Mac, the runtime allocates around 8.1 GB and produces about 3 tokens per second. On a 16 GB Mac, it uses around 10 GB and produces about 4 tokens per second. A 24 GB machine gives around 8 tokens per second, and a 48 GB M5 Pro produces about 12 tokens per second in warm decode, with a peak memory usage of 32 GB. Engine startup takes about 2 seconds because only the shared, non-expert part of the model loads, and a full 32k-token prompt takes around 3 minutes to process before the first output token.

The constraints

Slotstream only runs on Apple Silicon and macOS 14 or later, and the engine is tied to the specific structure of Qwen3.8-Flash-Next in 4-bit form. It does not run other models. The user needs around 110 GB of free disk space for the weights, which in practice means a 512 GB machine. The binary is MIT-licensed and the weights are under the Qwen community licence. It exposes an Ollama-compatible API and an OpenAI SDK-compatible interface, so an existing client that already works with Ollama can connect to it without code changes.

Disk space and waiting time now set the limit

The reason to notice this is the limit it changes. A 125-billion-parameter model on a laptop that has 16 GB of RAM has, until now, needed either a rented GPU or a workstation with far more memory than a laptop carries. Slotstream shows the limit is set by disk space and by how long you are willing to wait, and that a working Ollama-compatible endpoint is possible on hardware a developer already has. The tokens per second are low enough that this is a testing and evaluation tool for the near term, so the honest use is a local setup for prompt work and small-batch experiments before spending on a cloud instance.

Source

Primary source: carloslfu/slotstream on GitHub.