FreeToken runs a 35B model on an 8GB laptop GPU

Researchers from UC Berkeley and MIT released FreeToken, an open-source inference engine for running mixture-of-experts models on consumer hardware. InfoQ's Olimpiu Pop covered it on 29 August. Co-authors include Databricks co-founders Matei Zaharia and Ion Stoica, along with Song Han and Kurt Keutzer.

The problem it solves

A sparse mixture-of-experts model computes only a fraction of its parameters per token, but decoding still needs to route across hundreds of billions of inactive weights.

In a data centre, interconnects like NVLink hide the cost of moving those experts around. On a consumer machine, PCIe throughput of roughly 16 to 64 GB/s and host RAM latency become the step that limits speed. Existing edge runtimes keep inactive weights in system RAM and stream them to the GPU when they are needed, which stops execution completely on a cache miss.

What it does instead

Rather than halting the GPU on a miss, FreeToken splits the computation of a token between CPU cores and GPU tensor cores according to the interconnect throughput it measures at that moment. The paper calls this the q* policy.

A fast weight format plus full-layer double buffering lets weight streaming over PCIe overlap with active computation. An elastic memory manager reallocates VRAM between KV cache entries and resident expert slots at runtime, with no model reload.

There is also a part aimed specifically at agents. Coding assistants change the context window constantly through prompt edits, tool-call responses and thinking blocks, and a standard engine throws away its linear KV cache when the prefix changes, forcing a full recomputation. FreeToken checkpoints intermediate attention states at logical task boundaries, so editing a tool argument or injecting execution output reuses existing sub-sequence state instead of invalidating the cache.

The reported numbers

From the paper's benchmark section, as reported by InfoQ:

• Qwen3.6-35B at around 39 tokens per second on an 8GB RTX 4060 laptop
• DeepSeek-V4-Flash, 284B parameters, served on an RTX 5090 desktop
• GLM-5.2, 753B parameters, processed on a single workstation GPU
• 3 to 4 times faster decode and 6 to 30 times faster prefill than Ollama and llama.cpp on equivalent mixture-of-experts models

Those are the authors' own figures against their own baselines, which is worth stating plainly. InfoQ reports that community discussion on Hacker News and the LocalLLaMA forum has raised exactly that point, questioning whether the theoretical q* calculations remain true under real CPU dispatch latency, memory contention and expert residency under concurrent load, and whether the llama.cpp baselines were hand-tuned.

Treat the speedups as a reason to run the benchmark yourself.

No per-token bill, and the repository stays on your machine

Two reasons, and the second is the stronger one.

Cost. An agent loop against a local model has no per-token bill, which changes what is worth trying. Iterating on a prompt fifty times stops being a budget decision.

Where the code goes. A model running on your workstation does not send the repository anywhere. For teams with source code they cannot send to a third party, that has always been the constraint, and the hardware requirement is what made it impractical.

Available for NVIDIA RTX 30, 40 and 50 series on Linux and Windows, via FlashML.ai and GitHub.

Source

• FreeToken Unlocks Frontier MoE Inference on Consumer Hardware via Dynamic Co-Execution, InfoQ, 29 August 2026