A 48 GB Mac mini runs Qwen3.6-35B at 34 tokens per second

Kevin Lewis, writing on lws.io on 1 September, published a description of the local inference setup he runs on his own Mac mini, with the numbers he measured. The post drew 294 points on Hacker News.

The hardware and the models

The server is an M4 Pro Mac mini with 48 GB of unified memory, which Lewis notes carries 273 GB per second of memory bandwidth. Two four-bit models run on it. The main one is Qwen3.6-35B-A3B, quantised with OptiQ to about 20 GB in memory. It is a mixture-of-experts model: 35 billion parameters in total, only about three billion active on any given token, which is what allows it to run inside 48 GB at all. A smaller Gemma-4-E4B model at four bits runs alongside it at about 2.4 GB, for lighter tasks.

The measured numbers

For Qwen3.6-35B-A3B at four bits, Lewis reports 325 tokens per second while the model is reading a prompt and 34 tokens per second while it is writing a reply. He puts the quality cost of the four-bit quantisation at one to two points below BF16 on most benchmarks, quoting OptiQ.

How the parts are connected

The inference engine is oMLX, listening on port 8000. oMLX includes an admin dashboard, a Hugging Face model browser, and persistent KV cache: cached prefixes are written to the SSD and rebuilt in milliseconds on the next call rather than recomputed from the tokens. All three of Lewis' devices, the Mac mini, a MacBook, and an iPhone, are connected by a private Tailscale network, so the clients reach the server over an encrypted internal address without any port being exposed to the public internet.

On top of the server, an agent backend called Hermes runs on the Mac mini and is reachable from a Telegram bot on the phone and a desktop client on the laptop. Apollo on iOS and Raycast on macOS connect to the oMLX endpoint directly for quick chats. A separate agent called Pi handles coding work.

The number a team pricing local inference for a coding or agent workload wants is what the actual machine does on the actual model with the actual clients connected to it, rather than the vendor's main claim. 34 tokens per second of generation is enough for a personal assistant loop and slow for pair programming. Prompt processing at 325 tokens per second is what makes long-context work bearable on this hardware at all, and the KV cache persistence is what stops that prompt being re-read on every turn.

Nothing here is a review of the Mac mini or the models. It is one honest write-up of one working setup, and its value is that a team considering the same class of machine can now start from a real measurement rather than a marketing chart. The next step, if a team is serious about it, is to run the same model on the same hardware with the traffic pattern the team actually has, since prompt length, batching, and concurrent clients all change the number.

Source

Primary source: Kevin Lewis, "My local model setup on an M4 Pro Mac mini", 1 September 2026.