oMLX saves the KV cache to SSD so Macs skip recomputing context

oMLX is a local inference server for Apple Silicon, built on Apple's MLX framework and managed from the macOS menu bar. It appeared on Product Hunt this week with the line "Mac LLM server that cuts agent wait times from 90s to 5s". The repository is Apache 2.0, created on 13 February 2026, and had 21,152 stars and 1,801 forks when this was written.

The problem it targets

A coding agent sends many requests where the prompt prefix keeps shifting. Every turn edits something earlier in the conversation, injects a tool result, or adds a thinking block.

Existing MLX servers invalidate the KV cache when that happens, which forces a full recomputation of the whole context. After a few turns you are waiting a long time before the first token appears, every single turn.

That is the actual reason local models feel unusable for agent work. It is a caching problem rather than a speed problem, which is why buying a faster machine does not fix it.

What it does

oMLX persists KV cache across two tiers: recent data in memory and older data on SSD. When a previous prefix comes back, it is restored from disk rather than recomputed. The project says past context stays cached and reusable across requests even when the context changes mid-conversation.

Around that sits a FastAPI process wrapping MLX, with an engine pool that holds several models in memory at once and evicts the least recently used one under memory pressure. It supports text models, vision models, OCR and embeddings, and serves both OpenAI-compatible and Anthropic-compatible endpoints, so it can directly replace the backend for tools that support either.

The menu bar app is the part the author says he wanted and could not find: pin everyday models in memory, automatically load heavier ones when needed, set context limits, and manage it from one place.

On the numbers

The 90 seconds to 5 seconds figure is the product's own marketing line, and the community reports of time-to-first-token dropping from 30 to 90 seconds down to 1 to 3 seconds are users describing their own machines.

None of that has been independently measured, and it depends heavily on your model, your context length and your SSD. The project publishes benchmarks on its own site, which is the right place to start and not the same as verification.

What is checkable is the mechanism, and the mechanism is sound. Restoring cache blocks from disk is cheaper than recomputing attention over a long context, and the shifting-prefix pattern is exactly what agent loops produce.

Where it fits

Every reduction in what a long context costs to keep makes running the largest models on hardware you own more practical, and cache reuse from disk is one of those reductions.

Apache 2.0, Python 3.11 to 3.13, Apple Silicon only.

Source

• jundot/omlx on GitHub, Apache 2.0. Star, fork and date figures read from the GitHub API on 31 August 2026
• Found via Product Hunt