Open-source local model runtimes
Engines and servers that run open models on your own hardware and expose an API other programs can call, often shaped like OpenAI's. They are what the chat apps, agents and scripts elsewhere in this directory connect to when the model runs locally.
An MIT tool for downloading and running open models on your own machine with one command, built on llama.cpp, with a REST API, Python and JavaScript libraries, and a launcher that connects coding agents to local models.
The MIT C and C++ inference engine behind much of local AI, running GGUF models with 1.5 to 8-bit quantisation on CPUs, Apple Silicon and 17 hardware backends, with an OpenAI-compatible server and a web interface built in.
An Apache-2.0 engine from UC Berkeley's Sky Computing Lab for high-throughput model serving, with an OpenAI-compatible and Anthropic Messages API, 200+ model architectures, wide quantisation support, and parallelism across many GPUs.
An MIT engine that runs text, vision, voice, image and video models on any hardware, no GPU required, behind drop-in OpenAI, Anthropic and ElevenLabs-compatible APIs, over 60+ backends including llama.cpp, vLLM and whisper.cpp.
Questions people ask
What is the best way to run open models locally?
Ollama (MIT) for the quickest start: one command downloads and runs a model behind a local API. llama.cpp (MIT) for the engine itself on any hardware, with an OpenAI-compatible server. LocalAI (MIT) for chat, images, speech and embeddings behind OpenAI-compatible APIs. vLLM (Apache-2.0) for serving many users from GPUs.
Which local runtimes are compatible with the OpenAI API?
LocalAI, llama.cpp and vLLM state OpenAI-compatible APIs in their READMEs, and vLLM also serves the Anthropic Messages API. Ollama's README describes its own REST API and does not state OpenAI compatibility, so check its documentation for the endpoint your code needs. Jan and TextGen also serve local OpenAI-compatible APIs.
Do I need a GPU to run models locally?
No. llama.cpp is built for CPUs with quantisation down to 1.5 bits, LocalAI states no GPU is required, and Ollama builds on llama.cpp. A GPU speeds things up; vLLM is designed around GPUs for serving many users at scale from one server or several.
Are these runtimes free?
Yes. All four are OSI-licensed with no added conditions: Ollama, LocalAI and llama.cpp under MIT, vLLM under Apache-2.0, and none of their READMEs describes a paid tier. The open models you run carry their own licences, which you should read separately.
Running one of these inside your own environment, with your own data and your own security rules, is the kind of work Reveneau does. Read how a forward deployed engagement works.