llama.cpp
llama.cpp is an open-source inference engine under MIT for running language and vision models in plain C and C++ with no dependencies. It loads GGUF models straight from Hugging Face, quantises them from 1.5 to 8 bits to cut memory, runs on CPUs, Apple Silicon and 17 hardware backends including CUDA, Metal, Vulkan and ROCm, and starts an OpenAI-compatible API server with a web interface in one command.
What does llama.cpp do?
llama.cpp is "LLM inference in C/C++." Its README states the goal: "to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud." One command launches an OpenAI-compatible API server with a built-in web interface, loading a GGUF model straight from Hugging Face. Many projects in this directory, including Ollama, Jan, TextGen and LocalAI, run on it.
It is a "plain C/C++ implementation without any dependencies." Apple Silicon is "a first-class citizen" through ARM NEON, Accelerate and Metal; NVIDIA GPUs have custom CUDA kernels, with AMD through HIP. Its backend list has 17 entries, from CUDA, Metal, Vulkan, SYCL and OpenCL to CANN, Hexagon and WebGPU. Quantisation runs from 1.5 to 8 bits, and CPU and GPU hybrid inference runs models larger than your VRAM. It supports vision models and grammar-constrained output.
Key facts
- Licence: MIT, copyright the ggml authors, with no added conditions.
- An OpenAI-compatible API server with a built-in web interface, started with llama serve and a Hugging Face model name.
- Model format: GGUF, downloaded directly from Hugging Face.
- Quantisation at 1.5, 2, 3, 4, 5, 6 and 8 bits to reduce memory use.
- Hardware: Apple Silicon through NEON, Accelerate and Metal; x86 with AVX, AVX2, AVX512 and AMX; RISC-V; 17 backends including CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, CANN and WebGPU.
- CPU and GPU hybrid inference for models larger than total VRAM.
- Language and vision models; GBNF grammars for constrained output.
- Install from llama.app, Docker, prebuilt binaries or source. Latest tagged release when read: v0.4.1 on 2026-09-14; 128,615 stars on 2026-09-17.
What does it replace, and where does it fall short?
llama.cpp replaces the OpenAI API with an OpenAI-compatible server on your own hardware, down to a laptop CPU, with no dependencies and the widest hardware list on this site. It is the engine underneath several other entries, and running it directly gives the most control over quantisation and hardware. It sits on the open-source alternatives to the OpenAI API page as the pick for any hardware.
Where it falls short: it runs GGUF models, so other formats need converting, and choosing a quantisation is a trade-off you make yourself. It is an engine rather than an app: no model library, no accounts, no images or speech. For a friendlier wrapper, Ollama; for serving many users from GPUs, vLLM.
How does llama.cpp run?
Install from llama.app, use Docker, download a prebuilt binary from the releases page, or build from source. Then llama serve with a Hugging Face GGUF model starts the API server and web interface. An Android build guide and a multi-GPU guide are linked. No hosted or paid tier is described.
Who is llama.cpp for?
A developer who wants the engine itself: any hardware, exact control over quantisation, and an OpenAI-compatible server with nothing else attached. Someone who wants models managed for them should use Ollama, which is built on it.
What limits does the README state?
From the README: the OpenVINO backend for Intel CPUs, GPUs and NPUs is marked in progress.
Questions people ask
Is llama.cpp open source?
Yes. llama.cpp is MIT licensed, copyright the ggml authors, with no added conditions. Ollama, Jan, TextGen and LocalAI all build on it and credit it in their READMEs, so it is the engine underneath much of the local AI in this directory.
Does llama.cpp have an OpenAI-compatible API?
Yes. Its README gives one command to launch an OpenAI-compatible API server, llama serve with a Hugging Face model, and the server includes a web interface. Code written against the OpenAI SDK can point at it.
Can llama.cpp run on a laptop without a GPU?
Yes. It is a plain C and C++ implementation with no dependencies, optimised for x86 and ARM CPUs, with quantisation down to 1.5 bits to fit models in memory. On Apple Silicon it uses Metal, and CPU and GPU hybrid inference runs models larger than your VRAM.
How does llama.cpp compare with Ollama?
Ollama is built on llama.cpp and adds a model library, one-command downloads and its own API. llama.cpp gives the engine directly, with every quantisation and hardware option exposed. The open-source alternatives to the OpenAI API page compares both with LocalAI and vLLM.
What is a GGUF model?
The model file format llama.cpp loads, which bundles a model's weights and settings in one file, usually quantised to a smaller size. The README's server command downloads GGUF models directly from Hugging Face by name.
Sources
- llama.cpp README and MIT LICENSE: github.com/ggml-org/llama.cpp, read 2026-09-17.
Compared with the others
On the open-source alternatives to the OpenAI API page, llama.cpp is the pick for the engine on any hardware. MIT C and C++ engine with an OpenAI-compatible server and web interface, GGUF models at 1.5 to 8 bits, and 17 hardware backends from CPU to CUDA and Metal.
Also on that page: Ollama for the quickest local start, LocalAI for every modality behind openai-compatible apis, vLLM for serving many users from gpus.
More local model runtimes
An MIT tool for downloading and running open models on your own machine with one command, built on llama.cpp, with a REST API, Python and JavaScript libraries, and a launcher that connects coding agents to local models.
An Apache-2.0 engine from UC Berkeley's Sky Computing Lab for high-throughput model serving, with an OpenAI-compatible and Anthropic Messages API, 200+ model architectures, wide quantisation support, and parallelism across many GPUs.
An MIT engine that runs text, vision, voice, image and video models on any hardware, no GPU required, behind drop-in OpenAI, Anthropic and ElevenLabs-compatible APIs, over 60+ backends including llama.cpp, vLLM and whisper.cpp.
Added September 17, 2026. Every claim above comes from the project's README, LICENSE or model card, read on September 17, 2026, or from the GitHub API on the date shown in the panel. Found an error? Write to reveneau@licheo.com and it is fixed in the next weekly pass. Repository: github.com/ggml-org/llama.cpp.
Running one of these inside your own environment, with your own data and your own security rules, is the kind of work Reveneau does. Read how a forward deployed engagement works.