llama.cpp

llama.cpp is the inference engine that proved you do not need a datacenter GPU to run a large language model. Maintained by ggml.org and written in C/C++ under the MIT license, it compiles to a single small binary that runs on CPUs, NVIDIA GPUs (CUDA), AMD GPUs (ROCm), Vulkan-compatible hardware, and Apple Silicon (Metal). Its primary innovation is the GGUF quantization format, which reduces model file sizes dramatically while preserving usable accuracy.

First released in 2023, llama.cpp started the local model movement. Before it, running a 7B parameter model on a laptop was impractical without a high-end GPU. Today, over 118,000 GitHub stars and 1,776 contributors back its development. Many tools you already know — Ollama, LM Studio, and others — are built directly on top of llama.cpp, making it the de facto base layer for local LLM deployment.

This engine solves a specific problem: running open-weight models on whatever hardware you have, not the hardware you wish you had. It prioritizes portability and memory efficiency over peak throughput. If you need to serve a model on a CPU-only server, an Apple Silicon Mac, or a modest GPU with limited VRAM, llama.cpp is the most proven choice. It competes with engines like vLLM, SGLang, and TensorRT-LLM, but those are designed for high-throughput GPU clusters. llama.cpp fills the gap for edge, desktop, and single-user production workloads.

llama.cpp operates around the GGUF model format. You obtain a GGUF file (convert from Hugging Face, download directly, or use the built-in convert_hf_to_gguf.py script). The engine loads that file into memory, creates a computation graph using the underlying ggml tensor library, and runs inference on whatever backend you compiled it for.

There are three main ways to interact with it:

• CLI (`llama-cli`): Load a model and prompt it interactively or via a one-shot command. Best for quick testing and scripting.
• Server (`llama-server`): A built-in HTTP server that exposes an OpenAI-compatible API. It supports chat completions, completions, embeddings, and grammar-constrained output. You point any OpenAI client at it with a swapped base URL.
• Library (`libllama`): Embed llama.cpp directly into your own C/C++ or Python application via bindings.

To serve a model, you run:

1llama-server -m path/to/model.gguf --port 8080

That is it. The server handles continuous batching, context management, and concurrent requests out of the box. The API is compatible enough that you can replace OpenAI’s /v1/chat/completions with no code changes to your application.

llama.cpp is not the fastest engine on a 8x A100 cluster. It is the fastest on a 4GB Raspberry Pi or a 16GB MacBook Air. Its design philosophy trades raw throughput for hardware flexibility and memory efficiency.

Hardware support is its strongest feature. One codebase compiles for:

• CPU: AVX2, AVX512, NEON, and other SIMD optimizations. Runs entirely in system RAM, no GPU required.
• NVIDIA GPU: CUDA backend with support for compute capabilities 5.0+. Works on consumer cards (RTX 3060) and enterprise (H100).
• AMD GPU: ROCm backend, tested on RX 7900 XTX and MI series.
• Apple Silicon: Metal backend, using the unified memory architecture effectively. M1/M2/M3 machines can run models up to their total RAM.
• Vulkan: Cross-vendor GPU acceleration for hardware not covered by CUDA or ROCm.
• Multi-GPU: Split layers across multiple GPUs or combine CPU and GPU for larger contexts.

Memory is controlled via GGUF quantization. A 7B model in 4-bit (q4_K_M) uses roughly 4GB of RAM/VRAM. A 70B model in 4-bit fits in 40GB, making it feasible on a used RTX 3090. llama.cpp also supports K-quant (advanced quantization methods available in GGUF files). KV cache is managed with optional cell-based caching; paged attention is not native but context reuse is handled efficiently for small batch sizes.

Throughput: On a single GPU, llama.cpp delivers strong single-stream performance but does not scale to high concurrent loads like vLLM. Benchmarks from Red Hat and academic papers show it maintains consistent inter-token latency under low concurrency, but time-to-first-token increases with queue depth. For a single user or a handful of simultaneous requests, it is more than adequate. For 100+ concurrent requests, it is not the right tool.

• GGUF Quantization: The standard for local model distribution. Provides fine-grained control over size vs accuracy tradeoffs (q2, q3, q4, q5, q6, q8, f16). Enables running models on devices with as little as 512MB RAM.
• llama-server with OpenAI-compatible API: Drop-in replacement for OpenAI’s API. Supports streaming, function calling, structured output (via GBNF grammars), and multimodal inputs (images with vision models).
• Grammar-constrained output: Define a formal grammar (GBNF) to enforce structured JSON, code, or custom formats. Critical for applications requiring deterministic output like data extraction or program synthesis.
• Continuous batching: Dynamically batches incoming requests to improve GPU utilization while maintaining low latency for individual requests.
• Structured output: Beyond grammar, supports JSON schema mode and guided generation for reliable structured responses.
• Streaming: Full token-by-token streaming over HTTP SSE, matching the OpenAI streaming contract.
• Multi-GPU: Distribute model layers across multiple GPUs or backends. Useful for running a 70B model on two 24GB cards.
• One-line install: brew install llama.cpp on macOS. Prebuilt binaries for Linux and Windows via GitHub releases, Docker, or package managers.
• Multimodal support: Native support for LLaVA-style vision models, allowing image understanding in the same server.

CPU-only inference is the standout use case. Teams running models on cloud instances without GPUs (e.g., t3.xlarge in AWS) can serve a 7B model at 10-20 tokens/sec using CPU-only llama.cpp. This is cost-effective for low-traffic internal tools.

Edge and embedded devices: Quantized GGUF models run on Raspberry Pi 5, Jetson Nano, and even phones via Termux. llama.cpp is used in field robots, offline translators, and privacy-preserving medical assistants (as documented in the MedLocalGPT project).

A base for other tools: Ollama, LM Studio, and dozens of open-source projects wrap llama.cpp. If you are building a custom runner or a lightweight backend for an AI agent framework, using llama.cpp directly gives you full control without the overhead of a larger platform.

Private assistants and RAG pipelines: A single developer can run a local RAG pipeline on a laptop with llama.cpp serving the LLM and a vector database for retrieval. No cloud dependencies, no data leaving the machine.

Not a good fit for: High-throughput production APIs serving hundreds of requests per second. For that, use vLLM or SGLang. Also not ideal for training or fine-tuning (use Hugging Face Transformers or Axolotl).

The fastest path to a running model:

1. Install: brew install llama.cpp (macOS). For Linux, download a binary from GitHub releases or build from source with make. For Windows, use Docker or the winget package.

1. Get a model: Download a GGUF file from Hugging Face. For example, wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf.

1. Serve: llama-server -m llama-2-7b-chat.Q4_K_M.gguf --port 8080

1. Query: curl http://localhost:8080/v1/chat/completions -d '{"model":"llama","messages":[{"role":"user","content":"Hello"}]}'

You can also use llama-cli for interactive chat: llama-cli -m model.gguf -p "Once upon a time".

What you need: A quantized GGUF model file and enough RAM to hold it. No GPU required. If you have a GPU, the Metal, CUDA, or ROCm backend will be used automatically if compiled in.

Docs and community: The master repository at github.com/ggml-org/llama.cpp contains extensive docs under /docs. The community is active on GitHub Discussions and Discord. The llama-server changelog and libllama API changelog are maintained in separate issues linked from the README.

llama.cpp vs vLLM: vLLM is the throughput king for production GPU serving with PagedAttention and advanced scheduling. It requires NVIDIA GPUs with compute capability 7.0+ and optimized for high concurrency. Choose vLLM when you have a GPU cluster and need to serve many users. Choose llama.cpp when you have one GPU, a CPU, or Apple Silicon, and value portability over peak throughput.

llama.cpp vs Ollama: Ollama is a user-friendly wrapper around llama.cpp. It adds model management, a simpler CLI, and a polished experience. If you want a turnkey local server, use Ollama. If you need to embed inference into your own application, optimize backends, or control every detail, use llama.cpp directly.

llama.cpp vs LM Studio: LM Studio is a desktop GUI built on llama.cpp for macOS/Windows. Great for non-engineers. If you are an engineer scripting or serving in production, skip the GUI and use llama.cpp’s CLI or server.

When to avoid llama.cpp: You need flash attention, speculative decoding at scale, or advanced continuous batching optimizations found in vLLM or TensorRT-LLM. You are serving real-time applications with hundreds of concurrent users. You need training or fine-tuning capabilities. For those, use the appropriate specialized engine.