llama.cpp
More info
- Creator
- Georgi Gerganov
- Released
- Type
- LLM inference engine (C++)
- License
- MIT
- Source
- github.com/ggerganov/llama.cpp
- Wikipedia
- en.wikipedia.org
Pure-C++ implementation of LLM inference — runs Llama, Mistral, Gemma and others on CPU, GPU (CUDA, Metal, Vulkan, ROCm) or a mix, without Python or PyTorch.
Georgi Gerganov's hobby project (March 2023) that became the foundation of the entire "local LLM" ecosystem. Defined the GGUF format (a self-describing quantised model file) and supports 1.5–8 bit quantisation, KV cache quantisation, speculative decoding, mmap loading.
Ollama, LM Studio, Jan and most local LLM UIs are built on top of llama.cpp. Beats vLLM/SGLang for single-user desktop; loses for batched server throughput.