GGUF
A file format (successor to GGML) for packaging quantized language models in a single file, designed for llama.cpp and local running. The de facto standard for distributing models that hobbyists run on their own hardware.
Properties: a GGUF file contains the weights (at any quantization level, e.g. Q4_K_M, Q5_K_S, Q8_0), the architecture, and all metadata (tokenizer, hyperparameters) in one package → download one file and run. It supports running partly on CPU and partly on GPU ("offloading" layers), which lets even models larger than the GPU memory be run. The naming Q4_K_M etc. indicates the bits and quantization scheme (size vs quality trade-off). Ecosystem: Hugging Face full of GGUF conversions, run by llama.cpp, Ollama, LM Studio, Jan. Distinct from GPU-focused formats (safetensors + AWQ/GPTQ for vLLM). The backbone of the local LLM movement. Related to quantization and llama.cpp.