Quantization (LLM)
Storing a model's weights at lower numerical precision (e.g. 4 or 8 bits instead of 16) to reduce the memory need and speed up inference. It makes it possible to run large models on modest hardware — with a small quality loss.
Mechanic: the weights are mapped from floating point (FP16/BF16) to fewer bits (INT8, INT4, or formats like NF4). A 70B model that needs ~140 GB in FP16 can shrink to ~35 GB in 4-bit → fits on a single GPU. Win: lower memory, faster (less data to move), cheaper inference. Price: a certain accuracy loss that grows the more aggressively you quantize (4-bit is often a good balance; 2-bit starts to show clearly). Methods: GPTQ, AWQ (post-training, smart about which weights are sensitive), and the GGUF format for llama.cpp. Distinct from QLoRA (which quantizes for TRAINING); this is about inference. One of the most important techniques for running LLMs locally. Related to GGUF and QLoRA.