QLoRA
Quantized LoRA — a technique (2023) that makes it possible to fine-tune very large models on a single consumer GPU by keeping the base model in 4-bit quantization and training only small LoRA adapters on top. It democratized fine-tuning.
Mechanic: the base model's frozen weights are stored in a compact 4-bit format (NF4, "NormalFloat"), which drastically lowers the memory need; during training the weights are dequantized on the fly for computation, but the gradients only update the small LoRA adapters (not the base model). Result: you can fine-tune a 65B model on a single 48 GB GPU — something that previously required a cluster. Innovations in the paper: 4-bit NormalFloat, double quantization, and paged optimizers. Trade-off: a small quality loss from the quantization, but near full-precision results in practice. It built on LoRA and made fine-tuning accessible to hobbyists and small teams. Related to LoRA adapter and PEFT.