LoRA Low-Rank Adaptation
Low-Rank Adaptation — parameter-efficient fine-tuning. Microsoft paper, 2021. Freeze the base model, train only two small matrices per attention layer (rank r ~ 8-64). Trains millions of parameters instead of billions.
Insight: weight updates have low "intrinsic rank" — you don't need to change the whole weight matrix, just a low-rank approximation. Adapter size typically 10-100 MB instead of a full fine-tune at 10-100 GB. Hot-swappable: change LoRA mid-flight for different tasks. Quantized base + LoRA = QLoRA, can fine-tune 65 B on an RTX 4090. Stable Diffusion communities have spread the concept to image LoRAs (character, style). Standard in the Hugging Face PEFT library.