PEFT
Parameter-Efficient Fine-Tuning — the umbrella name for methods that adapt a large model by freezing the base weights and training only a small fraction of new parameters. It makes fine-tuning cheap enough to run on modest hardware. LoRA is the best-known PEFT method.
Idea: instead of updating all billions of weights (expensive in memory and compute), PEFT adds or adjusts only a fraction of the parameters — often <1% — and freezes the rest. The family includes LoRA/QLoRA (low-rank adapters), prefix tuning and prompt tuning (training "soft" prompt vectors), adapters (small inserted layers), and IA³. Win: drastically less memory, faster training, and you can have many small task-specific adapters for one shared base model (swap the adapter instead of the whole model). Hugging Face's peft library made it a standard tool. The dominant approach to practical fine-tuning today. Related to LoRA adapter and fine-tuning.