DPO (Direct Preference Optimization)
A method (2023) for aligning an LLM to human preferences directly from pairs of "preferred/not-preferred" answers — without first training a separate reward model and running reinforcement learning. A simpler, more stable alternative to RLHF.
Background: RLHF alignment happens in several steps (train a reward model on preference data, then optimize the policy with PPO) — powerful but complex and unstable. DPO shows mathematically that you can skip the reward model and the RL step: you optimize the model directly on the preference pairs with a simple classification-like loss function. Result: comparable or better alignment, much simpler and more stable to train, fewer moving parts. It quickly became popular for open source models (Zephyr and others) precisely because it's so much easier to run than PPO-based RLHF. Part of the broader "preference tuning" family (alongside IPO, KTO, ORPO). Related to RLHF and instruction tuning.