PPO Proximal Policy Optimization
The standard RL algorithm for LLM training (the RLHF phase) — updates the policy "carefully" so it doesn't drift too far from the previous version.
John Schulman et al. at OpenAI (2017). Uses a "clipped surrogate objective" that constrains policy updates to a region around the old policy — avoids the training crashes A2C suffered. Default choice in InstructGPT/ChatGPT training, Claude training, Llama 2/3 training. Drawback: requires a separate value model (as large as the policy) ⇒ memory-hungry. GRPO (DeepSeek) eliminates the value model and became popular in 2024+. DPO is another simpler alternative.