GRPO Group Relative Policy Optimization
DeepSeek's RL algorithm for training reasoning models — cheaper than PPO, eliminates the value model by comparing groups of sampled answers.
Introduced in DeepSeek-Math (April 2024), popularised by DeepSeek-R1 (January 2025). PPO needs a separate value model (as big as the policy model) — GRPO samples N answers for the same prompt, computes relative advantage against the group mean, skips the value model. Saves ~50 % memory during training. Combined with rule-based rewards (verifiable maths/code answers) it enables training reasoning models without expensive human preference labelling.