GPQA
Graduate-Level Google-Proof Q&A — benchmark of 448 PhD-level questions in biology/physics/chemistry. Created in 2023 by David Rein et al. "Google-proof" = experts must consult several sources for 30+ min to solve.
Intended as a hard test for LLMs — MMLU and similar "high school" tests saturated quickly. The Diamond subset (198 questions) is extra hard. Humans: ~65% accuracy for domain experts, ~34% for non-experts. GPT-4: ~39%, Claude 3 Opus: ~50%, GPT-4o: ~56%, o1: 78%, GPT-5: 88%. When frontier models pass 90%, new benchmarks are needed (Humanity's Last Exam, FrontierMath). Diamond scores are what every "AI benchmark" press release cites in 2024-2025.