Speculative decoding
LLM inference technique: small draft model generates N tokens forward, large model verifies batch in parallel. Accept if matches, reject on divergence. 2-3x speedup without quality loss. DeepMind paper 2022, widespread deployment 2023+.
Underlying insight: LLM inference is memory-bound; a batch of 5 tokens costs only marginally more than 1. Draft model guesses; large model verifies 5-in-one. Hardware: GPUs are bad at sequential decoding (50 µs syscall per token); parallel-verify uses SIMD. Implementations: vLLM, TGI, llama.cpp, OpenAI's API. Hand-in-hand with KV cache + flash-attention for best-in-class inference throughput.