RMSNorm
Simplification of LayerNorm — normalise by the root mean square only, skip the mean subtraction. Slightly faster, equal or better quality.
Zhang & Sennrich, 2019. x / sqrt(mean(x²) + ε) · g, where g is a learnable scaling vector. No mean computation, no centring step, less fp instability under bf16/fp16.
The default in modern LLM architectures: Llama 2/3, Mistral, Qwen, Gemma, DeepSeek. When pre-norm transformers are built today, RMSNorm is default — LayerNorm has started to look old-fashioned.