RoPE Rotary Position Embedding
How most modern LLMs encode token position — rotation in vector space instead of an additive embedding.
Proposed by Jianlin Su et al. (2021), now standard in Llama, Mistral, GPT-NeoX, Qwen, DeepSeek. Instead of adding a positional embedding you multiply the query/key vectors by rotation matrices that encode position as an angle. Effect: relative positioning falls out naturally, and you can post-hoc "stretch" the context window with YaRN, NTK scaling, LongRoPE, etc.