Top-k sampling
A sampling method where, at each step, the model may only choose among the k most likely tokens (e.g. the top 40), and the rest are excluded entirely. A simple predecessor to nucleus sampling (top-p).
Mechanic: sort all possible tokens by probability, keep the k highest, renormalize their probabilities, and sample among them. Win: it shuts out the long tail of unlikely tokens (which would otherwise occasionally give nonsense) but keeps variation among the reasonable ones. Weakness compared with top-p: k is fixed, which fits poorly at both extremes — when one token is nearly certain, top-k still lets in k-1 unlikely alternatives, and when many alternatives are equally good, k may be too small and prune good choices. That's why nucleus sampling (top-p), which adapts the number dynamically, is often preferred; they can also be combined. A fundamental sampling parameter, often exposed alongside temperature and top-p. Related to nucleus sampling and temperature.