Medusa

Classic speculative decoding needs a smaller model to produce proposals, which means two models to train, deploy and keep in memory. Medusa, published in 2024, instead mounts a handful of lightweight prediction heads on top of the frozen main model's final layer. Each head guesses the position two, three or four steps ahead, and the combinations form a tree of candidate continuations verified in a single forward pass using a purpose-built attention mask. Whatever matches what the model would have produced anyway is kept and the rest discarded — so the output is identical to ordinary decoding. Typical speedups land around two to three times, at the cost of a few extra parameters.