Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search

Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search The KV-cache note made the generation loop practical: prefill the prompt, keep each layer’s keys and values, and run only the newest token through the model at every decode step. But it deliberately skipped the final decision in that loop. After the final Transformer layer and language-model head, we have one logit per vocabulary token. Which token do we actually put into the answer? ...

November 21, 2026 · 16 min