Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search

Decoding Strategies: Greedy, Temperature, Top-k, Top-p, and Beam Search The KV-cache note made the generation loop practical: prefill the prompt, keep each layer’s keys and values, and run only the newest token through the model at every decode step. But it deliberately skipped the final decision in that loop. After the final Transformer layer and language-model head, we have one logit per vocabulary token. Which token do we actually put into the answer? ...

November 21, 2026 · 16 min

Speculative Decoding: Verifying Several Tokens in Parallel

Speculative Decoding: Verifying Several Tokens in Parallel The decoding-strategies note answered which token to choose from a next-token distribution. It did not remove the central inefficiency of generation: every accepted token normally requires another serial step through the large target model. KV caching makes each of those steps cheaper by reusing old keys and values. But the loop is still serial: target model → token 1 → target model → token 2 → target model → token 3 Speculative decoding adds a second, cheaper model called the draft model. The draft proposes several tokens serially. The large target model then examines the whole proposed suffix in one parallel forward pass and decides what to accept. ...

November 21, 2026 · 17 min

KV Cache: Making Autoregressive Generation Practical

KV Cache: Making Autoregressive Generation Practical The language-model training note ended with an asymmetry: training processes every position in parallel, but generation produces one token at a time. Each new token is appended to the context, and the model is run again to predict the next one. If we implement that loop literally, we repeat an enormous amount of work. To generate token 100, we recompute the embeddings, queries, keys, values, attention outputs, and feedforward outputs for tokens 1 through 99 — even though we already computed all of them in earlier iterations. ...

November 10, 2026 · 20 min