Speculative Decoding: Verifying Several Tokens in Parallel

Speculative Decoding: Verifying Several Tokens in Parallel The decoding-strategies note answered which token to choose from a next-token distribution. It did not remove the central inefficiency of generation: every accepted token normally requires another serial step through the large target model. KV caching makes each of those steps cheaper by reusing old keys and values. But the loop is still serial: target model → token 1 → target model → token 2 → target model → token 3 Speculative decoding adds a second, cheaper model called the draft model. The draft proposes several tokens serially. The large target model then examines the whole proposed suffix in one parallel forward pass and decides what to accept. ...

November 21, 2026 · 17 min