<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Speculative-Decoding on Sanketh's Blog</title><link>https://sankethbk.github.io/blog/tags/speculative-decoding/</link><description>Recent content in Speculative-Decoding on Sanketh's Blog</description><generator>Hugo -- 0.170.1</generator><language>en-us</language><lastBuildDate>Sat, 21 Nov 2026 00:00:00 +0530</lastBuildDate><atom:link href="https://sankethbk.github.io/blog/tags/speculative-decoding/index.xml" rel="self" type="application/rss+xml"/><item><title>Speculative Decoding: Verifying Several Tokens in Parallel</title><link>https://sankethbk.github.io/blog/posts/ml/2026-11-21-6-speculative-decoding-verifying-several-tokens-in-parallel/</link><pubDate>Sat, 21 Nov 2026 00:00:00 +0530</pubDate><guid>https://sankethbk.github.io/blog/posts/ml/2026-11-21-6-speculative-decoding-verifying-several-tokens-in-parallel/</guid><description>&lt;h1 id="speculative-decoding-verifying-several-tokens-in-parallel"&gt;Speculative Decoding: Verifying Several Tokens in Parallel&lt;/h1&gt;
&lt;p&gt;The &lt;a href="https://sankethbk.github.io/blog/posts/ml/2026-11-21-5-decoding-strategies-from-logits-to-a-generated-answer"&gt;decoding-strategies note&lt;/a&gt; answered which token to choose from a next-token distribution. It did not remove the central inefficiency of generation: every accepted token normally requires another serial step through the large target model.&lt;/p&gt;
&lt;p&gt;KV caching makes each of those steps cheaper by reusing old keys and values. But the loop is still serial:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span style="display:flex;"&gt;&lt;span&gt;target model → token 1 → target model → token 2 → target model → token 3
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;strong&gt;Speculative decoding&lt;/strong&gt; adds a second, cheaper model called the &lt;strong&gt;draft model&lt;/strong&gt;. The draft proposes several tokens serially. The large target model then examines the whole proposed suffix in one parallel forward pass and decides what to accept.&lt;/p&gt;</description></item></channel></rss>