https://sankethbk.github.io/blog/tags/representations/Recent content in Representations on Sanketh's BlogHugo -- 0.166.0en-usMon, 14 Sep 2026 00:00:00 +0000https://sankethbk.github.io/blog/posts/ml/2026-09-14-4-embeddings-from-one-hot-to-learned-representations/Mon, 14 Sep 2026 00:00:00 +0000https://sankethbk.github.io/blog/posts/ml/2026-09-14-4-embeddings-from-one-hot-to-learned-representations/<h1 id="embeddings-from-one-hot-vectors-to-learned-representations">Embeddings: From One-Hot Vectors to Learned Representations</h1> <p>Everything in the previous notes assumed the network&rsquo;s input was already a list of numbers — <code>x1 = 1, x2 = 2</code>, pixel intensities, whatever. But most interesting data is not numeric. &ldquo;cat&rdquo;, &ldquo;dog&rdquo;, &ldquo;bank&rdquo;, user IDs, product IDs, words of a sentence. Neural networks cannot multiply the string <code>&quot;cat&quot;</code> by a weight matrix. Somewhere between the raw symbol and the first linear layer, a translation to numbers must happen, and the way we do it — the <strong>embedding layer</strong> — turns out to be one of the most consequential ideas in modern deep learning.</p>