https://sankethbk.github.io/blog/tags/lstm/Recent content in Lstm on Sanketh's BlogHugo -- 0.169.0en-usTue, 27 Oct 2026 00:00:00 +0530https://sankethbk.github.io/blog/posts/ml/labs/2026-10-27-lab-2-3-gates-as-memory-engineering/Tue, 27 Oct 2026 00:00:00 +0530https://sankethbk.github.io/blog/posts/ml/labs/2026-10-27-lab-2-3-gates-as-memory-engineering/<h1 id="lab-23-gates-as-memory-engineering--lstm-from-scratch-on-a-task-that-was-supposed-to-kill-the-rnn">Lab 2.3: Gates as Memory Engineering — LSTM from scratch, on a task that was supposed to kill the RNN</h1> <p><strong>Language-model internals series.</strong> <a href="https://sankethbk.github.io/blog/posts/ml/labs/2026-10-27-lab-2-2-language-model-without-attention/">Lab 2.2</a> measured the diagnosis instead of reciting it: on a trained char-RNN, the forward echo of a character and the backward BPTT blame both halve every ~2.3 steps (per-step retention 0.74), because the transported term per step is (roughly) $W_{hh}^{\top}\mathrm{diag}(1-h^2)$ — every step multiplies your memory by something whose average magnitude is far below 1.</p>