Lab 2.2: A Language Model Without Attention — RNN, BPTT, and the Measured Death of Gradient Signal

Tue, 27 Oct 2026 00:00:00 +0530

Lab 2.2: A Language Model Without Attention — char-level RNN, backprop through time, and the measured death of gradient signal

Language-model internals series. Lab 2.1 gave words geometry without any sequence at all — windows don’t know order. This lab puts order back, the pre-2017 way: a recurrent network that reads one character at a time and carries everything it remembers in one vector.

The organizing question of this series, sharpened:

Lab 2.3: Gates as Memory Engineering — LSTM From Scratch on a Task That Was Supposed to Kill the RNN

Tue, 27 Oct 2026 00:00:00 +0530

Lab 2.3: Gates as Memory Engineering — LSTM from scratch, on a task that was supposed to kill the RNN

Language-model internals series. Lab 2.2 measured the diagnosis instead of reciting it: on a trained char-RNN, the forward echo of a character and the backward BPTT blame both halve every ~2.3 steps (per-step retention 0.74), because the transported term per step is (roughly) $W_{hh}^{\top}\mathrm{diag}(1-h^2)$ — every step multiplies your memory by something whose average magnitude is far below 1.

Bptt on Sanketh's Blog

Lab 2.2: A Language Model Without Attention — RNN, BPTT, and the Measured Death of Gradient Signal

Lab 2.2: A Language Model Without Attention — char-level RNN, backprop through time, and the measured death of gradient signal

Lab 2.3: Gates as Memory Engineering — LSTM From Scratch on a Task That Was Supposed to Kill the RNN

Lab 2.3: Gates as Memory Engineering — LSTM from scratch, on a task that was supposed to kill the RNN