Language-Model

Lab 2.2: A Language Model Without Attention — char-level RNN, backprop through time, and the measured death of gradient signal Language-model internals series. Lab 2.1 gave words geometry without any sequence at all — windows don’t know order. This lab puts order back, the pre-2017 way: a recurrent network that reads one character at a time and carries everything it remembers in one vector. The organizing question of this series, sharpened: ...

Lab 2.3: Gates as Memory Engineering — LSTM from scratch, on a task that was supposed to kill the RNN Language-model internals series. Lab 2.2 measured the diagnosis instead of reciting it: on a trained char-RNN, the forward echo of a character and the backward BPTT blame both halve every ~2.3 steps (per-step retention 0.74), because the transported term per step is (roughly) $W_{hh}^{\top}\mathrm{diag}(1-h^2)$ — every step multiplies your memory by something whose average magnitude is far below 1. ...