Posts tagged "delta-rule"
2 posts found
Targeted Memory: The Delta Rule, Gated DeltaNet, and Kimi Delta Attention
Building targeted memory for linear attention from scratch — the delta rule (erase only the key being overwritten, derived as online gradient descent on a reconstruction loss), the gated delta rule (combine surgical erasure with a global forget gate), and Kimi Delta Attention (replace the scalar gate with a per-dimension one), why uniform decay alone cannot solve the memory-collision problem, and how Kimi Linear interleaves KDA with full attention to outperform pure full-attention models at 48B parameters.
Mathematical Prerequisites for the Delta Rule
Building the foundations for the delta rule and gated attention — the Frobenius norm and squared reconstruction loss, the gradient of a quadratic loss with respect to a matrix state, online stochastic gradient descent as a one-step update, and the chain rule for gated identity paths — all derived step by step with one consistent two-dimensional key-value example.