Targeted Memory: The Delta Rule, Gated DeltaNet, and Kimi Delta Attention
Building targeted memory for linear attention from scratch — the delta rule (erase only the key being overwritten, derived as online gradient descent on a reconstruction loss), the gated delta rule (combine surgical erasure with a global forget gate), and Kimi Delta Attention (replace the scalar gate with a per-dimension one), why uniform decay alone cannot solve the memory-collision problem, and how Kimi Linear interleaves KDA with full attention to outperform pure full-attention models at 48B parameters.