The Kernel Zoo: Performers, Fast Weight Programmers, and the Capacity-Approximation Tradeoff in Linear Attention
How to choose the feature map φ in linear attention — the capacity bound that says a state of dimension d_φ stores at most d_φ orthogonal key-value pairs, the Performer's FAVOR+ that approximates softmax with positive orthogonal random features, the DPFP map that grows capacity deterministically through ReLU-gated quadrant projections, and the capacity-versus-approximation tradeoff that defines the design space, with the linear-attention recurrence reframed as a 1990s fast weight programmer.