Posts tagged "kernels"
2 posts found
The Kernel Zoo: Performers, Fast Weight Programmers, and the Capacity-Approximation Tradeoff in Linear Attention
How to choose the feature map φ in linear attention — the capacity bound that says a state of dimension d_φ stores at most d_φ orthogonal key-value pairs, the Performer's FAVOR+ that approximates softmax with positive orthogonal random features, the DPFP map that grows capacity deterministically through ReLU-gated quadrant projections, and the capacity-versus-approximation tradeoff that defines the design space, with the linear-attention recurrence reframed as a 1990s fast weight programmer.
Why Replace Attention? The Softmax Bottleneck and the Path to Linear Time
Why softmax is the bottleneck that every attention variant leaves untouched — and how rewriting attention as a kernel exposes an associativity trick that collapses the O(n²) cost to O(n) and turns the transformer into an RNN.