Posts tagged "rnn"
2 posts found
Hybrid Architectures: RetNet and the Three Computation Paradigms
Building RetNet from scratch — how a single retention formula admits three computation modes (parallel for training, recurrent for inference, chunkwise for long sequences) producing identical outputs, why exponential decay and complex-exponential position encoding together fix the two failure modes of linear attention, and what the impossible triangle of training parallelism, low-cost inference, and strong performance actually requires.
Why Replace Attention? The Softmax Bottleneck and the Path to Linear Time
Why softmax is the bottleneck that every attention variant leaves untouched — and how rewriting attention as a kernel exposes an associativity trick that collapses the O(n²) cost to O(n) and turns the transformer into an RNN.