Hybrid Architectures: RetNet and the Three Computation Paradigms
Building RetNet from scratch — how a single retention formula admits three computation modes (parallel for training, recurrent for inference, chunkwise for long sequences) producing identical outputs, why exponential decay and complex-exponential position encoding together fix the two failure modes of linear attention, and what the impossible triangle of training parallelism, low-cost inference, and strong performance actually requires.