Posts tagged "sparse-attention"
5 posts found
DeepSeek-V4 Hybrid Attention: CSA and HCA from Scratch
Building the long-context attention of DeepSeek-V4 from the ground up — dual KV streams, overlapping softmax compression, the lightning indexer, top-$k$ sparse selection, and heavier compression — all derived step by step on a six-token example.
The Efficient Transformer Design Space: Comparing All Variants and the Three Futures of Attention
A unified view of every efficient attention mechanism — each is a different way of replacing the dense N×N attention matrix (sparsify it, factorize it, approximate the softmax kernel, recurse it, or pool one of its dimensions), every variant computed on a single 8-token example so the differences are visible side by side, ranked on the impossible triangle of training parallelism, low-cost inference, and strong quality, and ending with the three credible futures: hybrid retention, selective state spaces, and latent attention.
DeepSeek Sparse Attention: Learned Token Selection from Scratch
Letting the model choose its own sparse pattern — a lightweight indexer scores every past token, top-k selects the keys that matter for this query, and two-stage training with KL alignment keeps the sparse model faithful to its dense teacher at $O(nk)$ cost.
Sliding Window Attention: From Local Windows to Global Context
Borrowing from convolutions — how a sliding local window keeps attention linear in sequence length, recovers a global receptive field by stacking layers, and reaches across the whole document through a small set of task-driven global tokens.
Why Full Attention Is Wasteful: Sparse Factorization from Scratch
Most learned attention weights are near zero, so why pay for all n² token pairs? Building sparse factorized attention from scratch — strided and fixed patterns that preserve full reachability in two hops while reducing the cost from O(n²) to O(n sqrt(n)).