Posts tagged "mla"
2 posts found
The Efficient Transformer Design Space: Comparing All Variants and the Three Futures of Attention
A unified view of every efficient attention mechanism — each is a different way of replacing the dense N×N attention matrix (sparsify it, factorize it, approximate the softmax kernel, recurse it, or pool one of its dimensions), every variant computed on a single 8-token example so the differences are visible side by side, ranked on the impossible triangle of training parallelism, low-cost inference, and strong quality, and ending with the three credible futures: hybrid retention, selective state spaces, and latent attention.
DeepSeek-V2 from Scratch: Multi-head Latent Attention and DeepSeekMoE
Building DeepSeek-V2's two core innovations from the ground up — low-rank KV joint compression, decoupled RoPE, matrix absorption, fine-grained expert segmentation, shared experts, and three-level load balancing — all derived step by step with concrete numbers.