Posts tagged "mamba"
3 posts found
The Efficient Transformer Design Space: Comparing All Variants and the Three Futures of Attention
A unified view of every efficient attention mechanism — each is a different way of replacing the dense N×N attention matrix (sparsify it, factorize it, approximate the softmax kernel, recurse it, or pool one of its dimensions), every variant computed on a single 8-token example so the differences are visible side by side, ranked on the impossible triangle of training parallelism, low-cost inference, and strong quality, and ending with the three credible futures: hybrid retention, selective state spaces, and latent attention.
Mamba and Mamba-2: Selective State Spaces and Structured State Space Duality
Building Mamba and Mamba-2 from scratch — why fixed-dynamics state space models cannot do content-based reasoning, how the selection mechanism fixes it, and the structured-matrix duality showing every selective SSM is masked attention in disguise.
Mathematical Prerequisites for Mamba
Building the math foundations you need for Mamba — first-order linear ODEs, the integrating factor method, the matrix exponential, the integral identity used in zero-order hold, the ZOH discretization itself, the unrolled time-varying recurrence, and the 1-semiseparable matrix view — all derived step by step with one consistent leaky-integrator example.