Posts tagged "mixture-of-experts"
8 posts found
The Efficient Transformer Design Space: Comparing All Variants and the Three Futures of Attention
A unified view of every efficient attention mechanism — each is a different way of replacing the dense N×N attention matrix (sparsify it, factorize it, approximate the softmax kernel, recurse it, or pool one of its dimensions), every variant computed on a single 8-token example so the differences are visible side by side, ranked on the impossible triangle of training parallelism, low-cost inference, and strong quality, and ending with the three credible futures: hybrid retention, selective state spaces, and latent attention.
DeepSeek-V2 from Scratch: Multi-head Latent Attention and DeepSeekMoE
Building DeepSeek-V2's two core innovations from the ground up — low-rank KV joint compression, decoupled RoPE, matrix absorption, fine-grained expert segmentation, shared experts, and three-level load balancing — all derived step by step with concrete numbers.
Mathematical Prerequisites for Mixture of Experts — Part 3
Building the math foundations you need for understanding why MoEs work — orthogonality, vector norms, asymptotic notation, Lipschitz continuity, and dispatch entropy — all derived step by step with one consistent example.
Mixture of Experts from Scratch — Part 3: Why MoEs Work and the Modern Landscape (2022–2024)
Why experts specialize instead of collapsing, the role of nonlinearity, exploration and router learning, load balancing theory, and a complete taxonomy of modern MoE in LLMs — all derived from first principles with concrete examples.
Mixture of Experts from Scratch — Part 2: Scaling to Billions (2017–2022)
From thousands of experts to trillion-parameter models — sparse gating, top-k routing, load balancing, the Switch Transformer, and the engineering behind scaling MoEs — all derived step by step with a concrete 4-expert example.
Mathematical Prerequisites for Mixture of Experts — Part 2
Building the math foundations you need for sparse MoEs and the Switch Transformer — softplus, top-k masking, mean and variance, coefficient of variation, indicator functions, argmax, differentiability, and the dot-product loss — all derived step by step with one consistent example.
Mixture of Experts from Scratch — Part 1: The Foundations (1991–1993)
Building Mixture of Experts from the ground up — adaptive expert networks, gating functions, the mixture-of-Gaussians interpretation, hierarchical mixtures, and the EM algorithm — all derived step by step with a 2-expert regression example.
Mathematical Prerequisites for Mixture of Experts
Building the math foundations you need for Mixture of Experts — expected value, Gaussian densities, likelihood, Bayes' theorem, softmax, mixture models, conditional probability, multinomial distributions, and the Sherman-Morrison-Woodbury formula — all derived step by step with one consistent example.