Posts tagged "mixture-of-experts"
10 posts found
Kolibri Pre-Training from Scratch: UniBPE, Hybrid Attention and Exact Quantile Balancing
Building Aleph Alpha's Kolibri base model from the ground up: the Unigram-loss merge score of UniBPE and why it is pointwise mutual information in disguise, the KV-cache arithmetic of 40 sliding-window and 10 NoPE blocks, sigmoid MoE routing with Exact Quantile Balancing and Load-Error Injection, the 1 − K/E signature of a router the bias has overridden, and the cumulative-learning-rate law behind hyperparameter transfer. Every step is derived on one German sentence and four tokens.
MiMo-V2.6: Scaling Reinforcement Learning from Scratch
Building Xiaomi's MiMo-V2.6 RL recipe from the ground up — the importance-sampled GRPO objective, group-relative advantages and dynamic sampling, hack zeroing, Groupwise Reward Synthesis, Groupwise Advantage Redistribution, the group-relative length penalty, mass-conserving segment penalties, top-p candidate replay, router freezing, and the Little's-law Sample Mixer — all derived step by step on one four-rollout coding group.
The Efficient Transformer Design Space: Comparing All Variants and the Three Futures of Attention
A unified view of every efficient attention mechanism — each is a different way of replacing the dense N×N attention matrix (sparsify it, factorize it, approximate the softmax kernel, recurse it, or pool one of its dimensions), every variant computed on a single 8-token example so the differences are visible side by side, ranked on the impossible triangle of training parallelism, low-cost inference, and strong quality, and ending with the three credible futures: hybrid retention, selective state spaces, and latent attention.
DeepSeek-V2 from Scratch: Multi-head Latent Attention and DeepSeekMoE
Building DeepSeek-V2's two core innovations from the ground up — low-rank KV joint compression, decoupled RoPE, matrix absorption, fine-grained expert segmentation, shared experts, and three-level load balancing — all derived step by step with concrete numbers.
Mathematical Prerequisites for Mixture of Experts — Part 3
Building the math foundations you need for understanding why MoEs work — orthogonality, vector norms, asymptotic notation, Lipschitz continuity, and dispatch entropy — all derived step by step with one consistent example.
Mixture of Experts from Scratch — Part 3: Why MoEs Work and the Modern Landscape (2022–2024)
Why experts specialize instead of collapsing, the role of nonlinearity, exploration and router learning, load balancing theory, and a complete taxonomy of modern MoE in LLMs — all derived from first principles with concrete examples.
Mixture of Experts from Scratch — Part 2: Scaling to Billions (2017–2022)
From thousands of experts to trillion-parameter models — sparse gating, top-k routing, load balancing, the Switch Transformer, and the engineering behind scaling MoEs — all derived step by step with a concrete 4-expert example.
Mathematical Prerequisites for Mixture of Experts — Part 2
Building the math foundations you need for sparse MoEs and the Switch Transformer — softplus, top-k masking, mean and variance, coefficient of variation, indicator functions, argmax, differentiability, and the dot-product loss — all derived step by step with one consistent example.
Mixture of Experts from Scratch — Part 1: The Foundations (1991–1993)
Building Mixture of Experts from the ground up — adaptive expert networks, gating functions, the mixture-of-Gaussians interpretation, hierarchical mixtures, and the EM algorithm — all derived step by step with a 2-expert regression example.
Mathematical Prerequisites for Mixture of Experts
Building the math foundations you need for Mixture of Experts — expected value, Gaussian densities, likelihood, Bayes' theorem, softmax, mixture models, conditional probability, multinomial distributions, and the Sherman-Morrison-Woodbury formula — all derived step by step with one consistent example.