Posts tagged "mathematics"
15 posts found
Mathematical Prerequisites for Mamba
Building the math foundations you need for Mamba — first-order linear ODEs, the integrating factor method, the matrix exponential, the integral identity used in zero-order hold, the ZOH discretization itself, the unrolled time-varying recurrence, and the 1-semiseparable matrix view — all derived step by step with one consistent leaky-integrator example.
Mathematical Prerequisites for the Delta Rule
Building the foundations for the delta rule and gated attention — the Frobenius norm and squared reconstruction loss, the gradient of a quadratic loss with respect to a matrix state, online stochastic gradient descent as a one-step update, and the chain rule for gated identity paths — all derived step by step with one consistent two-dimensional key-value example.
Mathematical Prerequisites for Sparse and Sliding Window Attention
Building the discrete math foundations for sparse attention patterns — the triangular number formula for counting pairs, modular arithmetic for strided patterns, the pigeonhole principle for existence proofs, and set union with inclusion-exclusion for combining connectivity sets — all derived step by step with one consistent 8-house example.
Mathematical Prerequisites for DeepSeek-V2
Building the linear algebra foundations for Multi-head Latent Attention — matrix-vector multiplication, transpose, associativity, non-commutativity, the bilinear identity, linearity over sums, low-rank factorization, and the dot product of concatenated vectors — all derived step by step with one consistent 3-dimensional example.
Mathematical Prerequisites for the Attention Series
Building the math foundations for the attention series — the tanh function, dot products as similarity measures, the standard normal distribution, variance, independence of random variables, and why the variance of a dot product equals the vector dimension — all derived step by step with one consistent 2-dimensional example.
Mathematical Prerequisites for Mixture of Experts — Part 3
Building the math foundations you need for understanding why MoEs work — orthogonality, vector norms, asymptotic notation, Lipschitz continuity, and dispatch entropy — all derived step by step with one consistent example.
Mixture of Experts from Scratch — Part 3: Why MoEs Work and the Modern Landscape (2022–2024)
Why experts specialize instead of collapsing, the role of nonlinearity, exploration and router learning, load balancing theory, and a complete taxonomy of modern MoE in LLMs — all derived from first principles with concrete examples.
Mixture of Experts from Scratch — Part 2: Scaling to Billions (2017–2022)
From thousands of experts to trillion-parameter models — sparse gating, top-k routing, load balancing, the Switch Transformer, and the engineering behind scaling MoEs — all derived step by step with a concrete 4-expert example.
Mathematical Prerequisites for Mixture of Experts — Part 2
Building the math foundations you need for sparse MoEs and the Switch Transformer — softplus, top-k masking, mean and variance, coefficient of variation, indicator functions, argmax, differentiability, and the dot-product loss — all derived step by step with one consistent example.
Mixture of Experts from Scratch — Part 1: The Foundations (1991–1993)
Building Mixture of Experts from the ground up — adaptive expert networks, gating functions, the mixture-of-Gaussians interpretation, hierarchical mixtures, and the EM algorithm — all derived step by step with a 2-expert regression example.
Mathematical Prerequisites for Mixture of Experts
Building the math foundations you need for Mixture of Experts — expected value, Gaussian densities, likelihood, Bayes' theorem, softmax, mixture models, conditional probability, multinomial distributions, and the Sherman-Morrison-Woodbury formula — all derived step by step with one consistent example.
Mathematical Prerequisites for Foundation Prior
Building the math foundations for understanding how LLMs reshape Bayesian priors — parameters, likelihood, Beta distributions, KL divergence, entropy, exponential tilting, and marginal likelihood — all derived step by step with one coin example.
MaxRL: From REINFORCE to Maximum Likelihood
Why dividing by the number of successes instead of the batch size changes what your gradient estimator optimizes — and how this connects REINFORCE, maximum likelihood, and pass@k through one clean mathematical identity.
Reinforcement Learning from Scratch
Building RL from the ground up — actions, rewards, policies, expected reward, the policy gradient theorem, and REINFORCE — all derived step by step with concrete examples.
Mathematical Prerequisites for Reinforcement Learning
Building the math foundations you need for RL — probability, expected value, derivatives, the log trick, and Monte Carlo estimation — all through one consistent example.