Blog
Breaking down research papers, deriving ideas from scratch, and making complex topics intuitive.
X-Token: Cross-Tokenizer Knowledge Distillation from Scratch
Building cross-tokenizer distillation from the ground up — why per-position KL breaks across tokenizers, DP span alignment, the chain-rule chunk merge, the projection matrix W, and two complementary losses (P-KL and H-KL) — with a full proof of GOLD's suppressive gradient and one running 2+3 example derived step by step.
DeepSeek-V4 Hybrid Attention: CSA and HCA from Scratch
Building the long-context attention of DeepSeek-V4 from the ground up — dual KV streams, overlapping softmax compression, the lightning indexer, top-$k$ sparse selection, and heavier compression — all derived step by step on a six-token example.
Prefill-as-a-Service: How KVCache Goes Cross-Datacenter
Deriving PrfaaS-PD from the ground up — why the KVCache transfer bandwidth wall confines PD disaggregation to a single RDMA island, how hybrid attention lowers the KV throughput enough to cross that wall, the selective offloading principle, the three-stage producer-consumer throughput model, the optimality conditions for routing threshold and prefill/decode ratio, and the dual-timescale scheduler — all derived step by step with one tiny two-cluster deployment.
The Efficient Transformer Design Space: Comparing All Variants and the Three Futures of Attention
A unified view of every efficient attention mechanism — each is a different way of replacing the dense N×N attention matrix (sparsify it, factorize it, approximate the softmax kernel, recurse it, or pool one of its dimensions), every variant computed on a single 8-token example so the differences are visible side by side, ranked on the impossible triangle of training parallelism, low-cost inference, and strong quality, and ending with the three credible futures: hybrid retention, selective state spaces, and latent attention.
Mamba and Mamba-2: Selective State Spaces and Structured State Space Duality
Building Mamba and Mamba-2 from scratch — why fixed-dynamics state space models cannot do content-based reasoning, how the selection mechanism fixes it, and the structured-matrix duality showing every selective SSM is masked attention in disguise.
Mathematical Prerequisites for Mamba
Building the math foundations you need for Mamba — first-order linear ODEs, the integrating factor method, the matrix exponential, the integral identity used in zero-order hold, the ZOH discretization itself, the unrolled time-varying recurrence, and the 1-semiseparable matrix view — all derived step by step with one consistent leaky-integrator example.
The Kernel Zoo: Performers, Fast Weight Programmers, and the Capacity-Approximation Tradeoff in Linear Attention
How to choose the feature map φ in linear attention — the capacity bound that says a state of dimension d_φ stores at most d_φ orthogonal key-value pairs, the Performer's FAVOR+ that approximates softmax with positive orthogonal random features, the DPFP map that grows capacity deterministically through ReLU-gated quadrant projections, and the capacity-versus-approximation tradeoff that defines the design space, with the linear-attention recurrence reframed as a 1990s fast weight programmer.
Targeted Memory: The Delta Rule, Gated DeltaNet, and Kimi Delta Attention
Building targeted memory for linear attention from scratch — the delta rule (erase only the key being overwritten, derived as online gradient descent on a reconstruction loss), the gated delta rule (combine surgical erasure with a global forget gate), and Kimi Delta Attention (replace the scalar gate with a per-dimension one), why uniform decay alone cannot solve the memory-collision problem, and how Kimi Linear interleaves KDA with full attention to outperform pure full-attention models at 48B parameters.
Hybrid Architectures: RetNet and the Three Computation Paradigms
Building RetNet from scratch — how a single retention formula admits three computation modes (parallel for training, recurrent for inference, chunkwise for long sequences) producing identical outputs, why exponential decay and complex-exponential position encoding together fix the two failure modes of linear attention, and what the impossible triangle of training parallelism, low-cost inference, and strong performance actually requires.
Why Replace Attention? The Softmax Bottleneck and the Path to Linear Time
Why softmax is the bottleneck that every attention variant leaves untouched — and how rewriting attention as a kernel exposes an associativity trick that collapses the O(n²) cost to O(n) and turns the transformer into an RNN.
Gated Attention: Replacing Residuals and ReLU with Learned Gates
Building gated transformer blocks from the ground up — why standard residual connections and ReLU activations leave performance on the table, identity map reordering (pre-norm), five gating variants from input gating to GRU-type gates, gated identity initialization, GLU and its variants (SwiGLU, GEGLU, ReGLU, Bilinear), the 2/3 parameter budget trick, and the unified view of gating as multiplicative control — all derived step by step with a 4-dimensional running example.
Mathematical Prerequisites for the Delta Rule
Building the foundations for the delta rule and gated attention — the Frobenius norm and squared reconstruction loss, the gradient of a quadratic loss with respect to a matrix state, online stochastic gradient descent as a one-step update, and the chain rule for gated identity paths — all derived step by step with one consistent two-dimensional key-value example.
DeepSeek Sparse Attention: Learned Token Selection from Scratch
Letting the model choose its own sparse pattern — a lightweight indexer scores every past token, top-k selects the keys that matter for this query, and two-stage training with KL alignment keeps the sparse model faithful to its dense teacher at $O(nk)$ cost.
Sliding Window Attention: From Local Windows to Global Context
Borrowing from convolutions — how a sliding local window keeps attention linear in sequence length, recovers a global receptive field by stacking layers, and reaches across the whole document through a small set of task-driven global tokens.
Why Full Attention Is Wasteful: Sparse Factorization from Scratch
Most learned attention weights are near zero, so why pay for all n² token pairs? Building sparse factorized attention from scratch — strided and fixed patterns that preserve full reachability in two hops while reducing the cost from O(n²) to O(n sqrt(n)).
Mathematical Prerequisites for Sparse and Sliding Window Attention
Building the discrete math foundations for sparse attention patterns — the triangular number formula for counting pairs, modular arithmetic for strided patterns, the pigeonhole principle for existence proofs, and set union with inclusion-exclusion for combining connectivity sets — all derived step by step with one consistent 8-house example.
DeepSeek-V2 from Scratch: Multi-head Latent Attention and DeepSeekMoE
Building DeepSeek-V2's two core innovations from the ground up — low-rank KV joint compression, decoupled RoPE, matrix absorption, fine-grained expert segmentation, shared experts, and three-level load balancing — all derived step by step with concrete numbers.
Mathematical Prerequisites for DeepSeek-V2
Building the linear algebra foundations for Multi-head Latent Attention — matrix-vector multiplication, transpose, associativity, non-commutativity, the bilinear identity, linearity over sums, low-rank factorization, and the dot product of concatenated vectors — all derived step by step with one consistent 3-dimensional example.
Grouped-Query Attention: Fewer KV Heads, Same Quality
Building GQA from the ground up — from multi-head attention to multi-query attention to grouped-query attention — showing exactly how sharing KV heads across query groups reduces the KV cache by a factor of h/g while preserving nearly all of MHA's quality. Every formula derived, every number verified.
The KV Bottleneck Explained: Why Inference Is Memory-Bound
Building from exact byte counts to the fundamental insight: autoregressive inference is bottlenecked not by arithmetic but by memory bandwidth from loading keys and values. Every KV cache optimization in the literature is a response to this single bottleneck — derived step by step with concrete numbers.
What Can We Actually Modify in Attention? A Taxonomy of Every Variant
Attention has five independent axes of variation: number of heads, KV representation, attention pattern, storage and caching, and layer-level architecture. Every variant in the literature is a modification of exactly one or two of these axes.
Why Vanilla Attention Breaks at Scale: The O(n²) Wall
A precise accounting of why standard attention becomes infeasible at long sequences: exact FLOP counts, memory costs, KV-cache growth, and the HBM bandwidth bottleneck that makes memory worse than compute.
From Soft Alignment to Queries, Keys, and Values: Deriving the Transformer's Attention
The Q/K/V abstraction derived from first principles — why Bahdanau's feedforward alignment collapses to a dot product, how the Transformer formalizes queries, keys, and values as separate projections, why we scale by √d_k (with a complete variance proof), what multi-head attention adds, and a full 3-token self-attention numerical walkthrough
What Attention is Really Doing: Weighted Memory Retrieval from Scratch
Building attention from first principles — the fixed-length bottleneck that broke RNN encoder–decoders on long sentences, the weighted-sum solution introduced by Bahdanau, Cho, and Bengio (2015), alignment score derivation, softmax normalization, and a complete 3-word numerical walkthrough — all derived step by step
Mathematical Prerequisites for the Attention Series
Building the math foundations for the attention series — the tanh function, dot products as similarity measures, the standard normal distribution, variance, independence of random variables, and why the variance of a dot product equals the vector dimension — all derived step by step with one consistent 2-dimensional example.
Mixture of Experts from Scratch — Part 3: Why MoEs Work and the Modern Landscape (2022–2024)
Why experts specialize instead of collapsing, the role of nonlinearity, exploration and router learning, load balancing theory, and a complete taxonomy of modern MoE in LLMs — all derived from first principles with concrete examples.
Mathematical Prerequisites for Mixture of Experts — Part 3
Building the math foundations you need for understanding why MoEs work — orthogonality, vector norms, asymptotic notation, Lipschitz continuity, and dispatch entropy — all derived step by step with one consistent example.
Attention Residuals: Replacing Fixed Skip Connections with Learned Depth-Wise Attention
Building Attention Residuals from scratch — why standard residuals dilute information, how softmax attention over depth fixes it, the block variant that makes it practical, and the structured-matrix view that unifies everything — all derived step by step with a 4-layer running example
Mixture of Experts from Scratch — Part 2: Scaling to Billions (2017–2022)
From thousands of experts to trillion-parameter models — sparse gating, top-k routing, load balancing, the Switch Transformer, and the engineering behind scaling MoEs — all derived step by step with a concrete 4-expert example.
Mathematical Prerequisites for Mixture of Experts — Part 2
Building the math foundations you need for sparse MoEs and the Switch Transformer — softplus, top-k masking, mean and variance, coefficient of variation, indicator functions, argmax, differentiability, and the dot-product loss — all derived step by step with one consistent example.
Mixture of Experts from Scratch — Part 1: The Foundations (1991–1993)
Building Mixture of Experts from the ground up — adaptive expert networks, gating functions, the mixture-of-Gaussians interpretation, hierarchical mixtures, and the EM algorithm — all derived step by step with a 2-expert regression example.
Mathematical Prerequisites for Mixture of Experts
Building the math foundations you need for Mixture of Experts — expected value, Gaussian densities, likelihood, Bayes' theorem, softmax, mixture models, conditional probability, multinomial distributions, and the Sherman-Morrison-Woodbury formula — all derived step by step with one consistent example.
Foundation Prior: How LLM Outputs Reshape Bayesian Beliefs
Deriving the Foundation Prior framework from scratch — why synthetic data is not real evidence, the constrained optimization that produces exponential tilting, the trust parameter λ, prompt heterogeneity through mixtures, calibration via real data, and the final posterior — all step by step with one coin example.
Mathematical Prerequisites for Foundation Prior
Building the math foundations for understanding how LLMs reshape Bayesian priors — parameters, likelihood, Beta distributions, KL divergence, entropy, exponential tilting, and marginal likelihood — all derived step by step with one coin example.
MaxRL: From REINFORCE to Maximum Likelihood
Why dividing by the number of successes instead of the batch size changes what your gradient estimator optimizes — and how this connects REINFORCE, maximum likelihood, and pass@k through one clean mathematical identity.
Reinforcement Learning from Scratch
Building RL from the ground up — actions, rewards, policies, expected reward, the policy gradient theorem, and REINFORCE — all derived step by step with concrete examples.
Mathematical Prerequisites for Reinforcement Learning
Building the math foundations you need for RL — probability, expected value, derivatives, the log trick, and Monte Carlo estimation — all through one consistent example.
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets (with the actual math)
From residual identity paths to Hyper-Connections and mHC — now with the paper's exact equations, fully unrolled products, and concrete numeric examples
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets
A deep dive into why residual connections work, how Hyper-Connections generalize them, and why constraining learned skip paths to doubly stochastic matrices solves the instability problem
Gradient Boosting: A Complete Guide
A deep dive into Gradient Boosting - from intuition and geometry to the math behind pseudo-residuals, stage-wise corrections, and practical implementation considerations.
Hello, World!
A quick introduction about me and what I do.