Posts tagged "deep-learning"
12 posts found
Kolibri Post-Training from Scratch: Effort Labels, Format Rewards and a Gradient-Norm Trust Region
Building Aleph Alpha's Kolibri post-training recipe from the ground up: soft reasoning-effort labels drawn from log-length percentiles, attention-cost ordering for packed SFT batches, segment-aware group advantages, the p-norm format multiplier, the B-TV trust region, a closed-form per-token bound on the logit gradient, targeted on-policy self-distillation, the three-bucket curriculum, and the Merlin–Arthur game against hallucination. Every piece is derived on one German baker problem and a group of four rollouts.
Kolibri Pre-Training from Scratch: UniBPE, Hybrid Attention and Exact Quantile Balancing
Building Aleph Alpha's Kolibri base model from the ground up: the Unigram-loss merge score of UniBPE and why it is pointwise mutual information in disguise, the KV-cache arithmetic of 40 sliding-window and 10 NoPE blocks, sigmoid MoE routing with Exact Quantile Balancing and Load-Error Injection, the 1 − K/E signature of a router the bias has overridden, and the cumulative-learning-rate law behind hyperparameter transfer. Every step is derived on one German sentence and four tokens.
MiMo-V2.6: Scaling Reinforcement Learning from Scratch
Building Xiaomi's MiMo-V2.6 RL recipe from the ground up — the importance-sampled GRPO objective, group-relative advantages and dynamic sampling, hack zeroing, Groupwise Reward Synthesis, Groupwise Advantage Redistribution, the group-relative length penalty, mass-conserving segment penalties, top-p candidate replay, router freezing, and the Little's-law Sample Mixer — all derived step by step on one four-rollout coding group.
X-Token: Cross-Tokenizer Knowledge Distillation from Scratch
Building cross-tokenizer distillation from the ground up — why per-position KL breaks across tokenizers, DP span alignment, the chain-rule chunk merge, the projection matrix W, and two complementary losses (P-KL and H-KL) — with a full proof of GOLD's suppressive gradient and one running 2+3 example derived step by step.
Mathematical Prerequisites for Mamba
Building the math foundations you need for Mamba — first-order linear ODEs, the integrating factor method, the matrix exponential, the integral identity used in zero-order hold, the ZOH discretization itself, the unrolled time-varying recurrence, and the 1-semiseparable matrix view — all derived step by step with one consistent leaky-integrator example.
From Soft Alignment to Queries, Keys, and Values: Deriving the Transformer's Attention
The Q/K/V abstraction derived from first principles — why Bahdanau's feedforward alignment collapses to a dot product, how the Transformer formalizes queries, keys, and values as separate projections, why we scale by √d_k (with a complete variance proof), what multi-head attention adds, and a full 3-token self-attention numerical walkthrough
What Attention is Really Doing: Weighted Memory Retrieval from Scratch
Building attention from first principles — the fixed-length bottleneck that broke RNN encoder–decoders on long sentences, the weighted-sum solution introduced by Bahdanau, Cho, and Bengio (2015), alignment score derivation, softmax normalization, and a complete 3-word numerical walkthrough — all derived step by step
Mathematical Prerequisites for the Attention Series
Building the math foundations for the attention series — the tanh function, dot products as similarity measures, the standard normal distribution, variance, independence of random variables, and why the variance of a dot product equals the vector dimension — all derived step by step with one consistent 2-dimensional example.
Mixture of Experts from Scratch — Part 3: Why MoEs Work and the Modern Landscape (2022–2024)
Why experts specialize instead of collapsing, the role of nonlinearity, exploration and router learning, load balancing theory, and a complete taxonomy of modern MoE in LLMs — all derived from first principles with concrete examples.
Attention Residuals: Replacing Fixed Skip Connections with Learned Depth-Wise Attention
Building Attention Residuals from scratch — why standard residuals dilute information, how softmax attention over depth fixes it, the block variant that makes it practical, and the structured-matrix view that unifies everything — all derived step by step with a 4-layer running example
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets (with the actual math)
From residual identity paths to Hyper-Connections and mHC — now with the paper's exact equations, fully unrolled products, and concrete numeric examples
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets
A deep dive into why residual connections work, how Hyper-Connections generalize them, and why constraining learned skip paths to doubly stochastic matrices solves the instability problem