Posts tagged "linear-algebra"
4 posts found
Mathematical Prerequisites for DeepSeek-V2
Building the linear algebra foundations for Multi-head Latent Attention — matrix-vector multiplication, transpose, associativity, non-commutativity, the bilinear identity, linearity over sums, low-rank factorization, and the dot product of concatenated vectors — all derived step by step with one consistent 3-dimensional example.
Attention Residuals: Replacing Fixed Skip Connections with Learned Depth-Wise Attention
Building Attention Residuals from scratch — why standard residuals dilute information, how softmax attention over depth fixes it, the block variant that makes it practical, and the structured-matrix view that unifies everything — all derived step by step with a 4-layer running example
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets (with the actual math)
From residual identity paths to Hyper-Connections and mHC — now with the paper's exact equations, fully unrolled products, and concrete numeric examples
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets
A deep dive into why residual connections work, how Hyper-Connections generalize them, and why constraining learned skip paths to doubly stochastic matrices solves the instability problem