Posts tagged "architecture"
4 posts found
What Can We Actually Modify in Attention? A Taxonomy of Every Variant
Attention has five independent axes of variation: number of heads, KV representation, attention pattern, storage and caching, and layer-level architecture. Every variant in the literature is a modification of exactly one or two of these axes.
Attention Residuals: Replacing Fixed Skip Connections with Learned Depth-Wise Attention
Building Attention Residuals from scratch — why standard residuals dilute information, how softmax attention over depth fixes it, the block variant that makes it practical, and the structured-matrix view that unifies everything — all derived step by step with a 4-layer running example
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets (with the actual math)
From residual identity paths to Hyper-Connections and mHC — now with the paper's exact equations, fully unrolled products, and concrete numeric examples
Manifold-Constrained Hyper-Connections: Stabilizing Deep Networks Beyond ResNets
A deep dive into why residual connections work, how Hyper-Connections generalize them, and why constraining learned skip paths to doubly stochastic matrices solves the instability problem