Gated Attention: Replacing Residuals and ReLU with Learned Gates
Building gated transformer blocks from the ground up — why standard residual connections and ReLU activations leave performance on the table, identity map reordering (pre-norm), five gating variants from input gating to GRU-type gates, gated identity initialization, GLU and its variants (SwiGLU, GEGLU, ReGLU, Bilinear), the 2/3 parameter budget trick, and the unified view of gating as multiplicative control — all derived step by step with a 4-dimensional running example.