03 · C · Block

The pre-norm Transformer block

Residual highways, RMSNorm, and the two sublayers that every modern decoder stacks.

Pre-norm residual block

Modern decoder LLMs stack $n$ identical pre-norm blocks. Each block writes a residual delta into the stream, twice: once from attention, once from the MLP.

Z^{(ℓ)} H^{(ℓ)} = H^{(ℓ - 1)} + Attn (Norm (H^{(ℓ - 1)})) = Z^{(ℓ)} + MLP (Norm (Z^{(ℓ)}))

(3.1)

$ℓ$: layer index, 1 … n
$H^{(ℓ)}$: residual stream after layer ℓ, T×d
$Z^{(ℓ)}$: stream after the attention residual

Why it mattersResiduals are the highway that lets gradients travel through dozens of layers (He et al., 2016). The skip has Jacobian I, so there is always a path that does not vanish.

RMSNorm

RMSNorm scales a vector by its root-mean-square and a learned gain (Zhang and Sennrich, 2019). It does not subtract the mean and has no bias. Llama-style models use it.

RMS (u) = \frac{1}{d} i = 1 \sum d u_{i}^{2} + ε

(3.2)

$u \in R^{d}$: one residual-stream vector
$ε$: small constant, e.g. 10^{-6}

RMSNorm (u) = γ ⊙ \frac{u}{RMS ( u )}

(3.3)

$γ \in R^{d}$: learned scale
$⊙$: elementwise multiply

Why it mattersNorms keep residual-stream scale stable so attention scores and MLP activations do not explode as depth grows.

LayerNorm

LayerNorm also subtracts the mean and usually has a bias. GPT-2 uses it. The extra projection orthogonal to $1$ is the only structural difference.

u^= \frac{u - μ 1}{σ ^{2} + ε}, LN (u) = γ ⊙ u^+ β

(3.4)

$μ$: mean of u
$σ^{2}$: variance of u
$β \in R^{d}$: learned bias