05 · C · Compute

MLP and SwiGLU

Per-token computation and the wide matrices that store features.

Two-layer GELU FFN (GPT-2)

Attention moves information between positions. The MLP is the per-token computation: feature mixing, and a large fraction of stored knowledge in the wide matrices.

MLP (u) = W_{↓} ϕ (W_{↑} u + b_{↑}) + b_{↓}

(5.1)

$ϕ$: GELU
$W_{↑}$: up-projection, typically to 4d
$W_{↓}$: down-projection, back to d

GELU (z) = z Φ (z) \approx z σ (1.702 z)

(5.2)

$Φ$: standard Gaussian CDF; Hendrycks and Gimpel, 2016

SwiGLU (Llama)

Gated linear units (Shazeer, 2020) replace the single nonlinearity with a product of a SiLU-gated branch and a linear “up” branch. There is no bias in the usual Llama formulation. The up-projection is written $W_{up}$ so it is never confused with the unembedding $W_{U}$ .

SwiGLU (u) = W_{D}^{⊤} (SiLU (W_{G}^{⊤} u) ⊙ (W_{up}^{⊤} u))

(5.3)

$W_{G}, W_{up}$: gate and up, d × d_ff
$W_{D}$: down, d_ff × d
$d_{ff}$: inner width, often ~ 8d/3 so parameter count matches 4d GELU

SiLU (z) = z \cdot σ (z), σ (z) = \frac{1}{1 + e ^{- z}}

(5.4)

SiLU^{'} (z) = σ (z) + z σ (z) (1 - σ (z))

(5.5)

Why it mattersYou need this derivative in the backward pass. The extra σ(z)(1−σ(z)) term is the sigmoid Jacobian.