10 · G · Step

AdamW, clip, and the schedule

Turning a gradient into a new θ without destroying a billion-parameter run.

Vanilla SGD is not what runs

θ \leftarrow θ - η \nabla_{θ} L

(10.1)

Raw gradients in a Transformer have wild scale differences across layers. LLMs use AdamW: per-parameter adaptive steps, decoupled weight decay, a warmup-cosine schedule, and global-norm clipping.

AdamW

Let $g_{t} = \nabla_{θ} L_{t}$ at optimizer step $t$ .

m_{t} v_{t} m^_{t} θ_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{⊙ 2} = \frac{m _{t}}{1 - β _{1}^{t}}, v^_{t} = \frac{v _{t}}{1 - β _{2}^{t}} = θ_{t - 1} - η_{t} (\frac{m ^ _{t}}{v ^ _{t} + ε} + λ θ_{t - 1})

(10.2)

$η_{t}$: scheduled learning rate
$β_{1}$: first-moment decay, typically 0.9
$β_{2}$: second-moment decay, 0.95 or 0.999
$λ$: weight decay, applied to θ directly (the “W”)
$ε$: 10^{-8}, avoid divide-by-zero
$g^{⊙ 2}$: elementwise square

Warmup then cosine decay

η_{t} = {η_{m a x} \cdot t / t_{warm} η_{m i n} + \frac{1}{2} (η_{m a x} - η_{m i n}) (1 + cos π \frac{t - t _{warm}}{t _{total} - t _{warm}}) t \leq t_{warm} t > t_{warm}

(10.3)

Gradient clipping

Before AdamW:

g \leftarrow g \cdot min (1, \frac{c}{∥ g ∥ _{2}})

(10.4)

$c$: clip threshold, often 1.0
$∥ g ∥_{2}$: global L2 norm of the concatenated gradient

Why it mattersAdam normalizes per-parameter. Weight decay stops ‖θ‖ drifting. Clipping stops rare batches from destroying the run. The schedule starts gentle, then anneals so the model settles.