06 · D · Readout

Unembedding to probabilities

A final norm, a linear map into V dimensions, and a categorical distribution per position.

Final norm and unembedding

After $n$ blocks we have $H^{(n)} \in R^{T \times d}$ . A last norm, then a linear map into the vocabulary, produces logits.

H ~ = Norm (H^{(n)}), Z = H ~ W_{U} \in R^{T \times V}

(6.1)

$W_{U}$: unembedding matrix, d × V — never the SwiGLU up-projection
$z_{t}$: row t of Z: logits at position t

Some models tie $W_{U} = W_{E}^{⊤}$ (Press and Wolf, 2017; Inan, Khosravi, and Socher, 2017). Tied embeddings save parameters and couple the geometry of “looking up a token” with “predicting a token.” Untied is the Llama default.

From logits to a categorical

p_{t} = softmax (z_{t}), (p_{t})_{k} = \frac{e ^{z_{t, k}}}{\sum _{j = 1}^{V} e ^{z_{t, j}}}

(6.2)

$p_{t}$: a probability distribution over the vocabulary
$p_{t} (x_{t + 1})$: probability assigned to the true next token

Why it mattersThis is the model’s entire prediction: one categorical distribution per position. Everything before this was geometry in R^d. This is the interface to discrete text.