00 · Object

The object of study

An LLM is a map from token IDs to logits. Training only ever sees next-token conditionals.

What is being built

An LLM, in the sense used throughout this treatise, is a single differentiable function from a sequence of discrete token IDs to a matrix of logits — one row per position, one column per vocabulary item.

f_{θ} : {1, \dots, V}^{T} \to R^{T \times V}

(0.1)

$V$: vocabulary size (number of distinct tokens)
$T$: sequence length of this forward pass
$d$: model / residual-stream width
$n$: number of Transformer layers
$θ$: every learnable number in the network
$x_{1 : T}$: token IDs, each x_t \in \{1,\dots,V\}

Why it mattersTraining never “understands a paragraph as a whole” as a primitive. It only learns next-token conditionals. Every later capability is a side effect of making those conditionals accurate.

The probabilistic model is autoregressive. The joint over a sequence factorizes into a product of next-token conditionals. There is no separate “sentence model.”

p_{θ} (x_{1 : T}) = t = 1 \prod T p_{θ} (x_{t} ∣ x_{< t})

(0.2)

$x_{< t}$: (x_1,\dots,x_{t-1}); empty when t = 1
$p_{θ} (x_{t} ∣ x_{< t})$: the model’s categorical over V tokens at step t

Two modes of the same network

The same function $f_{θ}$ is used in two incompatible ways. Training has a correct next token sitting in a dataset, so a loss exists, so a gradient exists, so $θ$ moves. Inference has no such token. The weights freeze. Only the forward map runs.

x₁:Ttokens

H⁽⁰⁾embed + RoPE

H⁽ⁿ⁾n blocks

Zlogits

p₁:Tsoftmax

Forward map. Training attaches CE → Jacobians → AdamW after p. Inference samples from p.

Notation that does not change

Every later chapter uses the same letters. Residuals are $H^{(ℓ)}$ with rows $h_{t}^{(ℓ)} \in R^{d}$ . Attention is the only mixer. Everything else is applied positionwise. The unembedding produces logits $Z \in R^{T \times V}$ , and softmax turns a row into $p_{t}$ .