07 · E · Signal

Loss, entropy, and scaling

Cross-entropy is maximum likelihood. It decomposes into entropy plus KL. The floor is E, not zero.

Cross-entropy is maximum likelihood

The autoregressive model is a probability distribution over sequences. Maximizing the likelihood of the training documents is exactly minimizing next-token cross-entropy.

lo g p_{θ} (x_{1 : T}) = t = 1 \sum T lo g p_{θ} (x_{t} ∣ x_{< t}) = - t \sum L_{t - 1}

(7.0)

Why it mattersThere is no extra objective hiding behind the loss. Pretraining is MLE of the next-token categorical. Every later capability is a side effect of making that likelihood large.

Token-level cross-entropy

PyTorch CrossEntropyLoss on logits is exactly this, in nats.

L_{t} = - lo g p_{t} (x_{t + 1}) = - z_{t, x_{t + 1}} + lo g j = 1 \sum V e^{z_{t, j}}

(7.1)

$L_{t}$: NLL of the true next token, in nats

L (θ) = \frac{1}{∣ M ∣} t \in M \sum L_{t}

(7.2)

$M$: positions that are not padding / not masked

Why it mattersThis scalar is the only training signal in pretraining. Backprop exists solely to make L smaller.

Entropy plus KL

If $P^{⋆}$ is the true next-token distribution of the data, the expected loss splits into two terms (Cover and Thomas). Training can only shrink the second.

E [L] = H (P^{⋆}) + D_{KL} (P^{⋆} ∥ p_{θ})

(7.3)

$H (P^{⋆})$: entropy of language: irreducible uncertainty
$D_{KL}$: extra loss from the model being wrong

$H (P^{⋆}) > 0$ for open language, so $L \to 0$ is not achievable on a general corpus. Fitted irreducible floors on web-scale English tokenizers are often around $E \approx 1.7 - 1.8$ nats/token.

Perplexity and bits-per-byte

PPL = exp (L)

(7.4)

Loss 2.0 nats/token is perplexity $e^{2.0} \approx 7.39$ . Loss 1.69 is about 5.4. Read it as: “as uncertain as choosing among about 5–7 equally likely tokens,” on average.

Loss is not comparable across tokenizers. Bits-per-byte is the fairer comparison. If a document of $B$ bytes is tokenized into $T$ tokens:

BPB = \frac{L}{ln 2} \cdot \frac{T}{B}

(7.4b)

$L / ln 2$: loss in bits per token
$T / B$: tokens per byte for this tokenizer and document

Classic estimates of English entropy are roughly 0.6–1.3 bits per character (Shannon, 1951); frontier models are often cited around ~0.7 bits per character on diverse English — already near those old bounds on many corpora.

Scaling-law shape

This is a forecast of pretraining loss, not a training step. DeepMind’s Chinchilla fit (Hoffmann et al., 2022):

L (N, D) = E + \frac{A}{N ^{α}} + \frac{B}{D ^{β}}

(7.5)

$N$: parameter count
$D$: training tokens
$E$: estimated irreducible loss
$A N^{- α}$: penalty for finite model size
$B D^{- β}$: penalty for finite data

Their original fit put $E \approx 1.69$ , $A = 406.4$ , $B = 410.7$ , $α = 0.34$ , $β = 0.28$ . A later replication (Besiroglu, Erdil, Barnett, 2024) put $E$ closer to $1.82$ and a larger data coefficient. Those numbers are estimates for a data mix and a tokenizer, not a universal constant.

Training compute is, to a standard approximation (Kaplan et al., 2020),

C \approx 6 N D

(7.6)

$6$: ≈ 2 FLOPs/param/token for the forward pass, 4 for the backward

Minimize (7.5) along the budget line $D = C / (6 N)$ . The compute-optimal allocation is

N_{opt} (C) = G (\frac{C}{6})^{a}, D_{opt} (C) = G^{- 1} (\frac{C}{6})^{b}

(7.7)

$a = β / (α + β)$: ≈ 0.46 on the printed Chinchilla fit
$b = α / (α + β)$: ≈ 0.54
$G$: (\alpha A / \beta B)^{1/(\alpha+\beta)}

Why it mattersThe IsoFLOP experiments, which do not depend on the parametric constants, said something simpler: scale N and D equally, about twenty tokens per parameter. Chinchilla-70B was trained that way (70B × 1.4T). Kaplan had recommended many fewer tokens; that is the disagreement the floor does not care about.

Stage	Loss (nats)	Meaning
Random initialization	ln V ≈ 10.8–11.8	Uniform over 50k–128k tokens
Early training	~4–6	Crude word and syntax statistics
Small modern pretrain	~2.0–2.5	Competent next-token model
Strong large pretrain	~1.7–2.1	Near fitted irreducible floors
Narrow SFT, regular data	~0.2–0.8	The task is less uncertain than web text
Overfit a tiny set	→ 0	Memorization, not general language