01 · A · Data

Tokenization and targets

Text becomes integers. Loss, perplexity, and “zero” are defined on this vocabulary, not on English words.

Raw text becomes integers

A tokenizer — almost always BPE, SentencePiece, or Unigram — cuts raw bytes into a finite vocabulary. The model never sees characters as characters. From this point on, every quantity that can go to zero is defined on those integers.

Byte-pair encoding (Sennrich, Haddow, and Birch, 2016) is the usual algorithm. Start with a vocabulary of bytes. Count adjacent pairs in the corpus. Merge the most frequent pair into a new token. Repeat until the vocabulary has size $V$ . Encoding a new string is the greedy application of those merges, left to right.

(a, b) \mapsto ab, (a, b) = ar g pairs max count (a, b)

(1.0)

$a, b$: adjacent tokens in the current segmentation
$V$: final vocabulary size: bytes plus the learned merges

Why it mattersThe tokenizer is not part of θ. It is a frozen preprocessing map. Changing it changes the units of every later number — loss, perplexity, the fitted floor E — which is why bits-per-byte is the fair comparison.

A training example is one sequence $x_{1 : T}$ . The supervision target at position $t$ is the next token $x_{t + 1}$ . Position $T$ either has no target, or sequences are packed so every position has a successor.

y_{t} \in {0, 1}^{V}, (y_{t})_{x_{t + 1}} = 1

(1.1)

$y_{t}$: one-hot target at position t
$x_{t + 1}$: the true next token in the document
$V$: vocabulary size, typically 32k–256k

Why it mattersLoss, perplexity, and “what zero loss would mean” live on this discrete vocabulary, not on English words. A larger vocabulary usually raises per-token loss even if the model is a better compressor.

The causal shift

In code this is a one-position shift: logits at index $t$ predict token $t + 1$ . The first token has no predecessor; the last logit is either unused or predicts the first token of the next packed span.

input x_{1 : T} ⟶ target x_{2 : T + 1}

(1.2)