An LLM, in the sense used throughout this treatise, is a single differentiable function from a sequence of discrete token IDs to a matrix of logits — one row per position, one column per vocabulary item.

The probabilistic model is autoregressive. The joint over a sequence factorizes into a product of next-token conditionals. There is no separate “sentence model.”

The same function $f_{\theta }$ is used in two incompatible ways. Training has a correct next token sitting in a dataset, so a loss exists, so a gradient exists, so $\theta$ moves. Inference has no such token. The weights freeze. Only the forward map runs.

Every later chapter uses the same letters. Residuals are $H^{(\ell )}$ with rows $h_t^{(\ell )}\in {\mathbb{R}}^d$. Attention is the only mixer. Everything else is applied positionwise. The unembedding produces logits $Z\in {\mathbb{R}}^{T\times V}$, and softmax turns a row into $p_t$.

A tokenizer — almost always BPE, SentencePiece, or Unigram — cuts raw bytes into a finite vocabulary. The model never sees characters as characters. From this point on, every quantity that can go to zero is defined on those integers.

Byte-pair encoding (Sennrich, Haddow, and Birch, 2016) is the usual algorithm. Start with a vocabulary of bytes. Count adjacent pairs in the corpus. Merge the most frequent pair into a new token. Repeat until the vocabulary has size $V$. Encoding a new string is the greedy application of those merges, left to right.

A training example is one sequence $x_{1:T}$. The supervision target at position $t$ is the next token $x_{t+1}$. Position $T$ either has no target, or sequences are packed so every position has a successor.

In code this is a one-position shift: logits at index $t$ predict token $t+1$. The first token has no predecessor; the last logit is either unused or predicts the first token of the next packed span.

A learned matrix $W_E\in {\mathbb{R}}^{V\times d}$ stores one row per token. Looking up token $x_t$ is a gather; equivalently, a one-hot times $W_E$.

Without a position signal the Transformer is a bag of tokens. Absolute embeddings add a learned vector per index.

Rotary position embeddings (Su et al., 2021) add no extra vector to the residual stream. After Q and K are formed, each even/odd pair of coordinates is rotated by an angle that depends on position. Inner products then depend on the difference $t-s$, not on absolute $t$ and $s$ separately.

Write $R_t$ for the block-diagonal matrix of all pair rotations at position $t$. Rotations compose, and $R_t^{\top }=R_{-t}$:

Modern decoder LLMs stack $n$ identical pre-norm blocks. Each block writes a residual delta into the stream, twice: once from attention, once from the MLP.

RMSNorm scales a vector by its root-mean-square and a learned gain (Zhang and Sennrich, 2019). It does not subtract the mean and has no bias. Llama-style models use it.

LayerNorm also subtracts the mean and usually has a bias. GPT-2 uses it. The extra projection orthogonal to $1$ is the only structural difference.

From normalized input $U\in {\mathbb{R}}^{T\times d}$, three linear maps produce Q, K, V. They are then split into $H$ heads of width $d_h=d/H$. If RoPE is used, Q and K are rotated now.

Future tokens must be invisible. The causal mask writes $-\infty$ above the diagonal so those entries become zero after softmax.

For one row $a=\mathrm{softmax}(z)\in {\mathbb{R}}^T$:

Multi-head attention uses $H_q=H_{kv}$ query and key/value heads. Multi-query (Shazeer, 2019) uses one KV head. Grouped-query (Ainslie et al., 2023) sits in between: Llama-3-class models share each KV head across a group of query heads. The score math is the same; the cache is smaller.

Attention moves information between positions. The MLP is the per-token computation: feature mixing, and a large fraction of stored knowledge in the wide matrices.

Gated linear units (Shazeer, 2020) replace the single nonlinearity with a product of a SiLU-gated branch and a linear “up” branch. There is no bias in the usual Llama formulation. The up-projection is written $W_{\mathrm{up}}$ so it is never confused with the unembedding $W_U$.

After $n$ blocks we have $H^{(n)}\in {\mathbb{R}}^{T\times d}$. A last norm, then a linear map into the vocabulary, produces logits.

Some models tie $W_U=W_E^{\top }$ (Press and Wolf, 2017; Inan, Khosravi, and Socher, 2017). Tied embeddings save parameters and couple the geometry of “looking up a token” with “predicting a token.” Untied is the Llama default.

The autoregressive model is a probability distribution over sequences. Maximizing the likelihood of the training documents is exactly minimizing next-token cross-entropy.

PyTorch CrossEntropyLoss on logits is exactly this, in nats.

If $P^{\star }$ is the true next-token distribution of the data, the expected loss splits into two terms (Cover and Thomas). Training can only shrink the second.

$H(P^{\star })>0$ for open language, so $\mathcal{L}\to 0$ is not achievable on a general corpus. Fitted irreducible floors on web-scale English tokenizers are often around $E\approx 1.7\text{–}1.8$ nats/token.

Loss 2.0 nats/token is perplexity $e^{2.0}\approx 7.39$. Loss 1.69 is about 5.4. Read it as: “as uncertain as choosing among about 5–7 equally likely tokens,” on average.

Loss is not comparable across tokenizers. Bits-per-byte is the fairer comparison. If a document of $B$ bytes is tokenized into $T$ tokens:

Classic estimates of English entropy are roughly 0.6–1.3 bits per character (Shannon, 1951); frontier models are often cited around ~0.7 bits per character on diverse English — already near those old bounds on many corpora.

This is a forecast of pretraining loss, not a training step. DeepMind’s Chinchilla fit (Hoffmann et al., 2022):

Their original fit put $E\approx 1.69$, $A=406.4$, $B=410.7$, $\alpha =0.34$, $\beta =0.28$. A later replication (Besiroglu, Erdil, Barnett, 2024) put $E$ closer to $1.82$ and a larger data coefficient. Those numbers are estimates for a data mix and a tokenizer, not a universal constant.

Training compute is, to a standard approximation (Kaplan et al., 2020),

Minimize (7.5) along the budget line $D=C/(6N)$. The compute-optimal allocation is

Conditional entropy falls as context grows. That is not controversial.

If the prompt is only “After tomorrow is”, the calendar is unpinned. Supply “Today is Sunday” and Tuesday is the unique calendar answer. Ambiguity was unfinished context. Semantic collapse is real.

What does not follow is: therefore the next token becomes unique. All of these can be true on Sunday:

• After tomorrow is Tuesday.
• After tomorrow’s Tuesday.
• After tomorrow it will be Tuesday.
• After tomorrow is the 16th.
• After tomorrow? Tuesday.

Same fact. Different tokens. The training objective is not “recover the unique true proposition.” It is “assign probability to the exact next token that happened to be in this document.”

Semantic collapse. Given enough context, the intended meaning often becomes unique. “After tomorrow,” plus “today is Sunday,” plus “we are talking about weekdays,” leaves one date.

Realizational leftover. How that meaning gets worded, whether the speaker continues the sentence at all, what they mention next, in what register — that usually stays open.

A Born-rule picture is a fair analogy for sense: many readings exist while the sentence is incomplete; context makes one reading definite. It is a weaker analogy for the next token, because the “measurement” here is “whatever this particular writer typed,” and many typings are compatible with the same collapsed meaning.

A huge fraction of remaining loss is missing context, not linguistic randomness. The data-generating process is many speakers, not one oracle with one lawful continuation. The model is not given the full world: it gets a window of text, not “it is Sunday and the speaker intends to name the weekday.”

Shannon already measured this with humans (1951). People who had seen the preceding text still could not guess the next character with certainty. Human conditional entropy of English is low, not zero. An LLM is trying to match that same process.

Training is gradient descent on $\mathcal{L}(\theta )$. Backprop is reverse-mode automatic differentiation: one backward sweep computes ${\nabla }_{\theta }\mathcal{L}$. If $u\in {\mathbb{R}}^m$, $v=g(u)\in {\mathbb{R}}^k$, and $\mathcal{L}$ is scalar:

In practice nobody materializes giant Jacobians. Frameworks compute VJPs. For a matrix multiply $Y=XW$, with incoming gradient $G=\partial \mathcal{L}/\partial Y$:

This is the most important gradient in the whole system. Start from the definitions:

The upstream vector is $(\partial \mathcal{L}/\partial p{)}_k=-y_k/p_k$. One-hot at $c$ makes that $-e_c/p_c$. The VJP through softmax is

Here $p^{\top }v=p_c\cdot (-1/p_c)=-1$ and $p_j{v}_j=-1[j=c]$, so

Then the final Norm Jacobian maps $\partial \mathcal{L}/\partial \tilde{H}$ back to $\partial \mathcal{L}/\partial H^{(n)}$.

Let $r=\mathrm{RMS}(u)$, $\hat{u}=u/r$, $v=\gamma \odot \hat{u}$.

Then $\partial \mathcal{L}/\partial u$ follows by the product rule with $\gamma$. LayerNorm is the same idea after projecting orthogonal to $1$ (mean removal).

Fix one head. Forward: $S=Q{K}^{\top }/\sqrt{d_h}$, $A=\mathrm{softmax}(S+M)$, $O=AV$. Given $G_O=\partial \mathcal{L}/\partial O$:

RoPE’s backward pass is the inverse rotation. Then $Q=U{W}_Q$ etc. give

The same for K, V. Multi-head: sum the $\partial \mathcal{L}/\partial U$ contributions.

Let $g=W_G^{\top }u$, $\mathrm{up}=W_{\mathrm{up}}^{\top }u$, $s=\mathrm{SiLU}(g)$, $h=s\odot \mathrm{up}$, $\mathrm{out}=W_D^{\top }h$.

Then ordinary matmul rules for $W_G,W_{\mathrm{up}},W_D,u$.

$\partial \mathcal{L}/\partial H^{(0)}$ is scattered back into the rows of $W_E$ (and $W_P$ if used):

Raw gradients in a Transformer have wild scale differences across layers. LLMs use AdamW: per-parameter adaptive steps, decoupled weight decay, a warmup-cosine schedule, and global-norm clipping.

Let $g_t={\nabla }_{\theta }{\mathcal{L}}_t$ at optimizer step $t$.

Before AdamW:

Repeat millions of steps:

1. Sample a batch of sequences. Shape: $B\times T$ token IDs.
2. Forward: embeddings → $n$ blocks → logits $Z\in {\mathbb{R}}^{B\times T\times V}$.
3. Compute $\mathcal{L}$ on next-token targets (ignore pad / masked positions).
4. Backward: reverse-mode AD gives $g={\nabla }_{\theta }\mathcal{L}$.
5. Optionally accumulate g over several micro-batches to fake a larger batch.
6. Clip g, AdamW update, zero grads.
7. Periodically evaluate validation loss on held-out data (no parameter update).

Embeddings and most projections start as small Gaussians. Residual-output projections are often scaled so the untrained skip path dominates at depth (GPT-2):

Batch loss is the mean of per-token losses. Mixed precision (BF16/FP16 forward, FP32 master weights and loss) is an implementation detail; the math above is the same. The loss itself is computed in FP32 so the log-sum-exp is not wrecked by underflow.

Training loss can keep falling by memorization. Validation loss estimates generalization. The entropy floor applies to both; validation is the one that tells you the model is not just reciting.

Once training is finished, $\theta$ is frozen. There is no “correct” next token sitting in a dataset, so there is nothing to compute a loss on and therefore nothing to back-propagate. Autoregressive generation:

1. Start with prompt tokens $x_{1:t}$.
2. Forward → logits $z_t$.
3. Form a sampling distribution from $z_t$.
4. Draw $x_{t+1}$, append, repeat.

Top-k. Keep the $k$ largest logits, set the rest to $-\infty$, then softmax.

Nucleus / top-p (Holtzman et al., 2020). Keep the smallest set of tokens whose cumulative probability is at least $p$, then renormalize.

No $\mathcal{L}$. No Jacobian. No Adam.

Attention at step $t$ only needs $q_t$ and all past $k_{1:t},v_{1:t}$. Store (K, V) per layer:

New token: compute new $q_t,k_t,v_t$, append $k_t,v_t$, attend. Cost per new token is $O(t)$ attention, not $O(t^2)$ from scratch.

This is the entire user-facing model: frozen $\theta$, forward pass, sample. Training was the process that carved $\theta$. Inference only reads it.

Reverse-mode AD never materializes most of these as matrices. It computes the vector-Jacobian products. Written out, they are the whole backward pass.

The whole machine is one differentiable map from token IDs to next-token distributions. Training differentiates that map and walks θ downhill on cross-entropy. Use-time throws the derivative away and only evaluates the map.

These are the papers that justify a step in the stack, in the order the treatise uses them. A citation earns its place by naming a map, a Jacobian, a floor, or a default this site actually writes. Broader surveys, systems papers, and post-training are out of scope on purpose.

The interactive archive — filter by step, open an arXiv HTML copy, follow a DOI — lives in the Papers tab. This chapter is the print bibliography.

1. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal 27.

   Entropy as the irreducible uncertainty of a discrete source. Every later claim that loss cannot hit zero on language is this paper, applied to tokens.

   doi:10.1002/j.1538-7305.1948.tb01338.x

2. Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal 30.

   The floor is a measurement. Humans who have seen the preceding text still cannot guess the next character with certainty. Conditional entropy of English is low, not zero.

   doi:10.1002/j.1538-7305.1951.tb01366.x

3. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI technical report.

   The decoder-only language model this treatise writes: causal mask, next-token MLE, GELU FFN. Capabilities as a side effect of the conditionals.

   official

1. Gage, P. (1994). A new algorithm for data compression. C Users Journal 12(2).

   Byte-pair encoding as a compression algorithm: repeatedly merge the most frequent adjacent pair. The tokenizer is this idea, frozen, pointed at text.

   closed

2. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. ACL 2016.

   BPE as the frozen map from bytes to the integers the treatise trains on. Loss, perplexity, and ‘what zero would mean’ are defined on this vocabulary.

   arXiv:1508.07909

3. Kudo, T., and Richardson, J. (2018). SentencePiece: a simple and language independent subword tokenizer and detokenizer. EMNLP 2018 (system demonstrations).

   The other common tokenizer: unigram LM / BPE trained from raw bytes, language-agnostic, the Llama-class default alongside BPE.

   arXiv:1808.06226

1. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021). RoFormer: enhanced Transformer with rotary position embedding. Neurocomputing / arXiv.

   RoPE. Inner products depend on t − s, not on absolute t and s. No extra residual vector. The rotation the embeddings chapter writes.

   arXiv:2104.09864

1. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016.

   The skip whose Jacobian is I. Why a 32-layer Transformer trains: there is always a path that does not multiply by a layer Jacobian.

   arXiv:1512.03385

2. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint.

   Per-token mean and variance. GPT-2’s norm. The RMSNorm Jacobian in the treatise is this operator with the mean step removed.

   arXiv:1607.06450

3. Zhang, B., and Sennrich, R. (2019). Root mean square layer normalization. NeurIPS 2019.

   RMSNorm: scale without mean-centering. The Llama default, and the Jacobian the backprop chapter writes.

   arXiv:1910.07467

4. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. (2020). On layer normalization in the Transformer architecture. ICML 2020.

   Pre-norm trains at depth; post-norm was the original and is harder. The block equation in the treatise is the pre-norm form this paper recommends.

   arXiv:2002.04745

5. Touvron, H., Lavril, T., Izacard, G., et al. (2023). LLaMA: open and efficient foundation language models. arXiv preprint.

   The modern stack this treatise writes in one notation: RoPE, RMSNorm, SwiGLU, no biases, pre-norm, decoder-only.

   arXiv:2302.13971

1. Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ICLR 2015.

   Attention as a content-based weighted sum over source states. Vaswani replaces the scoring function; the mixing primitive is already here.

   arXiv:1409.0473

2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. NeurIPS 2017.

   Scaled dot-product attention, the √d_h factor, multi-head, sinusoidal position, the original post-norm stack. The mixing equation the treatise writes is this one.

   arXiv:1706.03762

3. Shazeer, N. (2019). Fast Transformer decoding: one write-head is all you need. arXiv preprint.

   Multi-query attention. The KV cache, not the matmul, bounds generation. One K/V head shared across all query heads.

   arXiv:1911.02150

4. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. (2022). FlashAttention: fast and memory-efficient exact attention with IO-awareness. NeurIPS 2022.

   Same softmax Jacobian, different IO. The T×T matrix does not have to live in HBM. Exact attention, not an approximation.

   arXiv:2205.14135

5. Dao, T. (2023). FlashAttention-2: faster attention with better parallelism and work partitioning. arXiv preprint.

   The production kernel. Still exact. Still the same Jacobian. Parallelism and work partitioning, not a new formula.

   arXiv:2307.08691

6. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023). GQA: training generalized multi-query Transformer models from multi-head checkpoints. EMNLP 2023.

   Grouped-query attention: H_kv between 1 and H_q. The Llama-3-class default. Interpolates MHA and MQA.

   arXiv:2305.13245

1. Hendrycks, D., and Gimpel, K. (2016). Gaussian error linear units (GELUs). arXiv preprint.

   GPT-2’s nonlinearity. Smooth, not a hard gate. The older MLP in the treatise before SwiGLU.

   arXiv:1606.08415

2. Shazeer, N. (2020). GLU variants improve Transformer. arXiv preprint.

   SwiGLU: the gated MLP Llama uses. Gate, up, down — three matrices, no bias. The forward and backward the treatise writes.

   arXiv:2002.05202

1. Bridle, J. S. (1990). Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. Neurocomputing (Springer).

   Softmax as a categorical distribution, and the collapse of softmax-plus-cross-entropy to p − y. The treatise’s most important gradient is this identification.

   closed

2. Press, O., and Wolf, L. (2017). Using the output embedding to improve language models. EACL 2017.

   Weight tying: W_U = W_E^⊤. Two uses, one parameter. The unembedding chapter’s optional identification.

   arXiv:1608.05859

1. Cover, T. M., and Thomas, J. A. (2006). Elements of information theory (2nd ed.). Wiley.

   The identity the loss chapter writes without apology: cross-entropy = entropy + KL. Training is KL minimization in disguise.

   closed

2. Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv preprint.

   C ≈ 6ND. The compute unit. Power-law loss in N and D. Their N-versus-D allocation is the one Chinchilla later revised.

   arXiv:2001.08361

3. Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training compute-optimal large language models. NeurIPS 2022.

   The Chinchilla law. E is the fitted irreducible loss — the floor the treatise refuses to call zero. IsoFLOP said roughly 20 tokens per parameter.

   arXiv:2203.15556

4. Besiroglu, T., Erdil, E., and Barnett, M. (2024). Chinchilla Scaling: a replication attempt. arXiv preprint.

   The printed Approach-3 constants do not recover the 20:1 policy. E is closer to 1.82 on their refit. The floor is an estimate, not a constant of nature.

   arXiv:2404.10102

1. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323.

   The public statement that a scalar loss can be routed to every weight by the chain rule, layer by layer, in reverse.

   doi:10.1038/323533a0

2. Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research 18.

   Backprop as reverse-mode automatic differentiation: vector–Jacobian products, not materialized Jacobians. The treatise’s ‘nobody forms J’ is this paper’s point.

   arXiv:1502.05767

1. Loshchilov, I., and Hutter, F. (2017). SGDR: stochastic gradient descent with warm restarts. ICLR 2017.

   Cosine annealing of the learning rate. The schedule the treatise writes (warmup, then cosine to η_min) is this shape, usually without the restarts.

   arXiv:1608.03983

2. Kingma, D. P., and Ba, J. (2015). Adam: a method for stochastic optimization. ICLR 2015.

   Per-parameter first and second moments. The adaptive step the treatise starts from before the ‘W’.

   arXiv:1412.6980

3. Loshchilov, I., and Hutter, F. (2019). Decoupled weight decay regularization. ICLR 2019.

   AdamW: λθ sits outside the second-moment rescaling. That is the entire difference, and it is the LLM default.

   arXiv:1711.05101

1. Micikevicius, P., et al. (2018). Mixed precision training. ICLR 2018.

   BF16/FP16 forward, FP32 master weights. An implementation detail the training loop names so the math is not confused with the storage format.

   arXiv:1710.03740

1. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The curious case of neural text degeneration. ICLR 2020.

   Nucleus / top-p. Inference carves a sampling set from the leftover mass, then renormalizes. No gradient. Also: typical human text is not the mode.

   arXiv:1904.09751

2. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. (2023). Efficiently scaling Transformer inference. MLSys 2023.

   Prefill versus decode, and why the KV cache is the resource that bounds serving. The inference chapter’s operational picture.

   arXiv:2211.05102

1. Dubey, A., et al. (Llama Team) (2024). The Llama 3 herd of models. arXiv preprint.

   The current default: GQA, RoPE, RMSNorm, SwiGLU, document mask, a much larger data mix. Confirmation that the treatise’s stack is the one in production, not a 2017 museum piece.

   arXiv:2407.21783