A treatise · fifteen chapters · twelve laboratories · 35 sources
Irreducible
The mathematics of the autoregressive Transformer
A complete, notation-stable treatise on building, training, and running an LLM: tokens, residual stream, attention, SwiGLU, cross-entropy, Jacobians, AdamW, and inference. Every symbol defined where it is used. Loss is driven toward zero. It never arrives.
The object
An LLM is a function . Training drives the next-token cross-entropy toward zero. On open language it cannot arrive. The floor is the entropy of the data, not a failure of the optimizer.
Forward map. Training attaches CE → Jacobians → AdamW after p. Inference samples from p.
Treatise
Fifteen chapters, one notation.
- 00The object of studyAn LLM is a map from token IDs to logits. Training only ever sees next-token conditionals.Object
- 01Tokenization and targetsText becomes integers. Loss, perplexity, and “zero” are defined on this vocabulary, not on English words.A · Data
- 02Embeddings and positionTokens enter a vector space. RoPE injects order by rotating query/key pairs.B · Embed
- 03The pre-norm Transformer blockResidual highways, RMSNorm, and the two sublayers that every modern decoder stacks.C · Block
- 04Causal attentionThe only place tokens mix. Scale, mask, softmax, GQA, and the softmax Jacobian.C · Mix
- 05MLP and SwiGLUPer-token computation and the wide matrices that store features.C · Compute
- 06Unembedding to probabilitiesA final norm, a linear map into V dimensions, and a categorical distribution per position.D · Readout
- 07Loss, entropy, and scalingCross-entropy is maximum likelihood. It decomposes into entropy plus KL. The floor is E, not zero.E · Signal
- 08Why loss is not zeroContext collapses meaning. It does not collapse the next token. The floor is entropy, not tautology.E · Floor
- 09Backpropagation and JacobiansReverse-mode AD, the p − y collapse, RMSNorm, attention, SwiGLU, and the skip whose Jacobian is I.F · Reverse
- 10AdamW, clip, and the scheduleTurning a gradient into a new θ without destroying a billion-parameter run.G · Step
- 11The training loopBatch, forward, loss, backward, accumulate, clip, step. Train loss is not validation loss.H · Train
- 12Inference and the KV cacheFrozen θ. Temperature, top-k, nucleus. Prefill once, then decode with cached keys and values.I · Use
- 13End-to-end map and catalogueThe stacked forward equation, the train path, the infer path, Jacobians, and what each piece is for.Map
- 14Literature, in the order of the lessonThe papers that justify each step. Named where they earn a place. The Papers tab is the reading copy.Sources
Laboratory
Manipulate the same objects.
Temperature, shift-invariance, the gradient, RoPE, the causal mask, the scale, the skip whose Jacobian is I, Sunday versus the leftover wording, Chinchilla’s floor, AdamW, nucleus sampling, entropy versus KL, and the KV cache — not as pictures, as instruments.
Softmax and temperature
Logits become a distribution. Temperature rescales the energy. A constant shift does nothing.
RoPE rotation
Relative position as a 2D rotation of query/key pairs.
Causal mask
Future tokens are −∞. The model is a next-token predictor, not an encoder.
The √dₕ scale
Without 1/√dₕ, scores grow like √dₕ and softmax saturates. Gradients vanish.
Skip Jacobian I
A residual path whose Jacobian is I. Depth does not erase the error signal.
The p − y error signal
Cross-entropy plus softmax collapses to probability minus truth.
Entropy + KL
Loss splits into irreducible language entropy and model error.
Meaning versus token
Sunday pins the weekday. It does not pin the wording. That leftover is the floor.
Chinchilla scaling
L(N, D) = E + A N^{−α} + B D^{−β}. Compute is 6ND. The floor is E, not zero.
One AdamW step
Moments, bias correction, decoupled decay. Watch one coordinate walk.
Decoding: top-k and nucleus
How inference carves a sampling distribution from logits.
Prefill and decode
Why generation is O(t) per new token, not O(t²) from scratch.
Primary sources
The papers that named the maps.
Each step of the stack has the work that defined it — Shannon’s floor, Vaswani’s scaled dots, Zhang’s RMSNorm, Hoffmann’s E. Open-access copies are linked; we do not host publisher PDFs. Closed works go to the DOI.
- 1948A mathematical theory of communicationEntropy as the irreducible uncertainty of a discrete source. Every later claim that loss cannot hit zero on language is this paper, applied to tokens.
- 1951Prediction and entropy of printed EnglishThe floor is a measurement. Humans who have seen the preceding text still cannot guess the next character with certainty. Conditional entropy of English is low, not zero.
- 2006Elements of information theory (2nd ed.)The identity the loss chapter writes without apology: cross-entropy = entropy + KL. Training is KL minimization in disguise.
- 1990Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognitionSoftmax as a categorical distribution, and the collapse of softmax-plus-cross-entropy to p − y. The treatise’s most important gradient is this identification.
- 1986Learning representations by back-propagating errorsThe public statement that a scalar loss can be routed to every weight by the chain rule, layer by layer, in reverse.
- 2018Automatic differentiation in machine learning: a surveyBackprop as reverse-mode automatic differentiation: vector–Jacobian products, not materialized Jacobians. The treatise’s ‘nobody forms J’ is this paper’s point.
- 1994A new algorithm for data compressionByte-pair encoding as a compression algorithm: repeatedly merge the most frequent adjacent pair. The tokenizer is this idea, frozen, pointed at text.
- 2016Neural machine translation of rare words with subword unitsBPE as the frozen map from bytes to the integers the treatise trains on. Loss, perplexity, and ‘what zero would mean’ are defined on this vocabulary.