Literature, in the order of the lesson

These are the papers that justify a step in the stack, in the order the treatise uses them. A citation earns its place by naming a map, a Jacobian, a floor, or a default this site actually writes. Broader surveys, systems papers, and post-training are out of scope on purpose.

The interactive archive — filter by step, open an arXiv HTML copy, follow a DOI — lives in the Papers tab. This chapter is the print bibliography.

1. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal 27.

   Entropy as the irreducible uncertainty of a discrete source. Every later claim that loss cannot hit zero on language is this paper, applied to tokens.

   doi:10.1002/j.1538-7305.1948.tb01338.x

2. Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal 30.

   The floor is a measurement. Humans who have seen the preceding text still cannot guess the next character with certainty. Conditional entropy of English is low, not zero.

   doi:10.1002/j.1538-7305.1951.tb01366.x

3. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI technical report.

   The decoder-only language model this treatise writes: causal mask, next-token MLE, GELU FFN. Capabilities as a side effect of the conditionals.

   official

1. Gage, P. (1994). A new algorithm for data compression. C Users Journal 12(2).

   Byte-pair encoding as a compression algorithm: repeatedly merge the most frequent adjacent pair. The tokenizer is this idea, frozen, pointed at text.

   closed

2. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. ACL 2016.

   BPE as the frozen map from bytes to the integers the treatise trains on. Loss, perplexity, and ‘what zero would mean’ are defined on this vocabulary.

   arXiv:1508.07909

3. Kudo, T., and Richardson, J. (2018). SentencePiece: a simple and language independent subword tokenizer and detokenizer. EMNLP 2018 (system demonstrations).

   The other common tokenizer: unigram LM / BPE trained from raw bytes, language-agnostic, the Llama-class default alongside BPE.

   arXiv:1808.06226

1. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021). RoFormer: enhanced Transformer with rotary position embedding. Neurocomputing / arXiv.

   RoPE. Inner products depend on t − s, not on absolute t and s. No extra residual vector. The rotation the embeddings chapter writes.

   arXiv:2104.09864

1. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. CVPR 2016.

   The skip whose Jacobian is I. Why a 32-layer Transformer trains: there is always a path that does not multiply by a layer Jacobian.

   arXiv:1512.03385

2. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint.

   Per-token mean and variance. GPT-2’s norm. The RMSNorm Jacobian in the treatise is this operator with the mean step removed.

   arXiv:1607.06450

3. Zhang, B., and Sennrich, R. (2019). Root mean square layer normalization. NeurIPS 2019.

   RMSNorm: scale without mean-centering. The Llama default, and the Jacobian the backprop chapter writes.

   arXiv:1910.07467

4. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. (2020). On layer normalization in the Transformer architecture. ICML 2020.

   Pre-norm trains at depth; post-norm was the original and is harder. The block equation in the treatise is the pre-norm form this paper recommends.

   arXiv:2002.04745

5. Touvron, H., Lavril, T., Izacard, G., et al. (2023). LLaMA: open and efficient foundation language models. arXiv preprint.

   The modern stack this treatise writes in one notation: RoPE, RMSNorm, SwiGLU, no biases, pre-norm, decoder-only.

   arXiv:2302.13971

1. Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ICLR 2015.

   Attention as a content-based weighted sum over source states. Vaswani replaces the scoring function; the mixing primitive is already here.

   arXiv:1409.0473

2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. NeurIPS 2017.

   Scaled dot-product attention, the √d_h factor, multi-head, sinusoidal position, the original post-norm stack. The mixing equation the treatise writes is this one.

   arXiv:1706.03762

3. Shazeer, N. (2019). Fast Transformer decoding: one write-head is all you need. arXiv preprint.

   Multi-query attention. The KV cache, not the matmul, bounds generation. One K/V head shared across all query heads.

   arXiv:1911.02150

4. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. (2022). FlashAttention: fast and memory-efficient exact attention with IO-awareness. NeurIPS 2022.

   Same softmax Jacobian, different IO. The T×T matrix does not have to live in HBM. Exact attention, not an approximation.

   arXiv:2205.14135

5. Dao, T. (2023). FlashAttention-2: faster attention with better parallelism and work partitioning. arXiv preprint.

   The production kernel. Still exact. Still the same Jacobian. Parallelism and work partitioning, not a new formula.

   arXiv:2307.08691

6. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023). GQA: training generalized multi-query Transformer models from multi-head checkpoints. EMNLP 2023.

   Grouped-query attention: H_kv between 1 and H_q. The Llama-3-class default. Interpolates MHA and MQA.

   arXiv:2305.13245

1. Hendrycks, D., and Gimpel, K. (2016). Gaussian error linear units (GELUs). arXiv preprint.

   GPT-2’s nonlinearity. Smooth, not a hard gate. The older MLP in the treatise before SwiGLU.

   arXiv:1606.08415

2. Shazeer, N. (2020). GLU variants improve Transformer. arXiv preprint.

   SwiGLU: the gated MLP Llama uses. Gate, up, down — three matrices, no bias. The forward and backward the treatise writes.

   arXiv:2002.05202

1. Bridle, J. S. (1990). Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. Neurocomputing (Springer).

   Softmax as a categorical distribution, and the collapse of softmax-plus-cross-entropy to p − y. The treatise’s most important gradient is this identification.

   closed

2. Press, O., and Wolf, L. (2017). Using the output embedding to improve language models. EACL 2017.

   Weight tying: W_U = W_E^⊤. Two uses, one parameter. The unembedding chapter’s optional identification.

   arXiv:1608.05859

1. Cover, T. M., and Thomas, J. A. (2006). Elements of information theory (2nd ed.). Wiley.

   The identity the loss chapter writes without apology: cross-entropy = entropy + KL. Training is KL minimization in disguise.

   closed

2. Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv preprint.

   C ≈ 6ND. The compute unit. Power-law loss in N and D. Their N-versus-D allocation is the one Chinchilla later revised.

   arXiv:2001.08361

3. Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training compute-optimal large language models. NeurIPS 2022.

   The Chinchilla law. E is the fitted irreducible loss — the floor the treatise refuses to call zero. IsoFLOP said roughly 20 tokens per parameter.

   arXiv:2203.15556

4. Besiroglu, T., Erdil, E., and Barnett, M. (2024). Chinchilla Scaling: a replication attempt. arXiv preprint.

   The printed Approach-3 constants do not recover the 20:1 policy. E is closer to 1.82 on their refit. The floor is an estimate, not a constant of nature.

   arXiv:2404.10102

1. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323.

   The public statement that a scalar loss can be routed to every weight by the chain rule, layer by layer, in reverse.

   doi:10.1038/323533a0

2. Baydin, A. G., Pearlmutter, B. A., Radul, A. A., and Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research 18.

   Backprop as reverse-mode automatic differentiation: vector–Jacobian products, not materialized Jacobians. The treatise’s ‘nobody forms J’ is this paper’s point.

   arXiv:1502.05767

1. Loshchilov, I., and Hutter, F. (2017). SGDR: stochastic gradient descent with warm restarts. ICLR 2017.

   Cosine annealing of the learning rate. The schedule the treatise writes (warmup, then cosine to η_min) is this shape, usually without the restarts.

   arXiv:1608.03983

2. Kingma, D. P., and Ba, J. (2015). Adam: a method for stochastic optimization. ICLR 2015.

   Per-parameter first and second moments. The adaptive step the treatise starts from before the ‘W’.

   arXiv:1412.6980

3. Loshchilov, I., and Hutter, F. (2019). Decoupled weight decay regularization. ICLR 2019.

   AdamW: λθ sits outside the second-moment rescaling. That is the entire difference, and it is the LLM default.

   arXiv:1711.05101

1. Micikevicius, P., et al. (2018). Mixed precision training. ICLR 2018.

   BF16/FP16 forward, FP32 master weights. An implementation detail the training loop names so the math is not confused with the storage format.

   arXiv:1710.03740

1. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The curious case of neural text degeneration. ICLR 2020.

   Nucleus / top-p. Inference carves a sampling set from the leftover mass, then renormalizes. No gradient. Also: typical human text is not the mode.

   arXiv:1904.09751

2. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. (2023). Efficiently scaling Transformer inference. MLSys 2023.

   Prefill versus decode, and why the KV cache is the resource that bounds serving. The inference chapter’s operational picture.

   arXiv:2211.05102

1. Dubey, A., et al. (Llama Team) (2024). The Llama 3 herd of models. arXiv preprint.

   The current default: GQA, RoPE, RMSNorm, SwiGLU, document mask, a much larger data mix. Confirmation that the treatise’s stack is the one in production, not a 2017 museum piece.

   arXiv:2407.21783