Symbols

The network: token IDs to a T×V logit matrix.

Every learnable parameter (embeddings, projections, norms).

Vocabulary size — number of distinct tokens.

Sequence length of this forward pass (context window used).

Model / residual-stream width.

Number of Transformer layers.

Number of attention heads (MHA: H_q = H_kv = H).

Query heads and key/value heads. Equal in MHA; H_kv = 1 is MQA.

Head dimension, usually d/H.

Token-ID sequence, each x_t ∈ {1,…,V}.

Autoregressive next-token distribution of the model.

Token embedding matrix, V×d.

Absolute position embedding matrix (GPT-2 style).

Residual stream after embedding, T×d.

RoPE base frequency ω^{−2i/d_h}.

Relative RoPE rotation. ⟨R_t q, R_s k⟩ = qᵀ R_{s−t} k.

Learned RMSNorm / LayerNorm scale, in R^d.

Small constant for numerical stability (e.g. 10⁻⁶, 10⁻⁸).

Query, key, value projection matrices.

Attention scores QKᵀ / √d_h, T×T.

Causal mask: 0 on and below the diagonal, −∞ above.

Attention weights after softmax. Rows sum to 1.

Output projection that merges heads.

diag(a) − aaᵀ. Symmetric, null space along 1.

z · σ(z). Swish activation used in SwiGLU.

SwiGLU up-projection, d × d_ff. Not the unembedding.

SwiGLU gate and down projections.

Unembedding matrix, d×V. Sometimes tied to W_Eᵀ.

Logits at position t, in R^V.

Softmax of z_t. A categorical over the vocabulary.

Token-level cross-entropy / NLL, in nats.

Entropy of the true next-token distribution. Irreducible.

Extra loss from the model being wrong.

Perplexity e^L. Equivalent number of equally likely tokens.

Bits per byte: (L / ln 2) · (T / B). Tokenizer-fair loss.

Fitted irreducible loss in a scaling law.

Parameter count (scaling laws).

Training tokens (scaling laws).

Training compute. C ≈ 6ND FLOPs.

Jacobian ∂v/∂u of a map g.

Vector-Jacobian product. What reverse-mode AD actually computes.

Gradient of CE+softmax w.r.t. logits.

Learning rate (scheduled).

Adam moment decays. Typical 0.9 and 0.95/0.999.

Decoupled weight decay coefficient.

Sampling temperature. τ → 0 is greedy.

Stored keys for past tokens, per layer.