Instruments
Laboratory
Every control here is a term from the treatise. Nothing is simulated mysticism. The order follows the lesson: the maps, then the error signal, then the floor, then the step, then use-time.
Laboratory
Softmax and temperature
Logits become a distribution. Temperature rescales the energy. Adding a constant to every logit does nothing: softmax is shift-invariant, which is why implementations subtract the max first.
peaks on the max logit. flattens toward uniform. Entropy now 1.204 nats. Drag : the bars do not move.
- the0.173
- a0.032
- Paris0.636
- France0.070
- is0.023
- to0.012
- of0.047
- cat0.006
Laboratory
RoPE rotation
Each even/odd pair is a 2D rotation. The inner product of a query at t and a key at s depends only on t − s, not on absolute indices.
. For this pair, ⟨q̃, k̃⟩ = cos((t−s)θᵢ) = 0.498
From the treatise: embeddings
Laboratory
Causal mask
Future tokens are −∞. Row t may only look at positions s ≤ t. That is the entire difference between a decoder and an encoder.
Hover a cell.
From the treatise: attention
Laboratory
The √dₕ scale
A typical coordinate of q and k is O(1), so a typical dot product is O(√dₕ). Softmax of that saturates: one position gets all the mass, and the Jacobian vanishes. Dividing by √dₕ puts scores back at O(1).
Unscaled softmax(q·k)
- the1.000
- a0.000
- Paris0.000
- France0.000
- is0.000
- to0.000
- of0.000
- cat0.000
Entropy 0.000 nats
Scaled softmax(q·k / √dₕ)
- the0.707
- a0.042
- Paris0.017
- France0.048
- is0.039
- to0.112
- of0.022
- cat0.013
Entropy 1.107 nats
From the treatise: attention
Laboratory
Skip Jacobian I
Z = H + F(H) has Jacobian I + J_F. The first term is the skip. It does not shrink with depth. That is why a 32-layer decoder trains and a 32-layer plain stack does not.
No skip — αⁿ
0.058
Signal dies as αⁿ.
Residual — 1 + αⁿ
1.058
Skip path stays O(1).
Laboratory
The p − y error signal
Cross-entropy plus softmax collapses to probability minus truth. Overconfident wrong tokens get a large push. Already-correct peaks get a small one.
Loss 0.851 nats. Gradient at the true class is p_c − 1 (always ≤ 0). Drag logits below.
- the+0.19
- a+0.08
- Paris-0.57
- France+0.06
- is+0.04
- to+0.12
- of+0.06
- cat+0.03
Laboratory
Entropy + KL
Expected cross-entropy splits into the entropy of the true next-token distribution and the KL from that distribution to the model. Only the KL can be trained away.
H(P★) irreducible
0.819
DKL(P★ ‖ pθ)
0.127
Loss = H + KL
0.946
Ink is entropy. Steel is KL. Their sum is the loss.
Laboratory
Meaning versus token
The prompt is “After tomorrow is”. Context kills the calendar branching. It does not kill the wording. Next-token loss lives on the string, not the proposition.
No weekday is pinned. Several dates, several wordings.
Meaning
- Tuesday0.300
- Monday0.280
- Wednesday0.220
- other0.200
H(meaning) = 1.373 nats
Next token
- Tuesday0.160
- Tues.0.100
- the0.150
- it0.140
- 16th0.100
- Monday0.140
- will0.110
- ?0.100
H(token) = 2.062 nats
From the treatise: floor
Laboratory
Chinchilla scaling
L(N, D) = E + A N^{−α} + B D^{−β}. Compute C ≈ 6ND. The IsoFLOP headline is ~20 tokens per parameter. The floor E is entropy of the corpus, not a failure of AdamW.
Loss L
2.386 nats
Perplexity
10.87
Floor E
1.69 nats
Tokens / param
100.0
Training FLOPs C ≈ 6ND = 6.00e+20. Chinchilla’s 70B model used 1.4T tokens — twenty per parameter. Push N without D and you pay the data term; the reverse pays the parameter term. Neither term is the floor.
Steel bar is the fraction of L that is the irreducible floor. The rest is finite N and D.
Laboratory
One AdamW step
Quadratic well at θ = 1. SGD and Adam walk to the well. AdamW’s decoupled decay pulls toward the origin, so the settled point is 1/(1+λ), not 1. That is the whole difference from Adam.
Grey SGD · ink Adam (λ = 0) · steel AdamW. Final AdamW θ = 0.989.
Laboratory
Decoding: top-k and nucleus
Inference carves a sampling distribution from logits, then renormalizes. Nothing here is a gradient. The weights are frozen.
Top-k, renormalized
- the0.414
- a0.297
- Paris0.170
- France0.078
- is0.040
- to0.000
- of0.000
- cat0.000
Nucleus, renormalized
- the0.432
- a0.309
- Paris0.177
- France0.082
- is0.000
- to0.000
- of0.000
- cat0.000
From the treatise: inference
Laboratory
Prefill and decode
Without a cache, each new token redoes attention over the whole prefix, O(t²) from scratch. With a cache, you pay O(t) for the new query against stored keys.
Steel cells are cached (K, V). The last one is the new token.
Naive recompute
16,384
score entries this step
With KV cache
128
new scores this step
Ratio saved this step: 128.0×. Prefill still pays the full quadratic once, then decode is linear per token (and memory-bandwidth bound).
From the treatise: inference