12 · I · Use

Inference and the KV cache

Frozen θ. Temperature, top-k, nucleus. Prefill once, then decode with cached keys and values.

Weights freeze. Only the forward map runs.

Once training is finished, $θ$ is frozen. There is no “correct” next token sitting in a dataset, so there is nothing to compute a loss on and therefore nothing to back-propagate. Autoregressive generation:

Start with prompt tokens $x_{1 : t}$ .
Forward → logits $z_{t}$ .
Form a sampling distribution from $z_{t}$ .
Draw $x_{t + 1}$ , append, repeat.

Decoding math

(p_{t})_{k} = \frac{exp ( z _{t, k} / τ )}{\sum _{j} exp ( z _{t, j} / τ )}

(12.1)

$τ > 0$: temperature. τ → 0 is greedy: argmax_k z_{t,k}

Top-k. Keep the $k$ largest logits, set the rest to $- \infty$ , then softmax.

Nucleus / top-p (Holtzman et al., 2020). Keep the smallest set of tokens whose cumulative probability is at least $p$ , then renormalize.

V_{p} = ar g V min {∣ V ∣ : k \in V \sum p_{t} (k) \geq p}

(12.2)

No $L$ . No Jacobian. No Adam.

KV cache

Attention at step $t$ only needs $q_{t}$ and all past $k_{1 : t}, v_{1 : t}$ . Store (K, V) per layer:

K^{cache} \in R^{t \times d_{k}}, V^{cache} \in R^{t \times d_{v}}

(12.3)

New token: compute new $q_{t}, k_{t}, v_{t}$ , append $k_{t}, v_{t}$ , attend. Cost per new token is $O (t)$ attention, not $O (t^{2})$ from scratch.

Prefill

O(T²)

One forward over the whole prompt. Every position attends to the past. Cache is filled.

Decode

O(t) / token

New q attends to cached K, V. Append one row. No Jacobian. No optimizer.

This is the entire user-facing model: frozen $θ$ , forward pass, sample. Training was the process that carved $θ$ . Inference only reads it.