02 · B · Embed

Embeddings and position

Tokens enter a vector space. RoPE injects order by rotating query/key pairs.

Token embedding

A learned matrix $W_{E} \in R^{V \times d}$ stores one row per token. Looking up token $x_{t}$ is a gather; equivalently, a one-hot times $W_{E}$ .

e_{t}^{tok} = (W_{E})_{x_{t}, :} \in R^{d}

(2.1)

$W_{E}$: token embedding matrix, V×d
$e_{t}^{tok}$: the d-vector for token x_t

E^{tok} = X_{oh} W_{E} \in R^{T \times d}

(2.2)

$X_{oh}$: one-hot sequence, T×V

Absolute position (GPT-2 style)

Without a position signal the Transformer is a bag of tokens. Absolute embeddings add a learned vector per index.

h_{t}^{(0)} = e_{t}^{tok} + (W_{P})_{t, :}

(2.3)

$W_{P}$: learned positions, T_max × d
$h_{t}^{(0)}$: residual-stream vector at layer 0

RoPE (Llama-style default)

Rotary position embeddings (Su et al., 2021) add no extra vector to the residual stream. After Q and K are formed, each even/odd pair of coordinates is rotated by an angle that depends on position. Inner products then depend on the difference $t - s$ , not on absolute $t$ and $s$ separately.

θ_{i} = ω^{- 2 i / d_{h}}

(2.4)

$d_{h}$: head dimension (even)
$i$: pair index 0, 1, …, d_h/2 − 1
$ω$: base frequency, usually 10,000 (or a long-context variant)

R_{t, i} = (cos (t θ_{i}) sin (t θ_{i}) - sin (t θ_{i}) cos (t θ_{i}))

(2.5)

q ~_{t, (2 i : 2 i + 1)} = R_{t, i} q_{t, (2 i : 2 i + 1)}, k ~_{s, (2 i : 2 i + 1)} = R_{s, i} k_{s, (2 i : 2 i + 1)}

(2.6)

Write $R_{t}$ for the block-diagonal matrix of all pair rotations at position $t$ . Rotations compose, and $R_{t}^{⊤} = R_{- t}$ :

q ~_{t}^{⊤} k ~_{s} = (R_{t} q_{t})^{⊤} (R_{s} k_{s}) = q_{t}^{⊤} R_{s - t} k_{s}

(2.7)

Why it mattersThe rotation is orthogonal, so it preserves norms. The backward pass is the inverse rotation (the transpose). Relative position is baked into the attention score, which is why RoPE extrapolates farther than absolute embeddings.