The article provides a technical walkthrough of positional embedding methods in large language models, progressing from absolute to relative techniques with mathematical and PyTorch implementations. It begins by establishing why explicit positional information is necessary despite causal masking providing implicit order: causal attention cannot distinguish distance magnitudes (e.g., one vs. ten positions apart). The first method covered is GPT-2's learned absolute position embeddings, where a positional embedding table of size context_length × emb_dim is added element-wise to token embeddings before the first transformer block; GPT-2 Medium's 1,024-position table constitutes 0.30% of weights, but scaling to 100k context would exceed 22%, making it uneconomical. The original Transformer's fixed sinusoidal encodings are then detailed, using paired sine/cosine functions at geometrically decreasing frequencies (1, 0.1, 0.01, 0.001 radians/position for d_model=8) with token embeddings scaled by √d_model to match magnitude. This structure enables relative reasoning because PE_{pos+k} is a linear function of PE_{pos}, implemented as a rotation. The core of the article explains Rotary Position Embedding (RoPE), introduced in RoFormer (2021), which injects position inside attention by rotating query and key vectors per head. Using split-half pairing (features 0–3 paired with 4–7 in an 8-feature head), each pair is rotated by mθ_i where m is token position and θ_i = 10000^{-2i/d}. A worked example with θ=20°/position shows a query at position 5 (100° rotation) and key at position 2 (40° rotation) yielding a 60° angular difference; their dot product becomes |q||k|cos(60°)=13, down from 26 pre-rotation. Moving the key to position 3 reduces the angle to 40°, increasing the dot product to ~19.92, demonstrating that attention scores depend only on relative offset (m-n). The article notes MiniMax-M3 uses partial RoPE (rotating only part of each head) and pre-RoPE QK-Norm, and previews a survey of 2026 open-weight models' positional choices.
Read original
hackernews