rotary position embedding concept

Definition	A rotation applied to queries and keys that encodes where each token sits in the sequence. Chapter 0 section 0.11. Also rotary, RoPE, position.
Example	A query at position 5 and a key at position 3 score exactly as they would at positions 105 and 103.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `rotary-position-embedding`, kind concept.
Status	no ledger row names this entry. Corrections: none recorded.
Defining equation	Book equation 0.23.
Assumptions and scope	A rotation applied to queries and keys by an angle proportional to position. A rotation preserves length, and the score of a query at one position against a key at another depends only on the difference of positions, so a head reads relative position and shifting both positions by the same amount changes no score. A key quantizer that rounds the rotated key rounds its position as well as its content, which is part of why the direction-only quantizer moved the scores.
Prior art	none recorded
Evidence	`turboquant-pro/docs/KV_KEYS_FINDING.md:61-86`, `turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`, `lean/DataMiningAsObservation/Rope.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

A rotation by position, under which a head reads relative position only.

Equation

Book equation 0.23.

\[\operatorname{softmax}(z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}},\qquad \text{output}=\sum_i\operatorname{softmax}\!\Big(\frac{q\cdot k_i}{\sqrt{d}}\Big)_{\!i}\,v_i.\]

Conditions

A rotation applied to queries and keys by an angle proportional to position. A rotation preserves length, and the score of a query at one position against a key at another depends only on the difference of positions, so a head reads relative position and shifting both positions by the same amount changes no score.
A key quantizer that rounds the rotated key rounds its position as well as its content, which is part of why the direction-only quantizer moved the scores.

Conditions are curated in entries.toml rather than read from a record.

Ledger

none

First stated

Su and others, RoFormer, 2021, as chapter 0 section 0.11 of Data Mining as Observation states it, with the program’s key-side measurements in turboquant-pro/docs/KV_KEYS_FINDING.md:1-49.

Measurements

Where the book states it	Numbers, as the book’s sources table records them	Source
chapter 11 section 11.1	condition (A2), cosine satisfies it, post-rotary keys do not, the cone below cell size	`turboquant-pro/docs/KV_KEYS_FINDING.md:61-86`; `the-angular-observer/README.md:26-31`
chapter 11 section 11.2	fp16 12.24, values-only 13.12, PolarQuant K4 10643 and 0.095, per-channel uniform K4 14.91 and 0.062, per-channel NUQ K3 15.77 and 0.148, 2.4x and 670x, pre-rotary near 22000, 2 key heads serve 12 query heads	`turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`

Failures and corrections

none

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/Rope.lean, theorems rot_length, rot_dot, relative_position, shift_invariant, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation primer L, chapters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14.

attention; query, key, value; KV cache; dot product.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← ROC curveROUGE →