head concept

Definition	One attention operation. A model has many, each with its own queries, keys, and values. Chapter 0 section 0.11.
Example	One head reads a query against 512 keys and returns one weighted mix of the values.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `head`, kind concept.
Status	no ledger row names this entry. Corrections: none recorded.
Defining equation	Book equation 0.23.
Assumptions and scope	One attention operation, with its own queries, keys, and values. A head’s output lies between the smallest and largest value, it reads the keys only through their scores against its query, and a key change the query does not read leaves its output unchanged however large the change. Each head has its own read subspace, a few query-weighted directions of each key, so a compression that serves one head can fail another, and the serving-stack measurement of chapter 13 is per head.
Prior art	none recorded
Evidence	`turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`, `lean/DataMiningAsObservation/Attention.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

One attention head, a query against every key.

Equation

Book equation 0.23.

\[\operatorname{softmax}(z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}},\qquad \text{output}=\sum_i\operatorname{softmax}\!\Big(\frac{q\cdot k_i}{\sqrt{d}}\Big)_{\!i}\,v_i.\]

Conditions

One attention operation, with its own queries, keys, and values. A head’s output lies between the smallest and largest value, it reads the keys only through their scores against its query, and a key change the query does not read leaves its output unchanged however large the change.
Each head has its own read subspace, a few query-weighted directions of each key, so a compression that serves one head can fail another, and the serving-stack measurement of chapter 13 is per head.

Conditions are curated in entries.toml rather than read from a record.

Ledger

none

First stated

Chapter 0 section 0.11 of Data Mining as Observation, with the program’s head-level measurements in turboquant-pro/docs/KV_KEYS_FINDING.md:1-49 and the planted probe of chapter 6.

Measurements

Where the book states it	Numbers, as the book’s sources table records them	Source
chapter 11 section 11.2	fp16 12.24, values-only 13.12, PolarQuant K4 10643 and 0.095, per-channel uniform K4 14.91 and 0.062, per-channel NUQ K3 15.77 and 0.148, 2.4x and 670x, pre-rotary near 22000, 2 key heads serve 12 query heads	`turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`

Failures and corrections

none

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/Attention.lean, theorems output_le_max, min_le_output, output_congr, output_nuisance, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation primer S, chapters 0, 1, 2, 4, 6, 7, 8, 11, 12, 13.

attention; KV cache; read subspace; softmax.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← hashHessian →