KV cache concept

Definition	The stored keys and values of earlier tokens, kept so they need not be recomputed. Compressing it is where several of the book's examples go wrong. Chapters 0 and 11. Also KV serving, key quantizer.
Example	Serving 14000 tokens with 32 heads keeps 14000 keys and values per head for the next token to read.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `kv-cache`, kind concept.
Status	measures [demonstrated]. Corrections: none recorded.
Defining equation	Book equation 0.23.
Assumptions and scope	The stored keys and values of every earlier token for every layer, so its size is the product of layers, tokens, two, the head width, and the bits per number. Compressing it changes the keys the heads read. A compression that preserves every query-key score preserves every head output, and one that preserves the keys’ cosine need not, since two keys of the same direction and different lengths have cosine one and different scores. That is the attention-key finding in one sentence.
Prior art	none recorded
Evidence	`geometric-observation/claims/LEDGER.md:81`, `lean/DataMiningAsObservation/KVCache.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

One key and one value per token per head, kept for the next token.

Equation

Book equation 0.23.

\[\operatorname{softmax}(z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}},\qquad \text{output}=\sum_i\operatorname{softmax}\!\Big(\frac{q\cdot k_i}{\sqrt{d}}\Big)_{\!i}\,v_i.\]

Conditions

The stored keys and values of every earlier token for every layer, so its size is the product of layers, tokens, two, the head width, and the bits per number. Compressing it changes the keys the heads read.
A compression that preserves every query-key score preserves every head output, and one that preserves the keys’ cosine need not, since two keys of the same direction and different lengths have cosine one and different scores. That is the attention-key finding in one sentence.

Conditions are curated in entries.toml rather than read from a record.

Ledger

measures. GO-2/GO-12/GO-13 operational (KV serving, 077) [demonstrated]. Consumer-relative access width measured on a production serving stack (Qwen2.5-7B KV-cache eviction, matched budget): task quality tracks measured predictive uncertainty u about the consumer’s future reads, not nominal scorer width — the … geometric-observation/claims/LEDGER.md:81.

First stated

Chapter 0 section 0.11 and chapter 11 section 11.1 of Data Mining as Observation, with the program’s measurements in turboquant-pro/docs/KV_KEYS_FINDING.md:1-49 and the serving-stack ledger row.

Measurements

none

Failures and corrections

none

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/KVCache.lean, theorems cacheBits_tokens, cacheBits_half, ratios, output_of_scores_preserved, cosine_not_score, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation chapters 0, 11.

attention; eviction; the flip; budget.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← k-meanslag →