perplexity concept

Definition	Two to the power of the average number of bits a language model needs per token of a test text. Lower is better. Equation 0.22.
Example	An average of 3.585 bits per token gives perplexity 2 to the 3.585, which is 12.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `perplexity`, kind concept.
Status	no ledger row names this entry. Corrections: none recorded.
Defining equation	Book equation 0.22.
Assumptions and scope	Two to the power of the average number of bits a language model needs per token of a test text. It is at least one, equals the vocabulary size for a model that spreads its mass evenly, and is monotone in the bits. The attention-key finding raised it from 12.24 to 10643 at cosine 0.995, a rise of more than nine bits per token, and the recalibration that improved the reconstruction worsened it, the two standing negatives that keep reconstruction error off the list of acceptance metrics.
Prior art	none recorded
Evidence	`geometric-observation/chapters/ch02_failure_of_observer_free_measurement.md:40-60`, `geometric-observation/chapters/ch16_honest_negatives.md`, `turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`, `lean/DataMiningAsObservation/Perplexity.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

Two to the power of the average bits per token.

Equation

Book equation 0.22.

\[\mathrm{PPL}=2^{H},\qquad H=-\frac1T\sum_{t=1}^{T}\log_2 p\big(w_t\mid w_{<t}\big).\]

Book equation 11.1.

\[\cos\big(k,\hat k\big)=0.995\qquad\text{while}\qquad \mathrm{PPL}:\ 12.24\ \to\ 10643.\]

Conditions

Two to the power of the average number of bits a language model needs per token of a test text. It is at least one, equals the vocabulary size for a model that spreads its mass evenly, and is monotone in the bits.
The attention-key finding raised it from 12.24 to 10643 at cosine 0.995, a rise of more than nine bits per token, and the recalibration that improved the reconstruction worsened it, the two standing negatives that keep reconstruction error off the list of acceptance metrics.

Conditions are curated in entries.toml rather than read from a record.

Ledger

none

First stated

Chapter 0 section 0.11 of Data Mining as Observation, with the program’s case in turboquant-pro/docs/KV_KEYS_FINDING.md:1-49.

Measurements

Where the book states it	Numbers, as the book’s sources table records them	Source
chapter 1 section 1.4	cosine 0.995 and perplexity of order ten thousand, the recalibration negative	`geometric-observation/chapters/ch02_failure_of_observer_free_measurement.md:40-60`; `geometric-observation/chapters/ch16_honest_negatives.md` NEG-2 and NEG-4; `turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`
chapter 8 section 8.2	cosine 0.995, perplexity near 1e4	`turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`; `geometric-observation/chapters/ch16_honest_negatives.md` NEG-2

Failures and corrections

none

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/Perplexity.lean, theorems bits_nonneg, one_le_perplexity, perplexity_uniform, perplexity_mono, bits_of_finding, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation primer S, chapters 0, 1, 8, 11.

KL divergence; the flip; identity reader; read distortion.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← permutation testplanted →