KL divergence concept

Definition	A measure of how far one probability distribution is from another, zero when they are identical. Equation 0.24. Also Kullback, perplexity.
Example	A fair coin coded as if heads had probability 0.9 costs about 0.74 bits of divergence per toss.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `kl-divergence`, kind concept.
Status	no ledger row names this entry. Corrections: none recorded.
Defining equation	Book equation 0.24.
Assumptions and scope	The expected log ratio of two distributions’ masses under the first. It is nonnegative by Gibbs’ inequality, zero when the distributions agree, and not symmetric, so the direction of the comparison is part of the claim. The book’s use is through perplexity. A reconstruction at cosine 0.995 raised the perplexity by three orders of magnitude, which is the case that reconstruction error is not the consumer’s error.
Prior art	none recorded
Evidence	`geometric-observation/chapters/ch02_failure_of_observer_free_measurement.md:40-60`, `geometric-observation/chapters/ch16_honest_negatives.md`, `turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`, `lean/DataMiningAsObservation/KL.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

How many extra bits coding with q costs when the truth is p.

Equation

Book equation 0.24.

\[\mathrm{KL}(p\,\|\,q)=\sum_i p_i\ln\frac{p_i}{q_i}\ \ge 0.\]

Book equation 0.22.

\[\mathrm{PPL}=2^{H},\qquad H=-\frac1T\sum_{t=1}^{T}\log_2 p\big(w_t\mid w_{<t}\big).\]

Conditions

The expected log ratio of two distributions’ masses under the first. It is nonnegative by Gibbs’ inequality, zero when the distributions agree, and not symmetric, so the direction of the comparison is part of the claim.
The book’s use is through perplexity. A reconstruction at cosine 0.995 raised the perplexity by three orders of magnitude, which is the case that reconstruction error is not the consumer’s error.

Conditions are curated in entries.toml rather than read from a record.

Ledger

none

First stated

Kullback and Leibler, on information and sufficiency, 1951, as chapter 0 section 0.11 states it beside perplexity, with the program’s case in turboquant-pro/docs/KV_KEYS_FINDING.md:1-49.

Measurements

Where the book states it	Numbers, as the book’s sources table records them	Source
chapter 1 section 1.4	cosine 0.995 and perplexity of order ten thousand, the recalibration negative	`geometric-observation/chapters/ch02_failure_of_observer_free_measurement.md:40-60`; `geometric-observation/chapters/ch16_honest_negatives.md` NEG-2 and NEG-4; `turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`
chapter 8 section 8.2	cosine 0.995, perplexity near 1e4	`turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`; `geometric-observation/chapters/ch16_honest_negatives.md` NEG-2

Failures and corrections

none

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/KL.lean, theorems kl_nonneg, kl_self, kl_not_symm, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation primer S, chapters 0, 1, 8, 11.

identity reader; read distortion; calibration; the flip.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← kernelk-means →