calibration concept

Definition	The property of a score that a value of 0.8 means the row is positive eighty percent of the time. Measured by expected calibration error, equation 0.29. Also expected calibration error.
Example	If the rows scored near 0.8 are positive 80 percent of the time, the score is calibrated at 0.8.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `calibration`, kind concept.
Status	refutes or corrects [refuted]. Corrections: 1 item(s), see below.
Defining equation	Book equation 0.29.
Assumptions and scope	A score is calibrated when a value of 0.8 means the row is positive eighty percent of the time. The expected calibration error, the bin-weighted mean absolute gap between mean score and fraction of positives, lies in the unit interval and is zero exactly when every weighted bin is calibrated. Calibration is a property of the score values and not of their ranking, so a recalibration can change it while leaving AUROC, the reliability weight, and every threshold-swept metric unchanged. The two are reported side by side and neither stands in for the other. Recalibrating a reconstruction toward the input improved the reconstruction and worsened the consumer, the standing negative that keeps calibration off the list of acceptance metrics for a code.
Prior art	none recorded
Evidence	`geometric-observation/claims/LEDGER.md:97`, `gtc-prototype/docs/CALIBRATED_AUTHORITY.md:19-63`, `gtc-prototype/docs/CALIBRATED_AUTHORITY.md:1-65`, `lean/DataMiningAsObservation/Calibration.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

Score bins against the fraction of positives in each.

Equation

Book equation 0.29.

\[\mathrm{ECE}=\sum_b\frac{n_b}{n}\,\big|\bar s_b-\bar y_b\big|.\]

Book equation 0.38.

\[w=\max\big(0,\ 2\cdot\mathrm{AUROC}-1\big).\]

Conditions

A score is calibrated when a value of 0.8 means the row is positive eighty percent of the time. The expected calibration error, the bin-weighted mean absolute gap between mean score and fraction of positives, lies in the unit interval and is zero exactly when every weighted bin is calibrated.
Calibration is a property of the score values and not of their ranking, so a recalibration can change it while leaving AUROC, the reliability weight, and every threshold-swept metric unchanged. The two are reported side by side and neither stands in for the other.
Recalibrating a reconstruction toward the input improved the reconstruction and worsened the consumer, the standing negative that keeps calibration off the list of acceptance metrics for a code.

Conditions are curated in entries.toml rather than read from a record.

Ledger

refutes or corrects. NEG-4 [refuted]. Lightweight online (Lloyd) key calibration beats the calibration-free default on softmax-KL. geometric-observation/claims/LEDGER.md:97.

First stated

Chapter 0 section 0.14 of Data Mining as Observation, with the program’s calibrated authority in gtc-prototype/docs/CALIBRATED_AUTHORITY.md:19-63 and the recalibration negative of ledger row NEG-4.

Measurements

Where the book states it	Numbers, as the book’s sources table records them	Source
chapter 12 section 12.5	ECE 0.018 to 0.101 vs raw up to 0.223, reliability weight, audit binding	`xbse/README.md:175-195`; `gtc-prototype/docs/CALIBRATED_AUTHORITY.md:19-63`
chapter 14 section 14.2	reliability weights and calibration errors per axis, the collapsed family’s mean weight 0.559 against the general valence channel’s own 0.735, the three design rules, 0.048 to 0.049 and 0.089 to 0.101 at 696 pairs	`gtc-prototype/docs/CALIBRATED_AUTHORITY.md:1-65`

Failures and corrections

NEG-4, [refuted]. Lightweight online (Lloyd) key calibration beats the calibration-free default on softmax-KL. geometric-observation/claims/LEDGER.md:97.

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/Calibration.lean, theorems ece_nonneg, ece_le_one, ece_eq_zero_iff, ece_of_calibrated, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation primer S, chapters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14.

reliability weight; cross-corpus gate; Monotone Invariance Theorem; identity reader.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← cachecapacity →