expected calibration error instrument

Definition	The average over score bins of the absolute difference between the mean score and the fraction of positives in the bin. Equation 0.29.
Example	Bins with mean scores 0.2 and 0.8 and positive fractions 0.3 and 0.7, each holding half the rows, give an error of 0.1.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `expected-calibration-error`, kind instrument.
Status	no ledger row names this entry. Corrections: none recorded.
Defining equation	Book equation 0.29.
Assumptions and scope	The average over score bins, weighted by bin size, of the absolute difference between the bin’s mean score and its fraction of positives. It lies in the unit interval, it is zero exactly when every bin’s mean score equals its positive fraction, and a calibrated scorer has error zero. The program’s validated encoders carry errors of 0.018 to 0.101 on held-out splits against raw values up to 0.223, and the reliability weight that gates an encoder’s authority is computed beside it.
Prior art	none recorded
Evidence	`gtc-prototype/docs/CALIBRATED_AUTHORITY.md:19-63`, `lean/DataMiningAsObservation/Calibration.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

The bin-weighted gap between the mean score and the fraction of positives.

Equation

Book equation 0.29.

\[\mathrm{ECE}=\sum_b\frac{n_b}{n}\,\big|\bar s_b-\bar y_b\big|.\]

Book equation 0.38.

\[w=\max\big(0,\ 2\cdot\mathrm{AUROC}-1\big).\]

Conditions

The average over score bins, weighted by bin size, of the absolute difference between the bin’s mean score and its fraction of positives. It lies in the unit interval, it is zero exactly when every bin’s mean score equals its positive fraction, and a calibrated scorer has error zero.
The program’s validated encoders carry errors of 0.018 to 0.101 on held-out splits against raw values up to 0.223, and the reliability weight that gates an encoder’s authority is computed beside it.

Conditions are curated in entries.toml rather than read from a record.

Ledger

none

First stated

Chapter 0 section 0.14 of Data Mining as Observation, equation 0.29, with the program’s per-encoder errors in xbse/README.md:175-195 and gtc-prototype/docs/CALIBRATED_AUTHORITY.md:19-63.

Measurements

Where the book states it	Numbers, as the book’s sources table records them	Source
chapter 12 section 12.5	ECE 0.018 to 0.101 vs raw up to 0.223, reliability weight, audit binding	`xbse/README.md:175-195`; `gtc-prototype/docs/CALIBRATED_AUTHORITY.md:19-63`

Failures and corrections

none

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/Calibration.lean, theorems ece_nonneg, ece_le_one, ece_eq_zero_iff, ece_of_calibrated, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation chapters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14.

calibration; reliability weight; AUROC; threshold.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← expectationexplained variance →