cross-entropy concept
| Definition | The expected bits per outcome when outcomes drawn from p are coded as if from a model q. Its excess over the entropy is the KL divergence, and perplexity is two to the power of the cross-entropy per token. Primer S, equation S.20. Also cross entropy. |
|---|---|
| Example | Coding (0.5, 0.25, 0.25) with the uniform model costs log2 of 3, about 1.585 bits, 0.085 more than its entropy. |
| Book | Data Mining as Observation, draft 0.2, commit f3914f0; entry id cross-entropy, kind concept. |
| Status | no ledger row names this entry. Corrections: none recorded. |
| Defining equation | Book equation S.20. |
| Assumptions and scope | none |
| Prior art | none recorded |
| Evidence | none |
| Reviewed | semantic review 2026-09-09; generated 2026-09-10 from records at the commits on the provenance page. |
Equation
Book equation S.20.
\[H(p,q)=-\sum_i p_i\log_2 q_i,\qquad \mathrm{KL}(p\,\|\,q)=H(p,q)-H(p)=\sum_i p_i\log_2\frac{p_i}{q_i}\ \ge0.\]
Conditions
none
Ledger
none
First stated
Primer S section S.10 of Data Mining as Observation, added in draft 0.3 (2026-09-09) for the ECE 514 readers whose first courses are far behind. The idea is standard and TSK Appendix C covers it at length.
Measurements
none
Failures and corrections
none
Invariance envelope
none declared
Machine checked
none
Used in
Data Mining as Observation primer S.
Related
entropy; KL divergence; perplexity; likelihood, maximum likelihood.
See also
none
Status
Generated 2026-09-10 by encyclopedia/generate.py; book
at observation-data-mining f3914f0; the commit of every record is listed
in the encyclopedia’s provenance.