entropy concept
| Definition | The expected surprise of a distribution, the bits needed per outcome under the best code, at most the log base two of the number of outcomes. Primer S, equation S.19. |
|---|---|
| Example | A fair die has entropy log2 of 6, about 2.585 bits, and (0.5, 0.25, 0.25) has 1.5 bits. |
| Book | Data Mining as Observation, draft 0.2, commit f3914f0; entry id entropy, kind concept. |
| Status | no ledger row names this entry. Corrections: none recorded. |
| Defining equation | Book equation S.19. |
| Assumptions and scope | none |
| Prior art | none recorded |
| Evidence | none |
| Reviewed | semantic review 2026-09-09; generated 2026-09-10 from records at the commits on the provenance page. |
Equation
Book equation S.19.
\[H(p)=-\sum_i p_i\log_2 p_i.\]
Conditions
none
Ledger
none
First stated
Primer S section S.10 of Data Mining as Observation, added in draft 0.3 (2026-09-09) for the ECE 514 readers whose first courses are far behind. The idea is standard and TSK Appendix C covers it at length.
Measurements
none
Failures and corrections
none
Invariance envelope
none declared
Machine checked
none
Used in
Data Mining as Observation primer S.
Related
surprise; cross-entropy; KL divergence; perplexity; bit.
See also
none
Status
Generated 2026-09-10 by encyclopedia/generate.py; book
at observation-data-mining f3914f0; the commit of every record is listed
in the encyclopedia’s provenance.