bag of words concept
| Definition | A representation of a document as a vector with one coordinate per vocabulary term holding that term's count. Chapter 12. Also bag-of-words, BoW. |
|---|---|
| Example | The texts ‘the reader reads the row’ and ‘the row reads the reader’ have the same counts and are one document to a bag of words. |
| Book | Data Mining as Observation, draft 0.2, commit f3914f0; entry id bag-of-words, kind concept. |
| Status | no ledger row names this entry. Corrections: none recorded. |
| Defining equation | Book equation 0.36. |
| Assumptions and scope |
|
| Prior art | none recorded |
| Evidence | lean/DataMiningAsObservation/BagOfWords.lean |
| Reviewed | not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page. |
Equation
Book equation 0.36.
\[w_{t,d}=\mathrm{tf}_{t,d}\cdot\ln\frac{N}{\mathrm{df}_t},\qquad \mathrm{tf}_{t,d}=\frac{\text{count of }t\text{ in }d}{\text{length of }d},\qquad \mathrm{df}_t=\text{documents containing }t.\]
Book equation 12.3.
\[\text{validated}\iff \mathrm{AUROC}_{\text{cross}}-\max\big(\mathrm{AUROC}_{\text{untrained}},\ \mathrm{AUROC}_{\text{BoW}}\big)\ \ge\ 0.10.\]
Conditions
- A document as the vector of its term counts. Two documents that are rearrangements of each other have the same bag, so the bag is a quotient that declares word order irrelevant, the counts are nonnegative, and they sum to the document’s length.
- As a null model it reads only counts, and an encoder is validated only when its held-out AUROC clears the bag’s by the preregistered margin. The scorecard’s bag scores lie between 0.46 and 0.54, near chance.
Conditions are curated in entries.toml rather than read
from a record.
Ledger
none
First stated
Chapter 12 section 12.1 of Data Mining as Observation, with
the program’s bag-of-words null in xbse/README.md:104-130
and the chapter 14 scorecard.
Measurements
none
Failures and corrections
none
Invariance envelope
none declared
Machine checked
lean/DataMiningAsObservation/BagOfWords.lean,
theorems bag_perm, bag_example,
bag_sum, bag_absent, at
observation-data-mining f3914f0; what the check covers is stated in the
book’s appendix
C.
Used in
Data Mining as Observation chapters 0, 12, 14.
Related
TF-IDF; cross-corpus gate; quotient; nuisance.
See also
none
Status
Generated 2026-09-10 by encyclopedia/generate.py; book
at observation-data-mining f3914f0; the commit of every record is listed
in the encyclopedia’s provenance.