harness concept

Definition	The code that evaluates a model. It is itself a consumer with a read subspace, and chapter 8 shows how it reads the test fold when allowed to.
Example	Ten folds, five seeds, and 100 samples per fold is a budget of 5000 reads of the data.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `harness`, kind concept.
Status	no ledger row names this entry. Corrections: 1 item(s), see below.
Defining equation	Book equation 8.1.
Assumptions and scope	The harness is the code that evaluates a model, and it is a consumer with its own read subspace, its output metric, and a budget of folds, samples, and seeds. What it reads decides what the reported score measures. A harness that is allowed to see the test fold, through feature selection before the split, through repeated selection on one split, or through a metric that discriminates nothing, reports a score of itself. Repeated cross-validation folds are not independent, so the variance of a fold mean is understated unless corrected, and the program’s own retracted comparison is the case.
Prior art	none recorded
Evidence	`theory-radar/paper/REVISION_PLAN.md:39-45`, `lean/DataMiningAsObservation/Bonferroni.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

The evaluation reads folds, samples, and seeds, and what it reads it can leak.

Equation

Book equation 8.1.

\[O_{\mathrm{harness}}=\big(C_{\mathrm{score}},\ G_{\mathrm{metric}},\ B=\text{folds}\times\text{samples}\times\text{seeds}\big).\]

Conditions

The harness is the code that evaluates a model, and it is a consumer with its own read subspace, its output metric, and a budget of folds, samples, and seeds. What it reads decides what the reported score measures.
A harness that is allowed to see the test fold, through feature selection before the split, through repeated selection on one split, or through a metric that discriminates nothing, reports a score of itself.
Repeated cross-validation folds are not independent, so the variance of a fold mean is understated unless corrected, and the program’s own retracted comparison is the case.

Conditions are curated in entries.toml rather than read from a record.

Ledger

none

First stated

Chapter 8 section 8.1 of Data Mining as Observation, with the program’s own harness failures in openvector-bench/results/QUERY_COUPLING_ARTIFACT.md:1-20 and constraint-gap/review/FINDINGS.md:1-35.

Measurements

none

Failures and corrections

theory-radar/paper/REVISION_PLAN.md:39-45 at 37c4e6c. Fix: - Drop “σ significance” everywhere - Report: mean ΔF1, std, 95% CI, and corrected resampled t-statistic - Table columns: “Test F1”, “GB F1”, “ΔF1”, “95% CI”, “Direction” - Text: “the formula outperforms GB by ΔF1=0.031, 95% CI [0.028, 0.034]” - Add Nadeau-Bengio corrected t-test (accounts for CV fold correlation) - Keep effect size as supplementary but don’t call it “σ”

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/Bonferroni.lean, theorems family_error_le, bonferroni, one_look, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation chapters 1, 2, 4, 6, 7, 8, 10, 12, 14.

leakage; observer; preregistration; ledger class.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← graphhash →