softmax concept

Definition	The function that exponentiates a list of numbers and divides by their sum, so that they are positive and add to one. Equation 0.23.
Example	Scores 2, 1, and 0 give weights 0.665, 0.245, and 0.090, which sum to one.
Book	Data Mining as Observation, draft 0.2, commit `f3914f0`; entry id `softmax`, kind concept.
Status	refutes or corrects [refuted] ×2. Corrections: 2 item(s), see below.
Defining equation	Book equation 0.23.
Assumptions and scope	The softmax exponentiates a list of scores and divides by the sum, so the weights are positive and add to one. Adding the same constant to every score leaves it unchanged, and a larger score gets a larger weight. As a consumer it reads the scores through their differences, which is why the two proxies for softmax-KL in the ledger, the variance ratio and the projected-variance trace, were refuted.
Prior art	none recorded
Evidence	`geometric-observation/claims/LEDGER.md:101`, `geometric-observation/claims/LEDGER.md:102`, `turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`, `lean/DataMiningAsObservation/Softmax.lean`
Reviewed	not yet reviewed; generated 2026-09-10 from records at the commits on the provenance page.

Exponentials normalized to sum to one.

Equation

Book equation 0.23.

\[\operatorname{softmax}(z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}},\qquad \text{output}=\sum_i\operatorname{softmax}\!\Big(\frac{q\cdot k_i}{\sqrt{d}}\Big)_{\!i}\,v_i.\]

Conditions

The softmax exponentiates a list of scores and divides by the sum, so the weights are positive and add to one. Adding the same constant to every score leaves it unchanged, and a larger score gets a larger weight.
As a consumer it reads the scores through their differences, which is why the two proxies for softmax-KL in the ledger, the variance ratio and the projected-variance trace, were refuted.

Conditions are curated in entries.toml rather than read from a record.

Ledger

refutes or corrects. NEG-8 [refuted]. The Var-ratio tang_qproj is a ≥0.9-Spearman rank proxy for softmax-KL under every consumer. geometric-observation/claims/LEDGER.md:101.
refutes or corrects. NEG-9 [refuted]. The projected-variance trace tr(P_C·Σ_δ) is a complete rank statistic for softmax-KL across all arms. geometric-observation/claims/LEDGER.md:102.

First stated

Chapter 0 section 0.11 of Data Mining as Observation, with the program’s softmax reader in turboquant-pro/docs/KV_KEYS_FINDING.md:1-49.

Measurements

Where the book states it	Numbers, as the book’s sources table records them	Source
chapter 2 section 2.5	cosine 0.995 and the softmax reader	`turboquant-pro/docs/KV_KEYS_FINDING.md:1-49`

Failures and corrections

NEG-8, [refuted]. The Var-ratio tang_qproj is a ≥0.9-Spearman rank proxy for softmax-KL under every consumer. geometric-observation/claims/LEDGER.md:101.
NEG-9, [refuted]. The projected-variance trace tr(P_C·Σ_δ) is a complete rank statistic for softmax-KL across all arms. geometric-observation/claims/LEDGER.md:102.

Invariance envelope

none declared

Machine checked

lean/DataMiningAsObservation/Softmax.lean, theorems denom_pos, softmax_pos, softmax_sum, softmax_shift, softmax_lt_iff, at observation-data-mining f3914f0; what the check covers is stated in the book’s appendix C.

Used in

Data Mining as Observation chapters 0, 2, 3, 4, 8, 11.

attention; KL divergence; perplexity; consumer.

Status

Generated 2026-09-10 by encyclopedia/generate.py; book at observation-data-mining f3914f0; the commit of every record is listed in the encyclopedia’s provenance.

← skewnesssources table →