Chapter 12: Sensory Integration and the Geometry of Experience
“Flavor is not in the food; it is created by the brain.” — Gordon M. Shepherd, Neurogastronomy: How the Brain Creates Flavor and Why It Matters (2012)
RUNNING EXAMPLE — THE RED-DYED WHITE WINE
In 2001, in a laboratory at the Faculty of Oenology in Bordeaux, the doctoral student Frédéric Brochet handed fifty-four tasters a glass of white wine and asked them to describe its aroma in their own words. A few days later he handed the same panel another glass — the very same white wine, now dyed red with an odorless, tasteless anthocyanin food coloring. The tasters were not novices; they were students of oenology, trained to put language to what the nose and tongue report.
For the white wine they reached for the vocabulary of white wine — citrus, lychee, honey, straw. For the dyed wine, which was chemically the identical white, they reached for the vocabulary of red wine — cherry, blackcurrant, cedar, crushed leaves, tannic structure. The same molecules entered the same noses. The only thing that changed was a wavelength of reflected light, and the change rewrote the report. Vision did not merely tint the experience; it reached down into the olfactory percept and overruled it. The tasters were not lying, and they were not fools. They were telling the truth about what they perceived, and what they perceived had already been edited by their eyes before it reached awareness.
This chapter is about that editing. It asks how five separate sensory streams — taste, smell, texture, temperature, and sight — are fused into the single, seamless object we call “the flavor of the wine,” and why a fusion that is normally invisible can be hijacked by a few drops of dye. The answer is that the brain does not store the streams side by side. It projects them, jointly, into one low-dimensional space, and a point in that space has no memory of which sense supplied which coordinate. The dye did not fool the tongue. It moved the point.
One Percept From Many Senses
Pick up a strawberry and eat it, and ask yourself, honestly, where the experience lives. The sweetness is on the tongue. The bright, green-red volatiles are in the nose, delivered retronasally up the back of the throat. The seedy give and juicy collapse are in the jaw and the mechanoreceptors of the mouth. The faint coolness is thermal. And before any of this, the eye has already reported a deep red and a particular shape, and the ear has registered, dimly, the wet crush. Five physically distinct transduction systems, in five anatomically distinct populations of cells, fire in five different parts of the brain — and yet what you experience is not five things. It is one thing: the flavor of a strawberry, located, impossibly, in the mouth.
This is the central puzzle of flavor, and it is not a small one. Flavor is the textbook case of a multisensory percept: a unified experience assembled from inputs that share no common receptor, no common nerve, and no common cortical map. The unity is so complete that we forget it is an achievement — we say a strawberry “tastes” of strawberry, when the overwhelming majority of that taste is smell and a non-trivial fraction is sight and touch. As Shepherd argued in Neurogastronomy (2012), flavor is not a property of the food at all. It is a construction, performed by the brain, from materials the food provides.
The previous eleven chapters built the flavor vector $\mathbf{f}$ and the space $\mathcal{F}$ it lives in, and treated that space as given. This chapter asks where $\mathcal{F}$ comes from — not in chemistry, but in the brain. The thesis is that $\mathcal{F}$ is the output of an integration pipeline: a sequence of neural maps that takes the high-dimensional, multi-stream sensory input and projects it, jointly and irreversibly, onto a single low-dimensional manifold of experience. The binding of the senses, the cross-modal correspondences, the wine dye’s uncanny power, attention, and even Proust’s madeleine are all consequences of one geometric fact — that the senses converge before perception, not after, and that what converges is not five vectors but one.
12.1 The Binding Problem in Flavor
The general binding problem is one of the oldest in cognitive science: when the brain represents a red ball moving left, the redness, the roundness, and the leftward motion are computed by different neural populations, so what makes them features of one object rather than three loose facts? In vision the problem is hard. In flavor it is harder, because the features are not even in the same sensory modality. Sweetness (gustatory), strawberry aroma (olfactory), and juicy texture (somatosensory) must be bound into one object despite arriving along entirely separate cranial nerves — the facial and glossopharyngeal for taste, the olfactory for smell, the trigeminal for texture and temperature.
[Empirical.] The binding in flavor is unusually aggressive, and the diagnostic case is the oral referral of smell. Retronasal aroma is transduced in the olfactory epithelium, high in the nasal cavity, yet we experience it as located in the mouth, fused with taste. Pinch your nose and the aroma vanishes; release it and the “taste” floods back — but it never feels like it is happening in your nose. The brain has referred the olfactory signal to the site of the gustatory and tactile signals, because those signals co-occur in time and the brain’s prior is that co-occurring signals belong to one source. Rozin (1982) named this the “duality” of smell: the same molecule perceived orthonasally (sniffed) and retronasally (in the mouth) yields perceptually different objects, because the second is bound to taste and touch and the first is not.
We can state the problem geometrically. Let the raw sensory input be a tuple of vectors in distinct spaces: $$ \big(\mathbf{t} \in \mathcal{T},\ \mathbf{a} \in \mathcal{A},\ \mathbf{x} \in \mathcal{X},\ \boldsymbol{\theta} \in \Theta,\ \mathbf{v} \in \mathcal{V}\big), $$ taste, aroma, texture, temperature, and vision, each in its own receptor space. The naive concatenation lives in the product space $\mathcal{T} \times \mathcal{A} \times \mathcal{X} \times \Theta \times \mathcal{V}$, whose dimension is the sum — hundreds of dimensions, dominated by the ~400 of $\mathcal{A}$. The percept does not live there.
[Modeling Axiom.] Binding is a map $\Phi$ from the product of sensory spaces to a single low-dimensional flavor space: $$ \Phi:\ \mathcal{T} \times \mathcal{A} \times \mathcal{X} \times \Theta \times \mathcal{V}\ \longrightarrow\ \mathcal{F},\qquad \dim \mathcal{F} \sim 10\text{–}30. $$ The unity of the percept is the single-output property of $\Phi$: its codomain is one space, not five, so a point $\mathbf{f} = \Phi(\mathbf{t},\mathbf{a},\mathbf{x},\boldsymbol{\theta},\mathbf{v})$ carries no tag recording which input supplied which coordinate. This is the geometric content of “one percept from many senses,” and — as §12.4 will show — it is also exactly why the red dye works. Once the streams have been mixed by $\Phi$, no downstream process can un-mix them, because the mixing was a projection and projections are not invertible. The binding is the loss of the seam.
12.2 The Ventral Pathway as a Dimensionality-Reduction Pipeline
The map $\Phi$ is not a single operation but a pipeline, and its stages are anatomical. Following the aroma stream — the high-dimensional half of flavor — the signal leaves the ~400 olfactory receptor types and passes first to the olfactory bulb, where the axons of all neurons expressing a given receptor converge onto a small number of glomeruli. This is already a compression: the diffuse receptor sheet is reorganized into a spatial map of a few thousand glomeruli, a combinatorial image of the odorant (Malnic, Hirono, Sato & Buck, 1999).
From the bulb the signal reaches the piriform cortex, the primary olfactory cortex. Piriform is not a sensory map in the way primary visual cortex is; it is an association network, with broad, distributed, plastic connectivity. Here the combinatorial glomerular pattern is read out as a content-addressable representation: piriform behaves like an autoassociative memory, completing partial patterns and assigning odor objects rather than odor features (Wilson & Sullivan, 2011; Gottfried, 2010). The dimensionality has begun to collapse — from the receptor code toward a smaller set of recurring odor objects.
The decisive stage is the orbitofrontal cortex (OFC). The OFC is where the streams meet: it receives taste (via insula), retronasal smell (via piriform), texture and temperature (via somatosensory routes), and visual input, and it is the first region where single neurons respond to combinations across modalities — a neuron that fires to the taste and the smell and the sight of a food, but not to any one alone (Rolls, 2015; Small & Prescott, 2005). The OFC is, in the language of this book, the locus of $\Phi$: the place where the product space is mapped to the joint flavor space.
[Empirical.] The compression along this pathway is steep and measurable. The input is on the order of $10^4$ volatile compounds; the receptor layer is ~400; the bulb’s glomerular map is a few thousand spatial channels; and the perceptual output — the number of independent flavor dimensions a trained taster can reliably report — is on the order of tens. Each arrow discards information: $$ \underbrace{10^4\ \text{compounds}}_{\text{world}} \to \underbrace{\sim\!400\ \text{receptors}}_{\text{epithelium}} \to \underbrace{\text{bulb}}_{\text{glomeruli}} \to \underbrace{\text{piriform}}_{\text{odor objects}} \to \underbrace{\text{OFC}}_{\Phi} \to \underbrace{\sim\!10\text{–}30\ \text{dims}}_{\mathcal{F}}. $$
[Modeling Axiom.] We model the pipeline as a composition of maps, each reducing dimension and folding in a new modality: $$ \Phi\ =\ \pi_{\text{OFC}} \circ \pi_{\text{piriform}} \circ \pi_{\text{bulb}}, $$ with $\pi_{\text{OFC}}$ taking the additional taste, texture, temperature, and visual inputs as side channels. Two structural facts follow, and both will matter. First, because each stage is a many-to-one map, the whole is non-invertible: the percept cannot be decomposed back into its sensory sources, which is the abstract form of oral referral and of the wine dye. Second, because the integrating stage (OFC) is also the brain’s principal site for value and reward (Rolls, 2015), the flavor space $\mathcal{F}$ is delivered to consciousness already carrying a hedonic gradient (Chapter 3.6) — flavor and its pleasantness are computed in the same place, by the same neurons, which is why we cannot perceive a flavor without simultaneously liking or disliking it.
12.3 Cross-Modal Correspondences
If the senses are projected into a shared space, the projection should leave fingerprints: regularities by which a feature in one modality reliably maps to a feature in another, because both land near the same region of $\mathcal{F}$. Charles Spence and colleagues have catalogued exactly these crossmodal correspondences, and they are startlingly consistent across people (Spence, 2011; Spence, 2015).
[Empirical.] Sweetness is associated with red and pink, with round shapes, with higher musical pitch, and with creamy, legato timbres; bitterness is associated with dark colors, with angular and spiky shapes, with lower pitch, and with dissonance; sourness with yellow-green, high pitch, and sharp angles; carbonation with fast, staccato sound. These are not idiosyncratic. In a now-classic demonstration, the great majority of participants match a rounded form to one nonsense word and an angular form to another (the “bouba/kiki” effect, Köhler 1929; Ramachandran & Hubbard 2001), and the same rounded-versus-angular axis predicts sweet-versus-bitter matches. Spence’s group has shown that these are not parlor tricks: serving the same dessert from a round white plate versus an angular black one shifts its rated sweetness, and tasting against a “sweet” soundtrack raises perceived sweetness over a “bitter” one (the “sonic seasoning” effect).
The geometric reading is direct. A crossmodal correspondence is the statement that two inputs from different modalities project to nearby points in $\mathcal{F}$. If $\Phi(\text{red}) \approx \Phi(\text{sweet})$ in the relevant coordinate, then redness and sweetness are, in the integrated space, almost the same direction, and the brain — having only the projected point to work with — cannot tell a contribution from the eye apart from a contribution from the tongue along that axis. The correspondences are the low-dimensional structure of $\mathcal{F}$ made visible across senses: they reveal that the shared space has axes (rounded↔angular, light↔dark, high↔low) onto which every modality projects, so that “sweet,” “red,” “round,” and “high-pitched” are not five facts but one location.
[Conjecture.] The correspondences are learned statistics of the world, encoded as the geometry of $\Phi$. Ripe, sugar-rich fruit is statistically redder and rounder; bitter, alkaloid-bearing plants are statistically darker and more angular. If so, the shared axes of $\mathcal{F}$ are not arbitrary: they are the principal directions along which the sensory streams co-vary in natural foods, and the brain has aligned its integration map to them. The same logic that makes shared volatile compounds predict pairing (Chapter 4) makes shared environmental statistics predict crossmodal correspondence — both are the geometry of co-occurrence.
THE RED-DYED WINE, REVISITED — A CORRESPONDENCE WEAPONIZED
Brochet’s dye is a crossmodal correspondence turned into an attack. Deep red projects, in $\mathcal{F}$, onto the directions of dark fruit, tannin, and barrel — the directions a red wine actually occupies. The white wine’s true point sits elsewhere, among citrus and floral coordinates. When the eye delivers “red” and the nose delivers “white,” $\Phi$ must combine them, and because the visual channel into the OFC is strong and the correspondence is steep, the integrated point is dragged toward the red-wine region. The taster reads out that integrated point — and reports cherry and cedar, in perfect honesty. The vocabulary follows the geometry. Spence’s plates and soundtracks nudge the point by a little; Brochet’s dye, exploiting the most metrically aligned correspondence of all, nudges it across a category boundary.
12.4 The McGurk Effect for Flavor
The audiovisual McGurk effect (McGurk & MacDonald, 1976) is the canonical proof that multisensory integration is mandatory and pre-conscious: shown a face mouthing /ga/ while hearing /ba/, listeners perceive /da/ — a fusion that belongs to neither input, that they cannot suppress by knowing the trick, and that demonstrates vision editing audition before awareness. The red-dyed wine is the flavor analogue, and it makes the same three points.
[Empirical.] Morrot, Brochet & Dubourdieu (2001), reporting the result in Brain and Language, found that fifty-four oenology students described the dyed white wine using terms drawn overwhelmingly from the red-wine lexicon — and that the shift was not a relabeling of detected qualities but a change in the qualities detected: the olfactory descriptors used for the dyed white converged statistically on those used for genuine reds. In a companion line of work, the same mid-range wine poured from a grand-cru bottle versus a vin-de-table bottle drew correspondingly elevated or deflated descriptions. Visual input did not bias a verbal judgment downstream of a stable percept; it altered the percept.
The three McGurk lessons transfer exactly. Integration is mandatory: the tasters could not choose to ignore the color, just as you cannot choose to hear /ba/ when watching /ga/. Integration is pre-conscious: the editing happens in $\Phi$, at or before the OFC, so what reaches awareness is the already-fused point — there is no un-dyed percept to compare against. The fused percept can belong to neither input: just as /ba/+/ga/ yields /da/, white-aroma + red-color can yield a percept that is neither the honest white nor a true red but a blend located between them, dragged toward red by the strength of the visual channel.
This is precisely the non-invertibility of §12.2 cashed out as a phenomenon. Because $\Phi$ is a projection, the integrated point $\mathbf{f} = \Phi(\dots,\mathbf{v})$ does not record that $\mathbf{v}$ was a lie. Downstream cognition has access only to $\mathbf{f}$, not to the pre-image, so it cannot subtract the visual contribution even in principle. [Theorem.] If $\Phi$ is non-injective, then no function of $\mathbf{f}$ alone recovers the modality-of-origin of its coordinates. (Immediate: if two distinct inputs $u \neq u'$ satisfy $\Phi(u) = \Phi(u')$, any map of the output assigns them the same value, so the distinction between them — including which modality differed — is unrecoverable from the output.) The wine taster is not weak-willed; they are running a correct read-out of an irreversibly mixed signal. The dye is to flavor what the dubbed mouth is to speech.
12.5 Attention as Projection
If integration is mandatory and the senses are fused into one point, how can a sommelier nonetheless “concentrate on the finish,” or a taster “ignore the color and judge the nose”? The answer is that attention does not un-do $\Phi$ — it cannot, by the theorem above. Attention operates after integration, on the point $\mathbf{f} \in \mathcal{F}$, and it acts by projection onto a subspace.
[Modeling Axiom.] To attend to a flavor component is to apply a projection operator $P_S:\ \mathcal{F} \to S$ onto a subspace $S \subseteq \mathcal{F}$ spanned by the attended dimensions: $$ \mathbf{f}\ \longmapsto\ P_S\,\mathbf{f},\qquad P_S^2 = P_S. $$ Attending to acidity projects onto the sour/bright axis; attending to “the oak” projects onto the relevant aroma coordinates; attending to mouthfeel projects onto the texture block $\mathcal{X}$. The complementary projection $(I - P_S)$ is the suppressed remainder. This is the formalization promised in Chapter 1.3, where we noted that capsaicin “compresses the perceptual budget,” and it explains both what attention gains and what it costs. The gain is resolution: within the attended subspace $S$, small differences that were lost in the full norm become salient, because the metric of $\mathcal{F}$ restricted to $S$ magnifies them. The cost is the discarded complement: a taster fixated on tannin genuinely perceives less of the fruit, because the components along $(I - P_S)$ have been projected out of the report.
This reframes expert tasting as a disciplined sequence of projections — deconstruction of the integrated point by examining it one subspace at a time. The wine professional’s structured protocol (appearance, then nose, then palate, then finish) is a schedule of projection operators $P_{\text{visual}}, P_{\text{aroma}}, P_{\text{taste}}, P_{\text{temporal}}$ applied in turn to the same fused $\mathbf{f}$. Crucially, attention’s power is bounded by integration: the sommelier can project $\mathbf{f}$ however they like, but cannot recover the pre-image, and so cannot recover the un-dyed percept. They can attend to the color and consciously discount it — but the dye has already moved $\mathbf{f}$, and projecting the moved point onto the aroma subspace still yields the moved coordinates. This is why expert tasters who know Brochet’s experiment still fail it: attention is post-integration, and the dye’s damage is done before.
12.6 Memory and Flavor: The Madeleine as Associative Retrieval
The most famous sentence in the literature of flavor is Proust’s, and read carefully it is a precise claim about geometry. A spoonful of tea with a crumb of madeleine, and “the whole of Combray and its surroundings” rises into being — not as a fact recalled but as an involuntary flood, an entire remembered world unfolded from a single gustatory-olfactory cue. Why flavor, of all senses? And why so total a retrieval from so small a key?
[Empirical.] The phenomenon is real and modality-specific: odor- and flavor-evoked autobiographical memories are more emotional, more vivid, and older than memories cued by words or images — the “Proust effect” (Herz & Schooler, 2002; Chu & Downes, 2000). The anatomy explains the privilege. The olfactory pathway is the only sensory system with direct projections to the amygdala and hippocampus, bypassing the thalamic relay that gates the other senses; piriform cortex and the limbic memory structures are neighbors and partners. Smell is wired into memory in a way sight and sound are not.
The geometric model is the one developed for Volume 6, Geometric Cognition, which represents memory as an associative, content-addressable system: memories are points (or attractor basins) in a representational space, and retrieval is pattern completion — a partial cue settling, by the dynamics of an autoassociative network, into the nearest stored attractor. Recall the piriform cortex of §12.2, which we already described as behaving like an autoassociative memory (Wilson & Sullivan, 2011). The same machinery that completes a partial odor into a whole odor object completes a partial experience into a whole remembered episode.
[Modeling Axiom.] Let an autobiographical memory be a stored point $\mathbf{m} \in \mathcal{M}$ in an experiential space $\mathcal{M}$ that contains the flavor coordinates as a subspace, $\mathcal{F} \subseteq \mathcal{M}$. Encoding binds a flavor $\mathbf{f}^\ast$ to the rest of an episode — the place, the people, the feeling — as one joint point $\mathbf{m}^\ast = (\mathbf{f}^\ast, \text{context})$. Retrieval is the completion map $$ \mathbf{f} \approx \mathbf{f}^\ast\ \xrightarrow{\ \text{pattern completion}\ }\ \mathbf{m}^\ast, $$ which takes a present flavor near the stored flavor coordinate and returns the entire attractor, context and all. This is why the madeleine returns Combray and not merely “the taste of a madeleine”: the flavor was never stored alone. It was stored as one face of a high-dimensional episodic point, and supplying that face completes the whole — exactly the autoassociative recall of Volume 6, with the flavor space $\mathcal{F}$ of this volume as the cue subspace.
Two consequences close the loop. First, the madeleine’s potency is the non-invertibility of §12.2 working in our favor: because $\Phi$ binds the senses into one point, a flavor cue is also a multisensory cue, and the completion is rich because the binding was total. Second, the same associative geometry underwrites the dye of §12.4: the dyed wine recruits red-wine memories, so the taster’s red-wine vocabulary is not invented but retrieved — completed from a cue the eye corrupted. Memory, correspondence, integration, and the wine illusion are one phenomenon seen from four sides: the geometry of a space in which co-occurring things become near, and nearness becomes destiny.
From the Brain Back to the Plate
This chapter has reversed the book’s usual direction. Where Chapters 1 through 11 took the flavor space $\mathcal{F}$ as given, this chapter asked where $\mathcal{F}$ comes from — and found it to be the output of an integration pipeline $\Phi$ that projects five sensory streams, jointly and irreversibly, onto one low-dimensional manifold of experience. The binding problem (§12.1) is the single-output property of $\Phi$; the ventral pathway (§12.2) is its anatomy; crossmodal correspondences (§12.3) are the shared axes the projection exposes; the red-dyed wine (§12.4) is what happens when one input lies and the projection cannot tell; attention (§12.5) is the post-integration projection onto a subspace; and the madeleine (§12.6) is that same geometry running in reverse, connecting this volume to the associative memory of Volume 6: Geometric Cognition.
The lesson for the cook is the one Brochet’s tasters learned the hard way: you are not seasoning a chemical, you are seasoning a percept, and the percept is assembled from the plate, the glass, the light, the sound of the room, and the diner’s own buried memories. The color of the sauce, the weight of the fork, the music — these are not garnish. They are coordinates of $\mathbf{f}$, fed into the same $\Phi$ as the salt and the acid, and the integrated point is what the diner actually tastes. Chapter 13 turns this from diagnosis to design, asking how flavor houses and chefs deliberately navigate $\mathcal{F}$ — engineering not just compounds but the whole multisensory point — and what it means to solve the inverse problem: given a target experience, find the plate that lands there.