Chapter 23: Geometric Visual Art — Image-Embedding Clouds and the Geometry of Composition
RUNNING EXAMPLE — Hiroshi’s Acquisition
Hiroshi, senior curator at a mid-sized municipal museum, is three weeks from a board meeting at which he must defend six recommended acquisitions. Two of them are by artists the board has never heard of. He has written the usual connoisseur’s letters — formal analysis, biographical context, provenance notes — and he has rehearsed the usual defense: look at how the eye moves across the canvas, notice the weight of the left quadrant, feel the tension against the diagonal. He has said these things hundreds of times in his career. He also suspects they are true. What he does not have is any way to show that they are true, other than by asking his board to trust his eye. One of the board’s trustees, a retired physicist, has begun politely asking whether connoisseurship “has any structure at all, or is it what we pay you to have, on faith.” Hiroshi would like, just once, to answer that question with something other than a raised eyebrow.
This chapter asks whether the framework we have developed for text (Chapter 20) and audio (Chapter 21) has a natural extension to the visual arts. We do not have, at the time of writing, the direct empirical results for paintings that we have for books and tracks. What we have is a tightly constrained prediction: if the framework captures something real about aesthetic geometry, then replacing the LaBSE-encoded paragraph cloud with a CLIP- or DINOv2-encoded patch cloud should recover the same four structural channels — divergence, internal coherence, trajectory, and Lasso-discovered latent axes — with signs characteristic of a static, contemplative medium. The chapter sets out what the pipeline looks like, what the framework predicts, and what would refute it. We are careful here. The sign-flip between books and music (Chapter 21) is the single most important result in this book, and its generalization to painting is a hypothesis, not a report.
23.1 The Pipeline Analog
For text, the unit of analysis was the paragraph: a semantically coherent chunk, encoded by LaBSE into a 768-dimensional vector, stacked into a per-work cloud of typically 30–300 points. For music, the unit was a 30-second clip, encoded by MERT-v1-330M into a 45 × 1024 token sequence. For painting, the natural unit is the patch.
A vision transformer such as DINOv2 (Oquab et al., 2024) or CLIP (Radford et al., 2021) tiles an input image into non-overlapping patches — typically 14 × 14 or 16 × 16 pixels — and emits one contextualized embedding per patch. A 518 × 518 input to DINOv2-ViT-L/14 yields $37 \times 37 = 1369$ patch tokens in 1024-dimensional space. A painting, cropped and resized to the model’s input resolution, becomes a cloud of 1369 points in $\mathbb{R}^{1024}$. This is the natural analog of the paragraph cloud. We do not claim it is the correct analog — we claim it is the analog the framework tells us to try first.
Definition 23.1 (Image-Patch Cloud). Let $I$ be an image and let $\phi: \mathcal{I} \to \mathbb{R}^{N \times d}$ be a patch-token encoder producing $N$ contextualized embeddings in $d$ dimensions. The image-patch cloud of $I$ is the finite multiset $\{\phi(I)_1, \dots, \phi(I)_N\} \subset \mathbb{R}^d$. The painting manifold $M_{\text{paint}}$ is the corpus-level geometry induced by the patch clouds of a reference collection — for example, the open-access portions of the Metropolitan Museum, the National Gallery of Art, and Wikidata-linked museum holdings.
The corpus prior is the mean and covariance of all patch embeddings in the reference collection, computed either globally or per-genre (Byzantine icon, Dutch still life, Color Field painting, contemporary photography). The per-painting cloud is then evaluated against this prior exactly as the per-book paragraph cloud was evaluated against the Gutenberg corpus prior in Chapter 20.
The Four Channels, Instantiated for Painting
A. Spectral divergences. For each painting, fit a Gaussian approximation to its patch cloud and compute KL, JS, Hellinger, Bhattacharyya, Mahalanobis-mean, and TV divergences against the museum-wide patch prior. A high Hellinger distance means the painting occupies a region of visual representation space that the corpus on average does not — Chardin against a wall of Rubens, Agnes Martin against a wall of Abstract Expressionism. This is the painting analog of the 8σ divergence channel we observed for books (Chapter 17, §17.4).
B. Internal patch-coherence. Compute $\text{pair\_sim\_mean}$ and $\text{pair\_sim\_std}$ over patch embeddings within a painting. High pair_sim_mean means the painting’s patches are internally consistent: a Rothko color field, a Morris Louis veil, an Agnes Martin grid. Low pair_sim_mean means the painting contains sharply differentiated regions: a Bosch triptych, a Peter Blake collage, a Kehinde Wiley figure against ornamental ground. This channel is the novel finding of Chapter 17 — coherence is distinct from divergence — and we predict it is also the most load-bearing channel for painting.
C. Trajectory geometry. Paintings are not, on their face, temporal objects. But they are scanned: the eye moves. Recent saccade-prediction models (Jiang et al., Kümmerer & Bethge, Deep Gaze IIE) emit a saliency-ordered sequence of fixations for any input image, and with this ordering the patch cloud becomes a patch trajectory. Compute step_mean, step_std, recur_rate, path_eff, powerlaw_slope — exactly the features of Chapter 17, §17.4, now defined over the gaze-induced order. A painting with a well-composed entrance and a clear terminal fixation will have low path_eff and a characteristic powerlaw tail. A painting with no visual hierarchy will scan as a random walk through its own patches.
D. Lasso on latent directions. Run an L1-penalized regression from a 128-dimensional PCA of the patch-cloud mean against a curatorial target (acquisitions price, critical rating aggregated from published reviews, museum-visit dwell time). The surviving non-zero coefficients are the framework’s proposal for the genre-and-form axes of painting — the visual equivalents of the 71 interpretable axes we recovered for literature.
What the Rating-Analog Is
For books, the rating was Goodreads aggregate. For music, it was log listen count. For painting, there is no equally clean target. The four natural candidates, in descending order of cleanliness, are:
- Auction hammer prices, which have the advantage of being numerical and public but the disadvantage of confounding aesthetic judgment with market dynamics, provenance, and speculation;
- Museum wall time, the number of days per decade a work is on view rather than in storage, which is cleaner but noisy and highly institution-specific;
- Aggregate critical scores from published criticism, which is the closest analog to Goodreads but is sparse and not well standardized;
- Controlled human ratings collected specifically for this study, which is what we would want and do not yet have.
None of these is as clean as the Gutenberg-Goodreads match. We flag this as the first and largest empirical gap of the chapter.
23.2 What the Framework Predicts
The central empirical finding of this book (Chapter 17, §17.6) is that aesthetic geometry has directionality: the same geometric feature has opposite signs for text and for music. Books reward internal coherence ($\rho = +0.126$, 8.4σ) and small trajectory steps ($\rho = -0.096$, 6.4σ). Music rewards the opposite ($\rho = -0.076$ for coherence, $p = 5 \times 10^{-33}$; $\rho = +0.071$ for step size, $p = 4 \times 10^{-29}$). Books are continuity-rewarding. Music is contrast-rewarding.
Prediction 23.1 (Painting Aligns with Books). Easel painting — the genre of Western oil-on-canvas works from roughly 1420 to the present — should exhibit the book-like sign pattern: higher internal patch-coherence predicts higher aesthetic rating; smaller gaze-induced step sizes predict higher rating. The reasoning is that painting, like literature, is a sustained contemplative medium whose reception unfolds on a time scale in which the observer actively constructs unity from parts. Music is heard once through at the medium’s chosen pace; a novel is read at the reader’s pace; a painting is stood in front of at the viewer’s pace. Painting and literature share this structural feature, which we conjecture is what the sign-flip tracks.
This prediction is the chapter’s load-bearing claim. It is falsifiable. If a careful, genre-controlled study of patch-cloud features against any of the four rating analogs above produces the music sign — higher contrast, bigger steps, lower coherence predicting higher judgment — the prediction fails. The framework would not be destroyed by such a finding, but the simplicity of the coherence/contrast dichotomy would be, and the chapter would have to be rewritten to account for a three-way partition (text, music, painting) rather than a two-way one.
Prediction 23.2 (Genre Confound Will Be Severe). In Chapter 17, §17.4, 85% of the books’ headline $R^2$ dissolved under within-genre residualization; in music (§17.6), 91% dissolved. Genre — whether a book is fantasy or memoir, whether a track is rock or classical — is a first-order confound. The same should hold, and more so, in painting, where genre distinctions (icon, still-life, portrait, history painting, color field, photography) are themselves visually encoded: a DINOv2 can identify the genre of a painting from its patch cloud with near-perfect accuracy. We therefore predict that naive, unresidualized correlations between patch-cloud features and acquisitions price will be large and misleading, and that the honest residualized signal — pooled within-genre — will be on the order of $R = 0.1$–$0.2$, consistent with the books and music results. Any future study that reports an unresidualized $R = 0.5$ and calls it discovery is, we predict, reporting genre.
Prediction 23.3 (Divergences Will Saturate in Small-Sample, High-Dimensional Fits). The Hellinger saturation we encountered for music (Chapter 17, methodological notes) — 45 tokens in 128-dimensional space is ill-conditioned for Gaussian fitting — will recur for painting whenever per-image patch counts are low or dimensions are not PCA-reduced. Practitioners should operate in a $K \leq 32$ PCA subspace and prefer Bhattacharyya to Hellinger, exactly as we concluded for music. This is a methodological prediction, not an aesthetic one, but it is equally falsifiable.
23.3 What Would Refute the Framework in Painting
The framework is refuted in painting if any of the following occurs under a properly controlled study (author- or artist-disjoint folds, per-genre residualization, pre-registered feature set, $n \geq 2{,}000$ works spanning at least five genres).
First, if none of the four channels survive residualization — if divergence, coherence, trajectory, and Lasso-latent features all collapse to null once genre is partialed out — the claim that aesthetic judgment has structure above genre in this modality is false. The framework would retain its books and music results but lose the painting generalization.
Second, if the sign of the coherence channel matches music rather than books, the coherence/contrast dichotomy is wrong. We would then need a three-way theory (literature, painting, music) rather than a two-way one, and we would have to explain what feature of painting makes it contrast-rewarding. This would not be fatal to the framework but it would be fatal to the compact version of it that Prediction 23.1 commits to.
Third, if the Lasso on the 128-d PCA spectrum fails to recover interpretable genre/form axes — if the selected directions are visually uninterpretable even when inspected via top-weighted images — the claim that the latent geometry has human-readable structure would be weakened. We note that this would be consistent with the framework capturing something real that is, however, not what human critics are tracking.
Fourth, if cross-cultural invariance fails — if the geometry recovered from a Western museum corpus does not transfer to a Japanese woodblock corpus, a Mughal miniature corpus, or a Yoruba mask corpus, after controlling for style — then the translation-invariance result of Chapter 12 does not generalize to visual art. We suspect it does generalize, but the book corpus evidence ($\rho = +0.712$ for pair_sim_mean across 19 languages and 10 families) gives us no direct purchase here.
23.4 Connection to Existing Art-Theoretic Literature
The predictions above are not free-standing. They are in dialogue with three traditions.
Arnheim and Gestalt visual balance. Rudolf Arnheim’s Art and Visual Perception (1954, revised 1974) argues that a painting is a dynamic equilibrium of perceptual forces — weights, directions, tensions — organized around a structural skeleton. Arnheim’s system is geometric in an informal sense: it posits that compositional success has measurable structure, and that trained viewers can identify that structure. In the framework of this book, Arnheim’s visual weights are an early, pre-computational attempt to articulate what the internal-coherence channel (B) and the trajectory channel (C) capture quantitatively. Our pair_sim_std, for instance, is the machine-learned analog of Arnheim’s distinction between unified and unbalanced compositions. Our path_eff is the analog of his structural net. Arnheim wrote, memorably, that “a work of art is not a shape but a shape being lived.” The gaze-induced trajectory channel is one way to operationalize that claim.
Greenbergian formalism. Clement Greenberg’s formalist criticism, particularly in Art and Culture (1961) and the late essays on flatness and medium-specificity, insists that what is aesthetically decisive in painting is internal to the painting’s own formal organization — not its subject matter, not its historical context, not its narrative content. The framework is, at a philosophical level, a Greenbergian view implemented in a learned representation space: the patch-cloud divergence and coherence channels measure structural-formal properties of the image that are insensitive to caption, title, and context. A study that found these channels to be genuinely predictive would be partial vindication of Greenberg’s claim that formal properties carry aesthetic signal. Conversely, if the only channel that survives is a content-tracking Lasso axis (e.g., “is this image a landscape”), the result would be anti-Greenbergian: what matters is subject, not form.
Rothko, Reinhardt, and the color-field reception. Mark Rothko’s large color-field paintings — the Seagram murals, the Houston Chapel paintings — were designed to be stood in front of at close range for extended periods. Rothko claimed that viewers who wept in front of his paintings were having “the same religious experience I had when I painted them.” Ad Reinhardt’s black paintings push the claim further: the work emerges only after sustained, coherent attention. These painters stake a claim that is almost the literal version of the internal-coherence channel (B): a work whose patches are nearly indistinguishable from one another, whose pair_sim_mean approaches unity, is not a failure of differentiation — it is the work. If Prediction 23.1 holds, the color-field tradition is the extreme end of the book-like aesthetic: maximum coherence, minimum trajectory variance, and high corpus-level divergence from a museum whose average is much more differentiated. The predicted rating effect is positive on all three channels.
We also note the opposite pole. Bosch, Bruegel, and the tradition of narrative-packed surface (continuing through R. B. Kitaj, Peter Saul, and contemporary maximalists) should land on the low coherence, high trajectory-variance side. The framework does not predict that this pole is aesthetically inferior — it predicts that such works must pay for their low coherence with something else: extraordinary corpus divergence (they are unlike the museum average) or exceptionally rich Lasso-discovered content axes. A Bruegel panel survives not by being patch-coherent but by being, in the PCA sense, simply elsewhere.
23.5 Hiroshi’s Defense, Reconsidered
Return to Hiroshi and his board meeting. What does the framework actually give him?
Not a score. The framework does not collapse to a scalar, and this chapter’s central theoretical claim is that any curator who defends an acquisition with a single number has misunderstood the geometry. What the framework gives Hiroshi is a vector of structural claims, each of which can be computed, each of which can be interrogated, and each of which the board can choose to weight as it prefers.
For each of his six acquisitions, Hiroshi can report: where the work sits in the museum’s corpus divergence distribution; how internally coherent its patch cloud is relative to peer works; what its gaze-induced trajectory looks like against the collection’s median; which of the Lasso-discovered latent axes it loads on most heavily; and — if he has a rating model trained on the museum’s own acquisitions history — what that model predicts, along with the residual uncertainty the model itself admits. The physicist-trustee will recognize the form: it is error bars on an estimator. The question “is this painting good?” becomes “does this painting occupy a region of geometric space that this collection has historically rewarded, and if not, can you justify the departure?” That is a question curators already answer implicitly. The framework lets them answer it explicitly, and in a form that admits disagreement about weighting rather than disagreement about taste.
None of this replaces Hiroshi’s eye. But it gives his eye a language.
23.6 Honest Acknowledgment of the Empirical Gap
We have not run this pipeline. The books pipeline (Chapter 17, Phase 1–3) took eighteen months and required a matched Gutenberg–Goodreads corpus of 4,998 books and a cross-lingual extension with 4,683 non-English works. The music pipeline (Phase 4) required FMA Medium’s 24,801 tracks with artist-disjoint folds. The painting analog requires: a large, rights-cleared, metadata-rich image corpus; a rating signal cleaner than the four candidates listed in §23.1; an author- or artist-disjoint fold structure that respects period and genre; and compute budget for DINOv2 or CLIP inference over the full corpus. None of these is unobtainable. Several art-historical research groups (the Frick Digital Art History initiative, the PHAROS consortium, ImageNet-WikiArt crosses) have begun assembling the necessary infrastructure. But the study does not exist, at the time of this writing, and we do not write as if it does.
We commit, instead, to the following honesty condition. If the study is run by us or by any other group, and the sign of the coherence channel comes out opposite to Prediction 23.1 — that is, if painting behaves like music rather than like books — we will say so, and we will revise this chapter to reflect the finding, and the theoretical coherence/contrast dichotomy of §17.6 will have to be restated as a property specifically of books-versus-music rather than of sustained-contemplation-versus-temporal-flow. The chapter is written to be readable after such a revision; nothing in it depends on the prediction being confirmed. What depends on the prediction being confirmed is the simpler, stronger claim that the framework makes a sharp, advance-stated, empirically bite-sized commitment about a modality we have not yet studied.
23.7 Bridge to Chapter 24
Painting is the static-visual case. Architecture — the subject of the next chapter — is the visual case in three dimensions and at the scale of the built environment, where the viewer is not in front of the work but inside it. If painting is a patch cloud, architecture is a panoramic or point-cloud sweep; if painting’s trajectory is a saccade path over a canvas, architecture’s trajectory is a promenade through rooms. We expect continuity: the internal-coherence channel should operate in built space much as in painted space. We also expect one sharp new prediction specific to architecture, namely that proportion systems (Palladio, Le Corbusier’s Modulor, the golden section) are constrained-trajectory patterns in representation space rather than free proportions. We turn to that now.
For the mathematical preliminaries underlying the patch-cloud construction, see Chapter 4. For the tensor hierarchy into which painting’s five levels (patch, region, composition, work, corpus) fit, see Chapter 6. For the empirical evidence that grounds the book-like sign prediction of §23.2, see Chapter 17, especially §17.6 on cross-modality sign flips.