Characterizing Universal Object Representations Across Vision Models
arXiv:2605.13675v1 [cs.CV] 13 May 2026
Florian P. Mahner∗ Vision and Computational Cognition Group Max Planck Institute Justus-Liebig-University Giessen [email protected]
Johannes Roth∗ Vision and Computational Cognition Group Max Planck Institute Justus-Liebig-University Giessen
Ka Chun Lam Machine Learning Core National Institute of Mental Health Bethesda, MD, USA Francisco Pereira Machine Learning Core National Institute of Mental Health Bethesda, MD, USA
Michael F. Bonner Department of Cognitive Science Johns Hopkins University Baltimore, MD, USA Martin N. Hebart Vision and Computational Cognition Group Max Planck Institute Justus-Liebig-University Giessen
Abstract Deep neural networks trained with different architectures, objectives, and datasets have been reported to converge on similar visual representations. However, what remains unknown is which visual properties models actually converge on and which factors may underlie this convergence. To address this, we decompose the object similarity structure of 162 diverse vision models into a small set of nonnegative dimensions. To determine universal versus model-specific dimensions, we then estimate how often each dimension reappears across models. In contrast to model-specific dimensions, universal dimensions are more interpretable and more strongly driven by conceptual image properties, indicating the relevance of interpretability and semantic content as implicit factors driving universality across models. Differences in architecture, objective function, training data, model size, and model performance do not explain the emergence of universal dimensions. However, models with more universal dimensions also better predict macaque IT activity and human similarity judgments, suggesting that universality reflects representations relevant to biological vision. These findings have important implications for understanding the emergent representations underlying deep neural network models and their alignment with biological vision.
1
Introduction
Deep neural networks have become central to visual representation learning [1] and are increasingly used as computational models of the visual system [2–4]. Recent evidence suggests that deep neural networks trained on visual tasks converge on a common representational structure, despite substantial differences in architecture, objective function, and training data [5, 6]. This shared structure aligns more closely with neural responses in human visual cortex than model-specific structure [7, 8], suggesting that these recurring representations may matter for biological vision, as well. ∗ Equal contribution.
Preprint.
a Extract interpretable embeddings from vision models Stimuli
Symmetric NMF
Model embeddings
...
BiTExpert ViT-16 AlexNet RN50
Pairwise similarities
Penultimate representations
...
162 pretrained vision models
RBF
...
c Universality across models and dimensions
b Quantifying universality
...
.91 .35 .04
k=2 .82 .85 .12 .28
.08
.91 .35 .04
.82 .85 .12 .28 .85 .85 .05
k=4 .03 .11 .57 .19
.03 .11
k=4 0.57
M Re
.57 .19
j=1 j=2 j=3 j=4
...
j=1 j=2 j=3 j=4
.73
k=3 0.73
/16 ViT B IPCL
as
k=3 .85 .85 .05 .73
k=2 0.82
Un ive rs
k=1 .08
al
Universal
Predict and explain
ecific l-sp Mid
j=1 j=2 j=3
Wm ′
k=1 0.91
ing ad sh
Wm
Find unique match
e od
k=1 k=2 k=3
Compute similarity
O-ResNet 50 DIN
Embedding pair
0.0
0.2
0.4
...
Universality score
Macaque IT
Human behavior Model-specific
semantic visual mixed
Crowdsourcing
Figure 1: Overview of the analysis pipeline and universality framework. (a) For each of 162 vision models, we extract penultimate-layer representations for object images, compute pairwise object similarity matrices, and apply symmetric nonnegative matrix factorization to obtain non-negative embeddings. (b) We compute a universality score that quantifies how consistently each dimension of a model’s embedding reappears across all other models. (c) Left: distribution of per-dimension universality scores across all 162 models and 50 dimensions per model. Right: universal and modelspecific dimensions are used to predict neural activity in macaque IT cortex and human similarity judgments, and to label the dimension content via online crowdsourcing. However, what properties these shared representations capture and what factors determine their emergence remain poorly understood. First, it is unclear whether universal representations reflect high-level conceptual properties such as object categories, or perceptual image properties such as edges and colors that have long been known to be shared across models [6, 9]. Second, prior work has shown that training data and objective often matter more than architecture for alignment with brains and behavior [10, 11], but which of these factors drive universality across a large and diverse set of vision models, and how universal properties relate to biological vision, has remained open. Addressing these questions requires us to overcome several challenges. First, we need to make representations across a wide range of different model architectures directly comparable. Second, we need to find dimensions underlying these representations that decompose an image representation at a level of granularity suitable for cross-model comparisons, since overly fine-grained representations (e.g. individual neurons) lack a canonical correspondence across models. We address these questions by decomposing the internal representations of 162 diverse vision models [11] for 22,248 object images from the THINGS database [12] into a small set of non-negative dimensions, following cognitive-science work showing that human similarity judgments for natural objects are captured by a similarly small, interpretable, non-negative representation [13]. We then quantify the universality of each dimension by measuring how consistently it recurs across models. Together, the loadings and universality score make it possible to characterize what properties make up universal and modelspecific dimensions, which inductive biases are related to universality, and how universality aligns with biological vision. Our contributions are as follows: • We find that vision models, despite their differences, share a set of interpretable universal dimensions. • We demonstrate that universal dimensions are more interpretable and more strongly explained by conceptual image properties than model-specific dimensions, which are less interpretable and more strongly driven by perceptual image properties. • By isolating the role of architecture, objective function, and training data, we find that no single inductive bias accounts for universality, nor does model size or downstream performance. • Instead, we show that universal dimensions are highly predictive of multi-unit activity in macaque inferior temporal cortex and of human behavioral similarity judgments, suggesting that universal dimensions share constraints with biological vision. 2
2
Related Work
Our work relates to two broad research lines, one on representational alignment, and one on conceptbased interpretability. Work on representational alignment has shown that vision models can differ substantially in what they learn: They rely more on texture than on shape relative to humans [14, 15], produce distinct intermediate representations across random seeds [16], and differ in overall representational geometry [17]. The Platonic Representation Hypothesis proposes that, despite differences in architecture, objective function, and training data, model representations become increasingly similar with scale and even across modalities [5], though recent work has questioned this and suggested a more nuanced picture [18–20]. Other work has examined a related question within vision, asking whether diverse models share universal structure and whether these shared components are especially relevant for neural alignment [6–8]. Across work relating models to neural responses and behavior, training data and objective function often matter more than architecture or scale once other factors are controlled [10, 11, 21–24]. These studies, however, mainly establish whether models overlap, not what properties the shared representations actually capture. Concept-based interpretability instead asks what models have learned in human-understandable terms [25, 26], rather than merely visualizing the most predictive image regions [27]. A useful distinction has been made between concept discovery, which extracts interpretable units from a representation, and concept importance estimation, which measures their contribution to model outputs [28]. Early methods tested whether models represented fixed human-labeled concepts [25, 26], while later work recovered concepts directly from activations [29–35]. More recent work used sparse autoencoders to describe a model by a large set of directions in its activation space, where each direction corresponds to a concept the model has learned [36–40], and Thasarathan et al. [41] extended this to multiple models by jointly training a single sparse autoencoder across a small set of pretrained models to recover a shared set of directions. These approaches operate in each model’s activation space, so comparing many models requires aligning spaces and matching across thousands of features. We therefore describe each model by the similarity relationships its activations induce over a fixed set of objects, and decompose these similarities into a small number of non-negative dimensions defined as loadings over the same stimuli. Most closely related to our work, Hebart et al. [13] showed that human similarity judgments can be explained by a sparse, non-negative embedding whose dimensions are reproducible, interpretable, and predictive of behavior. Mahner et al. [9] derived sparse, non-negative embeddings for a small set of deep neural networks and found that their dimensions were dominated by visual rather than semantic properties. We extend this line of work to a large sample of vision models and introduce a measure of how consistently each dimension recurs across them.
3
Methods
3.1
Notation
Let X = {x1 , . . . , xN } be a dataset of N object images, and let F = {f1 , . . . , fM } be a collection of M vision models. For an image xi ∈ X , each model fm ∈ F extracts a dm -dimensional feature vector from its penultimate layer, denoted as zm (xi ) ∈ Rdm . Note that the dimensionality dm may vary across models. We used the penultimate layer because it is closest to the behavioral output and serves as the final stage of representation, integrating high-level conceptual and fine-grained perceptual information to produce an output. For each model fm , we collected the representations of all images in X into a matrix Zm ∈ RN ×dm , where the i-th row is given by zm (xi )⊤ . 3.2
Dataset and Models
We used the THINGS image database [12], which spans 1,854 basic-level object categories. For each object category, we selected 12 image exemplars, yielding a dataset X with a total of N = 22,248 images. We chose THINGS since it was designed to span a broad range of object concepts, is not part of common training sets, and has associated neural and behavioral data, making it well suited for evaluating learned representations against biological vision. Using THINGS, we extracted penultimate-layer representations from M = 162 vision models applied to the images in X . Our model set F comprised those in Conwell et al. [11] (excluding IPCL models), plus six OpenCLIP ViTL/14 models [42]. The resulting set spans four architecture classes (99 convolutional, 51 transformer, 9 MLP-Mixer, and 3 hybrid), a range of objective functions (supervised classification, self-supervised learning, vision-language contrastive learning, and one untrained baseline), and diverse training 3
datasets (including ImageNet, ImageNet-21k, YFCC15M, LAION, and Taskonomy). The full list of models is provided in Table S1. This diversity enables controlled comparisons that disentangle the effects of architecture, objective, and training data on our universality metric. 3.3
Symmetric Nonnegative Factorization
To make dimensions comparable across models with different feature bases and interpretable in terms of the images that define them, we characterize each model by the similarity structure it induces over a common image set and decompose that structure into a small set of nonnegative dimensions (Fig. 1a). Generate representational similarity matrix Models in F produce features in spaces of different dimensionalities and different bases, and several approaches exist for comparing such representations [17, 43]. We work at the level of similarity structure [44], characterizing each model by the pairwise similarities it induces over the images in X . This level of description is invariant to rotations of each model’s feature basis and yields a common N × N matrix for every model in F. Because similarity is determined by the pairwise distances between image representations, feature dimensions that vary little across X contribute little to these distances, so the similarity matrix emphasizes the structure along which the images actually differ. Similarity is defined over the shared image set, so any dimensions we extract from these matrices are automatically indexed by the same images across models and can be compared directly. For each model fm , we computed a symmetric, entrywise ×N nonnegative similarity matrix Sm ∈ RN via a radial-basis function (RBF) kernel: ≥0 ∥z (x ) − z (x )∥2 m i m j [Sm ]i,j = exp − , 2 2σm
(1)
where σm is chosen per model to jointly maximize factorization stability and the explained variance of the low-rank reconstruction. Concretely, we searched over multipliers α of the median pairwise Euclidean distance d˜m [45], setting σm = α∗ · d˜m , where α∗ maximizes the harmonic mean of both criteria (Appendix B.1). We used the RBF kernel because it is a common choice for similarity measures [17], and it maps similarities to the interval [0, 1] and thereby supports interpretability for the subsequent nonnegative factorization step. Furthermore, the RBF kernel guarantees that Sm is positive semi-definite and nonnegative, both of which are required properties for symmetric nonnegative factorization (Eq. 2). Decompose similarity matrix into interpretable image embeddings We decomposed each similarity matrix into r non-negative dimensions using symmetric NMF [46]. For a chosen rank r, ×r we estimated a nonnegative embedding Wm ∈ RN ≥0 of all images in X by solving: min 1 Wm ≥0 2
2
Sm − Wm W⊤ m F.
(2)
Each image is represented by a row of the matrix Wm . Each column wm,k ∈ RN ≥0 defines a dimension: its entries provide a nonnegative loading for each object image, and the numeric weight shows how strongly that image contributes to dimension k. The nonnegativity constraint promotes an interpretable parts-based factorization [46]. We optimized Eq. 2 using block-successive upperbound minimization [47], following the authors’ extension (Appendix B.2), independently for each model and for ranks r ∈ {10, 30, 50, 100, 200}. For each model, rank, and candidate bandwidth multiplier, we ran B = 5 random initializations. The bandwidth was selected by the harmonic mean of factorization stability and explained variance (Appendix B.1); within the selected bandwidth, we retained the most central seed (Appendix B.2). For all model comparisons, we fixed rank to r = 50. In Appendix B.3, we show that our main results are insensitive to this choice. 3.4
Universality
We quantified how consistently a dimension recurs across models (Fig. 1b). Given a dimension k in model fm and a dimension j in model fm′ , we measured their agreement via squared cosine 4
similarity, which is scale-invariant, bounded, and consistent with the Frobenius-norm objective of symmetric NMF: s(wm,k , wm′ ,j ) = cos2 wm,k , wm′ ,j
=
⊤ (wm,k wm′ ,j )2 ∈ [0, 1]. ∥wm,k ∥2 ∥wm′ ,j ∥2
(3)
Because symmetric NMF factors are identifiable only up to permutation, we established correspondences between models through a one-to-one assignment. Unlike a greedy best-match, which can allow multiple target dimensions to map onto the same source dimension, this formulation enforces a ′ bijection across the full set of dimensions (Appendix C.1). Pr For each model pair (m, m ), we deter∗ mined the optimal permutation πm,m′ ∈ arg maxπ∈Pr k=1 s(wm,k , wm′ ,π(k) ) via the Hungarian algorithm and defined the per-dimension universality as: universality(m, k) =
X 1 s wm,k , wm′ ,π∗ (k) . M −1 ′
(4)
m ̸=m
As all dimensions are nonnegative, any two of them will share some positive cosine similarity even if unrelated, and denser dimensions will have a higher baseline than sparser ones. This could distort the ranking of universality across dimensions. We corrected for this by calibrating against a permutation null that shuffles the row order of Wm′ , destroying stimulus correspondence while preserving column structure (Appendix C.1). After calibration, values near 1 indicate dimensions that are consistently recovered across models, while values near 0 indicate dimensions with chance-level cross-model agreement.
4
Results
4.1
Universal dimensions are shared across vision models
We first computed universality scores across 8,100 dimensions from 162 models. Scores ranged from near chance (0.0003) to 0.54 (Fig. 1c), with the upper end reaching 64% of the within-model stability ceiling, estimated from independent runs of the same model (median ceiling 0.84, Appendix C.2). Most models contained both highly universal and model-specific dimensions, with 80% having dimensions in both the top and bottom quartiles. To confirm that these findings are not driven by the specific choice of images or models, we recomputed universality scores after (i) changing the image set, (ii) bootstrap-resampling the model set, and (iii) excluding entire model families. As an alternative image set, we used ObjectNet [48], which depicts objects in cluttered real-world scenes with varied viewpoints and rotations, in contrast to the object-centered images in THINGS. Universality scores recomputed from ObjectNet embeddings largely preserved the model ranking (ρ = 0.81). Bootstrap resampling to 20% of models (n = 32) yielded a highly consistent ranking (ρ = 0.97; Fig. S6a), and excluding model families also left the ranking largely intact (ρ = 0.83; Fig. S6b). These controls confirm that universality is a stable property of how models organize object representations (see Appendix C.3 for details). 4.2
Universal dimensions are biased towards interpretable, conceptual image properties
We next asked what image properties distinguish universal from model-specific dimensions (Fig. 2a). Some dimensions have high values for images sharing high-level object concepts such as animals, food, or household objects, while others group images by lower-level perceptual properties such as color or texture, and some show no clear pattern at all. To quantify the relationship between the dimension content and its universality, we asked crowd-sourced participants to label 1,059 representative dimensions, selected via clustering from the full set of 8,100, as semantic, visual, both, or neither (see Appendix H for details). Ratings were reliable, with median per-rater agreement with the majority label of 0.67 and median split-half agreement of 0.57. Binning the dimensions by their universality score revealed a sharp change in interpretability (Fig. 2b). At low universality, 78% of dimensions were labeled neither, indicating that model-specific dimensions largely lack interpretable structure. At high universality this fraction collapsed to 4%, indicating that universal dimensions were more interpretable to raters. Among the dimensions deemed interpretable, the content also varied with universality. The semantic and both labels together accounted for 80% of dimensions at 5
Figure 2: Universal dimensions are more driven by conceptual object properties, and modelspecific dimensions by visual properties or lack interpretable structure. (a) Representative model dimensions for each label category (columns) at low, mid, and high universality (rows). Each grid shows the top-weighted THINGS images for that dimension. (b) Proportion of dimensions assigned each human label (semantic, visual, mix semantic-visual, neither) across all universality bins. At low universality, 78% of dimensions are rated as uninterpretable (neither); at high universality, semantic and mixed dimensions together account for 80%. (c) Between- and within-category image loading variance across universality deciles, pooled over all 162 models. The most universal dimensions (top decile) are dominated by between-category variance (∼78%), while the most model-specific dimensions (bottom decile) are dominated by within-category variance (∼71%). (d) Category consistency (η 2 ) vs. universality for all dimensions (ρ = 0.63). (e) Reconstruction importance (∆R2 ) vs. universality. The near-zero pooled correlation (ρ = 0.01 across all dimensions; within-model median ρ = 0.08) indicates that model-specific dimensions contribute as much to a model’s similarity structure as universal ones. high universality, while the visual label remained flat across the universality range. Thus, universal dimensions capture high-level object concepts, whereas the interpretable model-specific dimensions reflect low-level visual features. Moreover, if universal dimensions capture conceptual object structure, we would expect them to show stronger organization around basic object categories than model-specific ones. Because THINGS provides 12 exemplars for each of 1,854 categories, we can decompose each dimension’s loading variance into between- and within-category components. High between-category variance indicates that the dimension groups images by object category, while high within-category variance indicates that it varies independently of category. We formalize this as category consistency η 2 , the fraction of variance explained by category membership (see Appendix D): SSbetween η2 = . (5) SStotal A high η 2 indicates grouping by object category, whereas a low η 2 reflects within-category variation in visual features such as texture, viewpoint, or color. Category consistency (η 2 ) correlates strongly with universality across all 8,100 dimensions (ρ = .63; Fig. 2d). To see how this relationship 6
a
b
Architecture
Taskonomy
SLIP
SSL
0.5
Universality
0.4 0.3 0.2 0.1 0.0 N
c
xe Mi
ViT
CN
r
bri Hy
d
n ma Se
tic
3D
2D
o Ge
tric me
Co
tive as ntr
No
n -c
on
tr.
CL Sim
R
SL
IP
CL
IP
ViT-L/14
Scale
Universality
0.5 0.4 0.3 0.2 0.1 0.0
ag Im
eN
1K etag Im
eN
21 et-
K 40 NIO LA
0M taC Me
ll -Fu LIP
CC
2B NIO LA
taC Me
-40 LIP
0M e Op
0 I-4 nA
0M
2B NDF
ta Da
Co
-XL mp
Figure 3: Controlled comparisons of per-dimension universality. Each panel varies one factor while holding the others approximately constant. Box plots show the distribution of per-dimension universality scores; individual dots are dimensions, colored by model. The dashed line and gray band indicate the grand mean ± 1 SD across all models in that group. (a) Architecture: 34 CNNs, 21 transformers, 6 MLP-Mixers, and 3 hybrids, all trained on ImageNet-1K classification. (b) Objective function: Taskonomy ResNet-50 encoders grouped by task cluster (n = 23, excluding two singleton clusters); contrastive vs. non-contrastive self-supervised ResNet-50 models (n = 10); SLIP ViT models trained with SimCLR, SLIP, or CLIP (n = 9). (c) Training data. Left: ImageNet-1K vs. ImageNet-21K, matched by architecture (n = 12). Right: seven ViT-L/14 CLIP models trained on different web-scale datasets.
changes across the universality spectrum, we binned all dimensions into deciles and computed the fraction of variance that falls between versus within categories in each bin (Fig. 2c). The most universal dimensions (top decile) are dominated by between-category variance (∼78%), while for the most model-specific dimensions (bottom decile), within-category variance dominates (∼71%). This confirms that category-level object structure, a key component of the semantic labels assigned by human raters, is strongly associated with universality. Universal dimensions thus capture conceptual object structure and are more interpretable, but does this mean they play a larger role in a model’s similarity structure? We computed each dimension’s reconstruction importance, the drop in variance explained in the similarity matrix when that dimension is removed. Universality and reconstruction importance show near-zero correlation (within-model median ρ = 0.08; Fig. 2e), indicating that universality reflects conceptual content but not contribution to similarity structure. 4.3
Model inductive biases do not determine universality
If universality is a general property of learned visual representations, it should emerge across different architectures, objective functions, and training datasets rather than only under a specific design choice. To assess this, we first computed a single universality score per model, averaging across its 50 dimensions. This makes it possible to compare models that differ in one design choice while holding the others approximately constant. Model-level universality showed no reliable correlation with ImageNet top-1 accuracy (ρ = −0.04, p = 0.72; n = 101) or number of parameters (r = 0.08, p = 0.31; n = 162), ruling out model performance or size as trivial explanations (Fig. S5). We tested each of the five pre-specified controlled contrasts with a single uniform procedure. Two-group contrasts were tested with Welch’s t, multi-group contrasts with one-way ANOVA, and contrasts whose groups were all singleton were reported descriptively, all with Bonferroni multiple comparison correction. We report Hedges’ g for two-group contrasts and ω 2 for multi-group contrasts with 95% confidence intervals. 7
0.3 0.2
55
0.35
50
0.30 0.25 0.20 0.15
0.1 0.2 Model universality
0.3
Universal half
50
45 40 35 30
0.10
r = 0.76
chance
45 40 35 30
chance
25 0.1 0.2 Model universality
Specific half
Triplet accuracy [%]
0.4
b
0.40 Triplet accuracy [%]
Encoding [Pearson r]
r = 0.74 Encoding [Pearson r]
a 0.5
0.3
Universal half
Specific half
Figure 4: Universality predicts neural and behavioral alignment. (a) Left: model universality vs IT encoding accuracy (mean across two macaques); Right IT encoding accuracy from the universal vs specific half of each model’s dimensions. (b) Left: model universality vs human triplet accuracy; Right: triplet accuracy from the universal vs specific half. Dotted lines are chance level (1/3). Architecture. We compared 34 CNNs, 21 transformers, 6 MLP-Mixers, and 3 hybrids, all trained on ImageNet-1K classification (n = 64). Architecture had no statistically significant effect on universality (F (3, 60) = 2.54, p = 0.33, ω 2 = 0.07 with 95% CI [−0.02, 0.29]; Fig. 3a), indicating that universality is not tied to a specific architecture class. Objective function. We analyzed 25 ResNet-50 encoders from the Taskonomy benchmark [49], each trained on a different visual task with identical architecture and training data. These models showed low universality overall (cluster means ranging from 0.05 to 0.12, well below the full-set median of 0.22), reflecting the narrow distributions of their training tasks. Nevertheless, within this group, task cluster had a clear effect on the relative level of universality after excluding two singleton clusters (F (3, 19) = 12.7, p < .001, ω 2 = 0.60; Fig. 3b, left). Semantic tasks (object/scene classification) produced the most universal dimensions, followed by 3D, Geometric, and 2D tasks. Across individual tasks, universality spanned a range of 0.125, from denoising at the bottom to object classification at the top. We next examined whether the choice of self-supervised learning objective affects universality. Among 10 self-supervised ResNet-50 models trained on ImageNet, contrastive objectives (SimCLR, MoCo, Barlow Twins, SwAV, DeepCluster) showed nominally higher universality than non-contrastive methods (RotNet, Jigsaw, ClusterFit), but the effect was not significant after correction (Welch t = 3.21, p = .065, Hedges’ g = 1.76; Fig. 3b, center). Finally, to isolate the effect of language alignment, we compared ViT models from the SLIP family [50], all trained on YFCC15M but with different objectives, SimCLR (purely visual), CLIP (vision-language), and SLIP (combined) (n = 9). Although CLIP models scored lower in universality than SimCLR and SLIP, the difference was not statistically significant (F (2, 6) = 3.80, p = .43, ω 2 = 0.38; Fig. 3b, right). Training data. We compared six models trained on ImageNet-1K with six trained on ImageNet21K, matched by architecture family [11]. Data scale had no effect on universality (Welch t = −0.33, p > .99, Hedges’ g = −0.18; Fig. 3c, left). To further isolate the role of training data, we compared seven ViT-L/14 CLIP models that share the same architecture and objective but differ in training dataset (OpenAI WIT, LAION-400M, LAION-2B, DataComp-XL, DFN-2B, MetaCLIP-400M, MetaCLIP-FullCC). Universality scores within this group varied significantly less than expected from a random sample of seven models (bootstrap test on within-group standard deviation with null drawn from all 162 models, one-sided p = .009, 10,000 resamples; Fig. 3c, right). Together, neither dataset scale nor training-data choice substantially changed universality. Overall, although training objective may slightly modulate universality in specific model subsets, no single inductive bias accounts for universality. Universality thus emerges across diverse design choices, suggesting that it reflects a general property of learned visual representations. 4.4
Universality predicts neural and human behavioral alignment
So far, the results have established that universal dimensions capture high-level conceptual object properties and that they emerge across diverse architectures, objective function, and training datasets rather than being driven by any single design choice. This leaves open why different models converge 8
on these particular dimensions. A plausible explanation is that universal dimensions reflect aspects of the visual world itself rather than consequences of any particular model design. If so, universal dimensions should align with biological visual representations, since both are shaped by the same visual world, whereas model-specific dimensions reflect idiosyncrasies of particular models. We tested this using neural recordings from macaque inferior temporal (IT) cortex and human behavioral judgments, both collected on the THINGS image set. Prediction of macaque IT neural recordings To quantify the alignment between model dimensions and neural activity, we predicted multi-unit activity in macaque IT cortex from each of the 162 models’ dimensions using cross-validated ridge regression (see Appendix E for details). Models with more universal dimensions also predict IT activity more accurately (r = 0.74, p < .001; Fig. 4a). To test whether this is driven by universal dimensions specifically, we split each model’s dimensions at the median universality score into a universal and a model-specific half. The universal half predicts IT activity significantly better than the model-specific half (paired t-test, p < .001). Prediction of human behavior To quantify the alignment between model dimensions and behavior, we evaluated the 162 models against a dataset of odd-one-out judgments [13], in which participants viewed triplets of object images and chose the image least similar to the other two. For each model, we predicted the human choice from pairwise cosine similarities between the embeddings Wm of the three images (see Appendix F for details). Models with more universal dimensions also predict human judgments more accurately (r = 0.76, p < .001; Fig. 4b), and the universal half of each model’s dimensions yields significantly higher accuracy than the model-specific half (0.42 vs. 0.37, chance = 0.33, paired t-test, p < .001).
5
Discussion and Conclusions
We decomposed the representations of 162 vision models into interpretable dimensions using symmetric NMF and quantified how consistently each dimension reappears across models. Across this diverse set of models, we found that many dimensions appear repeatedly. These universal dimensions are also the ones that are most interpretable to human observers and most strongly tied to conceptual, between-category structure. In contrast, model-specific dimensions display higher within-category variation, are less interpretable, and relate more to lower-level visual properties. Our controlled comparisons show that no single inductive bias – architecture, objective function, or training data – accounts for universality. Taskonomy models show uniformly low universality relative to the broader model zoo, though, within this family, semantic tasks produce more universal dimensions than geometric or 2D tasks. Instead, universality is more closely aligned with biological vision than with any inductive bias. Universal dimensions predict both neural activity in macaque IT cortex and human behavioral similarity judgments more accurately than model-specific ones. The very same dimensions that align with brains and behavior are also the ones that human raters find interpretable and label as semantic. Interpretability, conceptual content, and biological alignment thus co-occur in the structure that diverse models converge on. This suggests that universality is an emergent property of successful visual representation learning. A growing body of work has provided evidence for convergence and shared structure across diverse models [5, 7, 8, 18–20]. Our results are consistent with the view that there may be universal dimensions of visual representation, with diverse models converging on conceptual object properties. However, this convergence is far from complete. The most universal dimensions do not have a perfect universality score, and model-specific dimensions, which capture lower-level visual properties, contribute equally to each model’s representational geometry. This work addresses a basic question about visual representation learning: what structure diverse models converge on when they learn to represent objects. We find that this structure is conceptual, interpretable, and aligned with biological vision. Understanding why it emerges so consistently across very different architectures, objectives, and training regimes may help clarify the underlying principles shared by artificial and biological vision. More broadly, our approach generalizes beyond vision and can be applied to any set of models that process a common set of inputs, offering an approach for understanding what is shared and what is specific across learned representations. 9
6
Limitations
Several limitations and open questions remain. Our model set is broad but biased toward systems suited to tasks that humans find useful, and may not characterize the full space of possible visual representations. The additive, non-negative decomposition also cannot recover features that encode concepts at multiple levels of abstraction [24] or are stored in superposition [39, 41], and other methods may reveal additional structure in the model-specific components. Additionally, we used penultimate-layer representations because they were closest to behavior, but representations from other layers might modulate universality differently. Whether universal dimensions can be used to improve vision models, for instance, through representation alignment [51] or universality-based regularization during training, is another open question worth pursuing.
Acknowledgments and Disclosure of Funding The authors thank Aria Wang and Drew Linsley for helpful feedback on the manuscript and KlausRobert Müller for fruitful discussions that motivated further analyses. Author contributions. F.P.M., and M.N.H. conceived the study. F.P.M., J.R., and F.P. designed the research. F.P.M. and K.L. developed the methodology and the mathematical framework. F.P.M. and J.R. conducted the experiments and performed the analyses. M.B., F.P., and M.N.H. provided critical feedback and contributed to the interpretation of results. F.P.M. and J.R. wrote the initial draft of the manuscript. All authors reviewed and edited the final manuscript. Funding. FPM, JR and MNH acknowledge support by a Max Planck Research Group grant of the Max Planck Society awarded to MNH. MNH was supported by the ERC Starting Grant COREDIM (ERC-2021-STG-101039712), a LOEWE Start Professorship by the Hessian Ministry of Higher Education, Research, Science and the Arts, and the Deutsche Forschungsgemeinschaft (German Research Foundation, DFG) under Germany’s Excellence Strategy (EXC 3066/1 “The Adaptive Mind”, Project No. 533717223). This study used the high-performance from the Raven and Cobra Linux clusters at the Max Planck Computing & Data Facility (MPCDF), Garching, Germany https://www.mpcdf.mpg.de/services/supercomputing/. KL and FP were supported in part by the Intramural Research Program of the National Institutes of Health (NIH) (ZIC MH002968). The contributions of the NIH author(s) were made as part of their official duties as NIH federal employees, are in compliance with agency policy requirements, and are considered Works of the United States Government. However, the findings and conclusions presented in this paper are those of the authors and do not necessarily reflect the views of the NIH or the U.S. Department of Health and Human Services. MFB acknowledges support through a Research Fellowship for Experienced Researchers from the Humboldt Foundation. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.
10
References [1] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521:436–444, 2015. [2] N. Kanwisher, M. Khosla, and K. Dobs. Using artificial neural networks to ask ’why’ questions of minds and brains. Trends in Neurosciences, 46:240–254, 2023. [3] A. Doerig, R. P. Sommers, K. Seeliger, B. Richards, J. Ismael, G. W. Lindsay, K. P. Kording, T. Konkle, M. A. J. van Gerven, N. Kriegeskorte, and T. C. Kietzmann. The neuroconnectionist research programme. Nature Reviews Neuroscience, 24:431–450, 2023. [4] D. L. K. Yamins and J. J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19:356–365, 2016. [5] M. Huh, B. Cheung, T. Wang, and P. Isola. Position: The platonic representation hypothesis. Proceedings of Machine Learning Research, 235:20617–20642, 2024. [6] F. Guth and B. Ménard. On the universality of neural encodings in CNNs. In ICLR 2024 Workshop on Representational Alignment (Re-Align), 2024. [7] E. Hosseini, C. Casto, N. Zaslavsky, C. Conwell, M. Richardson, and E. Fedorenko. Universality of representation in biological and artificial neural networks. bioRxiv preprint, 2024.12.26.629294, 2024. [8] Z. Chen and M. F. Bonner. Universal dimensions of visual representation. Science Advances, 11:eadw7697, 2025. [9] F. P. Mahner, L. Muttenthaler, U. Güçlü, and M. N. Hebart. Dimensions underlying the representational alignment of deep neural networks with humans. Nature Machine Intelligence, 7:848–859, 2025. [10] L. Muttenthaler, L. Linhardt, J. Dippel, R. A. Vandermeulen, K. Hermann, A. Lampinen, and S. Kornblith. Improving neural network representations using human similarity judgments. Advances in Neural Information Processing Systems, 36:50978–51007, 2023. [11] C. Conwell, J. S. Prince, K. N. Kay, G. A. Alvarez, and T. Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 15:9383, 2024. [12] M. N. Hebart, A. H. Dickter, A. Kidder, W. Y. Kwok, A. Corriveau, C. Van Wicklin, and C. I. Baker. THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLOS ONE, 14:e0223792, 2019. [13] M. N. Hebart, C. Y. Zheng, F. Pereira, and C. I. Baker. Revealing the multidimensional mental representations of natural objects underlying human similarity judgements. Nature Human Behaviour, 4:1173–1185, 2020. [14] R. Geirhos, K. Narayanappa, B. Mitzkus, T. Thieringer, M. Bethge, F. A. Wichmann, and W. Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885–23899, 2021. [15] K. L. Hermann, T. Chen, and S. Kornblith. The origins and prevalence of texture bias in convolutional neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 19000–19015. Curran Associates, 2020. [16] Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft. Convergent learning: Do different neural networks learn the same representations? In International Conference on Learning Representations, 2016. [17] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. Proceedings of Machine Learning Research, 97:3519–3529, 2019. [18] F. Gröger, S. Wen, and M. Brbić. Revisiting the platonic representation hypothesis: an aristotelian view. arXiv preprint, 2602.14486, 2026. [19] A. Kumar, J. Clune, J. Lehman, and K. O. Stanley. Questioning representational optimism in deep learning: The fractured entangled representation hypothesis. arXiv preprint, 2505.11581, 2025. [20] M. Tjandrasuwita, C. Ekbote, L. Ziyin, and P. P. Liang. Understanding the emergence of multimodal representation alignment. In International Conference on Machine Learning, 2025.
11
[21] A. Y. Wang, K. Kay, T. Naselaris, M. J. Tarr, and L. Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5: 1415–1426, 2023. [22] J. S. Prince, G. A. Alvarez, and T. Konkle. Contrastive learning explains the emergence and function of visual category-selective regions. Science Advances, 10:eadl1776, 2024. [23] T. Yerxa, J. Feather, E. P. Simoncelli, and S. Chung. Contrastive-equivariant self-supervised learning improves alignment with primate visual area IT. Advances in Neural Information Processing Systems, 37: 96045–96070, 2024. [24] L. Muttenthaler, K. Greff, F. Born, B. Spitzer, S. Kornblith, M. C. Mozer, K.-R. Müller, T. Unterthiner, and A. K. Lampinen. Aligning machine and human visual representations across abstraction levels. Nature, 647:349–355, 2025. [25] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: quantifying interpretability of deep visual representations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3319–3327. IEEE, 2017. [26] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). Proceedings of Machine Learning Research, 80:2668–2677, 2018. [27] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, volume 8689, pages 818–833. Springer, 2014. [28] T. Fel, V. Boutin, L. Béthune, R. Cadène, M. Moayeri, L. Andéol, M. Chalvidal, and T. Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. In Advances in Neural Information Processing Systems, volume 36, pages 54805–54818. Curran Associates, 2023. [29] A. Ghorbani, J. Wexler, J. Y. Zou, and B. Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, volume 32, pages 9273–9282. Curran Associates, 2019. [30] R. Zhang, P. Madumal, T. Miller, K. A. Ehinger, and B. I. P. Rubinstein. Invertible concept-based explanations for CNN models with non-negative concept activation vectors. Proceedings of the AAAI Conference on Artificial Intelligence, 35:11682–11690, 2021. [31] M. Graziani, A.-P. Nguyen, L. O’Mahony, H. Müller, and V. Andrearczyk. Concept discovery and dataset exploration with singular value decomposition. In ICLR 2023 Workshop on Pitfalls of Limited Data and Computation for Trustworthy ML, 2023. [32] J. Vielhaben, S. Blücher, and N. Strodthoff. Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023. [33] M. Kowal, A. Dave, R. Ambrus, A. Gaidon, K. G. Derpanis, and P. Tokmakov. Understanding video transformers via universal concept discovery. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10946–10956. IEEE, 2024. [34] T. Fel, A. Picard, L. Béthune, T. Boissin, D. Vigouroux, J. Colin, R. Cadène, and T. Serre. CRAFT: Concept recursive activation FacTorization for explainability. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2711–2721. IEEE, 2023. [35] D. A. Klindt, S. Sanborn, F. Acosta, F. Poitevin, and N. Miolane. Identifying interpretable visual features in artificial and biological neural systems. arXiv preprint, 2310.11431, 2023. [36] H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, 2024. [37] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. [38] L. Gao, T. Dupré la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations, 2025.
12
[39] T. Fel, E. S. Lubana, J. S. Prince, M. Kowal, V. Boutin, I. Papadimitriou, B. Wang, M. Wattenberg, D. Ba, and T. Konkle. Archetypal sae: adaptive and stable dictionary learning for concept extraction in large vision models. Proceedings of Machine Learning Research, 267:16543–16572, 2025. [40] U. Bhalla, A. Oesterling, S. Srinivas, F. P. Calmon, and H. Lakkaraju. Interpreting CLIP with sparse linear concept embeddings (SpLiCE). In Advances in Neural Information Processing Systems, volume 37, pages 84298–84328. Curran Associates, 2024. [41] H. Thasarathan, J. Forsyth, T. Fel, M. Kowal, and K. G. Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. Proceedings of Machine Learning Research, 267, 2025. [42] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829. IEEE, 2023. [43] A. H. Williams, E. Kunz, S. Kornblith, and S. Linderman. Generalized shape metrics on neural representations. In Advances in Neural Information Processing Systems, volume 34, pages 4738–4750. Curran Associates, 2021. [44] N. Kriegeskorte, M. Mur, D. A. Ruff, R. Kiani, J. Bodurka, H. Esteky, K. Tanaka, and P. A. Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60: 1126–1141, 2008. [45] B. Schölkopf, A. Smola, and K.-R. Müller. Kernel principal component analysis. In International Conference on Artificial Neural Networks, volume 1327, pages 583–588. Springer, 1997. [46] D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. Advances in Neural Information Processing Systems, 13:556–562, 2000. [47] Q. Shi, H. Sun, S. Lu, M. Hong, and M. Razaviyayn. Inexact block coordinate descent methods for symmetric nonnegative matrix factorization. IEEE Transactions on Signal Processing, 65:5995–6008, 2017. [48] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in Neural Information Processing Systems, 32:9448–9458, 2019. [49] A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese. Taskonomy: disentangling task transfer learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3712–3722. IEEE, 2018. [50] N. Mu, A. Kirillov, D. Wagner, and S. Xie. SLIP: Self-supervision meets language-image pre-training. In European Conference on Computer Vision, pages 529–544. Springer, 2022. [51] I. Sucholutsky, L. Muttenthaler, A. Weller, A. Peng, A. Bobu, B. Kim, B. C. Love, C. J. Cueva, E. Grant, I. Groen, J. Achterberg, J. B. Tenenbaum, K. M. Collins, K. L. Hermann, K. Oktar, K. Greff, M. N. Hebart, N. Cloos, N. Kriegeskorte, N. Jacoby, Q. Zhang, R. Marjieh, R. Geirhos, S. Chen, S. Kornblith, S. Rane, T. Konkle, T. P. O’Connell, T. Unterthiner, A. K. Lampinen, K.-R. Müller, M. Toneva, and T. L. Griffiths. Getting aligned on representational alignment. Transactions on Machine Learning Research, 2025. [52] P. Papale, F. Wang, M. W. Self, and P. R. Roelfsema. An extensive dataset of spiking activity to reveal the syntax of the ventral stream. Neuron, 113:539–553.e5, 2025. [53] O. Schoppe, N. S. Harper, B. D. B. Willmore, A. J. King, and J. W. H. Schnupp. Measuring the performance of neural models. Frontiers in Computational Neuroscience, 10:10, 2016.
13
A
Model Overview
While no finite benchmark can represent all conceivable vision models, our set of 162 models span major contemporary sources of variation in visual representation, including architecture, objective, scale, training set and training task. Together with the robustness analyses showing stable universality rankings under model subsampling, exclusion of whole architecture families, and changes in image set (See Appendix C.5), this suggests that the universality score captures a robust recurrence pattern rather than a fragile artifact of the particular benchmark composition. Table S1: Overview of all 162 vision models. Models are grouped by architecture class and sorted alphabetically within each group. Model
Class
Family
Objective
Task
Data
2D Keypoints 3D Keypoints Abstraction AlexNet Animal Arthropod Autoencoder Bird CLIP-ResNet101 CLIP-ResNet50 CSP-ResNet50 Camera Pose (Fixated) Camera Pose (Nonfixated) ConvNext-B ConvNext-B-IN21K ConvNext-L ConvNext-L-IN21K Curvatures DINO-ResNet50 DLA34 Denoising DenseNet121 ECA-NFNet-L0 EfficientNet-B1 EfficientNet-B3 Egomotion Euclidean Depth Faster-RCNN-ResNet50-FPN Flower Food GMLP-S16 GMixer-24 GhostNet100 GoogleNet HardCoreNAS-A HardCoreNAS-F Inception-V3 Inpainting Instrument Jigsaw Keypoint-RCNN-ResNet50FPN MNASNet1.0 Mammal Mask-RCNN-ResNet50-FPN MiDaS MobileNet-V2 MobileNet-V3-Large NF-Net-L0 NF-ResNet50 Object Object Classification
CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN
ResNet ResNet ResNet AlexNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ConvNeXt ConvNeXt ConvNeXt ConvNeXt ResNet ResNet DLA ResNet DenseNet NFNet EfficientNet EfficientNet ResNet ResNet R-CNN ResNet ResNet gMLP gMixer GhostNet Inception HardCoreNAS HardCoreNAS Inception ResNet ResNet ResNet R-CNN
Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Vision-Language Vision-Language Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Self-Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised
Taskonomy Taskonomy BiT-Expert Supervised BiT-Expert BiT-Expert Taskonomy BiT-Expert CLIP CLIP Supervised Taskonomy Taskonomy Supervised Supervised Supervised Supervised Taskonomy Self-Supervised Supervised Taskonomy Supervised Supervised Supervised Supervised Taskonomy Taskonomy Detection BiT-Expert BiT-Expert Supervised Supervised Supervised Supervised Supervised Supervised Supervised Taskonomy BiT-Expert Taskonomy Segmentation
Taskonomy Taskonomy BiT Transfer ImageNet-1K BiT Transfer BiT Transfer Taskonomy BiT Transfer OpenAI-400M OpenAI-400M ImageNet-1K Taskonomy Taskonomy ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-21K Taskonomy ImageNet-1K ImageNet-1K Taskonomy ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K Taskonomy Taskonomy COCO BiT Transfer BiT Transfer ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K Taskonomy BiT Transfer Taskonomy COCO
CNN CNN CNN CNN CNN CNN CNN CNN CNN CNN
MNASNet ResNet R-CNN MiDaS MobileNet MobileNet NFNet ResNet ResNet ResNet
Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised
Supervised BiT-Expert Segmentation Depth Supervised Supervised Supervised Supervised BiT-Expert Taskonomy
ImageNet-1K BiT Transfer COCO Depth Mix ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K BiT Transfer Taskonomy
14
Model
Class
Occlusion Edges CNN Point Matching CNN Random Weights CNN RegNet-128Gf-SEER CNN RegNet-128Gf-SEER-INFT CNN RegNet-32Gf-SEER CNN RegNet-32Gf-SEER-INFT CNN RegNet-64Gf-SEER CNN RegNet-64Gf-SEER-INFT CNN RegNetX-64 CNN RegNetY-64 CNN Relation CNN ResNet101 CNN ResNet152 CNN ResNet18 CNN ResNet50 CNN ResNet50-BarlowTwins CNN ResNet50-ClusterFit CNN ResNet50-DeepClusterV2 CNN ResNet50-JigSaw-Goyal19 CNN ResNet50-JigSaw-P100 CNN ResNet50-MoCo-V2 CNN ResNet50-PIRL CNN ResNet50-RotNet CNN ResNet50-SimCLR CNN ResNet50-SwAV CNN Reshading CNN RetinaNet-ResNet50-FPN CNN Room Layout CNN SEResNext50-32x4D CNN SKResNext50-32x4D CNN Scene Classification CNN SemNASNet100 CNN Semantic Segmentation CNN ShuffleNet-V2-x1.0 CNN SqueezeNet1.0 CNN Surface Normals CNN Texture Edges CNN Unsupervised 2.5D Segmenta- CNN tion Unsupervised 2D Segmenta- CNN tion VGG16 CNN Vanishing Point CNN Vehicle CNN Xception CNN YOLO-V5-L CNN YOLO-V5-M CNN YOLO-V5-S CNN Z-Buffer Depth CNN CLIP-ViT-B/16 ViT CLIP-ViT-B/32 ViT CLIP-ViT-L/14 ViT CoaT-Lite-Tiny ViT CrossViT-B ViT DINO-ViT-B16 ViT DPT-Hybrid ViT DeiT-B-P16-224 ViT JX-NesT-Tiny ViT LeViT128 ViT OpenCLIP-ViT-L/14 (DFN- ViT 2B) ViT OpenCLIP-ViT-L/14 (DataComp-XL) OpenCLIP-ViT-L/14 (LAION- ViT 2B)
Family
Objective
Task
Data
ResNet ResNet ResNet RegNet RegNet RegNet RegNet RegNet RegNet RegNet RegNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet ResNet R-CNN ResNet SENet SENet ResNet SemNASNet ResNet ShuffleNet SqueezeNet ResNet ResNet ResNet
Supervised Supervised Untrained Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised
Taskonomy Taskonomy Untrained SEER SEER SEER SEER SEER SEER Supervised Supervised BiT-Expert Supervised Supervised Supervised Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Self-Supervised Taskonomy Detection Taskonomy Supervised Supervised Taskonomy Supervised Taskonomy Supervised Supervised Taskonomy Taskonomy Taskonomy
Taskonomy Taskonomy Taskonomy Random-1B Random-1B Random-1B Random-1B Random-1B Random-1B ImageNet-1K ImageNet-1K BiT Transfer ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K Taskonomy COCO Taskonomy ImageNet-1K ImageNet-1K Taskonomy ImageNet-1K Taskonomy ImageNet-1K ImageNet-1K Taskonomy Taskonomy Taskonomy
ResNet
Supervised
Taskonomy
Taskonomy
VGG ResNet ResNet Inception YOLO YOLO YOLO ResNet ViT ViT ViT CoaT CrossViT ViT DPT DeiT NesT LeViT ViT
Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Vision-Language Vision-Language Vision-Language Supervised Supervised Self-Supervised Supervised Supervised Supervised Supervised Vision-Language
Supervised Taskonomy BiT-Expert Supervised YOLO YOLO YOLO Taskonomy CLIP CLIP CLIP Supervised Supervised Self-Supervised Depth Supervised Supervised Supervised CLIP
ImageNet-1K Taskonomy BiT Transfer ImageNet-1K COCO+VOC COCO+VOC COCO+VOC Taskonomy OpenAI-400M OpenAI-400M OpenAI-400M ImageNet-1K ImageNet-1K ImageNet-1K Depth Mix ImageNet-1K ImageNet-1K ImageNet-1K DFN-2B
ViT
Vision-Language CLIP
DataComp-XL
ViT
Vision-Language CLIP
LAION-2B
15
Model
Class
OpenCLIP-ViT-L/14 (LAION- ViT 400M) ViT OpenCLIP-ViT-L/14 (MetaCLIP-400M) ViT OpenCLIP-ViT-L/14 (MetaCLIP-FullCC) PiT-B-224 ViT PiT-T-224 ViT PoolFormer-S36 ViT Swin-B-P4-W7 ViT Swin-B-P4-W7-IN21K ViT Swin-L-P4-W7 ViT Swin-L-P4-W7-IN21K ViT Swin-T-P4-W7 ViT TnT-P16-224 ViT ViT-B-CLIP ViT ViT-B-P16 ViT ViT-B-P16-IN21K ViT ViT-B-P32 ViT ViT-B-P32-IN21K ViT ViT-B-R50-S16-IN21K ViT ViT-B-SLIP ViT ViT-B-SimCLR ViT ViT-L-CLIP ViT ViT-L-CLIP-CC12M ViT ViT-L-P16 ViT ViT-L-P16-IN21K ViT ViT-L-SLIP ViT ViT-L-SLIP-CC12M ViT ViT-L-SimCLR ViT ViT-S-CLIP ViT ViT-S-P16 ViT ViT-S-P16-IN21K ViT ViT-S-P32 ViT ViT-S-P32-IN21K ViT ViT-S-SLIP ViT ViT-S-SimCLR ViT ViT-T-P16 ViT Visformer ViT XCIT-N-12-P16 ViT XCIT-N-12-P8 ViT MLP-Mixer-B16 Mixer MLP-Mixer-B16-IN21K Mixer MLP-Mixer-L16 Mixer MLP-Mixer-L16-IN21K Mixer ResMLP-12 Mixer ResMLP-24 Mixer ResMLP-36 Mixer ResMLP-Big-24 Mixer ResMLP-Big-24-IN21K Mixer ConViT-B Hybrid ConViT-T Hybrid ConvMixer-768-32 Hybrid
Family
Objective
ViT
Vision-Language CLIP
LAION-400M
ViT
Vision-Language CLIP
MetaCLIP-400M
ViT
Vision-Language CLIP
PiT PiT PoolFormer Swin Swin Swin Swin Swin TnT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT ViT Visformer XCiT XCiT MLP-Mixer MLP-Mixer MLP-Mixer MLP-Mixer MLP-Mixer MLP-Mixer MLP-Mixer MLP-Mixer MLP-Mixer ConViT ConViT ConvMixer
Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Vision-Language Supervised Supervised Supervised Supervised Supervised Vision-Language Self-Supervised Vision-Language Vision-Language Supervised Supervised Vision-Language Vision-Language Self-Supervised Vision-Language Supervised Supervised Supervised Supervised Vision-Language Self-Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised
MetaCLIPFullCC ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-1K YFCC-15M ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-21K ImageNet-21K YFCC-15M YFCC-15M YFCC-15M YFCC-15M ImageNet-1K ImageNet-21K YFCC-15M YFCC-15M YFCC-15M YFCC-15M ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-21K YFCC-15M YFCC-15M ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-1K ImageNet-21K ImageNet-1K ImageNet-1K ImageNet-1K
B
Embedding generation
B.1
RBF Kernel and Bandwidth Selection
Task
Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised SLIP Supervised Supervised Supervised Supervised Supervised SLIP SLIP SLIP SLIP Supervised Supervised SLIP SLIP SLIP SLIP Supervised Supervised Supervised Supervised SLIP SLIP Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised Supervised
Data
We define the symmetric similarity matrix Sm ∈ Rn×n ≥0 via a radial basis function (RBF) kernel, ∥z 2 m,i − zm,j ∥ , (6) [Sm ]ij = exp − 2 2σm where the RBF kernel guarantees that Sm is positive semi-definite and nonnegative, both properties required by the symmetric nonnegative factorization (Eq. 2). 16
THINGS, Rank = 10
THINGS, Rank = 30
THINGS, Rank = 50
1.0
Score
0.8
0.6
Stability Expl. variance
0.4
Stability Expl. variance
THINGS, Rank = 100
Stability Expl. variance
THINGS, Rank = 200
ObjectNet, Rank = 50
1.0
Score
0.8
0.6 Stability Expl. variance
0.4
0.1
0.2
0.3
0.4
0.5
0.6
Stability Expl. variance 0.8
0.1
0.2
Bandwidth multiplier
0.3
0.4
0.5
0.6
Stability Expl. variance 0.8
Bandwidth multiplier
0.1
0.2
0.3
0.4
0.5
0.6
0.8
Bandwidth multiplier
Figure S1: Bandwidth selection. Factorization stability and explained variance as a function of the RBF bandwidth multiplier α, for THINGS ranks 10, 30, 50, 100, and 200, and ObjectNet rank 50. Thin lines show individual models (n = 162); thick lines show means ± 95% CI. Dashed vertical lines indicate the optimal α∗ (maximizing the harmonic mean of both criteria). The optimum is consistently near α = 0.4–0.5 across ranks. The bandwidth σm controls the emphasis on local versus global similarity structure. Rather than fixing it to the median pairwise distance, we set σm = α∗ · d˜m , where d˜m is the median pairwise Euclidean distance within model m [45] and α∗ is chosen per model to jointly maximize factorization stability and explained variance. Specifically, we search over a grid of multipliers α ∈ {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.8, 1.0}, run B = 5 random initializations per multiplier, and select the α∗ that maximizes the harmonic mean of factorization stability (mean pairwise correlation of aligned solutions across seeds) and explained variance of the low-rank reconstruction, thus penalizing imbalanced solutions in which one criterion is optimal because another is sacrificed. Fig. S1 shows that factorization stability decreases monotonically with α while explained variance increases, producing a consistent optimum near α∗ = 0.4–0.5 across all ranks. Note that we have ∂[Sm ]ij ∂ ∥zm,i − zm,j ∥2 ∥zm,i − zm,j ∥2 = exp − = [Sm ]ij · >0 ∂α ∂α 2α2 d˜2m α3 d˜2m
(7)
which implies when α increases, every off-diagonal similarity increases monotonically toward 1. More specifically, we can rewrite Sm with Taylor expansion around large α as Sm = 11⊤ −
1 2α2 d˜2m
D + O(α−4 )
(8)
where [D]ij = ∥zm,i − zm,j ∥2 is the squared pairwise Euclidean distance matrix. When α increases, less variance remains in higher-order directions, and Sm converges towards the leading term, which is rank 1. Consequently, for any fixed factorization rank r, the rank-r symmetric NMF can capture a larger fraction of similarity structure, leading to higher explained variance. At the same time, Sm contains less sharply differentiated structure as many pairs become similarly close. The matrix is dominated by broad global similarity. The optimization landscape for symmetric NMF becomes flatter in directions corresponding to splitting or merging of factors. The W W ⊤ decomposition is then weakly constrained and therefore stability across runs drops. B.2
Optimization
We optimize Eq. 2 via block successive upper-bound minimization [47]. For each model m, rank r, and candidate bandwidth multiplier, we run B = 5 random initializations. For each multiplier, we align solutions across initializations via the Hungarian algorithm and compute factorization stability 17
0
80
120
120
80
40
0
40
80
120
160 160
Model rank (k = 50)
120
0 = 0.89
= 0.93 Model rank (k = 100)
40
160 160
0 = 0.99
Model rank (k = 30)
Model rank (k = 10)
= 0.95
80
40
0
Model rank (k = 50)
40
Model rank (k = 200)
0
80
120
160 160
120
80
40
0
40
80
120
160 160
Model rank (k = 50)
120
80
40
0
Model rank (k = 50)
Figure S2: Rank stability. Model-level universality rankings at rank 50 compared against ranks 10, 30, 100, and 200. Each dot is one model; Spearman correlations are shown per panel. as the mean matched correlation across seed pairs. After selecting the bandwidth multiplier by the harmonic mean of factorization stability and explained variance (Appendix B.1), we retain the most central seed, defined as the solution with the highest average matched correlation to all other seeds. B.3
Rank Selection
We report results at rank r = 50 throughout the main text but repeat all analyses at r ∈ {10, 30, 100, 200}. Model-level universality rankings are highly stable across ranks (Fig. S2), with Spearman correlations between rank-50 and all other ranks at least ρ = 0.93 except for r = 200. This confirms that our conclusions do not depend on the particular factorization rank.
C
Metric validation
C.1
Universality Metric Details
Our universality metric is used to test factor identity: whether an individual dimension recovered from one model recurs as an individual dimension in another model. This means that matching should be one-to-one, and scores should be calibrated against chance stimulus-level correspondence. Why not greedy matching? A natural baseline assigns each target dimension its single bestmatching source dimension. This greedy strategy permits many-to-one collisions: multiple target dimensions can claim the same source dimension, leaving others unmatched. Across all 26,082 model pairs at rank 50, 41% of source dimensions are never selected as any target dimension’s best match (Fig. S3, a). Models with generic, broadly correlated dimensions benefit disproportionately from this inflation, while models with more distinctive dimensions are penalized. Because symmetric NMF factors are identifiable only up to permutation, a one-to-one assignment via the Hungarian algorithm is more principled: it respects the permutation structure of the factorization and ensures that each dimension receives exactly one match. Null calibration. At moderate ranks, even unrelated dimensions can achieve non-trivial cos2 scores by chance. To remove this floor, we construct a permutation null for each target model m: we randomly permute the rows of source embeddings Wm′ , destroying stimulus correspondence while preserving column structure, recompute the Hungarian assignment and cos2 -permutation scores on each shuffled dataset, and take the 95th percentile across B = 1,000 permutations as a per-dimension threshold am,k . The null-adjusted score is sadj (m, k; m′ ) =
sπ∗ (m, k; m′ ) − am,k , 1 − am,k
clipped to [0, 1],
(9)
and the final universality score averages these adjusted values across all remaining models. This calibration shifts the score distribution leftward and removes the positive floor present in the raw cos2 permutation scores (Fig. S3, b), ensuring that only dimensions with stimulus-specific correspondence contribute to universality. Factor identity versus shared subspaces. The permutation-based universality score tests whether each factor in model m has a unique counterpart in model m′ . Alternatively, one can ask whether 18
a
b 40
Null floor
4.0
Raw cos² permutation Null-adjusted
Never matched
3.0
30 Density
Source dimensions (%)
3.5
20
2.5 2.0 1.5
10
1.0 0.5
0
0.0 0
1
2
3
4
5
6+
0.0
Times selected as best match
0.2
0.4 0.6 Universality score
0.8
1.0
Figure S3: Universality metric calibration. (a) Distribution of how often each source dimension is selected as a target dimension’s best match across all model pairs at rank 50. Under greedy matching, 41% of source dimensions are never selected, while others are claimed by multiple target dimensions. The Hungarian algorithm enforces exactly one match per dimension. (b) Distribution of per-dimension universality scores before (gray) and after (blue) null correction. Raw cos2 -permutation scores exhibit a positive floor from chance alignment (dashed line); null adjustment removes this floor and rescales scores to [0, 1].
a factor lies in the nonnegative cone of another model’s factors, by projecting wm,k onto the cone C(Wm′ ) = {Wm′ a : a ≥ 0} via nonnegative least squares and measuring how much variance is captured. This cone projection variant captures shared representational subspaces rather than strict factor identity, including cases where one model splits or merges concepts across dimensions. However, while this can detect broader shared structure, it also mixes level of description: the source is an individual factor, while the target is a model-level subspace (or vice-versa). Consequently, a high cone score is ambiguous, as it may reflect a true one-to-one counterpart, a split/merge correspondence across multiple factors, or only a diffuse approximation by several partially related factors. We use permutation matching as our primary metric because it directly tests factor identity, that is, whether the same dimension recurs across models. C.2
Within-Model Stability Ceiling
To interpret universality on an absolute scale, we compute the within-model stability ceiling, defined as the typical agreement between independent NMF factorizations of the same model. Using the B = 5 seeds fit per model (Appendix B.2), we compute the cos2 -permutation score between each of the 52 = 10 seed pairs, average across pairs per dimension, and apply the same null calibration as for universality (Appendix C.1). The null-adjusted within-model stability has median 0.84 (IQR [0.75, 0.90], range [0.23, 0.995]) across all 8,100 dimensions. This is an empirical upper bound for cross-model agreement under cos2 -perm, since a dimension cannot be more consistent across different models than it is across independent fits of a single model. Note that this ceiling varies only the NMF initialization while holding the underlying model fixed. A stricter ceiling that also varied model training, for example across independent training reruns of the same architecture, would likely be lower, making our estimate a liberal upper bound on cross-model agreement. C.3
Universality Metric Validation
Pr We summarize universality at the model level as Um = 1r k=1 um,k and validate this metric in three ways (Fig. S4). First, we test whether universality generalizes to a different image set by recomputing scores from ObjectNet [48], which depicts objects in cluttered real-world scenes; model rankings are largely preserved. Second, we show that the cos2 -permutation metric agrees closely with a cross-validated ridge regression variant at the per-dimension level. Third, we confirm that universality correlates strongly with centered kernel alignment (CKA), establishing convergent validity across different representational similarity measures. 19
b = 0.81
0.8
40
0.7
60
0.6
Ridge R²
Model rank (ObjectNet)
20
80 100
c0
= 0.82
20
0.5 0.4
120
150
100 50 Model rank (THINGS)
0
80 100
140
0.2
160
60
120
0.3
140
= 0.89
40 Model rank (CKA)
a0
160 0.0 0.1 0.2 0.3 0.4 0.5 Dimension universality (cosine null-corrected)
150
100 50 Model rank (universality)
0
Figure S4: Metric validation. (a) Model-level universality rankings computed from THINGS vs. ObjectNet features (ρ = 0.81). (b) Per-dimension universality (null-corrected cosine) vs. crossvalidated ridge R2 (ρ = 0.82), confirming that our metric agrees with a more expensive regressionbased proxy. (c) Model-level universality rank vs. CKA rank (ρ = 0.89), establishing convergent validity with a standard representational similarity measure. b
a
r = 0.08, n = 162
90
8.5
80
8.0
Parameters (log10)
ImageNet top-1 accuracy (%)
= -0.04, n = 101
70
60
7.5
7.0
6.5 50
6.0 0.1
0.15
0.2
0.25
0.3
0.05
Universality
0.10
0.15
0.20
0.25
0.30
Universality
Figure S5: Universality is not explained by model accuracy or size. (a) Universality vs. ImageNet top-1 accuracy (n = 101; Spearman ρ = −0.04, p = 0.72). (b) Universality vs. number of parameters on a log scale (n = 162; Pearson r = 0.08, p = 0.31). Neither variable shows a reliable association with universality.
C.4
Universality vs. Model Accuracy and Size
Universality could in principle reflect model quality rather than shared representational structure. To rule this out, we correlate per-model universality with ImageNet top-1 accuracy (for the 101 models with reported accuracy) and with the total number of parameters (all 162 models). Neither shows a reliable association (Fig. S5). The Pearson correlation with accuracy is weak and non-significant (r = 0.19, p = 0.052), and the rank correlation is essentially zero (ρ = −0.04, p = 0.72), indicating that the marginal Pearson value is driven by a small number of outliers. The Pearson correlation with parameter count is similarly negligible (r = 0.08, p = 0.31). Models that are more accurate or larger are not systematically more universal. C.5
Model-Set Stability
We test whether the universality metric is overly dependent on the specific model set over which it is computed. Subsampling stability. We subsample the model set to 20% of models (n = 32, 1,000 iterations) and recompute universality scores. Model rankings based on overall universality are stable across iterations, with a median Spearman correlation between the full-set and subsampled rankings of ρ = 0.97 (95% range: [0.84, 0.99]). 20
a
b
0.35
0.35
= 0.97 (per iteration) Universality (20% subsample)
Universality (20% subsample)
median 0.30 0.25 0.20 0.15 0.10 0.05
= 0.83
0.30 0.25 0.20 0.15
CNN ViT Mixer Hybrid
0.10 0.05
0.05
0.10
0.15
0.20
0.25
0.30
0.05
0.35
0.10
0.15
0.20
0.25
0.30
0.35
Universality (full model set)
Universality (full model set)
Figure S6: Model-set stability. (a) Bootstrap stability: universality from the full model set (M = 162) vs the mean across 1,000 bootstrap resamples (20% of models, n = 32). (b) Leave-family-out stability: universality from the full set vs recomputed after excluding all models from the same architecture family.
Leave-family-out stability. We also test whether the universality of dimensions in a given model depends on other models from the same architectural family being present in the set. Recomputing universality using only models from other architecture classes yields scores that are highly correlated with those of the full set (ρ = 0.83).
D
Category Consistency
Our universality metric quantifies how consistently a dimension recurs across models, but does not reveal what a dimension represents. To characterize the content of each dimension, we exploit the categorical structure of the THINGS image set. The N = 22,248 images are organized into C = 1,854 object categories with J = 12 exemplars each (N = C × J). Each NMF dimension wm,k ∈ RN ≥0 assigns a nonnegative loading to every image. If a dimension captures a semantic category, exemplars of the same category should receive similar loadings. If it instead reflects a visual property such as texture or color, exemplars within a category may receive very different loadings. We formalize this intuition using one-way ANOVA to decompose each dimension’s loading variance into between- and within-category components. For a given dimension k of model m, let wm,k,cj denote the loading of the j-th exemplar in category PJ P c, let w̄m,k,c = J1 j=1 wm,k,cj be the mean loading for category c, and let w̄m,k = N1 c,j wm,k,cj be the grand mean over all images. The total sum of squares decomposes as C X J X
(wm,k,cj − w̄m,k )2 =
c=1 j=1
|
C X
J (w̄m,k,c − w̄m,k )2 +
c=1
{z
SStotal
}
|
J C X X (wm,k,cj − w̄m,k,c )2 .
(10)
c=1 j=1
{z
SSbetween
}
|
{z
SSwithin
}
Category consistency is the proportion of variance explained by category membership, 2 ηm,k =
SSbetween ∈ [0, 1]. SStotal
(11)
2 A dimension with ηm,k ≈ 1 assigns nearly identical loadings to all exemplars of the same category 2 and different loadings across categories. A dimension with ηm,k near the chance level of C/N ≈ 0.08 carries no more category information than expected from a random loading vector. 2 We compute ηm,k independently for each dimension k in each model m, yielding r scores per model. 2 To quantify reconstruction importance, we measure the drop in explained variance ∆Rm,k when ⊤ dimension k is removed from the low-rank reconstruction Wm Wm .
21
E
Neural Predictivity
We use multi-unit activity (MUA) recorded from inferior temporal (IT) cortex of two rhesus macaques (F and N) viewing the same 22,248 THINGS object images [52]. Responses were z-scored per channel per recording day and averaged across the response time window by the original authors. Split-half reliability was pre-computed from a set of 100 test images presented with ∼30 repetitions each. We average reliability across splits to obtain a single reliability estimate per channel and retain only IT channels with mean reliability > 0.3, yielding 157 neurons for monkey F (of 320 IT channels) and 141 neurons for monkey N (of 256 IT channels). For each of the 162 models, we fit a cross-validated ridge regression from the r = 50 NMF dimensions (Wm ∈ RN ×r ) to the response of each neuron independently. We use 5-fold cross-validation with ridge regularization selected within each training fold from α ∈ {10−2 , . . . , 106 } (20 log-spaced values). Encoding performance is the Pearson correlation between predicted and observed responses on held-out images, averaged across neurons and then across both monkeys to obtain a single encoding score per model. For the universal/specific-half comparison, we zeroed out the complementary 25 dimensions and reran the same cross-validated ridge procedure for each masked embedding. p We estimate a per-neuron noise ceiling as reliabj , where reliabj is the split-half reliability of neuron j [53]. Median noise ceilings are 0.71 (monkey F) and 0.83 (monkey N).
F
Behavioral Predictivity
Dataset. We use the THINGS odd-one-out triplet judgments of Hebart et al. [13], collected on the 1,854 THINGS object categories. On each trial, a participant is shown three object images and chooses the one least similar to the other two (the odd-one-out). We use the public train and validation splits, yielding 4,120,663 train and 457,430 validation triplets (4,578,093 total). For the zero-shot analyses reported in the main text we pool train and validation triplets into a single evaluation set, since no parameters are fit to the triplets. Image selection. For each of the 1,854 categories we use the single image on which the human cat behavioral ratings of Hebart et al. [13] were collected, yielding a category-level embedding Wm ∈ 1854×r R≥0 with r = 50. Triplet accuracy. Let T denote the set of triplets, where each triplet is indexed by the triple of images (i, j, k) for which the participant selected k as the odd one out, so that (i, j) is the humanm m chosen similar pair. For model m, let sm ab = cos(wa , wb ) denote the cosine similarity between cat rows a and b of the category-level embedding Wm . The model’s predicted similar pair is (a⋆ , b⋆ ) =
arg max
sm ab ,
(12)
(a,b)∈{(i,j), (i,k), (j,k)}
and triplet accuracy is the fraction of triplets on which the predicted and human-chosen similar pairs agree X 1 acc(m) = 1[(a⋆ , b⋆ ) = (i, j)] . (13) |T | (i,j,k)∈T
Universal vs. specific half. To test whether universal dimensions drive behavioral alignment, we split each model’s r = 50 dimensions at the median of the per-dimension universality score (Section 3.4) into a universal half (top 25 dimensions) and a model-specific half (bottom 25). For each half, we construct a masked embedding by zeroing out the complementary 25 columns of cat Wm , recompute cosine similarities on the masked embedding, and evaluate triplet accuracy exactly as above. Significance is assessed with a paired t-test over the 162 models on the difference accuniversal − accspecific .
G
Compute Resources
All experiments were run on an internal high-performance computing cluster. Feature extraction from 162 models on THINGS and ObjectNet images required approximately 6 GPU-hours on NVIDIA 22
A100 40GB GPUs. Symmetric NMF across 162 models × 5 ranks × 5 random seeds (Eq. 2) required approximately 75,000 CPU-hours on CPU compute nodes with 512 GB RAM. Beyond the reported experiments, preliminary and exploratory runs required approximately an additional 5,000 CPU-hours.
H
Human Dimension Rating Experiment
Goal. To characterize what each NMF dimension encodes from a human perspective, we collected crowd-sourced labels indicating whether each dimension reflects semantic content, visual properties, both, or neither. Dimension selection. We applied hierarchical agglomerative clustering to all 162 × 50 = 8,100 dimensions using the precomputed linkage matrix (correlation distance). Re-cutting the dendrogram at a threshold of 0.75 yielded 1,059 clusters. For each cluster, we selected the single representative dimension whose weight vector w had highest correlation with the cluster centroid (mean weight vector), reducing redundancy while preserving the diversity of dimension types. Stimuli. For each selected dimension, we constructed an image grid showing the 64 highestloading images, with at most one image per THINGS category (to prevent any single category from dominating). Images were sorted in descending order of loading weight. Task. Participants were shown image grids one at a time and asked to categorize each dimension by selecting one of four mutually exclusive labels: • Semantic — the images share an identifiable object category or concept (e.g. all dogs, all vehicles). • Visual — the images share a low-level perceptual property such as color, texture, or shape, without a clear shared concept. • Both — the images share both a conceptual and a perceptual property. • Neither — no clear shared property is apparent. Participants also rated the difficulty of labeling each dimension on a 1–7 scale (1 = very easy, 7 = very difficult). Experimental design. The 1,059 dimensions were distributed across 71 survey versions using a between-subjects design, with 15 real trials per version. Each version additionally contained one catch trial (a grid of dog images, expected response: semantic or both), one attention-check trial (explicit instruction: select both and difficulty = 4), and one repeated real trial (the first real trial, re-shown at a random position, used to estimate within-participant consistency). Trial order within each version was randomized. Each version was assigned to 8 participants, initially yielding 8 assigned ratings per dimension before quality-control exclusions. The study was administered online via the Connect platform (see Fig. S7; participants were compensated at C0.75 per task (approximately 6 minutes per session or C7.50/h). Quality control. Participants who failed the attention-check trial (i.e. did not select both with difficulty = 4) were excluded. Of 570 participants, 432 (76%) passed this criterion and were retained for analysis. Reliability by content type. Overall, raters agreed with the majority-vote label on 65% of trials on average (per-rater mean accuracy = 0.65, median = 0.67), and majority-vote labels were stable across independent half-samples of raters, with the two halves agreeing on the majority label in 57% of splits on average (per-dimension median split-half reliability = 0.57, mean = 0.55). Both measures varied systematically across content categories. Per-rater accuracy was highest for neither dimensions (µ = 0.72) and lowest for both dimensions (µ = 0.57), with semantic and visual intermediate (µ ≈ 0.63; Kruskal–Wallis H = 14.6, p = .002). Split-half reliability followed the same ordering: neither dimensions yielded the most stable majority-vote labels (median = 0.77), while both dimensions were least stable (median = 0.35), with visual (median = 0.49) and semantic (median = 0.43) intermediate. This convergence across two independent reliability measures confirms 23
a
b
Figure S7: Experiment instructions and task interface. (a) Instructions provided to participants before the task, including visual examples of semantic, visual, mix and neither. (b) Task interface and questions asked. that the observed pattern reflects genuine differences in label-boundary ambiguity rather than artifacts of any single metric. The structure of disagreements further validates the label space. When raters disagreed on a both dimension, they split nearly evenly between semantic (44%) and visual (42%), confirming that raters are decomposing a genuinely mixed-content dimension rather than responding randomly. When raters disagreed on a visual dimension, they most often substituted neither (43%), suggesting that the visual/neither boundary is the fuzziest in the label space – consistent with the observation that low-universality dimensions often encode subtle textural properties that can appear uninterpretable. Together, these patterns indicate that disagreements are structured and interpretable rather than random, supporting the validity of the majority-vote labels used throughout.
24