arXiv CS open access

Mapping and Measuring the Behavioral Evolution of Large Language Models

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Mapping and Measuring the Behavioral Evolution of Large Language Models Dong Qiao, Chris Ding, Jicong Fan∗ School of Data Science, The Chinese University of Hong Kong, Shenzhen, China dongqiao@link.cuhk.edu.cn, chrisding@cuhk.edu.cn, fanjicong@cuhk.edu.cn

arXiv:2608.11027v1 [cs.LG] 11 Aug 2026

Abstract Benchmark leaderboards summarize how well a language model performs, but not how its behavior relates to that of other models or changes across generations. We characterize the output behavior of 32 models from six families using their responses to a shared bank of 10,000 prompts. After embedding each response, we construct three complementary sentence-level dissimilarities: an aligned mean per-prompt distance, which is a pseudometric on observed model responses; a PCA-compressed summary of prompt-wise disagreement; and an alignment-free Gromov–Wasserstein discrepancy between models’ internal response geometries. We use these constructions to study static organization and temporal change on a release-date axis through behavioral maps, family-wise drift, hierarchical clustering, cross-family convergence, and response-cloud dispersion. Across the three constructions, model families form coherent clusters, with gpt-2 as a global outlier; cross-family distances decrease over time; and several recent reasoning-oriented models have comparatively compact response clouds. A token-level crosscheck based on per-prompt Maximum Mean Discrepancy closely agrees with the sentence-level mean distance (Spearman ρ = 0.98) and recovers the same qualitative findings. We organize these comparisons through a measure-theoretic lens making their alignment and invariance assumptions explicit. We also establish an architecture-agnostic sufficient condition linking behavioral similarity to inference-prompt coverage, small excess population log-loss, and similar effective target distributions—a possible training-side account rather than an empirical explanation of the observed trends. Our pipeline is label-free, and re-encoding every response with three further encoders—down to one 73× smaller—preserves the rank geometry, the outliers, and the sign of the time trend.

sors, or whether models from different vendors are becoming more alike. We therefore ask: Can the behavioral evolution of LLMs be mapped and measured directly from their outputs, without using ground-truth labels? We answer this question by applying the same 10,000prompt bank to 32 models, embedding their responses, and comparing the resulting response collections. Our contributions are fivefold. First, we introduce three complementary sentence-level constructions: an aligned mean per-prompt pseudometric, a PCA-compressed summary of prompt-wise disagreement, and an alignment-free Gromov–Wasserstein (GW) discrepancy over internal response geometry (Mémoli 2011; Peyré and Cuturi 2019). Second, we place the models on a release-date axis and quantify both static structure and temporal change using behavioral maps, drift curves, ordered heatmaps, hierarchical clustering, cross-family distance, and response-cloud dispersion. Third, we provide an empirical characterization of 32 models from six families and identify three patterns that are stable across constructions: family coherence, decreasing cross-family distance, and relatively compact response clouds for several recent reasoningoriented models. Fourth, we validate the sentence-level picture with an independent token-level Maximum Mean Discrepancy (MMD) analysis (Gretton et al. 2012). Fifth, we clarify the mathematical relationships among the constructions and prove a training-side sufficient condition that connects behavioral similarity to effective target-distribution similarity, excess population log-loss, and inference-prompt coverage. We additionally quantify tree-likeness and the lowdimensional structure of prompt-wise disagreement.

2 1

Introduction

The pace of large language model (LLM) releases has outstripped our tools for understanding how models relate to one another. Standard evaluation compresses a model’s performance on each benchmark into a scalar score (Liang et al. 2022; Srivastava et al. 2023). Such scores are essential for measuring capability, but they do not reveal behavioral relationships (Yax, Oudeyer, and Palminteri 2025; Goel et al. 2025; Zhang et al. 2026): whether two models answer in similar ways, whether a new release departs from its predeces∗

Corresponding author.

Related Work

Behavioral evaluation and black-box fingerprinting. Behavioral evaluation studies models through their observable responses rather than only through aggregate benchmark scores. CheckList (Ribeiro et al. 2020) introduced systematic behavioral testing for NLP models, while broader surveys organize model behavior across linguistic, reasoning, knowledge, and social dimensions (Chang and Bergen 2024). More recent work uses output behavior to identify or characterize LLMs. LLMmap (Pasquini, Kornaropoulos, and Ateniese 2025) actively selects prompts whose responses distinguish deployed model versions, whereas behavioral fingerprinting (Pei et al. 2025) constructs multidimensional profiles

of cognitive and alignment-related tendencies. These methods primarily target testing, identification, or trait profiling. Our objective is instead to construct a continuous geometry over models from a shared, broad prompt bank and use that geometry to measure lineage structure, temporal drift, and cross-family convergence. Are models converging? The Platonic Representation Hypothesis (Huh et al. 2024) holds that networks trained on different data and objectives are converging toward a shared statistical model of reality. The claim is now contested from two directions. Gröger, Wen, and Brbić (2026) advance an Aristotelian reading in which representations remain particularistic and task-dependent, reporting—via permutation tests and canonical correlation analysis—that networks diverge more than the universalist account predicts. Koepke et al. (2026) re-examine cross-modal alignment at scale and find that the mutual-nearest-neighbor agreement supporting convergence holds on small samples but degrades substantially as the data grow to millions, leaving the evidence for a common representation weaker than assumed. Our analysis is complementary to this debate. We measure convergence in what models emit, rather than in how they encode, and study change along a release-date axis rather than across modalities or objectives. Behavioral convergence is compatible with either conclusion about internal representations: output distributions may become more similar even when hidden representations remain distinct. Our GW analysis probes this distinction by comparing response-space geometry rather than absolute embedding coordinates. Comparing models by their internals. A parallel line compares models mechanistically. Wang et al. (2025) use sparse autoencoders to isolate interpretable features across Transformer (Vaswani et al. 2017) and Mamba (Gu and Dao 2023) architectures, find most features shared between the two, and show that Mamba’s induction circuits are structurally analogous to their Transformer counterparts— evidence for universality at the level of learned mechanism. Lan et al. (2024) apply representational-similarity metrics to sparse-autoencoder feature spaces across different LLMs and report substantial agreement, with subspaces for particular semantic concepts aligning especially closely. Representation-similarity analysis (Kornblith et al. 2019) compares internal activations in the same spirit. All of these require white-box access to weights or activations. Our distances need only generated text, which is what makes the closed frontier models in this study measurable at all. Tracing model evolution from behavior. Closest to our setting is LLM DNA (Wu et al. 2026), which derives a lowdimensional, bi-Lipschitz encoding of functional behavior from model responses and uses the induced distances to construct a phylogeny over many models. Both studies compare models through outputs rather than weights, but their goals differ. LLM DNA compresses behavior into a genotypelike representation to recover ancestry at scale, primarily among open-weight models. We retain pairwise behavioral geometry, compare constructions with different alignment and aggregation assumptions, and focus on temporal change

Family GPT Claude Gemini Qwen Llama Mistral

# 5 8 4 5 5 5

Release span 2019–2025 2025–2026 2024–2026 2024–2026 2024–2025 2024–2025

Latest (this study) gpt-5.2 claude-opus-4-7-think gemini-3.1-pro-preview qwen3.7-max llama-4-maverick mistral-large-2512

Table 1: The 32 evaluated models grouped by family. Dates marked as approximate in Table 3 are used only for ordering and horizontal placement on the time axis. among predominantly closed models: whether cross-family distances decrease with release date and how closely the resulting geometry resembles a tree.

Data and Methods

Models and prompts. Let M = 32 denote the number of evaluated models. They span six families: GPT (Radford et al. 2019), Claude (Anthropic 2024), Gemini (Gemini Team 2023), Qwen (Qwen Team 2024), Llama (Grattafiori et al. 2024), and Mistral (Jiang et al. 2023). Every model answers the same canonical bank of N =10,000 prompts drawn from 13 public benchmarks, including MMLU (Hendrycks et al. 2021a), ARC (Clark et al. 2018), HellaSwag (Zellers et al. 2019), WinoGrande (Sakaguchi et al. 2020), GSM8K (Cobbe et al. 2021), MATH (Hendrycks et al. 2021b), and TruthfulQA (Lin, Hilton, and Evans 2022). Thus, response index i always refers to the same prompt across models. The complete benchmark composition and model roster appear in the appendix (Tables 4 and 3); Table 1 summarizes the six families and their release spans. Response embeddings. We encode each complete response with Qwen3-Embedding-8B (Zhang et al. 2025) and mean-pool its token embeddings, following the sentenceembedding paradigm (Reimers and Gurevych 2019). The encoder has a maximum input length of 1024 tokens and produces a 4096-dimensional vector; we do not apply L2 normalization. For model m, the resulting matrix Em ∈ RN ×4096 contains one response embedding per prompt and is row-aligned with every other model’s matrix. Cross-model dissimilarities. For models m and m′ , we construct an M × M pairwise matrix in three ways (Fig. 1). • Mean per-prompt distance. We preserve the known prompt alignment and average Euclidean distances between paired response embeddings: N

mean Dm,m ′ =

1 X ∥Em [i] − Em′ [i]∥2 . N i=1

(1)

It is a pseudometric on models represented by their observed embedding matrices; it becomes a metric after quotienting out models with identical encoded responses on every prompt (Proposition 1 in Appendix). • PCA-compressed disagreement. For prompt i, let M (i) ∈ RM ×M contain the pairwise distances (i) Mm,m′ = ∥Em [i] − Em′ [i]∥2 . For each model pair,

(a) Mean per-prompt Dmean Em

Em′

(b) PCA-compressed Dpca stacked over N

Cm PCA

∥Em [i] − Em′ [i]∥

per-prompt 32 × 32

D: PC1 scores optimal coupling

variance-weighted compression over prompts

matches internal geometries; alignment-invariant

Figure 1: Three sentence-level comparison constructions. (a) The mean per-prompt distance compares embeddings of the same prompt and averages the Euclidean distances; it is an aligned pseudometric. (b) The PCA construction compresses each pair’s vector of prompt-wise distances onto the leading disagreement direction; the resulting score need not be a metric. (c) GW compares internal response geometry under an optimized coupling. With cosine costs, it is invariant to orthogonal transformations and positive rescaling of individual embedding vectors, and it does not require prompt indices to be matched. collect its prompt-wise disagreement profile xm,m′ = (1) (N ) (Mm,m′ , . . . , Mm,m′ )⊤ ∈ RN . Applying rank-one PCA to these pair profiles gives pca sm,m′ = xm,m′ − x̄, u1 , Dm,m (2) ′ = S(sm,m′ ),

where u1 is the leading principal direction and x̄ is the mean across model pairs, and S denotes the sign orientation, symmetrization, and rescaling used to form the displayed matrix. It is a one-dimensional summary of how pairs disagree across prompts, not necessarily a metric. • Gromov–Wasserstein discrepancy. For model m, let Cm ∈ RN ×N be the matrix of cosine dissimilarities among its own response embeddings, and let um be the uniform measure on those responses. We compute the squared GW discrepancy X 2 gw Dm,m min Cm [a, b]−Cm′ [c, d] πac πbd , ′ = π∈Π(um ,um′ )

a,b,c,d

(3) where Π(um , um′ ) = {π ≥ 0 : π1 = um , π ⊤ 1 = um′ }. GW compares internal response geometry under an optimized coupling and therefore does not require prompt indices to be matched. Because Cm uses cosine dissimilarity, the construction is unchanged by modelspecific orthogonal transformations and positive rescaling of the embedding vectors, but not by translations. We solve Eq. (3) with POT’s conditional-gradient solver (Flamary et al. 2021). Full N × N costs are computationally prohibitive, so all models use the same seeded subsample of 256 prompts. A measure-theoretic comparison lens. Let qm (· | x) denote model m’s response distribution for prompt x, and let x νm = ϕ# qm (· | x) be its pushforward under the response encoder ϕ, e.g., Qwen3-Embedding-8B. The aligned constructions condition on the same prompt x for both models. At the sentence level, the population quantity naturally associated with one independently generated response from each model is ′ x ⊗ν x ∥Z − Z ∥2 , Ex E(Z,Z ′ )∼νm (4) ′ m

When the prompt bank is viewed as a sample from the inference distribution and responses are sampled independently, Eq. (1) is the corresponding Monte Carlo estimate.

This quantity differs from the distance between conditional means, ∥EZ − EZ ′ ∥2 , unless decoding is deterministic or within-prompt variability is negligible. The PCA construction does not define a new per-prompt divergence; it compresses the vector of aligned per-prompt dissimilarities into a single score. The token-level MMD analysis, to be introduced in Section 5, remains prompt-aligned but replaces each pooled response vector by the empirical measure of its token embeddings. In contrast, GW discards the fixed prompt alignment and compares only the internal geometry of each model’s response cloud under an optimized coupling. Thus, the constructions form a hierarchy of comparison assumptions rather than four instances of a single metric: they differ in representation (pooled response versus token measure), aggregation (mean versus PCA compression), and alignment (fixed prompts versus optimized coupling). Their high empirical rank correlations therefore provide a robustness check: the principal organization of the models is stable despite these changes in representation and alignment. For token MMD, taking the square root recovers a metric on empirical measures when the kernel is characteristic. We report the conventional squared GW objective on cosine-dissimilarity networks; taking its square root leaves all rank-based analyses unchanged, although a formal metric interpretation of GW requires metric ground costs. Visualization and temporal summaries. We visualize each pairwise matrix with metric multidimensional scaling (MDS) (Kruskal 1964), t-SNE (van der Maaten and Hinton 2008), and UMAP (McInnes, Healy, and Melville 2018); perfamily panels reuse the corresponding global coordinates. To study evolution, we associate each model with its release date and report (i) cumulative drift from the first model in its family, (ii) step drift between consecutive family releases, (iii) an ordered distance heatmap and average-linkage dendrogram, (iv) mean distance to models from other families, and (v) response-cloud dispersion, defined as the mean off-diagonal entry of Cm . A few release dates are approximate and affect only the horizontal placement of points, not any pairwise dissimilarity. The analysis is related in spirit to representationsimilarity methods (Kornblith et al. 2019), but uses model outputs rather than hidden activations.

GPT Claude Gemini Qwen Llama Mistral

Figure 2: Behavioral map obtained by applying metric MDS to the mean per-prompt distance Dmean . Marker shape and color identify model family; darker shades denote later releases. Labels are omitted for readability and are provided in Fig. 6 and Table 3. gpt-2 (the lightest GPT marker) is the global outlier. The concentric rings are centered on the dark red dot, the mean map position of the six families’ latest releases; their radii are evenly spaced out to the 95th percentile of distances to that center.

Results

Behavioral cartography (Fig. 2). The mean per-prompt distance Dmean exhibits strong family structure: 21 of the 32 models have a nearest neighbor from the same family. The closest pair is gpt-3.5-turbo and gpt-4.1-mini (Dmean ≈ 6.8). In contrast, gpt-2 has the largest average distance to all other models (≈ 88.5) and is therefore the global outlier under this construction. Several cross-family nearest-neighbor relations involve recent GPT, Gemini, and Qwen models, which is consistent with the cross-family convergence examined below. Relational geometry via GW (Fig. 10, appendix). Because GW compares internal geometry rather than absolute embedding coordinates, its family signal is weaker: 14/32 models have a same-family nearest neighbor. Several recent models pair across vendors, including gemini-3.1-pro-preview with qwen3.7-max and an opus-class Claude model with gpt-5.2. These pairings indicate similar response-space geometry; they should not be interpreted as direct comparisons of capability. Drift (Fig. 3). Family-wise drift quantifies the observed displacement between releases. The gpt-2→gpt-3.5-turbo transition is substantially larger than every later step. Among releases dated 2024– 2026, step distances are smaller and broadly comparable across families. This is a descriptive pattern; the distance

step drift cumulative drift (distance from previous version)(distance from first version)

sentence_mean MDS

Behavioral drift over time 100

GPT Claude

source: sentence_mean

Gemini Qwen

Llama Mistral

50 25 0 100 75 50 25 0

201

202

release date

202

Figure 3: Behavioral drift over time (mean metric). Top: cumulative drift from each family’s first version; bottom: step drift between consecutive versions. The gpt-2→gpt-3.5-turbo jump dwarfs modern steps.

alone does not identify which changes in data, training, or decoding produced the drift. Block and hierarchical structure (Fig. 4). Ordering the distance matrix by family and date reveals low-distance within-family blocks and two high-distance rows and columns: gpt-2 and qwen-2.5-72b-instruct. The average-linkage dendrogram separates these models early; several subsequent clusters contain models from multiple vendors. The hierarchy therefore reflects lineage only partially, especially among recent releases. Convergence and dispersion (Fig. 5). Mean distance from each model to models in other families decreases with release date, yielding a negative fitted trend. This provides descriptive evidence of behavioral homogenization among later releases, although the observational analysis does not establish a causal temporal mechanism. Responsecloud dispersion provides a complementary within-model statistic. Several recent reasoning-oriented models, including claude-opus-4-7-think and gpt-5.2, have lower dispersion than some mid-generation predecessors; under the chosen encoder, their responses therefore occupy a more compact embedding cloud. Robustness across projections and constructions. The principal structure is stable across metric MDS, t-SNE, and UMAP (Fig. 13), so it is not specific to one projection. The PCA-compressed score tracks the mean distance closely (Spearman ρ = 0.92; 17/32 same-family nearest neighbors; Fig. 14), while GW is less correlated (ρ = 0.77), as expected from its removal of fixed prompt alignment and absolute coordinates. Family blocks, the gpt-2 outlier, and the downward trend nonetheless persist under all three sentence-level constructions.

source: sentence_mean

Convergence over time 100

distance

source: sentence_mean (cross-family trend: converging) GPT Claude Gemini

Qwen Llama Mistral

cross-family (slope=-1.37e-02/day) within-family (slope=-1.80e-02/day)

70 60 50 40

201

202

0.54

GPT Claude

Gemini Qwen

202

mean within-model response distance (dispersion)

gpt-2 gpt-3.5-turbo gpt-4o-mini gpt-4.1-mini gpt-5.2 claude-opus-4-0 claude-sonnet-4-0 claude-sonnet-4-5-think claude-haiku-4-5 claude-opus-4-5-think claude-sonnet-4-6 claude-opus-4-6 claude-opus-4-7-think gemini-2.0-flash gemini-2.5-pro gemini-3-flash-preview gemini-3.1-pro-preview qwen-2.5-72b-instruct qwen3-235b-a22b-instruct-2507 qwen3.5-397b-a17b qwen3.6-max-preview qwen3.7-max llama-3-8b-instruct llama-3.1-70b-instruct llama-3.3-70b-instruct llama-4-maverick llama-4-scout mixtral-8x22b-instruct mistral-nemo mistral-small-24b-instruct-2501 mistral-medium-3.1 mistral-large-2512

source: sentence_mean

80 60

202

Response-cloud dispersion over time Llama Mistral

0.52 0.50 0.48 0.46 0.44 0.42 0.40

201

202

release date

Hierarchical clustering (average linkage)

merge distance

mean cross-family behavioral distance

Distance matrix, ordered by family + release date gpt-2 gpt-3.5-turbo gpt-4o-mini gpt-4.1-mini gpt-5.2 claude-opus-4-0 claude-sonnet-4-0 claude-sonnet-4-5-think claude-haiku-4-5 claude-opus-4-5-think claude-sonnet-4-6 claude-opus-4-6 claude-opus-4-7-think gemini-2.0-flash gemini-2.5-pro gemini-3-flash-preview gemini-3.1-pro-preview qwen-2.5-72b-instruct qwen3-235b-a22b-instruct-2507 qwen3.5-397b-a17b qwen3.6-max-preview qwen3.7-max llama-3-8b-instruct llama-3.1-70b-instruct llama-3.3-70b-instruct llama-4-maverick llama-4-scout mixtral-8x22b-instruct mistral-nemo mistral-small-24b-instruct-2501 mistral-medium-3.1 mistral-large-2512

202

release date

202

claude-sonnet-4-0

llama-4-scout

claude-opus-4-0

claude-sonnet-4-5-think

llama-4-maverick

gemini-2.0-flash

claude-haiku-4-5

llama-3-8b-instruct

mistral-nemo

llama-3.3-70b-instruct

llama-3.1-70b-instruct

claude-opus-4-6

claude-sonnet-4-6

qwen3.7-max

claude-opus-4-5-think

qwen3.6-max-preview

gemini-3.1-pro-preview

gpt-5.2

gemini-2.5-pro

qwen3.5-397b-a17b

gpt-4.1-mini

gemini-3-flash-preview

gpt-4o-mini

gpt-3.5-turbo

mistral-large-2512

mistral-medium-3.1

mistral-small-24b-instruct-2501

claude-opus-4-7-think

mixtral-8x22b-instruct

qwen-2.5-72b-instruct

qwen3-235b-a22b-instruct-2507

gpt-2

Figure 4: Top: distance matrix ordered by family and release date; white lines separate families, labels are familycolored. Bottom: average-linkage hierarchical clustering; gpt-2 and qwen-2.5-72b-instruct split off first, and recent models mix across families.

Robustness to the response encoder. The constructions above share one encoder, so their agreement probes aggregation and alignment rather than the choice of ϕ. Re-encoding all 32 × 104 responses with bge-m3, multilingual-e5-large, and all-mpnet-base-v2—the last 73× smaller, none sharing a vendor with an evaluated model—preserves the geometry: ρ = 0.92–0.93 over the 496 pairs, 19–22 of 32 samefamily nearest neighbors against 21, and a negative trend throughout (r from −0.52 to −0.55). Every encoder returns gpt-2 and qwen-2.5-72b-instruct as the two outliers, so the Qwen-family outlier is recovered by encoders with no Qwen lineage (Table 7, Fig. 8).

Figure 5: Top: mean distance from each model to models in other families versus release date, together with the fitted downward trend. Bottom: within-model response-cloud dispersion over time. Several recent reasoning-oriented models have comparatively compact response clouds. Distance construction Mean per-prompt (pseudometric) PCA-compressed Gromov–Wasserstein

same-family NN 21/32 17/32 14/32

ρ vs. mean 1.00 0.92 0.77

Table 2: Agreement among sentence-level constructions. The first column reports the fraction of models whose nearest neighbor is from the same family; the second reports Spear man correlation over the 32 = 496 distinct model pairs. 2 Construction. For two responses with token embedbX = dings X = {xa }Ta=1 and Y = {yb }Sb=1 , define λ PT PS −1 −1 b T a=1 δxa and λY = S b=1 δyb . Their dissimilarity is the squared MMD under an RBF kernel: X 1 X bX , λ bY ) = 1 MMD2 (λ k(xa , xa′ ) + 2 k(yb , yb′ ) 2 T S ′ ′ a,a

b,b

2 X − k(xa , yb ). TS a,b

A Token-level Lens: Per-prompt MMD

The sentence-level analyses pool every response into one vector. As an independent cross-check, we retain the token embeddings within each response and compare the resulting empirical token measures prompt by prompt.

(5) This is the exact squared RKHS distance between the two observed empirical measures; viewed as an estimator of a latent population-token MMD, it is the usual biased plug-in estimator. It is well defined for every response length. For two single-token responses, Eq. (5) reduces to 2(1 − exp[−∥x − y∥22 /(2σ 2 )]), so no special case is needed.

gpt-2 gpt-3.5-turbo gpt-4o-mini gpt-4.1-mini gpt-5.2

claude-sonnet-4-0 claude-opus-4-0 claude-haiku-4-5 claude-sonnet-4-5-think claude-opus-4-5-think claude-sonnet-4-6 claude-opus-4-6 claude-opus-4-7-think

gemini-2.0-flash gemini-2.5-pro gemini-3-flash-preview gemini-3.1-pro-preview

qwen-2.5-72b-instruct qwen3-235b-a22b-instruct-2507 qwen3.5-397b-a17b qwen3.6-max-preview qwen3.7-max

Figure 6: Per-family detail for Fig. 2: a common ring-bounded frame (identical axes) with oldest→latest arrows. Left to right: GPT, Claude, Gemini, Qwen; all six families appear in Fig. 9 (appendix). We use the same estimator for all lengths because the unbiased U-statistic is undefined for a one-token response. The plug-in estimator’s finite-sample bias as an estimator of population MMD depends on response length (median: 74 tokens), so unequal lengths could perturb comparisons. The high empirical agreement with the sentence-level analysis reported below suggests that this issue does not materially alter the model-pair rankings in our data. For each prompt, Eq. (5) yields an M ×M matrix. Stacking over prompts gives an N ×M ×M tensor. Token embeddings are streamed rather than stored; only the approximately 40 MB tensor is written to disk. We aggregate it by the mean and by the same PCA procedure used for the sentence-level profiles. Agreement with the sentence-level analysis. The mean token-level matrix correlates strongly with the sentence-level mean distance: Spearman ρ = 0.98 and Pearson 0.97 over the 32 2 = 496 distinct model pairs, and 28/32 models share a nearest neighbor under the two constructions. The token analysis recovers the same qualitative structure: the gpt-2 and qwen-2.5-72b outliers, coherent family blocks (Figs. 7 and 15), and a decreasing cross-family trend (Fig. 16). That a pooled sentence representation and a token-distribution representation agree indicates the main conclusions are not driven by one representation alone.

A Theory of Behavioral Similarity

token_mmd_mean MDS GPT Claude Gemini Qwen Llama Mistral

Figure 7: Metric-MDS map of the token-level dissimilarity obtained by averaging per-prompt squared MMD. Visual encoding follows Fig. 2. The family structure and the gpt-2 outlier closely match the sentence-level map (Spearman ρ = 0.98); per-family details appear in Fig. 12.

Let ξ denote the inference-prompt distribution. For each The empirical analysis is output-side; it does not identify j ∈ {m, m′ }, assume the covariate-coverage condition the training causes of convergence. We nevertheless give a dξ sufficient training-side condition under which two models ξ ≪ Pj,X , ≤ κj < ∞. (7) dP must be behaviorally close. j,X ∞ Let Pm be an effective target population for model m, with Absolute continuity excludes inference regions that receive prompt marginal Pm,X and conditional response distribution no mass under the effective training prompt distribution, pm (r | x). For a model trained only by next-token prediction, where training risk gives no control. The density-ratio bound this may be interpreted as its training population. For a postlimits how strongly inference can upweight prompts that are trained system, treating Pm as an effective target distribution rare during training; κj therefore quantifies the severity of is an additional modeling assumption. The deployed model covariate shift. This assumption is weaker than equality of defines qm (r | x), including the end-of-sequence token, and the training and inference prompt distributions. has population negative-log-likelihood risk For probability measures P and Q on the response space, define Rm (q) = E(x,r)∼Pm [− log q(r | x)]. TV(P, Q) = sup |P (A) − Q(A)|, The population minimizer over all conditional distributions A is pm , and the excess risk is where the supremum is over measurable events A. Total variεm = Rm (qm )−Rm (pm ) = Ex∼Pm,X KL(pm (· | x)∥qm (· | x)) . ation is the largest probability discrepancy assigned to the (6) same event.

Theorem 1 (Behavioral similarity from population risk). For models m and m′ , define the effective target-distribution discrepancy txt ηm,m ′ = Ex∼ξ TV(pm (· | x), pm′ (· | x)) .

If the excess risks in Eq. (6) are finite and the coverage condition in Eq. (7) holds for both models, then Ex∼ξ TV(qm (· | x), qm′ (· | x)) r r κm εm κm′ εm′ txt ≤ ηm,m + . ′ + 2 2

7 (8)

The bound separates deployed-model difference into one discrepancy between the two effective targets and two learning-error terms. Conditional on these quantities, the bound does not depend explicitly on architecture, initialization, or optimizer. Small excess risk alone is insufficient: the models can remain different when their effective target distributions differ. The full proof appears in Appendix A.4. Textually different responses may nevertheless be semantically close. Let ϕ : R → Rd be a semantic encoder, and x x = ϕ# pm (· | x). For prob= ϕ# qm (· | x) and πm define νm d ability measures α and β on R with finite first moments, the 1-Wasserstein distance with Euclidean ground cost is Z W1 (α, β) = inf ∥u − v∥2 dγ(u, v), (9) γ∈Π(α,β)

where Π(α, β) is the set of couplings with marginals α and β. Thus, W1 is the smallest expected embedding distance achievable by jointly aligning the two distributions. Assume there exists a set Sϕ ⊂ Rd containing the supports of νjx and πjx for j ∈ {m, m′ } and ξ-almost every x, with diam(Sϕ ) ≤ Bϕ . Define sem x x ηm,m ′ = Ex∼ξ W1 (πm , πm′ ).

Corollary 1 (Semantic behavioral similarity). Under the assumptions of Theorem 1 and the bounded embeddingdiameter condition above, x x Ex∼ξ W1 (νm , νm ′) r r κm εm κm′ εm′ sem ≤ ηm,m + B + B . ′ ϕ ϕ 2 2

(10)

Corollary 2 (Conditional mean-embedding similarity). Let µm (x) = Er∼qm (·|x) [ϕ(r)] be the conditional mean response embedding. Under the assumptions of Corollary 1, Ex∼ξ ∥µm (x) − µm′ (x)∥2 r r κm εm κm′ εm′ sem ≤ ηm,m′ + Bϕ + Bϕ . 2 2

excess risks and increasingly similar effective target distributions are sufficient for relative semantic homogenization, potentially toward a nonzero floor. These are distributional results. A guarantee for deterministic greedy decoding requires an additional stability condition, such as a positive probability margin.

(11)

Corollary 2 controls the distance between conditional mean embeddings. The empirical distance in Eq. (1) instead uses one generated response from each model and prompt. Its expectation is bounded by the same right-hand side plus one within-prompt spread term for each model; Appendix A.4 states this connection precisely. Consequently, decreasing

Tree-likeness and Low-rank Structure

Approximate tree structure. We quantify tree-likeness using the four-point defect associated with Gromov δhyperbolicity (definition and qualification in Appendix A.3). After normalization by matrix diameter, the mean defect is 0.015 for both the sentence and token constructions. The maximum normalized defect is larger (0.12–0.16), indicating small average but localized departures from a tree-like geometry. Because the token matrix averages squared MMD, its four-point value is a descriptive diagnostic rather than a formal hyperbolicity constant. These localized defects are consistent with cross-family mixing in the dendrogram but do not establish a particular evolutionary mechanism. Low-dimensional disagreement structure. The promptwise token-MMD tensor admits a compact low-rank approximation: its leading component explains 63% of prompt-toprompt variance, and the first five explain 77%. A few axes thus summarize much of how model pairs disagree across prompts, supporting the PCA summary and suggesting that high-leverage prompts could yield smaller diagnostic sets, though prompt selection is not evaluated here.

Discussion and Limitations

The agreement across constructions indicates that the family structure and decreasing cross-family distances are not specific to one representation or alignment assumption. Behavioral similarity does not, however, imply correctness. Absolute distances depend on the prompt bank, encoder, and decoding configuration, though the reported structure survives the encoder swaps above. With one response per model– prompt pair, the distances also carry generation variability; release-date trends are observational, GW uses a 256-prompt subsample, and the model panel is not exhaustive. Finally, the training-side quantities are unobserved here, so the theory supplies a sufficient scenario rather than an empirical explanation.

Conclusion

We presented a label-free framework for mapping LLM behavior from responses to a shared prompt bank. Across sentence- and token-level constructions it reveals family coherence, outliers, decreasing cross-family distances, and compact response clouds for recent models. We also gave an architecture-agnostic sufficient condition for behavioral similarity that, with its training-side quantities unobserved, complements rather than explains the findings. Output geometry augments leaderboards when weights, activations, or labels are unavailable.

References Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. Model card, Anthropic. Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; et al. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732. Chang, T. A.; and Bergen, B. K. 2024. Language model behavior: A comprehensive survey. Computational Linguistics, 50(1): 293–350. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; et al. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168. Flamary, R.; et al. 2021. POT: Python Optimal Transport. Journal of Machine Learning Research, 22(78): 1–8. Gemini Team. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312.11805. Goel, S.; Strüber, J.; Auzina, I. A.; Chandra, K. K.; Kumaraguru, P.; Kiela, D.; Prabhu, A.; Bethge, M.; and Geiping, J. 2025. Great Models Think Alike and this Undermines AI Oversight. In Forty-second International Conference on Machine Learning. Grattafiori, A.; et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Gretton, A.; Borgwardt, K. M.; Rasch, M. J.; Schölkopf, B.; and Smola, A. 2012. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13: 723–773. Gröger, F.; Wen, S.; and Brbić, M. 2026. Revisiting the Platonic Representation Hypothesis: An Aristotelian View. In International Conference on Machine Learning (ICML). Gu, A.; and Dao, T. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021a. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR). Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021b. Measuring Mathematical Problem Solving With the MATH Dataset. In NeurIPS Datasets and Benchmarks Track. Huh, M.; Cheung, B.; Wang, T.; and Isola, P. 2024. Position: The Platonic Representation Hypothesis. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235, 20617–20642. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825.

Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL). Koepke, A. S.; Zverev, D.; Ginosar, S.; and Efros, A. A. 2026. Back into Plato’s Cave: Examining Cross-modal Representational Convergence at Scale. arXiv preprint arXiv:2604.18572. Kornblith, S.; Norouzi, M.; Lee, H.; and Hinton, G. 2019. Similarity of Neural Network Representations Revisited. In International Conference on Machine Learning (ICML). Kruskal, J. B. 1964. Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis. Psychometrika, 29(1): 1–27. Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; et al. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7: 452–466. Lan, M.; Torr, P.; Meek, A.; Khakzar, A.; Krueger, D.; and Barez, F. 2024. Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders. arXiv preprint arXiv:2410.06981. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. Lin, S.; Hilton, J.; and Evans, O. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Annual Meeting of the Association for Computational Linguistics (ACL). McInnes, L.; Healy, J.; and Melville, J. 2018. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv preprint arXiv:1802.03426. Mémoli, F. 2011. Gromov–Wasserstein Distances and the Metric Approach to Object Matching. Foundations of Computational Mathematics, 11(4): 417–487. Pasquini, D.; Kornaropoulos, E. M.; and Ateniese, G. 2025. {LLMmap}: Fingerprinting for large language models. In 34th USENIX Security Symposium (USENIX Security 25), 299–318. Pei, Z.; Zhen, H.-L.; Zhang, Y.; Yang, Z.; Li, X.; Yu, X.; Yuan, M.; and Yu, B. 2025. Behavioral Fingerprinting of Large Language Models. arXiv preprint arXiv:2509.04504. Peyré, G.; and Cuturi, M. 2019. Computational Optimal Transport. Foundations and Trends in Machine Learning, 11(5–6): 355–607. Qwen Team. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI. Rajpurkar, P.; Jia, R.; and Liang, P. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Annual Meeting of the Association for Computational Linguistics (ACL).

Reimers, N.; and Gurevych, I. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP). Ribeiro, M. T.; Wu, T.; Guestrin, C.; and Singh, S. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th annual meeting of the association for computational linguistics, 4902–4912. Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2020. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In AAAI Conference on Artificial Intelligence. Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; GarrigaAlonso, A.; et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research. van der Maaten, L.; and Hinton, G. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9: 2579–2605. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30. Wang, J.; Ge, X.; Shu, W.; Tang, Q.; Zhou, Y.; He, Z.; and Qiu, X. 2025. Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures. In International Conference on Learning Representations (ICLR). Wu, Z.; Zhao, H.; Wang, Z.; Guo, J.; Wang, Q.; and He, B. 2026. LLM DNA: Tracing Model Evolution via Functional Representations. In International Conference on Learning Representations (ICLR). Yax, N.; Oudeyer, P.-Y.; and Palminteri, S. 2025. PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks. In The Thirteenth International Conference on Learning Representations. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Annual Meeting of the Association for Computational Linguistics (ACL). Zhang, Y.; Li, M.; Long, D.; Zhang, X.; Lin, H.; Yang, B.; Xie, P.; Yang, A.; Liu, D.; Lin, J.; Huang, F.; and Zhou, J. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint arXiv:2506.05176. Zhang, Z.; Prasad, I. V.; Hu, Y.; Liu, Z.; Luo, H.; Ren, P.; and Yang, Y. 2026. Spectral Signatures of Large Language Models. arXiv preprint arXiv:2607.03377.

A A.1

Supplementary Material

Additional Proofs and Derivations

Full proof of Proposition 1. Proposition 1. The mean per-prompt distance in Eq. (1) is a pseudometric on the observed model representations {Em }M m=1 and a metric on the quotient space induced by equality of all encoded responses. The proof is given in Appendix A.1. PN mean −1 ′ For each prompt i, define di (m, m′ ) = ∥Em [i] − Em′ [i]∥2 . Then Dm,m ′ = N i=1 di (m, m ). Proof. For every prompt i, Euclidean distance implies nonnegativity, symmetry, and di (m, m′ ) ≤ di (m, ℓ) + di (ℓ, m′ ) mean for any third model ℓ. Averaging these pointwise inequalities over i = 1, . . . , N preserves all three properties, and Dm,m = 0. mean Hence D is a pseudometric. mean It may fail identity of indiscernibles at the model level. Specifically, Dm,m ′ = 0 if and only if Em [i] = Em′ [i] for every prompt i. Distinct underlying models can satisfy this equality if they produce encoder-equivalent responses on the evaluated prompts. After quotienting by the relation m ∼ m′ ⇐⇒ Em [i] = Em′ [i] for all i, identity of indiscernibles holds, so the induced function is a metric on the quotient space. bX = T −1 PT δx and λ bY = S −1 PS δy be the empirical token Token MMD for single-token responses. Let λ a b a=1 b=1 measures of two responses. The plug-in squared MMD is X 2 1 X 2 X \ (X, Y ) = 1 MMD k(xa , xa′ ) + 2 k(yb , yb′ ) − k(xa , yb ). (12) 2 T S TS ′ ′ a,a

b,b

a,b

For single-token responses, T = S = 1 and X = {x}, Y = {y}. With the RBF kernel k(u, v) = exp(−∥u − v∥22 /(2σ 2 )), Eq. (12) becomes 2 \ (X, Y ) = 2 1 − e−∥x−y∥22 /(2σ2 ) . MMD (13) It is zero when x = y, increases monotonically with ∥x − y∥2 , and approaches 2 as the distance diverges. Thus, a one-token response is handled naturally as a one-point empirical measure. Equation (12) is exact for the two observed empirical measures; if interpreted as an estimator of an underlying population-token MMD, it remains the biased plug-in estimator, with bias that can be substantial for short responses. Relationships among the comparison constructions. The constructions are related, but they are not all metrics of the same form. The mean sentence-level construction averages Euclidean distances under a fixed prompt alignment and is a pseudometric by Proposition 1. With a characteristic kernel, MMD is a metric on probability measures (Gretton et al. 2012); our token entries use its square, which is a dissimilarity whose square root is the corresponding metric. The PCA construction is instead a rank-one compression of the vector of aligned prompt-wise distances and need not satisfy metric axioms. GW replaces the fixed prompt alignment by an optimized coupling between internal cost matrices. When the within-space costs are metrics, the square root of the standard GW objective is a metric on metric-measure spaces modulo measure-preserving isometry (Mémoli 2011). Our within-model costs are cosine dissimilarities, which need not satisfy the triangle inequality; the appropriate object is therefore a measure-network discrepancy rather than a formal metric. These costs are nevertheless invariant to orthogonal transformations and positive rescaling of the embedding vectors. Accordingly, the four analyses vary along three axes: representation (pooled response embeddings or token empirical measures), aggregation (mean or PCA), and alignment (fixed prompts or optimized coupling). The empirical correlations in Table 5 show that the resulting rankings are similar, but they do not imply mathematical equivalence of the constructions.

A.2

Additional Experimental Details

Model roster. Table 3 lists all 32 models with the release dates used as the time axis; the figures abbreviate these identifiers. Dates marked ∗ are approximate (post-cutoff or unannounced at the time of the runs); they affect only the horizontal placement of points, never the distances. Question bank composition. Every model answers the same canonical bank of N =10,000 prompts, drawn as a stratified sample from 13 public benchmarks: MMLU (Hendrycks et al. 2021a), ARC (Clark et al. 2018), BIG-Bench (Srivastava et al. 2023), HellaSwag (Zellers et al. 2019), WinoGrande (Sakaguchi et al. 2020), MATH (Hendrycks et al. 2021b), GSM8K (Cobbe et al. 2021), SQuAD 2.0 (Rajpurkar, Jia, and Liang 2018), TriviaQA (Joshi et al. 2017), Natural Questions (Kwiatkowski et al. 2019), TruthfulQA (Lin, Hilton, and Evans 2022), HumanEval (Chen et al. 2021), and MBPP (Austin et al. 2021). Table 4 gives the per-benchmark counts. The multiple-choice and short-answer benchmarks are sampled to a common target (≈876 each); the two code benchmarks (HumanEval, MBPP) contribute their smaller available size. Response index i refers to the same prompt for every model. The bank stores prompts only, with no gold labels, consistent with the label-free design.

Family GPT

Claude

Gemini

Qwen

Llama

Mistral

Model ID gpt-2 gpt-3.5-turbo gpt-4o-mini gpt-4.1-mini gpt-5.2 claude-sonnet-4-0 claude-opus-4-0 claude-sonnet-4-5-think claude-haiku-4-5 claude-opus-4-5-think claude-sonnet-4-6 claude-opus-4-6 claude-opus-4-7-think gemini-2.0-flash gemini-2.5-pro gemini-3-flash-preview gemini-3.1-pro-preview qwen/qwen-2.5-72b-instruct qwen3-235b-a22b-instruct-2507 qwen/qwen3.5-397b-a17b qwen/qwen3.6-max-preview qwen3.7-max meta-llama/llama-3-8b-instruct meta-llama/llama-3.1-70b-instruct meta-llama/llama-3.3-70b-instruct meta-llama/llama-4-scout meta-llama/llama-4-maverick mistralai/mixtral-8x22b-instruct mistralai/mistral-nemo mistralai/mistral-small-24b-instruct-2501 mistralai/mistral-medium-3.1 mistralai/mistral-large-2512

Release date 2019-02-14 2022-11-30 2024-07-18 2025-04-14 2025-12-01∗ 2025-05-22 2025-05-22 2025-09-29∗ 2025-10-01∗ 2025-11-01∗ 2025-12-01∗ 2026-01-01∗ 2026-03-01∗ 2024-12-11 2025-03-25 2025-11-01∗ 2026-02-01∗ 2024-09-19 2025-07-21 2025-10-01∗ 2025-12-01∗ 2026-02-01∗ 2024-04-18 2024-07-23 2024-12-06 2025-04-05 2025-04-05 2024-04-17 2024-07-18 2025-01-30 2025-08-01∗ 2025-12-01∗

Table 3: The full 32-model roster: model identifier and the release date used on the time axis. Figures abbreviate these identifiers. ∗ : approximate date (post-cutoff or unannounced at the time of the runs); approximate dates affect only horizontal placement, never the distances.

Response embeddings. Responses are encoded with Qwen3-Embedding-8B. Each response is read as at most 1024 tokens (truncation limit on the input sequence length); the encoder returns one hidden vector per token in its native 4096-dimensional output space (the model’s hidden width, fixed by architecture and independent of response length), and we mean-pool over the token positions into a single 4096-dimensional vector per response. Thus a length-T response (T ≤ 1024) maps a (T, 4096) array of token embeddings to one 4096-vector; the two numbers are different axes—1024 counts input tokens, 4096 is the output width. We use no L2 normalization and an encoder batch size of 8, giving one matrix Em ∈ R10000×4096 per model, row-aligned across models by prompt. Cross-model dissimilarities. The mean per-prompt distance and PCA-compressed scores are computed directly from {Em } using Euclidean per-prompt distances. For Gromov–Wasserstein we build, per model, a cosine response-by-response cost matrix over a common seeded subsample of 256 responses (seed 0), assign uniform marginals, and solve GW2 with the conditionalgradient solver of POT. The token-level MMD uses the same encoder with at most 256 tokens per response, an RBF kernel with a global-median bandwidth (estimated from a sample of responses), and the biased (plug-in) estimator applied uniformly to all response lengths; the per-prompt 32 × 32 matrices are then averaged over prompts. Projections. 2D embeddings are computed from each precomputed 32 × 32 distance matrix with metric MDS (SMACOF, max_iter= 300, n_init= 4), t-SNE (perplexity= 5), and UMAP (n_neighbors= 5, min_dist= 0.1). All three use metric=precomputed and seed 0. The per-family panels reuse the overview’s 2D coordinates and share one ring center. Software and compute. Response generation is routed through OpenRouter, except the local gpt-2 baseline, which runs on-device; each model produces one response per prompt under a single decoding configuration. Embeddings are produced on a single local GPU, while all distances, projections, and fits run on CPU with NumPy, scikit-learn, and POT. Per-response embeddings are cached, so any distance can be recomputed without re-encoding.

Benchmark BIG-Bench ARC MMLU HellaSwag WinoGrande MATH GSM8K SQuAD 2.0 TriviaQA Natural Questions TruthfulQA MBPP HumanEval Total

# prompts 877 877 876 876 876 876 876 876 876 876 817 257 164 10,000

Table 4: Composition of the canonical 10k question bank by source benchmark.

Mean PCA GW Token

Mean 1.00

PCA .92 1.00

GW .77 .80 1.00

Token .98 .95 .83 1.00

Table 5: Spearman correlation between distance constructions over the 496 distinct off-diagonal model pairs (upper triangle). The Mean column matches Table 2.

A.3

Additional Results

Full agreement among distance constructions. Table 2 in the main text reports each construction’s Spearman correlation with the mean metric. Table 5 gives the complete pairwise matrix over the 32 =496 distinct off-diagonal model pairs. Beyond 2 the mean-metric column (which reproduces Table 2), two patterns stand out: the token-level MMD is the closest match to the sentence mean metric (ρ=0.98) and also tracks the PCA-compressed variant tightly (ρ=0.95), while Gromov–Wasserstein is the most distinct construction, correlating no higher than 0.83 with any other. This is consistent with GW discarding fixed prompt alignment and absolute embedding coordinates, and therefore capturing a different aspect of response geometry. Structure tests. For a quadruple (a, b, c, d), let S1 , S2 , S3 be the three sums Dab + Dcd , Dac + Dbd , and Dad + Dbc , ordered so that S(1) ≤ S(2) ≤ S(3) . We define its four-point defect as δ(a, b, c, d) =

S(3) − S(2) . 2

For a finite metric, every defect is zero exactly when the metric is 0-hyperbolic (equivalently, tree-additive on the observed points). Small normalized defects therefore quantify approximate tree-likeness when the input is a metric. We report the mean and maximum defect over evaluated quadruples, divided by matrix diameter. The sentence mean is a metric on its quotient space; the token matrix averages squared MMD values and need not satisfy the triangle inequality, so its defect is interpreted only as a descriptive tree-likeness diagnostic. Table 6 shows a small mean normalized defect (≈ 0.015) for both constructions and a larger maximum (0.12–0.16), indicating localized departures from tree structure. The prompt-wise token-MMD tensor also admits a compact low-rank approximation: the leading component explains 63% of prompt-to-prompt variance, increasing to 69%, 72%, 77%, and 85% at ranks 2, 3, 5, and 10, respectively. The rank-one approximation is the PCA summary used in the main text. Encoder robustness. All constructions in the main text share one response encoder, so their mutual agreement tests aggregation and alignment but not the choice of ϕ. We therefore re-encoded all 32 × 104 responses with three additional encoders and recomputed Dmean end to end. The alternatives span architecture, scale, and provenance: bge-m3 and multilingual-e5-large are XLM-RoBERTa encoders (568M and 560M parameters, 1024 dimensions), and all-mpnet-base-v2 is a 110M MPNet sentence encoder (768 dimensions), roughly 73× smaller than the 8B, 4096-dimensional Qwen3-Embedding-8B. None of the three shares a vendor with any evaluated model, which also addresses the concern that a Qwen encoder might favor Qwen models. Table 7 reports the outcome. Rank agreement with the reference encoder is high (ρ = 0.92–0.93 over the 496 distinct pairs), the same-family nearest-neighbor count is essentially unchanged (19–22 of 32 against 21), and every encoder returns gpt-2 and qwen-2.5-72b-instruct as the two global outliers—so the Qwen-family outlier is recovered by encoders with no Qwen

Sentence mean 0.015 0.125 111.8

δ̄/diam (mean rel.) δmax /diam (max rel.) diameter

Token MMD 0.015 0.155 0.317

Table 6: Normalized four-point defects. For the sentence mean, these are hyperbolicity statistics on the quotient metric; for the averaged squared-MMD matrix, they are descriptive tree-likeness diagnostics. Encoder Qwen3-Embedding-8B (ref.) bge-m3 multilingual-e5-large all-mpnet-base-v2

Dim 4096 1024 1024 768

Len 1024 1024 512 512

ρ — 0.933 0.922 0.930

same-fam. NN 21/32 21/32 22/32 19/32

trend r −0.522 −0.521 −0.551 −0.522

Table 7: Encoder robustness. ρ is the Spearman correlation of Dmean with the reference encoder over the 496 distinct model pairs; “same-fam. NN” counts models whose nearest neighbor is from their own family; “trend r” is the correlation between mean cross-family distance and release date (negative = converging; one-sided p ≤ 0.0011 in every row). All four encoders return gpt-2 and qwen-2.5-72b-instruct as the two global outliers. lineage. The convergence trend is likewise stable: its slope changes with the absolute distance scale, which is encoder-specific, but its correlation with release date is nearly invariant (r between −0.52 and −0.55, one-sided p ≤ 0.0011 throughout). Figure 8 shows the corresponding maps. Because metric MDS is determined only up to rotation, reflection, and scale, each map is Procrustes-aligned to the reference before plotting; the aligned configurations are visually near-identical. Two honest qualifications. First, multilingual-e5-large and all-mpnet-base-v2 accept at most 512 tokens rather than 1024; sampling 2,400 responses gives a median length of 80 tokens with 4.5% above 512, so a small minority of long responses is truncated more aggressively, which can only depress agreement. bge-m3 runs at the reference length of 1024 and is thus the exactly matched comparison. Second, the three alternatives agree with one another (ρ = 0.99) more closely than any of them agrees with Qwen3-Embedding-8B, which is plausibly the one decoder-based, 4096-dimensional encoder in the panel; the reported findings nonetheless hold under all four. Supplementary figures. Figure 9 completes Fig. 6 with the Llama and Mistral panels; Fig. 10 maps the ecosystem under the alignment-invariant GW distance; Figs. 11 and 12 give the per-family detail for the GW and token-level cartography maps; Figs. 13 and 14 show the projection and PCA robustness maps; and Figs. 15 and 16 the token-level structure and convergence.

A.4

Detailed Proofs for the Training-Side Account

This section proves Theorem 1 and Corollaries 1–2. It also states precisely how the mean-embedding result relates to the single-response distance used in the experiments. Setup and conventions. Let R be the countable space of finite token sequences terminated by an end-of-sequence token, and let Pm be a joint distribution over (X, R). We write Pm,X for its prompt marginal and pm (· | x) for the corresponding conditional response distribution. The deployed autoregressive model defines qm (r | x) =

|r| Y

qm (rt | x, r<t ),

t=1

where P the final token in r is the end-of-sequence token. On this countable space, TV(P, Q) = supA |P (A) − Q(A)| = 1 r |P (r) − Q(r)|. The arguments extend to standard measurable response spaces with regular conditional distributions. If 2 qm (r | x) = 0 on a set assigned positive probability by pm (· | x), the conditional KL divergence is infinite; the informative case is therefore finite excess risk. Lemma 1 (Cross-entropy excess equals conditional KL). For any conditional distribution q with finite risk, Rm (q) − Rm (pm ) = Ex∼Pm,X KL(pm (· | x)∥q(· | x)) . Consequently, pm minimizes population negative log-likelihood over all conditional response distributions. Proof. Conditioning first on X = x and expanding the definition of KL gives Er∼pm (·|x) [− log q(r | x)] − Er∼pm (·|x) [− log pm (r | x)] = Er∼pm (·|x) log

pm (r | x) = KL(pm (· | x)∥q(· | x)) . q(r | x)

Qwen3-Embedding-8B reference

bge-m3 = 0.933

multilingual-e5-large = 0.922

all-mpnet-base-v2 = 0.930

GPT Claude Gemini Qwen Llama Mistral

Figure 8: Metric-MDS maps of Dmean under four response encoders, drawn with the encoding of Fig. 2: family markers shaded light-to-dark by release order, oldest→latest arrows, and concentric rings about the latest-models centroid. Each map is Procrustes-aligned to the reference (left), since MDS is determined only up to rotation, reflection, and scale; the rings are recomputed per panel from that encoder’s own coordinates, so their agreement reflects the data rather than a reused reference geometry. The configurations are near-identical—the gpt-2 outlier, the family blocks, and the inward evolution paths—under an 8B decoder-based encoder and a 110M MPNet encoder alike. Taking expectation over x ∼ Pm,X proves the identity. Since KL is nonnegative, the minimum is attained at q(· | x) = pm (· | x) for Pm,X -almost every x. For an autoregressive model, the same identity can equivalently be expanded by the KL chain rule into the sum of next-token conditional KL terms over all prefixes. Lemma 2 (Transfer from training risk to inference prompts). Suppose Eq. (7) holds for model m, and write wm = dξ/dPm,X . Then r κm εm Ex∼ξ TV(pm (· | x), qm (· | x)) ≤ . 2 Proof. For each prompt x, Pinsker’s inequality yields r

1 KL(pm (· | x)∥qm (· | x)). 2 Let Km (x) = KL(pm (· | x)∥qm (· | x)). Because the square-root function is concave, Jensen’s inequality gives q p Ex∼ξ Km (x)/2 ≤ Ex∼ξ Km (x)/2. TV(pm (· | x), qm (· | x)) ≤

The density-ratio assumption then transfers the KL expectation from the inference distribution to the training marginal: Z Ex∼ξ Km (x) = Km (x)wm (x) dPm,X (x) Z ≤ κm Km (x) dPm,X (x) = κm εm , where the last equality is Lemma 1. Combining the three displays proves the claim. Full proof of Theorem 1. Proof. Fix a prompt x. Since total variation is a metric on probability measures, insert the two population conditionals and apply its triangle inequality: x x x x x x TV(qm , qm ′ ) ≤ TV(qm , pm ) + TV(pm , pm′ ) x + TV(pxm′ , qm ′) , txt where pxj = pj (· | x) and qjx = qj (· | x). Integrating with respect to x ∼ ξ, the middle term becomes ηm,m ′ . Lemma 2 bounds p p the first and third terms by κm εm /2 and κm′ εm′ /2, respectively. Adding the three bounds gives Eq. (8).

The result has the transparent decomposition model difference ≤ population difference + learning error of m + learning error of m′ . Small excess risks alone are not sufficient: even population-optimal models can differ when pm (· | x) and pm′ (· | x) differ on the evaluation prompts.

gpt-2 gpt-3.5-turbo gpt-4o-mini gpt-4.1-mini gpt-5.2

claude-sonnet-4-0 claude-opus-4-0 claude-haiku-4-5 claude-sonnet-4-5-think claude-opus-4-5-think claude-sonnet-4-6 claude-opus-4-6 claude-opus-4-7-think

gemini-2.0-flash gemini-2.5-pro gemini-3-flash-preview gemini-3.1-pro-preview

qwen-2.5-72b-instruct qwen3-235b-a22b-instruct-2507 qwen3.5-397b-a17b qwen3.6-max-preview qwen3.7-max

llama-3-8b-instruct llama-3.1-70b-instruct llama-3.3-70b-instruct llama-4-scout llama-4-maverick

mixtral-8x22b-instruct mistral-nemo mistral-small-24b-instruct-2501 mistral-medium-3.1 mistral-large-2512

Figure 9: All six per-family panels for Fig. 2 in a common ring-bounded frame (identical axes) with oldest→latest arrows. Left to right, top to bottom: GPT, Claude, Gemini, Qwen, Llama, Mistral. Fig. 6 shows the first four in the main text. Full proof of Corollary 1. Proof. For each prompt x, the triangle inequality for W1 gives x x x x x x x x W1 (νm , νm ′ ) ≤ W1 (νm , πm ) + W1 (πm , πm′ ) + W1 (πm′ , νm′ ).

We first bound the two outer terms. For arbitrary response distributions P, Q whose pushforwards are supported in Sϕ , take a maximal coupling (U, V ) satisfying Pr(U ̸= V ) = TV(P, Q) (which exists on the countable response space). Because diam(Sϕ ) ≤ Bϕ , ∥ϕ(U ) − ϕ(V )∥2 ≤ Bϕ 1{U ̸= V }. The pushforward of this coupling is a valid coupling of ϕ# P and ϕ# Q. Therefore, by Eq. (9), W1 (ϕ# P, ϕ# Q) ≤ Bϕ TV(P, Q).

(14) ′

Apply Eq. (14) to the first and third terms, average over x ∼ ξ, and invoke Lemma 2 for models m and m . This yields r κm εm x x sem Ex∼ξ W1 (νm , νm ) ≤ η + B ′ ϕ m,m′ 2 r κm′ εm′ + Bϕ , 2 which is Eq. (10). Full proof of Corollary 2. x x ′ Proof. Fix a prompt x and let γ ∈ Π(νm , νm ′ ) be any coupling. Writing (Z, Z ) ∼ γ and using the marginal constraints,

∥µm (x) − µm′ (x)∥2 = ∥Eγ [Z − Z ′ ]∥2 ≤ Eγ ∥Z − Z ′ ∥2 , where the inequality follows from Jensen’s inequality for the Euclidean norm. Taking the infimum over all couplings γ gives x x ∥µm (x) − µm′ (x)∥2 ≤ W1 (νm , νm ′ ).

Averaging Eq. (15) over x ∼ ξ and applying Corollary 1 proves Eq. (11).

(15)

sentence_gw MDS GPT Claude Gemini Qwen Llama Mistral

Figure 10: Alignment-invariant cartography from Gromov–Wasserstein distance (encoding as Fig. 2). GW compares each model’s internal response geometry, so frontier models pair across families and the family signal weakens. Detail: Fig. 11. Connection to the observed single-response distance. Equation (1) is computed from one generated response per model and prompt rather than from the conditional means. To state the connection precisely, define the within-prompt semantic spread x ∥Z − µm (x)∥2 . ωm = Ex∼ξ EZ∼νm

x x and Zm′ ∼ νm Conditional on x, let Zm ∼ νm ′ be generated independently. The triangle inequality through the two conditional means gives

E∥Zm − Zm′ ∥2 ≤ E∥Zm − µm (x)∥2 + ∥µm (x) − µm′ (x)∥2 + E∥Zm′ − µm′ (x)∥2 . After averaging over prompts and using Eq. (15), Ex∼ξ E∥Zm − Zm′ ∥2 r r κm εm κm′ εm′ sem ≤ ηm,m′ + Bϕ + Bϕ + ωm + ωm′ . 2 2

(16)

Thus, when responses are sampled from the model distributions, the empirical mean per-prompt distance estimates a quantity controlled by population similarity, learning errors, and within-prompt generation variability. The ω terms are essential in general: an independent-sample transport cost need not equal the optimal-coupling cost W1 . Bounded downstream scores. For any measurable score s(x, r) ∈ [0, 1], the variational characterization of total variation gives, for each x, Eqm (·|x) s(x, R) − Eqm′ (·|x) s(x, R) ≤ TV(qm (· | x), qm′ (· | x)) . Averaging over x ∼ ξ and applying Theorem 1 shows that behaviorally close models have close expected values for every bounded evaluation score. This does not imply high capability: the two models may be similarly incorrect. Effect of training-set size. The theorem is stated in terms of population excess risk. A finite-sample learning analysis may, under an appropriate oracle inequality for the chosen model class and algorithm, yield a bound of the schematic form εm ≤ am + τm + gm (nm , δ),

gpt-2 gpt-3.5-turbo gpt-4o-mini gpt-4.1-mini gpt-5.2

claude-sonnet-4-0 claude-opus-4-0 claude-haiku-4-5 claude-sonnet-4-5-think claude-opus-4-5-think claude-sonnet-4-6 claude-opus-4-6 claude-opus-4-7-think

gemini-2.0-flash gemini-2.5-pro gemini-3-flash-preview gemini-3.1-pro-preview

qwen-2.5-72b-instruct qwen3-235b-a22b-instruct-2507 qwen3.5-397b-a17b qwen3.6-max-preview qwen3.7-max

llama-3-8b-instruct llama-3.1-70b-instruct llama-3.3-70b-instruct llama-4-scout llama-4-maverick

mixtral-8x22b-instruct mistral-nemo mistral-small-24b-instruct-2501 mistral-medium-3.1 mistral-large-2512

Figure 11: Per-family detail for Fig. 10: each lineage under Gromov–Wasserstein distance, drawn in a common frame bounded by the outermost ring (identical axes in every panel). Panels, left to right and top to bottom: GPT, Claude, Gemini, Qwen, Llama, Mistral. where am is approximation error, τm is optimization error, and gm (nm , δ) is a generalization term that decreases with sample size nm . Substituting such a bound into Theorem 1 shows how more data can reduce the finite-sample contribution. This is not a universal decomposition without additional capacity and algorithmic assumptions, and large datasets alone do not eliminate approximation, optimization, population-discrepancy, or coverage errors. Scope and limitations of the sufficient condition. The theorem directly models log-loss training. Pretraining data alone may not capture instruction tuning, preference optimization, safety training, or system-level policies; applying the theorem to a post-trained system therefore requires a separate argument that its deployed distribution has small excess log-loss relative to an appropriate effective target population. The coverage assumption is also substantive: if the evaluation prompts place mass outside the support of the effective training marginal, no finite κm exists and the theorem gives no guarantee. Finally, the results concern probabilistic response distributions. They apply directly when evaluation responses are sampled from qm (and remain valid after a common response post-processing Markov kernel, since total variation contracts under such a kernel). Greedy decoding is a deterministic functional of the probability vector rather than a sample from it and can switch outputs discontinuously when the leading probabilities are nearly tied. A direct guarantee for greedy generation therefore needs sem an additional margin or decoder-stability assumption. Under these qualifications, decreasing excess risks and decreasing ηm,m ′ imply semantic convergence; a positive limiting population discrepancy or irreducible learning error instead yields relative homogenization toward a positive floor.

gpt-2 gpt-3.5-turbo gpt-4o-mini gpt-4.1-mini gpt-5.2

claude-sonnet-4-0 claude-opus-4-0 claude-haiku-4-5 claude-sonnet-4-5-think claude-opus-4-5-think claude-sonnet-4-6 claude-opus-4-6 claude-opus-4-7-think

gemini-2.0-flash gemini-2.5-pro gemini-3-flash-preview gemini-3.1-pro-preview

qwen-2.5-72b-instruct qwen3-235b-a22b-instruct-2507 qwen3.5-397b-a17b qwen3.6-max-preview qwen3.7-max

llama-3-8b-instruct llama-3.1-70b-instruct llama-3.3-70b-instruct llama-4-scout llama-4-maverick

mixtral-8x22b-instruct mistral-nemo mistral-small-24b-instruct-2501 mistral-medium-3.1 mistral-large-2512

Figure 12: Per-family detail for Fig. 7: each lineage under the token-level per-prompt MMD distance, drawn in a common frame bounded by the outermost ring (identical axes in every panel). Panels, left to right and top to bottom: GPT, Claude, Gemini, Qwen, Llama, Mistral.

sentence_mean t-SNE

sentence_mean UMAP GPT Claude Gemini Qwen Llama Mistral

(a) t-SNE

GPT Claude Gemini Qwen Llama Mistral

(b) UMAP

Figure 13: The same mean-metric distances under t-SNE (a) and UMAP (b), with the encoding of Fig. 2. Family grouping is stable across projections.

sentence_pca_scores MDS GPT Claude Gemini Qwen Llama Mistral

Figure 14: The PCA-compressed distance of Eq. (2) (metric MDS), with the encoding of Fig. 2. The compression closely matches the mean metric (Spearman ρ = 0.92).

0.00 llama-4-scout

gpt-5.2

llama-3.3-70b-instruct

llama-3.1-70b-instruct

gemini-2.0-flash

distance

0.15

mistral-nemo

qwen3.7-max

gemini-3.1-pro-preview

qwen3.6-max-preview

qwen3.5-397b-a17b

gemini-2.5-pro

claude-sonnet-4-6

claude-opus-4-6

Distance matrix, ordered by family + release date

mistral-medium-3.1

Hierarchical clustering (average linkage)

mistral-large-2512

mixtral-8x22b-instruct

gpt-4.1-mini

gpt-3.5-turbo

gpt-4o-mini

mistral-small-24b-instruct-2501

claude-opus-4-5-think

gemini-3-flash-preview

qwen3-235b-a22b-instruct-2507

claude-opus-4-7-think

qwen-2.5-72b-instruct

0.20

llama-4-maverick

claude-sonnet-4-0

claude-opus-4-0

claude-sonnet-4-5-think

llama-3-8b-instruct

claude-haiku-4-5

gpt-2

merge distance

source: token_mmd_mean 0.30

0.25

0.20

0.10

0.05

0.00

source: token_mmd_mean

0.15

0.10

0.05

Figure 15: Token-level MMD structure: family+date-ordered heatmap (top) and average-linkage dendrogram (bottom). As at the sentence level, gpt-2 and qwen-2.5-72b split off first and recent models mix across families.

Convergence over time mean cross-family behavioral distance

0.22

source: token_mmd_mean (cross-family trend: converging) GPT Claude Gemini

0.20

Qwen Llama Mistral

cross-family (slope=-3.54e-05/day) within-family (slope=-5.38e-05/day)

0.18 0.16 0.14 0.12 0.10 0.08

201

202

release date

202

Figure 16: Token-level convergence: mean cross-family MMD distance per model versus release date. The downward trend reproduces the sentence-level frontier homogenization (cf. Fig. 5).

Mapping and Measuring the Behavioral Evolution of Large Language Models

Analyze & explore related research in Conceptio

Related documents