Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
arXiv:2605.06656v1 [cs.LG] 7 May 2026
Jai Moondra∗† Carnegie Mellon University [email protected]
Ayela Chughtai† MIT Sloan School of Management [email protected]
Bhargavi Lanka‡ MIT Sloan School of Management [email protected]
Swati Gupta∗ MIT Sloan School of Management [email protected]
Abstract Ranking large language models (LLMs) via pairwise human feedback underpins current leaderboards for open-ended tasks, like creative writing and problem solving. We analyze ∼ 89K comparisons across 116 languages across 52 LLMs, from the Arena platform, and show that the best fit global Bradley-Terry (BT) ranking is largely misleading. Nearly two-thirds of the decisive votes cancel each other out, and even the top 50 models according to the global BT ranking are statistically indistinguishable (pairwise win probabilities are at most 0.53 within the top 50, i.e., near random outcomes). We trace this failure to strong, structured heterogeneity of opinions across language, task, and time. Moreover, we find an important characteristic - language plays an important role in this heterogeneity. Grouping by language (and language families) increases the agreement of votes massively, resulting in two orders of magnitudes higher spread in the ELO scores (i.e., very consistent rankings). What appears as global noise is in fact a mixture of coherent but conflicting subpopulations. To address such a heterogeneity in supervised machine learning, broadly, we introduce the framework of (λ, ν)-portfolios, which are small collections of models that achieve a prediction error at most λ, “covering” at least a ν fraction of users. We formulate this problem as a variant of the set cover problem, and provide guarantees by invoking the VC dimension of the underlying set system. On the Arena dataset, our algorithms recover just 5 distinct BT rankings that cover over 96% of votes at a modest λ, compared to the 21% coverage achieved by global ranking at the same threshold. We also provide a portfolio of 6 LLMs that cover twice as many votes as compared to choosing the top-6 LLMs from a global ranking. We further illustrate small portfolios for a classification problem on the COMPAS dataset using an ensemble of fairness-regularized classification models, and show that these portfolios can be used to detect blind spots or errors in the data, which might be of independent interest to policymakers.
1
Introduction
Large language models (LLMs) are increasingly deployed at scale to serve a global and diverse user base. Modern approaches evaluate them on specific tasks like creative writing, complexity, ∗ Corresponding authors: Jai Moondra ([email protected]) and Swati Gupta ([email protected]) † Equal contribution ‡ A part of this work was done when the author was a student at MIT Sloan School of Management.
Preprint.
Figure 1: Model differentiation across ranking types. Each point is a BT ranking fitted on a group of votes (< 50 votes excluded). The y-axis reports Elo-scaled score spread (max - min; see Eqn. 1 and Table 1), where higher spread indicates more aligned preferences and more predictive rankings. Language, language-family, and their task interactions induce substantially larger spreads than random, hourly, task-only, or global rankings. creativity, domain knowledge, problem solving, specificity, technical accuracy, code, math, and realworld use cases. As response quality from these models varies with prompt formatting and dataset contamination [Sclar et al., 2023, Voronov et al., 2024, Alzahrani et al., 2024], evaluation of LLMs through human feedback is often preferred. Moreover, recent studies highlight that human feedback can capture diversity more effectively than standard benchmarks [Kirk et al., 2024]. Open evaluation platforms like Arena [Chiang et al., 2024, Arena, 2026] (formerly, LMArena) rank large language models based on human feedback (in terms of preference amongst two displayed responses), and have become a transparent gold standard for assessing LLMs [The Wall Street Journal, 2024]. However – as we show – due to heterogeneity of the votes in the dataset, the global ranking constructed by such platforms is not representative of the user votes and opinions across the world. On the recent publicly available data from HuggingFace4 on 52 LLMs5 across 116 languages, we show that 72.28% of decisive non-tied votes (across ∼ 89k in the entire data) are actually ‘canceled’ 6 by each other. Moreover, since users only provide comparative feedback among a pair of LLM responses to their prompt, platforms like Arena convert votes to a global preference using the standard Bradley-Terry (BT) ranking model ([Bradley and Terry, 1952], also widely used for routing queries to the most appropriate LLM or inference-time scaling Damani et al. [2025]). This model estimates an Elo score θℓ ∈ R for each LLM ℓ (see Section 2), with higher scores indicating a greater probability that the user base prefers ℓ to other LLMs. Despite their usefulness in extrapolating pairwise feedback to a ranking, these models can end up erasing information about conflicting preferences or be dominated by the larger group’s preference. This leads to a peculiar hedging behavior, wherein the global model may have higher error across groups with different trends. In particular, for Arena’s case, we find that the global ranking is able to predict the winner of a vote with ≥ 70% probability for only 10.3% of all votes.7 We further note that even the highest-ranked models in the global BT ranking exhibit substantial variation across language family BT rankings. Among the global top five models, Gemini-2.5P, Gemini-2.5P3, Grok-4, o3, and Gemini-2.5-P5 differ from their global ranks by 5 to 14 positions on average across rankings restricted to votes from specific language-families; in fact Grok-4 even falls to rank 52 in the Afro-Asiatic family ranking. On the Arena website, e.g., as of May 4th, there is only a 50 point Elo score gap (see Section 2) between top 50 models, implying a probability of winning of 4 https://huggingface.co/datasets/lmarena-ai/arena-human-preference-140k 5 See a list of names in Table 2 in Appendix A.2. 6 That is, % of individual decisive votes that are offset by an opposing decisive vote for the same model pair, where one vote favors LLM 1 over LLM 2 and another favors LLM 2 over LLM 1. 7 Note that these are not unseen/test votes – these were used to train the global BT ranking (see Section 4). While some of this ‘hedging’ is by design – Arena’s algorithm [Chiang et al., 2024] is more likely to match similarly ranked models more often to be able to separate them – it does not explain varied preferences across language groups.
2
Figure 2: In-group predictive performance of 388 subpopulation BT rankings vs. the global ranking. Left: average probability assigned to the observed winner. Right: average log loss. Colors indicate subpopulation type. Language- and family-based rankings consistently assign sharper winner probabilities and lower log loss than the global ranking, while task-specific rankings remain nonrepresentative, similar to the global baseline. 0.53, of the top 1 model winning against the rank 50 model. Recent parallel work that also shows that these rankings are unreliable due to variance under vote dropout of <5% Huang et al. [2025a]. In this work, we dig deeper into the characteristics of heterogeneity in user preferences across the votes collected by Arena. In particular, we trace this failure to strong, structured heterogeneity of opinions across language, task, and time. However, we find an important characteristic - language plays an important role in this heterogeneity. Grouping by language (and language families) increases the agreement of votes massively, resulting in much lower log-loss per group and a much higher probability of sampling the votes within each group (see Figure 2). What appears as global noise is in fact a mixture of coherent but conflicting subpopulations. This phenomenon of ‘best-fit’ models hedging between heterogeneous groups and losing all statistical or predictive significance is observed more generally in all supervised learning problems. For example, Simpson’s paradox Simpson [1951] arises due to the reversal of certain trends in different groups of data when these groups are combined. We provide a way forward by proposing an alternate: to find a set of supervised ML models that together achieve a smaller loss on a diverse population. For the Arena case, we find a small set of rankings that critically use ‘more homogeneous clusters’ within the votes based on the prompt task, language, and language-family, thus solving the key challenge faced in this area. Our general framework addresses the following question: Given a large ensemble of ML models (or rankings) in a supervised learning setting, can we select a small subset or portfolio of models such that each user (or vote) is reasonably “covered” by at least one model in the portfolio? Outline and Summary of Key Contributions: In this work, we formalize this trade-off for supervised learning, and present an algorithmic framework for choosing small collections or portfolios [Gupta et al., 2023, Kim et al., 2025] of ML models that ensure that most (or all) users are satisfactorily ‘covered’ by some ML model in the portfolio (Section 3). Specifically, we show that the corresponding problem is an instance of the well-known “partial set cover problem” [Karp, 1975, Kearns, 1990, Slavík, 1997]. Next, in Section 3.1, we present algorithms with provable guarantees on the (approximately) smallest number of models that cover enough users. By exploiting the algorithmic connection between set cover and the VC dimension of the underlying set system Brönnimann and Goodrich [1994], we obtain novel guarantees on portfolio sizes for regression problems, and in particular for linear regression (Thm 2), and discuss the extension to ranking models (Section 3.2). Finally, we discuss our computational case studies on Arena and COMPAS in Section 4. For the Arena dataset (Section 4.1), we provide evidence of heterogeneity in user preferences, and homogeneity within language-based subgroups. We demonstrate that our framework can generate small portfolios (with only between 3 and 10 models each) for Arena that outperform any global model in capturing diverse users and reducing 3
performance gaps across families. Similar results hold for LLMs: we show most voters are reasonably well-served by a small collection of 4 to 8 LLMs (see Figure 4a). Simply collecting the top few LLMs in the global ranking does not achieve the same effect. We further show a proof of concept for other heterogeneous supervise ML settings, by constructing portfolios to predict recidivism of criminal defendants within 2 years using the COMPAS data Angwin et al. [2022] (Section 4.2), and discuss the characteristics of the defendants that are harder to cover with small portfolios (e.g., low priors, young), highlighting the need to develop better models. 1.1
Related work
Ranking Models and Preference Aggregation. A large literature studies aggregating pairwise preferences into global rankings (e.g., Bradley-Terry) Bradley and Terry [1952], but recent work highlights their limitations under heterogeneity. Group-level inconsistencies, such as those observed in hiring and subgroup fairness Cachel and Rundensteiner [2025], Kearns et al. [2018], show that aggregate rankings can mask conflicting preferences. In the context of LLM leaderboards, additional concerns arise: rankings are vulnerable to manipulation via injected votes Min et al. [2025], strategic voting Huang et al. [2025b], and gaming of LLM-based evaluators Zheng et al., Raina et al. [2024], as well as issues like data leakage Singh et al. [2025]. Complementing this line of work, we show that even without adversaries, global rankings can be inherently misleading due to structured heterogeneity, and are better viewed as a portfolio of homogeneous models so that opposing preferences are preserved. Individual Fairness. There is a long line of work advocating individual fairness, i.e., that similar individuals should be treating similarly Dwork et al. [2012, 2020], Dwork and Ilvento [2018], Anderson and Visweswaran [2025], Gupta and Kamble [2021]. There has been a lot of work on defining the metric under which similarity is defined (e.g., Ilvento [2020], Waller et al. [2025]), auditing in absence of such a metric Bechavod et al. [2020], and in even averaged versions of individual fairness that ask for similar classification accuracy across multiple tasks for each individual Sharifi-Malvajerdi et al. [2019]. In this work, we take the view that all individuals (i.e., the metric is zero everywhere) must enjoy an upper bound on the amount of error they receive from the model predictions, which is essential in assessing risk compliance of a tool deployed in practice. In the case of ranking LLMs, this amounts to a minimum representation provided by chosen large language models in the ranking setting for each vote. Our work therefore studies the choice of models from an ensemble that operationalizes individual fairness in terms of bounded error. This provides a way to address the tension between the two desiderata of fairness and accuracy, by asking for a bounded measurement error on each individual Kinney [2025]. Ensemble Methods. Literature on ensemble methods focuses on boosting and bagging methods for classification and regression models (e.g., Dietterich [2000] and ref. therein). Our work however focuses on selected a smaller subset of models from any given ensemble so that the maximum error from the best of the chosen models is bounded for each individual. Therefore, our work complements the sparsification literature in ML, where the typical motivation is to improve explainability and interpretability, e.g., Liu and Mazumder [2023], where the authors use minimization of aggregate least squares loss, rather than bounding the loss on each datapoint.
2
Preliminaries
Setup.Throughout the paper, we assume a supervised learning set up, and denote by X = (xi ) the set of n := |X | data points, and by Y = (yi ) the corresponding set of real-valued labels. For classification problems, the label yi ∈ {0, 1} for each i. A model h : X → R takes each data point xi to a prediction h(xi ) for its label yi . For classification models, h(xi ) ∈ [0, 1] can be interpreted as the model’s estimated probability of the label yi being 1. Arena. The Arena platform collects human preferences through pairwise comparisons between LLMs. Each time a user or a voter enters a prompt, the user is shown responses from two LLMs and can select a preferred output. Let L = {1, . . . , L} denote the set of LLMs. Each vote involves two distinct LLMs a, b ∈ L and produces one of four outcomes: a wins, b wins, tie, and ‘both bad’. We exclude votes labeled as ‘both bad’. We treat a tie as half a win for both LLMs. Hereafter, we assume n total votes, and refer to the ith vote as (ai ≻ bi ) where ai wins against bi , for i ∈ [n]. 4
Ranking model. The Bradley-Terry (BT) ranking model [Bradley and Terry, 1952] associates with each LLM a ∈ L a score θℓ ∈ R, with the probability that a wins against b in a vote computed as a) Pr(a ≻ b) = exp(θexp(θ = σ(θb − θa ), a )+exp(θb )
where σ(x) = (1 + 10−x )−1 denotes the standard logistic function, therefore larger gap in the scores corresponds to a sharper winning probability. Some model pairs appear more frequently than others in the data. To avoid overweighting commonly sampled pairs, we use inverse-probability weighted (IPW) win counts, as prescribed in Chiang et al. [2024]. Let P̂ ({a, fraction of votes P b}) denote the 1 comparing a and b in the dataset. We define the weight wab := votes where a P̂ ({a,b}) . beats b P The resulting IPW log-likelihood is ℓ(θ) := a,b∈L wab log σ(θa − θb ) + wba log σ(θb − θa ) . Scores θa , a ∈ L are obtained by optimizing the log-likelihood. Since winner predictions depend only on score differences, we can center the estimated scores to have zero mean. For interpretability, we report Elo-scaled scores Ratingm = 400 · θm + 1000
(1)
that correspond to a base-10 logistic scale commonly used in Elo rating systems, where a 400-point difference implies approximately 10:1 odds of preference [Elo, 1967]. Each vote in the Arena dataset contains additional data about the vote besides the names of the winning and losing LLMs. In this work, we will also use the (a) timestamp, (b) prompt/response language, and (c) task category of the prompt (as defined by Arena). In particular, the identity of the voter is not provided, and a single person may have cast multiple votes. For simplicity, we treat each vote as a ‘user’.
3
Framework and algorithms
We next propose small ‘portfolios’ of models that guarantee a reasonable ‘coverage’ across all data points or users. We first describe it generally for any supervised learning setup, and then specifically for Arena in Section 4, where users correspond to votes. Consider n data points X = {x1 , . . . , xn } with associated labels y1 , . . . , yn ∈ R. Given a margin λ ≥ 0, we say that a model h : X → R covers xi ∈ X if |h(xi ) − yi | ≤ λ. We next define (λ, ν)-portfolio, which is a collection of models such that at least ν fraction of data points have at least one model in this collection that λ-covers it: Definition 1 ((λ, ν)-portfolio). Consider a margin λ ≥ 0, a coverage fraction ν ∈ (0, 1], and set X of n = |X | data points. We say that a finite subset P = {h1 , . . . , hk } ⊆ H of models is a (λ, ν)-portfolio if there is a subset Sν ⊆ X of |Sν | ≥ νn data points such that for all xi ∈ Sν , there exists j ∈ [k] :
|h(xi ) − yi | ≤ λ.
(2)
Lower margin λ indicates better prediction quality. Higher ν indicates higher coverage, or fewer outliers. Ideally, we seek models with both good predictions (λ ∼ 0), and high coverage (large ν ∼ 1). This isn’t always possible when portfolio size k = 1, e.g., if ν = 1, the above definition requires a single model with maximum error ≤ λ across all data points. At the other extreme, portfolios of size k ≤ n exist8 trivially: for each data point i ∈ [n], choose any hypothesis hi ∈ H that satisfies hi (xi ) = yi . Therefore, there is a three-way trade-off between margin/quality λ, coverage fraction ν, and portfolio size (k). We are interested in finding portfolios where size k > 1 but is still very small (between 2 and 10). In particular, does allowing a few extra models dramatically increase quality, or cover many more people? That is, are there regimes where k and λ are small, and ν is close to 1? This is crucial in many applications: large ν ensures inclusivity, small k ensures cost-feasibility (only a few models can be trained and maintained), and small λ ensures meaningful predictions. 8We assume ‘reasonable’ classes H of models: each data point x and label y, there is always some model h ∈ H with
h(x) = y.
5
3.1
Algorithms to find minimum-size portfolio
Next, we discuss algorithms for the problem of finding the smallest (λ, ν)-portfolio, i.e., minimizing k for given λ and ν. As we have indicated, this is a covering problem where we seek the smallest subset P ⊆ H so that at least νn data points are λ-covered by some model in P. This is an instance of the Partial Set Cover problem [Kearns, 1990] and we will use corresponding algorithms and results for our setting, but omit the (straightforward) formal reduction. Greedy algorithm (Folklore). Given λ and ν, consider the following natural greedy algorithm: • Initialize portfolio P ← ∅. • While fewer than νn data points have been λ-covered by some model in the portfolio, find model hnew ∈ H that covers the most number of uncovered data points, and add hnew to P. We can use the results of Slavík [1997] in our setting, to derive directly that the greedy algorithm returns a (λ, ν)-portfolio of models with size at most O(log νn) · k ∗ , where k ∗ is the minimum size of a (λ, ν)-portfolio: Theorem 1 (Slavík [1997]). The greedy algorithm is O(log νn)-approximation for Partial Set Cover. Improved guarantees using VC dimension. While the greedy algorithm returns a (λ, ν)-portfolio with size within factor O(log νn) of the optimal, it does not exploit the geometry of the model class H. Brönnimann and Goodrich [1994] established a fundamental connection between Set Cover and learning theory, and gave an efficient algorithm for Set Cover that returns a portfolio of size at most O(d log dk ∗ ) · k ∗ , where d is the VC dimension of the class H of classifier models, or equivalently, d is the VC dimension of the underlying set system. To extend this to regression models, let us first reduce the regression problem to a classification problem: Definition 2 (Margin classification). Given a regressor h : X → R model for data points X = (xi ) with corresponding labels Y = (yi ), and a margin λ ≥ 0, define the corresponding classifier ĥ : X → R as the indicator for whether h correctly predicts yi within margin λ. That is, 1 if |h(xi ) − yi | ≤ λ, ĥ(xi ) = 0 otherwise. For a class H of regression models for dataset X , Y, define the class Ĥ := {ĥ : h ∈ H} of corresponding classifier models. The problem of determining the minimum (λ, ν = 1) portfolio is then the same as the set cover problem for set system with elements X = (xi ) and sets Sh := {xi ∈ X : ĥ(xi ) = 1} for all h ∈ H. Since the VC dimension of this set system is the same as the VC dimension of the corresponding hypothesis class Ĥ, we get that the following result follows from Brönnimann and Goodrich [1994]: Theorem 2. There is an efficient algorithm that given a class H of regression models, margin λ > 0, and data points (xi , yi ), i ∈ [n], returns a (λ, 1)-portfolio of size at most O(dk ∗ · ln(dk ∗ )), where k ∗ is the size of the smallest (λ, 1)-portfolio, and d is the VC dimension of the corresponding class Ĥ of margin-classifiers. As an immediate corollary, we get the following: Corollary 1. For d-dimensional data points X ⊆ Rd , and for the class H of all d-dimensional linear regressors (hyperplanes), a (λ, 1)-portfolio of size at most O(dk ∗ · ln(dk ∗ )) can be found efficiently. Integer program. When the model class H is finite,9 we can formulate an integer program (IP) for this problem. The optimal solution to this integer program gives the smallest (λ, ν)-portfolio, if such a portfolio exists. For each hypothesis hj ∈ H, binary variable tj ∈ {0, 1} indicates whether hj is included P in the portfolio. We seek to minimize the total number of models included in the portfolio, i.e., j tj . For each data point xi ∈ X , binary variable ui ∈ {0, 1} indicates whether xi is covered P by some model in the portfolio. Then, we must have that j:|hj (xi )−yi |≤λ tj ≥ ui . This gives the 9 Most model classes used in practice can be assumed to be finite using a fine-enough discretization of the parameter space.
6
following IP: min
X
tj
s.t.
(MIP)
j
X
ui ≥ νn,
(3)
i
ui ≤
X j:|hj (xi )−yi |≤λ
tj
ui , tj ∈ {0, 1}
∀ i,
(4)
∀ i, ∀ j.
Constraint (3) enforces that ≥ νn points are covered by the portfolio. While solving this IP requires exponential time in the worst-case, it is often practical when the number of data points n and the number of models |H| is reasonably small. For larger problems, we can relax the integer constrains and allow variables ui , tj to lie in [0, 1], obtaining a linear program and using a reasonable rounding scheme; see Appendix C for details. Despite having much higher runtime than the greedy algorithm, the IP formulation often produces smaller portfolios in our experiments on the Arena dataset (Section 4). 3.2
Arena as a prediction problem
Next, we describe this setup for Arena specifically, where data points correspond to votes. For different votes in the dataset, we wish to (1) find ‘good’ BT ranking(s) that are good predictors of the winner of the votes, and (2) find ‘good’ LLM(s) that are preferred by different voters. We view BT rankings as (probabilistic) classifiers for the votes, with the ‘error’ erri (h) of a BT ranking h for a vote xi = (ai ≻ bi ) defined as the probability that BT ranking assigns to the loser (bi ) winning the vote. Then, erri (h) ∈ [0, 1], and smaller values correspond to a better prediction. As before, we say that the BT ranking λ-covers this vote if erri (h) ≤ λ. A (λ, ν)-portfolio is then a (finite) subset of the BT rankings such that for at least a fraction ν of all votes, there is some BT ranking in the portfolio that correctly predicts the winner of the vote within error margin λ ∈ [0, 1]. Informally, this indicates that the preferences of the voter-base is always captured by one of these BT rankings within a given margin, apart from (1 − ν) × (#total votes) outliers. E.g., a (0.1, 0.99)portfolio of 5 rankings indicates that the preferences of 99% of the users is captured by 1 of these 5 rankings, up to a 90% confidence. Measuring the probability that an LLM is ‘preferred’ by a voter that cast a certain vote is trickier, since each vote consists of pairwise information among only 2 LLMs. We extrapolate this voter’s preferences to other LLMs using BT rankings, as described next. First, we compute a very large ensemble of BT rankings, each corresponding to a different subgroup of the votes based on language and task subfamilies. Then, we extrapolate the voter’s preferences to other based on BT rankings based on the assumption that if a BT ranking predicts a vote ‘accurately’, then the voter prefers higher-ranked LLMs in this BT ranking to the lower-ranked ones. We formalize this in Appendix A.6, and discuss this further in Section 4.
4
Computational case studies: Arena and COMPAS
We next present computational results on the Arena and COMPAS datasets. All computations were performed on an Apple MacBook computer with Apple M5 (base) processor, and 32 GB RAM. 4.1
Arena dataset with pairwise LLM comparisons
We describe our experimental setup for Arena. First, we fit a single ‘global’ BT model on all 89,193 decisive votes (winners assigned weight 1), and 19,375 ties (both models assigned weight 0.5). Next, we stratify the Arena human-preference data by five metadata partitions: Language, Language Family, Task, Language Family × Task and Language × Task. The 116 languages are mapped to coarse families (Germanic, Romance, Slavic, Indo-Iranian, Sino-Tibetan, Turkic, Uralic, Afro-Asiatic, Niger-Congo, Austronesian, Dravidian, etc; see Section A.3). Each prompt is also associated with one or more of 10 task categories (creative writing, complexity, creativity, domain knowledge, problem 7
Figure 3: Language-family BT rankings capture distinct preference structure beyond the global ranking. Coverage is evaluated over all decisive Arena votes as we vary the λ-risk threshold. The shaded regions show approximate 95% confidence bands around the family mean, computed across languages within each family (showing correlations within language family). Differences between the Germanic and Slavic curves indicate that the global ranking covers some language-family preferences better than others, with enough structure within language groups.
solving, real world, specificity, technical accuracy, code, and math). Each vote can belong to multiple subpopulations. To avoid sampling errors, we exclude any subpopulation with fewer than 50 votes. For each eligible subpopulation, we fit a standard BT model and each resulting ranking is evaluated on its own user slice. For instance, for a language BT ranking on German, coverage is computed only on the subsets of votes that use queries written in German. We recompute λ-coverage for each new BT ranking. Results. The global BT ranking leaves a large fraction of individual votes poorly explained, and predicts the winner of only ∼ 11% of all votes correctly with a > 70% confidence. For subpopulation-specific BT rankings, many subpopulations achieve much higher coverage than the global baseline when evaluated on their own data. Further, language family-level BT rankings are often also useful for languages with fewer than 50 votes for which we do not compute a dedicated BT ranking. Several language–task strata (e.g., (English, code) vs. (German, creative)) show different top models than the global ranking. This indicates heterogeneity across the whole population but relative homogeneity within subpopulations (see Figure 5 for rankings by some languages). - Portfolio of BT rankings. From our initial ensemble of 389 BT rankings, we seek a small portfolio that maximizes vote-level λ-coverage for margin λ ∈ {0.05, 0.10, . . . , 0.45, 0.50} ∪ {0.6, 0.7, 0.8, 0.9}. As before, a BT ranking λ-covers a vote (a ≻ b) if it predicts that a wins a vote against b with probability ≥ 1 − λ (equivalently, from Eqn. 1, the Elo scores for this ranking satisfy scorea − scoreb ≥ ln 1−λ λ ). Given some coverage fraction ν ∈ (0, 1], this sets up the portfolio optimization problem. As Figure 4 shows, the portfolio of 5 rankings obtained by the greedy algorithm massively outperforms the global ranking in terms of coverage, for all margins λ ∈ (0, 1), showing that the vast majority of all users can be put into one of 5 ‘preference clusters’. For ν = 0.95, Tables 4 and 5 respectively give the portfolio obtained by the greedy algorithm and MIP respectively. We remark that MIP’s portfolio outperforms or matches the greedy algorithm; e.g., for λ = 0.50, MIP obtains a portfolio of size 4 while greedy algorithm obtains a portfolio of size 5. - Portfolios to LLMs. Finally, we compute a portfolio of LLMs that together cover a large number of users using (1) the greedy algorithm’s ranking portfolio, (2) MIP’s ranking portfolio, and (3) using top-k LLMs in the global ranking, for various values of k. The ‘error’ of an LLM for a vote is computed based on a Bayesian approach, and depends on which of the original 389 BT ranking cover 8
(a) Cumulative vote coverage for k = 10 LLMs under the MIP portfolio, greedy portfolio, and global ordering at λ = 0.20.
(b) Vote coverage across λ thresholds for the greedy-selected BT portfolio vs. the global BT ranking. Dashed line marks λ = 0.35.
Figure 4: Coverage by global ranking and by our portfolios.
the vote well and where the LLM ranks in those BT rankings (see Appendix A.6 for details). Table 6 describes the LLM portfolios obtained by the greedy algorithm and MIP’s ranking portfolios. 4.2
Extensions to COMPAS Dataset
We briefly illustrate the same framework on the COMPAS dataset as an application to a classification setting, where the task is to predict recidivism within two years. We describe our experimental setup for COMPAS. First, we build an ensemble of 61 classification models trained with different objective functions. One model minimizes standard binary cross9
entropy (BCE) alone, while the remaining models incorporate a fairness regularization term as well: BCE + µ × EOgap (equalized odds gap). In the latter objective, µ controls the weight of the fairness penalty and changes from 10 to 300, in increments of ten. The equalized odds gap (EOgap ) is defined as the sum of disparities in true positive rates (TPRs) and false positive rates (FPRs) across groups: EOgap := (maxg T P Rg − ming T P Rg ) + (maxg F P Rg − ming F P Rg ), computed separately over protected attribute groups. This results in two sets of fairness-regularized models: 30 models use sex-based groups and 30 models use race-based groups. Each model generates a predicted probability of 2-year recidivism over every individual i in the dataset. Using these predictions, we compute λ-coverage at the individual level: a model j covers an individual i if the absolute difference between the predicted probability and the true binary outcome is at most λ (i.e., |hj (i) − yi | ≤ λ); otherwise, the individual is considered uncovered by that model. Results. We show that a greedily-constructed portfolio of 4 models achieves 90% coverage at λ = 0.45, with the selected models shown in Figure 10. This suggests that even a relatively small and manageable set of models is sufficient to capture heterogeneity in the population. Moreover, models trained with different levels of fairness regularization appear to specialize in different subsets of individuals. At λ = 0.4, the greedily-constructed portfolio achieves 83.56% coverage. Notably, the uncovered group comprises predominantly of younger individuals (under 45) with few prior offenses, and every individual is a repeat offender. These cases are missed by all models in the ensemble, including the global best model which only optimizes for BCE. This reflects a region of the feature space which is not well captured by the ensemble, indicating that such portfolios can be used to surface systematic blind spots in prediction coverage across both individuals and sub-populations, defined not only by protected attributes but also by behavioral characteristics.
5
Conclusion
We show that global Bradley–Terry rankings on pairwise preference data from Arena are uninformative at the level of individual votes due to strong, structured heterogeneity across subpopulations. Some artefacts of this heterogeneity show up as a small spread of Elo scores, and model winning probabilities close to 0.53 (i.e, nearly random). One of our key insights is that language is the driving factor of LLM performance, conditional on language, the votes (hence opinions) are much more homogeneous. This insight may be of independent interest to language scholars, from the perspective of differences of LLMs performance based on language families (see Figure 5). Further, these trends may connect with literature on (1) observed differences in reasoning across languages (e.g., Levinson [2003], Boroditsky [2001]), and (2) on the heterogeneity of opinions within each language family as their size (i.e., voters) grows (see Figure 8, and e.g., Kharkhurin [2012]). We propose (λ, ν)-portfolios as an alternative to heterogeneous supervised ML settings, where a model ensemble can be sparsified so that each individual is well covered or their prediction error is within allowable λ error. Small portfolios of BT rankings (and LLMs) achieve substantially higher vote coverage at comparable error thresholds. Similar gains hold in the COMPAS setting, suggesting that heterogeneous prediction problems are better served by a small set of specialized models. Limitations. Our analysis has some limitations, and leaves room for future work. In the Arena data, we do not observe voter identity, preventing separation of between-user heterogeneity from within-user noise. Further, although in applications like COMPAS it is easy to define subpopulations and fit models to them, in the Arena data, we assumed (and justified) that votes within a language-task subgroup can be meaningfully considered. However, despite our evidence of homogeneity within language and task-based groups, we invite future analysis that searches for better vote clusters in this dataset, perhaps with more context from the users. Moreover, certain language families do not have votes from cross-pair comparisons across all LLMs, and this can prevent a meaningful total order10 . Broader impact. Ranking systems shape decisions in science, technology, and society by distilling complex comparisons into simple leaderboards. While global rankings are appealing for their clarity and accessibility, our results show that they can obscure meaningful variation in preferences across populations, leading to potentially misleading conclusions. 10 For e.g., consider the Italian × creative writing subpopulation. We observe votes for (Llama-4ME > GPT-4.1M) and (o3 > Nova-Exp), but none of the cross-pairs were compared. The BT model nonetheless assigns scores to all four and induces a total ordering Llama-4ME > o3 > Nova-Exp > GPT-4.1M, despite no direct comparisons supporting the relative positions of models across the vote pairs.
10
We propose a simple alternative: small portfolios of models or rankings that better reflect diverse user needs while remaining interpretable. Beyond evaluation, our framework highlights populations that are poorly served and models that generalize broadly, offering a tool for auditing and more inclusive system design. These insights may be useful for policymakers and practitioners seeking to deploy AI systems in heterogeneous, real-world settings.
6
Acknowledgements
The authors would like to acknowledge the efforts of Akoua Orsot on preliminary versions of the code on the COMPAS case study.
References Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13787–13805, 2024. Joshua W Anderson and Shyam Visweswaran. Algorithmic individual fairness and healthcare: a scoping review. JAMIA Open, 8(1):ooae149, 02 2025. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics, pages 254–264. Auerbach Publications, 2022. Arena. Arena: An open platform for evaluating AI models, 2026. URL https://arena.ai. Accessed: 2026-04-22. Yahav Bechavod, Christopher Jung, and Steven Z Wu. Metric-free individual fairness in online learning. Advances in neural information processing systems, 33:11214–11225, 2020. Lera Boroditsky. Does language shape thought?: Mandarin and english speakers’ conceptions of time. Cognitive psychology, 43(1):1–22, 2001. Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Hervé Brönnimann and Michael T Goodrich. Almost optimal set covers in finite vc-dimension: (preliminary version). In Proceedings of the tenth annual symposium on Computational geometry, pages 293–302, 1994. Kathleen Cachel and Elke Rundensteiner. Group fair rated preference aggregation: Ties are (mostly) all you need. In ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 660–678, 2025. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 8359–8388. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/ v235/chiang24b.html. Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of lm computation. In The Thirteenth International Conference on Learning Representations, 2025. Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1–15. Springer, 2000. Cynthia Dwork and Christina Ilvento. Individual fairness under composition. In Proceedings of the Conference on Fairness, Accountability, and Transparency in Machine Learning, 2018. 11
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Innovations in Theoretical Computer Science, pages 214–226, 2012. Cynthia Dwork, Christina Ilvento, and Meena Jagadeesan. Individual fairness in pipelines. In 1st Symposium on Foundations of Responsible Computing (FORC 2020), pages 7–1. Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 2020. Arpad E Elo. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8):242–247, 1967. Swati Gupta and Vijay Kamble. Individual fairness in hindsight. Journal of Machine Learning Research, 22(144):1–35, 2021. Swati Gupta, Jai Moondra, and Mohit Singh. Which Lp norm is the fairest? Approximations for fair facility location across all "p". In Economics and Computation (EC) 2023, page 817, July 2023. ISBN 9798400701047. URL https://doi.org/10.1145/3580507.3597664. Jenny Y Huang, Yunyi Shen, Dennis Wei, and Tamara Broderick. Dropping just a handful of preferences can change top large language model rankings. arXiv preprint arXiv:2508.11847, 2025a. Yangsibo Huang, Milad Nasr, Anastasios Nikolas Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Christopher A Choquette-Choo, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Liu, et al. Exploring and mitigating adversarial manipulation of voting-based leaderboards. In International Conference on Machine Learning, pages 25654–25671. PMLR, 2025b. Christina Ilvento. Metric learning for individual fairness. In Symposium on Foundations of Responsible Computing (FORC), 2020. Richard M Karp. On the computational complexity of combinatorial problems. Networks, 5(1): 45–68, 1975. Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning, pages 2564–2572, 2018. Michael J Kearns. The computational complexity of machine learning. 1990. Anatoliy V Kharkhurin. Multilingualism and creativity. Multilingual Matters, 2012. Cheol Woo Kim, Jai Moondra, Shresth Verma, Madeleine Pollack, Lingkai Kong, Milind Tambe, and Swati Gupta. Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning, February 2025. URL http://arxiv.org/abs/2502.09724. arXiv:2502.09724 [cs]. David Kinney. Aggregating concepts of fairness and accuracy in prediction algorithms. In ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 464–472, 2025. Hannah R Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. Advances in Neural Information Processing Systems, 37:105236–105344, 2024. Stephen C Levinson. Space in language and cognition: Explorations in cognitive diversity, volume 5. Cambridge University Press, 2003. Brian Liu and Rahul Mazumder. FIRE: An optimization approach for fast interpretable rule extraction. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1396–1405, 2023. Rui Min, Tianyu Pang, Chao Du, Qian Liu, Minhao Cheng, and Min Lin. Improving your model ranking on chatbot arena by vote rigging. In International Conference on Machine Learning, pages 44252–44271. PMLR, 2025. 12
Vyas Raina, Adian Liusie, and Mark Gales. Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7499–7517, 2024. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324, 2023. Saeed Sharifi-Malvajerdi, Michael Kearns, and Aaron Roth. Average individual fairness: Algorithms, generalization and experiments. Advances in neural information processing systems, 32, 2019. Edward H Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2):238–241, 1951. Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D’souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A Smith, et al. The leaderboard illusion. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. Petr Slavík. Improved performance of the greedy algorithm for partial cover. Information Processing Letters, 64(5):251–254, 1997. The Wall Street Journal. The uc berkeley project that is the ai industry’s obsession, December 2024. URL https://www.wsj.com/tech/ai/ the-uc-berkeley-project-that-is-the-ai-industrys-obsession-bc68b3e3. Accessed: 2026-01-31. Anton Voronov, Lena Wolf, and Max Ryabinin. Mind your format: Towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6287–6310, 2024. Madeleine Waller, Odinaldo Rodrigues, and Oana Cocarascu. Beyond consistency: Nuanced metrics for individual fairness. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 2087–2097, 2025. Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Jing Jiang, and Min Lin. Cheating automatic llm benchmarks: Null models achieve high win rates. In The Thirteenth International Conference on Learning Representations.
13
A
Further details on Arena case-study
Here, we supply further details for the Arena case-study in Section 4. A.1
Conversion table from Elo rating scale to prediction probability
We include a table that maps the difference in Elo scores in a BT ranking and the BT ranking’s confidence that the model with the higher score wins, up to 2 decimal places. For example, for two models a, b with scores 1050 and 1450 respectively (score difference of 400), the BT ranking predicts −1 that a wins in a vote against b with probability 1 + exp( 1050−1450 ) = 1/(1 + e−1 ) ≃ 0.73. 400 Thus, the BT ranking predicts that a wins against b in roughly 3 out of every 4 votes. More generally, Eqn. 1 implies that an Elo rating difference of δ ≥ 0 implies that the model with higher score wins with probability 1 p(δ) := , 1 + exp(−δ) with p(0) = 12 (models with equal score are equally likely to win), and limδ→∞ p(δ) = 1. Table 1: Conversion between Elo score difference between two models and the predicted probability of the higher model winning in a vote. Elo score difference Win prob. (higher-ranked)
A.2
0 0.50
10 0.51
50 0.53
100 0.56
200 0.62
400 0.73
1000 0.92
1500 0.98
2000 0.99
LLM display name mapping
Table 2 lists the full model identifiers and release dates for all 52 LLMs referenced in this work, alongside the abbreviated display names used in figures throughout the paper. Models are ordered by version/release date. A.3
Language family mapping
Table 3 maps the 116 Arena languages to coarse language families used throughout the paper. A.4
Heterogeneity in language-specific rankings
Figure 5 shows ranks of top LLMs in the Arena dataset for the 15 most used languages in Arena. A.5
Portfolios of BT rankings
Greedy algorithm. Table 4 lists the greedy-selected BT ranking portfolios achieving ν = 0.95 coverage at each λ threshold, selected from 389 candidate rankings. Portfolio size k is the number of rankings selected; coverage reports the fraction of 89,193 decisive votes λ-covered by at least one ranking in the portfolio. MIP. Table 5 lists the MIP-selected BT ranking portfolios achieving ν = 0.95 coverage at each λ threshold, selected from 389 candidate rankings. Portfolio size k is the number of rankings selected; coverage reports the fraction of 89,193 decisive votes λ-covered by at least one ranking in the portfolio. A.6
BT rankings to LLM scores
Having computed the set R of 389 BT rankings, we show how to compute the score of each LLM ℓ for any given vote i = (a(i) ≻ b(i)). Recall that the LLM ℓ may not be one of ai or bi . For each ranking r, recall that the accuracy of the ranking for vote i is defined as p(i, r) := Prr (ai ≻ bi ) =
14
1 (r) (r) 1 + exp(θb(i) − θa(i) )
.
Table 2: Mapping of LLM display names to full model identifiers and version/release dates. Display name
Model identifier
Release date
GPT-4o-mini Claude-3.5H Claude-3.5S GPT-4o Nova-Pro Llama-3.3 Gemini-2FT Qwen-Max o3-mini Gemini-2F Claude-3.7S Claude-3.7ST Grok-3MB Grok-3MH Grok-3P QwQ-32B Gemma-3 Command-A Mistral-S-3.1 DeepSeek-V3 Gemini-2.5P3 ChatGPT-4o Llama-4ME Llama-4MI Llama-4SI GPT-4.1 GPT-4.1M Hunyuan-Turbos o3 o4-mini Gemini-2.5-FP Qwen3-235B Qwen3-235BNT Qwen3-30B Gemini-2.5P5 Mistral-M Claude-4 Claude-4T Claude-4S Claude-4ST Nova-Exp DeepSeek-R1 Magistral Mistral-S Gemini-2.5F Gemini-2.5FLT Gemini-2.5P MiniMax Gemma-3n Grok-4 Kimi-K2 Qwen3-235BI
gpt-4o-mini-2024-07-18 claude-3-5-haiku-20241022 claude-3-5-sonnet-20241022 gpt-4o-2024-11-20 amazon.nova-pro-v1:0 llama-3.3-70b-instruct gemini-2.0-flash-thinking-exp-01-21 qwen-max-2025-01-25 o3-mini gemini-2.0-flash-001 claude-3-7-sonnet-20250219 claude-3-7-sonnet-20250219-thinking-32k grok-3-mini-beta grok-3-mini-high grok-3-preview-02-24 qwq-32b gemma-3-27b-it command-a-03-2025 mistral-small-3.1-24b-instruct-2503 deepseek-v3-0324 gemini-2.5-pro-preview-03-25 chatgpt-4o-latest-20250326 llama-4-maverick-03-26-experimental llama-4-maverick-17b-128e-instruct llama-4-scout-17b-16e-instruct gpt-4.1-2025-04-14 gpt-4.1-mini-2025-04-14 hunyuan-turbos-20250416 o3-2025-04-16 o4-mini-2025-04-16 gemini-2.5-flash-preview-04-17 qwen3-235b-a22b qwen3-235b-a22b-no-thinking qwen3-30b-a3b gemini-2.5-pro-preview-05-06 mistral-medium-2505 claude-opus-4-20250514 claude-opus-4-20250514-thinking-16k claude-sonnet-4-20250514 claude-sonnet-4-20250514-thinking-32k amazon-nova-experimental-chat-05-14 deepseek-r1-0528 magistral-medium-2506 mistral-small-2506 gemini-2.5-flash gemini-2.5-flash-lite-preview-06-17-thinking gemini-2.5-pro minimax-m1 gemma-3n-e4b-it grok-4-0709 kimi-k2-0711-preview qwen3-235b-a22b-instruct-2507
2024-07-18 2024-10-22 2024-10-22 2024-11-20 2024-12-03 2024-12-06 2025-01-21 2025-01-25 2025-01-31 2025-02-05 2025-02-19 2025-02-19 2025-02-19 2025-02-19 2025-02-24 2025-03-05 2025-03-10 2025-03-13 2025-03-17 2025-03-24 2025-03-25 2025-03-26 2025-03-26 2025-04-05 2025-04-05 2025-04-14 2025-04-14 2025-04-16 2025-04-16 2025-04-16 2025-04-17 2025-04-28 2025-04-28 2025-04-28 2025-05-06 2025-05-07 2025-05-14 2025-05-14 2025-05-14 2025-05-14 2025-05-14 2025-05-28 2025-06-10 2025-06-10 2025-06-17 2025-06-17 2025-06-17 2025-06-17 2025-06-26 2025-07-09 2025-07-11 2025-07-21
15
Table 3: Mapping of languages to language families. Groupings follow standard linguistic taxonomy (Glottolog) at a coarse level, e.g., Indo-Iranian combines the Indo-Aryan and Iranian branches, and isolate languages are grouped pragmatically. Family
Family
Languages
English, German, Dutch, Swedish, Norwegian, Norwegian Nynorsk, Danish, Icelandic, Afrikaans, Luxembourgish, Scots, Western Frisian Romance French, Spanish, Portuguese, Italian, Catalan, Romansh, Occitan, Galician, Corsican, Romanian, Haitian, Seselwa Creole French, Latin Polish, Russian, Ukrainian, Slavic Czech, Slovak, Croatian, Bosnian, Serbian, Bulgarian, Belarusian, Macedonian, Slovenian Indo-Iranian Hindi, Urdu, Persian, Marathi, Gujarati, Bengali, Assamese, Sanskrit, Nepali, Sindhi, Kurdish, Tajik, Sinhala Sino-Tibetan Chinese, Tibetan, Burmese
Languages
Afro-Asiatic
Arabic, Hausa, Somali, Maltese, Oromo, Afar
Niger-Congo
Swahili, Zulu, Xhosa, Lingala, Wolof, Kinyarwanda, Ganda, Akan, Southern Sotho, Shona, Rundi, Sango
Austronesian
Malay, Tagalog, Cebuano, Malagasy, Waray, Tonga, Nauru, Indonesian
Dravidian
Tamil, Malayalam, Kannada
Turkic
Hellenic
Modern Greek
Uralic
Turkish, Uzbek, Azerbaijani, Kazakh, Kirghiz, Tatar, Uighur Finnish, Estonian, Hungarian
Baltic
Latvian, Lithuanian
Celtic
Irish, Welsh, Manx
Kartvelian
Georgian
Koreanic
Korean
Albanian
Albanian
Japonic
Japanese
Armenian
Armenian
Tai-Kadai
Thai
Eskimo-Aleut Kalaallisut
Isolate
Basque
Amerind
Constructed
Esperanto, Interlingue, Interlingua, Klingon, Volapük
Germanic
Austroasiatic Vietnamese, Khmer, Khasi
Quechua, Guarani
This allows us to construct a probability distribution σ(i) over the rankings, where σ(i, r) indicates that vote i is generated from ranking r ∈ R, or that the ranking r represents the true preference of the corresponding user. p(i, r) . s∈R p(i, s)
σ(i, r) := P
Then, we define the probability of an LLM ℓ satisfying the vote i as: q(i, ℓ) := Prσ(i) (ℓ is at least as good as the winner ai ) X = Prr (ℓ is at least as good as the winner ai |i is generated from r) · σ(i, r) r∈R
(P = 1
σ(i,r) r∈R 1+exp(θ (r) −θ (r) ) ℓ a(i)
if ℓ ̸= ai , otherwise.
Finally, we define the ‘error’ ε(i, ℓ) of LLM ℓ for vote i as the probability of ℓ not beating ai , i.e., ε(i, ℓ) := 1 − q(i, ℓ). 16
Figure 5: Heterogeneity in LLM rankings across languages. Rows: 15 languages with the most votes in Arena. Columns: top models by mean rank across these languages, ordered by release date. Each cell reports the model’s within-language rank (1 = best). Demonstrates how the same model can rank in the top few for one language and well outside it for another.
17
Table 4: The BT rankings selected by the greedy algorithm achieving ν = 0.95 coverage at each λ threshold. λ
k Selected Rankings
0.95 1 global 0.90 1 global 0.80 1 English × code 0.70 1 English × specificity 0.60 3 English × specificity; Vietnamese × real_world; Indonesian × complexity 0.50 5 global; Portuguese × math; Portuguese × creative_writing; Dutch × creativity; Persian × creative_writing 0.45 4 Persian × creative_writing; Portuguese × creative_writing; Romanian; Bengali × complexity 0.40 5 Thai; Dutch × creativity; Japanese × math; Portuguese × creative_writing; Romanian × domain_knowledge 0.35 5 Thai; Dutch × creativity; Japanese × math; Arabic × code; Romanian × domain_knowledge 0.30 6 Thai; Arabic × creative_writing; Dutch × complexity; Arabic × code; Malay; Portuguese × math 0.25 6 Hungarian × creativity; Arabic × creative_writing; Arabic × code; Thai × domain_knowledge; Dutch × complexity; Swedish × domain_knowledge 0.20 6 Arabic × creative_writing; Thai; Arabic × code; Malay; Dutch × complexity; Swedish × domain_knowledge 0.15 6 Arabic × creative_writing; Thai × domain_knowledge; Afro-Asiatic × code; Hungarian × creativity; Malay; Hungarian × specificity 0.10 6 Arabic × creative_writing; Thai × domain_knowledge; Arabic × code; Hungarian × creativity; Malay; Hungarian × specificity 0.05 7 Arabic × creative_writing; Thai × domain_knowledge; Arabic × code; Hungarian × creativity; Malay; Latin; Swedish × domain_knowledge
A.7
Pairwise win probability distributions
Figure 6: Distribution of pairwise win probabilities Pr(a ≻ b) by ranking type. Language-task rankings exhibit substantially more mass near 0 and 1 than random or hour-based baselines, indicating that stratified rankings induce sharper pairwise preferences. 18
Table 5: The BT rankings selected by the MIP algorithm achieving ν = 0.95 coverage at each λ threshold. λ k Selected Rankings 0.95 1 domain_knowledge 0.90 1 domain_knowledge 0.80 1 domain_knowledge 0.70 1 domain_knowledge 0.60 3 Japanese × technical_accuracy, Indonesian × complexity, Bengali × specificity 0.50 4 Turkish × technical_accuracy, Portuguese × creative_writing, Persian × creative_writing, Romanian × domain_knowledge 0.45 4 Turkish × technical_accuracy, Portuguese × creative_writing, Persian × creative_writing, Romanian × domain_knowledge 0.40 4 Portuguese × creative_writing, Persian × creative_writing, Romanian × domain_knowledge, Other × specificity 0.35 5 Thai, Portuguese × creative_writing, Dutch × creativity, Romanian × domain_knowledge, Japonic × math 0.30 5 Turkish × technical_accuracy, Portuguese × creative_writing, Persian × creative_writing, Romanian × domain_knowledge, Other × specificity 0.25 5 Danish × domain_knowledge, Portuguese × creative_writing, Persian × creative_writing, Romanian × domain_knowledge, Other × specificity 0.20 6 Malay, Portuguese × creative_writing, Dutch × creativity, Persian × creative_writing, Indonesian × complexity, Romanian × domain_knowledge 0.15 6 Danish × domain_knowledge, Dutch × technical_accuracy, Persian × creative_writing, Arabic × code, Romanian × domain_knowledge, Other × specificity 0.10 7 Thai, Malay, Portuguese × creative_writing, Japanese × math, Czech × creative_writing, Romanian × domain_knowledge, Uralic × real_world 0.05 7 Thai, Malay, Serbian, Portuguese × creative_writing, Czech × creative_writing, Romanian × domain_knowledge, Uralic × real_world
A.8
Portfolios of LLMs
Table 6 compares the portfolios of LLMs induced by (1) greedy algorithm’s portfolio of BT rankings, (2) MIP’s portfolio of BT rankings, and (3) choosing the top-k ranked models in the global BT ranking. The greedy and MIP portfolios are selected from the BT rankings at λ = 0.05 to achieve target coverage ν = 0.95. The posets reported here are evaluated at λ = 0.20 to see how much cumulative coverage is achieved with k = 10 LLMs. The global baseline corresponds to the global BT ordering. A.9
Coverage of language-specific votes by BT rankings
Figure 8 reports the coverage of the global BT ranking separately for each language at λ = 0.45. Each point represents one language, with the x-axis showing the number of votes on a log scale and the y-axis showing the fraction of votes covered. A.10
Portfolio error distributions
Figure 9 compares per-vote portfolio errors for the greedy and MIP portfolios. At tighter thresholds (e.g., λ = 0.4), the greedy portfolio concentrates more error mass near zero. Its first few picks target high-coverage language strata, yielding low error on the votes they cover, but leave a heavier tail of poorly covered votes. This results in a higher overall mean error than the MIP portfolio. As λ increases, this gap narrows and at λ = 0.8, the two methods achieve nearly identical mean errors. Greedy excels at covering easy votes cheaply, while MIP distributes coverage more evenly, so hybrid selection strategies may be a promising direction for future work. 19
Table 6: Model orderings induced by the greedy-selected, global, and MIP-selected BT ranking portfolios. Greedy portfolio Global ordering Rank LLM Coverage Rank LLM Coverage 1 2 3 4 5 6 7 8 9 10
deepseek-r1-0528 gpt-4.1-mini-2025-04-14 gemini-2.5-pro o3-2025-04-16 claude-sonnet-4-2025051 4 grok-3-preview-02-24 gemini-2.5-flash gemma-3-27b-it o4-mini-2025-04-16 deepseek-v3-0324
0.295 0.384 0.468 0.528 0.571
1 2 3 4 5
0.611 0.649 0.681 0.709 0.735
6 7 8 9 10
gemini-2.5-pro gemini-2.5-pro-preview03-25 grok-4-0709 o3-2025-04-16 gemini-2.5-pro-preview05-06 deepseek-r1-0528 chatgpt-4o-latest-202503 26 grok-3-preview-02-24 llama-4-maverick-03-26experimental gemini-2.5-flash
0.106 0.113 0.121 0.163 0.178 0.209 0.246 0.274 0.301 0.345
MIP portfolio Rank LLM 1 2 3 4 5 6 7 8 9 10
B
Coverage
gemini-2.5-pro grok-4-0709 claude-opus-4-20250514thinking-16k gemini-2.5-flash o3-2025-04-16 qwen3-235b-a22b-no-thi nking chatgpt-4o-latest-202503 26 mistral-medium-2505 deepseek-r1-0528 deepseek-v3-0324
0.162 0.223 0.269 0.313 0.354 0.393 0.429 0.461 0.490 0.519
Further details on COMPAS case-study
Here, we supply further details for the COMPAS case-study in Section 4.2. B.1
Model portfolio-sizes by λ values for different population coverage targets
Figure 10 shows the minimum number of models required to achieve a target coverage level over the full population for different values of λ where coverage is attainable. As expected, higher coverage targets require larger model portfolios and are achievable at looser thresholds (higher λ values). For example, to achieve 100% coverage, we need 8 models at λ = 0.5, whereas 70% coverage only requires 2 models at the same λ threshold. Similarly, portfolios converge to a single model at lower λ thresholds for lower coverage targets. B.2
Analysis of uncovered population
Table 7 shows the portfolio constructed at λ = 0.4. Here, the model portfolio achieves 83.56% coverage, with the remaining individuals assigned to “No model.” We explicitly retain this uncovered set to analyze the characteristics of individuals not captured by any model in the ensemble. We find that this group is predominantly younger individuals with few prior offenses, across all race and sex groups, and that every individual in this subset is a repeat offender (as shown in Table 8 for the female sub-population). This highlights that the portfolio approach can also be used as a tool for 20
Table 7: Assignments to model portfolio for the COMPAS data constructed at λ = 0.4, by subpopulation. Data point that were not covered by any regularized model are depicted in the last row. Model
F-AA
M-AA
F-C
M-C
F-O
M-O
Global Race(100) Race(140) Race(150) Race(180) Race(210) Race(230) Race(300) Sex(10) Sex(110) Sex(140) Sex(180) Sex(240) Sex(290) No model
0 1 0 0 0 8 2 17 0 69 1 0 0 5 13
0 0 0 0 0 8 7 15 2 41 0 0 0 3 10
0 0 0 0 0 3 2 4 0 17 0 0 0 0 5
11 0 1 1 1 32 6 78 0 277 2 0 0 22 94
4 6 0 0 1 24 9 49 0 180 2 2 2 5 57
0 0 0 0 0 5 4 18 0 76 0 0 0 9 24
Notes: F-AA = Female African-American, M-AA = Male African-American, F-C = Female Caucasian, M-C = Male Caucasian, F-O = Female Other, M-O = Male Other. Global model is the best fit linear regression model, with no regularization. The Sex(µ) and Race(µ) depict fairness-regularized models with penalty parameter µ set as indicated.
surfacing blind spots in predictive systems, perhaps offering policymakers insight into where models may systematically under-perform and, as a result, where greater attention in oversight might be required. B.3
FPR comparison between single best model and greedily-constructed portfolios
In Table 9, we compare model portfolios constructed under different λ thresholds for full population coverage against a global best model that optimizes only BCE. As expected, portfolios constructed at lower λ thresholds have larger sizes, reflecting a stricter notion of coverage that requires a more diverse set of models to account for variation across individuals. Under λ = 0.4, the constructed portfolio achieves zero overall and sub-population FPRs, suggesting that a wider, more diverse set of models improves the ability to capture differences across individuals, leading to more precise predictions. As λ increases, smaller portfolios can achieve the targeted coverage (100%) but the resulting portfolios have non-zero FPRs across the full population and within most sub-populations. This indicates that the selected model sets no longer capture all of the individual nuances. Despite this, we find that the portfolio approach consistently yields lower FPRs than the global best model for the overall population. Within the sub-populations, the FPRs from portfolios are also generally lower than those of the global model for all sub-populations and at all λ values, with the exception of where λ = 0.8. In this case, where a single model can be used to cover the entire population, the FPR increases slightly for the Male-Caucasian sub-population compared to the global best model but remains lower for the overall and all other sub-population FPRs.
C
Implementation details for MIP
Since MIP is an integer program, it may not be tractable to solve to optimality in a reasonable time. Instead, we solve the linear relaxation optimally, and obtain fractional variables ui , tj ∈ [0, 1] for data points i and models j. We use the following continuous threshold rounding scheme to round these to an integer solution and obtain a portfolio: • initialize portfolio P ← ∅ of models 21
Table 8: Features of the uncovered female sub-population, using the constructed portfolio at λ = 0.4 Age
Age Category
Sex
Race
Priors Count
Days
Length of Stay
Charge Type
2-Year Recid
24
Less than 25
Female
African-American
0
-1.0
0
F
1
21
Less than 25
Female
African-American
0
-1.0
1
F
1
23
Less than 25
Female
African-American
0
-1.0
3
F
1
23
Less than 25
Female
African-American
0
-1.0
2
F
1
21
Less than 25
Female
African-American
0
0.0
0
F
1
24
Less than 25
Female
Caucasian
0
-1.0
0
M
1
30
25 - 45
Female
Other
0
-1.0
1
M
1
32
25 - 45
Female
Other
0
-1.0
7
M
1
29
25 - 45
Female
Other
0
-1.0
1
F
1
29
25 - 45
Female
Other
0
-2.0
1
M
1
21
Less than 25
Female
Other
0
-1.0
0
M
1
20
Less than 25
Female
African-American
1
-1.0
1
F
1
21
Less than 25
Female
African-American
1
-1.0
0
F
1
26
25 - 45
Female
Caucasian
1
-1.0
0
F
1
34
25 - 45
Female
Caucasian
1
-1.0
12
F
1
33
25 - 45
Female
Caucasian
1
-5.0
4
M
1
31
25 - 45
Female
African-American
2
-1.0
0
F
1
23
Less than 25
Female
African-American
2
-2.0
1
F
1
24
Less than 25
Female
African-American
2
-1.0
1
F
1
39
25 - 45
Female
Caucasian
2
-1.0
0
F
1
31
25 - 45
Female
Caucasian
2
-1.0
1
F
1
29
25 - 45
Female
Caucasian
2
-1.0
8
F
1
27
25 - 45
Female
African-American
3
-1.0
0
M
1
38
25 - 45
Female
African-American
3
0.0
0
F
1
44
25 - 45
Female
Caucasian
3
0.0
11
F
1
33
25 - 45
Female
Caucasian
4
-1.0
11
M
1
53
Greater than 45
Female
Caucasian
4
-1.0
0
F
1
35
25 - 45
Female
African-American
5
-1.0
7
F
1
Features related to juvenile criminal history counts are not shown as they are primarily zero across the entire sub-population. Days denotes the number of days between screening and arrest. Charge-type M indicates misdemeanor. Charge-type F indicates felony. No model was able to cover these defendants within reasonable error.
• while fewer than νn points have been λ-covered by models in P , choose the model not in P that has the highest value of tj , and add it to P When each tj is integral, it is easy to check that this returns the corresponding portfolio P := {j : tj = 1}. Further, we also run a second phase of the MIP to obtain smaller overall errors: after the rounding, suppose the MIP returns a portfolio of size |P | = k, showing that a (λ, ν)-portfolio of size k exists. Then, we write another mixed-integer program to find the (λ, ν)-portfolio of fixed size k 22
Table 9: False Positive Rate (FPR) metrics across sub-populations for different model portfolios λ
Target Coverage
Overall FPR
F-AA
MAA
F-C
M-C
F-O
M-O
{Global} N/A {Sex(150)} 0.8 {Race(30), 0.7 Sex(10)} {Race(30, 40), 0.6 Sex(140)} {Global, 0.5 Race(210, 230, 280), Sex(160, 170, 230, 260)}
N/A 100% 100%
0.216 0.103 0.094
0.103 0.014 0.014
0.388 0.056 0.130
0.043 0.167 0.093
0.178 0.198 0.090
0.012 0.048 0.000
0.107 0.037 0.098
100%
0.074
0.014
0.091
0.093
0.080
0.000
0.073
100%
0.000
0.000
0.000
0.000
0.000
0.000
0.000
Model Portfolio
Reported FPR values are rounded to three decimal places. Global model is the best fit linear regression model, with no regularization. The Sex(µ) and Race(µ) depict fairness-regularized models with penalty parameter µ set as indicated. F-AA = Female African-American, M-AA = Male African-American, F-C = Female Caucasian, M-C = Male Caucasian, F-O = Female Other, M-O = Male Other.
that minimizes the overall mean-squared error across data points, where each data point is assigned to the model in the portfolio that has the smallest error on the data point.
D
Data sources
The Arena dataset data was provides by Arena on HuggingFace (https://huggingface. co/datasets/lmarena-ai/arena-human-preference-140k) under the CC BY 4.0 license. COMPAS data was obtained from ProPublica (https://github.com/propublica/ compas-analysis) under the CC BY 4.0 license.
23
Rank Model
BT Score (Elo-scaled)
1 Gemini-2.5P 2 Gemini-2.5P3 3 Grok-4 4 o3 5 Gemini-2.5P5 6 DeepSeek-R1 7 ChatGPT-4o 8 Grok-3P 9 Llama-4ME 10 Gemini-2.5F 11 Gemini-2.5-FP 12 Qwen3-235BNT 13 Hunyuan-Turbos 14 Kimi-K2 15 Qwen3-235B
1144 1114 1093 1093 1091 1088 1088 1080 1074 1072 1052 1044 1044 1039 1036
48 Mistral-S-3.1 49 Nova-Pro 50 GPT-4o-mini 51 Claude-3.5H 52 Magistral
892 890 876 871 867
Figure 7: Language Family BT rankings achieve better coverage than a global BT ranking. The top panel compares coverage under language-family-specific BT rankings against the global BT ranking, while the bottom table reports the top 15 models under the global BT ranking along with their BT scores.
24
Figure 8: Coverage of the global BT ranking by language at λ = 0.45. Each point corresponds to a language that has at least 100 votes, colored by language family. The x-axis shows the number of votes on a log scale, and the y-axis shows the fraction of votes covered by the global ranking. The variation across languages indicates that a single global ranking does not represent all language subpopulations uniformly. The dashed line shows an ordinary least squares fit, with the shaded region indicating the 95% confidence interval. The weak slope (-0.003) suggests that variation in global-ranking coverage is not simply explained by the number of votes available for each language.
Figure 9: Per-vote portfolio error distributions for MIP and greedy BT ranking portfolios at λ ∈ {0.4, 0.6, 0.8}. Portfolio error for each vote is the minimum error across selected rankings. Dashed lines: mean error; dotted lines: λ threshold.
25
Figure 10: Minimum number of models needed to achieve target population coverage across λ thresholds.The models in the greedily-constructed portfolios are listed for the largest portfolio-size for a given coverage target and for when the portfolio converges to a single model.
26