Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

AGGREGATE - THEN -C ALIBRATE FOR H UMAN - CENTERED A SSESSMENT WITH T HEORETICAL G UARANTEES

Zejun Xie1∗, Xintong Li2∗, Guang Wang3†, Desheng Zhang1 Rutgers University 2 Renmin University of China 3 Florida State University 1 {zx180, dz220}@cs.rutgers.edu 2 lixintong@ruc.edu.cn 3 guang.wang@fsu.edu 1

arXiv:2608.02455v1 [cs.LG] 3 Aug 2026

A BSTRACT Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features. We propose Aggregate-then-Calibrate (AtC), a two-stage framework that combines these complementary sources. Stage-1 aggregates heterogeneous comparative judgments into a consensus ranking π b using a rank-aggregation model that accounts for annotator reliability. Stage-2 calibrates any predictive model’s scores by an isotonic projection onto the order π b, enforcing ordinal consistency while preserving as much of the model’s quantitative information as possible. Theoretically, we show: (1) modeling annotator heterogeneity yields strictly more efficient consensus estimation than homogeneity; (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified; and (3) AtC asymptotically outperforms model-only assessment. Across semi-synthetic and real-world datasets, AtC consistently improves accuracy and robustness over human-only or model-only assessments. Our results bridge judgment aggregation with model-free calibration, providing a principled recipe for human-centered assessment when ground truth is costly, scarce, or unverifiable.

1

Introduction

Human-centered assessment tasks are essential for systematic decision-making in domains where ground truth is costly, unobservable, or only available in the future. Consider two scenarios: (1) Delivery platforms seek to estimate worker workload for fair compensation, yet true energy expenditure, though measurable via wearables, is impractical to collect at scale, necessitating reliance on worker judgments [1, 10, 31, 78]. (2) Conference committees must evaluate paper quality for acceptance decisions, yet the true quality only becomes observable years later through future impact, requiring committees to aggregate reviewer assessments in the present [18, 67, 69, 92]. These tasks share key characteristics: they seek standardized assessments from human judgments rather than individual preference satisfaction [11, 13, 36], and while ground truth exists conceptually, such as workload or future impact, it cannot be directly observed when decisions are needed [2, 81]. Existing approaches tend to rely either solely on human judgments or solely on model predictions. Methods that use only annotator-provided comparative judgments, such as judgment aggregation algorithms [12, 39, 96], can weight annotators by expertise but remain confounded by inconsistent rating scales. For example, a lenient expert and a strict novice might agree on an item’s relative quality yet assign very different absolute scores [37, 42]. Conversely, methods that use only model-generated scores from proxy labels face a supervision crisis: the true target values may correlate with latent factors that are impractical to measure at scale, such as cognitive load in task difficulty estimation [25, 44, 71, 80]. This forces models to learn from noisy surrogates, propagating systematic biases into the assessments. ∗ †

Equal contribution and shared co-first authorship. Corresponding author.

Intuitively, human judgments are easy to solicit but lack a universal scale, whereas model predictions are consistent by design but require supervised signals [14, 15]. Our goal is to combine the strengths of both sources: use the scale from the model assessment and the structure from human judgments [29]. A key insight is that people are typically more reliable at comparing items than at absolute scoring (also known as Weber–Fechner laws in psychophysics [63]). We therefore extract only the ordinal information from the noisy human inputs, rather than trusting their raw rating values or aggregated scores. We propose a two-stage framework called Aggregate-then-Calibrate (AtC) that integrates the complementary advantages of human judgments and model assessments. In the aggregation stage, AtC employs a heterogeneous rank aggregation approach [12, 38, 43] to derive a consensus ranking of items from multiple annotators, explicitly accounting for each annotator’s individual scale and reliability. In the calibration stage, AtC uses isotonic regression [19, 23, 67] to adjust an initial predictive model’s assessments so that they align with the consensus ranking. This two-step approach ensures that the final assessments are monotonic with respect to the human consensus judgments. By combining the ordinal structure (from human) with the scoring scale (from the model), we theoretically and empirically demonstrate that AtC leverages the best of both: the human judgments provide a reliable ranking backbone, and the model provides consistent scoring, resulting in assessments that outperform either approach alone. Our contributions are summarized as follows: • Conceptual: We formalize a specific class of human-centered assessment problems where judgment aggregation (not preference optimization) is central for systematic decision-making. AtC addresses these via a human-model complementarity: aggregated comparisons provide ordinal constraints, while models estimate metrically scaled scores. Moreover, it can be generalized to any predictive model and thus enables the use of any off-the-shelf predictive model without modification. • Theoretical: We provide a comprehensive analysis for AtC from three perspectives. First, to the best of our knowledge, Theorem 3.6 offers the first proof that heterogeneous rank aggregation models are strictly more statistically efficient than homogeneous methods when annotator abilities vary, validating our approach for the initial aggregation stage. Second, the risk analysis in Theorem 3.8 makes novel contributions to isotonic regression theory beyond prior work [5, 9, 88] by accounting for projection onto a random cone and managing biased effective noise, which are two necessary conditions for our problem setting. Finally, these components culminate in our optimality guarantee in Theorem 3.12, formally demonstrating the power of integrating human ordinal judgments with model-based scores. • Empirical: AtC consistently outperforms human-only and model-only assessments across semi-synthetic and real-world datasets, demonstrating superior accuracy and robustness under varying levels of data degradation.

2

Aggregate-then-Calibrate Framework

2.1

Overview

We consider a human-centered assessment task for n items using inputs from m human annotators and a predictive model p. Let s ∈ Rn denote the true (unobserved) latent scores of the n items, representing their ideal ground-truth quality (e.g., the true proficiency of a student’s answer or the true merit of a product). Since s cannot be directly observed, we collect two complementary sources of information to estimate it: • Human Judgments: Annotators provide comparative assessments (e.g., pairwise preferences or ratings) for various subsets of the items. These judgments are aggregated to produce a consensus score vector, which we denote by e s ∈ Rn . The vector e s represents the aggregated consensus over item scores derived purely from the given human input. We model e s as a noisy observation of the true scores s; specifically, we assume e s = s + ϵe, where ϵe ∼ N (0, σ e2 In ). ∗ In practice, e s is not observed directly; instead, we estimate this consensus via the Stage-1 procedure. Let s ∈ Rn denote the Stage-1 estimate of e b be the consensus ranking of the n items, s (as described in Stage-1 below). Let π obtained by sorting the entries of s∗ from lowest to highest. This ranking π b reflects the collective ordering of items according to the annotators’ comparative judgments. • Model Assessments: A predictive model p (e.g., a machine learning model analyzing item features) produces an initial score vector sp ∈ Rn for the items. We do not assume that sp is perfectly calibrated or aligned with the human judgments; indeed, the model may have been trained on proxy labels or may even be unsupervised due to the lack of ground truth. We model sp as a biased observation of the true scores, subject to an unknown systematic offset: specifically, sp = s + ν, where ν ∈ Rn is an unknown error (or bias) vector. Given the relationship e s = s + ϵe from above, we can relate the model scores to the consensus by writing sp = e s − ϵe + ν. For simplicity, we assume that the model’s noise is independent of the annotators’ noise (i.e., ν is independent of ϵe). 2

s ∈ Rn for the n items. The calibrated scores b s should (i) respect Our goal is to produce a final calibrated score vector b the consensus ordering π b (to maintain agreement with collective human judgment), and (ii) remain as close as possible to the model’s original scores sp (to preserve the model’s useful quantitative information). We formalize this goal by defining the AtC estimator as the projection of sp onto the set of score vectors that are consistent with π b. Let n c M := {y ∈ R : yπb(1) ≤ yπb(2) ≤ · · · ≤ yπb(n) } be the set of all length-n vectors that are nondecreasing according to the ranking π b. The AtC estimate is then given by: 2 b s = ΠM c(sp ) = arg min ∥y − sp ∥2 ,

(2.1)

c y∈M

c where ΠM s is the closest c(sp ) denotes the Euclidean projection of sp onto the constraint set M. By construction, b vector to sp (in ℓ2 -distance) that does not violate the consensus ordering. In other words, b s is obtained by an isotonic regression of the model’s scores onto the ordering π b. Figure 1 illustrates the roles of e s, sp , and b s in AtC. ´

n

¤

¤

^ = Sort(¡s ) s 2R ; ¼

( s¤ ; ° ¤ ) Homogeneous:

Fix ° = °0

³

s

(T ¡3)

¡ ¤ 0¢ s ;°

; °0

Thm: 3:3 §^shete ¹ §^shomo

c = fy : y¼^(1) · ¢ ¢ ¢ · y¼^(n)g M

Aggregate

´ f M

M

sp bs

sp

²~

bs

º

s

…

AÂC BÂA

m people, n items.

Consensus ¼ b:

AÂBÂC

Stage-2. Calibrate

sp

b s

sp = [8; 5; 7]

s

2

Thm: 3:5 E[k^s ¡ ~sk22] · E1;2;3

AÂB BÂC

CÂA

P = 1 ¡ ±1 ¡ ±2 c =M f =M M

Stage-1. Aggregate

AÂB

Calibrate

c M

e s

c M

o

; ° (T ¡3)

nose

(T ¡3)

eeaotoo

s

2

Thm: 3:9 k^s ¡ sk2 < ksp ¡ sk2

Figure 1: AtC Framework and Its Theoretical Guarantees.

b s = [8; 6; 6]

s

¤

sp

^s 🥇 0

³

Kendall ¿

s. Stage-1 aggregates human judgments into a ranking π b; Stage-2 calibrates model scores sp via isotonic projection onto M to obtain b Theorems 3.6, 3.8, and 3.12 provide efficiency, risk, and superiority guarantees.

2.2

Stage-1: Judgment Aggregation under Heterogeneous Thurstone Model

In Stage-1, we aggregate the human judgments into a single consensus ranking. We formulate this rank aggregation problem as finding a latent score vector (the consensus) that best explains the annotators’ comparative judgments. Each annotator u ∈ {1, . . . , m} provides pairwise preferences over certain item pairs (for example, i ≻ j indicates annotator u prefers item i over item j). A key design choice is how to handle observed inconsistencies when different annotators judge the same items. When couriers assess delivery route difficulty, their judgments differ—but this reflects heterogeneity in expertise rather than fundamental subjective preferences. Rather than filtering novice opinions, our approach learns each annotator’s reliability γu from data and applies optimal weighting to produce consensus closer to the latent target. Preliminary: Heterogeneous Thurstone Model (HTM). We assume these preferences follow a Thurstone choice model, extended to allow annotator-specific noise levels—this is HTM [39]. In particular, given any two items i and j that annotator u compares, the probability that u prefers i to j is modeled as: Pr{u : i ≻ j} = F(γu (si − sj )) , where F is a fixed symmetric cumulative distribution function (e.g., the standard normal CDF in Thurstone’s original formulation, or a logistic CDF in a Bradley–Terry model), and γu > 0 is annotator u’s precision (or consistency) parameter. A larger γu means annotator u is less noisy (more reliable) in their pairwise comparisons, whereas a smaller γu implies higher variance (lower consistency) for that annotator. To infer the consensus scores from the observed comparisons, maximum-likelihood estimation (MLE) is performed under the HTM. Let Du denote the set of observed pairwise comparisons from annotator u, where i ≻ j ∈ Du means u preferred Pm Pitem i over j. The log-likelihood of all annotators’ data, given parameters (s, γ), is: ℓ(s, γ) = − sj )) , with appropriate constraints on (s, γ) for identifiability (e.g., one can fix u=1 i≻j ∈ Du log F(γu ( siP P n m 1 1 ∗ ∗ ∗ i=1 si = 0 or constrain m u=1 γu = 1). The consensus score estimates s = (s1 , . . . , sn ) and the annotator n 3

reliabilities γ are obtained by maximizing this log-likelihood: maxs, γ ℓ(s, γ) , i.e. by finding the parameters that best explain the observed comparisons. This optimization can be carried out with iterative methods; for example, one can alternate between updating the score estimates s and the annotator precisions γ until convergence (each update can use gradient ascent on ℓ or other numerical routines). The result of Stage-1 is an estimated consensus score vector s∗ . Sorting the components of s∗ from lowest to highest yields the consensus ranking π b. In other words, π b is the permutation of item indices such that s∗πb(1) ≤ s∗πb(2) ≤ · · · ≤ ∗ sπb(n) . This Stage-1 outcome π b reconciles the varying, potentially inconsistent human judgments into a single ranked list of items.

Algorithm 1 Heterogeneous Rank Aggregation 1: Input: Comparative judgment data {i ≻ j ∈ Du : u = 1, . . . , m} from m annotators. 2: Output: Consensus score vector e s and consensus ranking π b. (0)

(0)

3: Initialize item scores si ← 0 for all i ∈ [n], annotator parameters γu 4: for t = 1, 2, . . . until convergence do 5: Update score estimates:  

← 1 for all u ∈ [m].

s(t) ← s(t−1) + ηs ∇s ℓ s(t−1) , γ (t−1) .

Identifiability: s(t) = (I − 11⊤ /n)s(t) . Update annotator parameters:   (t−1) (t−1) (t) + ηγ ∇γu ℓ s(t−1) , {γu } for all u ∈ [m]. γu ← γu P (t) (t) 1 8: (Optionally, project {γu } onto its feasible set, e.g., m u γu = 1.) 9: end for 10: s∗ ← s(tfinal ) 11: π b ← rank order(s∗ ) 12: return π b. 6: 7:

2.3

Stage-2: Assessment Calibration via Isotonic Regression

In Stage-2, we leverage the consensus ranking π b from Stage-1 to calibrate the model’s raw scores sp . The key idea is s respect the consensus ordering π to adjust the model’s scores so that the final outputs b b while remaining as faithful as possible to sp . We use the term calibration here to mean rescaling or aligning the model-generated scores with the human-derived ranking constraints. This notion of calibration differs from its conventional meaning in probability estimation or psychology: instead of aligning predicted probabilities with observed frequencies, here we enforce an ordinal consistency between sp and the consensus ranking. In practice, we implement this calibration by projecting sp onto the monotonicity constraint set defined by π b, as formalized in Eq. (2.1).

Preliminary: Isotonic regression. As shown in Eq. (2.1), the calibrated output can be obtained by solving an isotonic c that minimizes the squared error to sp . Equivalently, b regression problem: we seek s in M s is Pn the adjusted 2score vector b the solution to minx∈M (x − s ) , which is exactly the projection defined in Eq. (2.1). This optimization finds i p,i c i=1 the closest score vector to sp that does not violate the consensus ranking order. The solution to this isotonic regression is efficiently computed via the Pool-Adjacent-Violators (PAV) algorithm [17]. PAV iteratively scans through the scores sorted by π b to detect any adjacent pair that is out of order (i.e., violates the required non-decreasing condition), and fixes such violations by averaging the offending scores. This process repeats until no more violations remain. The result is the calibrated score vector b s, which by construction satisfies b sπb(1) ≤ b sπb(2) ≤ · · · ≤ b sπb(n) (i.e., it respects the consensus ranking). Notably, if the model’s score vector sp already happens to be perfectly consistent with π b, the isotonic regression will leave it unchanged. Otherwise, the model’s scores are adjusted in a minimally invasive way (with respect to squared-error) to remove any ranking inconsistencies. The calibrated scores b s can be interpreted as a revised set of model-generated scores that combine the model’s quantitative estimates with the human-derived ordinal constraints. 4

Algorithm 2 Isotonic Regression Calibration 1: Input: Model score vector sp = (sp,1 , . . . , sp,n ); consensus ranking π b over n items. 2: Output: Calibrated score vector b s = (b s1 , . . . , b sn ) that is monotonic w.rt. π b. b (so that index 1 corresponds to π b(1), etc.). Let (y1 , . . . , yn ) be the 3: Relabel item indices of sp according to π

reordered scores, where y1 = sp,bπ(1) , y2 = sp,bπ(2) , ..., yn = sp,bπ(n) .

4: Initialize b si ← yi for i = 1, . . . , n. 5: repeat 6: for i = 1 to n − 1 do 7: if b si > b si+1 then b b 8: si , b si+1 ← si +b2si+1 9: end if 10: end for 11: until b s1 ≤ b s2 ≤ · · · ≤ b sn 12: Undo the reordering: for each item j, set b sj to the value of the calibrated score assigned to item j’s position in the

sorted order.

13: return b s.

3

Main Theoretical Results

As illustrated in Figure 1, AtC’s intuitive design offers strong theoretical guarantees for human-centered assessment across three perspectives: heterogeneity, robustness, and optimality. In this section, we present the main results, with related work discussions in Section 5 and detailed assumptions and proofs in Appendix A, B, and C. 3.1

Heterogeneous Efficiency Guarantee

We now demonstrate the efficiency advantage of the HTM estimator over an estimator derived from a misspecified homogeneous model. When annotators exhibit varying levels of reliability, a correctly specified HTM is expected to yield score estimates with lower asymptotic variance compared to a homogeneous model that incorrectly assumes uniform annotator reliability. The following lemmas formalize the asymptotic covariance structures of the two estimators. shomo is the QMLE for the homogeneous model with Specifically, θbhete = (b shete , γ bhete ) is the MLE for the HTM. b fixed γ0 = 1. To establish the asymptotic properties of the estimators b shete and b shomo , we assume: Assumption 3.1 (Independent Observations). The pairwise comparison outcomes Yul are independent conditional on the true parameters. Assumption 3.2 (True Data Generating Process). The data are generated by a heterogeneous Thurstone model (HTM) with true parameters θ0 = (s∗ , γ ∗ ) ∈ Rn+m . To ensure identifiability of s∗ , we impose the constraint 1⊤ s∗ = 0, ∗ ⊤ where 1 denotes the all-ones vector. The annotator-specific parameter vector γ ∗ = (γ1∗ , . . . , γm ) contains at least two distinct values. The link function F (·) is assumed to be known and twice continuously differentiable. Assumption 3.3 (Regularity Conditions). Both the HTM and the homogeneous model satisfy the standard regularity conditions in [76, Assumptions 1–6], which ensure the consistency and asymptotic normality of the MLE and QMLE, respectively. These conditions include compactness of the parameter space, measurability and smoothness of the log-likelihood functions, existence of the required expectations, unique identifiability of the parameters (or QMLE limits), and non-singularity of the relevant matrices on the identifiable parameter subspace. Such conditions are typically satisfied by common link functions, such as the logistic CDF and the Gaussian CDF, provided that the parameter space is well-defined and the comparison design is non-degenerate. Furthermore, for the correctly specified HTM, of [76, Assumption 7] is assumed to hold at θ0 , ensuring the information matrix equivalence. Lemma 3.4. Under conditions in Assumption 3.1, 3.2 and 3.3, the score vector estimator b shete obtained from the MLE of the HTM has an asymptotic covariance matrix given by: + 1 −1 Σbshete = Iss − Isγ Iγγ Iγs , (3.1) N Pm where N = u=1 ku is the total number of observations. The matrices Iss , Iγγ and Isγ are the Fisher information −1 sub-matrices for s and γ. (Iss − Isγ Iγγ Iγs )+ is the Schur complement of the full Fisher information with respect to s, representing the effective Fisher information for s after accounting for the estimation of γ. The pseudoinverse addresses the identifiability constraint 1⊤ s = 0. 5

Lemma 3.5. Under conditions in Assumption 3.1, 3.2 and 3.3, consider a homogeneous Thurstone model assuming a known constant annotator accuracy γ0 . The quasi-MLE b shomo , derived from this potentially misspecified model, has an asymptotic covariance matrix given by the classic sandwich form:  1 Σbshomo = (3.2) Ahomo (s∗ )+ Bhomo (s∗ )Ahomo (s∗ )+ , N where s∗ is the probability limit of b shomo . The matrix Ahomo (s∗ ) is the negative of the expected Hessian and Bhomo (s∗ ) is the expected outer product of the scores from the misspecified model’s log-likelihood, with expectations taken under the true heterogeneous data-generating process. The pseudoinverse also addresses identifiability. Lemmas 3.4 and 3.5 allow us to formally compare the asymptotic efficiency of the two estimation strategies. The following theorem states that the estimator from the correctly specified HTM is asymptotically superior. The proof is given in Appendix A.3 and A.4. Theorem 3.6. (Heterogeneous Efficiency Guarantee). Let Σbshete and Σbshomo be the asymptotic covariance matrices of the score estimators as defined in Lemmas 3.4 and 3.5. Then, Σbshete ⪯ Σbshomo,

(3.3)

where ⪯ denotes the Loewner order. This implies that the matrix Σbshomo − Σbshete is positive semi-definite. Furthermore, if the true annotator accuracies γ ∗ are not all equal (i.e., genuine heterogeneity exists), then this inequality is strict within the identifiable subspace, meaning Σbshomo − Σbshete is positive definite on this subspace. Proof Sketch. The proof compares the asymptotic covariance of two estimators under the true heterogeneous Thurstone data-generating process. For the correctly specified heterogeneous model, the MLE (b shete , γ bhete ) is asymptotically normal and achieves the Cramér–Rao lower bound. In Lemma 3.4, since the score parameters s and the user-specific accuracy parameters γ are estimated jointly, the effective information for s is given by the Schur com−1 plement S = Iss − Isγ Iγγ Iγs , and the asymptotic covariance of b shete on the identifiable subspace {v : 1⊤ v = 0} + is S . While in Lemma 3.5, the homogeneous estimator b shomo is a QMLE under model misspecification, because it ignores genuine user-level heterogeneity in γu . By White’s misspecification theory, it converges to a pseudo-true parameter s∗ rather than the true score vector s∗ in general, and its asymptotic covariance takes the sandwich form + A+ homo Bhomo Ahomo , instead of the inverse Fisher information. The failure of the information matrix equality under misspecification is precisely what creates the efficiency gap. Comparing these two covariance expressions shows that the misspecified homogeneous estimator cannot be more efficient than the correctly specified heterogeneous estimator in the Loewner order. When the annotator accuracies are genuinely non-identical, this dominance is strict on the identifiable subspace, which yields the theorem. The full proof is in Appendix A.5. Theorem 3.6 demonstrates that when annotator heterogeneity is present, the correctly specified HTM yields a strictly more efficient estimator of the true scores than the misspecified homogeneous model. Explicitly modeling annotatorspecific differences provides a demonstrable advantage in estimation precision. 3.2

Robustness Guarantee under Model Misspecification

We next analyze how errors from the Stage-1 rank aggregation affect the final calibrated output in Stage-2: an upper bound on the expected squared error E[∥b s −e s∥22 ]. This bound quantifies the performance of the two-stage procedure, c aiming to estimate the where b s is the projection of model predictions sp onto the human-consensus-derived cone M, c subjective optimal scores e s. Our analysis accounts for the misspecification of M and the systematic bias ν in sp relative to e s. We first establish a sharp oracle inequality for least squares estimators, leveraging a lemma adapted from [5, Proposition 2.1]. This result bounds the estimation error based on the effective noise properties and the constraint set’s geometry. Lemma 3.7 (Oracle Inequality with Biased Effective Noise). Let e s ∈ Rn be the target score vector (itself random, c ⊂ Rn be a closed convex set. The observations for projection are sp , and the effective noise with E[e s] = s). Let M c relating sp to e s is ξ = sp − e s ∼ N (ν, σ e2 In ). The least squares estimator is b s = ΠM c(sp ). For any u ∈ M, the squared estimation error satisfies, almost surely: !2 1 2 2 T ∥b s−e s∥2 − ∥u − e s∥2 ≤ sup ξ θ , (3.4) n θ∈TM,u :∥θ∥2 ≤1 c c at u. where TM,u is the tangent cone to M c 6

Taking expectations and choosing u = ΠM s) leads to a decomposition of the total expected risk into three principal c(e components:  2      2e σ 2 2 2 e E[∥b s−e s∥22 ] ≤ EM,e ∥Π (e s ) − s ∥ + E + E (3.5) δ(T ) ∥Π (ν)∥ TM,Π cs c cs cs c c (e 2 2 . c M M,e M,e M,Π s) s) c (e M n n M | {z } | {z } | {z } Projection Error:=E 1

Statistical Error:=E2

Bias Error:=E3

c Term E2 arises from the zero-mean Term E1 reflects errors from projecting e s onto the potentially incorrect cone M. component of ξ, where δ(·) is the statistical dimension of the tangent cone. Term E3 captures error due to the systematic bias ν (details are in Appendix B.1). Further bounding these components yields our main robustness theorem. Theorem 3.8. (Robustness Guarantee). The total expected squared error of b s for e s is bounded by: 2e σ 2 (ln(n) + γE + O(1/n)) 2∥ν∥22 + . (3.6) n n Here, E[Inv(b π, π e)] is the expected ranking inversions from Stage-1. σ e2 is the variance of e s, ν is the fixed systematic bias, and γE is the Euler-Mascheroni constant. E[∥b s−e s∥22 ] ≤ Ees [nVar(e s)] E [Inv(b π, π e)] +

Proof Sketch. The key idea is to view Stage-2 calibration as least-squares projection from sp onto the random isotonic c induced by the Stage-1 ranking π cone M b. Since the calibration target is e s while the model output is sp = e s + ν − ϵe, the effective noise is ξ = sp − e s ∼ N (ν, σ e2 In ), which is both biased and centered around a random cone constraint. We start from an oracle inequality for least-squares projection onto a closed convex set in Lemma 3.7. Taking expectations c (2) a then decomposes the risk into three parts: (1) the projection error caused by using a misspecified cone M, statistical term controlled by the statistical dimension of the tangent cone, and (3) a bias term induced by the systematic offset ν. The first term is governed by the probability that Stage-1 produces an incorrect ordering, which can be related to the expected number of inversions. The second term is bounded by the statistical dimension of an isotonic cone, which scales as the harmonic number Hn = log n + O(1). The third term is bounded by non-expansiveness of Euclidean projection. Combining these bounds yields the theorem. The full proof is in Appendix B.3. Remark. The implications in Theorem are two-fold. Firstly, Theorem 3.5 shows that the robustness of AtC is controlled by three qualitatively different error sources. The first is structural: if Stage-1 returns a ranking close to the target ordering, then the projection error remains small, whereas severe ranking mistakes directly enlarge the first term through the expected inversion count. The second is statistical: even with the correct cone, calibration still incurs uncertainty from the zero-mean component of the effective noise, but this complexity is mild because isotonic projection only pays the statistical dimension of the tangent cone. The third is systematic: a nonzero bias in the model scores cannot be removed for free, yet its contribution is still controlled and decays as 1/n after projection. Next, this theorem also clarifies why our setting is more delicate than standard isotonic regression. Classical analyses typically assume a fixed cone and zero-mean noise relative to the target [6, 9, 88], whereas in AtC both assumptions fail: the cone is random because it is inferred from Stage-1, and the effective noise is biased because the model scores may contain systematic distortion. The theorem shows that isotonic calibration remains well-behaved even under these two departures, which is the regime needed for human-centered assessment. The first term in the bound of Theorem 3.8, which accounts for errors due to Stage-1 ranking inaccuracies, is directly related to the probability of the estimated ranking π b differing from the subjective optimal ranking π e. This probability can be further bounded as follows: Corollary 3.9 (Bound on Ranking Misspecification Probability). For any two items j, k such that e sj < e sk , let ∆kj = e sk − e sj be the true score gap. Let Σbs denote the covariance matrix of the estimation error b s −e s in Lemma 3.4, 2 and σX = (ej − ek )⊤ Σbs (ej − ek ). Let Ac be the event Ac := {b π ̸= π e}. The probability that the estimated ranking jk c f π differs from the subjective optimal ranking π e (i.e., M ̸= M) is bounded by: ! X σXjk (∆kj )2 c √ δ1 := P (A ) ≤ exp − . (3.7) 2 2 σX ∆kj 2π jk j,k: e s <e s j

k

Corollary 3.9 makes the first error term in Theorem 3.8 more explicit by translating the risk of cone misspecification into pairwise ranking inversion probabilities. In particular, it shows that the probability of using an incorrect isotonic cone can be controlled through the uncertainty of the Stage-1 estimator and the pairwise score gaps of the target ordering. Besides, it also serves as a technical ingredient for later analysis in Theorem 3.12. It provides a tractable way to upper bound the contribution of Stage-1 ranking errors, which will be invoked in the subsequent proof when establishing conditions under which the two-stage AtC pipeline yields an improvement over the raw predictor. 7

3.3

Optimality Guarantee

Finally, we establish that the AtC yields a calibrated output that exceeds the uncalibrated model’s output in accuracy with high probability. Intuitively, if the Stage-1 aggregation produces an almost-correct ordering of items and Stage-2 calibration correctly removes the bias, then the final result will be closer to the ground truth than using the original σXjk model predictions alone. Recall from Corollary 3.9 shows that δ1 decays exponentially in the gap-to-noise ratio ∆kj for each pair. If every true gap is not too small compared to the associated noise, then δ1 becomes negligible. The following proposition shows that Stage-1 produces the correct order of items with high probability. σmax Proposition 3.10. Let ∆min = min ∆kj and σmax = max σXjk . If ≤ ε for some sufficiently small ε > 0 j<k j<k ∆min (equivalently, the Stage–1 sample size satisfies mk ≫ 1/(∆min ε)2 ), then δ1 = o(1); i.e. the estimated ranking space c will equal the true ranking space M. M At the same time, Stage-2 may still fail if the projection is badly perturbed by the noise ϵe ∼ N (0, σ e2 In ). Let’s define f the event B = {M = M}, similarly we have: Lemma 3.11. Let s ∈ Rn be the ground truth score and e s = s + ϵe, where ϵe ∼ N (0, σ e2 In ). Let B c be the event B c := {π(e s) ̸= π(s)}. Then we have,   X σ e ∆2 √ exp − 4eσkj2 . δ2 := P (B c ) ≤ (3.8) ∆kj π j,k: sk >sj

Lemma 3.11 shows that the chance of calibration error decays once the post–aggregation score gaps exceed the noise level σ. Because the pairwise comparisons used in Stage-1 are independent of the Gaussian perturbations ϵ in Stage-2, the events A, B in Corollary 3.9 and Lemma 3.11 are independent. Theorem 3.12. (Optimality Guarantee). Let δ1 and δ2 be as in Corollary 3.9 and Lemma 3.11. Then, with probability at least 1 − δ1 − δ2 , 2

b s−s 2 <

2

sp − s 2 .

(3.9)

i.e. the calibrated output b s is strictly closer to the ground truth s than the uncalibrated model prediction sp . As the Stage–1 sample size grows and ∆min dominates the noise, both δ1 and δ2 approach 0, so the probability of improvement approaches 1. Proof Sketch. The proof is based on a simple projection geometry argument combined with a decomposition of ranking errors across the two stages. Previously, we define A is the event that Stage-1 recovers the correct ranking for its own target, while B is the event that the subjective target e s preserves the true ranking of s. On the intersection c Once s belongs to the A ∩ B, the estimated isotonic cone coincides with the true cone induced by s, and hence s ∈ M. projection set, the improvement claim follows from the Pythagorean identity for Euclidean projection onto a closed c the first term on the right-hand side is convex set ∥sp − s∥22 ≥ ∥sp − b s∥22 + ∥b s − s∥22 . Therefore, whenever sp ∈ / M, 2 2 strictly positive, which implies ∥b s − s∥2 < ∥sp − s∥2 . It remains to lower bound the probability of this favorable event. A union bound gives P (A ∩ B) ≥ 1 − δ1 − δ2 . Therefore, with probability at least 1 − δ1 − δ2 , projection onto the estimated cone yields a strictly better estimate than the raw predictor. The full proof is in Appendix C. Theorem 3.12 confirms that, once Stage-1 has gathered enough pairwise comparisons to rank items reliably, the monotone projection in Stage-2 invariably drives the (possibly biased) model scores toward the ground truth. In effect, AtC protects against misspecification: even if both aggregated consensus and predictive model are imperfect, enforcing the human–derived rank guarantees a strictly better estimator with high probability. Overall, in this section we establishes the theoretical foundation of AtC from three complementary perspectives: efficiency, robustness, and optimality. Theorem 3.6 shows that heterogeneous judgment aggregation is statistically preferable to homogeneous modeling when annotator reliability truly varies. Theorem 3.8 shows that isotonic calibration remains robust under an estimated ranking constraint and biased effective noise. Theorem 3.12 further demonstrates that these two ingredients combine to yield a strict improvement over the raw predictor with high probability. Together, these results formalize the central intuition behind AtC: reliable ordinal information from human judgments can be aggregated and then converted, through calibration, into more accurate final scores.

4

Experiments

We evaluate AtC on both semi-synthetic and real-world datasets, guided by 6 research questions: 8

• (RQ1) Does AtC’s output b s outperform human-only assessment s∗ and model-only assessment sp ? • (RQ2) Do heterogeneous rank aggregation methods outperform homogeneous ones, and is AtC’s design of using the ranking π b from s∗ rather than directly using the s∗ values correct? • (RQ3) Is AtC robust to aggregation errors? • (RQ4) Is AtC robust to model assessments (sp ) errors? • (RQ5) Does AtC’s assumption that rankings are more reliable than ratings hold true in practice? • (RQ6) How well does AtC perform on a real-world task? Table 1: Semi-synthetic results on Reading Level dataset.

s), Bold indicate the best performance among human-only assessment (s∗ ), model-only assessment (sp ), and AtC assessment (b respectively. The red denotes the optimal performance across all methods for the given metric. Stage-1 Method

Kendall τ ↑ ∗

Wasserstein↓ ∗

KS↓ ∗

MSE↓ ∗

s / sp / b s

s / sp / b s

s / sp / b s

s / sp / b s

0.375 / 0.399 / 0.410 0.375 / 0.399 / 0.403 0.368 / 0.399 / 0.399

2.250 / 2.831 / 0.839 2.243 / 2.831 / 0.827 2.351 / 2.831 / 0.738

0.500 / 0.300 / 0.163 0.498 / 0.300 / 0.163 0.563 / 0.300 / 0.192

8.658 / 29.00 / 8.122 8.658 / 29.00 / 8.191 8.985 / 29.00 / 7.919

CrowdBT 0.354 / 0.399 / 0.399 CrowdTCV 0.339 / 0.399 / 0.372

2.150 / 2.831 / 0.843 2.272 / 2.831 / 1.015

0.455 / 0.300 / 0.269 0.506 / 0.300 / 0.312

8.301 / 29.00 / 7.555 8.689 / 29.00 / 7.809

0.340 / 0.399 / 0.373 0.338 / 0.399 / 0.373

2.186 / 2.831 / 0.894 2.343 / 2.831 / 0.914

0.461 / 0.300 / 0.300 0.547 / 0.300 / 0.300

7.764 / 29.00 / 8.097 8.943 / 29.00 / 8.030

HRA-G HRA-E HRA-N

BTL TCV

Datasets. We consider two datasets in our experiments. (1) Reading-level (semi-synthetic dataset) [12] contains pairwise comparisons of text documents based on reading difficulty. 490 documents with known reading levels, 624 annotators providing 12,728 pairwise judgments. (2) Dots-activity (real-world dataset) [41], a benchmark for collective judgment aggregation. The task involves estimating the number of dots in images. The dataset contains two types judgments (ranking and rating) from 300 participants on 30 distinct images, which yield 8700 pairwise comparisons. Baselines and Metrics. For the semi-synthetic dataset, we compare 7 different aggregation methods: 3 HRA variants based on different noise distribution assumptions (HRA-G/E/N), 2 heterogeneous methods (CrowdBT/TCV) [12], and 2 homogeneous baselines (BTL/TCV) [7]. For the real-world dataset, we additionally compare against 3 human-centered assessment methods: GPPL [13], Rank-SVM [40], and BARCW [50]. We evaluate using 4 metrics: Kendall τ measures ranking accuracy; Wasserstein distance and KS statistic quantify distributional alignment between predicted and ground-truth scores; MSE captures absolute prediction error. Detailed descriptions are provided in Appendix D. Semi-Synthetic Evaluation. We first evaluate on semi-synthetic setting where ground-truth item scores are available. We simulate m annotators who provide pairwise comparisons, apply various Stage-1 aggregation methods (w/ or w/o modeling annotator heterogeneity) to obtain a consensus score s∗ , and then train a predictive model to produce an initial model score sp for each item (using a portion of the ground truth with added noise to emulate model error). We calibrate sp via AtC to obtain the output b s. In Table 1, we observe that b s consistently outperforms both s∗ and sp across nearly all metrics and methods (the b s entries are almost always bolded as best).3 Figure 2 further illustrates this improvement: the b s values align much more closely with the true distribution of s, whereas the uncalibrated s∗ and sp scores deviate significantly (same conclusion for all methods in Appendix D.4). To verify that this improvement stems from sp rather than simple rescaling, we conducted experiments matching s∗ ’s range to ground truth (Appendix D.5). In Figure 2: Score Distribution of summary, the AtC score dominates the human-only and model-only assessment, HRA-E confirming that combining the two sources produces more accurate judgments (answering RQ1). We examine the effect of modeling annotator heterogeneity on the consensus ranking quality. In Table 1, the methods that account for annotator heterogeneity (HRA-G/E/N, and CrowdBT/TCV) produce better (or the best) calibrated 3

The Kendall’s τ values for b s and s∗ differ due to tie-creation operation; see Appendix D.3 for details.

9

s than the homogeneous models (BTL, TCV). Furthermore, we observe that for s∗ , the heterogeneous HRA scores b models achieve more accurate rankings (higher Kendall τ ), despite performing worse on distance metrics compared to 4 baselines. Notably, after calibration with sp , the resulting b s scores from heterogeneous models surpass 4 baselines across Wasserstein distance and KS statistic, confirming that our design choice to prioritize ranking information from s∗ is indeed effective (answering RQ2). Figure 3 shows AtC’s robustness to ranking errors in s∗ . Our normalized results show that as we introduce pairwise judgment inversions, AtC degrades gradually until a critical threshold. With moderate noise (up to 400 inversions), Kendall τ decreases steadily while MSE increases, but AtC still produces meaningful outputs by leveraging model signals. However, beyond approximately 500 inversions, performance collapses completely—Kendall τ becomes negative, variance approaches zero, and the output distribution flattens (shown by declining block count). This synchronized degradation across all metrics confirms that while AtC can tolerate considerable noise in consensus rankings, extremely corrupted inputs will eventually render the calibration ineffective (answering RQ3). Real-World Evaluation. We evaluate the validity and robustness of the AtC framework on the Dots-activity dataset, a benchmark for counting dots in images. We assume models can only process corrupted images due to various constraints, while humans can still make reliable inferences based on experience, making this a valuable human-centered assessment problem. We introduce various image Figure 3: Robustness Analysis corruptions to simulate real-world scenarios with noisy or partially occluded data, then generate predictive scores sp using OpenCV’s contour detection [68] as an imperfect predictor. As illustrated in Figure 4, we apply 4 corruption types [30]: global Gaussian blur and three localized corruptions (blur, noise, and whiteout), with degradation intensity controlled by a hyperparameter to evaluate performance across different corruption levels.

(a) Original

(b) BLUR

(c) LOCAL BLUR

(d) LOCAL NOISE

(e) WHITEOUT

Figure 4: Examples of image corruption types applied to the Dots-activity dataset. Table 2 summarizes performance across AtC and 3 human-centered assessment baselines (GPPL, Rank-SVM, BARCW); comprehensive results including all Stage-1 methods appear in Figure 5, 6, and 7 presents the core robustness results for HRA-E, HRA-G and HRA-N, with similar trends for other baseline methods shown in Appendix D.7. The radar charts intuitively visualize the degradation of Kendall’s Tau correlation as damage intensity increases (clockwise from the top axis). Each chart compares the initial model (sp ), the uncalibrated human consensus (s∗ ), and the calibrated AtC outputs based on rankings (b s (Rank)) and ratings (b s (Rate)). Across the 4 types of corruption, our proposed primary method b s (Rank) demonstrates remarkable resilience. While the performance of the raw objective model sp degrades sharply with increasing noise, particularly under localized corruptions. This shows that by anchoring the model’s scores to the stable structure provided by human consensus, AtC effectively mitigates the impact of noise on the predictor (answering RQ4). Furthermore, these results underscore the superiority of using ordinal (ranking) information for calibration. In every scenario, b s (Rank) outperforms b s (Rate), the alternative calibrated using aggregated cardinal ratings. This suggests that the consensus ranking (s∗ ) provides a more robust calibration framework than aggregated numerical estimates, which are susceptible to individual biases and inconsistent scales (answering RQ5). In summary, the AtC framework demonstrates strong real-world performance by maintaining high accuracy even under severe input degradation (answering RQ6).

5

Additional Related Work

Judgement Aggregation and Rank Aggregation. Inferring a consensus from multiple noisy annotators is a common problem in crowdsourcing and psychometrics [36, 54, 61]. Classic models, like Dawid & Skene’s [16] model estimate a 10

Table 2: Real-world results on Dots dataset. Stage-1 Method

Kendall τ ↑

Wasserstein↓

KL↓

MSE↓

s / sp / b s

∗

s / sp / b s

∗

s / sp / b s

s∗ / sp / b s

HRA-G AtC HRA-E HRA-N

0.917 / 0.923 / 0.940 0.922 / 0.922 / 0.943 0.917 / 0.923 / 0.934

6.97 / 2.53 / 2.53 6.98 / 2.53 / 2.53 7.05 / 2.53 / 2.53

123.13 / 0.881 / 0.860 123.16 / 0.881 / 0.861 125.50 / 0.881 / 0.859

65.08 / 11.75 / 9.61 65.16 / 11.75 / 9.59 66.44 / 11.75 / 9.75

GPPL Rank-SVM BARCW

– / – / 0.931 – / – / 0.923 – / – / 0.940

– / – / 64.50 – / – / 61.20 – / – / 64.62

– / – / 126.62 – / – / 133.70 – / – / 24.09

– / – / 4220.36 – / – / 3814.49 – / – / 4236.16

1.5

1.0

2.0

0.5 2.5 0.6

0.7

0.8

0.9

∗

s (Rank) s (Rate) sp s*

1.0

0.4 0.5

0.6

0.0 5.0

0.8

0.65

(a) BLUR

0.7

0.4

s (Rank) s (Rate) sp 0.25 s *

0.9

0.35

0.45

1.0

0.5

0.2 0.8

0.6

4.5

4.0

0.7

0.3

0.55

3.0 3.5

0.35

0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.8

0.6 0.65

(b) LOCAL BLUR

0.7

0.35

0.5

0.2

0.55

0.75

1.0

0.4 0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.2

0.55

0.8

0.6

0.75

0.65

(c) LOCAL NOISE

1.0

0.7

0.75

(d) WHITEOUT

Figure 5: Radar plots under different noise conditions (HRA-E).

1.5

1.0

2.0

0.5 2.5 0.6

0.7

0.8

0.9

s (Rank) s (Rate) sp s*

1.0

0.4 0.5

0.6

0.0 5.0

0.8

0.65

(a) BLUR

0.7

0.4

s (Rank) s (Rate) sp 0.25 s *

0.9

0.35

0.45

1.0

0.5

0.2 0.8

0.6

4.5

4.0

0.7

0.3

0.55

3.0 3.5

0.35

0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.8

0.6 0.65

(b) LOCAL BLUR

0.7

0.35

0.5

0.2

0.55

0.75

1.0

0.4 0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.2

0.55

0.8

0.6

0.75

0.65

(c) LOCAL NOISE

1.0

0.7

0.75

(d) WHITEOUT

Figure 6: Radar plots under different noise conditions (HRA-G).

1.5

1.0

2.0

0.5 2.5 0.6

0.7

0.8

0.9

1.0

0.0

3.0 5.0 3.5 4.0

4.5

(a) BLUR

s (Rank) s (Rate) sp s*

0.4

0.35

0.45 0.5 0.6

0.7

0.3 0.8

0.9

1.0

0.8 0.65

0.7

0.35

0.45 0.5

0.2

0.55 0.6

0.4

s (Rank) s (Rate) sp 0.25 s *

0.6

0.7

0.3 0.8

0.9

0.8

0.6 0.65

(b) LOCAL BLUR

1.0

0.7

0.4 0.5

0.6

0.7

0.3 0.8

0.75

Figure 7: Radar plots under different noise conditions (HRA-N).

s (Rank) s (Rate) sp 0.25 s *

0.9

1.0

0.2

0.55

(c) LOCAL NOISE

11

0.35

0.45

0.2

0.55

0.75

s (Rank) s (Rate) sp 0.25 s *

0.8

0.6 0.65

0.7

0.75

(d) WHITEOUT

latent true label or score while capturing annotator error rates, with extensions incorporating bias and expertise. Ranking models, such as Bradley–Terry or Thurstone–Mosteller variants, account for heterogeneous annotator reliability [62]. Our Stage-1 algorithm accounts for rater heterogeneity to estimate the consensus ranking s∗ , which is supported by Theorem 3.6 and experiments, outperforms homogeneous annotator assumptions. Unlike prior methods focused solely on consensus, AtC uses it as an ordinal scaffold for predictive model calibration, enhancing downstream efficiency with formal guarantees on predictive performance. Model Calibration and Isotonic Regression. Calibrating model outputs to better align with empirical outcomes is a common practice in machine learning, especially for probabilistic predictions. Isotonic regression in particular is a non-parametric technique that enforces output monotonicity and has been used to calibrate scores to observed binary labels or probabilities [4, 57]. In AtC, we novelly apply isotonic regression to calibrate a continuous model predictions to a rank-based target derived from human judgments. This usage differs from standard calibration, as we are mapping model scores to an ordering rather than absolute ground-truth values. Prior work on isotonic regression and calibration [5, 9, 67] has not, to our knowledge, combined isotonic regression with a crowd-sourced ranking constraint. Theorem 3.8 provides calibration error analysis with imperfect rankings, demonstrating AtC’s ability to effectively calibrate using inferred consensus rather than fixed ground-truth labels. Unlike [5], our innovation lies in modeling subjective optimum e s as a random variable with allowable non-zero bias ν, enabling monotonic calibration when only relative judgments are available. We choose L2 projection instead of L1 or L∞ alternatives because it corresponds to maximum likelihood estimation under Gaussian noise and ensures unique solutions due to strict convexity [66]. Human-AI Complementarity. Our work is also related to methods that integrate human and machine decision-making to improve prediction quality. Such approaches seek to optimally fuse human labels and algorithmic learning [45, 64, 65, 84]. Notably, calibration via human judgment has emerged as a practical entry point for real-world assessment, such as logistics area difficulty assessment [78]. AtC extends this line of work in two ways: it generalizes the framework beyond domain-specific settings, and provides an optimality theorem (Theorem 3.12) that explains when and why the two-stage pipeline outperforms either component alone, which gap is common across prior human-in-the-loop methods [2, 81]. Unlike [50, 62], which impose linear model constraints, AtC places no restrictions on the aggregation method or the predictive model, making the design modular and broadly applicable.

6

Conclusion

We introduced AtC, a two-stage framework that addresses the fundamental challenge of human-centered evaluation when ground truth is costly, unobservable, or future-dependent. Our theoretical analysis establishes three key results: (1) heterogeneous annotator modeling yields more efficient consensus estimation than homogeneous assumptions, (2) isotonic calibration provides risk bounds even under misspecified rankings, and (3) AtC asymptotically outperforms model-only assessment. Empirically, AtC demonstrates consistent improvements in accuracy and robustness across semi-synthetic and real-world datasets, particularly under data degradation conditions common in practice. The framework suits human-centered assessment tasks where model-only evaluation faces limitations due to unobservable or costly ground truth, such as when models receive degraded inputs while humans access complete information, or when collecting human comparisons is more cost-effective than measuring ground truth directly. Future work can extend AtC in several directions: multi-dimensional assessments in which items are judged along multiple criteria, and domains with expert polarization that require multiple distinct consensus rankings. Another direction is to extend our formulation to LLM-as-a-judge settings, where generalizing the heterogeneous annotator to language model evaluators would help build automated and adaptive assessment systems. We believe that addressing these challenges will broaden the impact of collaborative human-AI assessment systems.

12

Statement This paper is the extended version of [79], incorporating additional content that could not be included in the conference submission due to page limitations. We release code and datasets at https://github.com/CAMELLIAxt/12500_ AtC_supp.

References [1] Samin Nili Ahmadabadi, Maryam Haghifam, Vahid Shah-Mansouri, and Sara Ershadmanesh. Design and evaluation of crowdsourcing platforms based on users’confidence judgments. Scientific Reports, 14(1):18379, 2024. doi: 10.1038/s41598-024-65892-7. URL https://doi.org/10.1038/s41598-024-65892-7. [2] Rohan Alur, Manish Raghavan, and Devavrat Shah. Human expertise in algorithmic prediction. In The Thirtyeighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/ forum?id=wpGJ2AX6SZ. [3] Dennis Amelunxen, Martin Lotz, Michael B McCoy, and Joel A Tropp. Living on the edge: Phase transitions in convex programs with random data. Information and Inference: A Journal of the IMA, 3(3):224–294, 2014. [4] Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. Stop measuring calibration when humans disagree. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1892–1915, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.124. URL https://aclanthology.org/2022.emnlp-main.124/. [5] Pierre C Bellec. Sharp oracle inequalities for least squares estimators in shape restricted regression. The Annals of Statistics, 46(2):745–780, 2018. [6] Pierre C. Bellec and Alexandre B. Tsybakov. Sharp oracle bounds for monotone and convex regression through aggregation. Journal of Machine Learning Research, 16(56):1879–1892, 2015. URL http://jmlr.org/ papers/v16/bellec15a.html. [7] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URL http://www.jstor.org/ stable/2334029. [8] Bryan Brancotte, Bo Yang, Guillaume Blin, Sarah Cohen-Boulakia, Alain Denise, and Sylvie Hamel. Rank aggregation with ties: experiments and analysis. Proc. VLDB Endow., 8(11):1202–1213, July 2015. ISSN 2150-8097. doi: 10.14778/2809974.2809982. URL https://doi.org/10.14778/2809974.2809982. [9] Sabyasachi Chatterjee, Adityanand Guntuboyina, and Bodhisattva Sen. On risk bounds in isotonic and other shape restricted regression problems. The Annals of Statistics, 43(4):1774–1800, 2015. ISSN 00905364. URL http://www.jstor.org/stable/43556660. [10] Sujoy Chatterjee, Anirban Mukhopadhyay, and Malay Bhattacharyya. A review of judgment analysis algorithms for crowdsourced opinions. IEEE Transactions on Knowledge and Data Engineering, 32(7):1234–1248, 2020. doi: 10.1109/TKDE.2019.2904064. [11] Siu Lun Chau, Javier Gonzalez, and Dino Sejdinovic. Learning inconsistent preferences with gaussian processes. In International Conference on Artificial Intelligence and Statistics, pp. 2266–2281. PMLR, 2022. [12] Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, WSDM ’13, pp. 193–202, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450318693. doi: 10.1145/2433396.2433420. URL https://doi.org/10.1145/2433396.2433420. [13] Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, pp. 137–144, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805. doi: 10.1145/1102351.1102369. URL https://doi.org/10. 1145/1102351.1102369. [14] Bo Cowgill. Bias and productivity in humans and algorithms: Theory and evidence from resume screening. Columbia Business School, Columbia University, 29, 2020. [15] Robyn M. Dawes, David Faust, and Paul E. Meehl. Clinical versus actuarial judgment. Science, 243(4899): 1668–1674, 1989. ISSN 00368075, 10959203. URL http://www.jstor.org/stable/1703476. 13

[16] A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20–28, 1979. ISSN 00359254, 14679876. URL http://www.jstor.org/stable/2346806. [17] Jan De Leeuw, Kurt Hornik, and Patrick Mair. Isotone optimization in r: pool-adjacent-violators algorithm (pava) and active set methods. Journal of statistical software, 32:1–24, 2010. [18] Parisa Dolati Neghabadi, Karine Evrard Samuel, and Marie-Laure Espinouse. Systematic literature review on city logistics: overview, classification and analysis. International Journal of Production Research, 57(3):865–887, 2019. [19] Richard L. Dykstra and Tim Robertson. An algorithm for isotonic regression for two or more independent variables. The Annals of Statistics, 10(3):708–716, 1982. ISSN 00905364. URL http://www.jstor.org/ stable/2240899. [20] Zhihan Fang, Guang Yang, Wenjun Lyu, Zhiqing Hong, Shuxin Zhong, Weijian Zuo, Yuelei Xie, Yu Yang, Guang Wang, Yunhuai Liu, et al. Cellular infrastructure sharing for network robustness: A citywide empirical study. IEEE Transactions on Mobile Computing, 2025. [21] Xinyue Feng, Shuxin Zhong, Jinquan Hang, Wenjun Lyu, Yuequn Zhang, Guang Yang, Haotian Wang, Desheng Zhang, and Guang Wang. Hierarchical structure sharing empowers multi-task heterogeneous gnns for customer expansion. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 4424–4434, 2025. [22] Xinyue Feng, Shuxin Zhong, Jinquan Hang, Yuequn Zhang, Guang Yang, Haotian Wang, Desheng Zhang, and Guang Wang. Neighsqueeze: Compact neighborhood grouping for efficient billion-scale heterogeneous graph learning. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pp. 5658–5665, 2025. [23] A. Fielding. Statistical Inference Under Order Restrictions. The Theory and Application of Isotonic Regression. Royal Statistical Society. Journal. Series A: General, 137(1):92–93, 12 2018. ISSN 0035-9238. doi: 10.2307/ 2345150. URL https://doi.org/10.2307/2345150. [24] William H Greene. Econometric analysis. Pretence Hall, 2003. [25] Baoshen Guo, Shuai Wang, Haotian Wang, Yunhuai Liu, Fanshuo Kong, Desheng Zhang, and Tian He. Towards equitable assignment: Data-driven delivery zone partition at last-mile logistics. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4078–4088, 2023. [26] Jinquan Hang, Zheng Dong, Hongke Zhao, Xin Song, Peng Wang, and Hengshu Zhu. Outside in: Market-aware heterogeneous graph neural network for employee turnover prediction. In WSDM ’22, pp. 353–362, Virtual Event, 2022. ACM. [27] Jinquan Hang, Zhiqing Hong, Xinyue Feng, Guang Wang, Dongjiang Cao, Jiayang Qiao, Haotian Wang, and Desheng Zhang. Complex-path: Effective and efficient node ranking with paths in billion-scale heterogeneous graphs. Proceedings of the VLDB Endowment, 17(12), 2024. [28] Jinquan Hang, Zhiqing Hong, Xinyue Feng, Guang Wang, Guang Yang, Feng Li, Xining Song, and Desheng Zhang. Paths2pair: Meta-path based link prediction in billion-scale commercial heterogeneous graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, New York, NY, USA, 2024. Association for Computing Machinery. [29] Patrick Hemmer, Max Schemmer, Niklas Kühl, Michael Vössing, and Gerhard Satzger. Complementarity in humanai collaboration: concept, sources, and evidence. European Journal of Information Systems, 34(6):979–1002, 2025. doi: 10.1080/0960085X.2025.2475962. URL https://doi.org/10.1080/0960085X.2025.2475962. [30] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=HJz6tiCqYm. [31] Chien-Ju Ho and Jennifer Vaughan. Online task assignment in crowdsourcing markets. In Proceedings of the AAAI conference on artificial intelligence, volume 26, pp. 45–51, 2012. [32] Qian Hong, Siyuan Chang, and Xiao Zhou. Wed-net: A weather-effect disentanglement network with causal augmentation for urban flow prediction. In Proceedings of the ACM Web Conference 2026, WWW ’26, pp. 9625–9633, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400723070. doi: 10.1145/3774904.3793059. URL https://doi.org/10.1145/3774904.3793059. [33] Qian Hong, Siyuan Chang, and Xiao Zhou. Wed-net: A weather-effect disentanglement network with causal augmentation for urban flow prediction. arXiv preprint arXiv:2601.22586, 2026. 14

[34] Zhiqing Hong, Guang Wang, Wenjun Lyu, Baoshen Guo, Yi Ding, Haotian Wang, Shuai Wang, Yunhuai Liu, and Desheng Zhang. Cominer: nationwide behavior-driven unsupervised spatial coordinate mining from uncertain delivery events. In Proceedings of the 30th International Conference on Advances in Geographic Information Systems, SIGSPATIAL ’22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450395298. doi: 10.1145/3557915.3560944. URL https://doi.org/10.1145/3557915.3560944. [35] Zhiqing Hong, Zelong Li, Shuxin Zhong, Wenjun Lyu, Haotian Wang, Yi Ding, Tian He, and Desheng Zhang. Crosshar: Generalizing cross-dataset human activity recognition via hierarchical self-supervised pretraining. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 8(2), May 2024. doi: 10.1145/3659597. URL https://doi.org/10.1145/3659597. [36] Robert Hu, Siu Lun Chau, Jaime Ferrando Huertas, and Dino Sejdinovic. Explaining preferences with shapley values. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=-me36V0os8P. [37] Shun Ito and Hisashi Kashima. Mitigating cognitive biases in multi-criteria crowd assessment, 2024. URL https://arxiv.org/abs/2407.18938. [38] Jiarui Jin, Ming Zhou, Weinan Zhang, Minne Li, Zilong Guo, Zhiwei Qin, Yan Jiao, Xiaocheng Tang, Chenxi Wang, Jun Wang, et al. Coride: joint order dispatching and fleet management for multi-scale ride-hailing platforms. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1983–1992, 2019. [39] Tao Jin, Pan Xu, Quanquan Gu, and Farzad Farnoud. Rank aggregation via heterogeneous thurstone preference models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4353–4360, 2020. [40] Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’02, pp. 133–142, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 158113567X. doi: 10.1145/775047.775067. URL https://doi.org/10.1145/775047.775067. [41] Ryan Kemmer, Yeawon Yoo, Adolfo Escobedo, and Ross Maciejewski. Enhancing collective estimates by aggregating cardinal and ordinal inputs. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pp. 73–82, 2020. [42] Urja Khurana, Eric Nalisnick, Antske Fokkens, and Swabha Swayamdipta. Crowd-calibrator: Can annotator disagreement inform calibration in subjective tasks? In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=VWWzO3ewMS. [43] Caitlin Kuhlman and Elke Rundensteiner. Rank aggregation algorithms for fair consensus. Proc. VLDB Endow., 13(12):2706–2719, jul 2020. ISSN 2150-8097. doi: 10.14778/3407790.3407855. URL https://doi.org/10. 14778/3407790.3407855. [44] Tinghao Lai, Yan Zhao, Weizhu Qian, and Kai Zheng. Loyalty-based task assignment in spatial crowdsourcing. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 1014–1023, 2022. [45] Jing Li, Rafał K. Mantiuk, Junle Wang, Suiyi Ling, and Patrick Le Callet. Hybrid-mst: A hybrid active sampling strategy for pairwise preference aggregation. In Neural Information Processing Systems, 2018. URL https://api.semanticscholar.org/CorpusID:53046957. [46] Jinlin Li and Xiao Zhou. Curegraph: Contrastive multi-modal graph representation learning for urban living circle health profiling and prediction. Artificial Intelligence, 340:104278, 2025. ISSN 0004-3702. doi: https: //doi.org/10.1016/j.artint.2024.104278. URL https://www.sciencedirect.com/science/article/pii/ S0004370224002145. [47] Jinlin Li and Xiao Zhou. Curegraph: Contrastive multi-modal graph representation learning for urban living circle health profiling and prediction. Artificial Intelligence, 340:104278, 2025. ISSN 0004-3702. doi: https: //doi.org/10.1016/j.artint.2024.104278. URL https://www.sciencedirect.com/science/article/pii/ S0004370224002145. [48] Jinlin Li, Xintong Li, and Xiao Zhou. Fap-cd: Fairness-driven age-friendly community planning via conditional diffusion generation. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27):28168–28176, Apr. 2025. doi: 10.1609/aaai.v39i27.35036. URL https://ojs.aaai.org/index.php/AAAI/article/view/ 35036. [49] Jinlin Li, Yuran Wang, Yifei Yuan, Xiao Zhou, Yingying Zhang, Xixian Yong, Yefeng Zheng, and Xian Wu. Beyond single models: Mitigating multimodal hallucinations via adaptive token ensemble decoding, 2025. URL https://arxiv.org/abs/2510.18321. 15

[50] Xinran Li, Dingdong Yi, and Jun S. Liu. Bayesian Analysis of Rank Data with Covariates and Heterogeneous Rankers. Statistical Science, 37(1):1 – 23, 2022. doi: 10.1214/20-STS818. URL https://doi.org/10.1214/ 20-STS818. [51] Xintong Li, Haoran Zhang, and Xiao Zhou. Spatio-temporal hierarchical causal models, 2025. URL https: //arxiv.org/abs/2511.20558. [52] Anjing Liu, Philip L. De Jager, David Bennett, The Alzheimer’s Disease Functional Genomics Consortium, Gao Wang, and William R. P. Denault. mfsusie enables multi-cell-type fine-mapping and multi-omic integration of chromatin accessibility qtls in aging brain. bioRxiv, 2025. doi: 10.1101/2025.11.25.690439. URL https: //www.biorxiv.org/content/early/2025/11/28/2025.11.25.690439. [53] Anjing Liu, Roulan Jiang, Ruixi Li, Xuewei Cao, Zining Qi, Ru Feng, Hao Sun, Masashi Fujita, Natacha Comandante-Lou, Chirag Lakhani, Jenny Empawi, David Knowles, Xiaoling Zhang, Kushal Dey, Philip Jager, David Bennett, The Consortium, Tianying Wang, and Gao Wang. Distributional genetic effects reveal contextdependent molecular regulation in human brain aging and alzheimer’s disease. Research square, 11 2025. doi: 10.21203/rs.3.rs-8219833/v1. [54] Ao Liu, Zhibing Zhao, Chao Liao, Pinyan Lu, and Lirong Xia. Learning plackett-luce mixtures from partial preferences. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):4328–4335, Jul. 2019. doi: 10.1609/aaai.v33i01.33014328. URL https://ojs.aaai.org/index.php/AAAI/article/view/4342. [55] Taichi Liu, Zhenyu Wang, Ruofeng Liu, Guang Wang, and Desheng Zhang. Towards 3d objectness learning in an open world. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=wEOmS8Aw1W. [56] Ziyang Lu, Yunqiang Pei, Guoqing Wang, Peiwei Li, Yang Yang, Yinjie Lei, and Heng Tao Shen. Scaneru: Interactive 3d visual grounding based on embodied reference understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3936–3944, 2024. [57] Ronny Luss, Saharon Rosset, and Moni Shahar. Efficient regularized isotonic regression with application to gene–gene interaction search. The Annals of Applied Statistics, 6(1):253 – 283, 2012. doi: 10.1214/11-AOAS504. URL https://doi.org/10.1214/11-AOAS504. [58] Wenjun Lyu, Haotian Wang, Zhiqing Hong, Guang Wang, Yu Yang, Yunhuai Liu, and Desheng Zhang. Rede: exploring relay transportation for efficient last-mile delivery. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 3003–3016. IEEE, 2023. [59] Wenjun Lyu, Xiaolong Jin, Haotian Wang, Yiwei Song, Shuai Wang, Yunhuai Liu, Tian He, and Desheng Zhang. Towards workload-constrained efficient order assignment in last-mile delivery. IEEE Transactions on Mobile Computing, 2024. [60] Wenjun Lyu, Shuxin Zhong, Guang Yang, Haotian Wang, Yi Ding, Shuai Wang, Yunhuai Liu, Tian He, and Desheng Zhang. Inco: Exploring inter-trip cooperation for efficient last-mile delivery. In Proceedings of the ACM on Web Conference 2025, pp. 5183–5191, 2025. [61] Thomas Brendan Murphy and Donald Martin. Mixtures of distance-based models for ranking data. Comput. Stat. Data Anal., 41:645–655, 2003. URL https://api.semanticscholar.org/CorpusID:42716709. [62] Yuangang Pan, Ivor W. Tsang, Weijie Chen, Gang Niu, and Masashi Sugiyama. Fast and robust rank aggregation against model misspecification. Journal of Machine Learning Research, 23(23):1–35, 2022. URL http: //jmlr.org/papers/v23/20-315.html. [63] Jose L. Pardo-Vazquez, Juan R. Castiñeiras-de Saa, Mafalda Valente, Iris Damião, Tiago Costa, M. Inês Vicente, AndréG. Mendonça, Zachary F. Mainen, and Alfonso Renart. The mechanistic foundation of weber’s law. Nature Neuroscience, 22(9):1493–1502, 2019. doi: 10.1038/s41593-019-0439-7. URL https://doi.org/10.1038/ s41593-019-0439-7. [64] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11(43):1297–1322, 2010. URL http://jmlr.org/papers/v11/raykar10a.html. [65] Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human-AI complementarity. Proc. Natl. Acad. Sci. U. S. A., 119(11):e2111547119, March 2022. [66] Quentin F. Stout. lp isotonic regression algorithms using an l0 approach, 2023. URL https://arxiv.org/abs/ 2107.00251. [67] Weijie Su. You are the best reviewer of your own papers: An owner-assisted scoring mechanism. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27929–27939. Curran Associates, Inc., 2021. 16

[68] Satoshi Suzuki and Keiichi Abe. Topological structural analysis of digitized binary images by border following. Computer Vision, Graphics, and Image Processing, 30(1):32–46, 1985. ISSN 0734-189X. doi: https: //doi.org/10.1016/0734-189X(85)90016-7. URL https://www.sciencedirect.com/science/article/ pii/0734189X85900167. [69] David Tran, Alexander V Valtchanov, Keshav R Ganapathy, Raymond Feng, Eric Victor Slud, Micah Goldblum, and Tom Goldstein. An open review of openreview: A critical analysis of the machine learning conference review process, 2021. URL https://openreview.net/forum?id=Cn706AbJaKW. [70] Di Wang, Jiahao Ding, Zejun Xie, Miao Pan, and Jinhui Xu. Differentially private (gradient) expectation maximization algorithm with statistical guarantees. CoRR, abs/2010.13520, 2020. URL https://arxiv.org/ abs/2010.13520. [71] Hai Wang, Shuai Wang, Yu Yang, and Desheng Zhang. Gcrl: Efficient delivery area assignment for last-mile logistics with group-based cooperative reinforcement learning. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 3522–3534. IEEE, 2023. [72] Pei Wang, Anlan Yu, Xujun Ma, Rong Zheng, Jingfu Dong, Zhaoxin Chang, Duo Zhang, Djamal Zeghlache, and Daqing Zhang. Wicg: In-body cardiac motion sensing based on a mix-medium wi-fi fresnel zone model. IEEE Transactions on Mobile Computing, 2025. [73] Shanshan Wang, Fan Wu, Jie Gao, Sijing Duan, Feng Lyu, Huaqing Wu, Yaoxue Zhang, and Xuemin Sherman Shen. Dynamic rrh-bbu mapping for c-ran: A data-driven approach. In GLOBECOM 2023 - 2023 IEEE Global Communications Conference, pp. 2656–2661, 2023. doi: 10.1109/GLOBECOM54140.2023.10437846. [74] Shanshan Wang, Sijing Duan, Shuxin Zhong, Zhiqing Hong, Zhiyuan Zhou, Hongyu Lin, Weijian Zuo, Desheng Zhang, and Yi Ding. Fraudulent delivery detection with multimodal courier behavior data in last-mile delivery. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, pp. 6110–6118, New York, NY, USA, 2025. Association for Computing Machinery. [75] Yao Wang, Zhaowei Wang, Zejun Xie, Nengwen Zhao, Junjie Chen, Wenchi Zhang, Kaixin Sui, and Dan Pei. Practical and white-box anomaly detection through unsupervised and active learning. In 2020 29th International Conference on Computer Communications and Networks (ICCCN), pp. 1–9, 2020. doi: 10.1109/ICCCN49398. 2020.9209704. [76] Halbert White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society, pp. 1–25, 1982. [77] Zejun Xie, Zhiqing Hong, Wenjun Lyu, Haotian Wang, Guang Wang, and Desheng Zhang. CoAlign: Uncertainty calibration of LLM for geospatial repartition. In Georg Rehm and Yunyao Li (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pp. 254–263, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-288-6. doi: 10.18653/v1/2025. acl-industry.19. URL https://aclanthology.org/2025.acl-industry.19/. [78] Zejun Xie, Wenjun Lyu, Yiwei Song, Haotian Wang, Guang Yang, Yunhuai Liu, Tian He, Desheng Zhang, and Guang Wang. Scalable area difficulty assessment with knowledge-enhanced ai for nationwide logistics systems. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD ’25, pp. 2713–2724, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400712456. doi: 10.1145/3690624.3709407. URL https://doi.org/10.1145/3690624.3709407. [79] Zejun Xie, Xintong Li, Guang Wang, and Desheng Zhang. Atc: Aggregate-then-calibrate for human-centered assessment. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=XNbVoi9mfr. [80] QianQian Xu, Jiechao Xiong, Xiaochun Cao, and Yuan Yao. False discovery rate control and statistical quality assessment of annotators in crowdsourced ranking. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1282–1291, New York, New York, USA, 20–22 Jun 2016. PMLR. URL https: //proceedings.mlr.press/v48/xua16.html. [81] Yixuan Even Xu, Hanrui Zhang, Yu Cheng, and Vincent Conitzer. Aggregating quantitative relative judgments: From social choice to ranking prediction. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 19933–19964. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ 2375085c9e5afdfcab8c564e42ec0019-Paper-Conference.pdf. [82] Guang Yang, Yuequn Zhang, Jinquan Hang, Xinyue Feng, Zejun Xie, Desheng Zhang, and Yu Yang. Carpg: Cross-city knowledge transfer for traffic accident prediction via attentive region-level parameter generation. 17

In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 2939–2948, 2023. [83] Guang Yang, Meiqi Tu, Zelong Li, Jinquan Hang, Taichi Liu, Ruofeng Liu, Yi Ding, Yu Yang, and Desheng Zhang. Behavior-aware hypergraph convolutional network for illegal parking prediction with multi-source contextual information. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2827–2836, 2024. [84] Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. Modeling crowdsourcing task complexity. In Proceedings of the Fourth AAAI Conference on Human Computation and Crowdsourcing, pp. 249–258. AAAI, 2016. [85] Qinchen Yang, Zejun Xie, Hua Wei, Desheng Zhang, and Yu Yang. Mallight: Influence-aware coordinated traffic signal control for traffic signal malfunctions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM ’24, pp. 2879–2889, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704369. doi: 10.1145/3627673.3679605. URL https://doi.org/ 10.1145/3627673.3679605. [86] Qinchen Yang, Zhiqing Hong, Dongjiang Cao, Haotian Wang, Zejun Xie, Tian He, Yunhuai Liu, Yu Yang, and Desheng Zhang. Addrllm: Address rewriting via large language model on nationwide logistics data. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD ’25, pp. 2756–2767, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400712456. doi: 10.1145/3690624.3709425. URL https://doi.org/10.1145/3690624.3709425. [87] Anlan Yu, Xuanzhi Wang, Pei Wang, Jinkun Li, Xujun Ma, Zhiqing Hong, Haotian Wang, Yi Ding, and Daqing Zhang. Finesat: Enhancing gnss signals for high-precision sensing. In 2025 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp. 25–36. IEEE, 2025. [88] Cun-Hui Zhang. Risk bounds in isotonic regression. The Annals of Statistics, 30(2):528 – 555, 2002. doi: 10.1214/aos/1021379864. URL https://doi.org/10.1214/aos/1021379864. [89] Hongwei Zhang, Tijin Yan, Zenjun Xie, Yuanqing Xia, and Yuan Zhang. Revisiting graph convolutional network on semi-supervised node classification from an optimization perspective, 2020. URL https://arxiv.org/ abs/2009.11469. [90] Lidi Zhang, Yinfeng Xiang, Wenjun Lyu, Zhiqing Hong, Haotian Wang, Desheng Zhang, Yunhuai Liu, and Tian He. D3-tr: Data-driven daily delivery task rescheduling for cost-effective last-mile delivery. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, pp. 6266–6273, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400720406. doi: 10.1145/3746252. 3761550. URL https://doi.org/10.1145/3746252.3761550. [91] Xiangxu Zhang, Xiao Zhou, Hongteng Xu, and Jianxun Lian. Hypemed: Enhancing medication recommendations with hypergraph-based patient relationships. ACM Trans. Inf. Syst., 44(4), May 2026. ISSN 1046-8188. doi: 10.1145/3803851. URL https://doi.org/10.1145/3803851. [92] Yichi Zhang, Fang-Yi Yu, Grant Schoenebeck, and David Kempe. A system-level analysis of conference peer review. In Proceedings of the 23rd ACM Conference on Economics and Computation, EC ’22, pp. 1041–1080, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450391504. doi: 10.1145/3490486. 3538235. URL https://doi.org/10.1145/3490486.3538235. [93] Yikang Zhang, Xiaomin Chu, Yelu Jiang, Hongjie Wu, and Lijun Quan. Semanticcap: Chromatin accessibility prediction enhanced by features learning from a language model. Genes, 13(4), 2022. ISSN 2073-4425. doi: 10.3390/genes13040568. URL https://www.mdpi.com/2073-4425/13/4/568. [94] Yikang Zhang, Junlong Chen, Wei Wang, Jia Liu, Nan Hu, and Haipeng Dai. Automated end-to-end model serving with cooperative compilation and scheduling. In Proceedings of the Twenty-First European Conference on Computer Systems, EuroSys 2026, Edinburgh, Scotland UK, April 27–30, 2026, pp. 1–16. ACM, 2026. doi: 10.1145/3767295.3769392. URL https://doi.org/10.1145/3767295.3769392. [95] Fei Zhao, Chonggang Lu, Zheyong Xie, Ziyan Liu, Haofu Qian, Jianzhao Huang, Fangcheng Shi, Zijie Meng, Hongcheng Guo, Mingqian He, et al. Redone: Revealing domain-specific llm post-training in social networking services. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 2648–2674, 2025. [96] Zhibing Zhao, Haoming Li, Junming Wang, Jeffrey O. Kephart, Nicholas Mattei, Hui Su, and Lirong Xia. A cost-effective framework for preference elicitation and aggregation. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 446–456. AUAI Press, 2018. URL http://auai.org/uai2018/ proceedings/papers/179.pdf. 18

[97] Zhiyuan Zhou, Li Lin, Hai Wang, Xiaolei Zhou, Gong Wei, and Shuai Wang. A cross domain method for customer lifetime value prediction in supply chain platform. In Proceedings of the ACM Web Conference 2024, pp. 4037–4046, 2024. [98] Zhiyuan Zhou, Xiaolei Zhou, Baoshen Guo, Shuai Wang, and Tian He. Multi-sensor data-driven route prediction in instant delivery with a 3-conversion network. ACM Transactions on Sensor Networks, 20(2):1–21, 2024. [99] Zhiyuan Zhou, Ruofeng Liu, Taichi Liu, Weijian Zuo, Shanshan Wang, Zhiqing Hong, and Desheng Zhang. Any to full: Prompting depth anything for depth completion in one stage, 2026. URL https://arxiv.org/abs/ 2603.05711.

19

A

Asymptotic Efficiency of the Heterogeneous Thurstone Model Estimator

This section details the asymptotic properties of the score estimators derived from the Heterogeneous Thurstone Model (HTM) and compares its efficiency against an estimator from a misspecified homogeneous model. A.1

Model Definitions and Problem Statement

We consider the task of estimating item scores s ∈ Rn from pairwise comparison data. P The comparisons are m provided by m users, each user u performs ku comparisons, leading to a total of N = u=1 ku observations Y = {Yul }u=1..m,l=1..ku . Each Yul ∈ {0, 1} represents the outcome of a comparison between items il and jl by user u. A.1.1

The Heterogeneous Thurstone Model

We assume the true data generating process follows a HTM. Under this model, the probability that user u prefers item il over item jl is given by: P (Yul = 1|s, γu ) = F (γu a⊤ l,u s),

(A.1)

where s ∈ Rn is the vector of true item scores, γu ∈ R+ is the accuracy parameter for user u, al,u = eil − ejl is a difference vector with ei being the i-th standard basis vector, and F (·) is a known link function, typically a Cumulative Distribution Function (CDF) such as the logistic function (leading to the Heterogeneous Bradley-Terry-Luce model, HBTL) or the Normal CDF (leading to the Heterogeneous Thurstone Case V model, HTCV). The complete parameter vector for the HTM is θ = (s, γ) ∈ Rn+m , where γ = (γ1 , ..., γm )⊤ . The log-likelihood for a single observation (Yul , al,u ) given parameters (s, γu ) is denoted by ℓ(Yul |s, γu ) = log P (Yul |s, γu ). The average log-likelihood for all N observations is: m

Lhete (θ; Y ) =

k

u 1 XX ℓ(Yul |s, γu ). N u=1

(A.2)

l=1

The Maximum Likelihood Estimator (MLE) for the HTM is: θbhete = (b shete , γ bhete ) = arg max Lhete (θ; Y ). θ

(A.3)

We are primarily interested in the score estimator b shete . A.1.2

The Homogeneous Thurstone Model

For comparison, we consider a misspecified HTM. This model incorrectly assumes that all users have the same accuracy parameter, γu = γ0 for all u, where γ0 is a fixed constant or a single parameter to be estimated. For simplicity and to highlight the effect of ignoring heterogeneity, we assume γ0 is a known constant (e.g., γ0 = 1). The probability of preference under this model is: P (Yul = 1|s, γ0 ) = F (γ0 a⊤ l,u s) m

Lhomo (s; Y, γ0 ) =

(A.4)

k

u 1 XX log P (Yul |s, γ0 ) N u=1

(A.5)

l=1

The estimator obtained by maximizing b shomo = arg maxs Lhomo (s; Y, γ0 ), the misspecified likelihood, is a QuasiMaximum Likelihood Estimator (QMLE)[24][Page 734]. The primary objective of Theorem 3.6 is to demonstrate that, when the true data generating process is heterogeneous, the estimator b shete derived from the correctly specified HTM is asymptotically more efficient than the estimator b shomo derived from the misspecified homogeneous model. This will be shown by comparing their asymptotic covariance matrices. 20

A.2

Core Idea and Asymptotic Framework

The efficiency comparison between b shete and b shomo is grounded in the asymptotic theory of MLE and QMLE. The core idea is that an estimator derived from a correctly specified model should be asymptotically more efficient than an estimator derived from a misspecified model. Under the stated assumptions, the MLE b shete from the correctly specified HTM is consistent for the true score vector s∗ , asymptotically normal, and asymptotically efficient. Its asymptotic covariance matrix, denoted Σbshete , achieves the Cramér-Rao Lower Bound (CRLB) for estimating s∗ under the HTM. This matrix is derived from the inverse (or pseudoinverse, to account for identifiability constraints) of the Fisher information matrix associated with the HTM, specifically from the Schur complement corresponding to the s parameters. shomo from the misspecified homogeneous model will converge to a parameter vector s∗ that Conversely, the QMLE b minimizes the Kullback-Leibler divergence between the true data generating process and the misspecified model; shomo , denoted Σbshomo , is generally, s∗ ̸= s∗ when true heterogeneity exists. The asymptotic covariance matrix of b given by the “sandwich” formula, which accounts for the misspecification. Due to the model misspecification, the standard information matrix equivalence does not hold for the homogeneous model, and its QMLE is typically not asymptotically efficient for s∗ . Moreover, the asymptotic covariance matrix of an estimator from a correctly specified model will be no larger than that of an estimator from a misspecified model in the Loewner sense. We aim to formalize this by showing that Σbshomo − Σbshete is a positive semi-definite matrix, and strictly positive definite in the identifiable subspace when true heterogeneity is present. To formally compare the efficiency of the estimators b shete and b shomo , we first derive their respective asymptotic covariance matrices by deriving Lemma 3.4 and Lemma 3.5. A.3

Proof of Lemma 3.4

The estimator b shete is derived from the HTM. The average negative log-likelihood function for the HTM, with parameters θ = (s, γ), is given by m

L(s, γ) = −

k

u 1 XX log P (Yul |s, γu ) N u=1

(A.6)

l=1

Pm where N = u=1 ku is the total number of observations, and P (Yul |s, γu ) = F (γu a⊤ l,u s) if we denote g(x; Y ) = − log P (Y |x) as the negative log-likelihood for a single observation. For simplicity, we adopt the notation from prior work where g(x) = − log F (x) is used when the outcome Yul is 1 or F (x) represents the probability of a specific outcome that leads to the form in the loss function. The score functions, which are the first-order partial derivatives of L(s, γ), are: m ku
1 XX g ′ γu a⊤ l,u s γu al,u N u=1 l=1    P  k1 ′ ⊤ ⊤ g γ a s a s 1 l,1 l,1 l=1  1    .. ∇γ L(s, γ) =   . N P    km ′ ⊤ ⊤ g γ a s a s m l,m l,m l=1

∇s L(s, γ) =

(A.7)

(A.8)

The Hessian matrix comprises the second-order partial derivatives: m

k

u
1 XX 2 ⊤ g ′′ γu a⊤ l,u s (γu ) al,u al,u N u=1 l=1 "k #m ! u X
⊤ 2 1 2 ′′ ⊤ ∇γ L(s, γ) = diag g γu al,u s (al,u s) N

∇2s L(s, γ) =

l=1

(∇s ∇γ L(s, γ))i,u =

ku 1 X

N

(A.9)

(A.10)

u=1


⊤
 ′ ⊤ (al,u )i g ′′ γu a⊤ l,u s (al,u s)γu + g γu al,u s

l=1

21

(A.11)

The Fisher information matrix for a single observation Ihete (θ0 ), evaluated at the true parameters θ0 = (s∗ , γ ∗ ), is defined as the negative expectation of the Hessian of the log-likelihood for that observation. For the average log-likelihood L, the corresponding expected Hessian (or information scaled by 1/N ) is I(s∗ , γ ∗ ) = −EY |s∗ ,γ ∗ [∇2 L(s∗ , γ ∗ )]. The blocks of this matrix are: Iss = −EY |s∗ ,γ ∗ [∇2s L(s∗ , γ ∗ )]

(A.12)

(Iγγ )uu = −EY |s∗ ,γ ∗ [(∇2γ L(s∗ , γ ∗ ))uu ] (Isγ )i,u = −EY |s∗ ,γ ∗ [(∇s ∇γ L(s∗ , γ ∗ ))i,u ]

(A.13) (A.14)

⊤ and Iγs = Isγ .

The log-likelihood for a single comparison Yul given s∗ and γu∗ is: ℓul (s∗ , γu∗ ) = Yul log F (x∗ul ) + (1 − Yul ) log(1 − F (x∗ul )),

(A.15)

∗ where x∗ul = γu∗ a⊤ l,u s . Let g(x; Y ) = −[Y log F (x) + (1 − Y ) log(1 − F (x))]. Under standard regularity conditions

for MLE and assuming the model is correctly specified (Assumption A3):

′

EYul |x∗ul [g ′ (x∗ul ; Yul )] = 0,

(A.16)

−EYul |x∗ul [g ′′ (x∗ul ; Yul )] = W (x∗ul ),

(A.17)

2

(F (x)) where W (x) = F (x)(1−F (x)) .

Substituting these expectations, the blocks of the Fisher information matrix (scaled by 1/N ) become: m

Iss =

k

u 1 XX W (x∗ul )(γu∗ )2 al,u a⊤ l,u N u=1

(A.18)

l=1

k

(Iγγ )uu =

u 1 X ∗ 2 W (x∗ul )(a⊤ l,u s ) N

(A.19)

l=1 k

(Isγ )i,u =

u 1 X ∗ ∗ W (x∗ul )(a⊤ l,u s )γu (al,u )i N

(A.20)

l=1

shete , γ bhete ) is N −1 [Ihete (θ0 )]−1 , where Ihete (θ0 ) is the The asymptotic covariance matrix of the MLE θbhete = (b Fisher information for a single observation. The matrix I(s∗ , γ ∗ ) derived above corresponds to N −1 Ihete (θ0 ) if we interpret the sums as averages per observation. We define: −1 Stotal = N · S = N · (Iss − Isγ Iγγ Iγs ), ∗

(A.21)

∗

where Iss , Isγ , Iγγ are the blocks of N · I(s , γ ) (i.e., sums without the 1/N scaling). √ The asymptotic covariance matrix for N (b shete − s∗ ) is then the top-left (n × n) block of [Ihete (θ0 )]−1 . Using the formula for the inverse of a partitioned matrix, this block is: (Stotal /N )+ = S + ,

(A.22)

′ ′ ′ ′ ′ ′ ′ where S = Iss − Isγ (Iγγ )−1 Iγs , and Iss , Isγ , Iγγ are the blocks of the Fisher information for a single observation (i.e., the expressions Eq. A.18-Eq. A.20 without the 1/N factor and with sums replaced by expectations or a single representative term if all ku = k and N = mk).

Due to the identifiability constraint: 1⊤ s∗ = 0,

(A.23)

the score s is estimable only up to an additive constant. This implies that the Fisher information matrix Ihete (θ0 ) is singular, and specifically, its block S is singular with S1 = 0. Let S = U ΛU ⊤ be the spectral decomposition of S, where U is an orthogonal matrix of eigenvectors and Λ = diag(λ1 , ..., λn−1 , 0) is the diagonal matrix of eigenvalues, with λ1 , ..., λn−1 > 0. The Moore-Penrose pseudoinverse is: S + = U Λ+ U ⊤ =

22

n−1 X

1 ui u⊤ i , λ i i=1

(A.24)

where Λ+ = diag(1/λ1 , 1/λ2 , ..., 1/λn−1 , 0). And this S + represents the inverse of S restricted to the identifiable subspace: √

V⊥1 = {v ∈ Rn : 1⊤ v = 0}.

(A.25)

The asymptotic covariance matrix for N (b shete − s∗ ) is therefore S + . This matrix is the Cramér-Rao Lower Bound ∗ for estimating s under the HTM, considering the estimation of γ ∗ and the identifiability constraint on s∗ . □ A.4

Proof of Lemma 3.5

The estimator b shomo is obtained as the QMLE from the HTM, which assumes a common accuracy parameter γ0 for all users. The average log-likelihood for this model is: m

Lhomo (s; Y, γ0 ) =

k

u 1 XX log P (Yul |s, γ0 ) N u=1

(A.26)

l=1

where P (Yul |s, γ0 ) = F (γ0 a⊤ l,u s). Under Assumption (A2), the true data generating process is an HTM with ∗ ∗ ⊤ heterogeneous accuracies γ = (γ1∗ , ..., γm ) , where not all γu∗ are equal to γ0 . Thus, the homogeneous model is misspecified. According to [76, Theorem 2.2], the QMLE b shomo converges in probability to a parameter vector s∗ , which is the unique minimizer of the Kullback-Leibler divergence between the true data generating process and the family of distributions defined by the misspecified homogeneous model. Equivalently, s∗ maximizes the expected misspecified log-likelihood: s∗ = arg max EY |s∗ ,γ ∗ [log P (Yul |s, γ0 )]

(A.27)

s

Due to the model misspecification , it is generally true that s∗ ̸= s∗ . Furthermore, according to [76, Theorem 3.2], the QMLE b shomo is asymptotically normally distributed: √ d N (b shomo − s∗ ) − → N (0, Chomo (s∗ ))

(A.28)

where CHomo (s∗ ) is the “sandwich” covariance matrix given by Chomo (s∗ ) = [Ahomo (s∗ )]−1 Bhomo (s∗ )[Ahomo (s∗ )]−1

(A.29)

The matrices Ahomo (s∗ ) and Bhomo (s∗ ) are defined for a single observation under the true data generating process (s∗ , γ ∗ ), evaluated at s∗ : Ahomo (s∗ ) = EY |s∗ ,γ ∗ [∇2s log P (Yul |s∗ , γ0 )]

(A.30) ⊤

Bhomo (s∗ ) = EY |s∗ ,γ ∗ [(∇s log P (Yul |s∗ , γ0 ))(∇s log P (Yul |s∗ , γ0 )) ]

(A.31) ∗

The expectation EY |s∗ ,γ ∗ [·] is taken with respect to the true probability distribution of Yul determined by s and γ ∗ . Due to the model misspecification, the information matrix equivalence [76, Theorem 3.3] generally does not hold for the homogeneous model, meaning Ahomo (s∗ ) ̸= −Bhomo (s∗ ). Moreover, to account for the identifiability constraint 1⊤ s =√0, the inverses in the sandwich formula are replaced by pseudoinverses. Thus, the asymptotic covariance matrix for N (b shomo − s∗ ) is more precisely written as: √ AsyVar( N (b shomo − s∗ )) = [Ahomo (s∗ )]+ Bhomo (s∗ )[Ahomo (s∗ )]+ (A.32) □ A.5

Proof of Theorem 3.6

shete and b shomo by We aim to compare the asymptotic efficiencies of b respective asymptotic covariance √ examining their matrices. From Lemma 3.4, the asymptotic covariance matrix for N (b shete − s∗ ) is S + . This matrix represents the CRLB for estimating s∗ under the correctly specified HTM, within the identifiable subspace defined by 1⊤ s = 0. Thus, √ AsyVar( Nb shete ) = S + (A.33) √ From Lemma 3.5, the asymptotic covariance matrix for N (b shomo − s∗ ) is given by the sandwich formula: √ AsyVar( Nb shomo ) = [Ahomo (s∗ )]+ Bhomo (s∗ )[Ahomo (s∗ )]+ 23

(A.34)

shomo under the misspecified homogeneous model, and Ahomo (s∗ ) and Bhomo (s∗ ) where s∗ is the probability limit of b are defined as in Lemma 3.5. To prove the first part of Theorem 3.6, we compare the matrices from Eq. A.34 and Eq. A.33. According to the theory of estimation under misspecified models, the asymptotic covariance matrix of a QMLE from a misspecified model cannot be smaller (in the Loewner sense) than the CRLB achieved by an MLE from the correctly specified model, when both are estimating parameters related to the true data generating process. The matrix S + is the CRLB for s∗ under the true HTM. Therefore, it must hold that: [Ahomo (s∗ )]+ Bhomo (s∗ )[Ahomo (s∗ )]+ ⪰ S +

(A.35)

N · Σbshomo ⪰ N · Σbshete .

(A.36)

This inequality establishes that:

For the second part of the theorem, concerning the strict inequality, Assumption (A2) states that true heterogeneity exists, meaning the true user accuracies γu∗ are not all equal to the fixed γ0 used in the homogeneous model. This implies that the homogeneous model is genuinely misspecified. Under such misspecification, the information matrix equivalence shomo fails to incorporate user-specific accuracy information, Ahomo (s∗ ) = −Bhomo (s∗ ) does not hold. The QMLE b leading to a loss of statistical efficiency compared to the MLE b shete from the correctly specified HTM. This efficiency loss manifests as a strict inequality in Eq. A.35 when considering the identifiable subspace V⊥1 = {v ∈ Rn : 1⊤ v = 0}. Specifically, for any non-zero vector v ∈ V⊥1 ,
v⊤ [Ahomo (s∗ )]+ Bhomo (s∗ )[Ahomo (s∗ )]+ v > v⊤ S + v (A.37) This occurs because the misspecification prevents the sandwich covariance matrix from collapsing to the simpler inverse Fisher information form and results in a larger variance for any linear combination of parameters within the identifiable subspace, compared to the CRLB achieved by the correctly specified model. Thus, N · Σbshomo − N · Σbshete is strictly positive definite on V⊥1 . □

B

Analysis of LSE under Model Misspecification and Non-zero Mean Noise

B.1

Preliminaries: Cones and Statistical Dimension

This section introduces fundamental concepts related to convex cones, tangent cones, and statistical dimension, which are essential for the subsequent analysis. Definition B.1 (Convex Cone). A set K ⊂ Rn is a convex cone if for any v1 , v2 ∈ K and any non-negative scalars t1 , t2 ≥ 0, the linear combination t1 v1 + t2 v2 is also in K. If, additionally, K is a closed set, it is a closed convex cone. Definition B.2 (Tangent Cone). Let K ⊂ Rn be a closed convex set and let u ∈ K. The tangent cone to K at u, denoted by TK,u , is defined as: TK,u = {t(v − u) : t > 0, v ∈ K}

(B.1)

where {·} denotes the closure of the set. Intuitively, the tangent cone TK,u comprises all feasible directions from u scaled by non-negative scalars, along which one can move infinitesimally while remaining within K. If K is a closed convex cone, the definition of the tangent cone at u ∈ K simplifies due to the cone property (i.e., if x ∈ K, then tx ∈ K for t ≥ 0). For a closed convex cone K and u ∈ K, the tangent cone is given by : TK,u = K + span({−u}) = {v − tu : v ∈ K, t ≥ 0}

(B.2)

where the sum is the Minkowski sum. Definition B.3 (Isotonic Cone). The isotonic cone (or non-decreasing cone) in Rn , denoted by M, is defined as: M := {x = (x1 , . . . , xn )T ∈ Rn : x1 ≤ x2 ≤ · · · ≤ xn } The set M is a closed convex polyhedral cone. For any u ∈ M, the tangent cone TM,u captures the local directional constraints at u. If ui < ui+1 for some i, the tangent cone allows more freedom in the i-th and (i + 1)-th coordinates compared to when ui = ui+1 , where the constraint xi ≤ xi+1 becomes active for directions in the cone. 24

Definition B.4 (Statistical Dimension). Let K ⊂ Rn be a closed convex cone. Its statistical dimension, denoted by δ(K), is defined as:   δ(K) := E |ΠK (g)|22 (B.3) where g ∼ N (0, In ) is a standard Gaussian vector in Rn , and ΠK (g) is the Euclidean projection of g onto K. Equivalently, the statistical dimension can also be expressed as:  !2   T  δ(K) = E g ΠK (g) = E  sup gT θ  (B.4) θ∈K:|θ|2 ≤1

The statistical dimension provides a measure of the “complexity” of the cone from a statistical perspective, often related to the effective number of parameters in a model constrained to the cone. B.2

Problem Setting and Relation to Prior Work

Let s ∈ Rn be an underlying ground truth score vector. The primary target of our estimation in the second stage is a s ∈ Rn , which can be thought of as a “subjective optimal point” or a noisy version of s. Specifically, e s is related vector e generated as: e s = s + ϵe,

(B.5)

where ϵe ∼ N (0, σ e2 In ). Thus, e s is a random vector with E[e s] = s. We assume the true isotonic cone relevant to our f s is M := Mπ(es) (or Mπ(s) , if the true ordering is dictated by the mean s; this distinction should be noted, but target e f in expectation or with high probability). for now, we assume e s∈M In the first stage of our overall procedure, a separate algorithm (e.g., Algorithm 1 from [39]) processes auxiliary data to produce an estimate of latent scores, s∗ ∈ Rn . The ranking induced by s∗ , denoted π b := π(s∗ ), defines a (random) c c isotonic cone M := Mπb . This cone M serves as the constraint set for the second stage. The observations for the second stage, denoted by sp ∈ Rn , are related to the ground truth s and a fixed, unknown systematic bias vector ν ∈ Rn as follows: sp = s + ν.

(B.6)

Using the relationship s = e s − ϵe, we can express sp in terms of our target e s: sp = (e s − ϵe) + ν = e s + ν − ϵe c The final estimator for the target e s is obtained by projecting sp onto the estimated cone M: b s := ΠM c(sp ) Our objective is to analyze the risk of this estimator with respect to e s: E[∥b s−e s∥22 ].

(B.7)

c random). The expectation is over all sources of randomness: ϵe (which makes e s random) and b salg1 (which makes M The effective noise when estimating e s from sp is: ξ := sp − e s = ν − ϵe,

(B.8)

in which ξ ∼ N (ν, σ e2 In ), as E[ξ] = ν and Cov(ξ) = Cov(−e ϵ) = σ e2 In . B.3

Proof of Theorem 3.8

In stage-1, the target is e s, and our observation is sp . The effective noise is ξ = sp − e s ∼ N (ν, σ e2 In ). By Lemma 3.7, we have: 1 ∥b s−e s∥22 − ∥u − e s∥22 ≤

n

25

!2 sup θ∈TM,u :∥θ∥2 ≤1 c

T

ξ θ

(B.9)

c and e Taking expectation with respect to ξ (conditional on M s), and choosing u = ΠM s), we have: c(e " # 2 1 2 2 c T ce s) − e s∥2 + Eξ Eξ [∥b s−e s∥2 |M, e s] ≤ ∥ΠM sup ξ θ M, s c(e n θ∈W ′

(B.10)

where W ′ = {θ ∈ TM,Π : ∥θ∥2 ≤ 1}. Let ϵ′ = ξ − ν = −e ϵ ∼ N (0, σ e2 In ). The expectation term is bounded c c (e s) M

using (a + b)2 ≤ 2a2 + 2b2 : " # " # 2 2  2 T T ′ T T c c Eξ sup ((ξ − ν) θ + ν θ) M, e s ≤ 2Eϵ′ sup (ϵ ) θ M, e s + 2 sup ν θ θ∈W ′

θ∈W ′

= 2e σ 2 δ(TM,Π ) + 2∥ΠTM,Π c c (e c s) M

(B.11)

θ∈W ′

s) c (e M

(ν)∥22

(B.12)

c and e Thus, given M s, the conditional risk is: 2 2e σ2 ce δ(TM,Π ) + ∥ΠTM,Π (ν)∥22 Eξ [∥b s−e s∥22 |M, s] ≤ ∥ΠM s) − e s∥22 + c c (e c(e c s ) s) c (e M n n M

(B.13)

c and e Taking the full expectation over M s:   s) − e s∥22 E[∥b s−e s∥22 ] ≤EM,e c(e c s ∥ΠM  2  2e σ + EM,e ) δ(T cs c c (e M,Π s) M n   2 2 + EM,e ∥Π (ν)∥ TM,Π cs 2 c s) c (e n M

(B.14) (B.15) (B.16)

We aim to bound the risk E[∥b s −e s∥22 ], where b s = ΠM s. Recall the key relationships: c(sp ) is the estimator for the target e • Target for estimation: e s = s + ϵe, where ϵe ∼ N (0, σ e2 In ). • Observation: sp = s + ν = e s − ϵe + ν. • Effective noise for estimating e s from sp : ξ = sp − e s = ν − ϵe ∼ N (ν, σ e2 In ). c = Mπ(s∗ ) , where s∗ is the output of a first-stage algorithm, estimating e • M s. f = Mπ(es) . The error term arises if s is π(E[e s]) = π(s). Let M • We assume the expected ordering relevant for e c ̸= M f and e c M s∈ / M. From Lemma 3.7 (or Eq. B.14- B.16 in previous discussions), the total risk is bounded by:   E[∥b s−e s∥22 ] ≤ EM,e s) − e s∥22 c s ∥ΠM c(e | {z }

(B.17)

Term 1: Projection Error from Misspecified Cone

 +

EM,e cs |

 2e σ2 δ(TM,Π ) c c (e s) M n {z }

(B.18)

Term 2: Statistical Error (Zero-Mean Noise Component)

  2 2 + EM,e ∥Π (ν)∥ T cs 2 c M,Π s) c (e n M | {z }

(B.19)

Term 3: Bias Error Component

We now bound each term.   Bounding Term 1: Projection Error from Misspecified Cone. Term 1 is EM,e s) − e s∥22 . This term c s ∥ΠM c(e c which is estimated from s∗ and may differ from the captures the error due to projecting the target e s onto the cone M, e true cone associated with s (or its mean s). 26

f = Mπe . The event of Let π e = π(e s) be the true ranking corresponding to the target of the first-stage algorithm. Let M c c (relative to the first stage’s own target) is A ≡ {b an incorrect cone M π ̸= π e}, where π b = π(s∗ ). h    i EM,e s) − e s∥22 = Ees EM|e s) − e s∥22 (B.20) c s ∥ΠM c(e c s ∥ΠM c(e c (i.e., the random target happens to satisfy the random cone’s constraints), then ∥Π c(e If e s∈M s) − e s∥22 = 0. This M ∗ occurs if π(s ) is compatible with e s. We follow the previous decomposition:     EM|e s) − e s∥22 = EM|e s) − e s∥22 Ac P (Ac ) (B.21) c s ∥ΠM c(e c s ∥ΠM c(e   (assuming ∥ΠM s) − e s∥22 is negligible or zero if π(s) = π e). The term EM|e s) − e s∥22 Ac is bounded by f(e c s ∥ΠM c(e sup ∥ΠM s) − e s∥22 ≤ nVar(e s), c(e

(B.22)

c M

where Var(e s) is the sample variance of the components of the specific realization of e s. Taking expectation over e s:   2 c EM,e s) − e s∥2 ≤ Ees [nVar(e s)]P (A ) (B.23) c s ∥ΠM c(e The term Ees [nVar(e s)] is related to the expected dispersion of e s. Since e s = s + ϵe, E[e si ] = si . " n # " n # X X ¯s)2 = Ees ¯s2 e Ees (e si − e s2 − ne i

i=1

(B.24)

i=1

This can be further analyzed, but for a simpler bound, we can use: Ees [nVar(e s)] ≤ Ees [∥e s∥22 ] = ∥s∥22 + ne σ2 .

(B.25)

si is bounded, nVar(e For a potentially looser but simpler worst-case, if the range of e s) can be bounded by a term related to the range, e.g., n(Res )2 /4 if e si ∈ [a, b] and Rse = b − a. A bound that depends on the properties of the ground truth s and noise σ e2 is needed here. For now, we denote it as E[nVar(e s)]. The probability of a ranking error P (Ac ) = P (π(s∗ ) ̸= π(e s)) is: X P (Ac ) ≤ P (s∗j > s∗k )

(B.26)

j,k: e sj <e sk 1 ∗ 2 e kj = e Let η = s∗ − e s ∼ N (0, Σs∗ ), where Σs∗ = mk sk − e sj , and σX = (ej − ek )⊤ Σs∗ (ej − ek ). Using S + . Let ∆ jk the Chernoff-Cramer bound: ! ∗ e kj )2 σX (∆ jk ∗ ∗ (B.27) P (sj > sk ) ≤ √ exp − ∗ 2 e kj ) 2π 2σXjk (∆

This completes the proof of Corollary 3.9. Thus, ∗ e kj )2 σX (∆ jk Term 1 ≤ Ees [nVar(e s)] √ exp − ∗ 2 e kj ) 2π 2σXjk (∆ j,k:e sj <e sk

!

X

Alternatively, using the expected number of inversions Inv(π ∗ , π e) for the first stage: X P (s∗j > s∗k ) = E[Inv(π ∗ , π e)]

(B.28)

(B.29)

j,k:e sj <e sk

Then, Term 1 ≤ Ees [nVar(e s)] · E[Inv(π ∗ , π e)] Bounding Term 2: Statistical Error.

Term 2 from the main risk decomposition Eq. B.15 is given by:  2  2e σ δ(T ) Term 2 = EM,e cs c c (e M,Π s) M n 27

(B.30)

(B.31)

c = Mπ(s∗ ) is the estimated isotonic cone, which is always a monotone cone . The point Π c(e Here, M s) is an element M c of M. A fundamental property of tangent cones is that for any closed convex set K and any point u ∈ K, the statistical dimension of the tangent cone TK,u is bounded by the statistical dimension of the set K itself: δ(TK,u ) ≤ δ(K) (B.32) This result is standard in the analysis of such problems (see, e.g., discussions related to statistical dimension in [3]). In c Since M c is always an isotonic cone, its statistical dimension is the same as that of the standard our case, K = M. isotonic cone M↑n . It is known (cf. [3, Eq. (D.12)]) that the statistical dimension of the standard isotonic cone M↑n is given by the n-th harmonic number Hn : n X 1 δ(M↑n ) = = Hn (B.33) k k=1

c and e Therefore, for any realization of M s: c = δ(M↑ ) = Hn δ(TM,Π ) ≤ δ(M) c c (e n s)

(B.34)

M

The harmonic number Hn has well-known asymptotic expansions. For n ≥ 1:   1 1 1 − +O (B.35) Hn = ln(n) + γE + 2n 12n2 n4 where γE ≈ 0.57721 is the Euler-Mascheroni constant. Thus, we can use the upper bound: 1 (B.36) Hn ≤ ln(n) + γE + 2n For a simpler, more common bound in such analyses, Hn ≈ ln(n) + γE , or Hn ≤ ln(n) + 1 for n ≥ 1. The bound log(en) = ln(n) + 1 used in [5] (Eq. 1.29) is a convenient and slightly looser upper bound for Hn . Using Hn ≤ ln(n) + γE + O(1/n):     2  2e σ 2e σ2 1 2e σ 2 Hn Term 2 ≤ EM,e Hn = = ln(n) + γE + O (B.37) cs n n n n Alternatively, using the log(en) bound for consistency with some literature: 2e σ 2 log(en) (B.38) Term 2 ≤ n This provides a bound for the statistical error component that depends on the variance σ e2 of the zero-mean noise component ϵe and logarithmically on n. Bounding Term 3: Bias Error Component.

Term 3 from the main risk decomposition Eq. B.16 is given by:   2 2 Term 3 = EM,e ∥ΠTM,Π (ν)∥2 (B.39) cs c s) c (e n M This term arises from the fixed systematic bias ν in the effective noise ξ = ν − ϵe. The expectation EM,e c s averages over c the randomness of the estimated cone M and the randomness of the target e s , which influences the point ΠM s) at c(e which the tangent cone is evaluated. Let uproj = ΠM s). We need to bound ∥ΠTM,u (ν)∥22 . A fundamental property of projection onto a closed convex c(e c proj set is that it does not increase the L2 norm. Therefore, ∥ΠTM,u (ν)∥22 ≤ ∥ν∥22 (B.40) c proj

c and e This inequality holds for any realization of M s. Substituting this into the expression for Term 3:   2 2 Term 3 ≤ EM,e ∥ν∥2 (B.41) cs n As ν is a fixed vector, ∥ν∥22 is a constant. Then the expectation operator does not change it: 2∥ν∥22 Term 3 ≤ (B.42) n This provides an upper bound for the bias error component, indicating that it scales with the squared norm of the fixed bias vector ν and decreases with 1/n. The factor of 2 arises from the use of the inequality (a + b)2 ≤ 2a2 + 2b2 when separating the zero-mean noise and bias components in the derivation of Lemma 3.7. 28

Combined Risk Bound.

Combining the bounds for the three terms: E[∥b s−e s∥22 ] ≤Ees [nVar(e s)] · E[Inv(π ∗ , π e)]

2e σ 2 log(en) n 2∥ν∥22 + n where E[Inv(π ∗ , π e)] is the expected number of inversions produced by the first-stage algorithm s∗ with respect to its true target e s, and can be expressed as: X E[Inv(π ∗ , π e)] = P (s∗j > s∗k ) +

j,k:e sj <e sk

with each P (s∗j > s∗k ) bounded using the Chernoff-Cramer bound: p   (ej − ek )⊤ Σs∗ (ej − ek ) (e sk − e sj )2 ∗ ∗ √ P (sj > sk ) ≤ exp − 2(ej − ek )⊤ Σs∗ (ej − ek ) (e sk − e sj ) 2π and Σs∗ is the asymptotic covariance matrix of s∗ in Lemma 3.4. Additional discussion. The core distinctions remain similar to previous discussions, with the key aspect being that our target for estimation e s is a noisy version of the underlying ground truth s, our observations sp are related to e s via an c (1) Differs from traditional isotonic effective noise term N (ν, σ e2 In ), and the projection is onto a random cone M. regression which assumes a fixed cone and zero-mean noise relative to its direct target. (2) Differs from fixed model misspecification as our cone is random and our effective observation noise has a fixed bias ν and variance σ e2 In relative to the target e s.

C

Probability of Estimation Improvement via Projection

In this section, we establish a probabilistic bound demonstrating that, under certain conditions related to the accuracy of c the projection-based estimator b s provides a strictly better estimate of the ground truth s than the the estimated cone M, direct observation sp . C.1

Proof of Corollary 3.9

Lemma C.1 (Restatement of Corollary 3.9). Let s∗ be the estimate from the first-stage algorithm for its true target e kj = e e s. For any two items j, k with e sj < e sk , let ∆ sk − e sj be the true score gap for the first-stage target, and let ∗ 2 T σXjk = (ej − ek ) Σs∗ (ej − ek ) be the variance of the estimated score difference from the first stage, where Σs∗ is c = Mπ(s∗ ) differs from the true the asymptotic covariance of s∗ − e s. Then the probability that the estimated cone M f = Mπ(es) is bounded by cone for the first-stage target M ! ∗ 2 X e σ ( ∆ ) X kj jk δ1 := P (π(s∗ ) ̸= π(e s)) ≤ exp − . (C.1) √ ∗ 2 e kj 2π 2 σX ∆ jk j,k: e s <e s j

k

Proof. The event π(s∗ ) ̸= π(e s) occurs if and only if there exists at least one pair of items (j, k) such that their true order according to e s is e sj < e sk , but their estimated order is s∗j > s∗k . Using the union bound over all such pairs: X P (π(s∗ ) ̸= π(e s)) ≤ P (s∗j > s∗k ) j,k: e sj <e sk ∗

Let η = s − e s. We assume η ∼ N (0, Σ ). The probability of a single incorrect pairwise ranking is P (s∗j > s∗k ) = ∗ 2 e kj ). P (ηj − ηk > e sk − e sj ). Let Xjk = ηj − ηk . Then Xjk ∼ N (0, σX ). The probability becomes P (Xjk > ∆ jk s∗

2

Applying the Chernoff-Cramer bound (see Appendix B.3 or a similar result), P (Z > t) ≤ t√12π e−t /2 for Z ∼ N (0, 1) e kj /σ ∗ . and t > 0. Let tjk = ∆ Xjk ! ∗ e kj )2 σX ( ∆ jk e kj ) ≤ P (Xjk > ∆ exp − √ ∗ 2 e kj 2π 2 σX ∆ jk Summing over all relevant pairs yields the stated bound for δ1 . 29

C.2

Proof of Lemma 3.11

Lemma C.2 (Restatement of Lemma 3.11). Let s ∈ Rn be the ground truth score and e s = s+e ϵ, where ϵe ∼ N (0, σ e2 In ). c c Let B be the event B := {π(e s) ̸= π(s)}. Then,   X σ e (s))2 √ exp − (∆kj δ2 := P (B c ) ≤ (C.2) 2 4e σ (∆kj (s)) π j,k: sk >sj

where ∆kj (s) = sk − sj . Proof. The event B c ≡ {π(e s) ̸= π(s)} occurs if there exists at least one pair of items (j, k) such that their true order according to s is (without loss of generality) sj < sk , but their order according to e s is e sj > e sk . Using the union bound over all such pairs (j, k) where sj < sk : X P (B c ) ≤ P (e sj > e sk ) j,k: sj <sk

Consider a single pair (j, k) such that sj < sk . The event e sj > e sk is equivalent to (sj + e ϵj ) > (sk + e ϵk ), which simplifies to: e ϵj − e ϵk > sk − sj Let Yjk = e ϵj − e ϵk . Since e ϵj and e ϵk are independent and identically distributed as N (0, σ e2 ), their difference Yjk follows a normal distribution: Yjk ∼ N (0, Var(e ϵj ) + Var(e ϵk )) = N (0, 2e σ2 ) Let ∆kj (s) = sk − sj . Since s√j < sk , we have ∆kj (s) > 0. The probability of interest for a single pair is P (Yjk > ∆kj (s)). Let Z = Yjk / 2e σ 2 . Then Z ∼ N (0, 1).   ∆kj (s) P (Yjk > ∆kj (s)) = P Z > √ 2e σ2 2

∆

(s)

Applying the Chernoff-Cramer bound P (Z > t) ≤ t√12π e−t /2 with t = √kj2eσ2 : √   (∆kj (s))2 2e σ2 √ exp − P (Yjk > ∆kj (s)) ≤ 2 · (2e σ2 ) ∆kj (s) 2π   σ e (∆ (s))2 √ exp − kj 2 = 4e σ ∆kj (s) π

(C.3) (C.4)

Summing this probability over all pairs (j, k) such that sj < sk (which is equivalent to summing over pairs j, k such that sk > sj by swapping indices if necessary, ensuring ∆kj (s) remains positive) gives the stated bound for δ2 . C.3

Proof of Theorem 3.12

Theorem C.3 (Restatement of Theorem 3.12). Let δ1 = P (π(s∗ ) ̸= π(e s)) be the probability of a ranking error in the first-stage estimation relative to its true target e s (as defined in Corollary 3.9). Let δ2 = P (π(e s) ̸= π(s)) be the probability that the ranking of the subjective optimal point e s differs from the ranking of the ground truth s (as defined in Lemma C.2). Assume that the true ranking for the first-stage target is consistent with the ground truth ranking, i.e., π(e s) = π(s). Then, with probability at least 1 − δ1 − δ2 , ∥b s − s∥22 < ∥sp − s∥22 ,

(C.5)

provided that sp ∈ / Mπ(s) . Proof. Consider the following events: c correctly identifies the cone Mes derived from the subjective • Let A be the event that the estimated cone M c = Mes }. optimal point e s, i.e., A := {M • Let B be the event that the cone Mes correctly identifies the true cone Ms derived from the ground truth s, i.e., B := {Mes = Ms }. 30

c ̸= Mes ) = P (π(s∗ ) ̸= π(e The probability P (Ac ) = P (M s)) = 1 − δ1 Thus, P (A) = 1 − δ1 . And P (B c ) = P (Mπ(es) ̸= Mπ(s) ) = P (π(e s) ̸= π(s)) = δ2 , as defined in Lemma C.2. Thus,P (B) = 1 − δ2 . Now we let C be the c = Ms . Event C occurs with probability at least 1 − δ1 − δ2 . event that M When event C occurs, since s defines the cone Ms , it follows that s ∈ Ms .Thus, c s ∈ M.

(C.6)

c for any vector x ∈ M, c we have: By the fundamental property of projection onto a closed convex set M, ∥sp − x∥22 ≥ ∥sp − b s∥22 + ∥x − b s∥22

(C.7)

∥sp − s∥22 ≥ ∥sp − b s∥22 + ∥s − b s∥22

(C.8)

Then we can set x = s:

This inequality holds when event C occurs. For the strict inequality ∥s − b s∥22 < ∥sp − s∥22 to hold, we require that the 2 c Thus, if event C occurs term ∥sp − b s∥2 is strictly positive. This occurs if and only if sp ̸= b s, which is true if sp ∈ / M. and it is also true that sp ∈ / Ms , then ∥sp − b s∥22 > 0. In this situation, we have: ∥s − b s∥22 < ∥sp − s∥22

(C.9)

c is met. The condition sp ∈ This strict inequality holds when event C occurs and the condition sp ∈ /M / Ms means that the biased observation s + ν violates the true ordering of s, which is generally true if ν is non-zero and not perfectly aligned with the cone Ms in a way that preserves membership. Remark C.4 (Conditions for Small Error Probabilities δ1 and δ2 ). The overall risk bound depends critically on the magnitudes of δ1 and δ2 . These probabilities become small under favorable conditions: e (∆

)2

• For δ1 (First-stage ranking accuracy): The probability P (s∗j > s∗k ) for e sj < e sk is small if the ratio σ∗kj 2 Xjk

is large. This occurs when: e kj = e 1. The true gaps ∆ sk − e sj between scores (for the first-stage target e s) are large, making items more distinguishable. ∗ 2 2. The variance of the score difference estimates σX = (ej − ek )T Σs∗ (ej − ek ) is small. This implies jk 1 S+ that the first-stage algorithm provides precise estimates of score differences. The matrix Σs∗ = mk indicates that precision increases with more users (m) or more comparisons per user (k). Thus, δ1 is small if the first-stage estimation is based on sufficient data and the items it aims to rank are well-separated. s and its mean s): The probability P (e sj > e sk ) for sj < sk is small if • For δ2 (Ranking consistency between e the ratio

(∆kj (s))2 is large. This occurs when: 4e σ2

1. The true gaps ∆kj (s) = sk − sj in the ground truth scores s are large. 2. The variance σ e2 of the subjective noise ϵe (which makes e s deviate from s) is small. Thus, δ2 is small if the ground truth scores s are well-separated and the process generating the subjective optimal point e s from s has low noise.

D

Additional Information of Experiments

D.1

Datasets and Settings

Reading Level: This dataset contains pairwise comparisons of text documents based on reading difficulty. The dataset includes 490 documents with known reading levels, serving as ground truth difficulty scores. A total of 624 annotators provided 12728 pairwise judgments, with each comparison indicating which document is easier to read. The dataset structure includes judge information, judgment outcomes (where “A” indicates document A is easier than document B), document identification numbers, and the corresponding reading levels for each document pair. We construct a semi-synthetic setting by generating pairwise comparisons from an underlying Bradley-Terry/Thurstone model using the true item scores, while adding annotator-specific noise to simulate heterogeneity. This provides us with realistic but 31

controlled pairwise preference data. We then obtain an “oracle” objective score sp for each item by perturbing its true score with additional noise or bias, mimicking the output of an imperfect predictive model. This noisy sp serves as the initial model assessment for Stage 2 of AtC. The availability of ground truth in these tasks allows us to quantitatively evaluate s∗ , sp , and b s under various metrics. Dots-activity: This dataset [41] contains judgments from 300 participants on 30 distinct images, yielding 8700 pairwise comparisons for estimating dot counts. We transformed this into a human-centered assessment scenario as follows: humans observe original complete images to provide comparative judgments, while the objective model (OpenCV’s contour detection) processes corrupted images with varying degradation levels (global blur, localized blur/noise/whiteout). The task is to estimate dot counts in the original images, so ground truth remains unchanged regardless of corruption applied to model inputs. This setup simulates realistic scenarios where human judgment helps calibrate models that receive degraded inputs (e.g., low-quality sensors, occluded views), while humans access complete information through experience or alternative channels. To our knowledge, this represents the first open-source study using real human data in this field; given the scarcity of suitable datasets, this transformation provides our best validation approach on authentic data. D.2

Rank Aggregation Baselines

We compare several methods for aggregating the crowdsourced pairwise judgments (Stage-1), including both homogeneous models and heterogeneity-aware models: • BTL (Bradley-Terry-Luce): A classic model for pairwise comparisons that assumes each item i has a latent score si i) such that the probability of i beating j is Pr(i ≻ j) = exp(sexp(s . We fit the scores s by maximum likelihood, i )+exp(sj ) given all pairwise preferences. BTL assumes all annotators are identical and noise is homogeneous (every comparison is an independent sample from the same underlying distribution). • Thurstone Case V (TCV): An alternative pairwise comparison model proposed by Thurstone, which assumes differences in item scores are normally distributed. In Case V, Pr(i ≻ j) = Φ(si − sj ) where Φ is the standard normal CDF. Like BTL, the homogeneous TCV model assumes a single common noise level for all annotators. We fit item scores by MLE under this probit-based model. BTL and TCV typically produce similar rankings; TCV can be slightly more robust to outliers in some cases (due to the normal vs. logistic noise assumption). • CrowdBT: A variant of the Bradley-Terry model that incorporates annotator-specific reliability parameters. In CrowdBT [12], each annotator u has an associated consistency weight or bias parameter that influences the comparisons they provide. Intuitively, this model down-weights votes from inconsistent annotators and up-weights those from reliable annotators, yielding a better aggregate score s∗ . We implement CrowdBT following the approach of [12], including a “virtual annotator”regularization strategy to stabilize the reliability estimates. • CrowdTCV: A heterogeneity-aware extension of Thurstone Case V that incorporates annotator-specific precision parameters. Each annotator u is assumed to have their own precision γu in the Thurstone model, such that Pr(u : i ≻ j) = Φ(γu (si − sj )). This model was introduced by [39]. We refer to our implementation simply as CrowdTCV. Like CrowdBT, it learns which annotators are more consistent (higher γu ) and which are noisier (lower γu ), improving the quality of the aggregated s∗ . • Heterogeneous Rank Aggregation (HRA) variants: We adopt the heterogeneous rank aggregation framework of Jin et al. [39], which explicitly models annotator-specific noise in the Bradley-Terry and Thurstone settings. Specifically, in the heterogeneous Bradley-Terry-Luce (HBTL) variant, Pr(u : i ≻ j) = σ(γu (si − sj )) where σ is the logistic CDF, whereas in the heterogeneous Thurstone Case V (HTCV) variant, the logistic is replaced by the normal CDF Φ. Both item scores s and annotator precisions {γu } are jointly estimated from data via iterative optimization. These methods achieve strong aggregation performance by accounting for individual differences: an annotator with a very small γu (unreliable) contributes nearly random comparisons, which the model down-weights, whereas an annotator with large γu is given more influence. We use the term “HRA” to refer to this class of heterogeneity-aware aggregators, with HBTL and HTCV as representative implementations (corresponding to HRA-E and HRA-N in Table 1). Each aggregation method produces an output consensus score vector s∗ (determined up to an arbitrary scale, which we normalize for evaluation). The s∗ ’s rank π b serve as inputs to Stage-2 of AtC. D.3

Evaluation Metrics

We evaluate the quality of the predicted scores and rankings using the following metrics. Let b s = (b s1 , . . . , b sn ) be the scores produced by a given method and s = (s1 , . . . , sn ) be the ground-truth scores (when available). For distribution32

based metrics, let p denote the true distribution of scores (or true outcome values) and q denote the distribution derived from b s. • Kendall’s Tau (τ ): A rank correlation coefficient between the ordering induced by b s and the true ordering. It is C −D
, where C is the number of concordant item pairs and D is the number of discordant pairs defined as τ = n 2

when comparing the rankings of b s and s. We have τ = 1 if the two rankings are identical, τ = 0 if the rankings are uncorrelated, and τ = −1 if one ranking is the exact reverse of the other. Higher τ indicates better agreement with the ground-truth ordering. v u n u1 X • MSE: The root-mean-squared error of the scores. We compute MSE = t (b si − si )2 . This is the square root of n i=1 the mean squared error (MSE), expressed in the same units as the scores. A lower L2 error indicates that the predicted scores b s are numerically closer to the true scores s. • Wasserstein Distance: Also known as the Earth Mover’s Distance, this measures the distance between two distributions. Treating the set of predicted si } and true scores {si } as empirical distributions, the Wasserstein-1 Z ∞scores {b distance is defined as W1 (b s, s) = Fbs (x)−Fs (x) dx, where Fbs and Fs are the cumulative distribution functions −∞

(CDFs) of the predicted and true score distributions. In practice, we compute W1 by sorting the scores and finding the area between the two empirical CDF curves. Smaller W1 implies that the distribution of predicted scores is closer to that of the true scores. • Kolmogorov–Smirnov (KS) Statistic: Another measure of distributional discrepancy between b s and s. The KS s and the CDF of s. statistic is DKS = sup Fbs (x) − Fs (x) , the maximum absolute difference between the CDF of b x

This represents the largest gap between the two cumulative distributions. Lower KS values (closer to 0) indicate a better alignment of the predicted score distribution with the true distribution. • Kullback–Leibler (KL) Divergence: For tasks where we compare probability distributions, we use KL divergence to measure how one distribution diverges from another. Let p = {pi } be the true distribution and q = {qi } be the s (e.g. by normalizing or exponentiating the b s values). The KL divergence of q predicted distribution derived from b n X pi from p is defined as DKL (p ∥ q) = pi log . We treat lower KL as better (with DKL = 0 indicating q exactly qi i=1 equals p). Note that KL is an asymmetric measure and is undefined if there exists some i with pi > 0 but qi = 0; in our implementation we add a small ϵ to predicted probabilities to avoid zeros.

Remark. An important phenomenon in AtC is that b s can achieve higher Kendall’s τ than s∗ despite both inducing the same ranking π b. The key is that b s often contains ties introduced by the Pool-Adjacent-Violators (PAVA) algorithm, which are absent in s∗ . When sp violates the consensus ranking π b, PAVA resolves conflicts by averaging violating scores, creating ties. This improves τ by converting discordant pairs into tied pairs, which Kendall’s τ penalizes less severely than discordant pairs [8]. Consider an illustrative example where ground truth is s = [10, 20, 30, 40] with order A < B < C < D. Stage-1 produces s∗ = [1, 2, 5, 4], inducing the ranking A < B < D < C where items C and D are inverted. The model generates sp = [12, 22, 29, 31] with the correct order. PAVA detects the violation at positions C and D, then averages them to produce b s = [12, 22, 30, 30]. Computing Kendall’s τ against ground truth yields τ (s∗ , s) = (5 − 1)/6 ≈ 0.67 but τ (b s, s) = (5 − 0)/6 ≈ 0.83. The tie at positions C and D eliminates the discordant pair, improving correlation despite identical ordinal constraints from Stage-1. D.4

Estimated Score Distributions on Reading Level Dataset

This section provides visualizations of the estimated item scores obtained by different methods on the Reading Level dataset. We present violin plots to illustrate the distribution of scores for each item. 33

(a) BTL-MLE

(b) CrowdBT

(c) TCV-MLE

(d) CrowdTCV

(e) HRA-N

(f) HRA-G

Figure 8: Violin plots of estimated scores for different baseline algorithms. D.5

Impact of Scale Inconsistency on Reading Level Dataset

A common concern in human judgment aggregation is whether scale inconsistency between consensus scores and ground truth inflates distributional divergence metrics. To address this, we conducted experiments scaling s∗ to match the range of ground truth s using linear transformation: s∗scaled =

s∗ − min(s∗ ) · (max(s) − min(s)) + min(s). max(s∗ ) − min(s∗ )

Table 3 presents results on the Reading Level dataset comparing four variants: unscaled consensus (s∗ ), scaled consensus (s∗scaled ), model predictions (sp ), and AtC output (b s). The results show that s∗scaled substantially improves performance ∗ over s (e.g., Wasserstein distance drops from 2.25 to 1.44 for HRA-G), confirming that scale inconsistency is a major issue. However, b s still outperforms s∗scaled across all metrics and methods, demonstrating that isotonic calibration c surface yields superior calibration provides value beyond simple rescaling. This validates that projection onto M’s compared to linear transformations that only address scale but not ordinal alignment with model predictions. D.6

Comprehensive Baseline Comparison on Real-World Dataset

We present two evaluations on the Dots-activity dataset: (1) score-level performance analysis comparing human-only (s∗ ), model-only (sp ), and AtC (b s) assessments across all Stage-1 aggregation methods, and (2) end-to-end comparison against three established human-centered calibration baselines: • GPPL [13]: Gaussian Process Preference Learning models pairwise preferences using GP regression with specialized covariance functions, learning latent utility functions from comparative judgments. • Rank-SVM [40]: Support Vector Machine adaptation for ranking problems that learns from pairwise preferences by optimizing margin-based objectives, commonly used in information retrieval and clickthrough data analysis. 34

Table 3: Scale Inconsistency Analysis on Reading Level Dataset.

We compare unscaled consensus (s∗ ), scaled consensus (s∗scaled ), model predictions (sp ), and AtC output (b s). Bold indicates the best performance among s∗scaled , s∗ , sp , and b s. The red denotes optimal performance across all methods. Stage-1 Method

Wasserstein↓

KS↓

s∗scaled / s∗ / sp / b s

s∗scaled / s∗ / sp / b s

HRA-G HRA-E HRA-N

1.44 / 2.25 / 2.83 / 0.84 1.46 / 2.24 / 2.83 / 0.83 1.45 / 2.35 / 2.83 / 0.74

0.31 / 0.50 / 0.30 / 0.16 0.34 / 0.50 / 0.30 / 0.16 0.31 / 0.56 / 0.30 / 0.19

BTL-MLE CrowdBT CrowdTCV TCV-MLE

1.78 / 2.19 / 2.83 / 0.89 1.72 / 2.15 / 2.83 / 0.84 1.76 / 2.27 / 2.83 / 1.02 1.78 / 2.34 / 2.83 / 0.91

0.42 / 0.46 / 0.30 / 0.30 0.40 / 0.46 / 0.30 / 0.27 0.42 / 0.51 / 0.30 / 0.31 0.43 / 0.55 / 0.30 / 0.30

• BARCW [50]: Bayesian Approach for Ranking with Comparison Weights, a recent method designed to handle inconsistent preferences by modeling annotator-specific biases and reliabilities using Gaussian processes. Table 4 demonstrates that AtC consistently outperforms both sp and s∗ across all seven aggregation methods. While the three baseline methods achieve competitive ranking accuracy (Kendall’s τ ≈ 0.92–0.94), they exhibit catastrophic failures in cardinal calibration: MSE deteriorates by 400–450× and Wasserstein distance by 24–26× compared to AtC. This stark contrast validates AtC’s unique strength—simultaneously achieving accurate absolute score calibration and maintaining ordinal consistency with human consensus, a capability not demonstrated by existing human-centered assessment approaches. Table 4: Real-world results on Dots dataset.

Bold indicates the best performance among human-only assessment (s∗ ), model-only assessment (sp ), and AtC assessment (b s), respectively. The red denotes the optimal performance across all methods for the given metric. Kendall τ ↑

Wasserstein↓

KL↓

MSE↓

s∗ / sp / b s

s∗ / sp / b s

s∗ / sp / b s

s∗ / sp / b s

HRA-G 0.917 / 0.923 / 0.940 0.922 / 0.922 / 0.943 HRA-E HRA-N 0.917 / 0.923 / 0.934 AtC CrowdBT 0.894 / 0.923 / 0.927 CrowdTCV 0.899 / 0.923 / 0.926 BTL 0.917 / 0.923 / 0.937 TCV 0.917 / 0.923 / 0.933

6.97 / 2.53 / 2.53 6.98 / 2.53 / 2.53 7.05 / 2.53 / 2.53 5.32 / 2.53 / 2.55 5.73 / 2.53 / 2.55 6.72 / 2.53 / 2.53 6.81 / 2.53 / 2.54

123.13 / 0.881 / 0.860 123.16 / 0.881 / 0.861 125.50 / 0.881 / 0.859 72.84 / 0.881 / 0.865 84.74 / 0.881 / 0.865 115.37 / 0.881 / 0.860 117.81 / 0.881 / 0.878

65.08 / 11.75 / 9.61 65.16 / 11.75 / 9.59 66.44 / 11.75 / 9.75 39.28 / 11.75 / 10.24 44.97 / 11.75 / 10.24 60.51 / 11.75 / 9.73 62.09 / 11.75 / 10.18

GPPL Rank-SVM BARCW

– / – / 64.50 – / – / 61.20 – / – / 64.62

– / – / 126.62 – / – / 133.70 – / – / 24.09

– / – / 4220.36 – / – / 3814.49 – / – / 4236.16

Stage-1 Method

– / – / 0.931 – / – / 0.923 – / – / 0.940

Remark. Our work is related to methods that integrate human and machine decision-making to improve prediction quality [48, 51, 77, 95]. Beyond the assessment scenarios examined in this paper, the AtC framework connects to a range of applied domains where reliable evaluation requires reconciling heterogeneous human expertise with automated predictions. In logistics and operations research, platforms must continuously assess latent service quality metrics such as route difficulty, courier workload, and delivery reliability [58–60, 74, 86, 90, 97, 98], where ground-truth measurements are expensive to obtain at scale, making principled fusion of operational data with frontline worker judgments a natural fit for the aggregate-then-calibrate paradigm. In urban and spatial computing, evaluating the effectiveness of intelligent transportation strategies [82, 83, 85], infrastructure robustness [20, 73, 94], and public health outcomes [33, 46, 47] similarly involves latent targets that are only partially observable through both sensor-driven models and domain expert assessments. The growing adoption of foundation models in sensing and perception tasks [32, 34, 35, 49, 55, 56, 72, 75, 87, 99] and in large-scale graph mining systems [21, 22, 26–28, 89] further amplifies the demand for systematic human-AI evaluation frameworks, as these powerful models require calibrated human oversight to ensure trustworthy deployment. More broadly, scientific disciplines such as computational biology [52, 53, 91, 93] and statistical learning under privacy constraints [70] increasingly rely on combining expert curation with computational evidence, calling for principled statistical and computational methods that can aggregate diverse signals under uncertainty. These settings 35

share the core structure that AtC addresses: ground truth is costly, delayed, or partially observable, and human judgments and model outputs carry complementary but imperfect information. D.7

Robustness Results for All Baseline Methods on Real-world Dataset.

This section provides supplementary results to complement the main analysis presented in Section 4. We present the complete set of radar plots for the six additional baseline aggregation methods: HRA-G, HRA-N, BTL, TCV, CrowdBT, and CrowdTCV. The experimental methodology, including the four types of image corruption and the evaluation metric (Kendall’s Tau), is identical to the one used for the HRA-E analysis in the main body of the paper. As illustrated in Figures 6 7 9 10 11 12, a consistent trend emerges across all baselines. The primary finding from the main text is strongly reinforced: the AtC framework, when calibrated using aggregated rankings (b s (Rank)), demonstrates superior robustness compared to all alternatives. In most cases, the performance of b s (Rank) remains high and degrades gracefully, while the raw objective model scores (sp ) are significantly more vulnerable to input corruptions. This consistent outperformance across a diverse set of aggregation algorithms underscores the general applicability and reliability of our proposed ranking-based calibration strategy. 1.5

1.0

2.0

0.5 2.5 0.6

0.7

0.8

0.9

s (Rank) s (Rate) sp s*

1.0

0.4 0.5

0.6

0.0 5.0

0.8

0.65

(a) BLUR

0.7

0.4

s (Rank) s (Rate) sp 0.25 s *

0.9

0.35

0.45

1.0

0.5

0.2 0.8

0.6

4.5

4.0

0.7

0.3

0.55

3.0 3.5

0.35

0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.8

0.6 0.65

(b) LOCAL BLUR

0.7

0.35

0.5

0.2

0.55

0.75

1.0

0.4 0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.2

0.55

0.8

0.6

0.75

0.65

(c) LOCAL NOISE

1.0

0.7

0.75

(d) WHITEOUT

Figure 9: Radar plots under different noise conditions (CrowdTCV). 1.5

1.0

2.0

0.5 2.5 0.6

0.7

0.8

0.9

s (Rank) s (Rate) sp s*

1.0

0.4 0.5

0.6

0.0 5.0

0.8

0.65

(a) BLUR

0.7

0.4

s (Rank) s (Rate) sp 0.25 s *

0.9

0.35

0.45

1.0

0.5

0.2 0.8

0.6

4.5

4.0

0.7

0.3

0.55

3.0 3.5

0.35

0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.8

0.6 0.65

(b) LOCAL BLUR

0.7

0.35

0.5

0.2

0.55

0.75

1.0

0.4 0.45

0.6

0.7

0.3 0.8

s (Rank) s (Rate) sp 0.25 s *

0.9

0.2

0.55

0.8

0.6

0.75

0.65

(c) LOCAL NOISE

1.0

0.7

0.75

(d) WHITEOUT

Figure 10: Radar plots under different noise conditions (BTL). 1.5

1.0

2.0

0.5 2.5 0.6

0.7

0.8

0.9

1.0

0.0

3.0 5.0 3.5 4.0

4.5

(a) BLUR

s (Rank) s (Rate) sp s*

0.4

0.35

0.45 0.5 0.6

0.7

0.3 0.8

0.9

1.0

0.8 0.65

0.7

0.35

0.45 0.5

0.2

0.55 0.6

0.4

s (Rank) s (Rate) sp 0.25 s *

0.6

0.7

0.3 0.8

0.9

0.8

0.6 0.65

(b) LOCAL BLUR

1.0

0.7

0.4 0.5

0.6

0.7

0.3 0.8

0.75

Figure 11: Radar plots under different noise conditions (TCV).

s (Rank) s (Rate) sp 0.25 s *

0.9

1.0

0.2

0.55

(c) LOCAL NOISE

36

0.35

0.45

0.2

0.55

0.75

s (Rank) s (Rate) sp 0.25 s *

0.8

0.6 0.65

0.7

0.75

(d) WHITEOUT

1.5

1.0

2.0

0.5 2.5 0.6

0.7

0.8

0.9

1.0

0.0

3.0 5.0 3.5 4.0

4.5

(a) BLUR

s (Rank) s (Rate) sp s*

0.4

0.35

0.45 0.5 0.6

0.7

0.3 0.8

0.9

1.0

0.8 0.65

0.7

0.35

0.45 0.5

0.2

0.55 0.6

0.4

s (Rank) s (Rate) sp 0.25 s *

0.6

0.7

0.3 0.8

0.9

0.8

0.6 0.65

(b) LOCAL BLUR

1.0

0.7

0.4

0.75

0.5 0.6

0.7

0.3 0.8

1.0

0.2 0.8

0.6

Figure 12: Radar plots under different noise conditions (CrowdBT).

s (Rank) s (Rate) sp 0.25 s *

0.9

0.55

(c) LOCAL NOISE

37

0.35

0.45

0.2

0.55

0.75

s (Rank) s (Rate) sp 0.25 s *

0.65

0.7

0.75

(d) WHITEOUT

Related documents

• An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration#200492
• Revisiting the SciPop Scale: Measurement Alignment, Scoring Rules, and Cross-Context Comparability#672352
• Revisiting the SciPop Scale: Measurement Alignment, Scoring Rules, and Cross-Context Comparability#223261
• TOOD: Task-Aware Out-of-Distribution Score Calibration for Continual Learners#422241
• A Ranking Approach for Measuring Calibration#1070599