A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection Gabriel Singer†*1,2 , Samuel Gruffaz†*2 , Olivier Vo Van 1 , Nicolas Vayatis 2 , Argyris Kalogeratos 2 1
SNCF, Paris, France Tampere University, Finland, Biostatistics group 3 Université Paris-Saclay, ENS Paris-Saclay, Centre Borelli, France
arXiv:2607.24622v1 [stat.ML] 27 Jul 2026
2
∗ Equal contribution. [email protected]
† Corresponding authors. [email protected]
Abstract We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists. Existing models only partially address this problem: they either capture class-dependent errors but ignore item difficulty, or they model item difficulty without capturing class-dependent errors. To fill this gap for imbalanced datasets in crowdsourcing, we introduce a generative aggregation model combining item difficulty with class-dependent annotator competence. The model allows both annotator abilities and item difficulties to vary across classes. We then revisit Condorcet’s Jury Theorem in the class-imbalanced setting. We also show that majority voting asymptotically preserves the underlying class proportion. We evaluate our model on 33 real-world crowdsourcing datasets, covering multiclass tasks such as images and text, as well as two large-scale regimes: large-scale annotation datasets, with many annotations per item, and large-scale item datasets, with a large number of annotated instances. Across these diverse settings, our model consistently achieves the highest minority recall while remaining competitive in balanced accuracy, making it particularly relevant when rare-label recovery is the primary objective.
1
Introduction
In many real-world inspection systems, the labels of interest are also the rarest ones. This creates a tension: large annotated datasets are needed to train reliable models, but expert annotation of rare events is costly, and difficult to scale. Crowdsourcing provides an attractive alternative by collecting multiple labels from non-expert annotators at lower cost and larger scale (Snow et al., 2008). In a balanced setting, majority voting and weighted majority voting enjoy consistency guarantees (Berend and Kontorovich, 2014; Berend and Paroush, 1998). However, imbalanced settings break part of this picture. Majority voting can achieve high overall accuracy, while missing rare but critical minority-class defects. Class imbalance also changes the structure of the crowd. In balanced binary settings, it is natural to distinguish reliable from unreliable annotators (Berend and Kontorovich, 2014; Berend 1
and Paroush, 1998). In imbalanced settings, this distinction is not enough to capture all possibilities. In fact, an annotator may be reliable on both classes, unreliable on both classes, reliable only on the majority class, or only on the minority class. Dawid–Skene’s model captures annotator confusion matrices and can therefore represent class-dependent error patterns (Dawid and Skene, 1979). Yet, this does not explicitly model item difficulty. The Rasch model was the first to introduce item difficulty and annotator ability modeling (Rasch, 1960); GLAD later adapted this principle by adding the estimation of the latent true labels (Whitehill et al., 2009). However, they both consider a single ability parameter per annotator, which prevents them from distinguishing majority-class competence from minority-class competence. Thus, Dawid–Skene is class-dependent, but not difficulty-aware, while GLAD is difficulty-aware but not class-dependent. Moreover, most aggregation methods Ustalov et al. (2024); Liu et al. (2026) are accuracy-driven rather than designed for minority-class recovery. Contributions.
Our contributions are as follows:
1. Theoretical analysis of majority vote under class imbalance. We provide interpretable conditions guaranteeing a Condorcet jury theorem in the class-imbalanced setting (Theorem 3.1) and characterize how majority voting asymptotically preserves the underlying minority-class proportion (Corollary 3.2). 2. Mixed-effects generative model. We propose a new generative model of annotator competence and item difficulty that captures how both vary across classes. We complete our analysis by proving that, the M-step of the EM algorithm admits a unique solution (Proposition 4.2). 3. Large-scale empirical evaluation. We evaluate CC-Rasch against four baselines on 33 diverse real-world crowdsourcing datasets, constituting, to the best of our knowledge, one of the largest benchmarks. CC-Rasch achieves consistent gains in minority recall across evaluation regimes, while controlled synthetic experiments confirm its robustness as minorityclass specialists become increasingly rare. In what follows: Section 2 reviews the Rasch and GLAD models. Section 3 analyzes majority voting under class imbalance. Section 4 introduces CC-Rasch, and Section 5 evaluates it on synthetic and real-world datasets. Section 6 concludes the paper. Complete proofs are provided in the supplementary material.
2
Background
The crowdsourcing problem We consider that each item has an unknown ground-truth label Y ∈ [K], where [K] := {0, . . . , K − 1} and K denotes the number of classes. As a reference example, one may think an item to be an image and of its ground-truth label as indicating whether an object appears in it; in this case K = 2. Let RR = {1, . . . , R} be the set of R annotators. Each item receives at most R noisy labels Y1 , . . . , YR ∈ [K]. A label aggregation rule combines these noisy labels into a single prediction Yb ∈ [K]. The simplest aggregation rule is majority voting, which outputs the class with most votes: Yb MV ∈ arg max R
k∈[K]
R X r=1
2
1{Yr = k} .
More generally, probabilistic aggregation methods such as GLAD or Rasch that will be presented later, estimate the posterior distribution P(Y = k | Y1 , . . . , YR ) and predict the most likely class; Yb ∈ arg max P(Y = k | Y1 , . . . , YR ) . k∈[K]
Rasch and GLAD models. This work belongs to the family of probabilistic approaches that jointly model annotator competence and item difficulty. One of the earliest and most influential models of this type is the Rasch model (Rasch, 1960), which is widely used in item response theory. Consider a set of m annotators and n items, with [n] := {1, . . . , n}. Each annotator r ∈ [m] is associated with an ability parameter θr⋆ ∈ R, while each item i ∈ [n] is associated with a difficulty parameter βi⋆ ∈ R. Let Yri denote the response provided by respondent r to item i, and let Yi denote the corresponding correct answer. We define the correctness indicator as Cri := 1{Yri = Yi } ∈ {0, 1},
(1)
where, for an event A, 1{A} equals 1 if A occurs and 0 otherwise. The Rasch model assumes: Cri ∼ Bernoulli (σ(θr⋆ − βi⋆ )) , where for any x ∈ R, σ(x) :=
1 , 1 + exp(−x)
(2)
(3)
is the logistic function. Thus, the probability of a correct response increases with the difference between the annotator’s ability and the item’s difficulty. In particular, when θr⋆ − βi⋆ is large, this probability is close to one. In educational testing, for example, Cri = 1 indicates that student r correctly answers question i, θr⋆ represents the student’s ability, and βi⋆ represents the difficulty of the question. A limitation of the Rasch model in crowdsourcing applications is that the correctness indicators Cri can only be observed when the correct answers, or equivalently the true item labels, are known. In standard label aggregation problems, however, these true labels are precisely the quantities that must be inferred. The GLAD model (Whitehill et al., 2009) extends the ability–difficulty principle to crowdsourcing by treating the true item labels as latent variables and estimating them jointly with annotator abilities and item difficulties. Let O ⊆ [m] × [n] denote the set of observed annotator–item pairs. For each item i ∈ [n], define the set of annotators who provided a label for item i: Ωi := {r ∈ [m] : (r, i) ∈ O} .
(4)
Each item i has an unknown binary true label Yi ∈ {0, 1}. For every observed pair (r, i) ∈ O, annotator r provides a noisy label Yri ∈ {0, 1}. Each annotator r is associated with an ability parameter αr ∈ R: when αr > 0, the annotator is more likely to provide the correct label than an incorrect one; when αr = 0, the annotator behaves as a random guesser; and when αr < 0, the annotator is more likely to provide an incorrect label. Moreover, each item i is associated with an inverse-difficulty parameter βi > 0, so larger values correspond to easier items. Conditionally on the latent true labels and the model parameters, the observed annotations are assumed to be independent. GLAD models the probability that annotator r correctly labels item i in a very similar way as Rasch model does. Given Cri := 1{Yri = Yi } , they define Cri | Yi ∼ Bernoulli (σ(αr βi )) . 3
(5)
Note that easier items having larger βi values amplify the effect of annotator expertise. Unlike in the Rasch model, the correctness indicators Cri are not directly observed in GLAD because the true labels Yi are unknown. The latent labels, annotator abilities, and item inverse difficulties must therefore be estimated jointly from the observed annotations. Other aggregation models. Dawid–Skene (Dawid and Skene, 1979) models each annotator through a class-dependent confusion matrix and jointly estimates the latent labels and annotator reliabilities using the EM algorithm. Unlike GLAD, it captures class-dependent annotation behavior but does not model item difficulty. More recently, CrowdFM (Liu et al., 2026) introduced a foundation-model-based approach to label aggregation. Imbalanced crowdsourcing. Very few works have explicitly investigated class imbalance in crowdsourcing, which is the setting this work focuses on. Zhang et al. (2015) introduced the imbalanced multiple noisy labeling setting, in which annotators may exhibit different class-conditional labeling accuracies, and proposed the PLAT algorithm. More recently, Gil-Gonzalez et al. (2026) proposed the CCGPFL algorithm, a correlated Gaussian-process model trained with a weighted variational objective. Unlike classical label aggregation methods, CCGPFL additionally relies on item features to model instance-dependent annotator reliability.
3
Majority voting under class-imbalance
Let the class-imbalance proportion be defined as: µ := min P(Y = k) ∈ ]0, k∈[K]
1 ]. K
We denote by jmin and jmax , respectively, the minority and majority classes, i.e. jmin := arg mink∈[K] P(Y = k) and jmax := arg maxk∈[K] P(Y = k). Crowd decomposition. For each annotator r ∈ RR , we define the probability of correctly labeling an item of class k as p(k) r := P(Yr = k | Y = k). and, we define for each class k ∈ [K] the subsets of good and bad annotators: (k)
1 GR := {r ∈ RR : p(k) r > 2 + δk }, (k)
1 BR := {r ∈ RR : U (k) ≤ p(k) r ≤ 2 }.
Here, δk ∈ [0, 12 ] is a minimum competence above random guessing for class k, while U (k) ∈ [0, 12 [ is a uniform (with respect to R) lower bound on the class-k accuracies of the annotators. Moreover, we call an annotator r ∈ RR a minority-class specialist when (j
)
r ∈ GR min ,
(6) (j
)
and respectively a majority-class specialist when r ∈ GR max . For example, consider the binary case K = 2 and assume that the minority class is indexed by (0) (1) jmin = 0. Then, GR denotes the minority specialists, while GR denotes the majority specialists.
4
Imbalanced Condorcet Jury Theorem. We first establish conditions on the crowd composition under which majority voting asymptotically recovers every class, including the minority classes, regardless of the severity of the underlying class imbalance. MV b For each k ∈ [K], let eR (k) := P YR ̸= Y | Y = k the conditional error and πk := P(Y = k) the ground truth distribution. Throughout the paper, we assume that the random variables (Yr )r∈RR are conditionally independent given Y . Theorem 3.1. Assume that for any k ∈ [K] (k)
(k)
GR B (1 + 2δk ) + 2 R U (k) > 1. R R
lim inf R→∞
(7)
It follows that lim P(YbRMV ̸= Y ) = 0.
R→∞
For every R ≥ 1, the law of total probability gives
P YbRMV ̸= Y
=
X
πk eR (k).
k∈[K]
Under condition (7), Proposition D.3 of Singer et al. (2026) applies to each class k ∈ [K], yielding lim eR (k) = 0. R→∞
Since K is finite,
lim P YbRMV ̸= Y
R→∞
=
X
πk lim eR (k) = 0. R→∞
k∈[K]
Since for any R ≥ 1
P YbRMV = Y it follows that:
= 1 − P YbRMV ̸= Y ,
lim P YbRMV = Y
R→∞
= 1.
This is a generalization to the multidimension and multiclass case of Proposition D.3 in Singer et al. (2026). Interpretation: Condition (7) provides for each class k, a lower bound on the number of class-k specialists required to compensate for the presence of adversarial annotators. Indeed, for sufficiently large R, the condition implies (k) R − 2 BR U (k) (k) GR > . 1 + 2δk In the extreme case U (k) ≈ 0, the contribution of the non-specialists becomes negligible, and the condition approximately reduces to R (k) GR > . 1 + 2δk Therefore, if the crowd is made of the non-specialists that are arbitrarily inaccurate on class k, consistency requires to have a sufficiently large fraction of specialists. 5
Asymptotic preservation of class imbalance. A desirable property of an aggregator is to preserve the underlying class distribution. This type of property has previously been studied by Zhang et al. (2015) in the binary case with a homogeneous crowd. Our result extends their analysis to heterogeneous multiclass crowds. Corollary 3.2. Let R ≥ 1 and µ be the imbalanced proportion of the ground truth Y . Define the majority vote imbalanced proportion: for any R ≥ 1,
b MV = k . µMV R := min P YR k∈[K]
Assume that Eq. (7) holds, then: lim µMV R = µ.
R→∞
Since (7) holds: limR→∞ P YbRM V ̸= Y
= 0. Plus, for any R ≥ 1 and any k ∈ [K] we have that
P YbRM V = k − P (Y = k) ≤ P YbRM V ̸= Y , thus for any R ≥ 1 : mR := max
n
k∈K
P YbRM V = k − P (Y = k)
o
→ 0.
R→∞
We conclude by noticing that, for any R ≥ 1:
b MV = k − min P(Y = k) µMV R − µ = min P YR k∈[K]
k∈[K]
≤ max P YbRMV = k − P(Y = k) k∈[K]
4
The CC-Rasch Model
4.1
Motivation
→ 0.
R→∞
Theorem 3.1 essentially shows that majority voting requires the crowd to be sufficiently competent on each class. In particular, performing sufficiently well on the minority class may be difficult to guarantee in practice under class imbalance. We therefore propose a probabilistic model that accounts for both annotation difficulty and the class-dependent ability of each annotator to recover the ground truth. Our second motivation comes from the observation that GLAD does not distinguish the probability of error of an annotator depending on the value taken by the latent ground truth label. Consequently, GLAD cannot account for class-dependent annotation behaviors, which are particularly important in the imbalanced setting. More formally we have: Proposition 4.1. Under the GLAD model Eq. 5, the sensitivity is equal to specificity. For any r ∈ [n] and for any i ∈ Ωr , we have that: P (Yri = 1 | Yi = 1) = P (Yri = 0 | Yi = 0)
6
4.2
The model
We introduce CC-Rasch, which stands for Class-Conditional Rasch. Let σ be the sigmoid function (Eq. (3)) and Yi ∈ {0, . . . , K − 1} the latent true class of item i, Yir ∈ {0, . . . , K − 1} the label given by annotator r on item i. Let, for all (i, r) ∈ Ωr × [R] Zir := 1{Yir = Yi } the indicator that annotator r recovers the true class of item i. We assume that: Zir | Yi = k ∼ Bernoulli(pir,k ),
pir,k = σ(αr,k − βi,k ).
Inspired by mixed-effects models (Mun et al., 2021), we split each term into a class-level average and an individual deviation from it: αr,k = µα,k + gr,k ,
βi,k = µβ,k + hi,k ,
where µα,k is the crowd average ability. While µβ,k is the average item difficulty for class k. gr,k and hi,k capture how far annotator r and item i deviate from their respective average. To interpret µα and P P µβ as the average ability (respectively difficulty) we impose for any k ∈ [K]: i hi,k = r gr,k = 0. Following Whitehill et al. (2009); Liu et al. (2026), we fix this translation invariance with Gaussian priors: 2 2 gr,k ∼ N (0, σα,k ), hi,k ∼ N (0, σβ,k ). The conditional law of the observed annotation follows: 1{y=Y } P(Yir = y | Yi ) = pir,k i
1 − pir,k K −1
1{y̸=Yi }
,
(8)
i.e. we spread the errors uniformly over the K − 1 remaining classes. We estimate the model parameters using the Expectation-Maximization (EM) algorithm (Dempster et al., 1977), a widely used approach for maximum-likelihood estimation in latent-variable models. EM alternates between computing the conditional distribution of the latent variables given the observed data and the current parameter estimates (E-step), and maximizing an auxiliary function that corresponds to the conditional expected log-likelihood (M-step). We now detail these two steps for our model. The unknown ground-truth label Yi ∈ [K], is treated as a latent variable. Let (πk )k∈[K] denote the ground-truth class distribution, where πk := P(Y = k). Here the CC-Rasch parameters that one need to estimate are: θ := (µα , µβ , G, H) , where µα , µβ ∈ RK and G = (gr,k )r,k ∈ Rm×K and H = (hi,k )i,k ∈ Rn×K . Where m denotes the number of annotators, n the number of items and K the number of classes. Conditional on Yi = k, the annotation likelihood is 1{y=k}
Pθ (Yri = y | Yi = k) = pir,k
1 − pir,k K −1
1{y̸=k}
.
Let ℓir,k (θ) := log Pθ (Yri | Yi = k). We estimate θ using an EM algorithm applied to a penalized complete- data log-likelihood, as it has been done in Whitehill et al. (2009). The latent variables handled by the E-step are the unknown labels Y1 , . . . , Yn ; the quantities µα , µβ , G, and H are optimized in the M-step. E-step.
Given the parameters at step s, θ(s) , the E-step computes the latent posterior: (s)
qik := Pθ(s) (Yi = k | (Yr,i )r ) . 7
M-step.
For ξ ∈ {α, β}, define 2 −1 λξ,µ := (2σξ,µ ) ,
2 −1 λξ,k := (2σξ,k ) ,
k ∈ [K],
and let Λξ := diag(λξ,1 , . . . , λξ,K ). Define also 1/2
2 2 R(θ) := λα,µ ∥µα ∥22 + λβ,µ ∥µβ ∥22 + ∥GΛ1/2 α ∥F + ∥HΛβ ∥F .
A direct calculation shows that at iteration s: Qs (θ) := EY |y; θ(s) [log pθ (y, Y )] − R(θ) n K−1 X X (s) X = ℓir,k (θ) − R(θ). q log πk +
(9)
ik
i=1 k=0
r∈Ωi
The parameter update is therefore θ(s+1) ∈ arg max Qs (θ),
(10)
θ∈C
where
(
C :=
(µα , µβ , G, H) :
m X
gr,k =
r=1
n X
)
hi,k = 0, k ∈ [K] .
i=1
(s)
Proposition 4.2. Let {qik }i,k be the fixed posterior latent and fixed class proportions πk , the M -step problem defined in Eq. (10) admits a unique maximizer θ(s+1) over C. The proof relies mainly on the following Theorem, the interested reader can find a proof in Allaire (2012): Theorem 4.3. Let C ⊆ Rd be a nonempty, closed, and convex set, and let f : C → R. If f is convex and coercive on C, then f attains its minimum on C. Lemma 4.4. Let C be the set defined as (
(µα , µβ , G, H) :
m X
gr,k =
r=1
n X
)
hi,k = 0, k ∈ [K] .
i=1
C is closed and convex. Proof. Let v = (µ′α , µ′β , G′ , H ′ ) ∈ C.
u = (µα , µβ , G, H) ∈ C, For any λ ∈ [0, 1] and every k ∈ [K], m X
′ λgr,k + (1 − λ)gr,k =λ
r=1
and similarly,
m X
gr,k + (1 − λ)
r=1
n X
r=1
λhi,k + (1 − λ)h′i,k = 0.
i=1
8
m X
′ gr,k = 0,
It follows that λu + (1 − λ)v ∈ C To prove closedness, let (uj )j≥1 ⊆ C such that uj → u. Writing uj = (µα,j , µβ,j , Gj , Hj ), we have, for every j and every k ∈ [K], m X
n X
gr,k,j = 0,
r=1
hi,k,j = 0.
i=1
Because these are finite sums, we may pass to the limit: m X
m X
gr,k = lim
j→∞
r=1
and similarly
n X
gr,k,j = 0,
r=1
hi,k = 0.
i=1
Therefore u ∈ C, and C is closed. The preceding lemma establishes that the feasible set C is closed and convex; it is also clearly nonempty, since it contains the zero parameter vector. We can therefore use Theorem 4.3 to study the M -step. More precisely, maximizing Qs over C is equivalent to minimizing fs := −Qs over the same set. It remains to show that fs is coercive and strictly convex. Coercivity will guarantee the existence of a minimizer, while strict convexity will ensure its uniqueness. (s)
Proof. First, fs is convex. For fixed posterior weights {qik }i,k , the Hessian of Qs with respect to θ = (µα , µβ , G, H) is ∇2θ Qs = −
n X X
(s) X
qik
i=1 k∈[K]
2 pir,k (1 − pir,k ) xir,k x⊤ ir,k − ∇θ R,
r∈Ωi
where xir,k := ∇θ ηir,k . Since pir,k (1 − pir,k ) ≥ 0 and xir,k x⊤ ir,k is symmetric positive semidefinite. Moreover R is strictly convex. Hence Qs is strictly concave and fs = −Qs is strictly convex. Next, fs is coercive. Since log πk ≤ 0
and
ℓir,k (θ) ≤ 0,
we have that: −Qs (θ) ≥ R(θ). R(θ) −→ +∞
as
9
∥θ∥ −→ ∞.
(11)
Therefore using: (11) it follows that: −Qs (θ) −→ +∞
as
∥θ∥ −→ ∞,
so fs is coercive. Finally, C is nonempty, closed, and convex by Lemma 4.4. Thus fs attains a minimum on C, and strict convexity guarantees that this minimizer is unique. Equivalently, Qs admits a unique maximizer on C.
5
Experimental Evaluation
Our experimental analysis is made of three parts. We first evaluate aggregation methods on a controlled synthetic crowdsourcing benchmark designed to reproduce two sources of imbalance: class imbalance in the ground-truth labels and class-dependent imbalance in annotator expertise. Then we evaluate our CC-Rasch model across 27 real-world datasets. Finally, we analyze how Gold Majority Vote (Ustalov et al., 2024) evolves as the number of available gold labels increases, and compare it performances on recall with CC-Rasch. In what follows, we define the class-imbalance ratio as maxk∈[K] P(Y = k) Imb := . mink∈[K] P(Y = k) A value of Imb = 1 corresponds to a balanced class distribution, while a large value indicate a stronge class imbalance.
5.1
Synthetic Data
Dataset. The synthetic dataset contains 3,000 items, with an imbalanced ratio of Imb ≈ 7 belonging to the minority class. Each item is annotated by 8 annotators. Annotators are partitioned into 4 accuracy profiles: majority-class specialists, minority-class specialists, annotators accurate on both classes, and annotators inaccurate on both classes. Items are assigned either an easy or a hard difficulty level. More formally: For each item i, we sample a ground-truth label Yi ∼ Bernoulli(π), where class 1 is the minority class, and an item-difficulty indicator Hi ∼ Bernoulli(ρ). Each annotator r belongs to a type Tr ∈ {good, maj, bad, min}, with fixed type proportions. For every observed annotation (i, r), P(Yir = Yi | Yi = k, Hi = h, Tr = t) = qt,k − hδ, where qt,k is the class-conditional reliability of annotator type t and δ is the hard-item penalty. Otherwise, the annotator outputs 1 − Yi . Each item receives Ni = 1 + Poisson(λ − 1) annotations, with annotators sampled uniformly without replacement. In the controlled sweep, the proportions of reliable and unreliable annotators are fixed, while the mass of majority and minority specialists is redistributed between the two groups. 10
Percentage of minority-specialist annotators 45
35
25
15
5
100
CC-Rasch PLAT DS
Minority Recall
80
60
CrowdFM GLAD
50% recall
40
20
MV
pmaj + pmin = 50% pgood = pbad = 25%
0 5
15
25
35
45
Percentage of majority-specialist annotators Figure 1: Minority-class recall as the proportion of majority-class specialists pmaj increases and the proportion of minority-class specialists pmin decreases. Their total proportion is fixed at pmaj + pmin = 50%, while pgood = pbad = 25%. Points report mean minority recall and error bars represent 95% confidence intervals. Robustness to fewer minority specialists. We study the robustness of aggregation methods when the number of minority-class specialists (see Eq. (6)) decreases while the number of majorityclass specialists increases. Results. Figure 1 shows that minority recall deteriorates for standard aggregation methods as minority expertise becomes scarce, while CC-Rasch remains more stable.
5.2
Real World datasets
Datasets. We evaluate CC-Rasch on a broad collection of real-world crowdsourcing datasets covering natural language processing, computer vision, and general classification tasks. The natural language processing benchmarks include Recognizing Textual Entailment (RTE), TREC and NIST– TREC relevance judgments, product classification, web relevance assessment, HITSpam, and several sentiment analysis datasets, including PosSent, Weather Sentiment–AMT, and Sentiment Popularity– AMT (Snow et al., 2008). We also include the Movie Reviews 4-class dataset and three variants of the Medical CrowdTruth benchmark: Medical CrowdTruth All, Cause, and Treat. We additionally consider common benchmarks such as CF, CF⋆ , MS, SP, and the ZC variants (ZC-all, ZC-in,
11
Method
Imb. ≥ 3
All
Ann./item ≥ 10
Ann./item < 5
Recall ↑
Bal.-Acc. ↑
Recall ↑
Bal.-Acc. ↑
Recall ↑
Bal.-Acc. ↑
Recall ↑
Bal.-Acc. ↑
Majority Vote
0.732 ± 0.209
0.809 ± 0.126
0.732 ± 0.138
0.787 ± 0.117
0.726 ± 0.233
0.801 ± 0.140
0.686 ± 0.206
0.786 ± 0.111
Dawid–Skene (1979)
0.811 ± 0.204
0.844 ± 0.123
0.803 ± 0.130 0.814 ± 0.102 0.830 ± 0.233
0.851 ± 0.142
0.681 ± 0.184
0.785 ± 0.093
GLAD (2009)
0.720 ± 0.271
0.812 ± 0.140
0.776 ± 0.161
0.806 ± 0.111
0.732 ± 0.278
0.813 ± 0.150
0.524 ± 0.338
0.732 ± 0.150
CrowdFM (2026)
0.739 ± 0.211
0.817 ± 0.126
0.734 ± 0.144
0.791 ± 0.107
0.754 ± 0.232
0.819 ± 0.138
0.621 ± 0.204
0.762 ± 0.109
PLAT
0.859 ± 0.199 0.821 ± 0.125
0.881 ± 0.118
0.781 ± 0.116
0.840 ± 0.247
0.818 ± 0.142 0.873 ± 0.153 0.792 ± 0.112
CC-Rasch (OURS) 0.859 ± 0.190 0.847 ± 0.127 0.888 ± 0.064 0.802 ± 0.106 0.851 ± 0.239 0.853 ± 0.149 0.815 ± 0.107 0.793 ± 0.097
Table 1: Performance on binary datasets. Method
Imb ≥ 3
All
Ann/item ≥ 10
Ann/item < 5
Recall ↑
Bal-Acc ↑
Recall ↑
Bal-Acc ↑
Recall ↑
Bal-Acc ↑
Recall ↑
Bal-Acc ↑
Majority Vote
0.659 ± 0.341
0.740 ± 0.119
0.185 ± 0.128
0.745 ± 0.043
0.520 ± 0.476
0.812 ± 0.128
0.831 ± 0.143
0.721 ± 0.048
Dawid–Skene (1979)
0.691 ± 0.316
0.765 ± 0.109
0.296 ± 0.128
0.739 ± 0.010
0.582 ± 0.418 0.826 ± 0.123 0.903 ± 0.093 0.772 ± 0.021
GLAD (2009)
0.675 ± 0.344
0.760 ± 0.112
0.185 ± 0.128
0.743 ± 0.031
0.520 ± 0.476
0.819 ± 0.126
0.877 ± 0.108
CrowdFM (2026)
0.656 ± 0.375
0.758 ± 0.121
0.139 ± 0.192
0.738 ± 0.039
0.479 ± 0.524
0.811 ± 0.130
0.881 ± 0.102
0.762 ± 0.044
CC-Rasch (OURS) 0.722 ± 0.295 0.770 ± 0.115 0.370 ± 0.257 0.754 ± 0.037
0.576 ± 0.412
0.823 ± 0.122
0.871 ± 0.138
0.771 ± 0.029
0.761 ± 0.040
Table 2: Performance on multiclass datasets. Method
Large-scale annotations Recall ↑
Bal-Acc ↑
Large-scale items F1 ↑
Recall ↑
Bal-Acc ↑
F1 ↑
Majority Vote
0.712 ± 0.206
0.796 ± 0.158
0.798 ± 0.158
0.747 ± 0.235
0.794 ± 0.204
0.794 ± 0.200
Dawid–Skene (1979)
0.766 ± 0.237
0.835 ± 0.146
0.831 ± 0.149
0.723 ± 0.302
0.822 ± 0.184
0.830 ± 0.185 0.818 ± 0.186
GLAD (2009)
0.699 ± 0.306
0.802 ± 0.175
0.793 ± 0.198
0.740 ± 0.252
0.806 ± 0.190
CrowdFM (2026)
0.722 ± 0.223
0.805 ± 0.162
0.804 ± 0.163
0.724 ± 0.257
0.794 ± 0.203
0.797 ± 0.199
CC-Rasch (OURS)
0.834 ± 0.176
0.837 ± 0.148
0.812 ± 0.156
0.803 ± 0.234
0.820 ± 0.191
0.793 ± 0.196
Table 3: Performance on large-scale datasets. and ZC-us) (Liu et al., 2026). For computer vision, we include Bird, Dog, Face, LabelMe, and CIFAR-10H, which have been widely used to study annotator expertise, item difficulty, and human uncertainty in visual crowdsourcing (Whitehill et al., 2009; Welinder et al., 2010; Rodrigues et al., 2013; Russell et al., 2008; Peterson et al., 2019). We also consider the challenging Duchenne smile dataset, referred to as the WSCM dataset, which was used in Whitehill et al. (2009). Overall, the benchmark contains both binary and multiclass tasks and covers a wide range of class-imbalance levels, crowd sizes, dataset sizes, and annotation densities. This diversity allows us to evaluate the aggregation methods in both sparse and highly redundant annotation regimes, as well as on large-scale datasets in terms of either the number of items or the number of annotations. Methodology. We compare CC-Rasch with five aggregation baselines: PLAT, specifically designed for imbalanced label aggregation, and four widely used methods, namely majority voting (MV), Dawid–Skene (DS), GLAD, and CrowdFM, a recent deep learning-based approach. CCGPFL (GilGonzalez et al., 2026) is not included because it requires item features to model instance-dependent annotator reliability. Since such features are unavailable for many datasets in our benchmark, CCGPFL cannot be evaluated under our common annotation-only setting. We use the official Crowd-Kit implementations of MV, DS, and GLAD (Ustalov et al., 2024), and the authors’ official implementation of CrowdFM. 12
For each method, we evaluate the aggregated labels against the available ground-truth labels. Values are reported as mean ± standard deviation across datasets after averaging each dataset over P 4 random seeds. Let nk := ni=1 1{yi = k} , Kobs := {k ∈ [K] : nk > 0} and let ymin denote the least frequent ground-truth class. The empirical recall of class k ∈ Kobs is Pn b k := R
bi = k} i=1 1{yi = k, y
nk
.
Minority recall and balanced accuracy are then defined by by , \ min := R Recall min
\ := BalAcc
X 1 bk . R |Kobs | k∈K obs
The multiclass benchmark contains 11 datasets, including 3 with imbalanced ratio greater than 3, 4 with Annotators/item ≥ 10, and 3 with Annotators/item < 5. The binary benchmark contains 16 datasets, including 4, 11, and 2 datasets in these respective regimes. We also consider large-scale subsets: 8 datasets with at least 20 000 annotations and 4 datasets with at least 4 000 annotated items. Since high global accuracy may hide poor recovery of rare classes, we treat minority recall as the primary metric. We report results separately for binary datasets and for the full benchmark. In Table 3, we report minority recall in the large-scale annotation and large-scale item regimes. Implementation details. We implement CC-Rasch in Python and optimize its penalized M-step objective using the L-BFGS-B algorithm through scipy.optimize.minimize (Virtanen et al., 2020). At each EM iteration, the optimizer is initialized with the parameter estimates obtained at the previous iteration. The prior (πk )k is initialized uniformly at the initialization of the EM. Results. Across real-world datasets, CC-Rasch shows the strongest and most consistent ability to recover the minority class. On multiclass datasets, (Table 2), it improves the average minority recall from 0.691 to 0.722 and the balanced accuracy from 0.765 to 0.770 over the strongest baseline. The gain is larger in the most imbalanced regime, reaching +7.4 recall points. On binary datasets (Table 1), CC-Rasch achieves the best overall balanced accuracy (0.847) and the best minority recall (0.859, tied with PLAT). Notice that CC-Rasch has the lowest variance across datasets compared with PLAT. It is also more stable across datasets, with the lowest recall variability overall and in the imbalanced and sparse-annotation regimes. Under strong imbalance, CC-Rasch reaches 0.888 recall, while for Ann./item ≥ 10, it obtains the best recall (0.851) and balanced accuracy (0.853).
5.3
CC-Rasch versus Gold-Supervised Methods
We investigate whether access to ground-truth labels is necessary for accurate minority-class recovery. We compare CC-Rasch, which is fitted using only the observed crowd labels and never accesses gold labels, with supervised aggregation methods whose annotator parameters are estimated from an increasing proportion of gold-labeled items. This kind of question has previously been investigated, for instance, by Jung and Lease (2015). We revisit it from the perspective of class imbalance, with a particular focus on minority recall. Notice that standard majority voting does not use gold labels and is therefore unaffected by their availability. In contrast, gold-supervised methods such as Gold Majority Vote and Dawid–Skene estimate annotator reliability from the available gold-labeled items, so their performance may improve as the amount of supervision increases. 13
WVSCM
0.9
Minority recall
Minority recall
RTE
0.8
0.7
0.7
0.6
0.5
Imb: = 1:0 0
10
20
30
40
Imb: = 1:7
50
0
10
20
30
40
Gold-labeled items (%)
Gold-labeled items (%)
Medical-cause
Product
50
0.9
Minority recall
Minority recall
1.0
0.8 0.7 0.6
0.8
0.6
0.4
Imb: = 3:0
0.5 0
10
20
30
40
Imb: = 7:2
50
0
Gold-labeled items (%) Gold Majority Vote
10
20
30
40
50
Gold-labeled items (%) Gold Dawid--Skene
CC-Rasch
Figure 2: Gold label boosted methods versus CC-Rasch. Methodology Let I = {1, . . . , n} be the set of items and let a real number x ∈]0, 1[. We sample a subset Ix ⊂ I, containing ⌊xn⌋ items, for which the ground-truth labels are assumed to be available, these are the gold labels. We define the gold training set as Dxgold :=
n
o
i, (Yir )r∈Ωi , Yi : i ∈ Ix ,
and the remaining test set as the complementary of Dxgold . Given Dxgold , we fit Gold Majority Vote (Ustalov et al., 2024) and a gold-supervised Dawid–Skene model. For Dawid–Skene, the annotator confusion matrices are estimated from the gold items and then kept fixed when aggregating the n annotations o of the test items. In parallel, CC-Rasch is fitted classically using only i, (Yir )r∈Ωi : i ∈ I \ Ix . All three methods are evaluated on the same (not observed) items. Results. According to Figure 2, in the class-balanced case, Gold Majority Vote achieves a minority recall higher than or close to that of CC-Rasch. However, as class imbalance increases, the gap between CC-Rasch and the two gold-supervised methods becomes larger. Gold Dawid–Skene and 14
Gold Majority Vote obtain similar minority recall on the most imbalanced datasets, although some differences remain on less imbalanced ones. This confirms in particular that an accurate estimation of annotator abilities is not always sufficient under strong class imbalance. Accounting for class-dependent item difficulty allows CC-Rasch to better aggregate labels.
6
Conclusion
In this paper, we studied label aggregation in the class-imbalanced crowdsourcing setting, where rare classes are often the most important ones. We provided a new type of Condorcet Jury Theorem adapted to the imbalanced setting. To address the limitations created by the imbalanced setting, we introduced CC-Rasch, a class-dependent aggregation model that combines annotator competence and item difficulty. Our experiments on synthetic and real-world datasets show that CC-Rasch consistently improves minority-class recall while remaining competitive in balanced accuracy and F1-score, with particularly strong gains under severe class imbalance, high annotation redundancy, and large-scale settings.
7
Detailed results on each dataset
7.1
Synthetic dataset generation
Here we provide a more detailed description of the generation process for synthetic dataset. For each item i, we sample a ground-truth label Yi ∼ Bernoulli(π), where class 1 is the minority class, and an item-difficulty indicator Hi ∼ Bernoulli(ρ). Each annotator r belongs to a type Tr ∈ {good, maj, bad, min}, with fixed type proportions. For every observed annotation (i, r), P(Yir = Yi | Yi = k, Hi = h, Tr = t) = qt,k − hδ, where qt,k is the class-conditional reliability of annotator type t and δ is the hard-item penalty. Otherwise, the annotator outputs 1 − Yi . Each item receives Ni = 1 + Poisson(λ − 1) annotations, with annotators sampled uniformly without replacement. In the controlled sweep, the proportions of reliable and unreliable annotators are fixed, while the mass of majority and minority specialists is redistributed between the two groups.
7.2
Detailed benchmark Table 4
Dataset
Method
Imb.
Ann./item
Min. Recall ↑
Macro F1 ↑
Loneliness (Older Adults)
Majority Vote Dawid–Skene
19.00 19.00
4.77 4.77
0.800 ± 0.000 0.600 ± 0.000
0.667 ± 0.000 0.634 ± 0.000
Continued on next page 15
Dataset
CF
CF*
Weather Sentiment - AMT
Amazon Sentiment (Negative)
product
Jigsaw
ZCall
ZCin
Method
Imb.
Ann./item
Min. Recall ↑
Macro F1 ↑
GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote
19.00 19.00 19.00 19.00 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 10.22 8.70 8.70 8.70 8.70 8.70 8.70 7.22 7.22 7.22 7.22 7.22 7.22 6.13 6.13 6.13 6.13 6.13 6.13 3.60 3.60 3.60 3.60 3.60 3.60 3.60
4.77 4.77 4.77 4.77 5.73 5.73 5.73 5.73 5.73 20.10 20.10 20.10 20.10 20.10 20.00 20.00 20.00 20.00 20.00 7.61 7.61 7.61 7.61 7.61 7.61 3.00 3.00 3.00 3.00 3.00 3.00 19.93 19.93 19.93 19.93 19.93 19.93 10.71 10.71 10.71 10.71 10.71 10.71 5.49
0.600 ± 0.000 0.600 ± 0.000 0.800 ± 0.000 1.000 ± 0.000 0.333 ± 0.000 0.444 ± 0.000 0.333 ± 0.000 0.361 ± 0.106 0.667 ± 0.000 0.111 ± 0.000 0.222 ± 0.000 0.111 ± 0.000 0.028 ± 0.056 0.222 ± 0.000 0.111 ± 0.000 0.222 ± 0.000 0.111 ± 0.000 0.028 ± 0.056 0.222 ± 0.000 0.827 ± 0.000 0.837 ± 0.000 0.827 ± 0.000 0.837 ± 0.000 0.942 ± 0.000 0.904 ± 0.000 0.613 ± 0.000 0.640 ± 0.000 0.518 ± 0.000 0.581 ± 0.007 0.825 ± 0.000 0.889 ± 0.000 0.916 ± 0.000 0.970 ± 0.000 0.976 ± 0.000 0.930 ± 0.000 0.959 ± 0.000 1.000 ± 0.000 0.605 ± 0.000 0.862 ± 0.000 0.824 ± 0.000 0.729 ± 0.027 0.887 ± 0.000 0.907 ± 0.000 0.555 ± 0.000
0.693 ± 0.000 0.693 ± 0.000 0.494 ± 0.000 0.545 ± 0.000 0.810 ± 0.000 0.734 ± 0.000 0.775 ± 0.000 0.777 ± 0.016 0.754 ± 0.000 0.723 ± 0.000 0.727 ± 0.000 0.728 ± 0.000 0.705 ± 0.021 0.728 ± 0.000 0.723 ± 0.000 0.727 ± 0.000 0.728 ± 0.000 0.705 ± 0.021 0.725 ± 0.000 0.906 ± 0.000 0.904 ± 0.000 0.901 ± 0.000 0.909 ± 0.000 0.762 ± 0.000 0.857 ± 0.000 0.766 ± 0.000 0.843 ± 0.000 0.799 ± 0.000 0.765 ± 0.005 0.707 ± 0.000 0.567 ± 0.000 0.975 ± 0.000 0.982 ± 0.000 0.971 ± 0.000 0.972 ± 0.000 0.918 ± 0.000 0.743 ± 0.000 0.758 ± 0.000 0.758 ± 0.000 0.782 ± 0.000 0.771 ± 0.005 0.739 ± 0.000 0.655 ± 0.000 0.658 ± 0.000
Continued on next page 16
Dataset
ZCus
Loneliness (Intervention)
medical crowdtruth cause
movie reviews 4class
hitspam crowdflower
Web
medical crowdtruth all
Method
Imb.
Ann./item
Min. Recall ↑
Macro F1 ↑
Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR)
3.60 3.60 3.60 3.60 3.60 3.60 3.60 3.60 3.60 3.60 3.60 3.17 3.17 3.17 3.17 3.17 3.17 2.97 2.97 2.97 2.97 2.97 2.97 2.95 2.95 2.95 2.95 2.95 2.23 2.23 2.23 2.23 2.23 2.23 2.09 2.09 2.09 2.09 2.09 1.92 1.92 1.92 1.92 1.92
5.49 5.49 5.49 5.49 5.49 5.98 5.98 5.98 5.98 5.98 5.98 14.36 14.36 14.36 14.36 14.36 14.36 13.42 13.42 13.42 13.42 13.42 13.42 4.96 4.96 4.96 4.96 4.96 22.97 22.97 22.97 22.97 22.97 22.97 5.84 5.84 5.84 5.84 5.84 13.49 13.49 13.49 13.49 13.49
0.747 ± 0.000 0.707 ± 0.000 0.597 ± 0.023 0.828 ± 0.000 0.828 ± 0.000 0.666 ± 0.000 0.853 ± 0.000 0.797 ± 0.000 0.686 ± 0.027 0.905 ± 0.000 0.894 ± 0.000 0.875 ± 0.000 0.917 ± 0.000 0.958 ± 0.000 0.917 ± 0.000 0.958 ± 0.000 0.625 ± 0.000 0.720 ± 0.000 0.904 ± 0.000 0.745 ± 0.000 0.749 ± 0.021 0.971 ± 0.000 0.996 ± 0.000 0.967 ± 0.000 0.995 ± 0.000 0.978 ± 0.000 0.969 ± 0.005 0.989 ± 0.000 0.097 ± 0.000 0.097 ± 0.000 0.097 ± 0.000 0.097 ± 0.000 0.097 ± 0.000 0.097 ± 0.000 0.918 ± 0.000 0.888 ± 0.000 0.924 ± 0.000 0.928 ± 0.077 0.970 ± 0.000 0.780 ± 0.000 0.910 ± 0.000 0.823 ± 0.000 0.817 ± 0.018 0.959 ± 0.000
0.701 ± 0.000 0.709 ± 0.000 0.661 ± 0.001 0.674 ± 0.000 0.614 ± 0.000 0.801 ± 0.000 0.784 ± 0.000 0.830 ± 0.000 0.805 ± 0.003 0.725 ± 0.000 0.700 ± 0.000 0.450 ± 0.000 0.508 ± 0.000 0.440 ± 0.000 0.489 ± 0.000 0.429 ± 0.000 0.496 ± 0.000 0.863 ± 0.000 0.900 ± 0.000 0.874 ± 0.000 0.876 ± 0.006 0.883 ± 0.000 0.746 ± 0.000 0.645 ± 0.000 0.709 ± 0.000 0.687 ± 0.000 0.691 ± 0.020 0.710 ± 0.000 0.469 ± 0.000 0.469 ± 0.000 0.469 ± 0.000 0.469 ± 0.000 0.469 ± 0.000 0.469 ± 0.000 0.773 ± 0.000 0.826 ± 0.000 0.826 ± 0.000 0.856 ± 0.006 0.869 ± 0.000 0.886 ± 0.000 0.929 ± 0.000 0.907 ± 0.000 0.902 ± 0.006 0.929 ± 0.000
Continued on next page 17
Dataset wvscm
nist-trec-relevance
LabelMe
Amazon Sentiment (Book)
Loneliness (Technology)
Dog
bird
Trec
Method
Imb.
Ann./item
Min. Recall ↑
Macro F1 ↑
PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR)
1.92 1.74 1.74 1.74 1.74 1.74 1.74 1.74 1.74 1.74 1.74 1.73 1.73 1.73 1.73 1.73 1.57 1.57 1.57 1.57 1.57 1.57 1.56 1.56 1.56 1.56 1.56 1.56 1.35 1.35 1.35 1.35 1.35 1.28 1.28 1.28 1.28 1.28 1.25 1.27 1.27 1.27 1.27 1.27
13.49 12.26 12.26 12.26 12.26 12.26 5.87 5.87 5.87 5.87 5.87 2.55 2.55 2.55 2.55 2.55 7.63 7.63 7.63 7.63 7.63 7.63 4.86 4.86 4.86 4.86 4.86 4.86 10.00 10.00 10.00 10.00 10.00 39.00 39.00 39.00 39.00 39.00 39.00 4.64 4.64 4.64 4.64 4.64
0.994 ± 0.000 0.741 ± 0.000 0.672 ± 0.000 0.621 ± 0.000 0.698 ± 0.017 0.690 ± 0.000 0.490 ± 0.000 0.333 ± 0.000 0.528 ± 0.000 0.441 ± 0.044 0.468 ± 0.000 0.843 ± 0.000 0.809 ± 0.000 0.764 ± 0.000 0.770 ± 0.030 0.719 ± 0.000 0.980 ± 0.000 0.977 ± 0.000 0.980 ± 0.000 0.982 ± 0.000 0.980 ± 0.000 0.990 ± 0.000 0.897 ± 0.000 0.949 ± 0.000 0.897 ± 0.000 0.897 ± 0.000 0.949 ± 0.000 0.949 ± 0.000 0.860 ± 0.000 0.884 ± 0.000 0.860 ± 0.000 0.860 ± 0.011 0.866 ± 0.000 0.553 ± 0.000 0.851 ± 0.000 0.383 ± 0.000 0.601 ± 0.032 0.872 ± 0.000 0.875 ± 0.000 0.432 ± 0.000 0.534 ± 0.000 0.080 ± 0.000 0.406 ± 0.033 0.688 ± 0.000
0.844 ± 0.000 0.714 ± 0.000 0.713 ± 0.000 0.759 ± 0.000 0.711 ± 0.014 0.721 ± 0.000 0.528 ± 0.000 0.568 ± 0.000 0.566 ± 0.000 0.534 ± 0.007 0.559 ± 0.000 0.765 ± 0.000 0.786 ± 0.000 0.770 ± 0.000 0.764 ± 0.006 0.768 ± 0.000 0.960 ± 0.000 0.961 ± 0.000 0.963 ± 0.000 0.962 ± 0.000 0.964 ± 0.000 0.942 ± 0.000 0.886 ± 0.000 0.887 ± 0.000 0.876 ± 0.000 0.886 ± 0.000 0.907 ± 0.000 0.897 ± 0.000 0.816 ± 0.000 0.845 ± 0.000 0.834 ± 0.000 0.820 ± 0.003 0.837 ± 0.000 0.738 ± 0.000 0.876 ± 0.000 0.671 ± 0.000 0.776 ± 0.017 0.914 ± 0.000 0.851 ± 0.000 0.632 ± 0.000 0.685 ± 0.000 0.432 ± 0.000 0.618 ± 0.012 0.700 ± 0.000
Continued on next page 18
Dataset Bird
MS
PosSent
medical crowdtruth treat
Sentiment popularity - AMT
SP
Face
RTE
Method
Imb.
Ann./item
Min. Recall ↑
Macro F1 ↑
PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR) Majority Vote Dawid–Skene GLAD
1.27 1.25 1.25 1.25 1.25 1.25 1.25 1.19 1.19 1.19 1.19 1.19 1.12 1.12 1.12 1.12 1.12 1.12 1.07 1.07 1.07 1.07 1.07 1.07 1.04 1.04 1.04 1.04 1.04 1.04 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
4.64 39.00 39.00 39.00 39.00 39.00 39.00 4.21 4.21 4.21 4.21 4.21 20.00 20.00 20.00 20.00 20.00 20.00 13.61 13.61 13.61 13.61 13.61 13.61 20.00 20.00 20.00 20.00 20.00 20.00 5.55 5.55 5.55 5.55 5.55 5.55 8.98 8.98 8.98 8.98 8.98 10.00 10.00 10.00
0.653 ± 0.000 0.562 ± 0.000 0.854 ± 0.000 0.396 ± 0.000 0.609 ± 0.031 0.875 ± 0.000 0.875 ± 0.000 0.683 ± 0.000 0.905 ± 0.000 0.889 ± 0.000 0.905 ± 0.000 0.905 ± 0.000 0.892 ± 0.000 0.934 ± 0.000 0.903 ± 0.000 0.926 ± 0.005 0.928 ± 0.000 0.956 ± 0.000 0.829 ± 0.000 0.925 ± 0.000 0.898 ± 0.000 0.898 ± 0.015 0.962 ± 0.000 0.973 ± 0.000 0.951 ± 0.000 0.943 ± 0.000 0.947 ± 0.000 0.944 ± 0.002 0.955 ± 0.000 0.951 ± 0.000 0.885 ± 0.000 0.920 ± 0.000 0.913 ± 0.000 0.874 ± 0.012 0.926 ± 0.000 0.885 ± 0.000 0.938 ± 0.000 0.904 ± 0.000 0.925 ± 0.000 0.933 ± 0.007 0.925 ± 0.000 0.910 ± 0.000 0.948 ± 0.000 0.948 ± 0.000
0.654 ± 0.000 0.742 ± 0.000 0.887 ± 0.000 0.678 ± 0.000 0.780 ± 0.016 0.915 ± 0.000 0.851 ± 0.000 0.709 ± 0.000 0.770 ± 0.000 0.793 ± 0.000 0.802 ± 0.004 0.801 ± 0.000 0.931 ± 0.000 0.960 ± 0.000 0.948 ± 0.000 0.949 ± 0.003 0.957 ± 0.000 0.910 ± 0.000 0.899 ± 0.000 0.944 ± 0.000 0.935 ± 0.000 0.935 ± 0.008 0.962 ± 0.000 0.957 ± 0.000 0.944 ± 0.000 0.944 ± 0.000 0.946 ± 0.000 0.943 ± 0.002 0.946 ± 0.000 0.944 ± 0.000 0.890 ± 0.000 0.915 ± 0.000 0.917 ± 0.000 0.898 ± 0.004 0.916 ± 0.000 0.890 ± 0.000 0.610 ± 0.000 0.626 ± 0.000 0.611 ± 0.000 0.622 ± 0.007 0.634 ± 0.000 0.919 ± 0.000 0.927 ± 0.000 0.926 ± 0.000
Continued on next page 19
Dataset
cifar10h
Method
Imb.
Ann./item
Min. Recall ↑
Macro F1 ↑
CrowdFM Φ-Rasch (OUR) PLAT Majority Vote Dawid–Skene GLAD CrowdFM Φ-Rasch (OUR)
1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
10.00 10.00 10.00 51.42 51.42 51.42 51.42 51.42
0.880 ± 0.015 0.943 ± 0.000 0.910 ± 0.000 0.999 ± 0.000 0.999 ± 0.000 0.999 ± 0.000 0.998 ± 0.001 0.994 ± 0.001
0.914 ± 0.006 0.926 ± 0.000 0.919 ± 0.000 0.992 ± 0.000 0.993 ± 0.000 0.992 ± 0.000 0.992 ± 0.001 0.989 ± 0.000
References Grégoire Allaire. Analyse numérique et optimisation. Éditions de lÉcole Polytechnique, Palaiseau, 2012. Daniel Berend and Aryeh Kontorovich. Consistency of weighted majority votes. In Advances in Neural Information Processing Systems, volume 27, 2014. Daniel Berend and Jacob Paroush. When is condorcet’s jury theorem valid? Social Choice and Welfare, 15:481–488, 08 1998. doi: 10.1007/s003550050118. Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28, 1979. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1–38, 1977. Julian Gil-Gonzalez, David Augusto Cárdenas-Peña, Alvaro Orozco-Gutiérrez, Enrique D. GuijarroEstelles, and Andres M. Álvarez-Meza. Learning from crowds using a focal loss function: Dealing with imbalanced annotations. Technologies, 14(6):370, 2026. Hyun Joon Jung and Matthew Lease. Modeling temporal crowd work quality with limited supervision. In AAAI Conference on Human Computation and Crowdsourcing, volume 3, page 83–91, 2015. Hao Liu, Jiacheng Liu, Feilong Tang, Long Chen, Jiadi Yu, Yanmin Zhu, Qiwen Dong, Yichuan Yu, and Xiaofeng Hou. Towards a foundation model for crowdsourced label aggregation. In The Fourteenth International Conference on Learning Representations, 2026. URL https:// openreview.net/forum?id=FF9QVQduAu. Byeong Min Mun, Paul H Kvam, and Suk Joo Bae. Mixed-effects nonhomogeneous poisson process model for multiple repairable systems. IEEE Access, 9:71900–71908, 2021. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust, 2019. URL https://arxiv.org/abs/1908.07086. Georg Rasch. Probabilistic Models for Some Intelligence and Attainment Tests. Studies in Mathematical Psychology. Danmarks Paedagogiske Institut, Copenhagen, 1960.
20
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Learning from multiple annotators: Distinguishing good from random labelers. Pattern Recogn. Lett., 34(12):1428–1436, September 2013. ISSN 0167-8655. doi: 10.1016/j.patrec.2013.05.012. URL https://doi.org/10.1016/j. patrec.2013.05.012. Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 77 (1–3):157–173, 2008. Gabriel Singer, Samuel Gruffaz, Olivier Vo Van, Nicolas Vayatis, and Argyris Kalogeratos. Optimal fair aggregation of crowdsourced noisy labels using demographic parity constraints, 2026. URL https://arxiv.org/abs/2601.23221. Rion Snow, Brendan O’connor, Dan Jurafsky, and Andrew Y Ng. Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks. In Conference on Empirical Methods in Natural Language Processing, pages 254–263, 2008. Dmitry Ustalov, Nikita Pavlichenko, and Boris Tseitlin. Learning from Crowds with Crowd-Kit. Journal of Open Source Software, 9(96):6227, 2024. ISSN 2475-9066. doi: 10.21105/joss.06227. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020. doi: 10.1038/s41592-019-0686-2. Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems, volume 23, pages 2424–2432, 2010. Jacob Whitehill, Paul Ruvolo, Ting-fan Wu, Jacob Bergsma, and Javier R. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, volume 22, pages 2035–2043. Curran Associates, Inc., 2009. Jing Zhang, Xindong Wu, and Victor S. Sheng. Imbalanced multiple noisy labeling. IEEE Transactions on Knowledge and Data Engineering, 27(2):489–503, 2015. doi: 10.1109/TKDE.2014. 2327039.
21