arXiv:2606.16868v1 [cs.CV] 15 Jun 2026
Federated Medical Image Segmentation under Real-World Label Noise: A Benchmark Suite for Noisy Label Learning Method Selection Markus Bujotzek1,2*, Dimitrios Bounias1 , Stefan Denner1 , Ralf Floca1,3 , Maximilian Fischer1,2 , Peter Neher1,4 , Klaus Maier-Hein1,2,4,5 1
Division of Medical Image Computing, Germany Cancer Research Center, Heidelberg, 69120, Germany. 2 Medical Faculty, University of Heidelberg, Heidelberg, 69120, Germany. 3 Heidelberg Institute of Radiation Oncology (HIRO), National Center for Radiation Research in Oncology (NCRO), Heidelberg, 69120, Germany. 4 Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, 69120, Germany. 5 Faculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, 69120, Germany. 6 National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and the university medical center Heidelberg, Heidelberg, 69120, Germany.
*Corresponding author(s). E-mail(s): [email protected]; Abstract Objective: While federated learning (FL) enables collaborative medical image segmentation without centralizing sensitive data, real-world deployment is frequently complicated by cross-site label imperfections such as contour disagreement, missing or additional structures, and confused labels. Federated noisy label learning (FNLL) aims to mitigate these effects, yet remains underused in practice as existing evidence is largely based on synthetic noise, simplified settings, and limited real-world noisy evaluation. We address this gap by introducing a benchmark suite that combines diverse real-world noisy datasets, deployment-relevant
1
client-noise scenarios, and label-noise-targeted evaluation to support systematic FNLL assessment and informed method selection. Materials & Methods: The suite combines curated real-world noisy medical image segmentation datasets from diverse sources with a comprehensive federated segmentation framework including various client-noise scenarios and noise-targeted evaluation. To demonstrate its capabilities, we compare representative FNLL methods across approaches, including noise-aware aggregation, robust personalization, label correction, and sample selection. Results: In-depth data analysis shows that real-world segmentation label noise occurs both in isolation and in combination of characterized noise types. The benchmark identifies FedSelect as the strongest overall FNLL method, underlines FedAvg as a competitive baseline, and provides an actionable decision guide to support selection of suitable FNLL strategies based on label-noise type and client-noise scenario. Discussion & Conclusion: The presented suite provides a realistic and discriminative basis for FNLL evaluation in medical image segmentation and establishes a reusable foundation for fair benchmarking, dataset-specific label-noise characterization, and future method development under realistic federated settings. Code is available at https://github.com/MIC-DKFZ/FedSegNoiseBench. Keywords: Federated Learning; Medical Image Segmentation; Label Noise; Data Quality; Benchmarking
1 Introduction Federated Learning (FL) enables collaborative model training across institutions without exchanging local data, by aggregating locally optimized model updates [1]. In medical imaging, FL leverages multi-institutional data quantity and diversity to improve robustness and generalization while respecting privacy and governance constraints [2], with growing translational evidence in radiology [3, 4] and beyond [5].
1.1 Motivation Medical image segmentation is a key component of automated diagnosis and computerassisted therapy, as it provides spatially meaningful representations of anatomical and pathological structures [6]. Supervised training of segmentation models relies on voxellevel annotation masks, which in clinical practice are often heterogeneous and noisy due to inter-rater variability, difficulty of target recognition, human annotation errors, and the increasing use of automatically generated labels [7]. As a result, segmentation label noise commonly manifests as contour inconsistencies, missing or additional instances, or confused class labels of targets, all of which degrade model performance [7] (Figure 1). Because segmentation outputs are frequently used in downstream tasks such as diagnosis, treatment planning, and quantitative biomarker extraction, such degradation can propagate into clinical workflows and affect decision-making reliability [8–10]. These noise-based challenges are further amplified in FL. First, both data and annotation heterogeneity increase naturally as image acquisition, curation, and 2
labeling are performed independently across institutions [11]. Second, the privacypreserving nature of FL restricts centralized inspection of data and labels, complicating noise detection and mitigation, such that noisy clients may only become apparent through performance deterioration, as reported in real-world deployments [4]. Third, distributed label noise can propagate through model aggregation, allowing partially or fully noisy clients to degrade learning on otherwise clean clients.
Fig. 1: Segmentation label noise of various forms degrades model performance and poses a particular challenge in FL, where noisy annotations are distributed across clients and cannot be centrally inspected. While FNLL methods aim to address this problem, existing literature is often limited to few and synthetic noise types, restricted client-noise scenarios, and narrow data scope. Our benchmark suite closes this gap by combining diverse real-world noisy segmentation datasets, a federated benchmarking framework, and comprehensive noise-targeted evaluation, thereby enabling FNLL method selection, dataset characterization, benchmarking on new data, and evaluation of newly developed FNLL methods. Dataset thumbnails adapted from [12–17].
1.2 Related work and problem statement Centralized Noisy Label Learning (NLL) is a widely studied domain beyond medical image segmentation [7, 18–20], and methods can be categorized into model- and data-centric approaches. Model-centric approaches include noise-robust architecture [21] or loss [22, 23] modifications, regularization via label smoothing [24] or data augmentation [25], and changed training paradigms like curriculum learning [26] or teacher-student frameworks [25, 27]. Data-centric approaches cover approaches like label correction or refinement [25, 26, 28, 29], data re-weighting [30–33], or sample selection [25, 27, 34–37]. 3
While centralized NLL has been extensively studied, many methods rely on pooleddata signals, like global loss statistics, class centroids, or clean validation sets, which are unavailable in cross-silo FL. Moreover, client-specific data and annotation heterogeneity couple label reliability to domain shift [38, 39], making local noise mitigation prone to miscalibration across clients and potentially biasing global aggregation. Federated Noisy Label Learning (FNLL) mitigates label noise under distributeddata constraints through four recurring, often combined mechanisms: Noise-aware federated model aggregation estimates client reliability and reweights updates during aggregation, typically using training dynamics, class or region losses, or class centroids to down-weight noisy clients [40–42]. In segmentation, aggregation can additionally be guided by contour-localized losses or completeness proxies [43–46]. Noise-robust personalization addresses data and label heterogeneity by adapting parts of the model to client-specific distributions. Methods use inter-site prediction inconsistency to retain local parameters and reweight uncertain regions [47], or fuse accumulated local-gradient information with the global model after training [48]. Label correction methods identify unreliable labels, typically on noisy clients, and refine them using global predictions [38], centroid-consistent pseudo-labels [49], or soft-label optimization [41]. In segmentation, correction is commonly performed pixel-wise through peer- or teacher-model agreement [45, 50], or by confidence-thresholded recovery of missing structures [44]. Sample selection methods prioritize informative or reliable samples during training, either by focusing on high-loss (not-yet-learned) samples [51] or by mitigating label noise through confidence- and consistency-based selection [49], including meta-learned strategies driven by loss dynamics [39]. While FNLL methods have been proposed across diverse approaches, their evaluation remains fragmented along critical axes. First, studies predominantly rely on few and largely synthetic noise models [38–45, 49, 50], despite evidence that conclusions may not transfer to real-world noisy data [52]. Second, only limited clientnoise scenarios are considered [38, 40–46, 50], rather than systematically assessing robustness under heterogeneous federated conditions. Third, evaluations are typically conducted on a small number of narrow datasets with limited modality diversity [38– 51], restricting generalization. As a result, FNLL remains underutilized in practice, where real-world FL deployments still rely on FedAvg-based training [3, 4, 53], while evidence for FNLL is largely confined to synthetic and simplified settings. Benchmarking endeavors have helped consolidate otherwise fragmented NLL research in centralized medical image classification [52] and instance segmentation [54], and have recently emerged for FNLL in classification [55, 56]. However, a benchmarking effort for federated medical image segmentation systematically covering diverse datasets, multiple client-noise scenarios, and numerous, real-world segmentation label noise types remains lacking, limiting informed FNLL method evaluation and selection.
1.3 Contribution We present a reusable and extensible benchmark suite for FNLL in cross-silo medical image segmentation, enabling systematic evaluation, fair comparison, and informed method selection under realistic conditions. Rather than proposing another narrowly
4
evaluated FNLL method, we provide a reusable community resource for deploymentrelevant evaluation, decision guidance, and structured extension to future datasets and methods. The suite combines four tightly integrated components: (1) six curated medical image segmentation datasets with inherent real-world label noise and corresponding noise-type analysis; (2) an nnU-Net-based federated segmentation framework [57, 58] with integrated state-of-the-art FNLL methods and deployment-relevant client-noise scenarios; (3) a label-noise-targeted evaluation protocol for comparison across heterogeneous datasets and noise settings; and (4) an actionable decision guide linking observed label-noise characteristics to suitable FNLL strategies. Using this suite, we benchmark representative FNLL methods to identify consistently strong performers, assess robustness across datasets and noise scenarios, and characterize method suitability for specific label-noise patterns. By releasing the suite as an open-source codebase along a detailed contribution guide, we provide a community resource for reproducible evaluation, dataset-specific noise characterization, and structured extension to future datasets and methods. The code is publicly available at https://github.com/MIC-DKFZ/FedSegNoiseBench.
2 Materials and Methods 2.1 Medical image segmentation datasets with real-world label noise We base our benchmark suite on six multi-rater medical image segmentation datasets with inherent inter-rater variability and real-world label noise, rather than synthetically generated noise. Noisy training labels are obtained by randomly selecting one rater annotation per sample [42], while clean labels are derived from majority voting, STAPLE fusion [59], or additionally provided expert labels. The curated datasets cover diverse clinical domains and modalities, including thoracic CT (LIDC-IDRI [12]), retinal fundus photography (RIGA [60]), prostate histopathology microscopy (GleasonXAI Harvard Dataverse [14, 61, 62]), micro-CT of mouse tumors (MouseTumor [15]), and multi-organ MR imaging (MMIS [16], MAMAMIA [17]). They vary in dimensionality, number of classes and instances per class, and label-noise types, including contour inconsistencies, missing or additional target instances, and confused class labels, providing a broad test bed for FNLL in medical image segmentation. Dataset and label-noise details are summarized in Table 1.
2.2 FNLL benchmarking framework for cross-silo medical image segmentation Federated segmentation framework. We build the benchmark model on a federated adaptation of nnU-Net [57, 58], whose data-driven self-configuration provides a strong and task-adaptive basis for comparing FNLL methods across diverse segmentation datasets. In the federated setting, local data fingerprints are aggregated into a global fingerprint for unified experiment planning and preprocessing, ensuring consistently configured networks and data processing across clients [4, 63].
5
Table 1: Overview of the six benchmark datasets, their inherent real-world segmentation label-noise characteristics, and representative thumbnails qualitatively illustrating the source and manifestation of label noise. Example thumbnails adapted from [12–17]. Modality Dimensions # classes
LIDC
RIGA
GleasonHD
MouseT
MMIS
MMIA
CT 3D 1
Fundus 2D 2
Microscopy 2D 3
micro-CT 3D 1
MR 3D 1
MR 3D 1
multi-rater contour, missed/extra labels, confused labels STAPLE
multi-rater contour, missed/extra labels
multi-rater contour, missed/extra labels
auto-generated contour, missed/extra labels
STAPLE
rater majority
expert
Noise origin Noise type
multi-rater contour, missed/extra labels
multi-rater contour
Clean origin
rater majority
rater majority
Example
Client-noise scenarios. FNLL robustness is evaluated in four client-noise scenarios: clean, where all clients use clean labels; noisy, where all clients use noisy labels; ratio of clients (roc), where a fraction p of clients is fully noisy and the remaining clients are clean; and ratio on all (roa), where each client contains a fraction p of noisy samples. Evaluation. While training labels are inherently noisy depending on the clientnoise scenario, evaluation is performed on clean masks to quantify robustness against label noise with respect to expert or consensus references [19]. General segmentation performance is assessed using the overlap-based Dice score. To characterize robustness to specific label-noise types, we additionally use noise-type-specific metrics: HD95 for contour disagreement, foreground-background instance-level F1 for missing or additional target instances, and voxel-based class confusion for label confusion. The F1 score matches instances at an IoU threshold of 0.1, while class confusion measures, for each ground-truth class, the fraction of its voxels predicted as another foreground class (Equation 1). Metric edge-case handling of absent classes and empty masks, is described in Appendix A.1. |{v ∈ Ω : yv = ci , ŷv ∈ Cfg \ {ci }}| (1) |{v ∈ Ω : yv = ci }| We perform bootstrap resampling of each evaluation set with 1000 iterations and sample size N , then average scores over classes, clients, and folds. To identify the best-performing FNLL methods, metric scores were averaged across classes, clients, folds, and datasets, then ranked per client-noise scenario and after averaging across all scenarios. To assess whether FNLL methods improve over FedAvg, we perform one-sided paired Wilcoxon signed-rank tests on case-level score differences matched by ClsConf ci =
6
case identifier within each dataset and pooled across datasets for each metric, clientnoise scenario, and method. Holm-Bonferroni correction is applied across method-wise comparisons against FedAvg to limit false-positive findings.
2.3 Benchmarked segmentation label noise mitigating methods We benchmark representative methods from major FNLL approaches, selected for applicability to real-world segmentation label noise, compatibility with nnU-Net, and feasibility in cross-silo healthcare settings without central expert validation data. Methods are compared against FedAvg as the default FL baseline [64]. Noise-aware aggregation: FedA³I [43]. FedA³I addresses heterogeneous segmentation label noise by estimating client-specific segmentation tendencies for aggregation. After a FedAvg warm-up, a server-side two-component GMM separates overand under-segmenting clients and enables quality-aware, layer-wise aggregation. We selected FedA³I as it targets general contour-related annotation bias, whereas alternatives focus on incompleteness-only noise [44] or combine aggregation with binary label correction [45]. Key hyperparameters are warm-up rounds and the inter-group weighting coefficient. Noise-robust personalization: IOP-FL [48]. IOP-FL personalizes federated training by combining local and global gradient information to guide optimization toward client-specific optima. We selected it for its architecture-agnostic design, enabling direct integration into our nnU-Net-based framework. We use the insidefederation variant, which combines current and historical model information through a single mixing factor α. Label correction: FedCorr [38]. FedCorr implements a staged, task-agnostic label-correction pipeline without requiring noise assumptions or clean central validation data [44, 45, 50]. After initial pre-processing, it identifies noisy clients via LID statistics, separates clean and noisy samples using a loss-based GMM, and corrects selected labels using global predictions with noise-adaptive proximal regularization. Key hyperparameters control pre-processing rounds, relabeling extent and confidence, and regularization strength. Sample selection: FedSelect [39]. FedSelect mitigates label noise via metalearned importance-aware sample and client selection, without requiring an external clean validation set. Sample importance is inferred from training dynamics and guides both sample selection and client weighting using client-side proxy validation data. Key hyperparameters control warm-up rounds, selected client and sample fractions, meta-margin momentum, and proxy validation set size.
2.4 Experimental setup Datasets were partitioned into 3-5 FL clients based on clinically or technically meaningful factors, including center, scanner, sub-dataset, rater identity, or animal assignment (Table 2). The dataset-specific client composition determines the partially noisy client-noise scenarios roa and roc. In roa, a nominal noise level of p = 50% is applied within each client. In roc, p specifies the proportion of noisy clients, with |Knoisy | = ⌈p |Kall |⌉.
7
Table 2: Overview of dataset-specific federated client partitioning and the resulting partially noisy client-noise scenarios. For roa and roc, sample counts are reported as nclean /nnoisy . In roc, C and N denote the sets of clean and noisy clients, respectively.
Partial noise
FL splitting
LIDC # clients Split criterion
4
RIGA 3 sub-dataset membership C0: 195 C1: 94 C2: 460
GleasonHD
MouseT
MMIS
3
5
4
Samples per client
roa (nc /nn ) proa eff
1062 / 1059 49.93%
375 / 374 49.93%
239 / 237 49.79%
227 / 225 49.78%
61 / 59 49.17%
754 / 752 49.93%
C = {0, 1} N = {2, 3} 1499 / 622 29.33%
C = {0} N = {1, 2} 195 / 554 73.97%
C = {0} N = {1, 2} 159 / 317 66.60%
C = {0, 1, 2} N = {3, 4} 305 / 147 32.52%
C = {0, 1} N = {2, 3} 63 / 57 47.50%
C = {0, 1} N = {2, 3} 462 / 1044 69.32%
roc (nc /nn ) proc eff
C0: 159 C1: 158 C2: 159
random mouse C0: 178 C1: 127 C2: 67 C3: 48 C4: 32
rater identity C0: 34 C1: 29 C2: 27 C3: 30
4 originating center C0: 291 C1: 171 C2: 980 C3: 64
scanner manufacturer C0: 1281 C1: 218 C2: 475 C3: 147
roc clients
random
MMIA
Since client counts and sample counts per client vary across datasets, we report the effective noise level peff as the fraction of noisy samples: P P nk k∈K k∈Kall nk p roc roa ≈ p, peff = P noisy . peff (p) = P k∈Kall nk k∈Kall nk Here, Kall = Kclean ∪ Knoisy and nk denotes the number of samples at client k . Corresponding sample counts and resulting effective noise levels peff are listed in Table 2. The federated framework uses the default nnU-Net full-resolution ResEncM configuration with dataset-specific patch and batch sizes (Appendix B.1). For each client configuration, communication rounds were set to match cumulatively the nnU-Net recommendation of 1000 local epochs [57], using one local epoch per round. Standard nnU-Net five-fold splits were generated per client; experiments used the first three folds, with metrics computed on the corresponding held-out validation sets. Prior to benchmarking, method-specific FNLL hyperparameters were optimized by grid search over predefined candidate configurations, with final settings selected by average performance across datasets. Details are provided in Appendix B.2.
3 Results 3.1 In-depth data and label noise analysis Figures 2 and 3 characterize the six inherently noisy benchmark datasets by consensusmask quality and noisy-mask properties. Consensus cleanliness is assessed using interrater and rater-consensus agreement via class-wise Fleiss’ kappa, HD95, instance-level F1, and class confusion. Noisy masks are analyzed in the joint space of these noisesensitive metrics to quantify noise-type prevalence and severity, complemented by representative examples from regions enriched for characteristic noise patterns. How clean is consensus? Consensus quality varies substantially across datasets. RIGA and MouseT show the cleanest consensus masks, with high inter-rater and raterconsensus agreement across metrics. LIDC and MMIS show intermediate consensus 8
quality, with increased variability mainly in boundary consistency reflected by HD95. GleasonHD has the lowest consensus cleanliness, with reduced Fleiss’ kappa, weaker rater-consensus agreement, and more pronounced off-diagonal class-confusion entries. MMIA is excluded because clean labels are provided as expert annotations rather than multi-rater consensus. How noisy is noisy? RIGA shows the simplest noise regime, with high instance-level agreement, negligible class confusion, and variable HD95, indicating predominantly contour-driven noise. LIDC, MouseT, and MMIS show mixed noise, dominated by contour disagreement but with reduced instance-level F1 indicating missed or additional target structures. MMIA is mainly characterized by missed or additional target structures, reflected by shifts along the instance-level F1 axis. GleasonHD shows the most severe and heterogeneous label noise, with broad dispersion across all three metric dimensions.
3.2 Comparative benchmarking of FNLL methods Which FNLL method performs best across datasets, noise types and scenarios? Across datasets and client-noise scenarios, FedSelect is the strongest overall FNLL method due to its high rank stability and consistent performance, whereas IOP-FL is the main competitor and achieves the best Dice scores in many individual dataset-scenario comparisons (Table 3, Figure 4, Appendix C.1). FedAvg remains a strong baseline, while FedCorr shows dataset-dependent gains and FedA3I performs weakest overall, both frequently failing to improve over FedAvg. Statistical testing confirms this mixed picture: improvements over FedAvg are only partially significant, with corrected significant Dice gains observed primarily for IOP-FL in the roa, roc, and noisy scenarios (Appendix C.3). Segmentation performance is strongly shaped by dataset difficulty and label-noise characteristics (Figures 2 and 3). RIGA and MouseT are comparatively easy, contour-dominated datasets with high Dice scores and limited separation between top methods, although FedSelect and IOP-FL remain strongest overall. GleasonHD, which combines all three noise types, is the most challenging dataset, with substantially lower performance, broader distributions, and strongest results for FedCorr. LIDC, MMIS, and MMIA form an intermediate regime; FedSelect and IOP-FL perform similarly under contour-related noise, while IOP-FL performs best on the predominantly instance-level noise in MMIA. The ranking analysis (Figure 4) confirms a stable overall ordering, with FedSelect ranked first, followed by IOP-FL; FedAvg remains competitive and outperforms the remaining FNLL methods. This identifies FedSelect and IOP-FL as the top-performing FNLL tier across noisy and even beneficial in clean scenarios. How robust are FNLL methods to partially and fully noisy training? Clean-referenced Dice drops show that all noisy regimes remain close to clean baselines, but differ by scenario and method (Figure 5). The roa setting, with ∼50% noisy samples per client, induces only minor degradation, indicating robustness to withinclient partial noise. In contrast, roc (29.33-73.97% effective noise; mean 53.2%, Table 2) causes larger losses and is more harmful than roa, while fully noisy training yields the strongest degradation. Method-wise, IOP-FL remains closest to the clean baseline in roa, whereas FedSelect shows the largest drop. In roc and fully noisy settings, FedCorr 9
Fig. 2: Consensus- and noisy label characteristics of LIDC, RIGA and GleasonHD datasets including representative examples illustrating characteristic label noise patterns. In left column, R and C abbreviate raters and consensus; in right column, clean/consensus contours in yellow and noisy contours in cyan.
achieves the smallest Dice degradation, while FedSelect again degrades most. This contrasts with absolute performance (Table 3, Figure 4), where FedSelect ranks best overall, indicating that highest Dice does not coincide with minimal degradation and that robustness is scenario- and dataset-dependent. Which FNLL method mitigates specific label noise types best? We assess noise-type-specific robustness using dedicated metrics (Section 2.2), including for each metric only datasets where the corresponding noise type is present according to Figures 2 and 3. This preserves noise-type specificity while retaining realistic mixed-noise settings.
10
Fig. 3: Consensus- and noisy label characteristics of MouseT, MMIS and MMIA datasets including representative examples illustrating characteristic label noise patterns (yellow=consensus/expert, cyan=noise). As MMIA comes out-of-the-box with expert (clean) and automatically generated (noisy) label masks, it is not included in the How clean is consensus considerations.
For contour disagreement, assessed by HD95, FedSelect ranks best overall and is the only FNLL method consistently matching or outperforming the strong FedAvg baseline (Figure 6, Appendix C.2.1). Separation is clearest in contour-dominated datasets (RIGA, MouseT, MMIS ), where FedSelect yields the lowest boundary deviations across scenarios, while FedA3I and FedCorr often show substantially higher HD95. For missing or additional target structures, assessed by instance-wise F1, FedSelect again ranks first overall, with IOP-FL and FedAvg as strongest competitors. Separation is clearest in the instance-noise-dominated MMIA, where FedSelect and IOP-FL
11
Fig. 4: Rank stability of Dice segmentation performances based on bootstrapping across datasets, client-noise scenarios and methods. Lower ranks indicate better performances, bubble size reflects the ranking frequency across datasets.
Table 3: Mean validation Dice values (×100) for each dataset, noise scenario, and method. Dice scores ×100; best values in bold and green, second best underlined, worst in red. Dataset LIDC
RIGA
GleasonHD
MouseT
MMIS
MMIA
Scenario clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy
FedAvg 56.4 56.7 52.6 49.5 93.2 92.8 89.0 86.1 32.1 33.5 32.4 30.6 92.5 92.7 91.0 90.5 81.2 80.5 77.4 74.8 68.3 61.5 63.3 63.1
FedA3I 52.9 54.1 51.6 49.1 84.7 81.5 80.3 77.3 28.4 30.6 29.4 30.4 89.9 87.7 83.1 86.1 81.0 79.6 77.1 74.6 64.8 65.5 66.4 63.8
IOP-FL 58.6 58.5 53.6 50.5 93.2 92.6 88.9 86.5 33.2 34.2 28.7 27.1 90.9 91.5 91.3 89.8 81.8 81.9 78.6 77.6 69.8 66.6 67.1 69.3
FedCorr 54.4 54.4 51.1 46.9 82.2 78.8 79.5 74.4 39.1 38.1 34.5 35.0 83.6 89.9 87.8 88.3 81.3 80.7 77.3 75.1 60.8 65.7 67.1 65.4
FedSelect 57.1 57.6 53.1 50.3 93.1 92.4 88.8 86.2 37.4 33.0 32.1 30.9 92.5 92.0 91.4 90.7 84.3 83.2 79.8 76.7 68.0 66.1 66.4 67.8
achieve the highest instance-wise agreement (Appendix C.2.2). For label swapping, assessed by class confusion, only GleasonHD is included. FedSelect ranks best overall, followed by FedCorr (Appendix C.2.3). Given the single dataset, mixed noise, low performance, and limited consensus due to strong inter-rater disagreement, conclusions should be interpreted with caution. Statistical testing supports these trends
12
Fig. 5: Clean-referenced robustness of FNLL methods across the client-noise scenarios roa, roc, and noisy, shown as the absolute Dice difference to clean performance. Black bars denote the mean across datasets, and light gray points the corresponding bootstrap distribution.
selectively: significant improvements over FedAvg are observed primarily for FedSelect in instance-level robustness (F1) across scenarios and for both FedSelect and IOPFL in contour-related settings (HD95, roc ), while no significant gains are found for class-confusion robustness (Appendix C.3). Overall, the noise-type-specific analysis identifies FedSelect as the most consistent method across all three noise dimensions, supporting the Dice-based results and confirming it as the strongest FNLL method in our benchmark. We translate the combined Dice and noise-sensitive ranking analyses into a decision guide (Table 4; Appendix C.1, C.2.1, C.2.2, C.2.3) to support targeted FNLL method selection in deployment-relevant noisy federated settings. Recommended methods are based on lowest mean bootstrap rank, while bold entries indicate cases with statistically significant improvements over FedAvg after Holm-Bonferroni correction.
Fig. 6: Rank stability of segmentation performances w.r.t. noise-type-specific metrics based on bootstrapping across datasets, client-noise scenarios and methods. Lower ranks indicate better performance, bubble size reflects the ranking frequency across datasets.
13
Table 4: Decision guide for selecting FNLL methods according to the dominant segmentation label-noise type and client-noise scenario. Noise type (metric)
Client-noise scenario clean
roa
roc
noisy
overall
Contour (HD95) Instance (F1) Confusion (ClsConf)
FedSelect FedSelect FedSelect
FedAvg IOP-FL FedCorr
FedSelect FedSelect FedAvg
FedAvg FedSelect FedSelect
FedSelect FedSelect FedSelect
General (Dice)
FedSelect
IOP-FL
FedSelect
FedSelect
FedSelect
Recommended methods are selected by the lowest mean bootstrap rank for each metric and scenario. Bold entries indicate Holm-Bonferroni corrected significance versus FedAvg.
4 Discussion This study addresses a central gap in FNLL for medical image segmentation: the lack of a standardized benchmark that combines diverse datasets, numerous realworld segmentation label noise types, multiple client-noise scenarios, and task-relevant evaluation to enable informed decision-making for FNLL method selection. The included datasets span diverse modalities, dimensionalities, target structures, and noise origins, and the in-depth analysis shows that real-world segmentation label noise occurs both in isolation and in combinations of contour disagreement, missing or additional target structures, and class confusion. Noise-sensitive metrics enable explicit noise-type characterization, supporting both dataset interpretation and analysis of method behavior across noise regimes. Consensus quality varies across datasets: GleasonHD shows the lowest consensus cleanliness and most severe heterogeneous noisy-mask characteristics, partly explaining its role as the most challenging benchmark case and warranting caution for label-confusing conclusions. Overall, however, the consensus analysis indicates that, except for GleasonHD, evaluation against the derived clean reference provides a fair basis for method comparison. The comparative benchmark shows that the combination of realistic data with noise and standardized evaluation is sufficiently discriminative to reveal meaningful differences between FNLL strategies. Across datasets and client-noise scenarios, FedSelect emerges as the strongest overall method, with IOP-FL as the only comparable alternative, while FedAvg remains a strong baseline and FedA3I and FedCorr are less consistently competitive. Importantly, dedicated FNLL methods improve over the widely used FedAvg baseline only in specific noise regimes, highlighting the need for informed method selection rather than assuming universal FNLL superiority. This is further reflected in the statistical analysis, where only a subset of these improvements are significant, indicating that performance gains over FedAvg are not consistently robust across all settings. At the same time, “best” depends on the notion of robustness considered. FedSelect achieves the highest absolute performance and is most consistent across noise-type-specific analyses, whereas FedCorr shows the smallest clean-referenced Dice degradation in the more harmful roc and fully noisy settings, largely driven by its performance on the challenging GleasonHD dataset. This distinction between absolute performance and clean-referenced robustness would 14
remain hidden in narrower benchmarks. To facilitate practical use, we summarize the relationships between label-noise characteristics, client-noise scenarios, and FNLL performance in an actionable decision guide (Table 4), combining rank-based recommendations with statistical evidence against FedAvg. The guide identifies FedSelect as a robust default and FedAvg or IOP-FL as preferable in specific settings, supporting informed FNLL method selection in deployment-relevant federated settings. Although the benchmark suite combines curated datasets, representative FNLL methods, and a comprehensive evaluation protocol, several limitations remain. The results represent a structured snapshot of the current FNLL landscape, with hyperparameters optimized for average performance across datasets, favoring comparability over dataset-specific optimality. Comparisons between partially noisy settings (roa vs. roc ) should be interpreted with care, as effective noise levels vary substantially in roc, making both settings complementary rather than directly comparable robustness probes. Noise representation is imbalanced: contour-related noise dominates, instancelevel noise is mainly represented by MMIA, and class-confusion noise primarily by GleasonHD. Accordingly, conclusions are strongest for contour robustness, moderate for instance-level noise, and preliminary for class-swapping. The decision guide should be interpreted as a data-driven heuristic, as optimal method selection may vary with dataset-specific characteristics. Future work should extend the suite with additional datasets featuring instance-level and multiclass confusion noise, and evaluate partial-noise settings across multiple noise levels. By combining real-world noisy datasets, explicit noise characterization, standardized federated training, and competitive FNLL baselines, the suite supports practical use beyond this comparative study, including FNLL method selection, characterization of new datasets, benchmarking on new data, and reproducible evaluation of future methods under deployment-relevant noisy conditions. Its main value therefore lies not only in the benchmark conclusions, but in providing a reusable foundation for FNLL method development, fair comparison, and dataset-driven analysis in realistic federated segmentation settings.
5 Conclusion We presented a benchmark suite for FNLL in cross-silo medical image segmentation, combining diverse real-world noisy datasets, clinically relevant client-noise scenarios, and targeted evaluation. The dataset analysis shows that the suite covers multiple practically relevant segmentation label-noise types and provides context for interpreting method behavior across datasets and noise regimes. The comparative benchmark identifies FedSelect as the strongest overall FNLL method and IOP-FL as the most competitive alternative, while showing that gains over the strong FedAvg baseline remain dataset-, noise-, and scenario-dependent. Through an actionable decision guide, the suite supports informed FNLL method selection in deployment-relevant federated settings. Beyond the present benchmark, the released suite provides a reusable foundation for fair comparison, label-noise characterization, and future method development, supporting more reliable federated segmentation and downstream clinical decision-making.
15
Supplementary information. Supplementary materials are provided in the Appendix below. Acknowledgements. Maximilian Zenk and Ünal Akünal for guidiance in the early phase of the project.
16
Statements and Declarations • Funding: This research was funded by the German Federal Ministry of Education and Research (BMBF) as part of the University Medicine Network (Project RACOON, 01KX2021), as part of the PrivateAIM project (01ZZ2316M), and as part of the Research Campus M2OLIE, within the Framework “Forschungscampus: Public-private partnership for Innovations” (13GW0388A). • Competing interests: The authors have no competing interests to declare that are relevant to the content of this article. • Author contributions: Markus Ralf Bujotzek: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review & editing. Dimitrios Bounias: Conceptualization, Formal analysis, Investigation, Validation, Writing – review & editing. Stefan Denner: Conceptualization, Formal analysis, Investigation, Methodology, Writing – review & editing. Ralf Floca: Conceptualization, Project administration, Supervision, Writing – review & editing. Maximilian Fischer: Writing – review & editing. Peter Neher: Supervision, Writing – review & editing. Klaus H. Maier-Hein: Funding acquisition, Project administration, Resources, Writing – review & editing. • Ethics approval: This study exclusively used publicly available datasets. All data were collected and made available by the original studies in accordance with relevant ethical guidelines and approvals. No additional ethical approval was required for this work. • Consent to participate: Not applicable. This study used only publicly available, de-identified data. • Consent to publish: Not applicable. This study did not involve any identifiable individual data. • Data availability: All datasets used in this study are publicly available from their original sources. The LIDC-IDRI dataset is available from The Cancer Imaging Archive at https://www.cancerimagingarchive.net/collection/lidc-idri/. The RIGA dataset is available through Deep Blue Data at https: //deepblue.lib.umich.edu/data/concern/data sets/3b591905z. GleasonXAI image data are available from Harvard Dataverse at https://dataverse. harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/OCYCMP, and the corresponding multi-rater annotations are available from Figshare at https://springernature.figshare.com/articles/dataset/Pathologist-like explainable AI for interpretable Gleason grading in prostate cancer/27301845. The MouseTumor dataset is described in Scientific Data and available through the resources listed in the publication at https://www.nature.com/articles/s41597-024-03814-y#Tab3. The MMIA dataset is available via Synapse at https://www.synapse.org/Synapse: syn60868042/wiki/628716. The MMIS dataset is available from the MMIS 2024 challenge website at https://mmis2024.vercel.app/. • Code availability: The benchmark suite code, including scripts for dataset integration, federated training, evaluation, and reproduction of the comparative benchmark, is publicly available at https://github.com/MIC-DKFZ/FedSegNoiseBench.
17
Appendix A A.1
Evaluation
Evaluation metric edge-case handling
Explicit edge-case handling is required in label-noisy segmentation benchmarks to avoid undefined metric behavior while ensuring that missing, spurious, or empty predictions are neither unfairly rewarded nor insufficiently penalized. For Dice, classes absent in both ground truth and prediction are assigned NaN and ignored during nanmean-based aggregation. If only one mask contains the class, or if both masks are non-empty but non-overlapping, Dice is set to 0. For HD95, empty masks are handled explicitly: if both masks are empty, HD95 is set to 0; if only one mask is empty, HD95 is set to 1000 as a fixed large penalty. Otherwise, HD95 is computed from symmetric surface distances using physical voxel spacing, with unit spacing used when spacing metadata is missing or inconsistent. For foreground-background instance-level F1, all non-background labels are merged into a foreground mask before instance extraction. If both masks contain no foreground instances, the score is NaN; if only one mask contains foreground instances, the score is 0. Instance matching is one-to-one with an IoU threshold of 0.1, thereby penalizing split or merged objects through unmatched components. For class confusion, the score is defined only for foreground classes present in the ground truth; absent classes are assigned NaN. Background predictions on foreground regions and foreground predictions in background regions are not counted as class confusion. Only foreground-to-foreground swaps are counted, so the metric is NaN for single-foreground-class tasks.
18
Appendix B B.1
Dataset- and method-specific training details
Dataset-specific nnU-Net hyperparameters
The benchmark suite’s federated segmentation framework is based on the selfconfiguring nnU-Net framework, which derives experiment-planning and training hyperparameters from the processed dataset characteristics. Table B1 summarizes the dataset-specific batch size, patch size, median image size, and target spacing used in the experiments. Notably, we had to adjust the batch size to 1 for the FedSelect experiments due to computational limitations.
Table B1: Dataset-specific nnU-Net experiment-planning hyperparameters used in the federated benchmark, including batch size, patch size, median image size, and target spacing. Batch size Patch size Median image size (voxels) Target spacing
B.2
LIDC
RIGA
GleasonHD
MouseT
MMIS
MMIA
20 64, 64, 64 64, 64, 64
12 512, 512 1458, 1458
2 (1) 2048, 2048 3100, 3100
2 256, 96, 96 480, 192, 192
4 24, 192, 160 24, 187, 147
2 56, 192, 160 80, 256, 256
1.0, 1.0, 1.0
1.0, 1.0
1.0, 1.0
0.21, 0.21, 0.21
3.0, 0.508, 0.508
2.0, 0.703, 0.703
FNLL method-specific hyperparameters
For each compared FNLL method, hyperparameters were optimized by grid search over predefined candidate values. Final settings were selected according to the best average validation performance across all benchmark datasets, as summarized in Table B2.
Table B2: Grid-searched FNLL hyperparameters. Final values in bold denote the final selected settings based on the best average validation performance across all benchmark datasets. Method
Hyperparameter
Grid search values
FedA3I
feda3i warmup rounds frac feda3i interw
0.05, 0.1, 0.2 0.3, 0.5, 0.7
IOP-FL
iopfl alpha
0.05, 0.1, 0.2, 0.4, 0.6
FedCorr
fedcorr preproc rounds frac fedcorr relabel ratio fedcorr relabel confidence thres fedcorr proxterm beta
0.05, 0.1, 0.2 0.3, 0.5, 0.7 0.3, 0.5, 0.7 3, 5, 8
FedSelect
fedselect warmup rounds frac fedselect client select ratio fedselect sample select ratio fedselect meta momentum fedselect reward data size frac
0.05, 0.1, 0.2 0.6, 0.8, 1.0 0.5, 0.75, 1.0 0.3, 0.5, 0.7 0.05, 0.1, 0.2
19
Appendix C C.1
Detailed experimental results
General segmentation performance evaluation
General segmentation performance was assessed using the Dice score. Detailed results are reported in Table 3 and Figure C1.
Fig. C1: Bootstrap Dice score distributions of all benchmarked methods across clientnoise scenarios and all datasets. Black bars denote medians, and diamonds denote means.
20
Table C3: Mean validation Dice values (×100) with 95% percentile bootstrap confidence intervals for each dataset, noise scenario, and method. Dataset LIDC
RIGA
GleasonHD
MouseT
MMIS
MMIA
Scenario clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy
FedAvg 56.459.0 53.9 56.759.3 54.1 52.655.2 49.8 49.552.3 46.7 93.293.5 92.8 92.893.1 92.4 89.089.7 88.2 86.186.9 85.2 32.136.4 27.4 33.538.0 29.0 32.437.4 27.5 30.634.4 26.7 92.593.2 91.9 92.793.3 92.0 91.091.9 90.1 90.591.3 89.7 81.283.0 79.3 80.582.3 78.7 77.479.2 75.7 74.876.8 72.9 68.370.4 66.0 61.563.8 59.1 63.365.1 61.3 63.165.4 61.0
FedA3I 52.955.5 50.1 54.156.6 51.6 51.654.3 48.9 49.152.1 46.1 84.787.1 82.0 81.583.8 79.1 80.382.7 77.6 77.379.2 75.5 28.431.7 24.8 30.634.8 26.3 29.433.2 25.5 30.434.2 26.5 89.990.7 88.9 87.789.4 85.9 83.184.5 81.6 86.187.5 84.7 81.082.8 79.1 79.682.5 76.6 77.178.9 75.3 74.676.7 72.8 64.867.0 62.5 65.568.2 62.8 66.468.5 64.2 63.866.0 61.6
21
IOP-FL 58.661.5 56.0 58.561.4 55.8 53.656.4 50.7 50.553.6 47.7 93.293.6 92.9 92.693.0 92.2 88.989.6 88.2 86.587.2 85.7 33.237.2 29.0 34.238.7 30.0 28.732.4 25.0 27.130.6 23.5 90.992.2 89.2 91.592.3 90.8 91.392.0 90.6 89.891.0 88.3 81.883.7 79.7 81.983.5 80.2 78.680.1 76.9 77.679.0 75.9 69.871.8 67.7 66.668.8 64.3 67.169.3 64.8 69.371.1 67.3
FedCorr 54.457.1 51.7 54.457.1 51.8 51.153.6 48.4 46.949.6 44.3 82.284.4 79.8 78.881.3 76.0 79.581.6 77.4 74.476.7 72.1 39.143.8 34.5 38.143.1 32.9 34.539.2 29.9 35.039.2 30.7 83.685.7 81.1 89.990.8 88.8 87.888.9 86.5 88.389.2 87.3 81.383.1 79.3 80.782.9 78.4 77.379.1 75.6 75.177.0 73.2 60.863.5 58.1 65.768.4 62.6 67.169.3 64.7 65.467.8 63.1
FedSelect 57.159.7 54.5 57.660.5 54.9 53.155.8 50.3 50.353.1 47.4 93.193.4 92.7 92.492.8 92.0 88.889.6 88.1 86.287.0 85.3 37.442.4 32.7 33.036.9 29.2 32.136.8 27.3 30.934.4 27.0 92.593.1 91.8 92.092.7 91.3 91.492.2 90.6 90.791.5 89.8 84.385.6 82.9 83.284.6 81.7 79.881.0 78.5 76.778.0 75.3 68.070.1 66.0 66.168.2 64.1 66.468.4 64.4 67.869.7 65.8
C.2 C.2.1
Noise-specific segmentation performance evaluation Contour-based segmentation label noise
Contour-related segmentation label noise was assessed using the HD95 metric on datasets featuring this noise type. Detailed results are reported in Table C4 and Figures C2, C3.
Fig. C2: Bootstrap HD95 distributions of all benchmarked methods across client-noise scenarios and all datasets. Black bars denote medians, and diamonds denote means.
Fig. C3: Rank stability of HD95 segmentation performances based on bootstrapping across client-noise scenarios and methods on LIDC, RIDA, GleasonHD, MouseT, MMIS, and MMIA. Lower ranks indicate better performances, bubble size reflects the ranking frequency across datasets.
22
Table C4: Mean validation HD95 values with 95% percentile bootstrap confidence intervals for each dataset, noise scenario, and method. Dataset LIDC
RIGA
GleasonHD
MouseT
MMIS
MMIA
Scenario clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy
FedAvg 279.9312.6 249.3 280.2312.8 249.8 322.7355.9 291.5 341.4375.9 307.0 16.927.2 11.1 12.815.1 11.4 18.119.8 16.5 26.537.3 20.5 561.7613.4 514.6 570.0619.3 522.4 577.9636.3 523.8 586.9636.6 537.9 10.715.5 6.4 4.56.3 3.1 16.824.6 10.4 12.818.3 8.0 8.311.0 5.3 6.89.2 4.5 9.011.6 6.4 11.015.0 7.2 86.391.4 81.4 98.4104.1 93.2 95.6100.6 90.4 96.9102.9 91.4
FedA3I 289.8321.2 260.1 282.0314.3 251.7 310.4343.6 278.6 324.1357.8 289.2 62.685.8 41.3 71.588.7 55.4 83.2102.7 64.3 81.9101.3 65.2 659.1711.0 610.0 575.2626.7 526.0 595.0651.6 540.4 618.4665.5 574.3 20.326.7 14.3 21.629.2 15.3 110.4127.5 94.5 41.252.5 31.1 8.911.9 6.0 8.111.3 5.2 9.813.2 6.6 10.715.0 7.1 86.691.0 81.9 82.687.1 77.9 71.176.1 65.9 87.992.9 82.5
23
IOP-FL 275.4308.7 244.4 278.0310.7 247.3 321.2354.4 289.6 347.1380.6 309.8 22.443.4 11.4 18.029.8 11.8 22.934.2 16.4 20.923.3 19.0 621.8665.9 579.6 582.1627.1 539.9 650.9695.7 607.9 708.4755.8 664.4 21.529.8 13.6 14.319.3 9.8 11.116.3 6.7 14.6 9.96.1 5.16.7 3.6 5.36.7 3.9 6.07.5 4.8 7.29.8 4.9 81.186.2 75.9 86.091.3 80.2 88.593.4 83.1 83.988.7 78.9
FedCorr 280.6313.2 250.1 280.4313.0 250.1 324.9357.9 292.9 345.5379.3 308.4 73.491.2 57.2 109.6140.6 82.7 90.0116.0 68.7 100.9121.3 82.2 529.6584.2 477.6 505.0555.6 457.9 561.8611.7 515.9 571.8620.5 527.8 85.9109.9 65.5 27.735.6 20.1 26.233.0 18.8 31.639.9 23.5 7.69.9 5.1 7.410.9 4.4 9.814.6 6.2 10.314.5 6.9 90.595.5 85.1 75.580.3 70.4 70.775.8 65.4 87.492.4 82.0
FedSelect 282.7315.4 252.7 282.7315.4 252.7 317.5350.3 285.4 340.0374.5 306.1 15.825.3 10.8 12.413.0 11.7 17.519.0 16.1 26.538.3 19.3 514.9564.3 466.8 586.0629.5 540.6 571.6620.4 527.9 610.0655.2 567.3 7.311.0 4.4 14.319.4 10.0 7.511.2 4.7 7.811.4 4.8 3.03.4 2.6 3.53.9 3.0 4.14.5 3.6 5.76.9 4.6 85.390.3 80.7 86.390.8 81.8 88.393.2 83.6 88.593.1 83.9
C.2.2
Instance-based segmentation label noise
Instance-related segmentation label noise was assessed using the foregroundbackground instance-level F1 score on datasets featuring this noise type. Detailed results are reported in Table C5 and Figures C4, C4.
Fig. C4: Bootstrap foreground-background instance-level F1 score distributions of all benchmarked methods across client-noise scenarios and all datasets featuring instancebased noise (LIDC, GleasonHD, MouseT, MMIS, MMIA). Black bars denote medians, and diamonds denote means.
Fig. C5: Rank stability of foreground-background instance-level F1 segmentation performances based on bootstrapping across client-noise scenarios and methods on LIDC, GleasonHD, MouseT, MMIS and MMIA. Lower ranks indicate better performances, bubble size reflects the ranking frequency across datasets.
24
Table C5: Mean validation foreground-background instance-level F1 values (×100) with 95% percentile bootstrap confidence intervals for each dataset, noise scenario, and method. Dataset LIDC
GleasonHD
MouseT
MMIS
MMIA
Scenario clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy clean roa roc noisy
FedAvg 70.473.8 67.2 70.874.0 67.5 66.369.5 62.9 63.667.2 60.3 0.50.6 0.5 0.50.6 0.5 0.30.3 0.2 0.60.7 0.5 90.492.8 88.1 95.997.5 94.2 88.190.8 85.6 89.091.4 86.7 81.286.3 76.2 83.287.2 79.1 83.489.4 77.7 73.676.7 70.1 32.735.4 30.2 25.627.8 23.5 18.920.5 17.3 16.718.3 15.2
FedA3I 67.070.2 63.4 68.872.0 65.7 66.169.7 62.6 64.067.9 60.1 0.60.8 0.5 0.30.4 0.3 0.50.5 0.4 0.70.9 0.6 85.087.6 82.3 87.189.5 84.5 60.964.0 57.9 79.582.3 76.4 78.583.6 73.4 86.392.5 80.3 76.482.1 70.8 72.476.9 68.3 24.126.3 21.9 22.924.9 21.0 20.122.6 17.9 16.117.7 14.6
25
IOP-FL 71.074.3 67.8 71.174.0 67.9 66.269.5 62.8 63.667.4 60.2 0.60.6 0.5 0.40.4 0.3 0.40.5 0.4 0.40.5 0.4 89.692.1 87.2 89.892.3 87.3 91.493.4 89.3 91.693.9 89.0 89.392.5 86.1 94.297.2 91.1 87.992.6 83.4 86.390.1 82.6 33.536.3 30.6 29.331.3 27.4 21.723.3 20.1 20.522.2 18.8
FedCorr 69.372.7 66.0 70.373.5 67.0 65.568.6 62.0 61.865.4 58.3 0.70.8 0.6 0.30.3 0.3 0.40.4 0.4 1.01.2 0.9 66.069.1 62.8 85.387.6 83.2 87.790.2 85.3 85.087.3 82.4 78.884.8 73.2 80.685.3 76.4 82.789.1 76.7 74.880.3 69.4 26.028.2 23.8 28.330.7 25.8 21.724.1 19.7 15.817.1 14.6
FedSelect 70.773.9 67.5 70.373.5 67.1 66.169.5 62.8 63.967.5 60.4 0.50.5 0.4 0.20.2 0.2 0.20.3 0.2 0.30.3 0.2 93.295.3 91.1 90.993.1 88.9 94.296.3 91.9 93.995.7 91.9 89.193.8 83.9 89.592.7 85.5 86.590.2 82.7 83.688.6 78.6 34.737.3 32.2 32.235.4 29.3 23.426.0 21.2 21.323.7 19.1
C.2.3
Confusion-based segmentation label noise
Confusion-related segmentation label noise was assessed using the voxel-level class confusion metric on datasets featuring this noise type. Detailed results are reported in Table C6 and Figures C6, C7.
Fig. C6: Bootstrap Class confusion score distributions of all benchmarked methods across client-noise scenarios and all datasets featuring confusion-based noise (GleasonHD). Black bars denote medians, and diamonds denote means.
Table C6: Mean validation ClsConf values with 95% percentile bootstrap confidence intervals for each dataset, noise scenario, and method. Dataset GleasonHD
Scenario clean roa roc noisy
FedAvg 31.235.2 27.9 27.932.4 23.9 23.828.0 20.0 35.639.7 31.7
FedA3I 30.834.6 27.3 26.831.5 22.7 31.135.9 26.5 29.733.8 26.0
26
IOP-FL 29.133.5 25.0 28.532.9 24.3 32.636.3 29.0 31.736.1 27.6
FedCorr 25.730.0 22.0 29.333.7 25.2 28.332.6 24.3 30.334.7 26.1
FedSelect 25.829.1 22.4 27.231.1 23.7 26.930.9 23.1 25.629.5 22.3
Fig. C7: Rank stability of Class confusion segmentation performances based on bootstrapping across client-noise scenarios and methods on GleasonHD. Lower ranks indicate better performances, bubble size reflects the ranking frequency across datasets.
27
C.3
Statistical evaluation of segmentation performances
All tests compare each FNLL method against FedAvg using one-sided paired Wilcoxon signed-rank tests on case-level scores (Table C7). For each metric, scenario, and method, scores are matched to FedAvg by case identifier within each dataset and pooled across datasets. Positive ∆ values indicate improvement over FedAvg, corresponding to lower values for HD95 and class confusion. Dice uses all datasets, whereas noise-sensitive metrics use only datasets featuring the respective noise type: all datasets for HD95, LIDC, GleasonHD, MouseT, MMIS, and MMIA for foregroundbackground instance-level F1, and GleasonHD for class confusion. Holm–Bonferroni correction is applied across the four method comparisons per test family; underlined p-values indicate uncorrected significance, and bold corrected p-values indicate significance after correction.
Table C7: Per-metric, per-scenario Wilcoxon signed-rank tests against FedAvg. Metric
Scenario
Method
n
∆mean
∆median
p
pHolm
clean
FedA3I IOP-FL FedCorr FedSelect
3261 3263 3266 3266
-0.0330 -0.0048 -0.0379 -0.0022
-0.0081 +0.0000 -0.0085 +0.0000
1.0000 0.6592 1.0000 0.9991
1.0000 1.0000 1.0000 1.0000
roa
FedA3I IOP-FL FedCorr FedSelect
3266 3265 3267 3266
-0.0129 +0.0097 -0.0101 +0.0029
-0.0028 +0.0000 -0.0035 +0.0000
1.0000 1.14e-06 1.0000 0.8275
1.0000 4.55e-06 1.0000 1.0000
roc
FedA3I IOP-FL FedCorr FedSelect
3257 3262 3264 3258
-0.0149 +0.0003 -0.0123 +0.0038
-0.0021 +0.0000 -0.0011 +0.0000
1.0000 0.0106 1.0000 0.7754
1.0000 0.0423 1.0000 1.0000
noisy
FedA3I IOP-FL FedCorr FedSelect
3250 3262 3259 3260
-0.0157 +0.0078 -0.0193 +0.0032
-0.0021 +0.0000 -0.0016 +0.0000
1.0000 1.07e-10 1.0000 0.3777
1.0000 4.26e-10 1.0000 1.0000
clean
FedA3I IOP-FL FedCorr FedSelect
3268 3268 3268 3268
-17.9889 -6.0093 -15.5687 +3.4931
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 0.9931 1.0000 0.5947
1.0000 1.0000 1.0000 1.0000
roa
FedA3I IOP-FL FedCorr FedSelect
3268 3268 3268 3268
-4.9229 +1.4570 -3.1700 -0.4616
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 0.1147 1.0000 0.8167
1.0000 0.4586 1.0000 1.0000
FedA3I
3268
-11.7246
+0.0000
1.0000
1.0000
Dice
HD95
Continued on next page
roc 28
Table C7: Per-metric, per-scenario Wilcoxon signed-rank tests against FedAvg. (continued) Metric
Method
n
∆mean
∆median
p
pHolm
IOP-FL FedCorr FedSelect
3268 3268 3268
-4.9786 -5.7519 +3.9923
+0.0000 +0.0000 +0.0000
0.0084 0.9856 0.0023
0.0253 1.0000 0.0091
noisy
FedA3I IOP-FL FedCorr FedSelect
3268 3268 3268 3268
-10.0181 -9.0351 -11.7081 -0.7138
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 0.0438 1.0000 0.1112
1.0000 0.1751 1.0000 0.3337
clean
FedA3I IOP-FL FedCorr FedSelect
3261 3263 3266 3266
-0.0461 -0.0018 -0.0610 +0.0077
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 0.5221 1.0000 0.0015
1.0000 1.0000 1.0000 0.0058
roa
FedA3I IOP-FL FedCorr FedSelect
3266 3265 3267 3266
-0.0344 +0.0021 -0.0287 +0.0072
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 0.1803 1.0000 2.18e-06
1.0000 0.5408 1.0000 8.74e-06
roc
FedA3I IOP-FL FedCorr FedSelect
3257 3262 3264 3258
-0.0401 -0.0005 -0.0201 +0.0132
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 0.3762 0.9162 2.43e-09
1.0000 1.0000 1.0000 9.71e-09
noisy
FedA3I IOP-FL FedCorr FedSelect
3250 3262 3259 3260
-0.0346 +0.0075 -0.0314 +0.0110
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 1.57e-08 1.0000 1.31e-07
1.0000 6.27e-08 1.0000 3.93e-07
clean
FedA3I IOP-FL FedCorr FedSelect
738 738 738 738
+0.0030 +0.0075 +0.0068 +0.0166
-0.0055 +0.0000 -0.0039 -0.0005
1.0000 0.2027 1.0000 1.0000
1.0000 0.8107 1.0000 1.0000
roa
FedA3I IOP-FL FedCorr FedSelect
738 738 738 738
-0.0050 -0.0003 -0.0024 +0.0006
-0.0018 +0.0000 -0.0002 -0.0021
1.0000 0.8516 1.0000 1.0000
1.0000 1.0000 1.0000 1.0000
roc
FedA3I IOP-FL FedCorr FedSelect
738 738 738 738
-0.0214 -0.0281 -0.0109 -0.0133
+0.0000 -0.0006 +0.0000 -0.0004
0.9885 0.9995 0.9326 1.0000
1.0000 1.0000 1.0000 1.0000
FedA3I
738
+0.0101
+0.0000
0.1729
0.5187
Scenario
F1
ClsConf
Continued on next page
noisy
29
Table C7: Per-metric, per-scenario Wilcoxon signed-rank tests against FedAvg. (continued) Metric
Scenario
Method
n
∆mean
∆median
p
pHolm
IOP-FL FedCorr FedSelect
738 738 738
+0.0038 +0.0011 +0.0194
+0.0000 -0.0000 +0.0000
0.0076 0.9888 0.2477
0.0305 0.9888 0.5187
30
To increase statistical power while preserving interpretability, we additionally pool case-level paired observations across datasets and client-noise scenarios within each metric (Table C8). The test direction, effect definition, and correction procedure remain identical, with Holm-Bonferroni correction applied across the four method comparisons per metric.
Table C8: Pooled dataset-by-scenario Wilcoxon signed-rank tests against FedAvg. Metric
Scenario
Method
n
∆mean
∆median
p
pHolm
Dice
ALL
FedA3I IOP-FL FedCorr FedSelect
13034 13052 13056 13050
-0.0191 +0.0032 -0.0199 +0.0019
-0.0038 +0.0000 -0.0036 +0.0000
1.0000 3.44e-11 1.0000 0.9874
1.0000 1.38e-10 1.0000 1.0000
ALL
FedA3I IOP-FL FedCorr FedSelect
13072 13072 13072 13072
-11.1636 -4.6415 -9.0497 +1.5775
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 0.0772 1.0000 0.0677
1.0000 0.2708 1.0000 0.2708
F1
ALL
FedA3I IOP-FL FedCorr FedSelect
13034 13052 13056 13050
-0.0388 +0.0018 -0.0353 +0.0098
+0.0000 +0.0000 +0.0000 +0.0000
1.0000 5.17e-04 1.0000 5.39e-21
1.0000 0.0016 1.0000 2.16e-20
ClsConf
ALL
FedA3I IOP-FL FedCorr FedSelect
2952 2952 2952 2952
-0.0033 -0.0043 -0.0013 +0.0058
-0.0003 +0.0000 -0.0000 -0.0003
1.0000 0.7221 1.0000 1.0000
1.0000 1.0000 1.0000 1.0000
HD95
Given the large number of paired case-level observations, the Wilcoxon test primarily reflects the consistency of directional improvements rather than their magnitude. Consequently, statistically significant results indicate systematic per-case gains over FedAvg, whereas non-significant results often reflect heterogeneous or dataset-dependent behavior.
31
References [1] Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H.R., Albarqouni, S., Bakas, S., Galtier, M.N., Landman, B.A., Maier-Hein, K., et al.: The future of digital health with federated learning. NPJ digital medicine 3(1), 119 (2020) [2] Sheller, M.J., Edwards, B., Reina, G.A., Martin, J., Pati, S., Kotrotsou, A., Milchenko, M., Xu, W., Marcus, D., Colen, R.R., Bakas, S.: Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific Reports 10 (2020) [3] Pati, S., Baid, U., Edwards, B., Sheller, M., Wang, S.-H., Reina, G.A., Foley, P., Gruzdev, A., Karkada, D., Davatzikos, C., et al.: Federated learning enables big data for rare cancer boundary detection. Nature communications 13(1), 7346 (2022) [4] Bujotzek, M.R., Akünal, Ü., Denner, S., Neher, P., Zenk, M., Frodl, E., Jaiswal, A., Kim, M., Krekiehn, N.R., Nickel, M., et al.: Real-world federated learning in radiology: hurdles to overcome and benefits to gain. Journal of the American Medical Informatics Association 32(1), 193–205 (2025) [5] Oldenhof, M., Ács, G., Pejó, B., Schuffenhauer, A., Holway, N., Sturm, N., Dieckmann, A., Fortmeier, O., Boniface, E., Mayer, C., et al.: Industry-scale orchestrated federated learning for drug discovery. In: Proceedings of the Aaai Conference on Artificial Intelligence, vol. 37, pp. 15576–15584 (2023) [6] Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., et al.: The medical segmentation decathlon. Nature communications 13(1), 4128 (2022) [7] Karimi, D., Dou, H., Warfield, S.K., Gholipour, A.: Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical image analysis 65, 101759 (2020) [8] Aliotta, E., Nourzadeh, H., Siebers, J.: Quantifying the dosimetric impact of organ-at-risk delineation variability in head and neck radiation therapy in the context of patient setup uncertainty. Physics in Medicine & Biology 64(13), 135020 (2019) [9] Poel, R., Rüfenacht, E., Ermis, E., Müller, M., Fix, M.K., Aebersold, D.M., Manser, P., Reyes, M.: Impact of random outliers in auto-segmented targets on radiotherapy treatment plans for glioblastoma. Radiation Oncology 17(1), 170 (2022) [10] Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature communications 15(1), 654 (2024)
32
[11] Fang, X., Ye, M.: Robust federated learning with noisy and heterogeneous clients. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10072–10081 (2022) [12] Armato III, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D.R., Henschke, C.I., Hoffman, E.A., et al.: The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics 38(2), 915–931 (2011) [13] Liao, Z., Hu, S., Xie, Y., Xia, Y.: Modeling annotator preference and stochastic annotation error for medical image segmentation. Medical Image Analysis 92, 103028 (2024) [14] Zhong, Q., Guo, T., Rechsteiner, M., Rüschoff, J.H., Rupp, N., Fankhauser, C., Saba, K., Mortezavi, A., Poyet, C., Hermanns, T., et al.: A curated collection of tissue microarray images and clinical outcome data of prostate cancer patients. Scientific data 4(1), 170014 (2017) [15] Jensen, M., Clemmensen, A., Hansen, J.G., Krimpen Mortensen, J., Christensen, E.N., Kjaer, A., Ripa, R.S.: 3d whole body preclinical micro-ct database of subcutaneous tumors in mice with annotations from 3 annotators. Scientific Data 11(1), 1021 (2024) [16] Wu, Y., Xie, Y., Luo, X., Wu, Q., Cai, J.: Dataset, challenge, and evaluation for tumor segmentation variability. In: Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11302–11303 (2024) [17] Garrucho, L., Kushibar, K., Reidel, C.-A., Joshi, S., Osuala, R., Tsirikoglou, A., Bobowicz, M., Riego, J.d., Catanese, A., Gwoździewicz, K., Cosaka, M.L., Abo-Elhoda, P.M., Tantawy, S.W., Sakrana, S.S., Shawky-Abdelfatah, N.O., Salem, A.M.A., Kozana, A., Divjak, E., Ivanac, G., Nikiforaki, K., Klontzas, M.E., Garcı́a-Dosdá, R., Gulsun-Akpinar, M., Lafcı, O., Mann, R., Martı́n-Isla, C., Prior, F., Marias, K., Starmans, M.P.A., Strand, F., Dı́az, O., Igual, L., Lekadir, K.: A large-scale multicenter breast cancer dce-mri benchmark dataset with expert segmentations. Scientific Data 12(1), 453 (2025) https://doi.org/10. 1038/s41597-025-04707-4 [18] Shi, J., Zhang, K., Guo, C., Yang, Y., Xu, Y., Wu, J.: A survey of label-noise deep learning for medical image analysis. Medical image analysis 95, 103166 (2024) [19] Wei, Y., Deng, Y., Sun, C., Lin, M., Jiang, H., Peng, Y.: Deep learning with noisy labels in medical prediction problems: a scoping review. Journal of the American Medical Informatics Association 31(7), 1596–1607 (2024) [20] Song, H., Kim, M., Park, D., Shin, Y., Lee, J.-G.: Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and
33
learning systems 34(11), 8135–8153 (2022) [21] Dgani, Y., Greenspan, H., Goldberger, J.: Training a neural network based on unreliable human annotation of medical images. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 39–42 (2018). IEEE [22] Hu, K., Huang, Y., Huang, W., Tan, H., Chen, Z., Zhong, Z., Li, X., Zhang, Y., Gao, X.: Deep supervised learning using self-adaptive auxiliary loss for covid-19 diagnosis from imbalanced ct images. Neurocomputing 458, 232–245 (2021) [23] Chen, H., Tan, W., Li, J., Guan, P., Wu, L., Yan, B., Li, J., Wang, Y.: Adaptive cross entropy for ultrasmall object detection in computed tomography with noisy labels. Computers in Biology and Medicine 147, 105763 (2022) [24] Pham, H.H., Le, T.T., Tran, D.Q., Ngo, D.T., Nguyen, H.Q.: Interpreting chest x-rays via cnns that exploit hierarchical disease dependencies and uncertainty labels. Neurocomputing 437, 186–194 (2021) [25] Qian, C., Han, K., Ding, J., Lyu, C., Yuan, Z., Chen, J., Liu, Z.: Adaptive label correction for robust medical image segmentation with noisy labels. arXiv preprint arXiv:2503.12218 (2025) [26] Zhou, T., Wang, S., Bilmes, J.: Robust curriculum learning: from clean label detection to noisy label self-correction. In: International Conference on Learning Representations (2020) [27] Li, P., Purkait, P., Ajanthan, T., Abdolshah, M., Garg, R., Husain, H., Xu, C., Gould, S., Ouyang, W., Van Den Hengel, A.: Semi-supervised semantic segmentation under label noise via diverse learning groups. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1229–1238 (2023) [28] Zheng, G., Awadallah, A.H., Dumais, S.: Meta label correction for noisy label learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11053–11061 (2021) [29] Huang, B., Lin, Y., Xu, C.: Contrastive label correction for noisy label learning. Information sciences 611, 173–184 (2022) [30] Ma, F., Wu, Y., Yu, X., Yang, Y.: Learning with noisy labels via self-reweighting from class centroids. IEEE Transactions on Neural Networks and Learning Systems 33(11), 6275–6285 (2021) [31] Li, J., Song, Y., Zhu, J., Cheng, L., Su, Y., Ye, L., Yuan, P., Han, S.: Learning from large-scale noisy web data with ubiquitous reweighting for image classification. IEEE transactions on pattern analysis and machine intelligence 43(5), 1808–1814 (2019)
34
[32] Mirikharaji, Z., Yan, Y., Hamarneh, G.: Learning to segment skin lesions from noisy annotations. In: MICCAI Workshop on Domain Adaptation and Representation Transfer, pp. 207–215 (2019). Springer [33] Xiao, L., Li, Y., Qv, L., Tian, X., Peng, Y., Zhou, S.K.: Pathological image segmentation with noisy labels. arXiv preprint arXiv:2104.02602 (2021) [34] Mindermann, S., Brauner, J.M., Razzak, M.T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A.N., Morisot, A., Farquhar, S., et al.: Prioritized training on points that are learnable, worth learning, and not yet learnt. In: International Conference on Machine Learning, pp. 15630–15649 (2022). PMLR [35] Xia, X., Liu, T., Han, B., Gong, M., Yu, J., Niu, G., Sugiyama, M.: Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.00445 (2021) [36] Kim, T., Ko, J., Choi, J., Yun, S.-Y., et al.: Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems 34, 24137–24149 (2021) [37] Ribeiro, V., Avila, S., Valle, E.: Less is more: Sample selection and label conditioning improve skin lesion segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 738–739 (2020) [38] Xu, J., Chen, Z., Quek, T.Q., Chong, K.F.E.: Fedcorr: Multi-stage federated learning for label noise correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10184–10193 (2022) [39] Zhang, Y., Miao, X., Li, B., Wu, Y., Shang, Y.: Proxy-validated importanceaware federated sample selection with meta learning. In: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 3855–3866 (2025) [40] Wu, N., Yu, L., Jiang, X., Cheng, K.-T., Yan, Z.: Fednoro: Towards noise-robust federated learning by addressing class imbalance and label noise heterogeneity. arXiv preprint arXiv:2305.05230 (2023) [41] Jiang, X., Sun, S., Li, J., Xue, J., Li, R., Wu, Z., Xu, G., Wang, Y., Liu, M.: Tackling noisy clients in federated learning with end-to-end label correction. In: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 1015–1026 (2024) [42] Chen, Z., Li, W., Xing, X., Yuan, Y.: Medical federated learning with joint graph purification for noisy label learning. Medical Image Analysis 90, 102976 (2023) [43] Wu, N., Sun, Z., Yan, Z., Yu, L.: Feda3i: annotation quality-aware aggregation for
35
federated medical image segmentation against heterogeneous annotation noise. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 15943–15951 (2024) [44] Xiang, Y., Wu, N., Yu, L., Yang, X., Cheng, K.-T., Yan, Z.: Fedia: Federated medical image segmentation with heterogeneous annotation completeness. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 373–382 (2024). Springer [45] Zhu, M., Chen, Z., Yuan, Y.: Feddm: Federated weakly supervised segmentation via annotation calibration and gradient de-conflicting. IEEE Transactions on Medical Imaging 42(6), 1632–1643 (2023) [46] Wang, Y., Ding, W., Lin, W., Tan, T., Gao, Z.: Fedhnr: Federated hierarchical resilient learning for echocardiogram segmentation with annotation noise. Expert Systems with Applications 273, 126841 (2025) [47] Wang, J., Jin, Y., Stoyanov, D., Wang, L.: Feddp: Dual personalization in federated medical image segmentation. IEEE Transactions on Medical Imaging 43(1), 297–308 (2023) [48] Jiang, M., Yang, H., Cheng, C., Dou, Q.: Iop-fl: Inside-outside personalization for federated medical image segmentation. IEEE Transactions on Medical Imaging 42(7), 2106–2117 (2023) [49] Yang, S., Park, H., Byun, J., Kim, C.: Robust federated learning with noisy labels. IEEE Intelligent Systems 37(2), 35–43 (2022) [50] Bai, L., Wang, D., Wang, H., Barnett, M., Cabezas, M., Cai, W., Calamante, F., Kyle, K., Liu, D., Ly, L., et al.: Improving multiple sclerosis lesion segmentation across clinical sites: A federated learning approach with noise-resilient training. Artificial Intelligence in Medicine 152, 102872 (2024) [51] Shin, J., Li, Y., Liu, Y., Lee, S.-J.: Fedbalancer: Data and pace control for efficient federated learning on heterogeneous clients. In: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, pp. 436– 449 (2022) [52] Ma, Y., Hou, J., Zhang, C., Zhou, Y., Ge, Z., Xie, H., Ju, L.: Benchmarking realworld medical image classification with noisy labels: Challenges, practice, and outlook. arXiv preprint arXiv:2512.09315 (2025) [53] Moradi, A., Abrahamsen, B.S., Geerdink, J., Yakar, D., Huisman, H., Bathen, T.F., Elschot, M.: Beyond the sandbox: Real-world federated learning for mri prostate cancer detection (2025) [54] Dalva, Y., Pehlivan, H., Altındiş, S.F., Dundar, A.: Benchmarking the robustness
36
of instance segmentation models. IEEE Transactions on Neural Networks and Learning Systems 35(12), 17021–17035 (2023) [55] Jiang, X., Li, J., Wu, N., Wu, Z., Li, X., Sun, S., Xu, G., Wang, Y., Li, Q., Liu, M.: Fnbench: Benchmarking robust federated learning against noisy labels. arXiv preprint arXiv:2505.06684 (2025) [56] Liang, S., Huang, J., Hong, J., Zeng, D., Zhou, J., Xu, Z.: Fednoisy: Federated noisy label learning benchmark. arXiv preprint arXiv:2306.11650 (2023) [57] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021) [58] Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., Jaeger, P.F.: nnu-net revisited: A call for rigorous validation in 3d medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 488–498 (2024). Springer [59] Warfield, S.K., Zou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (staple): an algorithm for the validation of image segmentation. IEEE transactions on medical imaging 23(7), 903–921 (2004) [60] Almazroa, A., Alodhayb, S., Osman, E., Ramadan, E., Hummadi, M., Dlaim, M., Alkatee, M., Raahemifar, K., Lakshminarayanan, V.: Retinal fundus images for glaucoma analysis: the riga dataset. In: Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications, vol. 10579, pp. 55–62 (2018). SPIE [61] Arvaniti, E., Fricker, K., Moret, M., Rupp, N., Hermanns, T., Fankhauser, C., Wey, N., Wild, P., Rüschoff, J.H., Claassen, M.: Replication Data for: Automated Gleason grading of prostate cancer tissue microarrays via deep learning. Harvard Dataverse (2018). https://doi.org/10.7910/DVN/OCYCMP . https://doi.org/10. 7910/DVN/OCYCMP [62] Mittmann, G., Laiouar-Pedari, S., Mehrtens, H.A., Haggenmüller, S., Bucher, T.-C., Chanda, T., Gaisa, N.T., Wagner, M., Klamminger, G.G., Rau, T.T., et al.: Pathologist-like explainable ai for interpretable gleason grading in prostate cancer. Nature communications 16(1), 8959 (2025) [63] Kades, K., Scherer, J., Zenk, M., Kempf, M., Maier-Hein, K.: Towards real-world federated learning in medical image analysis using kaapana. In: International Workshop on Distributed, Collaborative, and Federated Learning, pp. 130–140 (2022). Springer [64] McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In:
37
Artificial Intelligence and Statistics, pp. 1273–1282 (2017). Pmlr
38