GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees
arXiv:2604.12757v1 [cs.LG] 14 Apr 2026
Arya Shah IIT Gandhinagar [email protected]
Kaveri Visavadiya IIT Gandhinagar [email protected]
Manisha Padala IIT Gandhinagar [email protected]
Abstract Adversarial robustness is essential for deploying neural networks in safety-critical applications, yet standard evaluation methods either require expensive adversarial attacks or report only a single aggregate score that obscures how robustness is distributed across classes. We introduce the GF-Score (GREAT-Fairness Score), a framework that decomposes the certified GREAT Score into per-class robustness profiles and quantifies their disparity through four metrics grounded in welfare economics: the Robustness Disparity Index (RDI), the Normalized Robustness Gini Coefficient (NRGC), Worst-Case Class Robustness (WCR), and a FairnessPenalized GREAT Score (FP-GREAT). The framework further eliminates the original method’s dependence on adversarial attacks through a self-calibration procedure that tunes the temperature parameter using only clean accuracy correlations. Evaluating 22 models from RobustBench across CIFAR-10 and ImageNet, we find that the decomposition is exact, that per-class scores reveal consistent vulnerability patterns (e.g., “cat” is the weakest class in 76% of CIFAR-10 models), and that more robust models tend to exhibit greater class-level disparity. These results establish a practical, attack-free auditing pipeline for diagnosing where certified robustness guarantees fail to protect all classes equally. We release our code on GitHub.
1
Introduction
Deep neural networks are vulnerable to adversarial examples: imperceptible perturbations that cause confident misclassifications [Szegedy et al., 2014, Goodfellow et al., 2015]. This vulnerability poses serious risks in safety-critical settings such as autonomous driving and medical diagnosis, where a single misclassification can have catastrophic consequences. Adversarial training [Madry et al., 2019] remains the dominant defense, and substantial progress has been tracked through standardized benchmarks like RobustBench [Croce et al., 2021], which ranks models by their accuracy under the AutoAttack ensemble [Croce and Hein, 2020]. Yet a fundamental tension exists between robustness and standard accuracy [Tsipras et al., 2019], and recent work on certified defenses [Cohen et al., 2019] has shifted attention toward provable guarantees rather than empirical attack evaluations alone. A critical limitation of current evaluation practice is that robustness is almost always reported as a single aggregate number. Whether the metric is empirical robust accuracy or a certified lower bound, it averages over the entire test distribution and thereby conceals how robustness is distributed across classes. Several studies have shown that adversarial training induces pronounced classwise performance gaps: certain classes become far more vulnerable than others under the same model [Benz et al., 2021, Xu et al., 2021, Tian et al., 2021]. For instance, an autonomous perception
system may appear globally robust while being nearly defenseless on pedestrian classes. Despite this, no existing framework provides certified, attack-free, per-class robustness evaluation. The GREAT Score [Li et al., 2024], introduced at NeurIPS 2024, offers a certified global robustness bound using only generative model samples and forward passes, achieving roughly 2,000× speedup over attack-based methods. However, it reports only a single scalar, inheriting the same class-blindness problem. Concurrently, training-time fairness interventions [Wei et al., 2023, Sun et al., 2022, Li and Liu, 2023, Zhang et al., 2024] address the disparity during model optimization but offer no tools for post-hoc auditing of already-deployed models. In this paper, we introduce the GF-Score (GREAT-Fairness Score), a framework that bridges this gap through three components. First, we decompose the GREAT Score into per-class certified robustness profiles by partitioning samples according to their ground-truth labels and computing class-conditional confidence margins. This decomposition is exact: the weighted sum of per-class scores recovers the aggregate score with zero numerical error. Second, we quantify the disparity of these per-class profiles through four metrics grounded in welfare economics and fairness theory: the Robustness Disparity Index (RDI), the Normalized Robustness Gini Coefficient (NRGC), Worst-Case Class Robustness (WCR), and a Fairness-Penalized GREAT Score (FP-GREAT). Third, we eliminate the original method’s dependence on adversarial attacks for temperature calibration by introducing a self-calibration procedure that maximizes rank correlation with publicly available clean accuracies. We evaluate the GF-Score on 22 robust models from RobustBench spanning CIFAR-10 (17 ℓ2 models) and ImageNet (5 ℓ∞ models). Our experiments yield several notable findings. The class-conditional decomposition is exactly consistent across all 22 models, confirming the mathematical validity of the approach. Per-class analysis reveals that the class “cat” is the most vulnerable in 76% of CIFAR-10 models, while “automobile” is consistently the most robust, suggesting that class vulnerability is an intrinsic data property rather than a training artifact. We observe a positive correlation between aggregate robustness and the Robustness Disparity Index, providing new quantitative evidence for the tension between robustness and fairness identified by prior work [Xu et al., 2021, Benz et al., 2021]. Our attack-free self-calibration achieves a Spearman rank correlation of ρ = 0.871 on CIFAR-10 and ρ = 1.000 on ImageNet with RobustBench rankings, making the entire evaluation pipeline truly attack-free. In summary, we make the following contributions: 1. We propose a class-conditional decomposition of the GREAT Score that preserves the certified lower-bound guarantee at per-class granularity, with formal concentration bounds (Propositions 1 and 2). 2. We introduce four fairness-aware disparity metrics (RDI, NRGC, WCR, FP-GREAT) grounded in welfare economics that quantify how robustness is distributed across classes. 3. We propose an attack-free self-calibration procedure that replaces adversarial-attack-based temperature tuning with clean accuracy correlation, enabling fully attack-free evaluation. 4. We conduct extensive experiments on 22 models across two benchmarks, revealing consistent class vulnerability patterns and a quantifiable robustness-fairness tension that aggregate metrics conceal.
2
Related Work
Our work lies at the intersection of three active research areas: adversarial robustness evaluation, fairness in adversarial training, and inequality measurement in machine learning. We synthesize each area below and position our contribution relative to existing methods in Table 1. 2.1
Adversarial Robustness Evaluation
Since the discovery that neural networks are vulnerable to imperceptible perturbations [Szegedy et al., 2014, Goodfellow et al., 2015], a rich line of work has developed increasingly powerful attacks to evaluate robustness. Gradient-based methods such as PGD [Madry et al., 2019] and the C&W attack [Carlini and Wagner, 2017] became standard tools, though Athalye et al. [2018] showed that many defenses merely obfuscated gradients rather than achieving true robustness. The AutoAttack ensemble [Croce and Hein, 2020] addressed this by combining complementary attack strategies into 2