When AUC Misleads: Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
arXiv:2606.19184v1 [cs.CV] 17 Jun 2026
Dat NGUYEN1 , Cosmin RADOI1 , Romain HERMARY1 , Marcella ASTRID1 , Nesryne MEJRI1 , Enjie GHORBEL1,2 , and Djamila AOUADA1 CVI2 , SnT, University of Luxembourg Cristal Laboratory, National School of Computer Sciences, University of Manouba [email protected] [email protected] 1
2
Abstract. Recent advances in generative AI, such as diffusion models and face-swapping tools, have enabled the creation of highly realistic deepfakes, leading to real-world harms including financial fraud and nonconsensual explicit content. In response, deepfake detection has become an active research area, with recent methods increasingly focusing on improving generalization to unseen manipulations. This is typically evaluated using the Area Under the ROC Curve (AUC) measured separately across multiple datasets. However, such an evaluation fails to reflect realworld scenarios where detectors face a mixture of data sources and varying artifact types. To address this limitation, we introduce a novel metric, Cross-dataset AUC (Cross-AUC) that averages per-domain AUCs with a measure of prediction polarization for taking into account the robustness to domain shift. The polarization extent is quantified by the Wasserstein Distance between class score distributions. Cross-AUC not only assesses the generalization capabilities of deepfake detectors under domain shifts more realistically, but it is also interpretable as it better explains the reason behind a drop in performance. Experiments performed on seven benchmark datasets demonstrate its practical relevance. Keywords: Deepfake Detection · Evaluation Protocol · Domain Shift · Generalization
1
Introduction
Recent advances in generative AI, such as diffusion models and open-source face-swapping tools, have facilitated the creation of realistic forged facial images. Such manipulated data, also known as deepfakes, has already caused real harm, ranging from financial scams [3] to the diffusion of non-consensual content [30]. In response to this threat, the field of deepfake detection has become a very active research topic [1, 2, 4, 7, 12, 13, 23, 25–28, 31, 32, 42]. Earlier methods mostly employ binary classifiers using standard deep architectures (e.g., XceptionNet [6], EfficientNet [33]) that learn to distinguish between real and fake data. Nevertheless, these approaches tend to overfit specific artifact patterns introduced by the generation techniques used to create the fake training samples;
2
D. NGUYEN et al. Negative
Positive
Optimal Threshold (A)
Optimal Threshold (B)
Dataset A (AUC = 1)
Dataset B (AUC = 1)
Dataset A & B (AUC < 1) 0.0
0.2
0.4
0.6
0.8
1.0
Predicted Scores
Fig. 1: Example of optimal thresholds for predictions on different datasets, A (top) and B (middle). While a perfect AUC = 1 is reached on both datasets separately, an obvious drop in AUC will appear when combining them to simulate real-world settings (bottom), since the scores are no longer well ordered. The different nature of score distributions leads to a clear disparity between optimal thresholds (purple range), demonstrating the importance of taking into account the polarization of probability predictions in real-world scenarios.
thereby, achieving poor generalization capabilities when considering unseen generation techniques. Solving this issue is crucial because new generative methods are constantly emerging, introducing distinct and inherently different artifact traces. To overcome this problem, recent state-of-the-art methods have adopted multi-task learning [2,4,12,28,41] and/or data synthesis strategies [4,5,23,32,38], claiming increased robustness to unseen manipulations. To demonstrate the generalization capabilities of these methods, a cross-dataset evaluation protocol is typically followed, where different datasets are employed during training and testing phases [4, 23, 28, 32]. Given its robustness to imbalanced data, the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC), which estimates the class separability of a binary classification model, is commonly reported as a primary evaluation metric [1,2,4,7,12,13,23,27,28,32]. It is computed separately on each dataset and then averaged to assess the overall performance of deepfake detectors. Such an evaluation protocol, however, cannot fully reflect the relevance of a given deepfake detector in a real-world setting. Indeed, it assumes homogeneous score distributions across different data sources, ignoring the existence of domain shifts. In practice, an effective detector should be able to deal with uncontrolled deepfake data featuring different kinds of artifacts at the same time, regardless of their origin. As such, the encountered data in realistic scenarios likely resemble a mixture of multiple heterogeneous datasets. Therefore, evaluating AUC separately on each dataset, as commonly done, fails to capture this realistic setting and tends to be overoptimistic when simply averaged. As illustrated in Figure 1, the AUC metric can be high when evaluated on each dataset individually, while dropping drastically when considering two datasets incorporating a domain shift. Therefore, we propose a new metric, called Cross-dataset AUC (Cross-AUC), that can better assess the generalization capabilities of deep detectors under domain shift. It relies on the mean AUC calculated on diverse datasets while taking into account the polarization extent of the probability predictions. A stronger polarization of the predicted probabilities (i.e., values closer to the boundaries
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
3
0 and 1) indicates a clearer separation between classes in the score space. In such cases, a fixed threshold is more likely to remain effective under varied conditions, thereby contributing to enhanced generalization capabilities. The polarization magnitude is estimated using the Wasserstein Distance (WD) between the distributions of positive and negative probability predictions. Consequently, the proposed Cross-AUC captures both the relative ranking performance (via the mean AUC) and the separability extent of predictions (via the polarization), offering a more adequate estimate of generalization performance in real-world scenarios. Finally, we empirically show that the Cross-AUC is very close the AUC measured on a combination of all the considered datasets, making it a practical measure for evaluating cross-dataset generalization. While estimating the AUC on a combined dataset better reflects real-world settings with the available data, it lacks the interpretability provided by the introduced polarization. In particular, it remains difficult to explain the reason behind a drop in performance of this measure compared to the average AUC computed on isolated datasets. Therefore, Cross-AUC offers a better interpretation and understanding of the discrepancy between these two measures, demonstrating the important role of prediction polarization in the generalization capabilities of deepfake detectors. In other words, our work suggests that the separability extent between real and fake predictions across domains is crucial. This novel metric not only motivates the introduction of a new evaluation protocol in deepfake detection, but it also encourages researchers to develop robust deepfake detectors that consider prediction polarization as an additional learning criterion, besides traditional classification losses. In summary, this work presents the following contributions: – We discuss and analyze the sensitivity of AUC under domain shift and its unsuitability for assessing the generalization capabilities of deepfake detectors. – We rigorously define the notions of polar sets and polarity in the score space that are used to quantify the extent of polarization. – We introduce a new metric called Cross-dataset AUC (Cross-AUC) to evaluate the generalization performance of deepfake detectors, taking into account the mean AUC across different datasets, as well as the polarization of probability predictions. – We investigate the impact of prediction polarization between classes and show that similar AUC values can correspond to vastly different decision behaviors. This is quantified using the WD between score distributions. – We conduct an experimental evaluation and analysis based on the proposed Cross-AUC for comparing several deepfake detectors, including recent ones using seven well-known benchmarks [10, 11, 15, 24, 31, 37, 43].
2
Problem Formulation: AUC under Domain Shift
Let us denote by X ∈ Rc×h×w an input image and y ∈ {0, 1} its associated label, which follow a joint probability distribution p(X, y). We assume that y = 0
4
D. NGUYEN et al.
and y = 1 if X belongs to the negative and positive classes (i.e., real and fake), respectively. As such, we denote by pp (X) = p(X | y = 1) and pn (X) = p(X | y = 0) the conditional probability density of the positive and negative classes, respectively. Furthermore, let s : Rc×h×w 7→ [0, 1] be a scoring function (often computed using a deep neural network in the context of deepfake detection) that estimates the positive posterior probability of an input sample p(y = 1 | X). Given a fixed threshold τ ∈ R, the classifier that gives the predicted label ŷ is then defined as follows: ŷ = I(p(y = 1 | X) > τ ), where I is the indicator function. The Area Under the ROC Curve (AUC) is a standard metric often used in deepfake detection and more generally in binary classification. Specifically, it reflects how well a model is able to separate positive from negative samples in the score space and is independent of the classification threshold τ . It measures the probability that a randomly chosen positive instance (e.g., a fake video) receives a higher confidence score than a randomly chosen negative one (e.g., a real video) [18, 20, 22, 36]. Formally, the AUC can be defined as, \label {equa:auc_overall} \text {AUC}(s) = \mathbb {E}_{\mathbf {X}^p \sim p^p(\mathbf {X}),\, \mathbf {X}^n \sim p^n(\mathbf {X})} \left [ \mathbb {I}(s(\mathbf {X}^p) > s(\mathbf {X}^n)) \right ] \text {,}
(1)
This formulation emphasizes that AUC reflects the ability of a model to rank positive instances higher than negative ones, depending only on the relative ordering of the scores. Although AUC is effective from a dataset to another when the class-conditional distributions are stable, it becomes problematic in the presence of a domain shift. Let i and j denote two different datasets with their distribution of positive and negative prediction samples denoted as ppi (X), ppj (X) and pni (X), pnj (X), respectively. In the presence of a complete domain overlap between i and j, the classconditional distributions are approximately aligned: p_i^p(\mathbf X ) \approx p_j^p(\mathbf X ), \quad p_i^n(\mathbf X ) \approx p_j^n(\mathbf X ) \text {.}
(2)
Hence, the model produces comparable confidence scores across domains. The indicator function in the AUC formula, \mathbb {I}(s(\mathbf {x}^P) > s(\mathbf {x}^N)) , remains meaningful, and global AUC reliably reflects the model’s performance. Nevertheless, in the presence of a domain gap in the prediction probability space, the two distributions differ significantly, p_i^p(\mathbf X ) \not \approx p_j^p(\mathbf X ), \quad p_i^n(\mathbf X )\not \approx p_j^n(\mathbf X ) \text {.}
(3)
In this context, the positive and negative posterior probabilities pp (X) and p (X) can be expressed as a mixture of dataset-specific posterior probabilities as follows, n
p^p(\mathbf X) = \sum _i \alpha _i \, p^p_i(\mathbf X ) \not \approx p^p_i(\mathbf X ), \text { } \forall i \text {,} \quad p^n(\mathbf X) = \sum _i \beta _i \, p^n_i(\mathbf X ) \not \approx p^n_i(\mathbf X ), \text { } \forall i \text {,} (4) where αi and βi represent the mixture weights. This is typically the case when considering different deepfake detection datasets such as FF++ [31], CDF [24],
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
5
1.0
90
ρ = 0.96
0.8
FF++ CDF2 DFD DFW DFDCP DFDC DF40 Combined
0.6
0.4
0.2
Method
AUC (Combined)
Optimal Thresholds
Dataset
80
CADDM ForensicAdapter LAA-Net RECCE SBI SLADD Xception
70
60
50 ter
t Ne
ap Ad
ALA
Fo re
nsic
I
M DD CA
E
SB
D
CC RE
AD
ept Xc
SL
ion
0.0
0.1
0.2 0.3 Ed∈D Ps(d)
0.4
Fig. 2: Optimal thresholds for the stud- Fig. 3: Correlation between estimated poied SOTA methods on the seven different larity (averaged across datasets) and the datasets [10,11,15,24,31,37,43] (and all of target combined AUC. Pearson coefficient ρ = 0.96. them combined).
DFDC [10], and DFW [43]. This divergence can result from differences in deepfake generation methods, compression levels, facial attributes, resolution, or lighting conditions across datasets. This can lead to inconsistencies in how the model scores samples across domains, as we can deduce from Eq. (1). For example, the model might assign higher confidence scores to fake videos from one dataset, but not to the other dataset. These inconsistencies can distort global rankings and degrade AUC, even when performance within each domain is strong. This makes AUC an unreliable performance measure if considered on different domains separately, as experimentally demonstrated in Section 4.2.
3
Cross-Dataset AUC
We propose a revised evaluation protocol that more faithfully reflects the deployment challenges of deepfake detection. As discussed in Section 2, the conventional evaluation metric, AUC, is computed independently on individual datasets and provides an incomplete or overly optimistic view of model robustness. This practice overlooks two critical issues: (i) the instability of decision thresholds, and (ii) the implicit assumption of the non-existence of distribution shifts, across datasets. In this section, we propose to define the notion of polarization in the score space, which is used as a new measure for assessing threshold instability. The proposed polarization measure is then integrated into the newly introduced metric termed Cross-AUC, which is proposed as a replacement for the standard mean AUC under domain shift. 3.1
Threshold Instability and Polarization
We first expose score instabilities in deepfake detection models. We verify it empirically by locating the separation between the negative and positive probability predictions for a given method over multiple datasets. This separation is known as a decision threshold and is necessary for any deployment scenario. Its value can depend on the application (e.g., maximizing detection rate or reducing
6
D. NGUYEN et al. Detector X: Perfect AUC=1on-and but AUC <<1onA&B Dataset
_,
Detector Y: Perfect AUC =1onand_, and AUC=10nA&B
|
|
Dataset
|
Dataset
|
|
A&B
Real sample of
Real sample of
Fake sample of
Fake sample of
Polarization over
Polarization over
| Optimal threshold
Fig. 4: Example of the obtained optimal thresholds on two datasets, A (top) and B (middle). Although both detectors achieve perfect AUC = 1 when evaluated on each dataset separately, mixing A&B (bottom) can cause a sharp AUC drop (Detector X), while a more stable detector preserves ordering and maintains AUC = 1 (Detector Y). It can be observed that a larger gap exists between the two classes in the score space, reducing the impact of a varying threshold. This motivates adopting a polarizationaware evaluation.
false alarm rate), but in a generic study, an ideal threshold should be the best trade-off between correctly classified positives and negatives. Given a scoring function s and a dataset d, we denote the set of obtained scores on d using the function s as Os,d = {s(X), ∀X ∈ d}. We choose the corresponding optimal threshold τs,d that gives the closest score to perfect classification (i.e, FPR = 0, TPR = 1). From the set of all possible thresholds T , it is obtained by minimizing the distance between the FPR and TPR values from the ROC curve and the FPR and TPR obtained under perfect classification. \tau _{s,d} = \argmin _{t \in \mathcal {T}}\left (\sqrt {\left [1 - \text {TPR}_t(\mathcal {O}_{s,d})\right ]^2 + \text {FPR}_t(\mathcal {O}_{s,d})^2}\right ).
(5)
In Figure 2, we show the optimal thresholds found for the considered SOTA methods across several datasets. Our analysis reveals noticeable threshold variations, with standard deviations reaching 0.16 for Xception [31] and even 0.31 for LAA-Net [28], demonstrating the impact of shifts between testing data sources. This large discrepancy between optimal thresholds will cause lower performance than expected after deployment. Although the instability of the optimal threshold across datasets reflects the instability of a deepfake detector model, its stability does not guarantee strong generalization capabilities. In particular, if the predicted probabilities vary only within a restricted range, the variance of optimal thresholds might be small. Yet, the corresponding model can still be unstable across datasets with disruptive domain shifts in the score space. On the other hand, a model that is robust to high variance thresholds is more likely to be stable across domain shifts. This intuition is illustrated in Figure 4. Therefore, we propose measuring this robustness criterion via polarization. Herein, we start by introducing Polar Sets and Polarity as follows. Definition (Polar Sets): Let s ∈ S be a scoring function (e.g., a model) and d ∈ D a dataset. The polar sets associated to s and d are defined as Rs,d =
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
7
{s(X) | y = 0, ∀X ∈ d} and Fs,d = {s(X) | y = 1, ∀X ∈ d}. Thus, the sets Rs,d and Fs,d contain the probability predictions given by s of all real data and fake data, respectively. Definition (Polarity): The polarity Ps (d) of a scoring function s ∈ S on a dataset d ∈ D is a measure of divergence (or distance) between the estimated probability distributions Rs,d = Θ(Rs,d ) and Fs,d = Θ(Fs,d ), with Θ a given density estimation function. We refer to the measuring function as Π, such that Ps (d) = Π(Rs,d , Fs,d ). An ideal model s∗ would converge towards Rs,d = δ0 and Fs,d = δ1 , which represent Dirac impulses at both extremities of the scoring space [0, 1]. By assuming that the distance between the distribution is calculated using the Wasserstein distance (WD), which we will use throughout the whole study, we can intuitively find that s∗ has a polarity Ps∗,d of WD(δ0 , δ1 ) = |0 − 1| = 1, for any d ∈ D. On the other hand, the worst performing model s− would end up with Rs− ,d ≈ Fs− ,d and a polarity Ps− ,d ≈ 0. As we focus on the robustness of models approaching perfect convergence, we can safely assume both densities Rs,d and Fs,d to be close to the objective densities (Rs,d ≈ δα , Fs,d ≈ δ1−β , α and β representing small shifts). Upon this condition, we can derive the definition space of τs,d for each dataset d ∈ D, denoted as [αs,d , 1−βs,d ] and deduce a tolerance of 1−max(βs,d )−max(αs,d ) for the d∈D
d∈D
cross-dataset optimal threshold τs of the model s. This suggests that the lower α and β are across numerous datasets, the greater the tolerance over different thresholds will be. This directly translates to having a maximum polarity across datasets. The hypothesis is empirically verified and illustrated in Figure 3, where we visualize the correlation between individual dataset polarities and the AUC on a Combined dataset obtained by aggregating all the datasets. This allows simulating real-world settings where data are unconstrained, thereby assessing more faithfully the deployment readiness. Specifically, the AUC results on the Combined dataset are reported according to the empirical average of polarities, Ed∈D Ps (d) . In summary, a high correlation between the empirical average of polarities, Ed∈D Ps (d) and the AUC can be observed. 3.2
Leveraging Polarization and AUC for Cross-Dataset Evaluation
To better evaluate the reliability of a model and take into account conflicting optimal thresholds, we propose to use the defined polarity as a reward parameter. We further establish that a stable AUC and polarities throughout the datasets will ensure better reliability. To this end, we define a penalty-reward criterion C, such that we have i) a penalty when AUCs are unstable and ii) a reward for high polarities. For a model s ∈ S, we will thus define the criterion over its corresponding sets of AUC scores A = {AUC(s, d), ∀d ∈ D} and polarities P = {Ps (d), ∀d ∈ D}. To penalize any instability of AUCs and polarities, we quantify the variability of the given sets with a measure of central dispersion (e.g., standard deviation), which we denote as Φ : Rk+ 7→ R+ (where k = card(D) is the number of datasets).
8
D. NGUYEN et al.
Taking the opposite of this quantity, −Φ(·), yields the desired penalties. To reward the score polarization, we rely on the estimation of the global model behavior with a measure of central tendency (e.g., harmonic mean), which we denote as Ψ : Rk+ 7→ R+ . We define the criterion C by compiling both sub-criteria as follows: {C}(\mathcal A, \mathcal P) = - \Phi (\mathcal A) + \left | {\Psi }(\mathcal P) - {\Phi }(\mathcal P) \right |. (6) We apply the same estimator Ψ over the AUCs to obtain the performance tendency of s, and combine it with the previously defined criterion to form the proposed Cross-AUC: \label {eq:cross_auc} \text {Cross-AUC} ={\Psi }(\mathcal A) + \lambda \cdot {C}(\mathcal A,\mathcal P),
(7)
with λ a balancing parameter. We provide a comprehensive analysis of the selection of Cross-AUC components in the next section.
4
Experiments
In this section, we experimentally demonstrate that the proposed Cross-AUC is a reliable estimator of model stability and provides values that are very close to the AUC on the Combined dataset. We first present the experimental settings in Section 4.1 and discuss the obtained results in Section 4.2. 4.1
Experimental Settings
Deepfake Detectors. To conduct our evaluation study, seven SOTA methods [2, 4, 7, 12, 28, 31, 32] are selected. The selection is based on the following criteria: (1) the official codes of the considered methods are available and open-source for reliable reproducibility; (2) they are diverse in the way they approach deepfake detection: we include in our study an end-to-end binary classifier (Xception [31] ), a disentanglement learning approach (RECCE [2]), a method relying on data synthesis (SBI [32]), two hybrid methods combining multi-task learning and data synthesis strategies (SLADD [4], CADDM [12]), a fine-grained approach (LAA-Net [28]), and a CLIP-based [29] adapter method (ForensicAdapter [7]); (3) the considered baselines are recent and have been accepted in top-tier conferences (at the exception of [31] that was published in 2019). The summary of the compared deepfake detectors is presented in Table 1. More details are provided in the supplementary materials. Table 1: Summary of compared deepfake detectors. Detector
Backbone
Family
Respository
Xception [31] Xception [6] Binary Classifier https://github.com/ondyari/FaceForensics SLADD [4] Modified Xception [6] Multitask + Data synthesis https://github.com/liangchen527/SLADD RECCE [2] Designed Networks Disentanglement Learning https://github.com/VISION-SJTU/RECCE SBI [32] EfficientNet [33] Data synthesis https://github.com/mapooon/SelfBlendedImages CADDM [12] EfficientNet [33] + Multiscale Multitask + Data synthesis https://github.com/megvii-research/CADDM LAA-Net [28] EfficientNet [33] + E-FPN Multitask + Data synthesis + Fine-grained https://github.com/10Ring/LAA-Net ForensicAdapter [7] CLIP [29] + ViT [14] Multitask + Data synthesis + Adapting CLIP https://github.com/OUC-VAS/ForensicsAdapter
Weights Release
Venue
✓ × ✓ ✓ ✓ ✓ ✓
ICCV’19 CVPR’22 CVPR’22 CVPR’22 CVPR’23 CVPR’24 CVPR’25
Datasets. Seven standard and challenging datasets are used in our experiments, including FaceForensics++ (FF++) [31], Celeb-DF (CDF) [24], Google
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
9
Deepfake Detection (DFD) [15], WildDeepfake (DFW) [43], Deepfake Detection Challenge (DFDC) [10], Deepfake Detection Challenge Preview (DFDCP) [11], and DF40 [37]. FF++ is typically used for training, while the others serve as test sets in the cross-dataset evaluation protocol. These datasets are selected based on the following criteria: 1) they are widely used for evaluation in the research community; 2) they contain fakes from diverse sources, including those generated by black-box or undisclosed methods, for which no implementation details or prior knowledge are available; 3) they present increased complexity as a result of various image perturbations, such as compression, noise, or blur. An overview of the selected benchmarks and further details is provided in the supplementary materials. Implementation details. We extract 32 frames per video following the conventional test split [31]. Facial regions are cropped using Face-RetinaNet [9]. The bounding boxes are slightly enlarged by a factor of 1.25 around the center of the face and then resized to a fixed resolution of 256 × 256. For all experiments, we use the official pre-trained models of the selected methods. Since SLADD does not provide pre-trained weights, we retrain it on FF++ following its opensource implementation. Unless specified differently, we use the harmonic mean for Ψ , the standard deviation for Φ, the Quantile Cumulative Density Function (CDF−1 ) for the density estimation function D, and λ = 0.5 to provide the best results since they yield the best performance as demonstrated empirically in Section 4.2. Evaluation Protocol. We consider two evaluation settings. The first is the conventional protocol, in which each model is evaluated separately on each dataset. The resulting independent scores are used to compare the AUC with the proposed Cross-AUC metric. To better reflect the heterogeneity and multi-source nature of real-world data, the second protocol is based on the evaluation of each model on a unified dataset, referred to as the “Combined ” dataset, obtained by aggregating all seven datasets. This enables the exposure of deepfake detection methods to domain gaps that may remain hidden in single-dataset evaluation settings. Evaluation Metrics. The polarization is computed using the Wasserstein Distance (WD) for Π, measuring the distance between the estimated densities Rs,d and Fs,d (real and fake score distributions). To approximate these densities, we experimented with different density estimation functions Θ based on nonparametric and parametric approaches. For the former, we employ Kernel Density Estimation (KDE) and Quantile Cumulative Density Function (Q); for the latter, we consider two types of distributions, namely, Gaussian Mixture Model (GMM) and Beta Distribution (BD). To estimate the parameters of GMM and BD, Expectation-Maximization (EM) and numerical Maximum Likelihood Estimation (MLE) are used, respectively. By default, Q is chosen for its effectiveness. We mainly focus on comparing the averaged AUC (AUCa ), the AUC on the Combined dataset (AUCc ), and our Cross-AUC. Comparison with additional evaluation metrics is provided in the supplementary materials for a more in-depth
10
D. NGUYEN et al.
analysis, including Accuracy, Balanced Accuracy, Precision, Recall, Specificity, F1-Score, and Equal Error Rate. 4.2
Results and Discussion
In this section, we present and discuss the obtained results. Additional results are provided in the supplementary materials, such as the runtime analysis, the effect of the number of domains, and the performance obtained using additional metrics. Overall Performance Analysis. Table 2 summarizes per-dataset AUC, average AUC, AUC on a combination of seven considered datasets [10, 11, 15, 24, 31, 37,43], optimal thresholds (τ ), and their stability (ϕτ ), as well as score polarization (polarity) measured via WD using different density estimators. As shown, ForensicAdapter [7] and LAA-Net [28] emerge as the strongest performers under domain shift. ForensicAdapter attains the highest mean AUC (90.18%) while maintaining a high combined AUC (85.96%), and it consistently yields strong polarity, indicating well-separated score distributions that transfer across datasets. LAA-Net achieves the best combined AUC (86.51%) and leads in terms of polarity on most datasets, suggesting a strong discriminative capacity. However, its higher threshold variance ϕτ suggests a less stable decision boundary in crossdomain settings. RECCE [2] achieves consistent performance across domains with stable AUC (81.13%) and Combined AUC (76.91%), and low threshold variance. Nonetheless, its class separability capacity is slightly weaker as compared to LAA-Net and ForensicAdapter. SBI [32] and CADDM [12] perform moderately well in terms of both average and Combined AUC, but their optimal threshold vary importantly. Xception [6] and SLADD [4] exhibit lower AUCs and weak generalization, confirming their limitations under complex cross-domain settings. A more detailed evaluation including additional metrics is reported in the supplementary materials. ForensicAdapter
LAA-Net
CADDM
1.0
Dataset
Density
0.8
0.6
0.4
FF - Real
DFDCP - Fake
FF - Fake
DFDC - Real
CDF - Real
DFDC - Fake
CDF - Fake
DFW - Real
DFD - Real
DFW - Fake
DFD - Fake
DF40 - Real
DFDCP - Real
DF40 - Fake
0.2
0.0
SBI
RECCE
SLADD
Xception
1.0
Density
0.8
0.6
0.4
0.2
0.0 0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Fig. 5: Class-conditional score distributions of the evaluated deepfake detectors across seven datasets [10, 11, 15, 24, 31, 37, 43].
Cross-AUC versus Average AUC. Table 3 compares the conventional average AUC across individual datasets (\protect \text {AUC}_a ) with the AUC measured on the combined test set (\protect \text {AUC}_c ). Across all models, \protect \text {AUC}_a is consistently higher than \protect \text {AUC}_c , with differences ∆a ranging from 1.77pp (LAA-Net) to over 12.30pp (SLADD). This consistent drop highlights that average AUC often overstates
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
11
Table 2: Performance of the selected models on all datasets. τ : Optimal threshold; ϕτ : variance of the top–K optimal thresholds; WKDE , WQ , WGMM , WBD : model polarities using WD as Π over different density estimation (Θ) methods, including nonparametric (Kernel Density Estimation (KDE), Quantile (Q)), and parametric ones assuming two types of distributions, i.e., GMM and BD. We set Φ as the standard deviation, Ψ as the harmonic mean and λ = 0.5. Gray represents the datasets the model was trained on. Green highlights the best AUC and polarities. FF++ CDF DFD DFW DFDCP DFDC DF40 Average
Combined
Xception
AUC (%) 93.6 62.1 92.24 63.71 τ 0.4 0.24 0.7 0.57 ϕτ 0.0004 0.0004 0.0002 0.0007 WKDE 0.524 0.064 0.430 0.110 WBD 0.549 0.068 0.446 0.122 WQ 0.661 0.077 0.524 0.139 WGMM 0.616 0.069 0.496 0.149
70.56 58.98 81.59 74.68 68.95 (↓ 5.73) 0.4 0.3 0.24 0.4 0.31 0.0010 0.0001 0.0001 0.0004 0.0001 0.146 0.078 0.239 0.227 0.168 0.164 0.060 0.232 0.234 0.152 0.183 0.091 0.264 0.277 0.190 0.118 0.081 0.249 0.254 0.180
SLADD
AUC (%) 88.31 56.65 97.22 48.63 τ 0.51 0.53 0.57 0.55 ϕτ 0.0001 0.0001 0.0001 0.0001 WKDE 0.060 0.001 0.098 0.004 WBD 0.078 0.003 0.079 0.001 0.079 0.005 0.090 0.002 WQ WGMM 0.079 0.005 0.089 0.004
65.98 57.37 72.25 69.48 57.18 (↓ 12.3) 0.53 0.53 0.45 0.52 0.52 0.0001 0.0001 0.0001 0.0001 0.0001 0.022 0.012 0.003 0.028 0.013 0.007 0.008 0.024 0.029 0.013 0.015 0.010 0.026 0.032 0.014 0.015 0.010 0.027 0.033 0.014
RECCE
AUC (%) 99.56 69.02 97.82 65.19 τ 0.98 0.74 0.99 0.65 0.00001 0.0002 0.0001 0.0001 ϕτ WKDE 0.631 0.126 0.352 0.096 WBD 0.569 0.124 0.280 0.116 WQ 0.716 0.136 0.355 0.115 WGMM 0.692 0.134 0.349 0.111
82.24 71.19 82.94 81.13 76.91 (↓ 4.22) 0.53 0.58 0.97 0.77 0.86 0.0001 0.0001 0.0001 0.0001 0.0001 0.212 0.181 0.153 0.250 0.221 0.240 0.182 0.117 0.233 0.201 0.243 0.203 0.144 0.273 0.232 0.227 0.203 0.142 0.265 0.228
SBI
AUC (%) 98.23 84.65 96.78 60.83 τ 0.21 0.28 0.46 0.16 ϕτ 0.0006 0.0001 0.0008 0.0001 WKDW 0.503 0.215 0.516 0.078 WBD 0.513 0.219 0.500 0.089 0.576 0.238 0.577 0.085 WW WGMM 0.567 0.236 0.564 0.084
86.83 69.77 72.05 81.3 74.43 (↓ 6.87) 0.19 0.14 0.17 0.23 0.17 0.0002 0.0001 0.0001 0.0002 0.0001 0.220 0.097 0.103 0.248 0.152 0.224 0.116 0.112 0.253 0.182 0.244 0.108 0.116 0.278 0.172 0.245 0.105 0.115 0.274 0.171
CADDM
AUC (%) 99.26 80.7 99.52 76.31 τ 0.97 0.45 0.93 0.37 ϕτ 0.0253 0.0001 0.0001 0.0001 WKDE 0.661 0.203 0.436 0.210 WBD 0.607 0.203 0.386 0.238 WQ 0.726 0.226 0.466 0.246 WGMM 0.714 0.224 0.459 0.241
71 70.33 76.28 81.91 71.53 (↓ 10.38) 0.44 0.37 0.22 0.53 0.31 0.0011 0.0001 0.0001 0.0038 0.0001 0.156 0.155 0.137 0.280 0.176 0.182 0.165 0.146 0.275 0.214 0.179 0.174 0.151 0.310 0.198 0.174 0.173 0.150 0.305 0.197
LAA-Net
AUC (%) 99.45 95.45 98.47 79.52 τ 0.23 0.3 0.96 0.06 ϕτ 0.0161 0.0004 0.0001 0.0001 WKDE 0.831 0.511 0.633 0.262 WBD 0.749 0.495 0.547 0.248 WQ 0.927 0.582 0.708 0.285 WGMM 0.911 0.579 0.690 0.292
86.48 72.6 86.04 88.28 86.51 (↓ 1.77) 0.08 0.03 0.11 0.25 0.083 0.0005 0.0001 0.0001 0.0024 0.0001 0.334 0.169 0.281 0.431 0.302 0.357 0.189 0.272 0.408 0.296 0.446 0.156 0.299 0.486 0.318 0.417 0.154 0.286 0.475 0.317
AUC (%) 98.39 94.94 99.8 80.95 τ 0.76 0.64 0.65 0.5 ϕτ 0.0013 0.0003 0.0027 0.0001 ForensicAdapter WKDE 0.497 0.338 0.659 0.251 WBD 0.460 0.334 0.624 0.253 WQ 0.540 0.374 0.713 0.285 WGMM 0.521 0.371 0.709 0.284
87.04 0.43 0.0007 0.259 0.268 0.293 0.288
83.88 86.3 90.18 85.96 (↓ 4.22) 0.4 0.38 0.53 0.43 0.0001 0.0001 0.0007 0.0001 0.257 0.291 0.365 0.304 0.266 0.294 0.357 0.306 0.286 0.322 0.402 0.336 0.285 0.320 0.397 0.335
12
D. NGUYEN et al.
generalization, as it reflects performance under domain-specific conditions. However, models in deployment encounter inputs from mixed or unknown distributions, more closely represented by the combined test set. The lower \protect \text {AUC}_c values expose how model predictions degrade under such conditions, revealing weaknesses that per-domain averages mask. These results indicate that relying solely on average AUC can be misleading, highlighting the need for more realistic evaluation of generalization. Table 3: Comparison of AUCc (combined AUC) with AUCa (average per-dataset AUC), H-score [16], and our Cross-AUC. ∆ values are absolute deviations from AUCc with respect to other metrics (lower is better). Green highlights the closest performance to AUCc . AUCc AUCa Cross-AUC H-score ∆a ∆Cross-AUC ∆H-score Xception 68.95 SLADD 57.18 76.91 RECCE SBI 74.43 CADDM 71.53 LAA-Net 86.51 ForensicsAdapter 85.96
74.68 69.48 81.13 81.3 81.91 88.28 90.18
68.52 58.37 73.26 74.20 75.31 86.47 83.59
72.38 5.73 65.88 12.30 79.22 4.22 79.10 6.87 80.43 10.38 87.25 1.77 89.66 4.22
0.42 1.19 3.64 0.22 3.78 0.04 2.36
2.42 8.70 2.31 4.67 8.90 0.74 3.70
We also analyze the gap between the predicted Cross-AUC and the combined AUC (\protect \text {AUC}_c ), as shown in Table 3. Unlike the conventional average AUC (\protect \text {AUC}_a ), which consistently overestimates performance under distribution shift, Cross-AUC is explicitly designed to estimate generalization using per-dataset AUCs and score-level polarization reported in Table 2. Across all models, CrossAUC aligns more closely with \protect \text {AUC}_c than \protect \text {AUC}_a , achieving the smallest difference (\Delta _\text {Cross-AUC} ) in all cases. In summary, Cross-AUC and average AUC imply a close yet different performance ranking of deepfake detection methods. For instance, Table 2 shows a rank reversal between ForensicsAdapter and LAA-Net. Although ForensicsAdapter attains the highest average AUC. LAA-Net achieves a higher combined AUC and Cross-AUC. Such cases highlight that the average AUC is not sufficient for comparing the performance of deepfake detection methods in terms of generalization. An alternative radar-based view of the comparison is provided in the supplementary materials. Polarization Robustness under Domain Shift. We compare the polarization scores of each model by looking at their average values across individual datasets and their value on the combined dataset, as shown in Table 2. In general, most models show a lower polarization score on the combined set compared to the average across separate datasets. Specifically, LAA-Net has the highest average polarization (e.g., average WQ : 0.486) and performs well on several datasets such as CDF, DFW, DFDCP. However, it also shows the largest drop in the combined setting, with its score decreasing to 0.318. On the other hand, ForensicAdapter shows a smaller decrease (WQ : 0.402 to 0.336), indicating more stable score alignment across domains. SBI, CADDM, and Xception achieve strong polarity scores on specific datasets (e.g., FF++, DFD), but their combined scores are notably lower. Finally, SLADD yields low polarity in both
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
13
settings (WQ : 0.032 to 0.014), indicating poor class separability regardless of the domain. These observations are further supported by Figure 5. Overall, most models show a drop in polarization when moving from individual datasets to the combined set. This gap confirms that the performance of current detectors is affected by domain shifts, highlighting the importance of adopting a polarization-aware evaluation for assessing the generalization capabilities of deepfake detectors. Cross-AUC versus Generalization Metrics. To further demonstrate the effectiveness of Cross-AUC, we additionally compare it with the H-score [16] metric in Table 3. H-score was originally designed for universal domain adaptation and represents the harmonic mean of accuracy on common and unknown classes. To align with our setting, we replace accuracy with per-dataset AUC. The obtained results demonstrate that Cross-AUC is more clearly correlated with the combined AUC. Effect of Ψ . Table 4 compares Table 4: Estimated Cross-AUC with different three averaging functions Ψ , in- Ψ functions, i.e., harmonic, arithmetic, and gecluding harmonic, arithmetic, and ometric means. The difference (∆) between the geometric, used compute Cross- AUCc and the resulting Cross-AUC is reported AUC in Eq. (7). We evaluate their for each function. Cross-AUC effectiveness by measuring the ab- Model AUCc ΨHarmonic ∆ ΨArithmetic ∆ ΨGeometric ∆ solute difference from the comXception 68.95 68.52 0.42 69.93 0.98 67.29 1.66 bined AUC (\protect \text {AUC}_c ). We evaluate SLADD 57.18 58.38 1.19 60.80 3.61 59.72 2.53 76.91 73.27 3.64 77.40 0.49 73.98 2.92 their effectiveness by measuring RECCE SBI 74.43 74.21 0.22 77.45 3.02 73.18 1.24 71.53 75.32 3.78 80.61 9.07 77.54 6.00 the absolute difference between CADDM 86.51 86.47 0.04 82.52 3.98 85.37 1.13 Cross-AUC and combined AUC LAA-Net ForensicsAdapter 85.96 83.60 2.36 82.02 3.94 82.94 3.02 (\protect \text {AUC}_c ). The results show that the harmonic mean consistently produces the closest results, achieving the lowest difference (\Delta ) in 6 out of 7 models. For that reason, we adopt the harmonic mean in the rest of our experiments. Table 5: Effects of the balancing factor λ. AUCc Xception 68.95 SLADD 57.18 76.91 RECCE SBI 74.43 CADDM 71.53 LAA-Net 86.51 ForensicsAdapter 85.96
λ=0.1
0.2
0.3
0.4
Cross-AUC 0.5 0.6
0.7
0.8
0.9
1.0
71.61 70.83 70.06 69.29 68.52 67.75 66.98 66.20 65.43 64.66 64.37 62.87 61.37 59.87 58.37 56.87 55.37 53.87 52.37 50.86 78.03 76.83 75.64 74.45 73.26 72.07 70.88 69.69 68.50 67.30 78.12 77.14 76.16 75.18 74.20 73.22 72.24 71.26 70.28 69.30 79.40 78.38 77.36 76.34 75.31 74.29 73.27 72.25 71.22 70.20 87.09 86.94 86.78 86.62 86.47 86.30 86.15 85.99 85.83 85.67 88.45 87.23 86.02 84.81 83.59 82.38 81.16 79.95 78.74 77.52
Effect of the Balancing Factor λ. We also analyze the sensitivity of CrossAUC to the balancing factor λ defined in Eq. (7). As shown in Table 5, setting λ to very small values deviates Cross-AUC from the combined AUC, making it closer to a simple average AUC. On the other hand, the use of excessively large values for λ overweights polarization. A mid-range choice (λ ≈ 0.5) provides Cross-AUC values that closer to AUCc .
14 D. NGUYEN et al. Table 6: Estimated Cross-AUC with different polarization estimators, i.e., Wasserstein Distance (WD), KL Divergence (KL), and Jensen-Shannon (JS) Distance. Xception SLADD RECCE SBI CADDM LAA-Net ForensicsAdapter Avg. AUCc
68.95
57.18
76.91
71.53
86.51
85.96
74.49
Cross-AUCWD Cross-AUCKL Cross-AUCJS
68.52 68.29 54.61
58.37 60.07 65.73
73.26 74.20 75.31 74.22 73.88 76.80 60.82 66.05 62.22
74.43
86.47 86.54 67.48
83.59 87.05 63.45
74.24 75.26 62.90
Cross-AUC with Different Polarization Estimator Π. We examine the effect of replacing the WD in Cross-AUC with alternative measures, including KL Divergence (KL) and Jensen-Shannon Distance (JS). As shown in Table 6. Both WD and KL-based Cross-AUC remain closely aligned with the combined AUC, whereas the JS variant yields less consistent estimates. This suggests that while Cross-AUC is generally robust to the choice of estimator, WD offers the most stable results under distribution shift. OOD Evaluation with Mag- Table 7: An out-of-distribution scenario icBrush. We further evaluate the with MagicBrush [39]. relevance of Cross-AUC beyond faMagicBrush AUCc AUCa Cross-AUC cial deepfake detection. Specifically, AUC 44.90 66.70 70.96 65.54 Xception WQ 0.029 0.170 0.246 we consider an Out-Of-Distribution AUC 38.25 56.98 65.58 58.44 (OOD) detection task using the MagSLADD WQ 0.007 0.013 0.029 icBrush [39] dataset, which contains AUC 53.70 72.23 77.70 73.20 RECCE WQ 0.001 0.196 0.239 non-facial manipulations that are unAUC 86.71 72.79 81.98 78.65 seen in other benchmarks. We report SBI WQ 0.209 0.187 0.269 the obtained results in Table 7. It AUC 53.24 69.60 78.33 74.08 CADDM WQ 0.012 0.172 0.272 shows that the gap between the avAUC 55.73 85.26 84.21 80.03 erage AUC and the combined AUC is LAA-Net WQ 0.018 0.309 0.428 even more pronounced, while CrossAUC 59.42 83.22 86.34 83.01 AUC remains consistently close to the ForensicsAdapter WQ 0.061 0.298 0.359 combined AUC. This highlights that Cross-AUC provides a reliable measure for both facial and non-facial content. Limitations. While Cross-AUC is simple, effective, and practical, we acknowledge the lack of theoretically grounded justifications. Despite this issue, we believe that this work remains valuable for the community. In fact, it highlights a persistent problem in deepfake detection that is mostly ignored in the literature, which hinders the transition toward truly deployable solutions. Moreover, it suggests, as a potential solution, the exploitation of polarization. Another limitation of this work lies in its exclusive focus on the deepfake detection use case. However, the observed lack of robustness of AUC under domain shift is, in principle, not specific to this application. It represents a broader issue that can affect a wide range of binary classification tasks. Studying Cross-AUC for other binary classification tasks will be, therefore, investigated in the future.
5
Conclusion
This work addresses the need to assess the generalization capabilities of deepfake detectors more accurately, as current evaluation practices overlook the complexity of real-world scenarios. We empirically demonstrate that computing AUC
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
15
scores separately across datasets obscures the true capabilities of detection models, which are expected to perform on real-world, mixed-domain data. To address this, we characterize the polarization of deepfake detector models by quantifying the degree of separation between real and fake score distributions. We introduce Cross-dataset AUC (Cross-AUC), a metric that extends the AUC scores by explicitly incorporating these score polarizations. This approach better captures the robustness of the models under domain shifts, which is expected in realistic evaluations. Our experiments indicate that most state-of-the-art detectors fail to generalize as much as they claim, and that Cross-AUC is a practical metric for evaluating their generalization performance, thereby highlighting the relevance of emphasizing polarity.
References 1. Bai, W., Liu, Y., Zhang, Z., Li, B., Hu, W.: Aunet: Learning relations between action units for face forgery detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 24709–24719 (June 2023) 2. Cao, J., Ma, C., Yao, T., Chen, S., Ding, S., Yang, X.: End-to-end reconstructionclassification learning for face forgery detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4103–4112 (2022). https: //doi.org/10.1109/CVPR52688.2022.00408 3. Chen, H., Magramo, K.: Finance worker pays out $25 million after video call with deepfake "chief financial officer". https://edition.cnn.com/2024/02/04/asia/ deepfake-cfo-scam-hong-kong-intl-hnk/index.html (2024), [Online; accessed 4-February-2024] 4. Chen, L., Zhang, Y., Song, Y., Liu, L., Wang, J.: Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 18710–18719 (June 2022) 5. Chen, L., Zhang, Y., Song, Y., Wang, J., Liu, L.: Ost: Improving generalization of deepfake detection via one-shot test-time training. Advances in Neural Information Processing Systems 35, 24597–24610 (2022) 6. Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1251–1258 (2017) 7. Cui, X., Li, Y., Luo, A., Zhou, J., Dong, J.: Forensics adapter: Adapting clip for generalizable face forgery detection. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 19207–19217 (2025) 8. Deepfakes: Faceswapdevs. https://github.com/deepfakes/faceswap (2019) 9. Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., Zafeiriou, S.: Retinaface: Single-stage dense face localisation in the wild. CoRR abs/1905.00641 (2019) 10. Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., Canton-Ferrer, C.: The deepfake detection challenge dataset. CoRR abs/2006.07397 (2020), https://arxiv.org/abs/2006.07397 11. Dolhansky, B., Howes, R., Pflaum, B., Baram, N., Canton-Ferrer, C.: The deepfake detection challenge (DFDC) preview dataset. CoRR abs/1910.08854 (2019), http://arxiv.org/abs/1910.08854
16
D. NGUYEN et al.
12. Dong, S., Wang, J., Ji, R., Liang, J., Fan, H., Ge, Z.: Implicit identity leakage: The stumbling block to improving deepfake detection generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3994–4004 (June 2023) 13. Dong, X., Bao, J., Chen, D., Zhang, T., Zhang, W., Yu, N., Chen, D., Wen, F., Guo, B.: Protecting celebrities from deepfake with identity consistency transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9468–9478 (June 2022) 14. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020) 15. Dufour, N., Gully, A.: Contributing data to deepfake detection research. https: //ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection. html (2019) 16. Fu, B., Cao, Z., Long, M., Wang, J.: Learning to detect open classes for universal domain adaptation. In: European conference on computer vision. pp. 567–583. Springer (2020) 17. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples (2015) 18. Huang, J., Ling, C.: Using auc and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering 17(3), 299–310 (2005). https://doi.org/10.1109/TKDE.2005.50 19. Kowalski, M.: Faceswap. https://github.com/MarekKowalski/FaceSwap (2018) 20. Kumagai, A., Iwata, T., Takahashi, H., Nishiyama, T., Fujiwara, Y.: Auc maximization under positive distribution shift. In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., Zhang, C. (eds.) Advances in Neural Information Processing Systems. vol. 37, pp. 36071–36096. Curran Associates, Inc. (2024), https://proceedings.neurips.cc/paper_files/paper/2024/file/ 3f8c7eb848ffec848f3ed2b7ca44915d-Paper-Conference.pdf 21. Kurakin, A., Goodfellow, I.J., Bengio, S.: Adversarial machine learning at scale. ArXiv abs/1611.01236 (2016), https://api.semanticscholar.org/CorpusID: 9059612 22. Kwegyir-Aggrey, K., Gerchick, M., Mohan, M., Horowitz, A., Venkatasubramanian, S.: The misuse of auc: What high impact risk assessment gets wrong. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023), https://api.semanticscholar.org/CorpusID:258960540 23. Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., Guo, B.: Face x-ray for more general face forgery detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020) 24. Li, Y., Yang, X., Sun, P., Qi, H., Lyu, S.: Celeb-df: A large-scale challenging dataset for deepfake forensics. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020) 25. Lin, L., He, X., Ju, Y., Wang, X., Ding, F., Hu, S.: Preserving fairness generalization in deepfake detection. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 16815–16825 (2024), https://api. semanticscholar.org/CorpusID:268031845 26. Nguyen, D., Astrid, M., Ghorbel, E., Aouada, D.: Fakeformer: Efficient vulnerability-driven transformers for generalisable deepfake detection. arXiv preprint arXiv:2410.21964 (2024)
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
17
27. Nguyen, D., Astrid, M., Kacem, A., Ghorbel, E., Aouada, D.: Vulnerability-aware spatio-temporal learning for generalizable deepfake video detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10786–10796 (2025) 28. Nguyen, D., Mejri, N., Singh, I.P., Kuleshova, P., Astrid, M., Kacem, A., Ghorbel, E., Aouada, D.: Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 17395–17405 (June 2024) 29. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (2021), https://api.semanticscholar.org/CorpusID: 231591445 30. Reuters: South korea to criminalize watching or possessing sexually explicit deepfakes. https://edition.cnn.com/2024/09/26/asia/south- korea- deepfakebill-passed-intl-hnk/index.html (2024), [Online; accessed 26-September-2024] 31. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Nießner, M.: FaceForensics++: Learning to detect manipulated facial images. In: International Conference on Computer Vision (ICCV) (2019) 32. Shiohara, K., Yamasaki, T.: Detecting deepfakes with self-blended images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18720–18729 (2022) 33. Tan, M., Le, Q.V.: Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946 (2019), http://arxiv.org/abs/1905.11946 34. Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: Image synthesis using neural textures. CoRR abs/1904.12356 (2019), http://arxiv.org/abs/ 1904.12356 35. Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: Real-time face capture and reenactment of RGB videos. CoRR abs/2007.14808 (2020), https://arxiv.org/abs/2007.14808 36. Ueda, N., Fujino, A.: Partial auc maximization via nonlinear scoring functions. arXiv preprint arXiv:1806.04838 (2018) 37. Yan, Z., Yao, T., Chen, S., Zhao, Y., Fu, X., Zhu, J., Luo, D., Wang, C., Ding, S., Wu, Y., et al.: Df40: Toward next-generation deepfake detection. Advances in Neural Information Processing Systems 37, 29387–29434 (2024) 38. Yan, Z., Zhao, Y., Chen, S., Guo, M., Fu, X., Yao, T., Ding, S., Yuan, L.: Generalizing deepfake video detection with plug-and-play: Video-level blending and spatiotemporal adapter tuning. arXiv preprint arXiv:2408.17065 (2024) 39. Zhang, K., Mo, L., Chen, W., Sun, H., Su, Y.: Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems 36, 31428–31449 (2023) 40. Zhang, X., Wang, Q., Zhang, J., Zhong, Z.: Adversarial autoaugment. ArXiv abs/1912.11188 (2019), https://api.semanticscholar.org/CorpusID: 209460718 41. Zhao, E., Xu, X., Xu, M., Ding, H., Xiong, Y., Xia, W.: Learning self-consistency for deepfake detection. In: ICCV 2021 (2021), https://www.amazon.science/ publications/learning-self-consistency-for-deepfake-detection 42. Zhao, H., Zhou, W., Chen, D., Wei, T., Zhang, W., Yu, N.: Multi-attentional deepfake detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2185–2194 (2021)
18
D. NGUYEN et al.
43. Zi, B., Chang, M., Chen, J., Ma, X., Jiang, Y.G.: Wilddeepfake: A challenging realworld dataset for deepfake detection. Proceedings of the 28th ACM International Conference on Multimedia (2020)
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
6
19
Appendix
Our technical appendix is organized as follows. Section 6.1 presents additional results and extended analyses that complement Section 4 of the main paper. Section 6.2 details the dataset information and selection criteria. Section 6.3 describes the seven deepfake detectors used in our experiments. Finally, Section 6.4 summarizes additional evaluation metrics beyond AUC and polarization. Table 8: Performance of the selected models on all datasets. τ : Optimal threshold; ϕτ : variance of the top–K optimal thresholds in each dataset; WKDE , WQ , WGMM , WBD : model polarities using WD as Π over different density estimation (Θ) methods, including non-parametric (Kernel Density Estimation (KDE), Quantile (Q)), and parametric ones assuming two types of distributions, i.e., GMM and BD. Gray represents the dataset the model was trained on. Green highlights the best performance and polarities. FF++
Xception [6]
SLADD [4]
RECCE [2]
SBI [32]
CDF
DFD
DFW DFDCP DFDC DF40 Average
Combined
ACC (%) 85.32 43.19 82.59 58.06 BACC (%) 87.31 53.55 79.79 57.95 AUC (%) 93.6 62.1 92.24 63.71 P (%) 96.03 74.48 83.24 57.8 R (%) 81.34 21.72 90.58 64.62 93.28 85.38 69 51.28 S (%) F1 (%) 88.08 33.64 86.76 61.02 EER (%) 13.43 42.69 15.5 40.51 τ 0.4 0.24 0.7 0.57 ϕτ 0.0004 0.0004 0.0002 0.0007 WKDE 0.524 0.064 0.430 0.110 WBD 0.549 0.068 0.446 0.122 WQ 0.661 0.077 0.524 0.139 WGMM 0.616 0.069 0.496 0.149
53.44 63.7 70.56 94.54 50.48 76.92 65.82 33.33 0.4 0.0010 0.146 0.164 0.183 0.118
55.92 57.56 62.29 55.84 67.17 66.47 58.98 81.59 74.68 57.71 41.95 72.25 42.39 92.49 63.37 69.29 92.4 76.79 48.88 57.72 63.13 43.58 25.6 30.66 0.3 0.24 0.4 0.0001 0.0001 0.0004 0.078 0.239 0.227 0.060 0.232 0.234 0.091 0.264 0.277 0.081 0.249 0.254
58.53 61.6 68.95 (↓ 5.73) 75.2 47.26 75.94 58.04 35.68 (↑ 5.02) 0.31 0.0001 0.168 0.152 0.190 0.180
ACC (%) 81.84 64.49 65.92 51.15 BACC (%) 80.03 50.95 54 50.45 88.31 56.65 97.22 48.63 AUC (%) P (%) 87.07 66.73 64.8 51.05 R (%) 85.44 92.55 100 93.65 S (%) 74.62 9.35 8 7.25 F1 (%) 86.25 77.55 78.7 66.08 EER (%) 18.65 44.44 7.5 52.22 τ 0.51 0.53 0.57 0.55 0.0001 0.0001 0.0001 0.0001 ϕτ WKDE 0.060 0.001 0.098 0.004 WBD 0.078 0.003 0.079 0.001 WQ 0.079 0.005 0.090 0.002 0.079 0.005 0.089 0.004 WGMM
87.06 53.51 65.98 89.52 96.76 10.25 93 41.02 0.53 0.0001 0.022 0.007 0.015 0.015
53.75 50.9 65.01 53.9 59.55 57.48 57.37 72.25 69.48 52.3 82.25 70.53 79.14 36.86 83.48 28.65 82.24 31.48 62.98 50.91 73.63 45.21 29.59 34.09 0.53 0.45 0.52 0.0001 0.0001 0.0001 0.012 0.003 0.028 0.008 0.024 0.029 0.010 0.026 0.032 0.010 0.027 0.033
55.77 53.38 57.18 (↓ 12.3) 63.29 64.56 42.2 63.92 46.4 (↑ 12.31) 0.52 0.0001 0.013 0.013 0.014 0.014
ACC (%) 92.28 70.01 75.18 54.6 BACC (%) 88.8 58.57 66.5 54.05 AUC (%) 99.56 69.02 97.82 65.19 90.16 70.62 71.72 53.2 P (%) R (%) 99.25 93.75 100 88.43 S (%) 78.35 23.39 33 19.67 F1 (%) 94.49 80.56 83.53 66.43 EER (%) 1.49 36.25 8 39.81 0.98 0.74 0.99 0.65 τ ϕτ 0.00001 0.0002 0.0001 0.0001 WKDE 0.631 0.126 0.352 0.096 WBD 0.569 0.124 0.280 0.116 WQ 0.716 0.136 0.355 0.115 WGMM 0.692 0.134 0.349 0.111
75.28 70.4 82.24 94.42 76.69 64.1 84.64 28.2 0.53 0.0001 0.212 0.240 0.243 0.227
63.4 71.21 71.7 63.52 53.89 65.1 71.19 82.94 81.13 59.25 70.78 72.87 84.46 99.31 91.69 42.58 8.48 38.51 69.64 82.65 80.27 34.43 24.48 24.66 0.58 0.97 0.77 0.0001 0.0001 0.0001 0.181 0.153 0.250 0.182 0.117 0.233 0.203 0.144 0.273 0.203 0.142 0.265
68.12 61.4 76.91 (↓ 4.22) 67.17 92.82 29.98 77.94 31.85 (↑ 7.19) 0.86 0.0001 0.221 0.201 0.232 0.228
ACC (%) 80.59 58.57 90.92 55.06 BACC (%) 85.26 68.17 91.35 55.76 AUC (%) 98.23 84.65 96.78 60.83 P (%) 99.47 97.01 95.61 91.8 R (%) 71.26 38.69 89.7 12.69 S (%) 99.25 97.66 93 98.82 F1 (%) 83.04 55.31 92.56 22.31 EER (%) 6.71 24.56 9 46.83 τ 0.21 0.28 0.46 0.16 ϕτ 0.0006 0.0001 0.0008 0.0001 WKDE 0.503 0.215 0.516 0.078 WBD 0.513 0.219 0.500 0.089
35.91 63.91 86.83 100 27.83 100 43.54 20.51 0.19 0.0002 0.220 0.224
55.12 40.14 59.47 54.86 55.98 67.89 69.77 72.05 81.3 97.9 92.86 96.37 9.94 14.44 37.79 99.78 97.52 98 18.05 24.99 48.54 36.11 33.36 25.29 0.14 0.17 0.23 0.0001 0.0001 0.0002 0.097 0.103 0.248 0.116 0.112 0.253
51.7 59.95 74.43 (↓ 6.87) 95.84 21.34 98.57 34.9 32.57 (↑ 7.28) 0.17 0.0001 0.152 0.182
20
D. NGUYEN et al. FF++
CDF
DFD
DFW DFDCP DFDC DF40 Average
0.576 0.567
0.238 0.236
0.577 0.564
0.085 0.084
0.244 0.245
0.108 0.105
ACC (%) 95.27 72.19 80.74 69.01 BACC (%) 93.47 72.56 74 69.35 AUC (%) 99.26 80.7 99.52 76.31 P (%) 94.31 84.21 76.58 84.13 R (%) 98.88 71.43 100 48.07 S (%) 88.06 73.68 48 90.63 F1 (%) 96.54 77.3 86.73 61.18 EER (%) 2.24 27.49 3.5 30.91 τ 0.97 0.45 0.93 0.37 0.0253 0.0001 0.0001 0.0001 ϕτ WKDE 0.661 0.203 0.436 0.210 WBD 0.607 0.203 0.386 0.238 WQ 0.726 0.226 0.466 0.246 0.714 0.224 0.459 0.241 WGMM
52.01 68.5 71 97.33 47.25 89.75 63.61 38.46 0.44 0.0011 0.156 0.182 0.179 0.174
63.72 43.7 68.09 58.55 63.6 57.41 71.27 63.32 70.33 76.28 81.91 71.53 (↓ 10.38) 72.92 87.83 85.33 81.51 42.97 21.46 61.43 41.01 84.22 93.36 81.1 85.64 54.08 34.49 67.7 54.56 36.06 30.48 24.16 35.97 (↑ 11.81) 0.37 0.22 0.53 0.31 0.0001 0.0001 0.0038 0.0001 0.155 0.137 0.280 0.176 0.165 0.146 0.275 0.214 0.174 0.151 0.310 0.198 0.173 0.150 0.305 0.197
ACC (%) 96.67 84.72 92.5 64.52 BACC (%) 97.14 87.71 88.67 65.01 AUC (%) 99.45 95.45 98.47 79.52 P (%) 99.26 98.15 91.71 85.22 R (%) 95.71 78.24 98.19 36.59 S (%) 98.57 97.18 79.15 93.43 F1 (%) 97.45 87.07 94.84 51.19 EER (%) 2.86 11.86 5.96 27.27 τ 0.23 0.3 0.96 0.06 ϕτ 0.0161 0.0004 0.0001 0.0001 WKDE 0.831 0.511 0.633 0.262 0.749 0.495 0.547 0.248 WBD WQ 0.927 0.582 0.708 0.285 WGMM 0.911 0.579 0.690 0.292
59.74 72.36 86.48 97.4 55.83 88.89 70.98 18.52 0.08 0.0005 0.334 0.357 0.446 0.417
57.68 37.79 70.51 57.37 65.99 76.32 72.6 86.04 88.28 90.99 99.66 94.62 16.33 33.63 59.21 98.41 98.35 93.42 27.69 50.29 68.5 33.91 22.2 17.51 0.03 0.11 0.25 0.0001 0.0001 0.0024 0.169 0.281 0.431 0.189 0.272 0.408 0.156 0.299 0.486 0.154 0.286 0.475
45.35 66.15 86.51 (↓ 1.77) 98.27 35.59 96.71 52.25 21.4 (↑ 3.89) 0.083 0.0001 0.302 0.296 0.318 0.317
ACC (%) 89.3 85.21 97.96 76.61 79.94 97.56 76.72 BACC (%) 84.89 AUC (%) 98.39 94.94 99.8 80.95 P (%) 87.38 83.9 97.68 81.32 R (%) 98.13 96.13 99.11 70.07 S (%) 71.64 63.74 96 83.37 ForensicAdapter [7] F1 (%) 92.44 89.6 98.39 76.27 EER (%) 5.22 12.28 1.5 25.53 τ 0.76 0.64 0.65 0.5 ϕτ 0.0013 0.0003 0.0027 0.0001 WKDE 0.497 0.338 0.659 0.251 WBD 0.460 0.334 0.624 0.253 WQ 0.540 0.374 0.713 0.285 WGMM 0.521 0.371 0.709 0.284
70.11 75.33 87.04 96.8 68.61 82.05 80.3 23.08 0.43 0.0007 0.259 0.268 0.293 0.288
75.46 71.96 80.94 75.39 77.12 80.99 83.88 86.3 90.18 84.2 93.82 89.3 62.34 63.6 79.71 88.43 90.64 82.26 71.64 75.8 83.49 24.03 22.32 16.28 0.4 0.38 0.53 0.0001 0.0001 0.0007 0.257 0.291 0.365 0.266 0.294 0.357 0.286 0.322 0.402 0.285 0.320 0.397
76.19 78.15 85.96 (↓ 4.22) 89.36 68.99 87.32 77.86 22.39 (↑ 6.11) 0.43 0.0001 0.304 0.306 0.336 0.335
WQ WGMM
CADDM [12]
LAA-Net [28]
6.1
0.116 0.115
0.278 0.274
Combined 0.172 0.171
Additional Results and Discussions
In addition to AUC, optimal threshold (τ ), threshold variance (ϕτ ), and polarization scores are reported in Table 2 of the main paper. In this section, for the sake of a more in-depth analysis, we provide results using additional standard classification metrics, namely, Accuracy (ACC), Balanced Accuracy (BACC), Precision (P), Recall (R), Specificity (S), F1-score (F1), and Equal Error Rate (EER). Table 8 presents the results of the seven considered detection models [2, 4, 7, 12, 28, 31, 32], all trained on FF++ [31], across six unseen datasets [10, 11, 15, 24, 37, 43] and the Combined test set. We note that we set a fixed threshold of 0.5 to compute these classification metrics. Conventional Cross-Dataset Generalization Evaluation. This setup directly tests cross-dataset generalization, a key challenge in deepfake detection, where models often fail when faced with manipulations or domains not seen during training. It can be observed that results show substantial performance variation across datasets. ForensicAdapter [7] consistently performs well across all test sets, maintaining strong ACC, AUC, and F1, which reflects robust generalization. RECCE [2] also performs well, especially in R, achieving high values on datasets
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift Xception SLADD
RECCE SBI
CADDM LAA-Net
ForensicsAdapter AUC (Average)
DFW
21
AUC (Combined)
Cross-AUC
DFD LAA-Net
SLADD
DFDCP
CDF2
100 CADDM
90
100
80
90
70
80
60
70
50 40
DFDC
60
FF++
50 40
SBI
RECCE
DF40
AUC (Average)
Xception
Cross-AUC
AUC (Combined)
ForensicsAdapter
Fig. 6: Methods results (AUC) for each Fig. 7: Direct comparison of average AUC, dataset. Average AUC, combined AUC, combined AUC, and the proposed Crossand the proposed Cross-AUC are also re- AUC on the studied deepfake detectors. ported.
like DFD [15] and DF40 [37]. However, its AUC drops below 70% on CDF [24] and DFW [43], indicating that its generalization is not uniformly strong across all domains. LAA-Net [28] achieves very high AUC on some datasets, such as CDF (95.45%) and DFD (98.47%), but its performance decreases on others like DFDC [10] (72.60%) and DF40 (86.04%), showing it is more sensitive to certain shifts. CADDM [12] performs moderately overall, with solid AUC and accuracy on several datasets, but lower R and higher EER on DFDC and DF40 indicate weaker transfer to more challenging domains. In contrast, SLADD [4] and Xception [31] experience sharp performance drops outside of FF++. Xception, for example, drops from 85.32% ACC on FF++ to below 60% on most other datasets, showing poor generalizability. SLADD performs better in R but lacks overall consistency in F1 and AUC. This is partly explained by their weak score separability, i.e., both models show low polarity, meaning they struggle to distinguish real and fake samples confidently. Even though SLADD maintains stable thresholds, its overlapping score distributions lead to unreliable decisions when tested on unfamiliar data, as shown in Figure 5 of the main paper. In particular, the AUC trends highlight these generalization patterns more clearly. ForensicAdapter is the only model that maintains AUC above 80% across all test datasets, indicating consistent class separation under domain shifts. Other models, including RECCE, LAA-Net, SBI, and CADDM, perform well on specific datasets but show noticeable drops on others, reflecting varying sensitivity to the target domain. Xception achieves competitive results on datasets like DFD and DF40, though its performance varies more widely across others. Notably, SLADD shows the most limited generalization, with the lowest Combined AUC and consistently low polarity scores, suggesting that it struggles to separate real and fake samples reliably across domains. A visual comparison of performance between detectors is shown in Figure 6.
22
D. NGUYEN et al.
The Combined Test Set: A Realistic Generalization Benchmark. As discussed in the main paper, we additionally propose a “Combined” test set, created by merging predictions from all test datasets, to reflect real-world use cases in which content is highly mixed. It offers a challenging test of generalization, i.e., models are exposed to many manipulation types, compression levels, and domain shifts simultaneously. ForensicAdapter again stands out with high Combined F1 (77.86%), low EER (22.39%), and strong AUC (85.96%). Its performance is both accurate and stable across conditions. LAA-Net reaches the highest Combined AUC (86.51%) and excels in P and S but suffers from low R, leading to a lower Combined F1 (52.25%). RECCE records the highest R and F1, exhibiting consistent behavior even in the most varied testing condition. In contrast, models like SLADD, SBI, and Xception show noticeable performance drops on the Combined set. SLADD’s high R leads to many false positives, while SBI and Xception exhibit more conservative behavior or limited score separability, affecting their reliability under mixed-domain conditions. This comparison highlights the limitation of using per-dataset averages to assess generalization. While averages may suggest overall stability, they can mask poor performance on specific domains. For example, SBI shows a decent average AUC but fails on DFDC due to very low R. SLADD has high average R, but low S on several datasets leads to frequent false positives. In contrast, ForensicAdapter performs consistently across both average and Combined evaluations, confirming its robustness. Hence, the Combined test set provides a more realistic and reliable measure of performance under diverse conditions. Cross-AUC versus Average AUC: A Visual Comparison. In addition to the results reported in Table 3 of the main paper as well as discussed in Section 4.2 of the main paper, we provide in Figure 7 a radar-based comparison between our proposed Cross-AUC and the conventional average AUC. As shown, across all models, Cross-AUC aligns more closely with \protect \text {AUC}_c than \protect \text {AUC}_a , yielding the smallest difference (\Delta _\text {Cross-AUC} ) in all cases. This underscores the importance of prediction polarization for assessing the generalization capability of detectors in mixed-domain and unconstrained settings. Threshold Stability and Score Separability. We examine threshold variability using ϕτ and τ , which reflect how consistently a model applies its decision boundary and how confident it is in its predictions. It can be observed that RECCE and SLADD show the most stable threshold behavior. SLADD has identical ϕτ values of 0.0001 across all datasets, and RECCE maintains similarly low values between 0.00001 and 0.0002. Their τ values are also consistent. RECCE operates at higher τ (0.77 on average), indicating more cautious predictions. This stability is supported by its moderately high polarity on the Combined set (WQ = 0.232), suggesting reasonably wellseparated score distributions. SLADD, on the other hand, has the lowest polarity (WQ = 0.014), showing that even with stable thresholds, the model struggles to confidently separate real and fake predictions.
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
23
SBI and Xception also show low ϕτ values, but their lower polarity scores suggest that threshold consistency does not translate into well-separated scores. Their decisions are more stable than confident. ForensicAdapter shows slightly more variation in threshold placement (e.g., 0.0027 on DFD), including the highest ϕτ on the Combined set (0.0007), but its high polarity (WQ = 0.336) supports confident decisions despite small shifts in threshold. LAA-Net and CADDM show less consistent threshold behavior, with noticeably higher ϕτ values on FF++ (0.0161 and 0.0253), and more fluctuation in τ across datasets. While both achieve high polarity, the instability in thresholds suggests more erratic decision behavior under domain shifts. Overall, RECCE is the only model that combines both stable thresholds and clear score separability. Other models show strength in one aspect but not both, which may impact their reliability under unseen conditions. Score Polarization and Its Relationship to Classification Metrics. We analyze polarity W using WD as Π between the score distributions of real and fake samples to quantify how well a model separates the two classes. Since classification metrics like AUC, EER, and F1 are directly influenced by how confidently a model scores real versus fake data, we expect models with higher polarity to generally exhibit stronger performance. In our evaluation, we examine how this relationship holds across models under cross-domain conditions. On the Combined test set, we observe that models with higher polarity, such as ForensicAdapter (WQ = 0.336), LAA-Net (WQ = 0.318), and RECCE (WQ = 0.232), also achieve strong classification results. ForensicAdapter pairs the highest polarity with an AUC of 85.96% and F1 of 83.49%, indicating that well-separated score distributions contribute directly to confident and reliable decisions. RECCE follows a similar trend, where moderate-to-high polarity aligns with high R and competitive F1, further supported by its stable thresholds across domains. LAA-Net shows high polarity and the highest AUC (86.51%), which suggests good score separation. However, its R remains low, leading to a reduced F1 score (52.25%). This gap indicates that while the model distinguishes real and fake scores well, it tends to bias predictions toward the real class, possibly due to misaligned threshold behaviors or domain sensitivity. In contrast, SLADD and Xception exhibit much lower polarity, with WQ = 0.014 and 0.190 respectively. SLADD shows stable thresholds across datasets, but the score overlap between real and fake classes results in high false positive rates and low F1 on the Combined set. This confirms that stable thresholds alone do not guarantee good performance. Without clear separation in the score space, the models’ predictions become sensitive and less confident under domain shifts. Xception, while having slightly higher polarity than SLADD, struggles with both generalization and score separability. Its performance drops sharply outside of FF++, and the weak polarization across domains contributes to high EER and poor generalizability. SBI, meanwhile, offers a different failure mode. Despite modest polarity (WQ = 0.172), it achieves very high P and S, but extremely low R. Its behavior reflects the use of conservative thresholds rather than confident score separation, a pattern supported by its relatively low τ values
24
D. NGUYEN et al.
and narrow decision margins. While this approach minimizes false positives, it also prevents the model from detecting many fake samples, limiting its utility in general-purpose settings. These findings confirm that polarity is a strong indicator of how confidently a model distinguishes real and fake content, and it typically correlates with AUC and EER. However, it must be considered alongside threshold behavior and precision-recall balance to fully understand a model’s generalization capability. Models that combine high polarity with consistent thresholds and balanced decision strategies, such as ForensicAdapter and RECCE, are more reliable under cross-domain conditions. Precision-Recall Trade-off. While earlier sections analyze model performance from score separability and threshold stability perspectives. In this section, we focus on the practical trade-offs between P and R. Models like RECCE and SLADD tend to favor R, detecting more fake samples but at the cost of increased false positives. In contrast, SBI and LAA-Net show conservative behavior, prioritizing P and S while often missing actual deepfakes. ForensicAdapter offers the most balanced trade-off, maintaining high values for both P and R. These differences highlight how models vary not just in accuracy, but also in how cautiously or aggressively they make decisions, an important consideration for real-world deployment. Failure Case Analysis. We observe consistent performance drops on certain datasets, revealing where generalization tends to fail. These cases highlight the sensitivity of some models to domain-specific shifts not reflected in overall averages. As shown in Table 8, the DFDC dataset poses a common challenge. Models such as LAA-Net, RECCE, and CADDM show significant declines in performance. For example, LAA-Net drops to 16.33% R and 27.69% F1, despite achieving top AUCs on other datasets. RECCE and CADDM also report high EERs (34.43% and 36.06%) and reduced F1, indicating difficulty adapting to DFDC’s unique characteristics, such as higher compression or different manipulation styles. DF40 also reveals failure points. Xception and SBI, in particular, suffer from very low R (14.44% and 21.46%, respectively), despite maintaining high S. These patterns suggest that overly conservative models fail to detect fakes when the score distributions shift, even if their decision thresholds remain stable. Such failures emphasize the limits of models that rely heavily on training distribution priors. While models like ForensicAdapter maintain robust results even under domain variation, others show clear breakdowns when faced with unseen manipulation types or domain properties not covered by FF++. Table 9: Runtime (in seconds) for computing polarization using different estimators across datasets. FF++
CDF
DFD
DFW DFDCP DFDC DF40 Combined
WKDE 0.0014 0.0012 0.0012 0.0010 0.0012 0.0015 0.0023 0.0036 WBD 0.0013 0.0012 0.0011 0.0011 0.0014 0.0011 0.0012 0.0013 WQ 0.0008 0.0008 0.0008 0.0013 0.0007 0.0129 0.0148 0.0847 WGMM 2.2309 2.1118 2.2576 2.7162 2.0931 4.8132 5.2694 8.3977
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
25
Runtime Analysis. Table 9 reports the runtime required to compute polarization for each estimator. As shown, non-parametric variants (WKDE , WQ ) incur only millisecond-level overhead, while the GMM-based estimator (WGMM ) is slower. Nevertheless, all variants remain practical for offline evaluation. Given its strong empirical performance and low computational cost, we use the quantilebased WD estimator (WQ ) by default. Table 10: Effect of the number of test domains. No. of Domains
Method
AUCc AUCa Cross-AUC
7 (All)
Xception [31] 68.95 SLADD [4] 57.18 RECCE [2] 76.91 SBI [32] 74.43 CADDM [12] 71.53 LAA-Net [28] 86.51 ForensicsAdapter [7] 85.96
74.68 69.48 81.13 81.30 81.91 88.28 90.18
68.52 58.37 73.26 74.20 75.31 86.47 83.59
6 (w/o DF40)
Xception [31] 65.13 SLADD [4] 62.55 RECCE [2] 74.16 SBI [32] 75.62 CADDM [12] 76.42 LAA-Net [28] 84.79 ForensicsAdapter [7] 87.36
73.53 69.02 80.83 82.84 82.85 88.66 90.83
68.53 57.62 72.70 75.38 76.88 86.33 85.20
Xception [31] 73.32 76.44 SLADD [4] 67.58 71.35 RECCE [2] 76.76 82.76 SBI [32] 82.19 85.46 CADDM [12] 83.13 85.35 LAA-Net [28] 94.52 91.87 ForensicsAdapter [7] 91.45 92.22
73.06 64.09 78.51 80.81 82.60 93.77 93.53
Xception [31] 74.58 77.91 SLADD [4] 68.31 72.70 RECCE [2] 81.91 82.89 4 (w/o DF40, DFDC, DFDCP) SBI [32] 82.09 85.12 CADDM [12] 85.76 88.94 LAA-Net [28] 95.72 93.22 ForensicsAdapter [7] 92.75 93.52
73.56 65.87 79.33 81.36 87.57 93.25 92.12
5 (w/o DF40, DFDC)
Effect of the Number of Test Domains. In Table 10, we conduct an ablation study by progressively reducing the number of test domains from seven to four and measuring how Cross-AUC, average AUC (AUCa ), and Combined AUC (AUCc ) varies across different detectors. We found that Cross-AUC consistently remains close to AUCc , even as domain diversity decreases, whereas AUCa tends to overestimate the performance. This supports the stability of Cross-AUC under varying domain coverage, suggesting that it reflects real-world generalization risk more reliably than per-domain average. Extended Analysis: Cross-AUC with Different Polarization Estimators. As discussed in Section 4.2 of the main paper, to assess the influence of the polarization component, we evaluate Cross-AUC with three different estimators including Wasserstein Distance (WD), KL Divergence (KL), and JensenShannon Distance (JS). Table 6 of the main paper reports the resulting CrossAUC values, while Table 11 in this supplementary material details the underlying polarization scores from each estimator. The results show that both WD and KL yield polarization magnitudes that scale proportionally with the de-
26 D. NGUYEN et al. Table 11: Detailed polarization scores for different estimators, which are used to compute the Cross-AUC results reported in Table 6 of the main paper. Polarization Score DFW DFDCP DFDC DF40 Average Combined
Method
Estimator
Xception [31]
WD KL JS
0.661 0.077 0.524 0.139 0.183 23.597 17.919 16.039 15.340 20.217 0.797 0.712 0.794 0.699 0.792
0.091 4.925 0.413
0.264 0.277 11.431 15.638 0.554 0.680
0.190 2.999 0.345
SLADD [4]
WD KL JS
0.079 17.695 0.686
0.015 14.469 0.593
0.010 0.553 0.199
0.026 2.764 0.375
0.032 9.089 0.471
0.014 0.744 0.185
RECCE [2]
WD KL JS
0.716 0.136 0.355 26.704 17.120 4.266 0.824 0.701 0.755
0.115 0.243 12.087 20.226 0.628 0.791
0.203 4.199 0.464
0.144 1.922 0.518
0.273 12.361 0.669
0.232 1.187 0.418
SBI [32]
WD KL JS
0.576 0.238 0.577 0.085 21.895 18.545 22.352 9.984 0.813 0.738 0.813 0.557
0.244 20.633 0.809
0.108 5.346 0.381
0.116 6.802 0.426
0.278 15.080 0.648
0.172 5.859 0.389
CADDM [12]
WD KL JS
0.726 0.226 0.466 0.246 0.179 25.675 18.646 24.948 15.572 19.638 0.830 0.736 0.825 0.673 0.773
0.174 5.993 0.441
0.151 7.800 0.492
0.310 16.896 0.681
0.198 3.873 0.365
LAA-Net [28]
WD KL JS
0.927 0.582 0.708 0.285 0.446 25.423 21.371 23.247 19.313 19.500 0.822 0.800 0.804 0.742 0.749
0.156 0.299 0.486 6.680 12.617 18.307 0.397 0.562 0.696
0.318 9.745 0.528
ForensicsAdapter [7]
WD KL JS
0.540 0.374 0.713 0.285 0.293 0.286 0.322 0.402 23.222 21.466 24.085 17.018 20.781 11.225 13.890 18.812 0.819 0.798 0.830 0.708 0.815 0.557 0.590 0.731
0.336 10.989 0.535
FF++
CDF
DFD
0.005 0.090 0.002 4.914 21.158 2.073 0.374 0.767 0.303
gree of domain mismatch, leading to Cross-AUC values that remain close to the combined AUC. In contrast, JS produces less discriminative polarization scores, which translate into weaker alignment in Cross-AUC. This suggests that although Cross-AUC is generally robust to the choice of estimator, the WD formulation offers the most stable and interpretable behavior, with KL as a competitive alternative. Table 12: Overview of datasets. The number of “Methods” is categorized into four subsets: Face-Swapping (FS), Face-Reenactment (FR), Entire Face Synthesis (EFS), and Face Editing (FE). “Perturbs” denotes the number of perturbations applied, while “Unknown” indicates whether the source contains prior knowledge of manipulations. Dataset
Venue
FF++ [31] ICCV’19 DFD [15] None DFDCP [11] ArXiv’19 CDF [24] CVPR’20 DFW [43] ACMMM’20 DFDC [10] ArXiv’20 DF40 [37] NeurIPS’24
6.2
Real videos Fake videos FS FR EFS FE Methods Perturbs Unknown 1,000 363 1,131 590 3,805 23,564 1428
4,000 3,068 4,113 5,639 3,509 104,500 0.1M+
2 2 5 - 1 - 5 1 2 10 13 12
5
4 5 2 1 8 40
2 3 19 6
× × × × ✓ × ×
More Details of Datasets
Seven standard and challenging datasets are used in our experiments, including FaceForensics++ (FF++) [31], Celeb-DF (CDF) [24], Google Deepfake Detection (DFD) [15], WildDeepfake (DFW) [43], Deepfake Detection Challenge (DFDC) [10], and Deepfake Detection Challenge Preview
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
27
(DFDCP) [11], and DF40 [37]. These datasets are selected based on the following criteria: 1) they are widely used for evaluation in the research community; 2) they contain fakes from diverse sources, including those generated by black-box or undisclosed methods, for which no implementation details or prior knowledge are available; 3) they present increased complexity as a result of various image perturbations, such as compression, noise, or blur. An overview of the selected benchmarks is presented in Table 12. Specifically, FF++ is typically used for training, while the others serve as test sets in the cross-dataset evaluation protocol. It consists of 1000 real videos and 4000 fakes videos with different compression levels, which are generated by four manipulation techniques, i.e., Deepfakes (DF) [8], Face2Face (F2F) [35], FaceSwap (FS) [19], and NeuralTextures (NT) [34]. In our experiments, we adopt the raw version of FF++. CDF is one of the most common and challenging benchmarks typically used to assess the generalizability of deepfake detectors under the cross-dataset evaluation setting [1, 2, 4, 7, 13, 23, 28, 32]. It contains highly realistic deepfakes. DFD includes more than 3000 forged videos featuring 28 actors in various scenes. DFW is fully sourced from the internet, without prior knowledge of generation methods. DFDC and its preview version (DFDCP) are challenging, large-scale datasets that contain numerous distorted videos applied by several perturbation techniques such as compression, noise, etc. The recent DF40 benchmark, a highly diverse and large-scale dataset comprising 40 distinct deepfake techniques, enables more comprehensive evaluations for the next generation of deepfake detection. 6.3
More Details of Deepfake Detectors
In this section, we describe the seven state-of-the-art (SOTA) deepfake detection models [2, 4, 6, 7, 12, 28, 32] evaluated under our proposed protocol (see Section 4.1 in the main paper). XceptionNet 3 : In this work, the well-known benchmark FF++ [31] is introduced. The Xception architecture proposed in [6] was trained in an end-to-end manner using both real and fake data from FF++ [31]. SLADD 4 : In [4], the authors introduce an approach called SLADD [4] that leverages adversarial training [17, 21, 40]. In particular, it relies on two components: a generator and a discriminator (detector). While the generator attempts to synthesize facial images based on splicing operations, the detector aims to discriminate between fake and real faces. SLADD is trained using the generated pseudo-fakes as well as real and deepfake images from FF++. This approach can be seen as a hybrid method since it simultaneously takes advantage of a data synthesis strategy as well as a multi-task learning framework. This approach is claimed to be generic, given the high AUC achieved under the cross-dataset protocol. 3 4
https://github.com/ondyari/FaceForensics https://github.com/liangchen527/SLADD
28
D. NGUYEN et al.
RECCE 5 : This method, referred to as RECCE [2], is based on a reconstructionclassification schema. While the reconstruction learning over real images allows for extracting forgery-aware features, the classification learning mines the main discrepancy between real and fake images. Both real and fake data from FF++ [31] are used during training. This multi-task learning framework is shown to achieve competitive AUC performance under the cross-dataset setting. SBI 6 : This approach, called SBI [32], is based on a novel data synthesis. This data synthesis produces pseudo-fakes by blending source and target images from a single pristine image. The main idea is that such an augmentation generates pseudo-fake images with very subtle artifacts, thereby pushing the classifier to learn more generic representations. Only real data from FF++ [31] and pseudofakes are used for training a binary classifier. Similar to [2], an impressive AUC is registered for the cross-dataset setting. CADDM 7 : The proposed method, ID-unaware Deepfake Detection Model, aims to improve the generalization of deepfake detectors by addressing the issue of Implicit Identity Leakage (IIL), the unintended reliance on identity-specific features. To mitigate this, the authors introduce an Artifact Detection Module (ADM) that encourages the model to focus on local visual artifacts rather than global identity features. ADM uses a multi-scale anchor-based detection approach to identify regions in an image that may contain manipulation traces, helping the model distinguish between real and fake content at a finer, more localized level. To support ADM training, they design a Multi-scale Facial Swap (MFS) technique that generates fake images with labeled artifact areas by blending regions of source and fake images using different window sizes and blending strategies. This synthetic data provides ground-truth artifact locations, allowing the model to learn more generalized features. By shifting attention away from identity and toward manipulation cues, this method significantly improves cross-dataset performance and robustness to unseen forgery techniques. LAA-Net 8 : LAA-Net (Localized Artifact Attention Network) is a fine-grained deepfake detection framework designed to effectively detect high-quality and unseen manipulations by focusing on localized visual inconsistencies. At its core, LAA-Net introduces an explicit attention mechanism within a multi-task learning framework composed of three parallel branches: a binary classification branch to distinguish real from fake images, a heatmap branch to localize regions with potential blending artifacts guided by “vulnerable points”, and a self-consistency branch that evaluates similarity between vulnerable pixels and their surroundings. These branches are trained using pseudo-fake data generated through blending-based synthesis from real images, eliminating the need for large-scale fake datasets. Additionally, LAA-Net integrates an Enhanced Feature Pyramid Network (E-FPN) to capture and preserve discriminative low-level fea5
https://github.com/VISION-SJTU/RECCE https://github.com/mapooon/SelfBlendedImages 7 https://github.com/megvii-research/CADDM 8 https://github.com/10Ring/LAA-Net 6
Polarization-Aware Evaluation of Deepfake Detectors under Domain Shift
29
tures across multiple scales while minimizing redundancy. By combining precise attention guidance with robust feature representation, LAA-Net significantly improves the generalizability and accuracy of deepfake detection, even under challenging cross-dataset and quality-agnostic scenarios. ForensicAdapter 9 : ForensicsAdapter is a lightweight and task-specific adapter network designed to transform CLIP into a powerful and generalizable face forgery detector. Unlike previous CLIP-based approaches that treat CLIP as a frozen feature extractor, this method introduces a dedicated adapter module that operates in parallel with CLIP to detect subtle manipulation traces, especially blending boundaries characteristic of forged faces. The adapter is trained using a combination of objectives: masked blending boundary detection, patch-wise contrastive learning, and sample-wise contrastive learning, each tailored to enhance sensitivity to forgery cues. A novel interaction mechanism is also proposed to enable bidirectional knowledge exchange between the adapter and CLIP: the adapter absorbs low-level CLIP visual tokens to enrich its learning, and in return, uses attention bias to guide CLIP’s focus toward forgery-specific features without altering CLIP’s core parameters. Despite having only 5.7M trainable parameters, ForensicsAdapter achieves state-of-the-art performance across multiple benchmarks and proves highly robust to domain shifts and image perturbations, making it a strong baseline for future CLIP-based forgery detection work. 6.4
Additional Evaluation Metrics
In addition to score polarization measured by Wasserstein Distance (W ), AUC, and our proposed Cross-AUC, we report additional metrics to support a more in-depth analysis. These include Accuracy (ACC), Balanced Accuracy (BACC), Precision (P), Recall (R), Specificity (S), F1-score (F1), and Equal Error Rate (EER). We consider the Fake and Real classes as the Positive and Negative classes, respectively. Equal Error Rate (EER): is a common metric for evaluating the performance of biometric authentication systems, such as speaker verification or facial recognition. It represents the point at which the False Acceptance Rate (FAR) and the False Rejection Rate (FRR) are equal. At this threshold, the system is equally likely to incorrectly accept an unauthorized user (false positive) as it is to incorrectly reject an authorized user (false negative). A lower EER indicates better overall system performance, as it reflects a more optimal trade-off between FAR and FRR. In the context of Deepfake Detection, EER is used to evaluate the ability of a detector to distinguish between real and manipulated (fake) content. Here, the FAR corresponds to the rate at which fake content is incorrectly classified as real, and the FRR corresponds to the rate at which real content is incorrectly classified as fake. EER provides a single value that reflects the detector’s robustness and balance between these two types of errors. 9
https://github.com/OUC-VAS/ForensicsAdapter
30
D. NGUYEN et al.
Accuracy (ACC): It is computed based on the percentage of correct predictions as follows, \text {ACC} = \frac {\text {TP}+\text {TN}}{\text {TP}+\text {FP}+\text {TN}+\text {FN}} \text {,}
(8)
where TP, FP, TN, and FN denote True Positives, False Positives, True Negatives, and False Negatives, respectively. While commonly used, ACC can be misleading when the dataset is imbalanced [20, 36]. For instance, in the context of deepfake detection, standard datasets such as FF++ contain four times more fake than real samples (see Table 12). As a result, a model biased toward predicting the majority class (Fake) may achieve high ACC without truly distinguishing between real and fake content. To address this, we also report Balanced Accuracy (BACC), which provides a more informative view under class imbalance. Balanced Accuracy (BACC): It is calculated based on the average recall values for both positive and negative classes. \text {BACC} = \frac {1}{2} \left ( \frac {\text {TP}}{\text {TP} + \text {FN}} + \frac {\text {TN}}{\text {TN} + \text {FP}} \right ) \text {.}
(9)
Specificity (S): It describes how well the model detects the negative samples. It is calculated as the proportion of correct negative predictions over the actual total number of negative samples. The formula is defined below, \text {S} = \frac { \text {TN}}{ \text {TN}+ \text {FP}}\text {.}
(10)
Precision (P): It describes how correct the positive predictions are. It is calculated as the proportion of correctly predicted positive samples over the total number of predictions classified as positive. The formula is defined below, \text {P} = \frac { \text {TP}}{ \text {TP}+ \text {FP}} \text {.}
(11)
Recall (R): It describes how well the model detects the positive samples. It is calculated as the proportion of correct positive predictions over the actual total number of positive samples. The formula is defined below, \text {R} = \frac { \text {TP}}{ \text {TP}+ \text {FN}}\text {.}
(12)
F1-score (F1): It is often used to assess the overall performance of a classification model. It is computed as follows, \text {F1} = 2.\frac {\text {P}.\text {R}}{\text {P}+\text {R}} \text {.}
(13)