arXiv CSopen access

Backdoor Mitigation in Object Detection via Adversarial Fine-Tuning

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

cryptography cybersecurity privacy security

cryptography, security, privacy, cybersecurity

Backdoor Mitigation in Object Detection via Adversarial Fine-Tuning Kealan Dunnett1 , Reza Arablouei2 , Dimity Miller1 , Volkan Dedeoglu2 , and Raja Jurdak1

arXiv:2605.05928v1 [cs.CV] 7 May 2026

Queensland University of Technology, Brisbane Australia 2 CSIRO, Pullenvale QLD, Australia

Abstract Backdoor attacks can implant malicious behaviours into deep models while preserving performance on clean data, posing a serious threat to safety-critical vision systems. Although backdoor mitigation has been studied extensively for image classification, defenses for object detection remain comparatively underdeveloped. Adversarial fine-tuning is a common backdoor mitigation approach in classification, but adapting it to detection is nontrivial as classification-oriented adversarial generation does not match the detection attack space, where attacks may cause object misclassification or disappearance, and standard detection losses can dilute the repair signal across many predictions. We address these challenges through a detection-aware adversarial fine-tuning framework for mitigating object-detection backdoors when the defender has access only to a compromised detector and a small clean dataset, without knowing the attack objective. For adversarial generation that does not require knowledge of the attack objective, we introduce soft-branch minimisation, which uses a soft gate to combine objectives aligned with misclassification and disappearance attacks, together with a detection-aware classification-loss maximisation. For targeted repair, we introduce a dual-objective fine-tuning loss applied to target-matched predictions, concentrating the defensive update on predictions most relevant to the backdoor behaviour. Experiments across CNN- and Transformer-based detectors show that our approach more effectively reduces attack success while preserving true detections, compared with classification-oriented baselines, and maintains competitive clean detection performance.

Introduction

The rapid deployment of deep learning in vision systems has heightened concerns about security vulnerabilities that can directly compromise downstream decisions in the physical world [14]. Among the most concerning threats are backdoor attacks. In a backdoor attack, an adversary tampers with the training pipeline (e.g., by poisoning a small subset of the training samples) so that the trained model behaves normally on clean inputs but exhibits attacker-chosen behavior whenever a specific trigger pattern is present (e.g., a small colored patch) [10]. Although backdoors have been studied extensively in image classification, their behavior and mitigation in object detection remain much less understood, despite the central role of detectors in safety-critical applications such as autonomous driving and surveillance. Backdoors in object detection differ fundamentally from those in classification. Rather than producing a single class, modern detectors produce a set of predictions, namely bounding boxes with associated class scores, which expands both the attack surface and the range of possible attack objectives. Among the threat models studied in prior work [2, 15, 7], object targeting attacks (i.e., attacks that target the predictions of specific objects in an image) are particularly prominent. In region misclassification Preprint.

attacks (RMAs), trigger-bearing objects are misclassified into an attacker-chosen target class. In object disappearance attacks (ODAs), the predictions for attacked objects are suppressed [2]. However, defenses for backdoor attacks in object detection have not kept pace. Existing countermeasures are dominated by two directions: (i) inference-time trigger detection [21] and (ii) trigger synthesis or pattern-recovery approaches [16, 3]. By contrast, direct backdoor mitigation (i.e., repairing a compromised detector using only limited clean data) remains largely unexplored. While [23] proposes a mitigation method, their approach is restricted to older two-stage detectors (e.g., Faster R-CNN), and assumes access to clean and backdoor data. As a result, methods that propose a more general-purpose design applicable to a wider range of detector classes without assuming access to backdoor data are required moving forward. Among classification-oriented approaches, adversarial fine-tuning has emerged as a prominent mitigation paradigm, using adversarial examples as surrogate triggers to identify and suppress the vulnerable pathways exploited by a backdoor [18, 24, 20]. However, adapting this paradigm to object detection is not straightforward. Object detectors produce multiple coupled predictions with intertwined localization and classification components, and the relevant attack behaviours are heterogeneous. RMA requires a non-target class score to cross the detection threshold, while ODA requires all foreground class scores to fall below it. A second difficulty arises during fine-tuning. Detection losses are aggregated across many candidate predictions, most of which correspond to background regions or non-target objects. When fine-tuning on an adversarially perturbed image, the gradient associated with the targeted object can therefore be diluted by contributions from unrelated predictions, limiting the effectiveness of the standard detection loss as an object-level repair signal. These observations suggest that adversarial fine-tuning for detection requires two detection-specific components: (i) adversarial generation aligned with the score-threshold structure of both RMA and ODA without requiring the defender to know which the adversary is using, and (ii) a concentrated repair objective focused on predictions matched to the targeted object. Motivated by this, we develop a detection-aware adversarial fine-tuning approach for backdoor mitigation in object detection. Our approach addresses two structural mismatches between classification and detection settings: (i) adversarial generation must handle an unknown attack objective, and (ii) the repair signal must concentrate on the targeted object. To address the first, we introduce soft-branch minimisation (SBM), which softly combines RMA- and ODA-aligned objectives when the attack objective is unknown to the defender. We contrast SBM with classification-loss maximisation (CLM), a detection-aware adaptation of the standard maximisation paradigm. To address the second, we propose a dual-objective defence loss applied to target-matched predictions. This loss complements the standard detection loss by directly restoring the ground-truth class score while suppressing competing class scores at the object level. We evaluate the approach across CNN- and Transformer-based detectors (FCOS, YOLOv5, DINO) on Pascal VOC, MTSD, and COCO, and against both RMA and ODA. The results show that our approach improves the trade-off between attack success rate (ASR) and true detection rate (TDR) relative to classification-oriented baselines while maintaining a favourable clean detection performance.

Background and Related Work

Backdoor Attacks in Object Detection: BadDet [2] introduced backdoor attacks for object detection and formalized threat models such as RMA and ODA. Subsequent work expanded these attack behaviors and improved their practicality, for example, by extending disappearance to the untargeted setting [15], inducing disappearance without explicit annotation editing [4], and evaluating more realistic scene-level manipulations and real-world benchmarks [5]. More recently, BadDet+ [7] revisited this line of work and showed that existing data-poisoning attacks are substantially less effective than originally reported: BadDet remains a viable RMA attack but frequently produces duplicate detections in which the original class persists alongside the target, while existing ODA attacks [15, 4, 5] fail to reliably suppress objects even at extreme poisoning ratios. To address this, BadDet+ proposes a unified training-time loss manipulation that reliably achieves both RMA and ODA across detectors and datasets. Backdoor Defenses for Object Detection: Compared with image classification, defenses tailored to backdoors in object detection remain limited. Existing efforts fall into two main directions: (i) 2

Figure 1: Overview of the proposed adversarial fine-tuning approach. (a) and (b) Adversarial example generation for a target object using the proposed maximisation and soft minimisation strategies. (c) Detection-aware fine-tuning objective used to mitigate the induced RMA- or ODA-style behaviour. test-time detection of triggered inputs or objects, e.g., by checking prediction consistency under input transformations [21]; and (ii) trigger inversion or scanning methods that synthesize object-level perturbations to elicit backdoor behavior [16, 3]. Both directions face practical limitations. The testtime approach of [21] relies on two independent and expensive search procedures, one for RMA and another for ODA, each requiring multiple forward passes per input. Even assuming perfect detection of attacked objects, running both procedures on every input would incur prohibitive computational cost. The trigger-inversion methods of [16, 3] assume the adversary commits to a specific victim class, an assumption violated by attacks such as BadDet, BadDet+, Morph, UBA, and Align, which train the backdoor to affect arbitrary classes. Methods that explicitly mitigate backdoors in compromised detectors are far scarcer. To the best of our knowledge, [23] proposes the only such mitigation method for object detection, but it is tailored specifically to Faster R-CNN and requires access to both clean and backdoor data. A natural fallback is to reuse classification-oriented defenses, which are dominated by two paradigms: fine-tuning-based methods that retrain or regularise the model on a small clean subset, and pruningbased methods that identify and remove parameters most implicated in the backdoor [19, 6]. Two features prevalent across this literature obstruct direct transfer to detection. First, many state-ofthe-art methods in both families rely on an adversarial-example generation step that perturbs an input to elicit the backdoor’s predicted class [18, 24, 20, 1]; this presumes a single-prediction, single-label abstraction that detection violates as each image produces many joint localisationclassification predictions, and under ODA the objective is the absence of class expression rather than misclassification. Second, pruning-based methods predominantly assume a channel-level structure tied to CNN classifiers and do not extend cleanly to transformer-based detectors. These limitations motivate detector-aware mitigation strategies that provide (i) adversarial generation that addresses both RMA and ODA without assuming which the adversary is using, and (ii) a concentrated repair objective.

Preliminaries

Let fθ denote an object detector with parameters θ. For an image x, let B = {(bi , yi )}N i=1 be the ground-truth set, where bi ∈ R4 and yi are the bounding-box coordinates and class label of the i-th object, respectively. The detector produces N̂ candidate predictions B̂ = {(b̂j , zj )}N̂ j=1 , where 4 C b̂j ∈ R are predicted bounding boxes and zj ∈ R are class logits over the C foreground classes. The corresponding inference-time class scores are denoted by sj ∈ [0, 1]C where sj,c is the score for class c. Final detections are obtained by applying the detector’s standard post-processing to {(b̂j , sj )}N̂ j=1 , including score thresholding at τ and non-maximum suppression (NMS). During training, the detector’s matching strategy M assigns predictions to ground-truth objects, yielding a mapping π : {1, . . . , N̂ } → {0}∪{1, . . . , N }. Here π(j) = i > 0 indicates that prediction j is matched to ground-truth object i, while π(j) = 0 marks an unmatched (background) prediction. The predictions are then supervised by a detection loss that we abstract as LOD = LLOC + LCLS , 3

(1)

where LLOC penalizes bounding-box regression error between each matched prediction b̂j and its assigned ground-truth box bπ(j) , and LCLS penalizes classification error between the predicted logits zj and the target label yπ(j) . This two-term decomposition is a simplification, as specific detector architectures may include additional components. Object-localized perturbation: Given a target object i⋆ , we optimize an ℓ∞ -bounded perturbation δ restricted to the target box via a binary mask M(bi⋆ ): x′ = Π[0,1] x + M(bi⋆ ) ⊙ δ , ∥δ∥∞ ≤ ϵ. (2) Matched predictions for the target: Let πx′ denote the assignment computed on the perturbed image x′ using the matching strategy M. We define the set of predictions matched to the target object as Ji⋆ (x′ ) = {j | πx′ (j) = i⋆ }, (3) which is typically a singleton for set-based detectors and may contain multiple anchors or spatial locations for dense detectors. In Appendix A.2, we provide a gradient-alignment rationale for restricting the inner optimisation to this target-matched set. Threat Model: We consider the standard repair setting used in backdoor mitigation: the defender has access only to a compromised detector and a limited set of clean data. We assume that the attack is object-specific, namely an ODA or RMA, but that the defender does not know the attack objective. These are object-targeting attacks in the sense that they affect predictions for trigger-bearing objects, but class-untargeted in that the attack is not tied to a specific victim class. Accordingly, we do not consider attacks that target background regions within existing images, such as the object-generation attack of [2]. Therefore, the defender’s goal is to mitigate the unknown backdoor behaviour using only clean data. Fine-tuning in Classification: Since the defender typically lacks prior knowledge of the trigger or attacker objective, adversarial fine-tuning has emerged as a prominent mitigation paradigm in image classification [18, 24, 20], using adversarial examples as surrogate backdoor data. The approach is formulated as a minimax problem: min max LCE (fθ (x + δ), y), θ

∥δ∥p ≤ϵ

(4)

where the inner maximization seeks the most harmful perturbation δ within an ℓp -bounded neighborhood of input x with ground-truth label y. Because a backdoored model relies on trigger-activated shortcut features, this inner step is biased toward perturbations that exploit related vulnerable pathways; the outer minimization then fine-tunes θ to remain correct under such perturbations, weakening the shortcut behavior and thereby mitigating the backdoor.

Proposed Approach

Building on the need for adversarial generation that handles an unknown attack objective and a concentrated repair objective identified in Section 1, we propose an adversarial fine-tuning approach for backdoor mitigation in object detection that has two components. First, we reformulate the conventional maximisation-based adversarial example generation used in classification-oriented defences and introduce a complementary soft-branch minimisation alternative. Second, we redesign the outer fine-tuning objective to account for the multi-prediction structure of object detection models. The resulting fine-tuning objective is compatible with both adversarial generation strategies, and is summarised in Fig. 1. 4.1

Adversarial Example Generation

We consider two detection-aware adversarial generation strategies: classification-loss maximisation (CLM), which adapts the standard adversarial maximisation paradigm to matched detector predictions, and soft-branch minimisation (SBM), which softly selects between RMA- and ODA-oriented objectives. 4.1.1

Classification-Loss Maximisation

As discussed in Section 3, loss maximisation is a standard strategy in classification-based backdoor mitigation for generating surrogate backdoor data. However, in object detection, naively maximising 4

the full detection loss, LOD , is sub-optimal for two reasons: it unnecessarily perturbs localisation and it does not isolate the effect of δ to the predictions relevant to the target object i⋆ . To address this, we adopt a detection-aware formulation that focuses on LCLS and restricts the influence of δ to the relevant prediction set Ji⋆ (see Fig. 1 (a)). In Appendix C.4, we provide an ablation showing the impact of using the full detector loss LOD . Therefore, we define the CLM strategy as X 1 LCLS (j). (5) max ′ δ |Ji⋆ (x )| ′ j∈Ji⋆ (x )

′

Restricting the adversarial loss to Ji⋆ (x ) concentrates the optimisation on predictions matched to the target object, hence reducing interference and gradient noise from unrelated objects or background proposals. However, this strategy does not explicitly distinguish between the objectives of RMA and ODA. In particular, maximising LCLS is only an indirect surrogate for either inducing a dominant incorrect class or suppressing foreground detections. This motivates introducing an alternative objective that more directly reflects these two behaviours. In Appendix A.3, we provide a supporting score-level analysis. 4.1.2

Soft-Branch Minimisation

As an alternative, we introduce SBM (see Fig. 1 (b)). SBM accommodates an unknown attack objective by defining two optimisation branches (i.e., 4a and 4b in Fig. 1 (b)), corresponding to RMA and ODA, and combining them through a soft gate that favours the branch that is easier to satisfy. The intuition is that the resulting perturbation δ is biased toward the branch that better aligns with the detector’s underlying backdoor behaviour, without requiring the defender to know the attack objective. For each matched prediction j ∈ Ji⋆ (x′ ), we define three quantities from the class scores sj,c (x′ ): ′ sgt j = sj,yi⋆ (x ),

soth = max sj,c (x′ ), j c̸=yi⋆

smax = max sj,c (x′ ), j c

(6)

which denote the ground-truth class score, the highest non-target class score, and the maximum foreground score, respectively. Misclassification (RMA): To encourage a false class to exceed the detection threshold τ , we define LRMA (j) = softplus(τ − soth j ),

(7)

where softplus(t) = log(1 + e ) ensures non-zero gradients everywhere. However, minimising this term alone may spread score mass across several incorrect classes, much like the maximisation objective above. To encourage a single dominant incorrect class, we add an entropy-based concentration penalty over the normalised non-target scores s̄j,¬y : X s̄j,c log(s̄j,c + ζ), (8) LCON (j) = − c̸=yi⋆

where ζ > 0 is a small stabilising constant. Minimising LCON (j) encourages the non-target score distribution to concentrate on a single incorrect class. In Appendix A.5, we show that the proposed soft minimisation objective can be interpreted as an entropy-regularised branch selector between the RMA and ODA paths, while in Appendix A.4, we clarify the role of LCON . Disappearance (ODA): To suppress all foreground detections, we define LODA (j) = softplus(smax − τ ), j

(9)

which encourages all class scores below τ , making the target object less likely to survive score filtering. Joint objective: Since the attack objective is unknown, we combine both branches using a soft gate that prioritises the objective that is closer to satisfaction: gjRMA , gjODA = softmax [−LRMA (j), −LODA (j)]. The resulting per-prediction adversarial loss is LADV (j) = gjRMA [LRMA (j) + LCON (j)] + gjODA LODA (j), and the full soft minimisation objective, averaged over matched predictions, is X 1 min LADV (j). (10) ′ δ |Ji⋆ (x )| ′ j∈Ji⋆ (x )

We solve this optimization problem using K-step projected gradient descent with sign-gradient updates and projection. As in the maximisation formulation, restricting optimisation to Ji⋆ (x′ ) focuses the gradients on matched predictions. This soft gate provides a differentiable mixture of the RMA and ODA objectives via softmax-normalized negative losses. The auxiliary concentration term, LCON , is branch-specific and shapes the RMA path by encouraging a single dominant incorrect class. j 4.2

Adversarial Fine-Tuning

Given an adversarially perturbed image x′ produced by either method in Section 4.1, we fine-tune θ using the standard detection loss LOD together with a dual-objective defence loss computed directly on the target-matched predictions Ji⋆ (x′ ) (see Fig. 1 (c)). Because LOD aggregates over all candidate predictions, the repair signal at i⋆ is otherwise diluted by background and non-target contributions; restricting the defence loss to Ji⋆ (x′ ) concentrates this signal on the predictions that need to have their classification restored. This loss comprises two terms that explicitly counteract the score-level events defining the two attack modes: a recovery term that encourages the ground-truth class score to exceed τ , and a suppression term that drives competing class scores below τ . Recovery: We encourage the ground-truth class score to recover a margin above the detection threshold via LREC (j) = softplus(τ − sgt (11) j ). Minimising this term counters ODA and RMA induced backdoors by increasing the target object’s ground-truth score on the perturbed input, encouraging at least one matched prediction to survive score filtering (see Fig. 1 (c) 4). Suppression: We penalise high non-target activations through LSUP (j) = softplus(soth − τ ). j

(12)

Minimising this term prevents a degenerate solution in which the detector restores the groundtruth score but simultaneously assigns large scores to competing classes, which would otherwise lead to misclassification after thresholding. This safeguard is particularly relevant when δ induces RMA-aligned behaviour, as depicted by the ’δ causes ODA’ path in Fig. 1(c). We combine LOD with the defence loss LDEF (j) = LREC (j) + LSUP (j), over target-matched predictions as LFULL (θ) = LOD (x; θ) + LOD (x′ ; θ) +

λ |Ji⋆ (x′ )|

LDEF (j).

(13)

j∈Ji⋆ (x′ )

Restricting LDEF to Ji⋆ (x′ ) mirrors the target-matched restriction used during adversarial generation in Section 4.1, ensuring that both the inner and outer loops concentrate on the same set of predictions. 4.3

Optimisation Framework

Our defence alternates between (i) generating object-localised adversarial examples, using either Fig. 1 (a) or (b), and (ii) fine-tuning the compromised detector to restore correct classifications while suppressing spurious activations, see Fig. 1 (c). This yields a bilevel-style procedure in which an inner loop approximately solves for δ ⋆ and an outer loop updates θ. Target object selection: Since each input may contain multiple objects, adversarial generation requires selecting a target (bi⋆ , yi⋆ ) ∈ B. We consider two selection strategies: 1. Random selection (RS): sample i⋆ uniformly from {1, . . . , N }. 2. Filtered weighted selection (FWS): restrict candidates to objects that are correctly detected on the clean input (matched prediction IoU above the relevant threshold, with no high-scoring overlapping incorrect predictions), and then sample proportionally to their clean inference confidence. RS provides unbiased coverage across objects, whereas FWS prioritises objects that are already cleanly detected, hence avoiding adversarial updates that merely exploit pre-existing detection ambiguity. In Appendix A.7, we provide an expected-informativeness rationale for FWS. 6

Inner loop: Given a clean image x and target index i⋆ , we form x′ (δ) = Π[0,1] (x + M(bi⋆ ) ⊙ δ). Depending on the adversarial generation strategy, the inner problem either solves Equation 5 or 10 in the CLM or SBM cases respectively. Outer loop: Given the perturbed image x′ , we minimise LFULL (θ) with respect to θ. This jointly optimises the standard detection objective on both x and x′ while applying the targeted recovery and suppression terms to the matched predictions Ji⋆ (x′ ), thereby steering the detector toward correct classification of the target object under adversarial perturbation.

Evaluation

In this section, we evaluate the proposed method against standard fine-tuning and pruning, adapted from image-classification backdoor mitigation [13]. Following the evaluation methodology of BadDet+ [7], we conduct a comprehensive study across diverse settings, covering four datasets and three detector architectures. We make our benchmarking framework publicly available on GitHub1 . We evaluate untargeted ODA and RMA. For ODA we use BadDet+, the only attack reliably inducing object disappearance; for RMA we use both BadDet and BadDet+, which achieve high ASR but differ in retaining the original-class detection alongside the target. Experiments are conducted on Pascal VOC [9] and MTSD [8] with DINO [22], FCOS [17], and YOLOv5m6 [11]; COCO ablations [12] appear in Appendix D.1, and real-world transfer of MTSD-trained mitigations to PTSD [5] in Appendix D.2. For MTSD, we use traffic-sign meta-class labels and exclude other-sign to alleviate class imbalance. Triggers are placed randomly for MTSD/PTSD and centred in bounding boxes for Pascal VOC, where high object density makes random placement impractical. We use a blue trigger for PTSD compatibility, with alternatives explored in Appendix D.4. Our main experiments use ∼5% clean training data (500 samples for Pascal VOC, 100 for MTSD) across five random splits, with 2.5% and 1.25% additionally tested. We compare CLM and SBM against FT, FP, and FP+FT [13], and FT-SAM [24], evaluating RS and FWS strategies (Section 4.3) with and without the defence terms. Full implementation details, hyperparameters, and additional results are in Appendices B, C, and D. We evaluate each method using three metrics: ratio mAP (RmAP), post-mitigation clean mAP divided by the pre-mitigation value, as well as ASR, and TDR reported using IoU 0.5. An RMA attack succeeds when the target class is positively detected; an ODA attack succeeds when no positive detection of the original class occurs. TDR is therefore omitted for ODA since ASR = 1 − TDR. Main Results: Figure 2 summarises performance with 5% clean training data, with Appendix Tables 3 and 4 reporting the median performance. Compared to FT, FP, FP+FT, and FT-SAM, the proposed CLM and SBM reduce ASR and raise TDR, at a modest cost to RmAP. The strongest configuration, SBM-FWS with LDEF , reaches median (ASR, TDR) of (0.163, 0.676) on VOC and (0.037, 0.861) on MTSD, against the best baselines at (0.945, 0.156) and (0.653, 0.427) respectively, at a RmAP cost of at most 0.10. Adding LDEF reduces median ASR by 67% for SBM-FWS and 29% for CLM-FWS on VOC, and by 80% and 33% respectively on MTSD, while reducing TDR variance with some additional RmAP cost. This is consistent with the motivation in Section 4.2: applying the repair signal directly to Ji⋆ (x′ ) avoids the dilution that target-object gradients undergo within the aggregated LOD . Similarly, FWS lowers ASR relative to RS. For SBM with LDEF , median ASR drops by a further 21% on VOC and 76% on MTSD, while the effect is smaller and mixed for CLM. This comes at the cost of a small reduction in RmAP and occasionally TDR, reflecting that restricting candidates to cleanly-detected objects reduces the probability that the inner loop exploits pre-existing detection ambiguity rather than backdoor behaviour (Appendix A.7). Comparing the FWS variants directly, SBM-FWS improves median ASR and TDR by roughly 36% and 20% relative to CLM-FWS on VOC, while on MTSD the two achieve similar ASR and TDR. This reflects the design distinction in Section 4.1: CLM maximises LCLS over Ji⋆ (x′ ), an indirect surrogate for the score-threshold events characterising RMA and ODA, whereas SBM optimises LRMA and LODA directly via a soft gate favouring the branch closer to satisfaction (Appendix A.3). Attack Objective: Figure 3 reports separate results for ODA and RMA, with Appendix Tables 5, 6 reporting the median performance. The proposed defense terms are particularly beneficial against RMA, reducing median ASR by roughly 80% on VOC (0.649 → 0.126) and 94% on MTSD 1 The code is included with the submission and will be released upon acceptance.

(a) Pascal VOC

(b) MTSD

Figure 2: Performance comparison of four adversarial fine-tuning variants (SBM-FWS, SBM-RS, CLM-FWS, CLM-RS) alongside four baseline defenses (FT, FP, FP+FT, FT-SAM).

(a) Pascal VOC RMA

(b) MTSD RMA

VOC

(d) MTSD ODA

Figure 3: Performance comparison of four adversarial fine-tuning variants (SBM-FWS, SBM-RS, CLM-FWS, CLM-RS) seperated by attack objective. (0.239 → 0.014) for SBM-FWS, while also decreasing variability in both ASR and TDR. The ODA results are more nuanced. On VOC, adding LDEF barely changes ODA ASR for SBM-FWS (0.364 → 0.379), while on MTSD it raises ODA ASR (0.138 → 0.251). Because the defender does not know the attack objective, the relevant criterion is robustness across both attack objectives, under which SBM-FWS with the proposed defense terms is the most reliable choice. If ODA were known in advance, the preferred configuration would be dataset-dependent: SBM-RS on the more complex Pascal VOC, and CLM-FWS on MTSD. Data Availability: Figure 4 illustrates the impact of decreasing the amount of clean data for SBMFWS and CLM-FWS, with Appendix Tables 7, 8 reporting the median performance. Even with the clean-data budget halved or quartered, the proposed defense remains highly effective: as the budget shrinks from 5% to 1.25%, median ASR for SBM with LDEF rises only from 0.163 to 0.269 on VOC and from 0.037 to 0.108 on MTSD. Notably, these results are still below the strongest baseline ASR (0.945 on VOC, 0.653 on MTSD) measured at the full 5% budget. Variance in TDR and ASR does grow as clean data shrinks; however, SBM with LDEF is more insulated from these median and variance changes than CLM. Key Findings: Taken together, the proposed adversarial fine-tuning framework delivers: (i) median ASR reductions of 83% (VOC) and 94% (MTSD) over the strongest baseline for SBM-FWS with LDEF , with TDR gains of 0.52 and 0.43 respectively at a RmAP cost of at most 0.10; (ii) LDEF 8

(a) Pascal VOC

(b) MTSD

Figure 4: Performance comparison of SBM-FWS and CLM-FWS separated by attack objective. provides a major contribution, reducing median ASR by 67–80% for SBM-FWS and rising to 80– 94% on the RMA attack subset; (iii) further ASR reductions from FWS over RS, most strikingly for SBM with LDEF (21% on VOC, 76% on MTSD), with a smaller and mixed effect for CLM; and (iv) SBM-FWS with LDEF is more insulated from data scarcity than CLM, still outperforming the strongest baseline (measured at 5% clean data) when fine-tuned on as little as 1.25% clean data.

Conclusion and Future Work

We proposed a detection-aware adversarial fine-tuning method for mitigating object-level backdoors under a practical repair setting, where the defender has access only to a compromised detector and a small clean dataset. Our approach addresses two structural mismatches between classification and detection settings: (i) adversarial generation must handle an unknown attack objective, and (ii) the repair signal must concentrate on the targeted object. We address the first by restricting adversarial generation to target-matched predictions through two strategies, CLM and SBM, the latter softly combining RMA- and ODA-aligned objectives to account for the attack objective being unknown. We address the second through a dual-objective defence loss applied directly to the target-matched predictions during fine-tuning. Across CNN- and Transformer-based detectors and multiple datasets, this combination improves the ASR-TDR trade-off relative to classification-oriented baselines while maintaining a favorable trade-off with clean detection performance. Because the defender does not know whether the compromised model is affected by RMA or ODA, an effective mitigation strategy must perform reliably across both. Our results indicate that SBM with the proposed defence terms, particularly when paired with filtered weighted selection of target objects (FWS, Section 4.3), provides the most robust overall trade-off. This reflects the closer alignment of SBM’s objectives with the events characterising RMA and ODA, and is also more stable than CLM in low-data regimes. Nevertheless, fully recovering correct detections for trigger-bearing objects remains challenging, consistent with limitations observed for fine-tuning-based defences in the classification setting [6], with TDR retaining a long lower tail in some settings. Building on these findings, two directions for future work seem particularly promising. The first is developing mechanisms that more directly improve TDR while maintaining low ASR and competitive RmAP. Pruning-based strategies are a natural candidate; however, existing pruning defences are largely designed for CNN architectures, and extending them to Transformer-based detectors, in particular, how to prune or regularise attention mechanisms effectively, remains open. The second is broadening mitigation beyond object-centric attacks. Object-generation attacks [2] manipulate background regions to induce phantom detections, a behaviour that is not directly addressed by our object-specific framework. Developing unified mitigation strategies that handle both localised triggers and background-induced false detections is an important next step. 9

References [1] Shuwen Chai and Jinghui Chen. One-shot neural backdoor erasing via adversarial weight masking. Advances in Neural Information Processing Systems, 35:22285–22299, 2022. [2] Shih-Han Chan, Yinpeng Dong, Jun Zhu, Xiaolu Zhang, and Jun Zhou. Baddet: Backdoor attacks on object detection. In European conference on computer vision, pages 396–412. Springer, 2022. [3] Siyuan Cheng, Guangyu Shen, Guanhong Tao, Kaiyuan Zhang, Zhuo Zhang, Shengwei An, Xiangzhe Xu, Yingqi Li, Shiqing Ma, and Xiangyu Zhang. Odscan: Backdoor scanning for object detection models. In 2024 IEEE Symposium on Security and Privacy (SP), pages 1703–1721. IEEE, 2024. [4] Yize Cheng, Wenbin Hu, and Minhao Cheng. Attacking by aligning: Clean-label backdoor attacks on object detection. arXiv preprint arXiv:2307.10487, 2023. [5] Bao Gia Doan, Dang Quang Nguyen, Callum Lindquist, Paul Montague, Tamas Abraham, Olivier De Vel, Seyit Camtepe, Salil S Kanhere, Ehsan Abbasnejad, and Damith C Ranasinghe. On the credibility of backdoor attacks against object detectors in the physical world. In 2024 Annual Computer Security Applications Conference (ACSAC), pages 940–956. IEEE, 2024. [6] Kealan Dunnett, Reza Arablouei, Dimity Miller, Volkan Dedeoglu, and Raja Jurdak. Countering backdoor attacks in image recognition: A survey and evaluation of mitigation strategies. arXiv preprint arXiv:2411.11200, 2024. [7] Kealan Dunnett, Reza Arablouei, Dimity Miller, Volkan Dedeoglu, and Raja Jurdak. Baddet+: Robust backdoor attacks for object detection. arXiv preprint arXiv:2601.21066, 2026. [8] Christian Ertler, Jerneja Mislej, Tobias Ollmann, Lorenzo Porzi, Gerhard Neuhold, and Yubin Kuang. The mapillary traffic sign dataset for detection and classification on a global scale. In European conference on computer vision, pages 68–84. Springer, 2020. [9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88 (2):303–338, 2010. [10] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733, 2017. [11] Glenn Jocher. Ultralytics YOLOv5. https://github.com/ultralytics/yolov5, 2020. Version 7.0, AGPL-3.0 License. [12] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. [13] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International symposium on research in attacks, intrusions, and defenses, pages 273–294. Springer, 2018. [14] Ximeng Liu, Lehui Xie, Yaopeng Wang, Jian Zou, Jinbo Xiong, Zuobin Ying, and Athanasios V Vasilakos. Privacy and security issues in deep learning: A survey. IEEE Access, 9:4566–4593, 2020. [15] Chengxiao Luo, Yiming Li, Yong Jiang, and Shu-Tao Xia. Untargeted backdoor attack against object detection. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. [16] Guangyu Shen, Siyuan Cheng, Guanhong Tao, Kaiyuan Zhang, Yingqi Liu, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Django: Detecting trojans in object detection models via gaussian focus calibration. Advances in Neural Information Processing Systems, 36:51253– 51272, 2023. 10

[17] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627–9636, 2019. [18] Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. Advances in Neural Information Processing Systems, 36:25876–25909, 2023. [19] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. Advances in Neural Information Processing Systems, 35:10546–10559, 2022. [20] Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. arXiv preprint arXiv:2110.03735, 2021. [21] Hangtao Zhang, Yichen Wang, Shihui Yan, Chenyu Zhu, Ziqi Zhou, Linshan Hou, Shengshan Hu, Minghui Li, Yanjun Zhang, and Leo Yu Zhang. Test-time backdoor detection for object detection models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24377–24386, 2025. [22] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022. [23] Xianda Zhang, Siyuan Liang, and Chengyang Li. Towards robust object detection: Identifying and removing backdoors via module inconsistency analysis. In International Conference on Pattern Recognition, pages 343–358. Springer, 2024. [24] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4466–4477, 2023.

Theoretical Analysis

We provide supporting analysis for the design choices in Sections 4.1–4.3. The goal is not to establish a full end-to-end robustness theorem for modern object detectors, which would require strong architecture- and post-processing-specific assumptions. Instead, we show that the proposed objectives are aligned with the score-level events that characterize object-level backdoor attacks, and that the outer defence terms induce a meaningful repaired margin against those events. Throughout, we analyse the detector scores sj (x′ ) ∈ [0, 1]C associated with predictions matched to the target object. This abstraction is agnostic to whether the detector is anchor-based, anchor-free, or set-based, and to whether class scores arise from sigmoid, softmax, or detector-specific scoring heads. We assume that the training-time assignment mechanism yields a nonempty set Ji⋆ (x′ ) of predictions matched to the target object i⋆ , as is standard for common dense and set-based assigners. If an implementation can produce an empty set, the definitions below can be interpreted using the usual practical fallback to the clean-image assignment. Unless stated otherwise, gradient statements are conditioned on the current matched set used to evaluate the loss and are taken away from score ties introduced by max operators; at ties, the same conclusions hold in the usual subgradient sense. A.1

Score-Level Margins and Attack Events

For each matched prediction j ∈ Ji⋆ (x′ ), recall that ′ sgt j = sj,yi⋆ (x ),

soth = max sj,c (x′ ), j c̸=yi⋆

smax = max sj,c (x′ ). j c

(14)

We define the corresponding score margins as mRMA = soth − τ, j j

mODA = τ − smax , j j

mREC = sgt j j − τ,

mSUP = τ − soth j j .

(15)

These margins encode the score-threshold conditions associated with the attack and defence objectives: 11

• mRMA ≥ 0 indicates that at least one non-ground-truth class score is at or above the detection j threshold τ ; • mODA ≥ 0 indicates that all foreground class scores are at most τ ; j REC • mj ≥ 0 indicates that the ground-truth class score is at or above τ ; • mSUP ≥ 0 indicates that all non-ground-truth class scores are at most τ . j Because the defender does not know the attacker’s target class in an RMA, we use the strongest non-ground-truth score soth as a defender-agnostic surrogate. This condition upper-bounds targeted j RMA behaviour in the sense that, if any attacker-chosen target class crosses the threshold, then soth j also crosses the threshold. The proposed inner and outer losses are smooth surrogates of these margin conditions: LRMA (j) = softplus(−mRMA ), j

LODA (j) = softplus(−mODA ), j

(16)

LREC (j) = softplus(−mREC ), j

LSUP (j) = softplus(−mSUP ). j

(17)

Proposition A1 (order consistency of the score surrogates). For each of the four losses above, minimising the loss is equivalent to maximising the corresponding score margin in the sense that the ordering of candidate perturbations is preserved. In particular, for any two perturbed inputs x′1 , x′2 and any matched prediction j, mRMA (x′1 ) > mRMA (x′2 ) ⇐⇒ LRMA (j; x′1 ) < LRMA (j; x′2 ), j j

(18)

mODA (x′1 ) > mODA (x′2 ) j j

(19)

⇐⇒

LODA (j; x′1 ) < LODA (j; x′2 ),

and the same equivalence holds for LREC and LSUP with their corresponding margins. Proof. The function softplus(t) = log(1 + et ) is strictly increasing in t. Hence softplus(−m) is strictly decreasing in m, which gives the stated ordering equivalences. □ Proposition A1 shows that the proposed losses preserve the score-threshold logic of the corresponding attack and defence events. They are monotone surrogates of the relevant margins, differentiable away from score ties and subdifferentiable at ties. A.2

Gradient-Alignment Rationale for Target-Matched Restriction

A central design choice in both classification-loss maximisation and soft branch minimisation is to restrict adversarial generation to predictions matched to the target object. This restriction can be understood through a gradient-alignment argument. Let Ltar (δ) denote the average loss over the target-matched set Ji⋆ (x′ (δ)), and let Lnui (δ) collect the remaining prediction terms that would appear in an unrestricted image-level objective. Writing Lall (δ) = Ltar (δ) + Lnui (δ),

(20)

the corresponding unrestricted gradient direction can be decomposed as gall = gtar + gnui ,

gtar = ∇δ Ltar ,

gnui = ∇δ Lnui .

Proposition A2 (target restriction reduces gradient dilution). and suppose there exist ρ ∈ [0, 1] and β ≥ 0 such that |⟨gtar , gnui ⟩| ≤ ρ∥gtar ∥2 ,

(21)

Assume gtar ̸= 0 and gall ̸= 0,

∥gnui ∥ ≤ β∥gtar ∥.

(22)

Then the cosine alignment between the unrestricted gradient and the target gradient satisfies cos(gtar , gall ) ≥

1−ρ . 1+β

(23)

In contrast, using only Ltar yields perfect alignment with the target objective, since cos(gtar , gtar ) = 1. 12

Proof.

By definition, ⟨gtar , gtar + gnui ⟩ . ∥gtar ∥ ∥gtar + gnui ∥

cos(gtar , gall ) =

(24)

Using the assumed bound on the inner product, ⟨gtar , gtar + gnui ⟩ = ∥gtar ∥2 + ⟨gtar , gnui ⟩ ≥ (1 − ρ)∥gtar ∥2 .

(25)

Using the triangle inequality and the norm bound on gnui , ∥gtar + gnui ∥ ≤ ∥gtar ∥ + ∥gnui ∥ ≤ (1 + β)∥gtar ∥.

(26)

Combining these two inequalities gives cos(gtar , gall ) ≥

(1 − ρ)∥gtar ∥2 1−ρ = . (1 + β)∥gtar ∥2 1+β

(27) □

Proposition A2 formalises the intuition that unrelated objects and background proposals can dilute the target-object gradient unless their gradients are sufficiently aligned with the target objective. Restricting optimisation to Ji⋆ (x′ (δ)) therefore improves the alignment of the inner-loop update with the intended target-object objective. A.3

Partial Alignment of Classification-Loss Maximisation

The classification-loss maximisation baseline in Section 4.1 provides a useful detection-adapted surrogate, but it does not directly optimise the score-threshold events that define RMA and ODA. For detectors with independent sigmoid-based classification heads, the classification loss for a matched prediction decomposes into class-wise terms, possibly with focal, label-smoothing, or implementation-specific weights. The following BCE form illustrates the basic mechanism: X LCLS (j) = − log σ(zj,yi⋆ ) − log(1 − σ(zj,c )). (28) c̸=yi⋆

Its gradient with respect to the input is X ∇x LCLS (j) = σ(zj,yi⋆ ) − 1 ∇x zj,yi⋆ + σ(zj,c )∇x zj,c .

(29)

c̸=yi⋆

Thus, maximising LCLS (j) tends to decrease the ground-truth logit while increasing the class-wise penalties associated with non-ground-truth logits. This makes classification-loss maximisation an indirect surrogate for the thresholded score events that define RMA and ODA. By contrast, the score-level events associated with RMA and ODA are governed by extremal class scores: soth = max sj,c , smax = max sj,c . (30) j j c

c̸=yi⋆

Away from ties, the corresponding gradients depend only on the class attaining the relevant maximum: ∇x soth = ∇x sj,c⋆ , j

c⋆ ∈ arg max sj,c , c̸=yi⋆

∇x smax = ∇x sj,c̃⋆ , j

c̃⋆ ∈ arg max sj,c . c

(31)

At score ties, the same reasoning applies in the subgradient sense. Proposition A3 (partial misalignment of class-loss maximisation). For additive multi-label classification heads, the update direction induced by maximising LCLS (j) generally need not coincide with a direction tailored to either the RMA score-threshold event or the ODA score-threshold event. Alignment occurs only under restrictive local conditions, for example when the relevant class-gradient contributions are collinear and have compatible signs. 13

Proof. We prove the claim by exhibiting local configurations in which the classification-loss ascent direction is not aligned with the natural score-event directions for RMA and ODA. Consider the BCE illustration above. For a matched prediction j, the input-gradient of the classification loss is X ∇x LCLS (j) = σ(zj,y ) − 1 ∇x zj,y + σ(zj,c )∇x zj,c , (32) c̸=y

where we write y = y for brevity. Thus, the classification-loss ascent direction combines the ground-truth logit gradient with a weighted sum of all non-ground-truth class-gradient contributions. i⋆

For RMA, the relevant score-threshold event is controlled by soth = max sj,c . j

(33)

c̸=y

Away from ties, let c⋆ = arg maxc̸=y sj,c . A direction tailored to increasing the RMA margin is therefore aligned with ∇x soth = ∇x sj,c⋆ . (34) j For sigmoid scores, ∇x sj,c⋆ = σ ′ (zj,c⋆ )∇x zj,c⋆ , so this direction is collinear with ∇x zj,c⋆ . Now consider a local configuration with two non-ground-truth classes c1 , c2 , where c1 = c⋆ is the strongest competing class, and suppose ∇x zj,c1 = e1 ,

∇x zj,c2 = e2 ,

∇x zj,y = 0,

(35)

with e1 and e2 linearly independent. Since σ(zj,c1 ) > 0 and σ(zj,c2 ) > 0, the classification-loss ascent direction contains the component σ(zj,c1 )e1 + σ(zj,c2 )e2 ,

(36)

which is not collinear with e1 . However, the RMA-tailored direction is collinear with ∇x soth j , and hence with e1 . Therefore, the classification-loss ascent direction need not coincide with the RMA-oriented direction. For ODA, the relevant score-threshold event is controlled by smax = max sj,c . j c

(37)

Away from ties, let c̃⋆ = arg maxc sj,c . A direction tailored to increasing the ODA margin mODA = j τ − smax is aligned with j −∇x smax = −∇x sj,c̃⋆ . (38) j Suppose, for example, that the ground-truth class is currently the maximum, so c̃⋆ = y. Let ∇x zj,y = e1 ,

∇x zj,c1 = e2 ,

(39)

with e1 and e2 linearly independent. The ODA-tailored direction is collinear with −e1 . By contrast, the classification-loss ascent direction contains both σ(zj,y ) − 1 e1 and σ(zj,c1 )e2 . (40) Since σ(zj,c1 ) > 0, this direction is not collinear with −e1 unless the non-ground-truth contribution is absent or collinear with e1 . Thus, classification-loss maximisation is not generally aligned with the ODA-oriented direction either. These examples show that alignment between classification-loss maximisation and the RMA or ODA score-threshold directions requires special local structure, such as collinearity of the relevant classgradient contributions or negligible gradients from non-controlling classes. In general, classificationloss maximisation is therefore only an indirect surrogate for RMA- and ODA-oriented adversarial generation. □ The same qualitative issue can arise for softmax-based class heads in set-based detectors: moving probability mass away from the correct class is not equivalent to directly controlling the thresholded extremal score events that define RMA and ODA. 14

A.4

Role of the Concentration Penalty

The concentration term LCON (j) = −

s̄j,c log(s̄j,c + ζ)

(41)

c̸=yi⋆

is designed to discourage the RMA branch from distributing non-target score mass diffusely across many incorrect classes. When the total non-target score is positive, we define the normalised non-target scores as sj,c s̄j,c = P , c ̸= yi⋆ . ′ ′ c ̸=yi⋆ sj,c

(42)

Ignoring the small stabilising constant ζ for the moment, LCON (j) is the Shannon entropy of the normalised non-target score vector s̄j,¬y . Minimising this entropy encourages the non-target score mass to concentrate on a single incorrect class rather than spreading across multiple classes. This is consistent with an RMA-style misclassification event, where a dominant incorrect class should exceed the detection threshold. If all non-target scores are zero, we set LCON (j) = 0 by convention, since there is then no non-target mass to concentrate. Proposition A4 (entropy minimisation encourages a dominant false class). Let s̄j,¬y ∈ ∆C−2 denote the normalised non-target score vector. Then X 0≤− s̄j,c log s̄j,c ≤ log(C − 1), (43) c̸=yi⋆

with the lower bound attained if and only if all mass is concentrated on a single non-target class, and the upper bound attained if and only if the non-target distribution is uniform. Proof. This is the standard extremal property of Shannon entropy on the probability simplex: entropy is minimised at the vertices of the simplex and maximised by the uniform distribution. □ Thus, when the RMA branch is active, minimising LCON encourages the non-target score distribution to concentrate on a single incorrect class rather than spreading diffusely across multiple classes. This promotes an RMA-style misclassification pattern, where one incorrect class becomes dominant, which generic classification-loss maximisation does not explicitly enforce. A.5

Soft Minimisation as Entropy-Regularised Branch Selection

We now formalise the role of the soft gate in the soft minimisation objective. For a matched prediction j, define aj = LRMA (j), bj = LODA (j), cj = LCON (j). (44) The branch weights are computed as gjRMA , gjODA = softmax(−aj , −bj ).

(45)

Thus, the gate assigns greater weight to the branch with the smaller loss, corresponding to the branch whose score-level condition is closer to being satisfied. Proposition A5 (exact decomposition of the gated objective). For each matched prediction j, the gated RMA–ODA objective satisfies gjRMA aj + gjODA bj = − log e−aj + e−bj + H(gj ), (46) where

H(gj ) = −gjRMA log gjRMA − gjODA log gjODA ∈ [0, log 2] is the entropy of the two-way gate. Consequently, since

we have

(47)

LADV (j) = gjRMA (aj + cj ) + gjODA bj ,

(48)

LADV (j) = − log e−aj + e−bj + H(gj ) + gjRMA cj .

(49)

Proof.

Let Zj = e−aj + e−bj .

(50)

By the definition of the softmax gate, gjRMA =

e−aj , Zj

e−bj . Zj

(51)

− log gjODA = bj + log Zj .

(52)

gjODA =

Therefore, − log gjRMA = aj + log Zj ,

Multiplying these identities by the corresponding gate weights and summing gives H(gj ) = −gjRMA log gjRMA − gjODA log gjODA = gjRMA aj + gjODA bj + log Zj .

(53)

gjRMA aj + gjODA bj = − log Zj + H(gj ) = − log e−aj + e−bj + H(gj ).

(54)

Rearranging yields

Substituting this identity into LADV (j) = gjRMA (aj + cj ) + gjODA bj

(55)

gives the stated decomposition of LADV (j).

□

Corollary A5.1 (soft lower envelope). For every matched prediction j, − log e−aj + e−bj ≤ gjRMA aj + gjODA bj ≤ − log e−aj + e−bj + log 2.

(56)

Moreover, as |aj − bj | → ∞, the gate entropy satisfies H(gj ) → 0, so the gated average approaches the log-sum-exp soft minimum. In the same limit, this soft minimum approaches min{aj , bj }. This corollary clarifies the role of the gate. The branch-selection component is not merely a heuristic average of the RMA and ODA losses; it is an entropy-regularised smooth selector that assigns greater weight to the branch closer to satisfaction. When one branch is clearly easier to satisfy, the gate becomes nearly deterministic. The additional term gjRMA cj applies the concentration penalty in proportion to the activation of the RMA branch. A.6

Repaired Margin Induced by the Outer Defence Terms

The outer loss augments the standard detection loss with LREC (j) = softplus(τ − sgt j ),

LSUP (j) = softplus(soth − τ ). j

(57)

These terms target the score inequalities that counter ODA and RMA at the matched-prediction level: the recovery term encourages the ground-truth score to exceed the detection threshold, while the suppression term encourages all non-ground-truth scores to remain below it. We define the repaired score margin as ′ oth ′ mj (x′ ) = min{sgt j (x ) − τ, τ − sj (x )}.

Proposition A6 (positive repaired margin excludes both score-level failure modes). 0 for some matched prediction j ∈ Ji⋆ (x′ ), then ′ sgt j (x ) > τ

and

′ soth j (x ) < τ.

(58) If mj (x′ ) > (59)

Therefore, at the pre-NMS score-filtering stage, this prediction survives score thresholding under the ground-truth class and has no threshold-crossing non-ground-truth class score. 16

Proof.

The result follows directly from the definition ′ oth ′ mj (x′ ) = min{sgt j (x ) − τ, τ − sj (x )}.

If mj (x′ ) > 0, then both arguments of the minimum are positive. Hence ′ sgt j (x ) − τ > 0

and

′ τ − soth j (x ) > 0,

′ oth ′ which gives sgt j (x ) > τ and sj (x ) < τ .

□

Proposition A6 is intentionally modest but useful. It shows that the added defence terms encourage precisely the score margin that the generic detection loss does not explicitly enforce: the recovery term pushes the correct class above threshold, while the suppression term pushes competing classes below threshold. Connecting this score-level statement to the final detector output requires a local stability assumption on localisation, matching, and post-processing. Corollary A6.1 (local score stability under fixed-match/fixed-box conditions). Suppose that, in a neighbourhood U of x′ , the following conditions hold for some matched prediction j: 1. the prediction remains matched to the target object, and its box remains compatible with the IoU and NMS conditions used at inference; and oth 2. the score functions sgt j (·) and sj (·) are L-Lipschitz on U with respect to the perturbation norm. If mj (x′ ) = γ > 0, then for any additional perturbation η supported inside the target box and satisfying ∥η∥ < γ/L, the repaired score inequalities are preserved: ′ sgt j (x + η) > τ,

Proof.

′ soth j (x + η) < τ.

(60)

Since mj (x′ ) = γ, we have ′ sgt j (x ) ≥ τ + γ,

′ soth j (x ) ≤ τ − γ.

By Lipschitz continuity, gt ′ ′ sgt j (x + η) ≥ sj (x ) − L∥η∥ > τ + γ − γ = τ,

(61)

′ oth ′ soth j (x + η) ≤ sj (x ) + L∥η∥ < τ − γ + γ = τ. Thus, both repaired score inequalities are preserved.

(62) □

and similarly,

This corollary gives a local stability interpretation of the outer loss: larger repaired margins provide greater tolerance to additional perturbations at the score level, conditional on localisation, matching, and post-processing remaining stable. A.7

Expected-Informativeness Justification for Filtered Weighted Selection

Filtered weighted selection (FWS) allocates the adversarial-generation budget to target objects that are already cleanly detected. This helps avoid cases where the inner loop primarily exploits pre-existing detection ambiguity rather than backdoor-related behaviour. Let Iclean denote the set of candidate objects that pass the clean-detection filter. For each i ∈ Iclean , let ci > 0 denote its clean confidence, and let qi ∈ [0, 1] denote the probability that selecting object i yields an informative adversarial example. Here, “informative” refers to cases where the selected object provides stable matching, unambiguous supervision, or a repair gradient more likely to target backdoor behaviour than ordinary detection failure. The following argument justifies the confidence-weighting step conditional on the filtered candidate set. Under uniform selection over Iclean , X 1 qi . (63) Eunif [q] = |Iclean | i∈Iclean

Under confidence-weighted selection with ci

(64)

P ci qi pi qi = Pi∈Iclean . i∈Iclean ci

(65)

pi = P

k∈Iclean ck

the expected informativeness becomes X

EFWS [q] =

i∈Iclean

Proposition A7 (confidence weighting improves expected informativeness under positive correlation). If the covariance between clean confidence and informativeness is nonnegative over Iclean , then EFWS [q] ≥ Eunif [q]. (66) Proof. Let c̄ and q̄ denote the uniform averages of {ci } and {qi } over Iclean . Since ci > 0 for all i ∈ Iclean , we have c̄ > 0. Then EFWS [q] =

Eunif [cq] Covunif (c, q) + c̄ q̄ Covunif (c, q) = = q̄ + . Eunif [c] c̄ c̄

(67)

Therefore, if Covunif (c, q) ≥ 0, then EFWS [q] ≥ q̄ = Eunif [q].

(68) □

This proposition captures the role of the weighting step within the filtered candidate pool. If objects with higher clean confidence are more likely to yield stable and informative target-object supervision, then confidence-weighted selection improves the expected quality of the selected inner-loop target. Relative to unfiltered random selection, FWS also excludes objects that fail the clean-detection filter by construction. Proposition A7 isolates the additional effect of confidence weighting among the retained candidates. A.8

Summary of Analytical Findings

The analysis above supports four main points. First, the proposed inner and outer losses are monotone-aligned with the score-threshold margins that describe object-level backdoor attack and repair events. Second, restricting optimisation to Ji⋆ (x′ (δ)) reduces gradient dilution from unrelated detections under mild alignment assumptions. Third, the soft branch minimisation objective can be interpreted as an entropy-regularised smooth selector between the RMA and ODA branches, while the concentration term encourages a dominant incorrect class when the RMA branch is active. Fourth, the outer defence terms induce a repaired score margin that excludes both score-level failure modes and provides local score stability once such a margin is achieved. These results should be interpreted as a mechanism-level justification rather than a global robustness guarantee for modern object detectors. Final detection outcomes also depend on localisation, matching, and post-processing, which are architecture-specific and highly nonlinear. The role of this analysis is therefore to justify the design of the proposed method and to complement, rather than replace, empirical evaluation.

Hyperparameter Selection

In the following section we provide experiments that show how sensitive the proposed methods are to variations in the chosen hyperparameters. For these experiments, we use SBM-FWS. In the main text, as well as the additional experiments provided in the subsequent sections C and D, the hyperparameters used for training are defined in Table 1. In all cases, τ is set to the confidence threshold used by each model. 18

Table 1: Model Configuration Summary. Model

Hyperparameters λ β ϵ IoU

YOLOv5 DINO FCOS

0.01 0.1 0.1

B.1

1.0 1.0 0.5

0.0314 0.0314 0.0314

0.60 0.60 0.60

Design Choices Train BB θUP Match. Strat. No No No

No Yes No

Yes No Yes

LR 0.000025 0.000091 0.001

Training Parameters Epochs Batch Size Adv. Steps 30 30 30

16 16 16

30 30 30

Step Size 0.0078 0.0078 0.0078

Lambda (λ)

In Section 4.2, we proposed adding the term LREC and LSUP when fine-tuning the model on adversarial examples. To integrate this penalty with the normal detector losses LOD , λ is used to balance its contribution. For each model architecture, we evaluate the impact that λ has on performance across each model architecture and for each attack objective. We show the results in Fig. 5. In general, we find that the value of λ has smooth performance changes as its value increases and decreases. Moreover, we find that as λ increases (i.e., the strength of the proposed defence penalty increases), the ASR-RmAP is greater than at lower value of λ, with this specifically being true for FCOS. For FCOS, BadDet+ ODA, the opposite is true, as more RmAP is traded off, ASR increases. This is largely due to large reductions in main task performances, influencing the model’s ability to detect objects. For DINO and FCOS, λ = 0.1 is optimal, while for YOLOv5, λ = 0.01 is optimal.

Figure 5: Performance of each model when λ is varied. High RmAP and TDR are better, while low ASR is preferred.

B.2

Beta (β)

In Section 4.1.2, we use a softplus(·) as part of the proposed term LREC and LSUP , as well as in Section 4.2 as part of LODA and LRMA . As a result, β is used in each definition to control the shape of the penalty in each case. For each model architecture, we evaluate the impact of β on performance for each attack objective. We show the results in Fig. 6. In general, we find that the value of β has little impact on the performance of each model architecture. Similar to λ, we observe an ASR-RmAP trade-off when β decreases and the softplus penalty becomes less piecewise-linear near the threshold τ . For DINO and YOLOv5, we use the default value of β = 1 and use β = 0.5 in the case of FCOS. B.3

Epsilon (ϵ)

As part of both the proposed adversarial objectives proposed in Section 4.1, an upper bound is placed on the perturbation δ. As a result, the value of ϵ has a critical impact on the possible values of δ that are possible and thus the subsequent effectiveness of subsequent minimisation. We show the results in Fig. 7. For DINO, we find that the value of ϵ is stable under both small and large values. In contrast, 19

Figure 6: Performance of each model when β is varied. High RmAP and TDR are better, while low ASR is preferred. the FCOS and YOLO results are much more varied. In general, smaller values of ϵ provide limited ASR reduction. Performance improves as ϵ increases to 8, but degrades again when ϵ is increased further to 32. As a result, we find that ϵ = 8 offers the best trade-off for all three architectures, and that smaller and larger perturbations either do not allow for solutions of δ that facilitate classification errors that align with the attack, or expand the set of possible solutions to be so large that the number of useful solutions (i.e., solutions that provide information about the attack) is diluted by adversarial examples that are unrelated to the attack.

Figure 7: Performance of each model when ϵ is varied. High RmAP and TDR are better, while low ASR is preferred. B.4

IoU Value

In Section 4.3, we propose the Filtered weighted selection (FWS) strategy. As part of this strategy, an IoU threshold is used to select objects that have positive predictions. As a result, this threshold has an impact on how strict this selection is, and subsequently has the potential to impact performance. We show the results in Fig. 8. In general, we find that IoU values above 0.5, the minimum value used 20

by measures like mAP, cause small increases in ASR performance across most model architectures. While TDR shows slight increase for FCOS and YOLOv5, DINO shows some decrease in TDR as the IoU threshold increases. As a result, we chose a value of 0.6 in all experiments.

Figure 8: Performance of each model when the IoU threshold used by FWS is varied. High RmAP and TDR are better, while low ASR is preferred.

Additional Ablations

In the subsections below, we provide results that justify particular design choices made as part of our implementation. As part of our evaluation, we compare the performance of SBM-FWS when factors like the backbone are fine-tuned, the updated parameters (θUP ) are used to generate adversarial examples, and the DINO matching strategy is replaced with a strategy more similar to FCOS and YOLOv5. Moreover, we also show the performance difference between using the full loss (FLM) instead of just the classification loss in CLM, and demonstrate that FLM is equivalent to CLM in most cases. C.1

Backbone Training

The evaluated architectures employ a convolutional neural network (CNN) backbone, pretrained on large-scale image classification tasks, for robust feature extraction. To address scale invariance, these features are processed through a Feature Pyramid Network (FPN), which supplies multi-scale representations to a task-specific detection head. As a result, when employing the proposed adversarial fine-tuning, we considered if it is beneficial to fine-tune the entire model or just the detection head. In Fig. 9 we show the performance of SBM-FWS under each paradigm. In the case of YOLOv5, substantial ASR benefits are gained when the backbone is not fine-tuned, particularly in both the RMA BadDet and BadDet+ cases. For FCOS, some modest RmAP, ASR, and TDR gains are made in all three cases. For DINO, ASR and TDR performance is similar in both cases; however, some increased RmAP and TDR variance is observed. As a result, we do not fine-tune the backbone for any of the considered model architectures. C.2

Adversarial Generation

When solving the inner loop problem defined in Section 4.3, we must choose whether to generate adversarial examples at each step using the original model parameters (θORG ) or the updated parameters (θUP ). If θORG is used, perturbations (δ) are sampled based on their effect on the original model, and θUP is subsequently updated to minimize the impact of δ. Conversely, if θUP is used for both steps, δ is crafted to cause misclassification or disappearance against the current θUP . The parameters θUP are then updated again to mitigate this new adversarial effect. 21

(a) FCOS

(b) DINO

Figure 9: Performance of each model when the backbone is included and excluded from fine-tuning.

(a) FCOS

(b) DINO

Figure 10: Performance of each model when θORG or θUP is used to generate the adversarial examples. In Fig. 10 we show the performance of SBM-FWS under each paradigm. For FCOS and YOLOv5, we find that using θORG provides significant ASR and TDR performance improvements, while for DINO, the opposite is true. In the case of FCOS and YOLOv5, θORG provides a biased estimate of the true trigger distribution, and adversarial fine-tuning succeeds. Moreover, when θUP is used instead, the initial mitigation of the backdoor is achieved when θUP ≈ θORG ; however, after multiple iterations, this effect is lost due to catastrophic forgetting. That is, θUP introduces a moving target, causing adversarial examples to drift away from their backdoor objective over multiple iterations. For DINO, the opposite is true; using θUP is optimal. This is potentially due to the transformer detection head of DINO being more sensitive to adversarial examples, and thus the moving target introduced by θUP is helpful, or that it is less susceptible to catastrophic forgetting compared to FCOS and YOLOv5. C.3

DINO Matching Strategy

In Fig. 11, we compare the performance when DINO’s original matching strategy and a simple IoU-based matching strategy is used when solving the inner objective. Note, the IoU-based matching strategy matches all predictions with IoU>0.5 with the targeted object. For both ODA and RMA, as well as when the defence term is an is not included, using the IoU-based matching strategy significantly improves ASR and TDR performance, while also providing some RmAP gains. This improvement can be attributed to a fundamental incompatibility between DINO’s Hungarian matching strategy and adversarial objectives that manipulate classification outputs. Hungarian matching jointly considers both localisation and classification scores when assigning predictions to ground-truth objects, meaning that as the adversarial objective alters a prediction’s classification, the matched set of predictions can shift between optimisation steps. This creates a moving-target problem: the gradient updates intended to suppress or redirect a particular prediction may instead act 22

(a) RMA

(b) ODA

Figure 11: Performance of DINO when its original and IoU-based matching strategy.

(a) CLM

(b) FLM

Figure 12: Performance of (a) CLM and (b) FLM. (c) Shows the average contribution of the classification term to FLM relative to other terms.

on a different set of predictions in the next iteration, destabilising the attack. An IoU-based strategy avoids this by assigning matches purely on geometric overlap, decoupling the matching process from the classification output being manipulated. This is consistent with the design of FCOS and YOLOv5, whose assignment strategies are similarly geometry-driven and classification-agnostic, providing a more stable set of matched predictions throughout optimisation.

C.4

Comparison of FLM and CLM

In Figure 12 (a) and (b), we compare the performance of Classification Loss Maximization (CLM) against the alternative Full Loss Maximization (FLM) across three model architectures. For RMA attacks, constraining the maximization to the classification loss (CLM) yields a measurable performance improvement over maximizing the cumulative loss (FLM). For ODA, however, the performance gains are marginal. Figure 12(c) summarizes the mean loss deltas for FLM by model and attack objective. For FCOS and YOLOv5, the classification term dominates the cumulative loss during FLM, rendering FLM functionally similar to CLM. In contrast, DINO exhibits a more even distribution between classification and other loss terms. This discrepancy is likely rooted in model architecture: FCOS and YOLOv5 utilize distinct branches for classification and localization in the detection head, whereas DINO shares attention layers across these tasks. Ultimately, filtering the maximization objective to classification terms only is a practical and effective design choice. For YOLO and FCOS, this filtering further concentrates gradients on the already dominant classification components. For DINO, it assists the perturbation δ in specifically targeting classification features. Critically, for both RMA and ODA, classification disruption is the primary objective, as unintended interference with localization terms is undesirable. 23

Figure 13: Performance of CLM-FWS and SBM-FWS on COCO for RMA and ODA.

Additional Results

D.1

COCO Evaluation

Fig. 13 shows the performance of CLM-FWS and SBM-FWS during a COCO evaluation, utilizing BadDet+ RMA and ODA attacks on the FCOS and DINO models. Due to the large number of classes present in COCO and the size of its original training dataset, a subset of 1000 images was used. In general, we find that both CLM-FWS and SBM-FWS are still effective, with median ASR performance remaining below 0.33 in both instances. Compared to the Pascal VOC results presented in Section 5, the result distributions are roughly similar. The COCO results, however, show slightly higher median ASR performance and longer variance in the lower tail when the defence term is applied. Comparing the results with and without the defense term, we find that its inclusion is beneficial. For SBM-FWS, both the median ASR and its variance decrease when the defense term is included. For CLM-FWS, however, while the median performance marginally improves with the defense term, removing it yields a stronger lower-tail performance. For both CLM-FWS and SBM-FWS, this evaluation demonstrates that the methods generalise to larger-scale object detection datasets. However, it also reveals that in cases where the models’ baseline mAP performance is lower, as is the case for both DINO and FCOS compared to Pascal VOC and MTSD, mitigation via adversarial fine-tuning becomes more challenging. Specifically, it becomes more difficult to find high-quality adversarial examples that exploit backdoor behaviour, rather than exploiting a fragile detector that already produces brittle predictions on benign images. D.2

PTSD Evaluation

In Fig. 14 we show the performance of each method on MTSD and PTSD. Note, PTSD is a collection of images containing physical triggers placed on real-world objects, and is used to validate whether a synthetic-to-real-world performance gap is present. In general, all methods show strong transferability between the two datasets, suggesting that MTSD serves as a reliable proxy for real-world trigger behaviour. For ASR, PTSD performance is typically the same or stronger than MTSD, indicating that physically realised triggers are at least as effective at inducing misclassification as their synthetic counterparts. For TDR, while median performance between MTSD and PTSD is typically similar, PTSD exhibits increased long-tail variance. This likely reflects real-world factors such as variation in lighting conditions, viewing angle, and print fidelity, which can introduce inconsistency in whether the defender successfully recovers correct classification, particularly in the case of RMA. D.3

Computational Efficiency and Timing Analysis

In Table 2, we report the empirical computational cost breakdown for a single batch across our three evaluated architectures: FCOS, DINO, and YOLOv5. All experiments were conducted on a single H100 GPU to ensure consistency in timing measurements. This analysis serves to quantify the overhead introduced by the adversarial generation process and our proposed defense mechanism. 24

(a) ODA

(b) RMA

Figure 14: Performance of each method on the PTSD and MTSD dataset. As expected in adversarial training and attack frameworks, the Adversarial Generation (δ optimization) consumes the vast majority of the computational budget, accounting for over 85–99% of the total execution time per batch. Conversely, the inclusion of our proposed defense loss (LDEF ) adds negligible overhead (typically < 0.2%), demonstrating that the robustness gains reported in Section 5 are essentially “free” in terms of training latency once the adversarial example is produced. Comparative Efficiency of FWS vs. RS: A key observation is that FWS consistently yields lower total execution times compared to RS, despite FWS requiring an initial inference pass and a filtering heuristic. This efficiency stems from the quality of the targeted region in the inner loop optimization. By isolating a confidently predicted box and filtering out boxes containing false positives, the subsequent PGD steps benefit from speedup, particularly during the matching process used to isolate the predictions matched to the selected target object. While FWS incurs an initial “filtering cost” to identify the subset of predictions matching the targeted box, the resulting reduction during each PGD step means these savings compound, leading to a lower total time per batch compared to the RS. Architectural Variance: We note a significant variance in absolute execution time between models. DINO, being a Transformer-based detector, exhibits the highest total cost (up to 6.15s per batch). We observe that DINO spends a lower relative percentage of time on adversarial generation compared to the CNN-based FCOS and YOLOv5. This is attributed to the inherent computational overhead of the self-attention mechanisms. Despite these differences in base architecture, the relative efficiency of FWS over RS remains consistent across all models. D.4

Additional Trigger Types

Figure 16 illustrates the performance of CLM-FWS and SBM-FWS across five distinct trigger types. While our primary evaluation in Section 5 utilized a blue trigger to align with PTSD dataset validation requirements, we trained DINO and FCOS models on MTSD data poisoned with various triggers to ensure our defenses are robust to trigger variations. Overall, both defense methods demonstrate stable performance regardless of the trigger used. While RMA ASR remains largely consistent across triggers for both methods, ODA ASR exhibits more 25

(a) SBM-FWS

(b) CLM-FWS

Figure 15: Performance of CLM-FWS and SBM-FWS on the PTSD when data availability is reduced. Table 2: Computational cost breakdown per batch (averaged across 64 batches of size 8) for each defense variant across models. Adv Gen corresponds to the cost of performing the inner loop PGD steps. Def. Loss is the cost of calculating LDEF . Total is the execution time in seconds. FCOS

DINO

YOLOv5

Method

Adv Gen

Def. Loss

Total

Adv Gen

Def. Loss

Total

Adv Gen

Def. Loss

Total

CLM-RS CLM-FWS SBM-RS SBM-FWS

99.46% 98.68% 96.38% 93.48%

<0.01% <0.01% <0.01% <0.01%

2.40 2.27 2.76 2.30

91.71% 85.89% 91.97% 92.46%

<0.01% <0.01% <0.01% <0.01%

6.15 4.11 6.10 4.91

94.86% 89.63% 94.41% 88.38%

0.12% 0.16% <0.01% 0.11%

2.14 1.35 2.02 1.98

fluctuation; specifically, CLM-FWS shows greater median variance across the settings than SBMFWS. Importantly, ODA ASR remains below 0.5 across all tested cases. Regarding TDR, SBM-FWS displays higher median variance than CLM-FWS under the RMA settings. Finally, clean performance (RmAP) for both methods remains highly stable across all trigger types. D.5

Pruning Threshold

In Fig. 17, we evaluate the impact of the pruning threshold (τp ) on Fine-Pruning (FP) performance. Across all evaluated architectures, τp fails to provide a viable trade-off between reducing ASR and maintaining clean performance (RmAP). While ASR does fall below 50% for DINO, FCOS, and YOLOv5 at τp > 60%, 70%, and 50%, respectively, this reduction is accompanied by severe RmAP degradation. Moreover, TDR does not improve at these thresholds. This indicates that the drop in ASR is a byproduct of overall detector collapse rather than effective backdoor mitigation. Furthermore, pruning is entirely ineffective against ODA-style attacks, where ASR strictly increases as τp rises. 26

(a) SBM-FWS

(b) CLM-FWS

Figure 16: CLM-FWS and SBM-FWS performance when different triggers (y-axis) are used.

Figure 17: FP performance when pruning threshold is increased from 10 to 90%. D.6

Additional Results

Table 3: Median performance for defense on Pascal VOC dataset using 5% clean data. Method Loss RmAP ↑ TDR ↑ ASR ↓ SBM-FWS SBM-RS CLM-FWS CLM-RS FT FP FP+FT FT-SAM

w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF

0.969 0.900 0.981 0.956 0.993 0.953 0.994 0.982

0.711 0.676 0.731 0.712 0.600 0.565 0.653 0.630

0.490 0.163 0.548 0.207 0.356 0.254 0.380 0.239

– – – –

0.981 0.721 0.972 0.995

0.098 0.031 0.092 0.156

0.945 0.967 0.947 0.948

Table 4: Median performance for defense on MTSD dataset using 5% clean data. Method Loss RmAP ↑ TDR ↑ ASR ↓ SBM-FWS SBM-RS CLM-FWS CLM-RS FT FP FP+FT FT-SAM

w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF

0.980 0.905 0.982 0.951 0.992 0.950 1.009 0.994

0.892 0.861 0.898 0.891 0.922 0.912 0.917 0.916

0.183 0.037 0.221 0.157 0.088 0.059 0.081 0.072

– – – –

1.053 1.018 1.066 0.931

0.359 0.132 0.348 0.427

0.722 0.870 0.747 0.653

Table 5: Median performance for defense on Pascal VOC dataset using 5% clean data for each attack. RMA ODA Method SBM-FWS SBM-RS CLM-FWS CLM-RS

Loss

RmAP ↑

TDR ↑

ASR ↓

RmAP ↑

ASR ↓

w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF

0.968 0.897 0.979 0.947 0.998 0.944 0.994 0.982

0.711 0.676 0.731 0.712 0.600 0.565 0.653 0.630

0.649 0.126 0.728 0.185 0.319 0.174 0.355 0.225

0.971 0.903 0.983 0.966 0.991 0.957 0.997 0.987

0.364 0.379 0.379 0.404 0.367 0.385 0.387 0.378

Table 6: Median performance for defense on MTSD dataset using 5% clean data for each attack. RMA ODA Method SBM-FWS SBM-RS CLM-FWS CLM-RS

Loss

RmAP ↑

TDR ↑

ASR ↓

RmAP ↑

ASR ↓

w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF w/o LDEF w LDEF

0.967 0.894 0.975 0.948 0.986 0.940 1.009 0.989

0.892 0.861 0.898 0.891 0.922 0.912 0.917 0.916

0.239 0.014 0.331 0.167 0.032 0.009 0.057 0.020

0.992 0.931 0.992 0.968 1.002 0.979 1.015 1.001

0.138 0.251 0.136 0.155 0.102 0.112 0.091 0.100

Table 7: Median performance for defense on Pascal VOC dataset using 5%, 2.5% and 1.25% clean data Method Samples Loss RmAP ↑ TDR ↑ ASR ↓

SBM

CLM

w/o LDEF w LDEF

0.969 0.900

0.711 0.676

0.490 0.163

2.5%

w/o LDEF w LDEF

0.970 0.880

0.695 0.656

0.576 0.239

1.25%

w/o LDEF w LDEF

0.967 0.862

0.598 0.543

0.611 0.269

w/o LDEF w LDEF

0.993 0.953

0.600 0.565

0.356 0.254

2.5%

w/o LDEF w LDEF

0.995 0.955

0.662 0.609

0.596 0.283

1.25%

w/o LDEF w LDEF

0.984 0.935

0.591 0.508

0.822 0.376

Table 8: Median performance for defense on MTSD dataset using 5%, 2.5% and 1.25% clean data Method Samples Loss RmAP ↑ TDR ↑ ASR ↓

SBM

CLM

w/o LDEF w LDEF

0.980 0.905

0.892 0.861

0.183 0.037

2.5%

w/o LDEF w LDEF

0.965 0.876

0.960 0.931

0.132 0.151

1.25%

w/o LDEF w LDEF

0.969 0.918

0.834 0.869

0.312 0.108

w/o LDEF w LDEF

0.992 0.950

0.922 0.912

0.088 0.059

2.5%

w/o LDEF w LDEF

0.939 0.821

0.893 0.893

0.101 0.089

1.25%

w/o LDEF w LDEF

0.935 0.847

0.670 0.660

0.472 0.433

NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction state the two contributions, namely the CLM/SBM adversarial generation strategies and the dual-objective detection-aware defense loss, and the experimental scope (CNN- and Transformer-based detectors across multiple datasets and attack types) matches the empirical evaluation reported in Section 5. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitations are discussed in Section 5 and in the Conclusion, noting that fully restoring correct detections for trigger-bearing objects remains challenging (TDR@50 retains a long lower tail) and that the framework does not directly address object-generation (background-region) attacks. The theoretical analysis in Appendix A also explicitly notes it is a mechanism-level justification rather than a global robustness guarantee. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 30

3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The theoretical results (Propositions A1–A7) are stated with their assumptions and accompanied by full proofs in Appendix A, which is cross-referenced throughout the design discussion in Sections 4.1–4.3. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 5 and Appendix B (with Table 1) report all per-architecture hyperparameters, optimizer settings, the number of epochs, batch size, adversarial steps, step size, data splits (5%, 2.5%, 1.75% with five random splits), and trigger configurations needed to reproduce the main results, alongside the publicly available datasets and architectures used. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 31

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: As stated in Section 5, the benchmarking framework code is included with the submission and will be released publicly on GitHub upon acceptance. All datasets used (Pascal VOC, MTSD, COCO, PTSD) are publicly available from their original providers. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //neurips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: The experimental setup (datasets, attack settings, baselines, metrics, and cleandata budgets) is described in Section 5, and full training details (learning rate, epochs, batch size, adversarial steps and step size, λ, β, ϵ, IoU threshold, matching strategy) are provided in Table 1 of Appendix B, with sensitivity analyses justifying how each hyperparameter was chosen. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] 32

Justification: All main results (Figures 2, 3, 4, and the appendix figures) are reported as box plots that show median, interquartile range, and the overall distribution computed across five random data splits, capturing variability due to the choice of clean-data subset. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Section 5 and the timing analysis in Appendix (Table 2) report that experiments were conducted on a single H100 GPU and provide a per-batch timing breakdown across FCOS, DINO, and YOLOv5 for each defence variant. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research uses publicly available benchmark datasets, does not involve human subjects, and is aimed at improving the security of object detectors against backdoor attacks; we have reviewed the NeurIPS Code of Ethics and believe our work conforms to it. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics. 33

• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The Introduction motivates the work in terms of safety-critical applications such as autonomous driving and surveillance, where backdoor mitigation in object detectors yields direct positive impact; because the contribution is a defensive mitigation method (not a new attack or generative capability), the path to direct negative societal impact is limited. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: The paper does not release any pre-trained generative models, scraped data, or other assets that pose a high risk of misuse; the released artifact is a defensive backdoormitigation framework built on existing public detectors and benchmark datasets. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets 34

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All datasets (Pascal VOC, MTSD, COCO, PTSD) and detector architectures (DINO, FCOS, YOLOv5) used in the paper are cited to their original publications/sources in Section 5, and they are used in accordance with their respective public terms of use. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The new asset is the benchmarking and mitigation code, which is included with the submission and will be released on GitHub upon acceptance; method, hyperparameters, and design choices are documented in Section 5 and Appendices B and C. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve any crowdsourcing or research with human subjects; all experiments are conducted on existing public benchmark datasets. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. 35

• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve crowdsourcing or research with human subjects, so IRB approval is not applicable. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: LLMs are not used as a component of the core methodology; the work concerns adversarial fine-tuning of object detectors (FCOS, DINO, YOLOv5) and does not rely on large language models in any important, original, or non-standard way. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

Record · ID 160733 · SHA-256 acf440b9535ad464
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.