arXiv CSopen access

Improving Certified Robustness via Adversarial Distillation

Improving Certified Robustness via Adversarial Distillation

arXiv:2606.31653v1 [cs.LG] 30 Jun 2026

Matteo Melis∗ School of EEECS Queen’s University Belfast [email protected]

Jesus Martinez Del Rincon School of EEECS Queen’s University Belfast [email protected]

Vishal Sharma School of EEECS Queen’s University Belfast [email protected]

Abstract Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set. For neural networks, certified training methods based purely on tight relaxation bounds produce networks that are amenable to certification, but sacrifice standard accuracy. Conversely, adversarial training often yields stronger empirical robustness and standard accuracy, but the resulting models are generally difficult to certify with neural network verifiers. Recently, the literature has shown that better standard-certified accuracy trade-offs can be achieved by combining adversarial training objectives with loose over-approximations based on Interval Bound Propagation (IBP), effectively interpolating between lower and upper bounds of the worst-case loss. Building on this, we introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound. We show that distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate for certified training, with AD-CERT achieving state-of-the-art certified performance on several robustness benchmarks. Furthermore, in a unified setup, distilling adversarial information at the logit-level is shown to improve certified accuracy over a robust feature-space distillation objective by up to 5.40 percentage points.

1

Introduction

While deep neural networks have seen great success across various disciplines, adversarial examples [2, 12, 31] raise important questions about their adversarial robustness, i.e., a network’s ability to preserve its prediction under small input perturbations. This is of particular concern in safety-critical domains such as autonomous driving and medical diagnostics. Motivated by this, neural network verification methods [9, 17] aim to provide formal certificates of robustness for a given network. Neural network verifiers broadly fall into two categories, complete verifiers [17, 32], which compute exact bounds but have an exponential worst-case runtime, and incomplete verifiers [30, 40], which rely on convex relaxations to obtain approximate bounds. Modern state-of-the-art verifiers [10, 34, 37, 42] typically combine the two, using convex relaxations to accelerate complete ∗ Corresponding author.

Preprint.

LAD-CERT

Pθs (xadv )

Pθt (x)

Student fθs

Frozen teacher fθt

  α LCE −z∆ θs (x, y), y

+

Bound propagation

Adversarial distillation

(1 − α) LKL (Pθt (x) ∥ Pθs (xadv ))

PGD attack

z∆ θs (x, y)

IBP

Student fθ s

Bϵ (x) = [x − ϵ, x + ϵ]

x Figure 1: Overview of AD-CERT (§3.1) loss formulation, which combines adversarial distillation (blue) with IBP (orange). verification within a branch-and-bound framework [3]. Since verification only certifies robustness after training, robust training methods are needed to produce networks that are both accurate and certifiable. Training neural networks for adversarial robustness is typically approached either empirically, through adversarial examples, or through certified training, using over-approximations of the network’s reachable set over a perturbation region. Adversarial training [22] is an empirical approach that aims to improve a network’s adversarial robustness by augmenting the training objective with adversarial examples, thereby optimising a lower bound on the worst-case loss. However, adversarially trained networks are hard to formally verify and often fall short when faced with stronger or adaptive attack strategies [4, 33]. In contrast, certified training directly optimises an upper bound on the worst-case loss, producing networks that are more amenable to verification, often at the cost of standard accuracy. While earlier certified training methods [13, 26, 29, 41] trained networks against sound upper bounds of the robust loss, recent state-of-the-art methods [6, 8, 23, 27] effectively optimise unsound approximations of the worst-case loss by combining adversarial training with loose IBP-based over-approximations [13]. Due to IBP’s favourable optimisation properties [16, 24] and adversarial training’s empirical strengths, such certified training methods have been shown to provide state-of-the-art standard-certified accuracy trade-offs in recent benchmarks [25]. Motivated by the success of robust objectives that combine adversarial and certified training, we hypothesise that transferring adversarial knowledge from an empirically robust teacher to a certified student using knowledge distillation could further improve certified performance when paired with IBP bounds. This leads to the following contributions: • We introduce Adversarial Distillation for Certification (AD-CERT) (§3.1), a novel certified training objective that combines adversarial logit-level distillation from an empirically robust teacher with IBP bounds (depicted in Figure 1). • We show that AD-CERT is a scalarisation between a teacher-guided lower bound surrogate and a certified IBP upper bound on the robust loss (§3.2). • We present extensive experimental evaluations of AD-CERT across standard certified training benchmarks (§4.2), showing that it achieves state-of-the-art certified accuracy against current certified training methods. • We provide a systematic analysis of multiple knowledge distillation approaches for empirical endpoints (§4.3), showing that adversarial logit-level distillation preserves teacher robustness 2

more effectively than clean or feature-space distillation over the same architecture, and translates best into certified training when combined with an IBP loss (§4.2).

2

Background

Here, we discuss some necessary background for this work. Let fθ denote a multi-class neural network classifier parametrised by θ such that fθ : Rd 7→ RC maps an input x ∈ X ⊆ Rd to a numerical prediction score [fθ (x)]i for each class i ∈ {1, 2, . . . , C}. Let (x, y) ∼ D denote a sample input and true label from a data distribution D and let Bεp (x) denote a perturbation set based on an ℓp norm threat model. In this work, as standard with certified training [1, 6, 7, 8, 13, 23, 27], we only consider the ℓ∞ case, and therefore put Bε (x) := {x′ : ∥x′ − x∥∞ ≤ ε}, where x′ denotes the perturbed input and ε is the perturbation radius. 2.1

Neural Network Verification

A network is said to be adversarially robust for an input x if for all x′ ∈ Bε (x), we have [fθ (x′ )]i < C−1 [fθ (x′ )]y ∀ i ̸= y. Let z∆ denote the logit-difference between the true class and all other θ ∈ R ∆ classes, i.e., zθ (x, y) := ([fθ (x)]y − [fθ (x)]i )i̸=y . Then, a network’s adversarial robustness can be verified by solving: ′ min min [z∆ (1) θ (x , y)]i , ′ x ∈Bε (x)

i̸=y

and checking if the solution to Equation (1) is greater than zero. Finding an exact solution to Equation (1) was shown to be NP-hard [17], thus, neural network verifiers either approximate the bounds [30, 40], or perform complete verification at a larger computational cost [10, 17, 32, 34, 37, 42]. 2.2

Training for Robustness

Standard training of neural network classifiers optimises network parameters θ by minimising the expected cross-entropy loss, i.e., min E(x,y)∼D [LCE (fθ (x), y)] , θ

(2)

PC where LCE (fθ (x), y) = log(1 + i̸=y exp([fθ (x)]i − [fθ (x)]y )). In contrast, training for robustness can be viewed as the following min-max objective:   min E(x,y)∼D ′max LCE (fθ (x′ ), y) . (3) θ

x ∈Bε (x)

Concretely, for an input x, the inner maximisation finds the most malicious perturbation x′ ∈ Bε (x) that maximises the loss. The outer minimisation seeks the optimal network parameters θ such that these threats are less effective against the network predictions. Since the inner maximisation problem is non-convex, we generally solve this by under- or over-approximating the worst-case loss. In general, if we let L∗rob (fθ (x), y) := ′max L (fθ (x′ ), y) denote a network objective solved for the x ∈Bε (x)

optimisation problem in Equation (3), then we have: Lrob (fθ (x), y) ≤ L∗rob (fθ (x), y) ≤ Lrob (fθ (x), y),

(4)

with Lrob (fθ (x), y) and Lrob (fθ (x), y) corresponding to adversarial and certified training, respectively, as described below. Adversarial Training (AT) AT first approximates a solution to the inner maximisation problem of Equation (3) by performing an adversarial attack, such as projected gradient descent (PGD) [22], within Bε (x) for each input x. Following this, the outer minimisation problem of Equation (3) is solved by augmenting the network’s training objective with the adversarial example xadv ∈ Bε (x) obtained by the attack. In its standard form, AT aims to minimise: LAT := LCE (fθ (xadv ), y) .

(5)

This can be seen as optimising a lower bound of the worst-case loss, i.e., Lrob (fθ (x), y) from Equation (4). While AT leads to strong empirical robustness, it produces complex networks that are difficult to formally verify and remain susceptible to stronger attacks [4, 33]. 3

Certified Training and Bound Propagation Certified training optimises an upper bound of the inner-maximisation problem in Equation (3) by over-approximating the worst-case loss. A simple approach is Interval Bound Propagation (IBP) [13], leveraging interval arithmetic to approximate the output range of a given layer. Given an input x, IBP propagates the upper and lower bounds of the perturbed input space, Bε (x), (corresponding to x := x + ε and x := x − ε, respectively) through the entire network layer by layer, providing a hyper-rectangle that is monotonically increasing every layer. The final layer output of this propagation, which we denote with B OX, is an over-approximation of the possible output space, i.e., B OX ⊇ fθ (Bε (x)). Hence, using the B OX relaxation, we can compute the worst-case logit difference between the true class and all other classes by putting  ′ ′ )] (x, y) := [f (x )] − z∆ [f (x and obtain a robust loss given by: θ θ θ i y i̸=y

LIBP := LCE (−z∆ θ (x, y), y),

(6)

where LIBP is a sound upper bound, Lrob (fθ (x), y), from Equation (4). Further details on IBP can be found in Appendix A. Interestingly, alternative certified training methods that utilise tighter relaxations, e.g., linear relaxations [35, 41], consistently yield worse performance than imprecise IBP-based training. Jovanović et al. [16] attribute this phenomenon to tighter relaxations adding discontinuity and difficulty to the optimisation objective. Moreover, Mao et al. [24] explicitly show that IBP-based training leads to better propagation tightness, while non-IBP-based methods do not. Additionally, they note that over-regularisation provides worse standard-certified accuracy trade-offs. In line with these findings, all current state-of-the-art certified training methods [6, 8, 23, 27] move away from previous over-regularised approaches [1, 7, 13] and instead shift their focus to combining AT with IBP-based training, achieving state-of-the-art standard-certified accuracy trade-offs while preserving certification properties. De Palma et al. [8] further elucidate this in their definition of expressive losses whereby expressivity is attained through interpolation between lower and upper bounds of Equation (4) during training, providing an unsound but favourable solution to Equation (3). 2.3

Knowledge Distillation

Hinton et al. [15] introduced knowledge distillation as a method for transferring knowledge from a large teacher model, Tθt , to a smaller, more compact student model, Sθs . This allows the student to retain much of the teacher’s performance while reducing computational cost. Knowledge distillation utilises Kullback-Leibler (KL) divergence to compare the predictive distributions of the student and teacher, which is defined by: ! C
X [Pθτt (x)]i τ τ τ KL Pθt (x) ∥ Pθs (x) = [Pθt (x)]i log , (7) [Pθτs (x)]i i=1   T (x) where τ is a temperature applied to the softmax operation and Pθτt (x) := softmax θtτ ,   Sθs (x) τ Pθs (x) := softmax are the respective predictive distributions of the teacher and student τ after softmax. Knowledge distillation then optimises the following objective: 
 min E(x,y)∼D (1 − α)LCE (Sθs (x), y) + ατ 2 KL Pθτt (x) ∥ Pθτs (x) , (8) θs

where α ∈ [0, 1] is a balancing factor between standard training and learning the teacher’s predictive distribution. Knowledge distillation has since been applied in the AT setting to boost empirical robustness of a student using an adversarially trained teacher [5, 11, 44, 45]. This was first explored by Goldblum et al. [11], through adversarially robust distillation (ARD), which injects adversarial examples into the KL term from Equation (8). More specifically:
LARD := (1 − α)LCE (Sθs (x), y) + ατ 2 KL Pθτt (x) ∥ Pθτs (xadv ) . (9) Additionally, De Palma [6] recently introduced knowledge distillation within the field of certified training, where knowledge is distilled from an empirically robust teacher over the feature space (further details in Appendix G). In the coming section, we build on previous certified training [1, 6, 7, 8, 13, 23, 27] and adversarial knowledge distillation [11, 45] works, presenting a novel certified training objective that distils the predictive distribution of an empirically robust teacher onto a student training for certifiable robustness. 4

3

Methodology

In this section, we introduce Adversarial Distillation for Certification (AD-CERT), a novel certified training scheme that interpolates between adversarial distillation (AD) and IBP [13]. We leverage the key insight that replacing a hard-label adversarial lower bound with a teacher-guided surrogate could further improve certified training. In this way, the empirical teacher shapes the student on concrete adversarial examples, while certification is enforced directly through IBP. Full proofs and additional theoretical results are deferred to Appendices B and C, respectively. Pseudocode for the complete training procedure is provided in Appendix E and further details on AD-CERT’s computational cost can be found in Appendix F.5, where AD-CERT corresponds to performing IBP and PGD training, with the slight added cost of a single forward pass on the detached teacher. 3.1

AD-CERT Formulation

Going forward, we denote by fθt a fixed teacher network trained via AT, and let fθs be the student model that we aim to certify. Note that both the student and teacher share the same architecture in this distillation setting. Moreover, for all experiments, we use a temperature setting of τ = 1, and henceforth, omit the τ notation from all softmax operations, writing Pθ (x) := softmax(fθ (x)) for the predictive distribution of a network. Excluding additional regularisation, AD-CERT attains the following loss objective:
LAD-CERT := (1 − α) KL (Pθt (x) ∥ Pθs (xadv )) +α LCE −z∆ α ∈ [0, 1]. (10) θs (x, y), y , | {z } | {z } Adversarial Distillation

IBP

The AD component of Equation (10) is exactly the KL portion of Equation (9), where the student is evaluated on an adversarially perturbed input, and tasked to match the clean distribution of an adversarially trained teacher, aiming to preserve standard and empirically robust accuracy. On an intuitive level, AD-CERT is designed to keep the teacher signal completely outside of verified bounds, using it purely for its empirical strengths, while leveraging IBP to tighten the worst-case logit differences. This aims to make the student amenable to verification, while still retaining a large proportion of the teacher’s adversarial and standard accuracy. 3.2

Teacher-Guided Lower Bound Surrogates

In this subsection, we take an analytical view of the objective in Equation (10) and formalise the role of AD within certified training. Unlike AT and IBP, which, respectively, provide formal lower and upper bounds from Equation (4), the AD term in AD-CERT is used as a teacher-guided surrogate for the adversarial lower bound. This is an intentional relaxation of the lower bound objective, using an empirical teacher only to improve the adversarial endpoint, while leaving the verifiable IBP upper bound objective unchanged. We begin by showing that, up to a teacher-dependent constant, the distillation component of AD-CERT is exactly AT with soft teacher labels. Proposition 3.1 (Equivalence of AD and soft-label AT). Let xadv be fixed, and let the teacher predictive distribution Pθt (x) be fixed. Then the AD objective: LAD := KL(Pθt (x) ∥ Pθs (xadv )) ,

(11)

is equal, up to an additive constant independent of the student parameters, to AT with soft teacher labels, namely: Lsoft-AT := LCE (fθs (xadv ), Pθt (x)) . (12) Moreover, both objectives induce identical backpropagation updates with respect to the student logits. In particular, for each class index i: ∂LAD ∂Lsoft-AT = [Pθs (xadv )]i − [Pθt (x)]i = . (13) ∂zθi s ∂zθi s Proof. See Appendix B. Given Proposition 3.1, we have shown that the distillation component of AD-CERT preserves AT at the gradient level, while being conditioned by an empirical teacher. We further elucidate this connection through an analytical view of our loss function from Equation (10). 5

Proposition 3.2 (Analytical form of AD-CERT). Let H(P ) and H(Q, P ) denote the entropy and cross entropy respectively for probability distributions P and Q. That is: H(P ) = −

C X

H(Q, P ) = −

pi log(pi ),

i=1

C X

qi log(pi ).

(14)

i=1

Then, LAD-CERT from Equation (10) can be re-written as: LAD-CERT = (1 − α)H(Pθt (x), Pθs (xadv )) + αLIBP − (1 − α)H(Pθt (x)).

(15)

Therefore, up to a constant independent of θs , AD-CERT is a scalarisation between (i) a soft-label adversarial lower bound surrogate and (ii) a certified IBP upper bound. Proof. See Appendix B. Given Proposition 3.2, AD-CERT fits naturally within the broader class of modern certified training objectives [6, 8, 23, 27] that combine empirical and certifiable endpoints. Its distinguishing feature is that the adversarial endpoint is no longer a formal lower bound of Equation (3), but a relaxed surrogate induced by AD from an empirically robust teacher. Additionally, this positions AD-CERT as distinct and complementary to the recent work of CC-DIST [6], which computes both IBP bounds and adversarial examples within their robust feature-space distillation objective. Instead, AD-CERT integrates teacher distillation directly into the lower bound endpoint, while keeping IBP bounds purely for the certified endpoint. While Proposition 3.2 characterises AD-CERT at the level of its objective, the next result highlights a key geometric difference between soft-label and hard-label adversarial supervision. In particular, the AD branch of AD-CERT induces a finite target in the logit space, whereas standard AT only approaches its optimum as the correct class margin grows indefinitely. ∗ Lemma 3.3. For AD, the optimal logit margin, ∆zij , between any two classes i, j is:   [Pθt (x)]i ∗ ∆zij = log . (16) [Pθt (x)]j Moreover, standard AT has no finite minimiser, and its infimum 0 is approached only as [fθs (xadv )]y − [fθs (xadv )]j → +∞, ∀j ̸= y. Proof. See Appendix B. Lemma 3.3 suggests that teacher-guided soft-label supervision may be a principled approach to enhance certified training, since it arguably induces a smoother lower bound objective, not pushing for arbitrarily large class margins. We evaluate our hypotheses experimentally in the next section.

4

Experimental Evaluation

In this section, we conduct experiments to assess the effectiveness of our proposed certified training method, AD-CERT (§3.1). Across our experiments, we report standard, certified and AUTOATTACK (AA) [4] accuracy. Certified accuracy is the primary metric used in certified training literature [1, 6, 7, 8, 13, 23, 25, 27], defined as the proportion of the validation set for which a verifier proves robustness within Bε (x). Standard accuracy is the clean accuracy on the validation set without perturbation. AA accuracy refers to the proportion of the validation set that could not be attacked by the AA strategy. As such, AA accuracy is strictly greater than or equal to certified accuracy. 4.1

Settings

We implement AD-CERT on top of CTBench [25], a unified certified training library using PyTorch [28]. We adopt three datasets, MNIST [21], CIFAR-10 [19] and TinyImageNet [20]. We train on the corresponding training sets and certify on the validation sets. To align with recent literature [6, 8, 23, 25, 27, 29], we use a 7-layer convolutional network (CNN7) for both the teacher and 2 8 student models, and the perturbation radii applied are ε = {0.1, 0.3} for MNIST, ε = { 255 , 255 } 6

1 for CIFAR-10 and ε = { 255 } for TinyImageNet. For verification, we use the α,β-CROWN library [34, 36, 37, 40], which combines linear bound propagation with branch-and-bound [3] for complete neural network verification. Full details on training of both student and teacher models, certification and the experimental setup can be found in Appendix F.

4.2

Comparison with State-of-the-Art

Table 1 compares the performance of AD-CERT (§3.1) to current state-of-the-art certified training methods [6, 8, 13, 23, 27, 38]. For each method, we report the standard and certified accuracy on the validation set. When available, we use the benchmark results from CTBench [25], as they generally report the strongest certified accuracies of each reported certified training method compared to the original paper [6, 8, 13, 23, 27, 38], while also providing a fair comparison to AD-CERT, which is built on top of the unified library. Overall, AD-CERT achieves state-of-the-art certified accuracy for ReLU-based architectures across all evaluated settings, while retaining competitive standard accuracy. On MNIST, AD-CERT obtains modest but consistent improvements over prior methods at both perturbation radii. While the gains are small, this is expected given the relative simplicity of MNIST and the already saturated performance of existing methods. For CIFAR-10 and TinyImageNet, AD-CERT improves the best prior ReLU-based certified accuracy by an average of 0.37 percentage points, 2 8 with the largest gain observed at ε = 255 on CIFAR-10. For CIFAR-10 at ε = 255 , SORTNET [38] obtains higher certified accuracy, however, it relies on a specialised 1-Lipschitz architecture known to perform particularly well on this setting. We therefore distinguish it from the ReLU-based methods, with further discussion in related work (§5). On TinyImageNet, AD-CERT achieves the highest certified accuracy among all compared methods, while substantially improving standard accuracy compared to previous non-distillation-based certified training methods [8, 13, 23, 27]. This suggests that AD-CERT scales favourably beyond the smaller MNIST and CIFAR-10 settings, where certified training is often closer in performance. An interesting comparison is that of AD-CERT and CC-DIST [6], the only distillation-based certified training methods. In general, CC-DIST retains higher standard accuracy, while AD-CERT obtains higher certified accuracy. We investigate this trade-off further through a controlled comparison in Table 2. Comparison of Distillation-based Certified Training Methods Table 2 compares AD-CERT against alternative distillation-based certified training methods under the exact same experimental setup for a direct comparison. This controlled comparison allows us to isolate the effect of the distillation mechanism itself by training and evaluating all methods in the CTBench library [25] with the same teacher model, training pipeline and verifier. We compare AD-CERT with CC-DIST [6], which performs robust feature-space distillation. Additionally, we introduce RSLAD-CERT, an alternative logit-level objective obtained by replacing the adversarial distillation term in AD-CERT with RSLAD [45], but retaining the same IBP interpolation, i.e., LRSLAD-CERT := (1 − α)LRSLAD + αLIBP . For all three methods, we use the exact same hyperparameter values as those used for AD-CERT in Table 1, which we report in detail in Appendix F.3, Table 6. The only exception is wrob for TinyImageNet, where we used an earlier untuned value of 0.7, as per MTL-IBP’s [8] reported value in CTBench [25]. Overall, the results support the design choice of AD-CERT, showing that directly distilling the teacher’s predictive distribution at the adversarial endpoint provides a simple and effective empirical branch for certified training. The addition of the clean distillation term in RSLAD-CERT reduces certifiability by 0.6 percentage points on average, while only improving standard accuracy by an average of 0.38 percentage points compared to AD-CERT. Consistent with the pattern observed in Table 1, CC-DIST is generally strongest at transferring standard accuracy, but shows weaker certified accuracy under the controlled setup. This results in worse standard-certified accuracy trade-offs, with CC-DIST improving standard accuracy over AD-CERT by an average of 1.69 percentage points, but reducing certifiability by 3.52 percentage points. Both logit-level objectives, i.e., RSLAD-CERT and AD-CERT, improve certified accuracy over the feature-space distillation objective of CC-DIST, despite not matching the formal definition of expressivity [8]. One possible explanation is that CC-DIST induces a more regularised objective, since it computes adversarial examples and IBP bounds during distillation, before being coupled with another expressive loss CC-IBP [8]. This suggests that tightly coupling distillation-based certified training objectives with the formal definition 7

Table 1: Comparison of AD-CERT with prior state-of-the-art certified training methods. Bold indicates the best standard or certified accuracy on each benchmark. When the best ReLU-based architecture result differs from the best overall result, the top ReLU-based accuracy is bold-underlined. Dataset

ε

0.1

MNIST

Method

Source

Std. acc. [%]

Cert. acc. [%]

AD-CERT

Ours

99.25

98.53

Mao et al. [25] Zhang et al. [38] Mao et al. [25] Mao et al. [25] Mao et al. [25]

98.87 99.01 99.08 99.16 99.18

98.26 98.14 98.40 98.52 98.37

IBP SORTNET† SABR TAPS/STAPS‡ MTL-IBP

Ours

98.85

93.98

Mao et al. [25] Zhang et al. [38] Mao et al. [25] Mao et al. [25] Mao et al. [25]

98.54 98.46 98.66 98.56 98.74

93.80 93.40 93.68 93.95 93.90

AD-CERT 0.3

IBP SORTNET† SABR TAPS/STAPS‡ MTL-IBP

Ours

79.21

65.08

Mao et al. [25] Zhang et al. [38] Mao et al. [25] Mao et al. [25] Mao et al. [25] De Palma [6]

67.49 67.72 77.86 77.05 78.82 81.55

55.99 56.94 63.61 64.21 64.41 64.60

Ours

53.69

35.84

Mao et al. [25] Zhang et al. [38] Mao et al. [25] Mao et al. [25] Mao et al. [25] De Palma [6]

48.51 54.84 52.71 49.96 54.28 55.13

35.28 40.39 35.34 35.25 35.41 35.52

Ours

40.06

28.19

Mao et al. [25] Zhang et al. [38] Mao et al. [25] Mao et al. [25] Mao et al. [25] De Palma [6]

26.77 25.69 30.58 30.63 35.97 43.78

19.82 18.18 20.96 22.31 27.73 27.88

AD-CERT 2 255

CIFAR-10

IBP SORTNET† SABR TAPS/STAPS‡ MTL-IBP CC-DIST AD-CERT

8 255

IBP SORTNET† SABR TAPS/STAPS‡ MTL-IBP CC-DIST AD-CERT

TinyImageNet

1 255

IBP SORTNET† SABR TAPS/STAPS‡ MTL-IBP CC-DIST

† Non-ReLU-based architecture. ‡ We report the stronger of TAPS and STAPS based on certified accuracy.

of expressivity may not translate into stronger certification properties. In contrast, AD-CERT keeps the distillation branch as a pure adversarial endpoint, while leaving the certified component as the standard IBP loss. This relaxes the definition of expressivity by extending it to surrogate lower bounds (§3.2), prioritising simplicity and smoothness in the loss objective. Finally, we acknowledge that further exhaustive hyperparameter and teacher tuning on CC-DIST could potentially reduce the gap between the results reported in Table 2 and those in Table 1. However, the most important shared hyperparameters, namely the IBP coefficient α and L1 regularisation, coincide with the original CC-DIST setup [6] (Table 3). The main remaining difference is therefore the teacher model. For completeness, we include an additional ablation on the sensitivity of AD-CERT and CC-DIST to different teachers in Appendix D, which shows a similar trend to the above ablation. 4.3

Ablations & Further Discussions

Teacher-Student Comparison Table 3 compares each AD-CERT student with its corresponding empirically robust teacher. Overall, AD-CERT converts models that are empirically robust but not certifiable into students with strong certified accuracy, while retaining a substantial fraction of the teacher’s standard and adversarial performance. This effect is particularly clear on MNIST, where AD-CERT stays within one percentage point of the teacher in both standard and AA accuracy across 8

Table 2: Comparison of AD-CERT against distillation-based certified training methods under the same experimental setup. Dataset

ε

Method

Method source†

Dist. space

Std. [%]

AA [%]

Cert. [%]

2 255

AD-CERT RSLAD-CERT CC-DIST

Ours Ours De Palma [6]

Logits Logits Features

79.21 79.87 81.04

68.38 68.27 69.97

65.20 64.36 59.80

8 255

AD-CERT RSLAD-CERT CC-DIST

Ours Ours De Palma [6]

Logits Logits Features

53.49 53.95 54.32

36.46 36.50 35.69

35.60 35.19 34.55

1 255

AD-CERT RSLAD-CERT CC-DIST

Ours Ours De Palma [6]

Logits Logits Features

41.38 41.39 43.79

30.68 30.90 32.66

27.68 27.14 23.57

CIFAR-10

TinyImageNet

† The “Method source” column indicates the origin of the training objective, all results are from our implementation and

evaluation in the CTBench library [25].

both perturbation radii, while introducing strong certifiability. On TinyImageNet, the teacher-student gap remains moderate, with losses below seven percentage points in standard accuracy and close to two percentage points in AA accuracy, though notably achieving 28.19% in certified accuracy. The 8 largest gap occurs on CIFAR-10 at ε∞ = 255 , where the student sees a decrease of almost twenty percentage points in standard accuracy relative to the teacher, although the drop in AA accuracy is much smaller. Nevertheless, AD-CERT still achieves the strongest certified accuracy among ReLU-based architectures in this setting. However, this highlights that closing the remaining gap between empirically robust teachers and certifiably robust students at larger perturbation radii remains an important direction for future work. Motivated by this observation, we next ablate different pure AD techniques on CIFAR-10, aiming to better understand the difficulty of transferring empirical robustness into certifiable robustness and to identify potential directions for stronger distillation-based certified training objectives in future. Table 3: Comparison between teacher and student models of AD-CERT for each dataset entry in Table 1. Deltas are reported relative to the teacher at the same perturbation radius. Dataset

ε∞

Model

Std. acc. [%]

∆ Std.

AA acc. [%]

∆ AA

Cert. acc. [%]

0.1

Teacher Student

99.47 99.25

– -0.22

98.97 98.55

– -0.42

≈ 0† 98.53

0.3

Teacher Student

99.43 98.85

– -0.58

93.83 94.64

– +0.81

≈ 0† 93.98

2 255

Teacher Student

88.67 79.21

– -9.46

72.41 68.39

– -4.02

≈ 0† 65.08

8 255

Teacher Student

73.55 53.69

– -19.86

41.99 36.69

– -5.30

≈ 0† 35.84

1 255

Teacher Student

46.78 40.06

– -6.72

33.16 31.14

– -2.02

≈ 0† 28.19

MNIST

CIFAR-10

TinyImageNet

† None of the first 10 samples were verified within a 1000 second time-limit.

Exploring Adversarial Distillation Techniques Table 4 compares several distillation objectives on CIFAR-10 [19] after removing any certified training component and trained against a PGD-trained teacher on the same architecture. This ablation isolates the empirical distillation endpoint, allowing us to study which teacher-student objectives best preserve the robustness of the PGD teacher before introducing certifiability. Natural knowledge distillation [15], denoted Nat-KL, transfers the teacher’s clean predictive distribution to the student. AD corresponds to removing the IBP term from AD-CERT (§3.1), while RSLAD [45] extends adversarially robust distillation [11] using soft-labels. Finally, adversarial feature-space knowledge distillation (AF-KD) corresponds to setting α = 0 in CC-DIST [6], thereby removing the IBP components from the objective. For RSLAD and AF-KD, we use λ = 56 and β = k5 , respectively, following the suggestions of the original works [6, 45], where k denotes the dimensionality of the feature space. 9

From the results, it can be observed that Nat-KL improves standard accuracy at both perturbation radii, but dramatically fails to transfer adversarial robustness, losing more than 41 percentage points in AA accuracy in both settings. This confirms that matching an adversarially trained teacher only on clean inputs is insufficient for transferring adversarial robustness. In contrast, objectives that expose the student to adversarial inputs preserve teacher-level AA accuracy much more effectively. Across both perturbation radii, AD and RSLAD remain close to the teacher’s AA accuracy, while 8 AF-KD is competitive at the smaller radius but degrades more noticeably at ε∞ = 255 . Among these methods, AD provides the most direct and consistently effective adversarial endpoint, improving teacher-level AA accuracy at the smaller radius and remaining within one percentage point at the larger radius. These results further motivate our use of AD as the empirical endpoint of AD-CERT, since it provides a simple but effective mechanism for retaining adversarial accuracy from an empirically robust teacher, while remaining straightforward to combine with an IBP-certified upper bound. This yields a loss objective that resembles previous state-of-the-art expressive losses [8], but in a distillation setting with lower bound surrogates, as discussed in §3.2.

Table 4: Effect of different AD techniques without a certification component on CIFAR-10. Method

Objective

Nat-KL AD† RSLAD‡ AF-KD†

KL(Pθt (x) ∥ Pθs (x)) KL(Pθt (x) ∥ Pθs (xadv )) (1 − λ)KL(Pθt (x) ∥ Pθs (x)) + λKL(Pθt (x) ∥ Pθs (xadv )) LCE (−zθs (xadv , y), y) + β ∥hθs (xadv ) − hθt (x)∥22

Dataset

ε∞

Method

Std. acc. [%]

∆ Std.

AA acc. [%]

∆ AA

2 255

PGD Teacher Nat-KL AD† RSLAD‡ AF-KD†

88.67 89.89 88.64 87.88 89.52

– +1.22 -0.03 -0.79 +0.85

72.41 30.72 73.62 73.32 73.55

– -41.69 +1.21 +0.91 +1.14

8 255

PGD Teacher Nat-KL AD† RSLAD‡ AF-KD†

78.95 81.32 77.08 76.47 78.95

– +2.37 -1.87 -2.48 0.00

42.48 0.52 41.64 42.38 37.49

– -41.96 -0.84 -0.10 -4.99

CIFAR-10

† IBP terms removed from AD-CERT/CC-DIST, i.e., α = 0. ‡ Adversarial examples are generated against soft-labels as in original work [45].

IBP Coefficient Sensitivity Analysis In certified training, it is well known that higher certified accuracy often comes at the cost of standard accuracy. Recent methods fall into a family of expressive losses [8], e.g., SABR [27], CC-IBP & MTL-IBP [8], and CC-DIST [6], including hyperparameters that regulate this trade-off by interpolating between empirical and certifiable training objectives. Our goal in this study is to empirically support our methodology claim (§3.2) that the AD term in AD-CERT provides a sensible teacher-guided surrogate for the lower bound endpoint used in prior expressive losses [6, 8, 27]. Figure 2 shows the trade-off between standard, AA, and certified accuracy for AD-CERT 1 on TinyImageNet with ε = 255 as we decrease the IBP coefficient, α, from Equation 10. Consistent with the sensitivity analyses of Müller et al. [27] (Figure 7), De Palma et al. [8] (Figure 1) and Mao et al. [25] (Figure 7), standard and AA accuracy increase as the strength of IBP regularisation is reduced. Up to a point, this also benefits certified accuracy, before the model starts to dramatically lose its certification properties when α is decreased beyond this. This behaviour suggests that the adversarial distillation endpoint in AD-CERT acts analogously to the hard-label adversarial endpoints used in prior expressive losses, while also achieving state-of-the-art certified accuracy (§4.2). 10

45

Accuracy [%]

40 35 30

Standard acc. AA acc. Certified acc.

25 10 2 IBP coefficient

10 3

Figure 2: Sensitivity analysis of AD-CERT with respect to the IBP coefficient, α, on TinyImageNet at ϵ = 1/255. We report standard, AA, and certified accuracy over eight values of α ∈ [5 × 10−4 , 5 × 10−2 ].

5

Related Work

Certified Training As previously mentioned (§2.2), IBP [13, 26] performs interval arithmetic over the bounds of a simple bounding-box relaxation that over-approximates the possible output ranges of each layer, before computing an upper bound of the worst-case loss based on the output range of the final layer. Shi et al. [29] later improved IBP-based training by introducing a parameter initialisation technique that induces a constant growth rate of IBP bounds, along with specially designed regularisation to stabilise initial phases of training and improve overall performance. Müller et al. [27] propose SABR, which computes the IBP bounds over a small adversarially selected region rather than over the full perturbation set. It defines a smaller box of radius τ = λτ ε, with λτ ∈ [0, 1], centred around a projected adversarial point, ensuring the local region Bτ (xτ ) = {x′ : ∥x′ − xτ ∥∞ ≤ τ } is fully contained within Bε (x). Final IBP bounds are then τ computed over Bτ (xτ ) and used to calculate the final IBP loss, i.e., LSABR = L(−z∆ θ (x, y), y). The intuition is that propagating a smaller adversarially selected region reduces the approximation error induced by IBP bounds, which grows with respect to network depth [24], thereby providing an unsound but effective approximation of the robust loss over the full perturbation set. Mao et al. [23] introduced TAPS, which splits a network into two sub-parts before training the first part through IBP and the second through AT. The idea is that the PGD under-approximation can compensate for some of the over-approximation error induced by IBP bounds. STAPS [23] carries out an identical procedure, only changing the perturbation region to a smaller adversarially selected one as in SABR. De Palma et al. [8] formalise a family of expressive losses, which interpolate between empirical lower bounds and certified upper bounds of the worst-case robust loss. MTL-IBP is one such method that achieved relative state-of-the-art performance through leveraging a task balancing coefficient α ∈ [0, 1] for fine-grained control over a linear interpolation of PGD and IBP based training, i.e., LMTL−IBP = (1 − α)LPGD + αLIBP . De Palma [6] introduced CC-DIST, which splits the network into a feature extractor and classification head, using an empirically trained teacher for feature-space distillation before learning final classifications through their prior CC-IBP objective [8] on the full model. CC-DIST is closest to our work, since it distils knowledge from an empirically robust teacher. However, CC-DIST is tightly coupled with the notion of expressivity [8], combining a robust feature-space distillation loss with the expressive CC-IBP objective. Moreover, CC-DIST calculates both adversarial examples and IBP bounds for distillation. In contrast, AD-CERT uses a simple logit-level AD branch as the empirical endpoint, while keeping the certified branch as the baseline IBP loss. All of the previously mentioned works train standard feed-forward ReLU networks. A related but distinct line of work instead studies robustness by construction. S ORTNET [39] is a notable such model that utilises ℓ∞ -distance functions to provide a 1-Lipschitz architecture. While S ORTNET 8 performs particularly well on the CIFAR-10 dataset with ε = 255 , it is less competitive with standard certified training methods on ReLU architectures, particularly against smaller ε values (see §4.2). The non-smoothness and gradient sparsity introduced by 1-Lipschitz architectures leave room for 11

exploration in their practical applicability to certified training but remains outside the scope of this work.

6

Conclusions

In this paper, we introduced AD-CERT, a novel certified training algorithm that combines the empirical strengths of an adversarially trained teacher with the verifiability of IBP bounds through a simple linear combination of adversarial distillation over the logit-level and an IBP loss objective. AD-CERT achieves overall state-of-the-art certified accuracy, outperforming all prior methods across five standard certified training benchmarks, while remaining competitive in standard accuracy. Our ablations further suggest that adversarial logit-level distillation provides an effective empirical endpoint for certified training, acting analogously to previous hard-label lower bound endpoints while improving certified performance. Nevertheless, a substantial gap remains between empirically robust teachers and certifiably robust students in terms of standard and adversarial accuracy, particularly on harder settings, highlighting the need for future work on stronger mechanisms for transferring empirical robustness into certifiable robustness.

References [1] Mislav Balunovic and Martin Vechev. Adversarial training and provable defenses: Bridging the gap. In International Conference on Learning Representations, 2020. [2] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Advanced Information Systems Engineering, 2013. [3] Rudy Bunel, Jingyue Lu, Ilker Turkaslan, Philip H. S. Torr, Pushmeet Kohli, and M. Pawan Kumar. Branch and bound for piecewise linear neural network verification. J. Mach. Learn. Res., 2020. [4] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning, 2020. [5] Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Decoupled kullback-leibler divergence loss. In Advances in Neural Information Processing Systems, 2024. [6] Alessandro De Palma. Learning better certified models from empirically-robust teachers, 2026. URL https://arxiv.org/abs/2602.02626. [7] Alessandro De Palma, Rudy Bunel, Krishnamurthy Dvijotham, M. Pawan Kumar, and Robert Stanforth. Ibp regularization for verified adversarial robustness via branch-and-bound. In International Conference on Machine Learning, 2022. [8] Alessandro De Palma, Rudy Bunel, Krishnamurthy Dvijotham, M. Pawan Kumar, Robert Stanforth, and Alessio Lomuscio. Expressive losses for verified robustness via convex combinations. In International Conference on Machine Learning, 2024. [9] Ruediger Ehlers. Formal verification of piece-wise linear feed-forward neural networks. Automated Technology for Verification and Analysis, 2017. [10] Claudio Ferrari, Mark Niklas Mueller, Nikola Jovanović, and Martin Vechev. Complete verification via multi-neuron relaxation guided branch-and-bound. In International Conference on Learning Representations, 2022. [11] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Association for the Advancement of Artificial Intelligence (AAAI) Conference on Artificial Intelligence, 2020. [12] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. [13] Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. In Advances in Neural Information Processing Systems, 2018. 12

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In International Conference on Computer Vision, 2015. [15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems, 2015. [16] Nikola Jovanović, Mislav Balunović, Maximilian Baader, and Martin Vechev. On the paradox of certified training. Transactions on Machine Learning Research, 2022. [17] Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In Computer Aided Verification, volume 10426 of Lecture Notes in Computer Science, pages 97–117. Springer, 2017. [18] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. [19] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. [20] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. [21] Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. MNIST handwritten digit database, 2010. URL http://yann.lecun.com/exdb/mnist. [22] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. [23] Yuhao Mao, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Connecting certified and adversarial training. In Advances in Neural Information Processing Systems, 2023. [24] Yuhao Mao, Mark Niklas Müller, Marc Fischer, and Martin Vechev. Understanding certified training with interval bound propagation. In International Conference on Learning Representations, 2024. [25] Yuhao Mao, Stefan Balauca, and Martin Vechev. Ctbench: A library and benchmark for certified training. In International Conference on Machine Learning, 2025. [26] Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, 2018. [27] Mark Niklas Müller, Franziska Eckert, Marc Fischer, and Martin Vechev. Certified training: Small boxes are all you need. In International Conference on Learning Representations, 2023. [28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019. [29] Zhouxing Shi, Yihan Wang, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Fast certified robust training with short warmup. In Advances in Neural Information Processing Systems, 2021. [30] Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract domain for certifying neural networks. Proc. ACM Program. Lang., 3(POPL), 2019. [31] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. [32] Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019. [33] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems, 2020. [34] Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. In Advances in Neural Information Processing Systems, 2021. 13

[35] Eric Wong and J. Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference of Machine Learning, 2018. [36] Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems, 2020. [37] Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. Fast and Complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In International Conference on Learning Representations, 2021. [38] Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: A boolean function perspective. In Advances in Neural Information Processing Systems, 2022. [39] Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Boosting the certified robustness of l-infinity distance nets. In International Conference on Learning Representations, 2022. [40] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems, 2018. [41] Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. In International Conference on Learning Representations, 2020. [42] Huan Zhang, Shiqi Wang, Kaidi Xu, Linyi Li, Bo Li, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. General cutting planes for bound-propagation-based neural network verification. In Advances in Neural Information Processing Systems, 2022. [43] Minxing Zhang, Michael Backes, and Xiao Zhang. Generating less certain adversarial examples improves robust generalization. Transactions on Machine Learning Research, 2025. [44] Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. In International Conference on Learning Representations, 2022. [45] Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. In International Conference on Computer Vision, 2021.

14

A

Interval Bound Propagation (IBP)

IBP has been at the core of all recent state-of-the-art certified training methods [6, 7, 8, 23, 27, 29]. Here, we formalise the bound propagation for a single layer, as proposed by Gowal et al. [13]. For an affine layer (e.g., convolutional or linear layer) represented by hk (zk−1 ) = W zk−1 + b, we can define the bound calculation as follows: z̄k−1 + z k−1 µk−1 = , µk = W µk−1 + b, (17) 2 z̄k−1 − z k−1 rk−1 = , rk = |W |rk−1 , (18) 2 (19) z k = µk − rk , z̄k = µk + rk , where µ denotes the centre of the interval box, r its radius, and z k , z̄k denote lower and upper bounds of the layer output, respectively. When hk is an element-wise increasing activation function, e.g., ReLU, sigmoid, or tanh, we instead trivially propagate the bounds as z k = hk (z k−1 ) and z̄k = hk (z̄k−1 ). For ReLU networks, an unstable ReLU activation implies the ReLU activation may be either active or inactive given the input interval, i.e., z k−1 < 0 < z̄k−1 . Hence, incomplete verifiers must over-approximate the possible activation outputs. Figure 3 compares the loose IBP box relaxation with the tighter convex hull relaxation for an unstable ReLU. ReLU(z)

z k−1

ReLU(z)

z̄k

z̄k

z

z

z k−1

z̄k−1

(a) IBP Relaxation

z̄k−1

(b) Convex Hull

Figure 3: Comparison of IBP relaxation and the tighter convex hull relaxation. The shaded regions represent convex over-approximations of an unstable ReLU function. The dashed line represents the actual possible activation outputs.

B

Deferred Proofs of Theoretical Results

In this section, we provide proofs for each of the theoretical results from our methodology section (§3) of the main text. In each case, we restate the claim for the ease of the reader. Proposition B.1 (Equivalence of AD and soft-label AT). Let xadv be fixed, and let the teacher predictive distribution Pθt (x) be fixed. Then the AD objective: LAD := KL(Pθt (x) ∥ Pθs (xadv )) ,

(20)

is equal, up to an additive constant independent of the student parameters, to AT with soft teacher labels, namely: Lsoft-AT := LCE (fθs (xadv ), Pθt (x)) . (21) Moreover, both objectives induce identical backpropagation updates with respect to the student logits. In particular, for each class index i: ∂LAD ∂Lsoft-AT = [Pθs (xadv )]i − [Pθt (x)]i = . ∂zθi s ∂zθi s

(22)

Proof. For brevity, we let zθi s := [fθs (xadv )]i and Pθis := [Pθs ]i denote the i-th entry of the raw logit outputs and predictive distributions of the student classifier, respectively. First, recall the derivative of 15

the softmax function: ∂P j = ∂z i



P i (1 − P i ), −P i P j ,

i = j, i ̸= j.

(23)

For ease of notation, we can rewrite this in terms of the Kronecker delta function: ∂P j = P j (δij − P i ), ∂z i

 where δij =

1, 0,

i = j, i ̸= j.

(24)

Now, we compare the derivatives of LAD and Lsoft-AT with respect to zθi s . Given that LAD =  j  PC PC PC Pθ (x) j t = j=1 Pθjt (x) · log(Pθjt (x)) − j=1 Pθjt (x) · log(Pθjs (xadv )), j=1 Pθt (x) · log P j (xadv ) θ PC s we note that the j=1 Pθjt (x) · log(Pθjt (x)) term is independent of student parameters, so: 



C X

∂LAD ∂ = i − Pθjt (x) log(Pθjs (xadv )) , ∂zθi s ∂zθs j=1

(25)

∂Pθjs (xadv ) , ∂zsi Pθj (xadv )

(26)

=−

C X j=1

=−

=−

C X

Pθjt (x)

1

s

Pθjt (x)

j j=1 Pθs (xadv ) C X

Pθjs (xadv )(δij − Pθis (xadv )),

Pθjt (x)(δij − Pθis (xadv )),

(27)

(28)

j=1

= −Pθit (x) + Pθis (xadv )

C X

Pθjt (x),

(29)

j=1

= Pθis (xadv ) − Pθit (x).

(30)

Finally, recall that Lsoft-AT = LCE (fθs (xadv ), Pθt (x)), and, C X

LCE (fθs (xadv ), Pθt (x)) = −

Pθjt (x) log(Pθjs (xadv )).

(31)

j=1

Hence, the result falls immediately from Equations (25)-(30), and by extension, the two objectives must only differ by an additive constant independent of θs . Proposition B.2 (Analytical form of AD-CERT). Let H(P ) and H(Q, P ) denote the entropy and cross entropy respectively for probability distributions P and Q. That is: H(P ) = −

C X

pi log(pi ),

H(Q, P ) = −

i=1

C X

qi log(pi ).

(32)

i=1

Then, LAD-CERT from Equation (10) can be re-written as: LAD-CERT = (1 − α)H(Pθt (x), Pθs (xadv )) + αLIBP − (1 − α)H(Pθt (x)).

(33)

Therefore, up to a constant independent of θs , AD-CERT is a scalarisation between (i) a soft-label adversarial lower bound surrogate and (ii) a certified IBP upper bound. 16

Proof. The proof is immediate since we can write: LKL (Pθt (x) ∥ Pθs (xadv )) =

C X

Pθit (x) · log

i=1

=

C X

Pθit (x) Pθis (xadv )

! ,

(34)

Pθit (x) · log(Pθit (x)) − log(Pθis (xadv )) ,

(35)

i=1

=−

C X

Pθit (x) log(Pθis (xadv )) +

i=1

C X

Pθit (x) log(Pθit (x)),

(36)

i=1

= H(Pθt (x), Pθs (xadv )) − H(Pθt (x)).

(37)

Hence, substituting this identity into Equation (10) gives: LAD-CERT = (1 − α)H(Pθt (x), Pθs (xadv )) + αLIBP − (1 − α)H(Pθt (x)),

(38)

as required. ∗ Lemma B.3. For AD, the optimal logit margin, ∆zij , between any two classes i, j is:   [Pθt (x)]i ∗ ∆zij = log . [Pθt (x)]j

(39)

Moreover, standard AT has no finite minimiser, and its infimum 0 is approached only as [fθs (xadv )]y − [fθs (xadv )]j → +∞, ∀j ̸= y. Proof. First note that, from Equation (30), LAD is minimised iff Pθs = Pθt , or equivalently, Pθis (xadv ) = Pθit (x) ∀ i ∈ {1, . . . , C}. Rewriting this using the definition of softmax, we have: i

ezs PC

j

zs j=1 e

= Pθit (x)

∀ i ∈ {1, . . . , C},

(40)

where zθi s = [fθs (xadv )]i , which implies:     C C X X
j j zsi = log Pθit (x) · ezs  = log Pθit (x) + log  ezs  j=1

(41)

j=1

Then, we can write the optimal logit for a given class i as zsi∗ := log(Pθit (x)) + c for some shared constant, c ∈ R. Since softmax is invariant to adding a constant to each logit, the optimal logit margin between any two classes i and j is: ! Pθit (x) j ∗ i∗ j∗ i ∆zij := zs − zs = log(Pθt (x)) − log(Pθt (x)) = log , (42) Pθjt (x) which is finite since Pθit (x) > 0 ∀ i. In contrast, standard AT loss is defined by LAT = − log(Pθys (xadv )), with y denoting the index of the true class label here. Hence, we have:     ! y C X X j y j y ezs LAT = − log PC = log  ezs −zs  = log 1 + ezs −zs  . (43) zsj e j=1 j=1 j̸=y j

y

For any finite logits, we have ezs −zs > 0, and hence X j y 1+ ezs −zs > 1 =⇒ LAT > 0.

(44)

j̸=y

Therefore, LAT has no finite minimiser. Since we have established that LAT > 0 for all finite logits, its global minimum is never attained. However, we can show that its infimum is exactly 0. Let 17

m = min(zsy − zsj ) denote the minimum logit margin of the correct class over all incorrect classes. j̸=y

We can bound the loss as follows: 



0 < LAT = log 1 +

X

e

−(zsy −zsj )

 ≤ log 1 + (C − 1)e−m ,

(45)

j̸=y

where C is the total number of classes. Taking the limit as the minimum margin grows to infinity (m → +∞), we have:
lim log 1 + (C − 1)e−m = log(1 + 0) = 0. (46) m→+∞

Thus, inf LAT = 0. This infimum is not reachable for any finite network weights, but is approached asymptotically as the correct-class logit margins tend to positive infinity.

C

Additional Results

In this section, we present additional theoretical results complementary to those in the main text. Proposition C.1 (Teacher-margin transfer in logit space). Assume the teacher fθt predicts the true class y on the natural input x. Then, if: ∥fθs (xadv ) − fθt (x)∥∞ <

1 min[zθ (x, y)]j , 2 j̸=y t

(47)

it follows that argmax[fθs (xadv )]i = y. i

Proof. First, define δ := ∥fθs (xadv ) − fθt (x)∥∞ . Then, −δ ≤ [fθs (xadv )]i − [fθt (x)]i ≤ δ, for any class index i. Which means, [fθs (xadv )]y ≥ [fθt (x)]y − δ and [fθs (xadv )]i ≤ [fθt (x)]i + δ for any i ̸= y. Moreover: =⇒ [fθs (xadv )]y − [fθs (xadv )]i ≥ [fθt (x)]y − [fθt (x)]i − 2δ, =⇒ [zθs (xadv , y)]i ≥ [zθt (x, y)]i − 2δ.

(48) (49)

Hence, if δ < 12 mini̸=y [zθt (x, y)]i , then clearly [zθt (x, y)]i − 2δ > 0, which implies [zθs (xadv , y)]i > 0 and argmax[fθs (xadv )]i = y. i

Lemma C.2. Let LAT be bounded by some U ∈ R and U > 0, i.e., LAT = LCE (fθs (xadv ), y) = − log(Pθys ) ≤ U,

(50)

where y ∈ {1, . . . , C} is the true label index and Pθys = [softmax(fθs (xadv ))]y . Then, for every j ̸= y, the corresponding true-vs-false logit margin satisfies

zsy − zsj ≥ − log(eU − 1).

(51)

Proof. Notice that AT loss can be written as: LAT = − log

PC

= log 1 +

k

zs k=1 e





!

y

ezs

X

e

zsj −zsy

.

(52)

j̸=y

Now assume that LAT ≤ U . Then, 



log 1 +

X

e

zsj −zsy

 ≤ U,

(53)

j̸=y

=⇒ 1 +

X

j

y

j

y

ezs −zs ≤ eU ,

(54)

j̸=y

=⇒

X

ezs −zs ≤ eU − 1.

j̸=y

18

(55)

Since each term in the sum is non-negative, it follows that for every j ̸= y, we have: j

=⇒ =⇒

y

ezs −zs ≤ eU − 1,

(56)

zsj − zsy ≤ log(eU − 1), zsy − zsj ≥ − log(eU − 1),

(57) ∀j ̸= y,

(58)

as required.

D

Additional Experiments

In this section, we evaluate the sensitivity of AD-CERT and CC-DIST [6] to teachers trained under different adversarial training methods and schedules. The sensitivity results from Table 5 suggest that the choice of teacher can affect the standard and adversarial accuracy of the resulting students, but the certified accuracy is comparatively stable across teacher variants, particularly for AD-CERT. Here, short-cycle refers to the training procedure described in De Palma [6], where we use the SGD optimiser with momentum set to 0.9, a 30-epoch training cycle with a cyclic learning rate, which linearly increases from 0 to 0.2 during the first half of the training, and then decreases back to 0. Long-cycle corresponds to the standard AT settings used in the CTBench library [25], where PGD [22] and EDAC [43] models are trained for 240 epochs using the Adam optimiser. Short-cycle teachers tend to slightly improve the certified results of AD-CERT and CC-DIST, which coincides with the observations of De Palma [6] in their work. In general, CC-DIST appears to benefit more noticeably from short-cycle teachers, with AD-CERT being slightly less sensitive. Nevertheless, AD-CERT remains consistently stronger in certified and AA accuracy across all teacher protocols, while CC-DIST tends to achieve slightly higher standard accuracy in most settings, consistent with previous observations in §4.2. 8 Table 5: Sensitivity analysis of AD-CERT and CC-DIST on CIFAR-10 at ε = 255 with different teacher training protocols.

Teacher

E

Teach. Std. [%]

Teach. AA [%]

Method

Std. [%]

AA [%]

Cert. [%]

53.17 52.35

35.95 34.80

35.21 33.75

PGD-Long

78.71

35.93

AD-CERT CC-DIST

PGD-Short

76.14

42.55

AD-CERT CC-DIST

53.90 54.44

36.52 36.05

35.61 34.81

EDAC-Long

78.95

42.48

AD-CERT CC-DIST

53.49 54.32

36.46 35.69

35.60 34.55

EDAC-Short

73.55

41.99

AD-CERT CC-DIST

53.69 54.33

36.69 35.76

35.84 34.68

P SEUDO - CODE

In this section, we outline the pseudocode for the training procedure of AD-CERT (§3.1) in Algorithm 1.

F

E XPERIMENTAL S ETUP

F.1

Dataset

We conduct experiments on MNIST [21], CIFAR-10 [19], and TinyImageNet [20]. These datasets are open-source and freely available, with unspecified licenses. We follow the data preprocessing directly available in the CTBench library [25]. No preprocessing is applied to MNIST. CIFAR-10 and TinyImageNet are normalised using the dataset mean and standard deviation, and random horizontal flips are applied. For CIFAR-10, additional random cropping to 32 × 32 is applied after zero-padding each image by 2 pixels on all sides. For TinyImageNet, random cropping to 64 × 64 is applied 19

Algorithm 1 AD-CERT Training Procedure Require: PGD teacher fθt , data D, perturbation radius ϵtrain , epochs Nwarm , N , IBP coefficient α 1: Initialisation 2: Initialise student fθs with parameters θs 3: Detach teacher fθt gradients from training

▷ Shi et al. [29] initialisation

4: Training 5: for epoch = 1 to N do 6: if epoch ≤ Nwarm then 7: Update ϵcurr and αcurr 8: end if 9: for (x, y) in D do 10: Compute xadv via PGD attack 11: Compute IBP bounds z∆ θs (x, y) using ϵcurr

▷ ramp-up to ϵtrain , α

Ldist ← KL (softmax(fθt (x)) ∥ softmax(fθs (xadv ))) Lrobust ← LCE (−z∆ θs (x, y), y) 14: Ltotal ← (1 − αcurr )Ldist + αcurr Lrobust 15: Update θs ← θs − η∇θs Ltotal 16: end for 17: end for 18: return fθs 12: 13:

after zero-padding each image by 4 pixels on all sides. We train on the corresponding training sets and certify on the validation sets, following common practice in the certified training literature [6, 8, 23, 25, 27, 29]. F.2

Model Architectures

We use the standard CNN7 architecture, a convolutional network consisting of seven convolutional and linear layers. Each layer, except for the final linear layer, is followed by Batch Normalisation and a ReLU activation. This architecture has been shown to perform consistently well across settings [25, 29], and is therefore widely adopted in the certified training literature [6, 8, 23, 25, 27, 29]. For TinyImageNet, we double the stride of the final convolutional layer to reduce computational cost. F.3

Training Details

Implementation We implement AD-CERT in CTBench using PyTorch. The training loss follows the CTBench decomposition into a natural loss, a robust loss, and regularisation terms. For AD-CERT, the robust loss is given by Equation 10, where the AD term uses a fixed adversarially trained teacher, and the certified term is computed with IBP bounds. Unless stated otherwise, the teacher and student use the same CNN7 architecture. Initialisation Adversarial teacher models are initialised by Kaiming uniform [14], while certified student models are initialised using the IBP initialisation from Shi et al. [29]. Training Schedule We follow the standard CTBench training schedule. Certified models are first trained for one epoch with ε = 0, and then use a warm-up phase where ε is smoothly increased from 0 to the target value. The warmup phase is 20 epochs for MNIST with ε = 0.1 and ε = 0.3, 2 8 80 epochs for CIFAR-10 with ε = 255 , 120 epochs for CIFAR-10 with ε = 255 , and 80 epochs for 1 TinyImageNet with ε = 255 . We use the IBP regularisation proposed by Shi et al. [29], with weight 0.5 on MNIST and CIFAR-10, and 0.2 on TinyImageNet, during warmup. In total, we train for 70 2 8 epochs on MNIST, 160 epochs on CIFAR-10 with ε = 255 , 240 epochs on CIFAR-10 with ε = 255 , and 160 epochs on TinyImageNet. Optimisation In general, we follow the exact optimisation setup from CTBench. We use Adam [18] with a learning rate of 5 × 10−4 . The learning rate is decayed by a factor of 0.2 at epochs 50 and 20

2 60 for MNIST, epochs 120 and 140 for CIFAR-10 with ε = 255 , epochs 200 and 220 for CIFAR-10 8 with ε = 255 , and epochs 120 and 140 for TinyImageNet. We use a batch size of 256 for MNIST and 128 for CIFAR-10 and TinyImageNet. Gradients are clipped to 10 in ℓ2 norm before every optimiser step. No weight decay is applied, and L1 regularisation is applied only to the weights of linear and convolutional layers. Batch normalisation follows the CTBench population-statistics setting. We apply Stochastic Weight Averaging (SWA) using the settings in accordance with MTL-IBP’s training setup in CTBench since both methods use an unsound robust objective, making direct model selection on the robust loss impractical.

Teacher Models We use the pre-trained PGD checkpoints provided by CTBench as teacher models, 8 except for CIFAR-10 with ε = 255 , where our ablation teacher sensitivity analysis from Table 5 showed that a 30-epoch short-cycle PGD model trained with a cyclic learning-rate schedule exactly as described by De Palma [6] alongside an EDAC [43] step size of 0.3, gives marginally better performance. All teachers are kept fixed during AD-CERT training. Tuning of Hyperparameters In general, we perform minimal hyperparameter tuning due to the computational overhead of running complete verification. We ran initial experiments following the exact hyperparameters and settings used in CTBench [25] for the MTL-IBP [8] method. For the tuning of α, we use a manually selected search range guided by some reported quantities in the CTBench library, namely, unstable ReLU ratio, IBP certification rate, and standard, adversarial, and certified accuracy. These metrics help indicate whether the objective places too much or too little weight on the IBP branch, allowing us to adjust α accordingly. We find that slightly reducing 1 α is beneficial on MNIST at ε = 0.1 and on TinyImageNet at ε = 255 . For TinyImageNet, we also observe improvements from increasing wrob and enlarging the PGD attack region used in the empirical branch. The final hyperparameters used for AD-CERT across all settings are reported in Table 6. Table 6: Best hyperparameters for AD-CERT across dataset settings. MNIST

F.4

CIFAR-10

TinyImageNet

Hyperparameter

0.1

0.3

2 255

8 255

1 255

L1 regularisation wrob IBP coefficient (α) Train ε PGD steps Attack range scale

1 × 10−6 0.7 7.5 × 10−3 0.2 10 1.0

1 × 10−6 1.0 0.5 0.3 1 1.0

3 × 10−6 1.0 0.01

0.0 1.0 0.5

5 × 10−5 0.9 5 × 10−3

1 255

8 255

1 255

8 2.0

1 1.0

1 2.0

Certification Details

Certification is performed within CTBench [25] using its wrapper for the α, β-CROWN verification library. For each example, we first check whether the model predicts the correct clean label. Incorrectly classified examples are marked as uncertified. For correctly classified examples, we run AUTOATTACK [4] and if an adversarial counterexample is not found, we attempt certification using a sequence of progressively stronger verifiers. Namely, IBP [13], CROWN [40], α-CROWN [37], and finally, the complete α, β-CROWN branch-and-bound verifier [34]. Each stage is only invoked when the previous stage fails to certify the example. We use a timeout of 400 seconds for the α, β-CROWN branch-and-bound stage and an overall timeout of 1000 seconds per example. F.5

Computational Setup & Cost

Table 7 reports the training and certification runtime of AD-CERT under the training and certification setups described in Sections F.3 and F.4, respectively. All experiments were run on an internal cluster with access to NVIDIA V100, A100, A100 MIG slices, and H100 GPUs. Training jobs were run on either 2g.20GB or 3g.40GB A100 MIG slices, using 8 CPU cores. For certification, we used MIG slices in most cases, but occasionally used full A100 or H100 GPUs for harder settings, such as 2 CIFAR-10 with ε = 255 and TinyImageNet. For clarity, Table 7 reports runtimes for experiments 21

run on a 3g.40GB A100 MIG slice. Training experiments used 64GB of RAM, while certification experiments used 128GB. The training times range from ∼ 1 hour on the simplest MNIST setting, to ∼ 1 day on TinyImageNet. Certification times range from ∼ 12 minutes to over 3 days. Additionally, we report the training cost complexity associated with an AD-CERT student, assuming a pre-trained teacher, compared to previous adversarial and certified training methods [6, 8, 13, 22, 23, 27, 43] in Table 8. On a high level, AD-CERT corresponds to performing IBP and PGD, with the small added cost of a single forward pass of a teacher model that is detached from training. Note that Table 8 is a direct extrapolation of the table provided in the original CTBench paper [25] with added entries for AD-CERT and CC-DIST. Table 7: Training and certification time for AD-CERT on different datasets and ε. Dataset

ε

Train Time (seconds)

Certification Time (seconds)

MNIST

0.1 0.3

1.20 × 104 4.08 × 103

7.67 × 102 5.32 × 104

2 255 8 255 1 255

2.66 × 104 1.59 × 104

2.61 × 105 3.58 × 104

8.97 × 104

1.54 × 105

CIFAR-10 TinyImageNet

Table 8: Detailed breakdown of training costs for each certified training method. Method

Training cost per batch

Details

Standard PGD / EDAC IBP SABR MTL-IBP AD-CERT† CC-DIST† TAPS STAPS

T (M + 1)T 2T (M + 2)T (M + 2)T (M + 2)T + F (M + 2)T + Fh 2t + K(M + 1)(T − t) 2t + K(M + 1)(T − t) + (M + 1)T

Forward + backward M attack steps + standard loss computation Lower and upper bounds propagation IBP + PGD IBP + PGD IBP + PGD + detached teacher forward CC-IBP + PGD + detached teacher forward on feature split IBP for first split and PGD for second split for each class TAPS + PGD

Legend T F Fh M K t

Time cost for standard training, including forward + backward pass Time cost for a forward pass through the teacher network Time cost for a forward pass through the teacher feature extractor network Number of adversarial attack steps, including repeats Number of classes Time cost for standard training in the first network split in TAPS

† Added entries from original table in CTBench [25].

G

CC-IBP Distillation (CC-DIST)

De Palma [6] formulate the CC-DIST loss function as a linear combination of an expressive robust distillation loss in the feature space, Rfθ (α; x, y), and their prior expressive loss formulation, CC-IBP [8]. Let fθ : Rd 7→ RC be a classification model as expressed earlier in §2. Then, we can rewrite fθ as a composition of a feature extractor hθ : Rd 7→ Rk and a classification head gθ : Rk 7→ RC , yielding fθ = gθ ◦ hθ (note that we abuse notation of θ for brevity). Then, the expressive robust distillation loss over the feature space is defined by: Rfθ (α; x, y) :=

k X

max

n

[hθ (α; x)]i − [hθt (x)]i

2

2

, ([hθ (α; x)]i − [hθt (x)]i )

o

,

(59)

i=1

where hθ (α; x) := (1 − α)hθ (xadv ) + αhθ (x) and hθ (α; x) := (1 − α)hθ (xadv ) + αhθ (x), with hθ (x) and hθ (x) denoting the IBP upper and lower bounds of hθ (x), respectively. Note that hθt denotes the feature extractor of a pre-trained PGD teacher model. 22

Now, let β be the distillation coefficient, determining the relative weight of Rfθ (α; x, y) from Equation (59). Omitting any regularisation, the training loss for CC-DIST takes the following form: LCC-DIST (α, β; x, y) := LCC-IBP (α; x, y) + β Rfθ (α; x, y), (60) fθ fθ where LCC-IBP (α; x, y) is the CC-IBP loss from De Palma et al. [8]. Denoting k as the dimensionality fθ of the feature space, β = 5/k was used for the majority of experiments.

23