Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography, security, privacy, cybersecurity

Adversarial Frontiers: Minimum-Norm Attack Ensembles for Robustness Evaluation Luca Scionisa,b,1 , Luca Melisa,b,1 , Maura Pintorb,∗, Fabio Braub , Ambra Demontisb , Giorgio Fumerab , Fabio Rolib,c , Battista Biggiob a Sapienza University of Rome, Italy b University of Cagliari, Italy c University of Genoa, Italy

Abstract Adversarial robustness is commonly evaluated with predefined attack ensembles, such as AutoAttack, at a single perturbation budget ε and on a selective choice of perturbation norms. We argue this formulation is fundamentally limited. First, robustness– perturbation curves may intersect or decay at different rates across models, making single-ε rankings unstable. Second, current

arXiv:2607.19855v1 [cs.LG] 22 Jul 2026

ensembles provide no evidence of optimality, leaving an unknown gap to worst-case performance. Third, fixed attack configurations provide no systematic control over the trade-off between attack strength and evaluation cost. To address these limitations, we introduce a unified evaluation framework based on a comprehensive pool of minimum-norm attacks and robustness–perturbation curves across ℓ0 , ℓ1 , ℓ2 and ℓ∞ norms. We define the attack frontier as the worst-case robustness estimate the attack pool produces

against a model. We then formalize evaluation as a frontier-approximation problem, constructing minimum-norm attack ensembles, optimized subsets of the comprehensive pool, that approach the frontier under a controllable query budget, with larger budgets monotonically tightening the estimate. Furthermore, we define the defense frontier as the maximum robustness across the model set at each perturbation size. We finally propose the Defense Optimality Index to rank defenses by their gap to the defense frontier, providing a ranking without selecting a reference ε. On CIFAR-10 and ImageNet, our ensembles match or exceed AutoAttack on most defenses at every budget tier, at fixed and controllable query cost, offering practitioners a query-controlled, curve-based alternative to fixed-ε evaluation. 1. Introduction Deep neural networks (DNNs) are vulnerable to adversar-

ial examples [1, 2], carefully-perturbed inputs that induce misclassification. Reliably measuring how robust a model is to such perturbations is critical for deployment. The main metric adopted for this evaluation is the robust accuracy, i.e., the fraction of inputs correctly classified by a model against adversarial perturbations bounded by a specific ℓ p -norm and a maxi-

mum perturbation size ε. Common evaluation frameworks rely on adversarial attacks to estimate this metric by perturbing each input to search for a misclassifying perturbation. Yet, a single weak attack may fail to identify a perturbation even when one exists, overestimating robustness. Therefore, state-of-theart frameworks use ensembles of diverse attacks to compensate for individual failures, yielding more reliable robustness estimates. The most relevant example adopting this approach is RobustBench [3], a common state-of-the-art benchmark of ro-

∗ Corresponding author

bust models, whose ranking relies on AutoAttack (AA) [4], a

Email address: [email protected] (Maura Pintor) 1 These authors contributed equally to this work.

fixed ensemble of four sequential attacks. This methodology,

Robust accuracy ρ(‖δ‖p )

εref

nor their relative cost budgets. AA is parameter-free and pre-

Model fA Model fB

configured by design, but this means evaluation quality and computational cost are coupled and non-adjustable. A practitioner with a small budget cannot obtain an approximate evaluation, while one with a large budget cannot improve beyond the

Perturbation size ‖δ‖p

fixed pipeline.

Figure 1: The limitation of fixed-ε evaluations. This plot shows the robustness–perturbation curves, ρ(ε), for two hypothetical models, whose robustness decays at different rates. At a fixed reference budget εref , the red model is more robust than the blue one. Yet, the latter appear stronger at larger ε. In Section 5 we demonstrate empirically how single-point evaluations are incomplete, motivating the need for ε-free evaluation frameworks.

We address the aforementioned limitations with the following contributions: (i) We introduce a unified evaluation framework based on minimum-norm adversarial attacks and robustness–

however, is characterized by the following limitations:

perturbation curves to characterize robustness across the ℓ0 ,

(i) Robustness is a function, not a point. The complete ro-

ℓ1 , ℓ2 , and ℓ∞ perturbation norms. We call attack frontier the

bustness profile of a model is defined by the robustness–

worst-case robustness estimate any given combination of at-

perturbation curve, which plots the robust accuracy as a func-

tacks can produce.

tion of the perturbation size ε, for a given ℓ p -norm. Best-

(ii) We formalize robustness evaluation as a frontier-

practice guidelines have long recommended reporting the com-

approximation problem. Starting from a pool of minimum-

plete curve [5, 6, 7]. However, current benchmarks evaluate

norm attacks and building on the Attack Optimality Index of At-

models at a single, arbitrary ε. As shown in Figure 1, this is in-

tackBench [8], we construct near-optimal attack ensembles, i.e.,

herently problematic, since robustness–perturbation curves can

specific subsets of Attacks taken from the initial pool. These

intersect or decay at different rates. The selective choice of

ensembles are built to approximate the empirical attack fron-

the perturbation norm further aggravates this problem. Each

tier as closely as possible under any given query budget, con-

ℓ p norm captures different types of attack. A model robust to

sistently across all the models we evaluated. Importantly, this

dense perturbations spread across the input (ℓ2 , ℓ∞ norm) might

approach measures how tight the approximation of the attack

be substantially weaker to sparse perturbations (ℓ0 , ℓ1 ). Robust-

frontier is, for each target model. We demonstrate that increas-

ness to sparse perturbations is not covered by available bench-

ing the query budget systematically reduces this gap, thereby

marks.

establishing a principled trade-off between computational cost

(ii) It is not possible to evaluate how close the attack is to the

and the tightness of the approximation.

worst case. AA is provided with no tools to check how close the

(iii) We introduce the defense frontier and the Defense Opti-

robust accuracy it finds is to the empirical optimum (obtained

mality Index (DOI) as a defense-side counterpart to the attack

considering also other attacks that are not in AA). A high ro-

frontier. Analogous to the attack frontier, we define the defense

bust accuracy may therefore reflect a strong defense or simply

frontier as the empirical upper bound of robustness achiev-

a weak attack, and the two are indistinguishable within the cur-

able across a given pool of evaluated models. This provides

rent framework.

an objective baseline against which any individual defense can

(iii) The pipeline cannot be adapted by changing the attacks

be measured. We then formalize the gap between a model’s 2

robustness–perturbation curve and the defense frontier as the

Section 3. Section 4 extends this framework by introducing the

Defense Optimality Index (DOI). Unlike the robust accuracy

defense frontier and the Defense Optimality Index. We validate

at a single ε, the DOI aggregates this gap across the entire per-

our approach in Section 5, discuss related work in Section 6,

turbation range. This yields a stable, ε-independent ranking of

and conclude in Section 7.

defenses. 2. Background

(iv) We leverage our framework to establish a new, budgetaware evaluation benchmark.

Building on our frontier-

We first discuss the main tool that should drive [5, 6, 9, 7]

approximation strategy, we construct attack ensembles approxi-

robustness evaluations: the robustness–perturbation curve. Re-

mating the frontier for the ℓ0 , ℓ1 , ℓ2 , and ℓ∞ perturbation norms,

liably evaluating a model’s robustness requires characterizing

at three nested query budget tiers. We deploy these ensem-

how it degrades at increasing perturbation strength. This is

bles to compute the DOI of existing defenses, proposing the

what the robustness–perturbation curve captures, reporting the

resulting benchmark as a principled, budget-aware alternative

fraction of inputs that remain correctly classified as the allowed

to fixed-pipeline evaluations. Evaluation quality is thus decou-

perturbation grows.

pled from computational cost, allowing practitioners to trade-

We then introduce the tools we use to compute the

off their query budget against the tightness of the robustness

robustness–perturbation curve, i.e., minimum-norm adversarial

estimate.

attacks. Since these attacks are empirical, they yield only an

We evaluate our approach on 13 held-out defenses on

estimate of the curve. Given that, we ground the evaluation on

CIFAR-10 and ImageNet. Our minimum-norm attack ensem-

the notion of optimality, first introduced by AttackBench [8], to

bles match or exceed AA’s optimality on the majority of mod-

assess how effectively any given attack estimates it.

els at every budget tier, with the margin generally improving

Notation. Let fθ : X → RC denote a classifier with parameters

as queries increase. The advantage is largest in ℓ2 , where our

θ, where X = [0, 1]d is the input domain and Y = {1, . . . C} is

ensemble already matches or exceeds AA on 12/13 defenses at

the set of class labels. Given an input x ∈ X, the model outputs

Q = 4k queries, despite AA spending up to 7 566 queries at

a vector of logits fθ (x) = [ f1 (x), . . . , fC (x)], and the predicted

its reference ε. In ℓ1 it climbs from 8/13 at Q = 4k to 13/13

class is taken as arg maxc fc (x). Given the true label y ∈ Y,

at Q = 12k, and in ℓ∞ it reaches 10/13. We also show that

an adversarial example x′ = x + δ satisfies arg maxc fc (x′ ) ,

AA rankings are highly unstable: varying ε beyond the Robust-

y, with x′ ∈ X. The perturbation size is measured by the ℓ p

Bench reference reorders nearly every defense. On CIFAR-10,

norm ∥δ∥ p , where p ∈ {0, 1, 2, ∞}.

17 defenses shift by at least 5 positions under ℓ2 , and on Ima-

The robustness–perturbation curve. Standard evaluations re-

geNet all 10 models shift by at least 2 positions under ℓ∞ . The

port the robust accuracy at a fixed budget ε, i.e., the fraction

DOI, instead, produces a single, ε-independent ranking.

of samples whose prediction remains correct under any per-

The remainder of the paper is organized as follows. After re-

turbation of size at most ε. Varying ε traces the complete

viewing minimum-norm attacks and the attack frontier in Sec-

robustness–perturbation curve ρ(ε; fθ ), which characterizes ro-

tion 2, we formalize the budget allocation problem and propose

bustness across a wide range of budgets rather than at a single

a greedy approach for constructing optimal attack ensembles in

operating point. Formally, let δ⋆ (x, fθ ) denote the minimal per3

as:

turbation sufficient to induce misclassification for a given input

ρ(ε; fθ ) =

x. The ideal robustness–perturbation curve evaluated over a

1 X I[d(x; fθ ) > ε] . |D| x∈D

(3)

dataset D of input–label pairs (x, y) ∈ D is then defined as: 2.2. Empirical Attack Frontier and Attack Optimality ρ⋆ (ε; fθ ) =

1 X I[ ∥δ⋆ (x, fθ )∥ p > ε ] , |D| (x,y)∈D

Minimum-norm attack implementations, however, come

(1)

with no measure of how effective they are: different solvers,

where I[·] is the indicator function, i.e., 1 if the condition is met

or even different configurations of the same one, may return

and 0 otherwise.

different solutions for the same sample. Comparing them, and knowing how close any comes to the smallest achievable pertur-

2.1. Minimum-norm Adversarial Attacks

bation, requires a quantitative notion of how close to optimality

To estimate the robustness–perturbation curve, and in partic-

an attack is. AttackBench [8] provides a quantitative measure of

⋆

ular the minimal distance ∥δ (x, fθ )∥ p , we use minimum-norm

optimality that is, by definition, relative to a predefined pool A

attacks.

of N attacks. Concretely, it first constructs the attack frontier.

Attack objective. For a model fθ , the goal of a minimum-norm

Given a model fθ and a fixed query budget Q (i.e., the maxi-

adversarial attack a is to find the smallest perturbation that in-

mum allowed forward and backward passes through the model

duces misclassification:

an attack can make), the frontier is defined as the per-sample

δ⋆ ∈ arg min δ

s.t.

minimum distance da⋆ (x) = mina∈A da (x) evaluated over each

∥δ∥ p

sample x ∈ D. Here, a⋆ represents the resulting per-sample

arg max fc (x + δ) , y,

(2)

optimal attack under the given budget constraint. The curve

x + δ ∈ X.

ρa⋆ (ε; fθ ) is the empirical robustness–perturbation curve relative to A, the tightest estimate of ρ⋆ (ε; fθ ), obtained using all

The constraint arg maxc fc (x + δ) , y asks only for some mis-

attacks in the pool. The closer an attack’s own curve lies to

classification, without fixing the resulting class; we call this the

this frontier, the higher the optimality. This is formalized as the

untargeted formulation. A targeted run instead drives the opti-

Attack Optimality Index (AOI) below.

mization toward a chosen class t , y. As any δ that satisfies the

Let ρa (ε; fθ ) denote the robustness–perturbation curve ob-

constraint is still a valid solution, targeting serves only to guide

tained by attack a against model fθ . The overall effectiveness

the trajectory, and any misclassification it produces remains ad-

of a is quantified by the Area Under the Robustness Evaluation

missible.

Curve (AUREC):

This problem has no closed-form solution and is typically solved with iterative gradient-based solvers, which use the gra-

AURECa ( fθ ) =

Z εmin ( fθ )

ρa (ε; fθ ) dε,

(4)

dient of Eq. (2) to refine the perturbation. Each returns a

distance that upper-bounds the true minimal norm, d(x; fθ ) ≥

where εmin ( fθ ) = min{ε ≥ 0 : ρa⋆ (ε; fθ ) = 0} is the smallest

⋆

∥δ (x, fθ )∥ p . Substituting d(x; fθ ) for ∥δ (x, fθ )∥ p in Eq. (1)

perturbation budget at which the frontier reaches zero robust

yields the empirical robustness–perturbation curve we compute

accuracy. Stronger attacks have smaller AUREC, i.e., they succeed with smaller budgets. 4

The per-model Attack Optimality Index (AOI) of attack a is

the same allocation regardless of its strength, so queries spent

then obtained by comparison with a⋆ :

on weaker or redundant attacks are effectively wasted. An example can be found in our experiments (Table 3): some attacks

ρ(0; fθ ) εmin ( fθ ) − AURECa ( fθ ) AOI(a, fθ ) = , ρ(0; fθ ) εmin ( fθ ) − AURECa⋆ ( fθ )

(5)

are selected with 100-200 queries rather than using the maximum budget of 1000, as optimizing further rarely improves the

where ρ(0; fθ ) is the clean accuracy of the model. By construc-

perturbation found for a sample. A uniform allocation would

tion, AOI(a, fθ ) ∈ [0, 1], equals 1 only if ρa (ε; fθ ) = ρ (ε; fθ )

instead spend the same budget on these members as on the dom-

for all ε ∈ [0, εmin ], and otherwise measures the normalized

inant ones, wasting queries the frontier does not need.

a⋆

area gap to the empirical frontier.

To address these limitations, we extend the evaluation from

Since attack optimality is model-dependent, an attack may

individual attacks to attack ensembles and allocate a single to-

closely approximate the frontier for one defense but not for an-

tal query budget across the attacks in the pool. The ensemble

other. To summarize its performance across a set of models M

combines the complementary strengths of its members, thereby

evaluated under the same ℓ p norm, we define the global AOI

extending the search coverage without increasing the overall

(AOI) as the mean of the corresponding per-model values:

evaluation budget. This allows us to study both how the alloca-

AOI(a) =

1 X AOI(a, fθ ). |M| f ∈M

tion of queries affects ensemble effectiveness and the budget re(6)

quired for the ensemble to approach the tightest empirical fron-

tier attainable by the considered attack pool. The central quesBy definition, AOI(a) ∈ [0, 1] and quantifies how closely a ap-

tion is therefore how to distribute a fixed query budget across

proaches the empirical attack frontiers on average across mod-

attacks so as to approximate this frontier as closely as possible.

els. In summary, given a pool of attacks, AttackBench compares

3. Approximating the Attack Frontier with Attack Ensem-

their relative effectiveness by measuring the AOI of each attack

bles

independently over a set of models, using a fixed query budget. Our goal is to construct attack ensembles that approximate

This makes it possible to identify the attack that is, on average,

the attack frontier as closely as possible, under a given query

the most effective for robustness evaluation. However, this ap-

budget a practitioner can control. The resulting ensembles can

proach has two important limitations:

then estimate the robustness–perturbation curve of any given

(i) It evaluates attacks in isolation. No individual attack is ex-

model. In the following, we present a greedy budget allocation

pected to approach the empirical attack frontier equally well

approach that optimizes the AOI of the resulting ensemble.

across all samples and models. Different attacks may therefore be complementary: taking the best perturbation found by each

3.1. Problem formulation

attack on a per-sample basis can produce a tighter robustness

Given a set of attacks A = {a1 , . . . , aN } and a set of mod-

curve than any individual attack.

els M, we aim to approximate the empirical attack frontier by

(ii) It assigns a fixed budget independently to every attack. Be-

combining a subset of the attacks in A, under a fixed computa-

cause the query budget is split uniformly, each attack receives

tional budget. To make computation comparable across attacks, 5

we measure cost in queries, i.e., the sum of forward and back-

same attack with multiple variants, discussed in Sect. 5.1. We

ward passes through the target model. This provides a simple

denote by E⋆ the ensemble induced by q⋆ that attains the high-

and consistent unit of computation and allows different attacks

est AOI achievable given the available query budget Q. Since

to be evaluated under the same budget constraint. We denote by

q⋆ cannot be computed exactly, in Section 3.2 we present a

T the maximum number of queries any attack can spend, and

b that approximates greedy algorithm that returns an ensemble E

by Q the total number of queries available to the whole ensem-

E⋆ .

ble. Ensemble construction thus reduces to deciding how many 3.2. Greedy algorithm

queries to assign to each attack in A. This leads to a budget

Problem (8) is a discrete budgeted maximization whose

allocation formulation.

search space grows exponentially with the pool size, therefore

Under this view, an ensemble is fully specified by a vector of

cannot be solved exhaustively. To find a suitable solution, we

per-attack query budgets q = (qa )a∈A ∈ NN , where N = |A| and

propose the greedy procedure reported in Algorithm 1.

each entry qa specifies the number of queries allocated to attack

We start from the empty allocation q ≡ 0 (Line 1), i.e., the

a. The attacks actually included in the ensemble are those with

induced ensemble is empty. For a given attack a ∈ A, a candi-

a non-zero allocation:

date update raises its budget to a strictly larger value q̂ ∈ S, i.e., E = {a ∈ A : qa > 0} ⊆ A.

it sets qa ≡ q̂ while leaving every other entry of q unchanged.

(7)

We denote by q′ the vector resulting from this update. We conWe can measure the optimality of the ensemble E directly as

sider only updates for which the resulting query vector satisfies

a function of the query vector, AOI(q), extending Eq. (6) to a

the constraints in Eq. (8). Given the current query allocation q,

complete budget allocation.

the feasible updates are:

Optimization Problem. The empirical attack frontier corresponds to the ideal ensemble Emax , obtained from the query

n o P N(q) = (a, q̂) ∈ A×S : q̂ > qa , q′ ∈ C, b∈A qb −qa +q̂ ≤ Q ,

vector qmax , which sets qa = T for every attack a ∈ A, i.e.,

(9)

runs the whole pool at the per-attack cap, so that Emax = A. By

Each feasible update is scored by its AOI gain, scaled by the

construction, AOI(qmax ) = 1, though a smaller budget may suf-

added queries, i.e.,

fice to reach it. This regime defines a reference upper bound of ga,q̂ =

performance but is not attainable under practical computational

AOI(q′ ) − AOI(q) . q̂ − qa

(10)

constraints. We therefore seek for q⋆ , the best feasible approxiThe normalization promotes query-efficient improvements over

mation to qmax with a limited total query budget Q ≤ N · T . We

raw gains.

formalize this as the following optimization problem:

We iteratively select the update in N(q) with the largest ga,q̂ q⋆ ∈ arg max AOI(q) q∈NN

s.t.

(Line 3) and apply it (Line 4), achieving a heuristic algorithm

qa ≤ Q, qa ≤ T ∀a, q ∈ C. | {z } a∈A | {z } per-attack cap

that approximates the frontier within the query budget Q. The

total budget

(8)

algorithm stops (Line 2) when it reaches the maximum allo-

where C ⊂ NN encodes additional structural rules to include the

cated budget Q (i.e., when the feasible set of candidate updates 6

Algorithm 1: Greedy Ensemble Construction.

APGDmin retains the sparse-sign direction, no-momentum up-

Input : attack pool A; models M; query budget Q. Output : query vector b q maximizing AOI subject to the b budget (Eq. 8), whose induced ensemble is E. 1 q ≡ 0 ▷ empty ensemble 2 while ∃ (a, q̂) ∈ N(q) with ga,q̂ > 0 do 3 (a⋆ , q̂⋆ ) ← arg max ga,q̂ ▷ best gain per added query

date, exact projection onto B1 (εt ) ∩ [0, 1]d , and sparsity-ratio checkpoint of ℓ1 -APGD. The implementation can be found in our released code (see Section 5).

(a,q̂)∈N(q)

qa⋆ ≡ q̂⋆

return b q←q

▷ update queries

4. Ranking Defenses: the Defense Optimality Index Section 3 showed how to construct, for a given query budget,

N(q) is empty), or when there is no update that improves the

an attack ensemble that best approximates the attack frontier.

overall AOI of the ensemble. Finally, the algorithm returns the

We now use this ensemble to evaluate and rank a given set of

query vector b q it has constructed (Line 5), whose induced en-

defenses. To do so, we introduce the defense frontier and the

b semble is E.

Defense Optimality Index (DOI), the defense-side counterparts

b obtained from Algorithm 1 can be used The ensemble E

of the attack frontier and the AOI introduced in Section 2.2.

to evaluate the robustness of any given model, yielding its

We detail below how we build the frontier and compute the

robustness–perturbation curve. The allocation procedure is ag-

DOI, which scores each defense by how close its robustness–

nostic to the attacks in the pool. To strengthen the pool used in

perturbation curve is to the defense frontier.

our experiments, we add APGDmin .

Defense Frontier. Let dE (x; fθ ) be the distance per-sample found by the attack ensemble E on a model fθ ∈ M. Retain-

3.3. Extending the Pool: Minimum-Norm Auto-Projected Gra-

ing the maximum of these minima across all models defines the

dient Descent

defense frontier, i.e., an oracle model that misclassifies each Auto-Projected Gradient Descent (APGD) [4, 10] is one of

sample with the largest minimum-norm perturbation:

the best attacks for ℓ1 and ℓ∞ , despite having been designed for fixed-radius evaluation. This motivates APGDmin , our native

⋆ ddef (x) = max dE (x; fθ ).

(11)

minimum-norm adaptation of APGD for ℓ1 , ℓ2 , and ℓ∞ , which

fθ ∈M

searches for small perturbations in one run rather than repeat-

The corresponding robustness–perturbation curve ρ⋆def (ε) fol-

edly invoking a point-wise attack at different radii.

lows from Eq. (3), and its area, AUREC⋆def , from Eq. (4). The

APGDmin retains APGD’s projected-gradient updates but

integration in this case runs up to εdef = min{ε ≥ 0 : ρ⋆def (ε) =

adapts the radius εt per sample. It initializes the radius from

0}, the smallest budget at which the defense frontier reaches

a first-order boundary estimate using the dual-norm gradient

zero. The same upper limit is used for every model in M, mak-

⋆

∥gt ∥q . Following FMN [11], it tracks the best perturbation δ ,

ing the areas directly comparable.

⋆

shrinks the radius toward ∥δ ∥ p after a successful iterate, and

Defense optimality. Let AURECE ( fθ ) be the area of model fθ ’s

expands it otherwise. The update rate uses cosine decay, the

robustness–perturbation curve computed with the ensemble E.

step size scales with εt+1 /εt , and checkpoints halve the step

We define the Defense Optimality Index (DOI) as:

size and restart from the best adversarial example. For ℓ2 and DOI( fθ ) =

ℓ∞ , these are the only changes to the original APGD. For ℓ1 , 7

AURECE ( fθ ) . AUREC⋆def

(12)

Table 1: Benchmark model pool. CIFAR-10 (C1–C20) and ImageNet (I1–I10) defenses used in our experiments. The Norm columns denote the perturbation norm ℓ p each model was adversarially trained on; BS is the batch size.

CIFAR-10

ImageNet

Model

Norm BS

Model

Norm BS

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10

Addepalli [12] ∞ 500 Augustin [15] 2 500 Chen [14] ∞ 500 Cui [20] ∞ 500 Engstrom 2 500 Gowal [25] ∞ 250 Gowal [25] ∞ 500 Jiang [28] 1 500 Maini [29] 1,2,∞ 500 Rade [13] ∞ 500

C11 C12 C13 C14 C15 C16 C17 C18 C19 C20

Rade [13] 2 Rebuffi [16] ∞ Sehwag [18] 2 Stutz [21] ∞ Zhong [23] (PGD) 0k=120 Zhong [23] (TRADES) 0k=120 Pang [26] ∞ Engstrom ∞ Wang [30] ∞ Xu [31] ∞

Table 2: Minimum-Norm Attack Pool. Minimum-norm solvers for our evaluation ensembles, and their supported ℓ p perturbation norms. Restarts indicates the number of random ℓ p -ball initializations (− if unsupported). All solvers natively support targeted searches.

I1 I2 I3 I4 I5 I6 I7 I8 I9 I10

Chen [14] Debenedetti [17] Rodriguez-Munoz [19] Singh [22] Wong [24] Engstrom Salman [27] Salman [27] Singh [22] Singh [22]

Norm BS ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞

100 100 100 100 200 200 200 200 100 250

mation improving consistently as the budget grows across tiers. We then use these ensembles to compute the DOI of a large pool of defenses, showing how this ε-independent ranking over-

Norms Solver

ℓ0

ℓ1

ℓ2

ℓ∞

Restarts

Primal-Dual Proximal Gradient Descent (PDPGD) [32] Primal-Dual Gradient Descent (PDGD) [32] Fast Minimum-Norm (FMN) [11] Auto-Projected Gradient Descent (APGDmin ) (Ours) Augmented Lagrangian Method for Adversarial (Alma) [33] Decoupled Direction and Norm (DDN) [34] σ-Zero (σ-zero) [35]

✓

✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓

5 5 5 5 — — 5

✓

500 500 500 500 500 500 250 500 500 500

ID Model

✓ ✓

comes the brittleness of single-budget evaluation.

5. Experiments We validate our framework empirically in two steps: we first

By construction, DOI( fθ ) ∈ [0, 1] and equals 1 only if

show that our ensembles closely approximate the attack fron-

ρE (ε; fθ ) = ρ⋆def (ε) for all ε ∈ [0, εdef ]. Lower values indicate a

tier at a controllable query cost, and then use them to com-

larger empirical gap from the best achievable robustness within

pute the DOI and demonstrate its stability against the brittle-

M. As for AOI, the metric is pool-relative and empirical. Its

ness of single-ε evaluation. We run these experiments on a sin-

values depend on the perturbation norm, the strength of the en-

gle NVIDIA H100 NVL (94 GB). Our code and benchmark are

semble used to compute the distances, and the diversity of the

accessible at adversarial-frontier.github.io.

models in M. 5.1. Setup Taken together, Sections 3 and 4 complete our framework. Datasets and models. We evaluate our approach on CIFAR-

Given a query budget, the greedy algorithm of Section 3 con-

10 and ImageNet using a fixed 1 000-sample validation subset

structs an attack ensemble that approximates the attack frontier.

for each model–norm pair. We evaluate 30 robust models, 20

As the budget increases, the algorithm yields a nested family of

on CIFAR-10 and 10 on ImageNet, drawn from RobustBench

ensembles, each an incremental extension of the previous one.

and their original sources. We label CIFAR-10 models as C1–

In Section 4 we use these ensembles to compute the DOI of a

C20 and ImageNet ones as I1–I10, as listed in Table 12 . We

set of defenses, and evaluate their gap to the defense frontier. In Section 5, we validate this framework empirically. We con-

2 Models C5, C18, and I6 are sourced from https://github.com/mad

rylab/robustness. For I10, the batch size is reduced to 100 for standard AutoAttack and for the ablation grid on ℓ1 and ℓ2 .

struct ensembles increasing query budget tiers and show that they closely approximate the attack frontier, with the approxi8

Table 3: Deployable attack ensembles produced by the greedy construction. Indented Restarts and Targeted rows report cumulative queries for the attack above; an en-dash denotes an inactive group. Details: https://github.com /adversarial-frontier.github.io.

attack is executed for 500 optimization steps (equivalent to a maximum per-attack budget of T = 1000 queries), recording the perturbation distance every 10 queries. Recall from Sec-

Configuration

12k

Configuration

ℓ0 σ-zero Restarts Targeted PDPGD Restarts Targeted FMN Restarts

12k

tion 3.1 that the feasible set C ⊂ NN restricts the admissible

ℓ1

800 2400 – 600 – – 200 –

900 3400 1600 700 – 1200 200 –

900 4400 3200 700 500 2000 200 100

APGDmin -CE Restarts APGDmin -DLR Restarts PDPGD Restarts FMN Restarts

1000 900 1000 – 400 400 100 200

ℓ2 Alma Targeted PDGD Restarts Targeted APGDmin -DLR Restarts APGDmin -CE Restarts Targeted

1000 2000 1000 2000 600 1100 100 200

1000 4000 1000 3000 600 2100 100 200

query allocation vectors q. Here, C enforces two structural dependencies tied to the pool expansion: (i) a targeted run against the k-th most likely incorrect class cannot receive more queries than the run against the (k−1)-th class, and (ii) a random restart

ℓ∞

1000 – 400 1000 – 400 700 500 – –

1000 – 500 1600 2200 700 1100 600 300 –

1000 2000 1000 2600 2500 700 1100 600 300 200

FMN

800

1000

may only receive queries if its corresponding untargeted base

PDPGD Restarts APGDmin -CE Restarts Targeted APGDmin -DLR Restarts

– – 700 100 – 700 1700

– – 700 2100 – 700 3500

800 2700 700 2400 100 700 3600

run is active. Candidate

levels

are

drawn

{100, 200, . . . , 1000}, defining a grid of J

from

= 10 evenly

3 Attack implementations are adopted from https://github.com/jerom erony/adversarial-library.

split these 30 defenses into two partitions, train and held-out,

spaced checkpoints up to the cap T .

stratified across datasets (seed 0): 17 training and 13 held-out

Because our greedy

strategy monotonically increases budget allocations, we record

models (12/8 on CIFAR-10, 5/5 on ImageNet). We compute

the resulting ensembles at three distinct query tiers, i.e.,

the ensembles on the train pool, and test them on the the unseen

Q ∈ {4, 8, 12}k queries. This yields a nested hierarchy of

models of the held-out set. Attack pool.

budget

ensembles E4k ⊆ E8k ⊆ E12k . A user can then pick the tier

Table 2 summarizes the pool of minimum-

matching their compute budget and later extend it by resuming

norm solvers3 we used across the four ℓ p norms, and expanded

the same run.

by introducing our novel minimum-norm adaptation of Auto-

Metrics and baselines. For each model and query tier, we re-

Projected Gradient Descent (APGDmin ), we described in Sec-

port the per-model AOI and robust accuracy at standard AA

tion 3.3. All baseline solvers are executed with their default

reference budgets εref . For CIFAR-10, these are set to 12, 0.5,

hyperparameters, The only exception is PDPGD in ℓ0 , where

and 8/255 for ℓ1 , ℓ2 , and ℓ∞ , respectively; for ImageNet, they

we replace the standard hard-thresholding with the closed-form

are 40, 3.0, and 4/255. Finally, we use the E12k ensemble to

ℓ2/3 concave surrogate proximal operator to improve sparsity.

rank all 30 models by computing the DOI.

Finally, each solver’s untargeted execution (seed 42) is aug-

We compare our approach against AA [4]. Since AA reports

mented with two search variants. First, we execute targeted at-

robust accuracy at a single fixed ε and returns no per-sample

tacks directed at the top-9 highest-confidence incorrect classes,

minimum-norm distance, it provides one evaluation point rather

yet still looking for an untargeted misclassification. Second, we

than a complete curve, and cannot be scored by AOI directly.

employ random restarts by evaluating 5 independent initializa-

We thus approximate its curve by evaluating AA over a per-

tions sampled from the ℓ p ball (seeds 43 to 47).

model grid of ε values, derived from our attack frontiers as 9

We measure attack cost in queries, defined as the total num-

distance percentiles plus the reference budget εref . We then lin-

ber of forward and backward passes through the model. Each 9

ℓ0

AOI (%)

100

ℓ1

ℓ2

ℓ∞

96 4k

Query budget Q

12k 4k

Query budget Q

12k 4k

Train models

Query budget Q

12k 4k

Query budget Q

12k

Held-out models

Figure 2: Generalization of attack optimality across query budgets. This plot shows AOI as a function of the query budget Q across the four evaluated perturbation norms. Blue lines denote the Attack Optimality Index on average across the set of 17 training models, while red lines denote the same measure on average across 13 held-out models. The AOI monotonically improves as the budget increases from 4k to 12k queries. Crucially, the narrow gap between the train and held-out curves demonstrates that our greedy budget allocation generalizes effectively to unseen defenses without overfitting.

early interpolate these points to reconstruct AA’s robustness–

cation concentrates on APGDmin and PDPGD restarts, with a single

perturbation curve and compute its AOI.

FMN base run. The learned allocations also generalize beyond the 17 training models. Figure 2 compares the AOI on these

5.2. Results models with that on the 13 held-out models from 4k to 12k We first validate the three ensembles on 13 held-out dequeries. The curves track closely across all norms and tiers, fenses, showing that their AOI generalizes to unseen models with a maximum gap below 0.4 percentage points, indicating and matches or exceeds AA’s, both on the complete robustness– no substantial degradation on unseen models.

perturbation curve and at the single reference budget εref . Once

Optimality analysis. Table 4 shows that, at Q = 12k, our en-

validated, E12k becomes our evaluation tool: we use it on all 30

sembles match or exceed AA’s AOI on all 13 held-out models defenses to show how the DOI resolves the limitations of fixed-

on ℓ1 , on 12 out of 13 on ℓ2 , and on 10 out of 13 on ℓ∞ ; their

ε evaluation, assessing every defense across all four perturba-

mean AOI is higher and its variation lower on all three norms. tion norms, even beyond the one used for its own adversarial

The largest improvement is on ℓ2 , where the 4k ensemble al-

training, and beyond RobustBench’s coverage.

ready matches or exceeds AA on 12 of 13 held-out models. BeEnsemble composition and generalization. Table 3 summacause AOI evaluates complete robustness–perturbation curves rizes the composition at each query tier. Base rows report the against the empirical attack frontier, these results demonstrate queries assigned to one run, whereas Restarts and Targeted an improved frontier approximation rather than only robust acrows report the total across all runs in that group; per-run details curacy at a selected perturbation radius. are available on our benchmark website. Composition varies by

In Table 5, we report the AOI on ℓ0 , where AA provides no

norm and budget. The ℓ0 ensembles center on σ-zero, with

baseline. The AOI on held-out models at 12k ranges from 98.1–

smaller contributions from PDPGD and FMN. Under ℓ1 , they rely

99.8 across all models, showing that the ensembles reach the mainly on the base and restart variants of both APGDmin losses. frontier reliably even on a norm with no established benchmark

The broader ℓ2 ensembles are led by PDGD restarts and targeted

to compare against.

runs, complemented by Alma and APGDmin . Under ℓ∞ , the allo10

Table 4: Our greedy ensembles vs. AA: AOI (%) per model as the query budget grows (4k→12k), train and held-out models. Green: our AOI matches or exceeds AutoAttack.

ℓ1

ℓ2

ℓ∞

E8k

E12k

E4k

E8k

E12k

E4k

E8k

E12k

Train C1 96.54 C2 98.25 C5 99.32 C7 98.50 C8 98.82 C9 97.61 C11 98.84 C13 98.88 C14 93.61 C15 98.97 C17 97.33 C20 97.45 I1 98.39 I2 98.32 I4 97.68 I5 99.19 I7 98.59

96.97 98.42 97.79 98.16 97.96 98.28 97.96 97.58 97.53 98.42 97.83 97.45 98.74 98.18 97.56 99.02 98.58

97.95 99.07 98.56 98.83 98.59 98.76 98.91 98.51 98.72 98.94 98.66 98.54 99.12 98.68 98.34 99.28 98.91

98.54 99.39 98.86 99.07 98.82 99.08 99.16 98.82 99.20 99.21 99.04 98.89 99.32 98.89 98.66 99.42 99.11

Train C1 97.65 C2 98.79 C5 99.32 C7 98.24 C8 99.75 C9 99.12 C11 99.76 C13 99.25 C14 97.67 C15 99.86 C17 98.11 C20 98.15 I1 97.15 I2 97.77 I4 95.49 I5 97.34 I7 97.57

99.07 99.49 99.71 99.28 99.70 99.53 99.74 99.52 98.60 99.74 99.30 99.08 99.04 99.07 97.68 98.79 99.15

99.49 99.71 99.87 99.66 99.92 99.72 99.88 99.81 99.24 99.82 99.68 99.50 99.56 99.52 98.74 99.30 99.63

99.72 99.83 99.92 99.79 99.95 99.86 99.91 99.87 99.41 99.94 99.81 99.65 99.72 99.66 99.07 99.56 99.75

Train C1 99.63 C2 99.72 C5 99.87 C7 99.59 C8 99.59 C9 99.45 C11 99.53 C13 99.84 C14 98.64 C15 99.32 C17 99.59 C20 99.67 I1 99.61 I2 99.24 I4 98.69 I5 98.49 I7 99.42

99.59 99.56 99.58 99.52 99.57 99.42 99.78 99.59 99.13 99.65 99.72 99.62 99.30 99.19 98.34 98.71 99.21

99.67 99.71 99.65 99.60 99.63 99.53 99.82 99.69 99.67 99.77 99.80 99.76 99.44 99.31 98.84 98.79 99.30

99.70 99.74 99.71 99.63 99.69 99.55 99.84 99.78 99.74 99.79 99.81 99.81 99.62 99.56 99.18 99.22 99.57

Held-out C3 97.54 C4 97.57 C6 97.33 C10 97.53 C12 97.76 C16 99.22 C18 98.17 C19 97.41 I3 96.26 I6 98.36 I8 98.38 I9 97.93 I10 98.31

97.98 97.83 97.95 96.58 97.80 98.52 97.48 97.59 97.08 98.40 98.24 97.87 98.44

98.78 98.68 98.70 98.20 98.64 99.01 98.43 98.53 97.95 98.85 98.71 98.51 98.94

99.19 99.07 99.00 98.77 98.97 99.22 98.78 98.90 98.31 99.04 99.04 98.83 99.15

Held-out C3 98.06 C4 97.83 C6 97.19 C10 96.61 C12 98.33 C16 99.99 C18 98.01 C19 97.85 I3 93.67 I6 96.80 I8 96.82 I9 95.60 I10 96.04

99.26 99.03 98.93 99.17 99.25 99.86 98.96 99.06 96.74 98.69 99.02 98.14 98.62

99.63 99.43 99.49 99.63 99.64 99.92 99.49 99.45 97.91 99.26 99.45 98.98 99.38

99.78 99.62 99.69 99.80 99.77 99.95 99.69 99.62 98.42 99.52 99.64 99.21 99.54

Held-out C3 99.62 C4 99.46 C6 99.46 C10 99.00 C12 99.50 C16 99.52 C18 99.65 C19 99.45 I3 98.31 I6 99.55 I8 99.56 I9 98.29 I10 98.62

99.66 99.57 99.44 99.60 99.50 99.77 99.44 99.58 98.53 98.88 99.15 98.69 98.97

99.74 99.68 99.57 99.72 99.60 99.84 99.58 99.68 99.00 99.06 99.30 99.11 99.25

99.75 99.69 99.63 99.78 99.63 99.87 99.62 99.71 99.12 99.37 99.50 99.24 99.46

Robust accuracy ρ(‖δ‖2 )

0.8

0.2 1

Perturbation size ‖δ‖2

0.0 0.00

(a) Best case: I3 on ℓ2 Attack Frontier

Ensemble (4k)

0.01

0.02

0.03

Perturbation size ‖δ‖∞

0.0 0.0

0.04

(b) Worst case: I6 on ℓ∞ Ensemble (8k)

0.2

0.4

Ensemble (12k)

Model I1 Model I3

0.8 0.6

0.6

0.4

Model C9 Model C19

0.8

0.6

0.0

ℓ∞

1.0

Robust accuracy ρ(‖δ‖2 )

ℓ2

1.0

Robust accuracy ρ(‖δ‖∞ )

E4k

Robust accuracy ρ(‖δ‖∞ )

AutoAttack εref =0.5

0.2

0.4

0.6

0.8

1.0

Perturbation size ‖δ‖2

1.2

0.0 0.00

AutoAttack εref =4/255

0.01

0.02

0.03

Perturbation size ‖δ‖∞

0.04

Figure 4: Curves can cross past the reference ε. Robust accuracy ρ(ε) vs. perturbation budget for defense pairs whose curves intersect at a budget larger than AA’s reference ε (dashed line): at the reference one defense leads, beyond it the order flips. Left: ℓ2 on CIFAR-10 (C9/C19); right: the single ℓ∞ case we observe, on ImageNet (I1/I3).

AutoAttack

Figure 3: Ensemble robustness curves vs. AA. Our curve-based evaluations at Q ∈ {4k, 8k, 12k} compared against AA’s point-wise baseline, on I3 (a) and I6 (b) as best and worst cases. Our ensemble wins in I3, while remaining competitive on I6.

We also examine how much increasing the query budget improves AOI on the held-out pool of models, from 4k to 12k 11

Table 5: Our greedy ensembles: Attack Optimality Index (AOI, %) per model as the query budget E grows (4k→12k); train and held-out models. ℓ0 has no AutoAttack baseline.

Train

Held-out

E4k

E8k

E12k

E4k

E8k

E12k

E4k

E8k

E12k

E4k

E8k

E12k

C1 C2 C5 C7 C8 C9 C11 C13 C14

96.6 98.5 98.3 96.6 96.8 97.7 98.6 97.2 97.2

98.5 99.5 99.4 98.3 98.6 99.1 99.4 98.8 98.3

99.2 99.8 99.9 99.2 99.4 99.8 99.7 99.6 99.0

C15 91.1 97.0 C17 97.1 98.6 C20 97.2 98.5 I1 99.3 99.6 I2 98.6 99.3 I4 98.3 99.1 I5 99.1 99.5 I7 98.7 99.2

98.2 99.4 99.3 99.8 99.7 99.6 99.7 99.7

C3 C4 C6 C10 C12 C16 C18

96.6 95.8 96.3 97.0 96.5 92.5 96.7

98.4 97.7 98.2 98.7 98.2 96.8 98.3

99.2 98.7 99.0 99.5 99.0 98.1 99.1

C19 I3 I6 I8 I9 I10

96.8 98.0 99.1 99.4 99.1 99.0

98.4 98.8 99.5 99.6 99.5 99.5

99.2 99.3 99.8 99.8 99.8 99.8

Table 6: Our greedy ensembles vs. AutoAttack: Robust Accuracy ρ(εref ) (%) at the reference ε per model as the query budget grows (4k→12k), for train and held-out models. QAA is the (variable) query cost of AA. Green: our ensembles are at least as strong as AA.

ℓ1

ℓ2

ℓ∞

QAA

E4k

E8k

E12k

QAA

E4k

E8k

E12k

QAA

E4k

E8k

E12k

Train C1 16.5 C2 31.0 C5 26.2 C7 11.1 C8 52.0 C9 41.3 C11 43.1 C13 34.4 C14 0.0 C15 0.0 C17 8.5 C20 9.5 I1 8.6 I2 36.0 I4 40.9 I5 3.8 I7 1.7

1824 3219 2736 1286 5179 4195 4362 3535 198 187 1051 1174 1015 3700 4160 508 299

17.2 30.8 27.8 12.4 52.0 40.7 44.5 36.5 0.0 0.0 8.8 10.9 9.1 37.8 42.5 4.9 2.1

16.0 29.8 27.3 12.0 51.7 40.5 43.8 34.9 0.0 0.0 7.6 9.9 8.2 36.5 41.1 4.4 1.9

15.4 29.2 27.1 11.7 51.7 40.3 43.6 34.5 0.0 0.0 7.5 9.5 8.1 36.1 40.7 4.1 1.9

Train C1 58.4 C2 73.4 C5 70.3 C7 65.8 C8 62.2 C9 58.9 C11 77.1 C13 73.3 C14 0.0 C15 0.9 C17 64.6 C20 69.2 I1 6.5 I2 27.5 I4 34.3 I5 2.6 I7 1.4

5767 7217 6918 6486 6116 5807 7566 7205 195 275 6377 6828 799 2823 3488 384 266

58.3 73.5 70.5 65.7 62.2 58.9 77.1 73.3 0.0 1.0 63.5 68.5 5.6 25.9 32.3 2.3 1.0

58.3 73.5 70.4 65.6 62.2 58.9 77.1 73.3 0.0 1.0 63.5 68.4 4.9 25.6 31.6 2.1 1.0

58.3 73.4 70.4 65.6 62.2 58.9 77.1 73.3 0.0 0.9 63.3 68.2 4.8 25.6 31.3 2.1 1.0

Train C1 51.1 C2 37.6 C5 29.6 C7 58.6 C8 35.5 C9 35.5 C11 44.1 C13 39.2 C14 0.0 C15 0.0 C17 62.5 C20 64.8 I1 45.2 I2 46.0 I4 51.3 I5 27.7 I7 29.5

5073 3822 3031 5799 3567 3587 4415 3944 195 183 6174 6411 4496 4574 5077 2794 2961

51.2 38.1 30.2 58.6 35.7 36.0 44.0 39.4 0.0 0.0 62.6 64.7 45.5 46.1 51.3 27.4 30.1

51.2 37.9 30.1 58.6 35.7 36.0 44.0 39.3 0.0 0.0 62.6 64.7 45.5 46.1 51.3 27.4 30.0

51.2 37.8 30.1 58.5 35.7 36.0 44.0 39.2 0.0 0.0 62.6 64.7 45.3 46.1 51.3 27.2 29.7

Held-out C3 10.7 C4 10.3 C6 9.1 C10 5.3 C12 14.7 C16 0.2 C18 4.7 C19 11.3 I3 34.5 I6 6.8 I8 3.0 I9 36.1 I10 35.2

1271 1230 1100 733 1644 219 667 1324 3590 821 453 3682 3605

11.4 10.8 8.7 7.0 16.5 0.3 6.2 11.1 35.2 7.6 4.0 38.2 36.4

9.7 10.0 8.2 4.9 15.3 0.3 5.3 10.7 32.9 6.9 3.5 36.6 35.0

9.5 9.5 7.8 4.4 15.0 0.3 4.7 10.4 32.0 6.7 3.2 35.7 34.6

Held-out C3 64.9 C4 70.9 C6 64.2 C10 65.7 C12 67.3 C16 7.4 C18 59.3 C19 71.1 I3 20.3 I6 4.3 I8 2.2 I9 31.6 I10 31.4

6402 6983 6338 6473 6633 903 5872 7006 2204 578 372 3231 3196

64.2 70.2 63.4 65.6 67.0 7.4 58.7 70.5 16.8 4.1 2.0 28.9 28.9

64.2 70.1 63.0 65.5 66.7 7.4 58.7 70.3 15.7 3.4 1.6 27.6 27.7

64.1 70.1 62.7 65.3 66.7 7.4 58.6 70.3 15.5 3.4 1.6 27.4 27.5

Held-out C3 57.9 C4 67.8 C6 66.5 C10 57.6 C12 61.9 C16 0.0 C18 51.0 C19 67.8 I3 53.9 I6 32.2 I8 39.1 I9 53.8 I10 53.0

5733 6687 6558 5701 6114 192 5074 6691 5347 3244 3904 5320 5244

58.0 67.9 66.7 57.5 61.9 0.0 51.6 67.8 54.1 33.7 39.9 54.0 53.3

57.9 67.9 66.7 57.5 61.9 0.0 51.5 67.8 54.1 33.6 39.8 54.0 53.2

57.9 67.9 66.7 57.4 61.9 0.0 51.5 67.8 54.1 33.1 39.6 54.0 53.1

queries. On ℓ1 , the mean AOI rises from 97.83 ± 0.53 to

varies substantially across the held-out models, ranging from

98.94 ± 0.23 (+1.12), the largest gain among the three norms.

219 to 3 682 queries on ℓ1 , from 372 to 7 006 on ℓ2 , and from

On ℓ∞ , the gain is the smallest, from 99.29±0.39 to 99.57±0.21

192 to 6 691 on ℓ∞ , whereas our tiers use the predefined budgets

(+0.28). This likely reflects ℓ∞ ’s prevalence in robustness eval-

Q. At AA’s reference ε, the ensemble advantage is smaller than

uation and adversarial training among the studied defenses: its

in the curve-based comparison: at 12k queries, our ensembles

attack configurations are near saturation even at low budgets. ℓ2

match or exceed AA on 28 of 39 held-out model–norm pairs

falls in between, improving from 98.83 ± 0.71 to 99.56 ± 0.37

(62 of 90 across the full pool).

(+0.73), while retaining the largest gap to AA.

Crossings and ranking instability. Figure 4 shows two ob-

Robust accuracy comparison. Here we directly compare our

served cases in which a pair of defenses reverses order only

ensembles fixed-ε performance at AA’s reference value. This

beyond AA’s reference ε: the ranking at the standard operating

favors AA’s fixed-ε design because our allocation maximizes

point therefore need not characterize their relative robustness

AOI over complete curves on the 17 training models rather than

over the complete curve. To assess the effect on rankings, we

optimizing specifically at εref , which can disadvantage our en-

recompute AA’s ordering over a grid of perturbation budgets

sembles at low budgets. Nevertheless, our ensembles remain

(Figure 5). The changes are substantial: under ℓ2 on CIFAR-

competitive at these reference values, as detailed in Table 6 and

10, 17 defenses move by at least five ranks across the evaluated

illustrated by the best- and worst-case frontiers from the held-

range of perturbation budgets, while under ℓ∞ on ImageNet, all

out models (Figure 3).

10 models move by at least two ranks. Thus, current bench-

ℓ2 is the norm where we perform best: our ensembles match

marks that rank defenses by robust accuracy at one fixed ref-

or exceed AA on all 13 held-out models, at every query tier,

erence ε, including AA, can miss ranking changes that arise

with a mean improvement of 1.5 ± 1.6 points at 12k (positive

across the complete robustness–perturbation curves.

values indicate a lower, hence better, robust accuracy than AA).

DOI Rankings. Table 7 reports the norm-specific DOI rank-

ℓ1 shows instead a mixed picture: at 8k queries we already

ings across all query tiers. Because each ranking summarizes

match or exceed AA on 7 out of 13 held-out models, rising

complete robustness–perturbation curves, it does not require se-

to 10 out of 13 at 12k, with a mean improvement of 0.6 ± 0.7

lecting a reference ε for that norm.

points at 12k. This is in line with what we observed in the op-

Comparing the resulting ranks reveals strong cross-norm

timality analysis, where ℓ1 was the norm benefiting the most

asymmetry. C6 [25], adversarially trained on ℓ∞ , ranks first

from additional queries. ℓ∞ is instead the hardest case for us:

there but falls in the bottom half under the other norms. C15

we match or exceed AA on 5 out of 13 held-out models, at both

and C16 [23], trained on ℓ0 , show the converse pattern, while

8k and 12k, with a mean difference of −0.2 ± 0.3 points in AA’s

C8 [28], trained on ℓ1 , ranks near the top under ℓ0 , ℓ1 , and ℓ2 but

favor at 12k. The same pattern holds on the complete pool of 30

near the bottom under ℓ∞ . Thus, a high rank under one norm

models: at 12k queries, we match or exceed AA on 29 defenses

does not imply a high rank under the others.

on ℓ2 , 20 on ℓ1 , and 13 on ℓ∞ .

Broad cross-norm robustness also does not follow directly

Beyond point-wise robust accuracy, the methods offer differ-

from the norm used for adversarial training. Of the two ResNet-

ent control over cost and evaluation quality. AA’s realized cost

18 models C10 and C11 [13], C11 ranks fifth under ℓ0 and in 13

Rank

CIFAR-10 ⋅ ℓ1

1 3 5 7 9 11 13 15 17 19

Rank

.33 6.67 13 2

CIFAR-10 ⋅ ℓ2

1 3 5 7 9 11 13 15 17 19

0.5

0.2

Rank

0.7

1.5

1.2

1.7

0.0

C13 C5 C6 C11

0.0

C20 C8 C2 C9

2.2

2.5

0.0

.33 06.7 93 1

3 0 12 133.

2 5 7 9 1 4 6 8 1 3 05 .010 .015 .020 .026 .031 .036 .041 .047 .052 0.0 0 0 0 0 0 0 0 0 0

05 0.1

C1 C12 I2 I7

ImageNet ⋅ ℓ∞

1 2 3 4 5 6 7 8 9 10

0.0

.33 6.67 53 6

ImageNet ⋅ ℓ2

1 2 3 4 5 6 7 8 9 10

CIFAR-10 ⋅ ℓ∞

1 3 5 7 9 11 13 15 17 19

ImageNet ⋅ ℓ1

1 2 3 4 5 6 7 8 9 10

I10 I5 I3 I9

I4 I1 I6

I8 Other models AA standard reference ε

Figure 5: Rank instability across perturbation budgets. Defense rankings (y-axis) computed via AA across ten different perturbation sizes ε (x-axis). In each panel, colored traces highlight the five models exhibiting the highest rank instability, while the remaining models are shown in gray. The vertical dashed line indicates the standard benchmark ε.

the top two under ℓ1 and ℓ2 , whereas C10 ranks 19th under ℓ0

On ImageNet, I4, I9, and I10 [22] rank near the top under all

but ninth under ℓ∞ . C9 [29], trained on the union of ℓ1 , ℓ2 , and

four norms, and I2, I3, I4, I9, and I10 consistently form the

ℓ∞ , ranks near the top under ℓ0 and ℓ1 but only 15th under ℓ∞ . 14

Robust accuracy ρ(‖δ‖p ) Robust accuracy ρ(‖δ‖p ) Robust accuracy ρ(‖δ‖p ) Robust accuracy ρ(‖δ‖p )

(a) CIFAR-10 (b) ImageNet

ℓ0

1.0

ℓ1

0.8 0.6 0.4 0.2 0.0

100

Perturbation size ‖δ‖0

ℓ2

1.0

Perturbation size ‖δ‖1

ℓ∞

0.8 0.6 0.4 0.2 0.0 0.0

0.5

1.0

1.5

2.0

2.5

Perturbation size ‖δ‖2

3.0

3.5

0.00

0.05

0.10

ℓ0

1.0

0.15

Perturbation size ‖δ‖∞

0.20

ℓ1

0.8 0.6 0.4 0.2 0.0

250

500

750

1000

1250

1500

Perturbation size ‖δ‖0

200

ℓ2

1.0

400

600

800

Perturbation size ‖δ‖1

ℓ∞

0.8 0.6 0.4 0.2 0.0

Perturbation size ‖δ‖2 0.0

0.2

0.4

DOI

0.00

0.02

0.6

0.8

0.04

0.06

Perturbation size ‖δ‖∞

0.08

1.0

Figure 6: Robustness–perturbation curves for all 30 evaluated defenses (E12k ). For each dataset, the top row shows the ℓ0 and ℓ1 norms, while the bottom row shows ℓ2 and ℓ∞ . The curves plot the robust accuracy against the perturbation size, colored by their DOI score. The red dashed line represents the defense frontier.

Table 7: DOI Evaluations. DOI computed with E at 4k, 8k, and 12k queries. The minimal variation between 4k and 12k demonstrates that the DOI provides stable robustness estimates.

ℓ0 Norm ID

ℓ1 Norm 12k

ℓ2 Norm 12k

ℓ∞ Norm

12k

81.7 74.2 73.7 70.3 63.5 60.8 60.3 59.7 58.5 58.1 57.5 57.1 56.7 56.1 56.0 54.8 48.9 19.8 14.4 1.5

82.0 74.3 73.8 70.4 63.7 60.6 60.4 59.6 58.4 57.9 57.3 57.0 56.4 56.0 55.7 54.3 48.5 19.9 14.4 1.5

82.0 74.2 73.9 70.4 63.7 60.5 60.4 59.5 58.3 57.9 57.3 57.0 56.4 56.0 55.7 54.2 48.5 19.9 14.4 1.5

C6 C4 C19 C17 C20 C12 C7 C3 C10 C1 C18 C11 C13 C2 C9 C8 C5 C16 C15 C14

87.8 83.7 83.5 81.5 79.5 77.9 75.1 74.3 70.6 69.2 62.4 54.3 49.6 48.9 45.5 43.2 43.0 13.4 10.2 1.3

87.7 83.7 83.6 81.5 79.4 77.9 75.2 74.3 70.6 69.2 62.4 54.4 49.6 48.9 45.6 43.3 43.1 13.5 10.3 1.3

87.7 83.8 83.7 81.7 79.5 78.0 75.2 74.5 70.7 69.3 62.5 54.5 49.6 49.0 45.6 43.3 43.2 13.5 10.3 1.3

86.3 82.7 80.4 77.2 66.9 40.9 35.8 30.3 26.5 23.9

86.3 82.9 80.6 78.1 66.9 41.2 35.9 30.5 26.7 24.1

86.3 82.8 80.6 78.2 66.7 41.2 35.9 30.5 26.7 24.1

I9 I10 I4 I3 I1 I2 I8 I6 I7 I5

83.8 82.0 79.6 75.1 73.8 65.6 61.8 51.0 44.6 44.0

83.7 82.0 79.5 74.9 74.0 65.8 61.9 51.1 44.7 44.1

83.9 82.0 79.5 75.0 74.1 65.9 61.9 50.9 44.6 43.7

CIFAR-10 C16 C9 C8 C15 C11 C2 C13 C12 C1 C7 C4 C19 C6 C5 C20 C3 C17 C18 C10 C14

72.6 60.8 59.8 60.3 46.2 40.4 40.1 40.4 37.4 36.9 37.0 35.8 35.8 33.4 34.3 33.8 29.5 24.2 22.7 7.8

72.2 63.9 62.6 57.0 49.2 42.2 42.0 41.3 37.8 37.8 37.6 36.4 36.1 35.3 35.3 34.3 30.2 24.7 23.4 8.3

72.2 64.6 63.4 56.8 50.1 42.9 42.5 41.2 37.8 37.7 37.4 36.3 36.0 35.8 35.2 34.3 30.1 24.7 23.5 8.4

C8 C11 C9 C13 C2 C5 C12 C1 C7 C19 C20 C4 C3 C6 C17 C10 C18 C16 C15 C14

81.5 64.7 59.8 55.5 49.8 46.6 32.2 31.3 29.7 28.7 28.8 28.3 28.2 26.1 25.6 25.2 23.1 17.2 11.7 1.3

81.7 64.1 60.0 55.1 49.3 46.4 31.8 30.7 29.3 28.2 28.1 27.8 27.7 25.7 25.2 24.5 22.5 17.3 11.7 1.3

81.9 64.1 60.0 55.0 49.2 46.4 31.6 30.3 29.1 28.0 27.8 27.6 27.4 25.5 24.9 24.2 22.2 17.3 11.7 1.2

C11 C13 C8 C2 C5 C12 C9 C19 C4 C7 C20 C3 C10 C1 C17 C6 C18 C16 C15 C14

ImageNet I4 I3 I9 I10 I2 I1 I6 I8 I5 I7

73.3 70.7 66.0 65.9 61.0 19.9 17.1 12.0 11.3 8.5

73.7 71.6 67.5 66.8 62.1 20.7 17.5 12.5 11.6 8.8

73.8 72.6 68.5 68.2 63.5 21.3 18.0 12.8 11.9 8.9

I4 I9 I2 I10 I3 I1 I6 I8 I5 I7

82.3 74.7 70.1 69.4 58.3 21.9 18.2 13.1 12.4 9.5

82.0 74.5 71.0 69.2 57.8 21.8 18.1 13.0 12.3 9.6

81.6 74.1 71.5 69.3 57.6 21.9 18.2 12.9 12.3 9.6

I4 I9 I10 I2 I3 I1 I6 I8 I5 I7

top half under ℓ0 , ℓ1 , and ℓ2 . These patterns are also visible in

antee, since the frontier and DOI can change with the query

Figure 6.

budget, as seen under ℓ0 .

The rankings are stable across the evaluated tiers: six of the 5.3. Using the ensembles in practice

eight (norm, dataset) combinations retain the same strict ordering at 4k, 8k, and 12k. On CIFAR-10, ℓ1 differs only by a

Suppose a practitioner has an established defense fA , already

C19/C20 tie at 4k, while ℓ0 contains small local swaps at 4k

evaluated with the full ensemble E12k , and wants to decide

that resolve by 8k. This is empirical stability rather than a guar-

whether a new candidate fB improves on it. Rather than committing the full budget upfront, they can first attack fB with the 16

cheapest tier E4k , on the same samples and norm, and compare

discarded on the same early-stopping grounds as above, since a

the two robustness–perturbation curves. This early comparison

smaller area already signals lower overall robustness before the

is already conclusive in one direction. Since the ensemble dis-

estimate has even tightened. This view is complementary, not

tances upper-bound the true minimal perturbations, and adding

a substitute, for comparing the models directly: AUREC sum-

queries can only tighten them, the curve of fB can only move

marizes robustness as an average over the whole perturbation

downward as the budget grows from 4k to 12k: spending more

range, but it does not reveal whether a defense is more robust at

queries can only reveal fB to be weaker than it already appears,

small perturbations than at large ones, or vice versa. A practi-

never stronger. Therefore, if the E4k curve of fB already lies

tioner should instead choose between the two defenses also by

below the E12k curve of fA , no additional query can reverse the

considering where along the curve robustness matters most for

outcome, and fB can be rejected after only 4k queries, without

their specific use case.

requiring further computations. Figure 7 shows one such case on ℓ∞ taken from our model

6. Related Work

pool: the curve of C9 evaluated at E4k already sits entirely be-

Fixed-ε Ensemble evaluation. Carlini et al. [6] showed that

low that of C6 evaluated at E12k , so C9 is less robust than C6

fixed-ε evaluation is fragile: it can conflate attack failure with

across every perturbation size, and the remaining 8k queries

genuine robustness. They advocated for testing across multiple

would only widen the gap. The practitioner discards C9 having

perturbation budgets and adapting attacks to defense-specific

spent a third of the full budget. If instead the two curves are

properties. AA [4] introduced a fixed, parameter-free ensemble

close or overlap at 4k, the comparison is not yet decisive: the

of four attacks (Auto-Projected Gradient Descent with Cross-

practitioner can then resume the same run and extend it to 8k or

Entropy and Difference-of-Logits-Ratio losses, Fast Adaptive

12k, refining the estimate only where it is needed.

Boundary, and Square attacks), establishing the de facto stan-

A complementary view for comparing the two models di-

dard for robustness benchmarking. Adaptive-AA [36] extends

rectly is to compare their AUREC (Eq. (4)). If the AUREC of

Robust accuracy ρ(‖δ‖∞ )

1.0

AA by automatically tuning attack hyperparameters. SparseAA [23] applies the fixed-ensemble approach to pixel-wise ℓ0

fA : C6 @ Q = 12k fB : C9 @ Q = 4k

0.8

sparsity, constrained to grouped-channel perturbations. Finally, AutoAE [37] proposes a greedy algorithm to allocate iterations

0.6

across attacks at a single, fixed ε, thus it does not extend to

0.4

curve-based evaluation. We provide evidence of the brittleness

0.2 0.0 0.00

of fixed-ε evaluations in our experiments. Moreover, none of these methods covers the optimality of the algorithms used for 0.02

0.04

0.06

0.08

Perturbation size ‖δ‖∞

0.10

the evaluations. Adversarial Benchmarks. RobustBench [3] maintains stan-

Figure 7: Incremental evaluation with a controllable budget. Robustness– perturbation curves on ℓ∞ for a defense fA (C6) evaluated with the full E12k ensemble and a candidate fB (C9) evaluated with the cheaper E4k tier. The new curve is below the reference at 4k queries; more queries can only lower it further, thus fB can be directly rejected.

dardized leaderboards reporting robust accuracy at fixed budgets on ℓ2 and ℓ∞ norms via AA. Guo et al. [38] propose a multiview evaluation framework with 23 data- and model-oriented

fB at 4k is already much smaller than that of fA at 12k, fB can be 17

metrics, including test-example integrity, model structure, and

tances, black-box attacks and semantic or patch-based perturba-

adversarial behavior. AttackBench [8] takes a complementary

tions can be added to the pool at the sole cost of generalizing

view, proposing the Attack Optimality Index (AOI) to measure

the query unit.

how closely individual attacks approach the smallest achiev-

We believe this work positions curve-based, optimality-

able perturbation across models. However, none of these two

aware, and configurable evaluation as a practical replacement

approaches propose a method for combining attacks.

for fixed-ε leaderboards.

7. Conclusion and Future Work

Acknowledgements

We introduce a new framework for adversarial robustness

This work has been carried out while L. Scionis and L. Melis

evaluation. Our greedy algorithm selects a minimum-norm at-

were enrolled in the Italian National Doctorate on AI run by the

tack ensemble, including APGDmin , within a query budget to

Sapienza University of Rome in collaboration with the Univer-

estimate the complete robustness–perturbation curve. Its AOI

sity of Cagliari. This research has been partially supported by

quantifies how closely the ensemble approaches the empirical

the Horizon Europe projects Sec4AI4Sec (GA no. 101120393),

attack frontier. The ensembles generalize beyond the models

and CoEvolution (GA no. 101168560), and by project FISA-

used to construct them. Our 12k-query ensembles match or ex-

2023-00128 funded by the MUR program Fondo italiano per le

ceed AA’s AOI in most cases. The DOI summarizes the com-

scienze applicate.

plete curve relative to the empirical defense frontier, avoiding

Declaration on the use of generative AI and AI-assisted

the selection of one ε for the ranking. The nested tiers make

technologies. During the preparation of this work, the author(s)

both the query cost and the tightness of the frontier approxima-

used Claude to complement the initial literature review and to

tion controllable.

write, and Claude Code to polish the plots and visualizations.

Our framework has three main limitations, each pointing

The author(s) reviewed and edited the output as needed and take

to a natural extension. First, evaluations are pool-based: the

full responsibility for the content of the published article.

frontiers are empirical estimates and the DOI is relative to the shared defense pool. We mitigate this by fixing and re-

References

leasing the reference pool, so that new defenses are ranked [1] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić,

against a common frontier. Second, constructing the frontier

P. Laskov, G. Giacinto, F. Roli, Evasion attacks against

requires pre-computing distance–query trajectories for every

machine learning at test time, in: Machine Learning and

attack on the reference models, an offline cost that Q does

Knowledge Discovery in Databases, Vol. 8190 of Lecture

not control. Cached trajectories make extensions incremental,

Notes in Computer Science, Springer, 2013, pp. 387–402.

since adding an attack only requires its own runs, and warmstarting new runs from existing ones would reduce this cost fur-

[2] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan,

ther. Finally, the pool contains only white-box, gradient-based

I. J. Goodfellow, R. Fergus, Intriguing properties of neural

attacks, so defenses relying on gradient obfuscation or non-

networks, in: International Conference on Learning Rep-

differentiable components are not faithfully evaluated. Since

resentations, 2014.

the min-composition framework only consumes per-sample dis18

[3] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti,

[11] M. Pintor, F. Roli, W. Brendel, B. Biggio, Fast minimum-

N. Flammarion, M. Chiang, P. Mittal, M. Hein, Robust-

norm adversarial attacks through adaptive norm con-

Bench: A standardized adversarial robustness benchmark,

straints, Advances in Neural Information Processing Sys-

in: NeurIPS Datasets and Benchmarks, 2021.

tems 34 (2021) 20052–20062.

[4] F. Croce, M. Hein, Reliable evaluation of adversarial ro-

[12] S. Addepalli, S. Jain, G. Sriramanan, R. Venkatesh Babu,

bustness with an ensemble of diverse parameter-free at-

Scaling adversarial training to large perturbation bounds,

tacks, in: International Conference on Machine Learning,

in: European Conference on Computer Vision, Springer,

2020.

2022, pp. 301–316.

[5] B. Biggio,

F. Roli,

Wild patterns:

rise

adversarial

after

the

ing,

Pattern

Recognition

Ten years

machine (2018)

[13] R. Rade, S.-M. Moosavi-Dezfooli, Helper-based adversar-

learn-

ial training: Reducing excessive margin to achieve a better

317–331.

accuracy vs. robustness trade-off, in: ICML 2021 Work-

doi:https://doi.org/10.1016/j.patcog.2018.07.023.

shop on Adversarial Machine Learning, 2021.

[6] N. Carlini, A. Athalye, N. Papernot, W. Brendel,

[14] E.-C. Chen, C.-R. Lee, Data filtering for efficient adver-

J. Rauber, D. Tsipras, I. Goodfellow, A. Madry,

sarial training, Pattern Recognition 151 (2024) 110394.

A. Kurakin, On evaluating adversarial robustness, arXiv [15] M. Augustin, A. Meinke, M. Hein, Adversarial robustness

preprint arXiv:1902.06705 (2019).

on in-and out-distribution improves explainability, in: Eu[7] N. Risse, C. Göpfert, J. P. Göpfert, How to compare ad-

ropean Conference on Computer Vision, Springer, 2020,

versarial robustness of classifiers from a global perspec-

pp. 228–245.

tive, in: International Conference on Artificial Neural Net[16] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg,

works, Springer, 2021, pp. 29–41.

O. Wiles, T. Mann, Fixing data augmentation to improve [8] A. E. Cinà, J. Rony, M. Pintor, L. Demetrio, A. Demontis, adversarial robustness, arXiv preprint arXiv:2103.01946 B. Biggio, I. Ben Ayed, F. Roli, AttackBench: Evaluating (2021). gradient-based attacks for adversarial examples, Proceed[17] E. Debenedetti, V. Sehwag, P. Mittal, A light recipe

ings of the AAAI Conference on Artificial Intelligence

to train robust vision transformers, in:

39 (3) (2025) 2600–2608.

2023 IEEE

conference on secure and trustworthy machine learning

[9] C. Göpfert, J. P. Göpfert, B. Hammer, Adversarial ro-

(SaTML), IEEE, 2023, pp. 225–253.

bustness curves, in: Joint European Conference on Ma-

[18] V. Sehwag, S. Mahloujifar, T. Handina, S. Dai, C. Xiang,

chine Learning and Knowledge Discovery in Databases,

M. Chiang, P. Mittal, Robust learning meets generative

Springer, 2019, pp. 172–179.

models: Can proxy distributions improve adversarial ro-

[10] F. Croce, M. Hein, Mind the box: ℓ1 -APGD for sparse

bustness?, arXiv preprint arXiv:2104.09425 (2021).

adversarial attacks on image classifiers, in: International Conference on Machine Learning, 2021. 19

Advances in Neural Information Processing Systems 33

[19] A. Rodríguez-Muñoz, T. Wang, A. Torralba, Characteriz-

(2020) 3533–3545.

ing model robustness via natural input gradients, in: European Conference on Computer Vision, Springer, 2024,

[28] Y. Jiang, C. Liu, Z. Huang, M. Salzmann, S. Susstrunk,

pp. 161–178.

Towards stable and efficient adversarial training against ℓ1

[20] J. Cui, Z. Tian, Z. Zhong, X. Qi, B. Yu, H. Zhang, Decou-

bounded adversarial attacks, in: International Conference

pled kullback-leibler divergence loss, Advances in Neural

on Machine Learning, PMLR, 2023, pp. 15089–15104.

Information Processing Systems 37 (2024) 74461–74486. [29] P. Maini, E. Wong, Z. Kolter, Adversarial robustness [21] D. Stutz, M. Hein, B. Schiele, Confidence-calibrated ad-

against the union of multiple perturbation models, in:

versarial training: Generalizing to unseen attacks, in: In-

International Conference on Machine Learning, PMLR,

ternational conference on machine learning, PMLR, 2020,

2020, pp. 6640–6650.

pp. 9155–9166. [30] Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, S. Yan, Bet[22] N. D. Singh, F. Croce, M. Hein, Revisiting adversarial

ter diffusion models further improve adversarial training,

training for imagenet: Architectures, training and gener-

in: International conference on machine learning, PMLR,

alization across threat models, Advances in Neural Infor-

2023, pp. 36246–36263.

mation Processing Systems 36 (2023) 13931–13955. [31] Y. Xu, Y. Sun, M. Goldblum, T. Goldstein, F. Huang, Ex[23] X. Zhong, Y. Huang, C. Liu, Towards efficient training

ploring and exploiting decision boundary dynamics for

and evaluation of robust models against ℓ0 bounded ad-

adversarial robustness, arXiv preprint arXiv:2302.03015

versarial perturbations, in: Forty-first International Con-

(2023).

ference on Machine Learning, 2024. [32] A. Matyasko, L.-P. Chau, PDPGD: Primal-dual proxi[24] E. Wong, L. Rice, J. Z. Kolter, Fast is better than mal gradient descent adversarial attack, arXiv preprint free:

Revisiting adversarial training, arXiv preprint arXiv:2106.01538 (2021).

arXiv:2001.03994 (2020). [33] J. Rony, E. Granger, M. Pedersoli, I. Ben Ayed, Aug-

[25] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A.

mented lagrangian adversarial attacks, in: Proceedings of

Calian, T. A. Mann, Improving robustness using generated

the IEEE/CVF International Conference on Computer Vi-

data, Advances in neural information processing systems

sion, 2021, pp. 7738–7747.

34 (2021) 4218–4233.

[34] J. Rony, L. G. Hafemann, L. S. Oliveira, I. Ben Ayed,

[26] T. Pang, M. Lin, X. Yang, J. Zhu, S. Yan, Robustness

R. Sabourin, E. Granger, Decoupling direction and norm

and accuracy could be reconcilable by (proper) definition,

for efficient gradient-based ℓ2 adversarial attacks and de-

in: International conference on machine learning, PMLR,

fenses, in: Proceedings of the IEEE/CVF Conference

2022, pp. 17258–17277.

on Computer Vision and Pattern Recognition, 2019, pp. [27] H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, A. Madry,

4322–4330.

Do adversarially robust imagenet models transfer better?, 20

[35] A. E. Cinà, F. Villani, M. Pintor, L. Schönherr, B. Big-

[37] S. Liu, F. Peng, K. Tang, Reliable robustness evaluation

gio, M. Pelillo, σ-zero: Gradient-based optimization of

via automatically constructed attack ensembles, in: Pro-

ℓ0 -norm adversarial examples, in: The Thirteenth Interna-

ceedings of the AAAI Conference on Artificial Intelli-

tional Conference on Learning Representations, 2025.

gence (AAAI), 2023.

[36] C. Yao, P. Bielik, P. Tsankov, M. Vechev, Automated

[38] J. Guo, W. Bao, J. Wang, Y. Ma, X. Gao, G. Xiao, A. Liu,

discovery of adaptive attacks on adversarial defenses,

J. Dong, X. Liu, W. Wu, A comprehensive evaluation

Advances in Neural Information Processing Systems 34

framework for deep model robustness, Pattern Recogni-

(2021) 26858–26870.

tion 137 (2023) 109308.

Record · ID 394340 · SHA-256 307f8dac648bc51f
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.