Surrogate assisted diversity estimation in neural ensemble search⋆ Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
arXiv:2607.26940v1 [cs.LG] 29 Jul 2026
{udeneev.av, p.k.babkin, bakhteev.o} (at) gmail.com
Abstract. Ensembles are a standard way to improve the performance and robustness of deep neural networks, but their effectiveness crucially depends on both the quality and the diversity of individual models. Most neural architecture search (NAS) methods are computationally expensive. Extending them to neural ensemble search (NES), which requires joint optimization of individual architectures and their ensemble composition, leads to an exponential growth of the search space and makes the problem computationally intractable. To address this, we introduce a dual-objective surrogate-guided ensemble search: candidate architectures are represented as directed acyclic graphs, and two surrogate models are trained independently to estimate predictive accuracy and diversity potential. Their combined estimates guide an NES framework that efficiently identifies architectures that are both individually strong and collectively diverse. Our final ensemble achieves competitive or superior performance compared to standard baselines such as Deep Ensembles and Random Search on FashionMNIST, CIFAR-10, and CIFAR-100. Keywords: neural ensemble search · ensemble diversity · surrogate function · triplet loss.
1
Introduction
Neural network ensembles consistently demonstrate improved predictive accuracy, robustness, and uncertainty estimation compared to single models, particularly in classification and regression tasks [1, 2]. This observation motivates the study of Neural Ensemble Search (NES), which aims to construct ensembles of neural networks in a principled and computationally efficient manner [3, 4]. In practice, NES relies on Neural Architecture Search (NAS) techniques to identify suitable candidate architectures, a task that remains challenging due to the vastness of the search space and the high cost of training individual models [5]. A widely used baseline for ensemble construction is Deep Ensembles (DeepEns) [6], which combine multiple independently trained instances of the same architecture. While simple and effective, this approach offers limited control over ensemble diversity, often resulting in highly correlated models. More recent NES ⋆
The final publication is available at Springer: https://link.springer.com/chapter/ 10.1007/978-3-032-30612-8_12. DOI: https://doi.org/10.1007/978-3-032-30612-8_12.
2
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
methods adapt NAS techniques to ensemble construction [4, 7], but diversity in these approaches typically emerges implicitly through optimization dynamics or heuristic similarity measures, rather than being explicitly modeled and optimized alongside accuracy.
0 0 1
1 2
2 3
3 ep
αp 1
1
1
1
d1
αa
1 1
1
1
0
1
ea
d1 ≤ d 2
d2 0
1
0
0
0
αn 0 1
Architecture space
en
Latent space
2 3
Fig. 1: Illustration of the surrogate diversity function. Neural architectures are embedded into a latent space such that architectures with similar predictive behavior are mapped close together, while dissimilar ones are separated by larger distances. This enables efficient diversity estimation without training candidate models.
To address these limitations, we draw inspiration from surrogate-assisted NAS methods, which employ lightweight predictive models to estimate architecture performance without full training [8, 9]. While surrogates are commonly used to predict accuracy, recent work suggests that they can also capture structural and functional relationships between architectures [10]. This observation motivates their use for explicit diversity modeling in NES. We propose a dual-objective surrogate-guided framework for Neural Ensemble Search. The framework separates the search into two complementary predictive objectives: accuracy estimation for individual architectures and diversity modeling via latent architecture embeddings. Candidate architectures are encoded as graphs and mapped into a continuous latent space using surrogate models, where geometric distances correspond to predictive dissimilarity. As illustrated in Fig. 1,
Surrogate assisted diversity estimation in neural ensemble search
3
this representation enables explicit, differentiable, and training-free evaluation of ensemble diversity. We implement the surrogate models using Graph Attention Networks (GATs) [11] and structure the latent space using the triplet loss [12] for metric learning. Our main contributions are: 1. We introduce a novel surrogate-based methodology for training diversity predictors that map discrete neural architectures to a structured latent space. 2. We present the surrogate-assisted framework for Neural Ensemble Search that jointly optimizes predictive performance and architectural diversity via dual-objective guidance. To the best of our knowledge, this is one of the first methods to explicitly model diversity in NES using surrogate-based latent representations. 3. We provide a proof-of-concept validation of our framework on FashionMNIST [13], CIFAR-10 [14], and CIFAR-100 [14], demonstrating its capability to balance predictive performance and architectural diversity effectively. 4. We provide datasets of approximately 3,000 trained models per dataset (including both weights and architectures) for FashionMNIST, CIFAR-10, and CIFAR-100 [15]. These datasets enable evaluation of ensemble methods without the need to train large numbers of models from scratch. Training details are provided in Section 4.1.
2
Problem statement
Neural Architecture Search methods have proven effective at discovering highperforming individual models. However, many predictive tasks benefit from ensemble strategies that aggregate multiple models rather than relying on a single architecture. The challenge lies in the fact that an optimal ensemble requires not merely a collection of strong models, but a set of architectures that actively complement one another. This motivates our formulation of Neural Ensemble Search as a unified optimization problem, where architectural selection and ensemble composition are jointly optimized. We begin by reviewing the standard NAS framework, then extend it to the ensemble setting. 2.1
Neural Architecture Search
We follow the standard Neural Architecture Search (NAS) formulation [16], where the goal is to identify an architecture α ∈ A that minimizes the validation loss after training: min Lval (α, ω ∗ (α)), α∈A (1) s.t. ω ∗ (α) = arg min Ltrain (α, ω). ω∈W
Here, A denotes a search space of neural architectures represented as directed acyclic graphs with operations assigned to edges. The primary challenge in NAS lies in the enormous size of the search space (e.g., ∼ 1025 in DARTS [16]), making exhaustive search infeasible.
4
2.2
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
Neural Ensemble Search
The primary objective of Neural Ensemble Search (NES) is to identify a set of complementary architectures S ⊂ A that jointly minimize the validation loss. We denote by α ∈ A a neural architecture and by ω(α) its corresponding parameters obtained by minimizing the training loss. The prediction of a model with architecture α on input x is denoted by fα (x, ω(α)). The NES problem can then be formulated as: ! 1 X ∗ min Lval fα (x, ω (α)) , S⊂A |S| (2) α∈S
s.t.
∀α ∈ S : ω ∗ (α) = arg min Ltrain (fα (x, ω)). ω
Compared to NAS, NES requires not only identifying high-performing individual architectures, but also selecting a subset of models that complement each other, significantly increasing the complexity of the search problem.
3
Surrogate-based ensemble search
In this work, we adapt the DARTS architecture space representation into a graphstructured form that can be processed by Graph Attention Networks (GATs) (see Section 3.1). In Section 3.2, we present the architecture of the surrogate functions and describe their operating principles. Section 3.3 then details the ensemble construction method based on these surrogate models, explaining how surrogate predictions are used to promote architectural diversity within the ensemble. Together, these components enable the construction of ensembles that jointly consider predictive accuracy and architectural diversity. 3.1
Architecture Search Space
Following the conventions of [16], we instantiate the search space A using a cell-based approach. Each architecture α ∈ A is composed of two functional units: a normal cell αnorm and a reduction cell αred , such that α = (αnorm , αred ). Both units are DAGs adhering to the formalisms established in Section 2.1. Unlike the original DARTS, which relies on continuous relaxation, our method explores A via discrete random sampling. We consider a reduced configuration with n = 5 nodes and m = 10 edges, where each node has exactly two incoming and two outgoing edges. This choice follows the general structural design of DARTS, while reducing the size of the search space and the computational cost of training the surrogate dataset. Given an operation set size |O| = 7, the cardinality of the cell-level search space is: |Acell | =
n−1 Y k=2
k · |O|m ≈ 109 . 2
(3)
Surrogate assisted diversity estimation in neural ensemble search
5
Consequently, the full architecture space |A| = |Acell |2 ≈ 1018 remains computationally intractable for exhaustive search, necessitating surrogate-guided exploration. To train the dual-surrogate models, we construct a dataset Dtrain by evaluating N sampled architectures: Dtrain = {(αi , yi , acci )}N i=1 ,
(4)
where yi = fαi (Xval , ω ∗ (αi )) is the vector of model predictions on a fixed validation subset Xval , and acci is the corresponding validation accuracy. These observations serve as the ground truth for our surrogate functions. 3.2
Surrogate Function
To facilitate ensemble construction (Section 3.3), we estimate both the performance and diversity of candidate architectures using surrogate models. Due to the size of the search space, these properties cannot be obtained via full training. We therefore introduce two surrogate functions: θ facc : A → R,
θ fdiv : A → Rd .
(5)
Each architecture is represented as a directed acyclic graph. For surrogate modeling, we use an operation-centric graph representation, where each edge of the original architecture is treated as a node, and connections are introduced between sequential operations. Operations are represented via one-hot encoding. Both surrogates are implemented as Graph Attention Networks (GATs) with residual connections, GraphNorm [17], and global pooling. The accuracy surrogate facc is trained via supervised regression using the mean squared error: ∗ θacc = arg min θ
N X
θ facc (αi ) − acci
2
.
(6)
i=1
The diversity surrogate fdiv is trained using a triplet loss to enforce proximity between architectures with similar predictions and separation otherwise. Positive and negative pairs are defined based on high and low agreement in model predictions. Given model predictions {y (i) }N i=1 on a validation set, we define a similarity matrix: K 1 X (i) (j) cij = I yk = yk , K k=1
which is discretized into D ∈ {−1, 0, 1}N ×N using quantile thresholds. The surrogate maps architectures into a latent space: θdiv ea = fdiv (αa ),
(7)
6
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
and is optimized via the triplet loss: X Ldiv = max ∥ea − ep ∥22 − ∥ea − en ∥22 + m, 0 .
(8)
(a,p,n)
This training encourages a latent space where Euclidean distances reflect predictive diversity. The full training procedure is summarized in Fig. 2.
Input: Dtrain : training dataset of architectures {(αi , yi , acci )}N i=1 ; N : number of architectures in the dataset; n: number of training epochs; D: diversity matrix where Djk ∈ {−1, 0, 1} (negative/neutral/positive); m: triplet loss margin; η: learning rate; B: batch size. Output: Optimized parameters θdiv for surrogate model fdiv 1 Initialize parameters θdiv 2 for epoch e ← 1 to n do 3 for step t ← 1 to N/B do 4 Sample a minibatch of anchor indices J ⊂ {1, . . . , N }, where |J | = B 5 for each j ∈ J do 6 Identify candidate sets: Pj = {k | Djk = 1} and Nj = {k | Djk = −1} 7 Sample positive kp ∼ Uniform(Pj ) and negative kn ∼ Uniform(Nj ) 8 Compute embeddings: θdiv θdiv θdiv 9 ea , ep , en ← fdiv (αj ), fdiv (αkp ), fdiv (αkn ) 10 Compute triplet loss for instance j: 11 ℓj ← max 0, ||ea − ep ||22 − ||ea − en ||22 + m 12 end P 13 Compute batch loss: L = B1 ℓ j∈J j 14 Update parameters: θdiv ← θdiv − η∇θdiv L 15 end 16 end 17 return θdiv
Fig. 2: Training procedure for the diversity surrogate function.
3.3
Ensemble Construction
θdiv θacc Once the surrogate models facc and fdiv are trained, they enable an efficient search for the optimal ensemble without requiring the prohibitive computational cost of training intermediate candidate architectures. The proposed ensemble construction (Fig 3) proceeds in two distinct phases: candidate pool filtering and diversity-driven greedy selection.
Surrogate assisted diversity estimation in neural ensemble search
7
Input: K: target ensemble size; N : number of initial candidates; λ: accuracy threshold; facc , fdiv : trained surrogate models. Output: Abest : selected ensemble of K architectures. // Phase 1: Candidate Pool Construction 1 Acand ← RandomSample(N ) 2 P ← (α, a, e) | α ∈ Acand , a = facc (α), e = fdiv (α), a ≥ λ // Phase 2: Diversity-driven Greedy Selection ∗ ∗ ∗ 3 Select triplet with highest predicted accuracy: (α , a , e ) = arg max(α,a,e)∈P a ∗ ∗ ∗ ∗ ∗ 4 Abest ← {α }; Esel ← {e }; P ← P \ {(α , a , e )} 5 while |Abest | < K and P ̸= ∅ do 6 For each candidate (αi , ai , ei ) ∈ P, compute P 1 ∥ei − esel ∥2 di = |Esel | esel ∈Esel 7 Select triplet with maximum diversity: (αi∗ , ai∗ , ei∗ ) = arg max(αi ,ai ,ei )∈P di 8 Update ensemble: Abest ← Abest ∪ {αi∗ }; Esel ← Esel ∪ {ei∗ } 9 P ← P \ {(αi∗ , ai∗ , ei∗ )} 10 end 11 return Abest
Fig. 3: Surrogate-Assisted Ensemble Construction.
In the first phase, we generate a large set of random candidate architectures Acand via discrete sampling from the search space. We then use the dual-surrogate framework to predict the accuracy a and the latent diversity embedding e for each candidate. To ensure the base quality of the ensemble, we apply an accuracy threshold λ, forming a pruned candidate pool P that contains only high-performing models. The second phase employs a greedy forward-selection strategy to maximize architectural diversity. Starting with the most accurate model as an anchor, the algorithm iteratively selects subsequent architectures that exhibit the maximum average Euclidean distance from the currently selected set in the latent space. By leveraging the geometric organization of the latent space provided by fdiv , this selection process explicitly favors architectures with dissimilar prediction patterns. This approach effectively identifies a diverse set of high-performing models, balancing individual predictive power with collective ensemble robustness.
4
Computational Experiments
In this section, we present the experimental results as well as the metrics used for comparison. In Section 4.1, we describe the dataset collection procedure. Section 4.2 details the training setup for the surrogate functions. Section 4.3 presents ablation studies analyzing the contribution of each surrogate function and the
8
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
number of trained models required for surrogate learning. Finally, in Section 4.4, we compare our proposed method against DeepEns [6] and Random Search [18] for FashionMNIST, CIFAR-10 and CIFAR-100, respectively. 4.1
Construction of the Training Dataset for Surrogate Models
The models in our dataset are trained according to the process described in Fig. 4. Training follows [16] with minor modifications; the main hyperparameters are summarized in Table 1.
Table 1: Training Hyperparameters and Performance per Dataset Dataset
Num. Cells Initial Width Num. Epochs Avg. Acc. (%) Avg. Top-1 Agreement FashionMNIST 3 16 125 89.6 ± 0.5 0.900 ± 0.004 CIFAR-10 8 16 200 75.8 ± 0.6 0.693 ± 0.006 CIFAR-100 8 16 200 37.6 ± 1.1 0.324 ± 0.008
It is important to note that the architectures used to construct the surrogate training dataset are intentionally trained in a reduced configuration (in terms of depth and width) to limit computational cost. Moreover, these models are not trained to full convergence, but only for a fixed number of epochs sufficient to obtain reliable relative performance estimates. In contrast, the final ensemble models are trained using a larger configuration, with increased number of cells and channel width. This design allows the surrogate functions, learned on lightweight and partially trained models, to effectively guide the selection of higher-capacity architectures during ensemble construction, significantly reducing the overall computational cost. To improve efficiency, we employ an iterative surrogate-assisted refinement: we first train the accuracy surrogate on N1 = 1000 randomly sampled architectures, then restrict the search to the top 10% candidates and sample an additional N2 = 2000 architectures from this reduced space. Constructing the surrogate dataset remains the main computational bottleneck. In our experiments, training approximately 3,000 architectures for each dataset, CIFAR-10 and CIFAR-100, required about three days per dataset on four NVIDIA A100 GPUs (80 GB each). 4.2
Training of the Surrogate Functions
Each architecture is represented as a cell with n = 5 nodes and m = 10 edges, where operations are sampled from a standard DARTS search space, including separable and dilated convolutions, pooling, and skip connections. We use a Graph Attention Network (GAT) with four convolutional layers and two fully connected layers to model both surrogate functions, trained with a cosine annealing learning rate schedule.
Surrogate assisted diversity estimation in neural ensemble search
9
Input: ADART S : architectures available through DARTS; X : dataset for train and validate architectures; N : number of architectures. Output: Model dataset D = {(αi , yi , acci )}N i=1 . 1 Split dataset X into training and validation subsets: Xtrain , Xval with a 20%/80% ratio 2 for i ← 1 to N do 3 Sample architecture αi ∼ Uniform(ADART S ) 4 Train model fαi (Xtrain , ω(αi )) 5 Evaluate fαi on Xval 6 prediction vector yi = fαi (Xval , ω ∗ (αi )) 7 validation accuracy P 1 acci = |Xval I arg maxc fαi (x; ω ∗ (αi )) = y | (x,y)∈Xval 8 D ← D ∪ (αi , yi , acci ) 9 end 10 return D
Fig. 4: Construction of the Model Evaluation Dataset
Full models are constructed by stacking normal and reduction cells in a 2:1 ratio with predefined initial channels. 4.3
Ablations of the Surrogate Functions
The main computational bottleneck of the proposed framework is the need to construct a large dataset of trained architectures. In this section, we analyze how the performance of the surrogate models depends on the training set size. All experiments are conducted on CIFAR-100 as the most challenging dataset. We begin with the accuracy surrogate, which is used to filter candidate architectures based on their predicted performance. We hold out 600 architectures and evaluate how well the surrogate retrieves high-performing models using Spearman correlation and Recall@K. The results are presented in Table 2. Both Spearman correlation and Recall@K improve as the number of training architectures increases, indicating that the surrogate becomes more effective at identifying strong models. The diversity surrogate is the core component of our method. To evaluate it, we again hold out 600 architectures. We measure (i) Spearman correlation between distances in the latent space and prediction similarity, and (ii) Recall@K, which reflects how well similar models are placed close to each other in the embedding space. For the diversity surrogate, the Spearman correlation is negative, as larger distances in the latent space correspond to lower similarity between model predictions. We observe that the diversity surrogate rapidly learns a meaningful latent structure: both correlation and Recall@K improve significantly when increasing
10
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
Table 2: Surrogate performance as a function of training set size. Results are averaged over 10 runs; subscripts denote standard deviation. Type
Train Size Spearman
R@10
R@50
R@100
Accuracy
0 250 500 750 1000 1500 2000
0.017±0.051 0.106±0.106 0.220±0.058 0.218±0.079 0.368±0.050 0.535±0.060 0.675±0.021
0.010±0.030 0.076±0.017 0.167±0.021 0.040±0.049 0.130±0.039 0.228±0.052 0.090±0.083 0.196±0.064 0.291±0.034 0.120±0.125 0.204±0.061 0.291±0.046 0.120±0.087 0.264±0.073 0.330±0.025 0.170±0.135 0.294±0.073 0.366±0.032 0.200±0.089 0.324±0.032 0.430±0.031
Diversity
0 250 500 750 1000 1500 2000
−0.001±0.029 0.019±0.003 0.092±0.005 0.179±0.006 −0.490±0.042 0.029±0.004 0.131±0.008 0.249±0.011 −0.545±0.035 0.031±0.003 0.136±0.005 0.261±0.008 −0.574±0.025 0.029±0.003 0.135±0.006 0.259±0.010 −0.602±0.025 0.031±0.003 0.142±0.007 0.271±0.012 −0.639±0.022 0.032±0.003 0.147±0.005 0.281±0.007 −0.637±0.025 0.032±0.004 0.145±0.007 0.278±0.008
the number of training architectures up to approximately 500–1000 samples, after which the gains become marginal. This suggests that reliable diversity estimation can be achieved without requiring the full surrogate training dataset. We additionally perform a component ablation to assess the contribution of each surrogate function. The results are presented in Table 3.
Table 3: Component ablation on CIFAR-100. Method Random Accuracy only Diversity only Accuracy + Diversity
4.4
Top-1 Acc. (%) Avg. Model Acc. (%) Pred. Disagreement 84.41±0.09 85.07±0.10 84.77±0.07 85.17±0.16
79.57±0.19 80.54±0.12 80.08±0.21 80.50±0.12
0.430±0.004 0.406±0.002 0.419±0.008 0.411±0.004
Ensemble Performance Across Datasets
Across all datasets, the proposed Surrogate Ensemble consistently achieves the best or competitive performance in terms of Top-1 accuracy and NLL (Table 4). The DeepEns baseline is constructed by repeatedly training a single architecture obtained via DARTS with different random initializations. On FashionMNIST, all methods perform similarly due to the low task complexity; however, the Surrogate Ensemble attains the highest accuracy and average
Surrogate assisted diversity estimation in neural ensemble search
11
model performance. While Random Search yields higher diversity, this does not translate into improved ensemble accuracy.
Table 4: Ensemble performance on FashionMNIST, CIFAR-10, and CIFAR-100.
FashionMNIST
Top-1 Acc. (%) Avg. Model Acc. (%) NLL Oracle NLL Brier Score ECE Ambiguity Norm. Disagreement Pred. Disagreement FGSM AUC PGD AUC
95.02±0.35 94.55±0.43 0.266±0.006 0.206±0.003 0.091±0.006 0.121±0.001 0.0047±0.0009 0.062±0.008 0.125±0.016 0.056±0.0036 0.024±0.0029
95.01±0.19 94.32±0.24 0.265±0.003 0.191±0.010 0.092±0.002 0.120±0.004 0.0069±0.0013 0.077±0.009 0.154±0.018 0.056±0.0012 0.022±0.0010
CIFAR-10
Surrogate Ens. DeepEns Random Search
Top-1 Acc. (%) 97.80±0.08 97.64±0.09 Avg. Model Acc. (%) 96.73±0.10 96.71±0.14 NLL 0.208±0.003 0.210±0.003 Oracle NLL 0.156±0.002 0.154±0.004 Brier Score 0.055±0.001 0.056±0.001 ECE 0.136±0.002 0.135±0.002 Ambiguity 0.011±0.001 0.009±0.001 Norm. Disagreement 0.044±0.001 0.046±0.003 Pred. Disagreement 0.087±0.003 0.093±0.007 FGSM AUC 0.0569±0.0012 0.0555±0.0021 PGD AUC 0.0273±0.0015 0.0251±0.0020
97.56±0.11 96.48±0.09 0.212±0.002 0.158±0.003 0.057±0.001 0.136±0.001 0.011±0.001 0.046±0.001 0.092±0.003 0.0559±0.0012 0.0256±0.0018
CIFAR-100
Dataset Metric
Top-1 Acc. (%) Avg. Model Acc. (%) NLL Oracle NLL Brier Score ECE Ambiguity Norm. Disagreement Pred. Disagreement FGSM AUC PGD AUC
84.41±0.09 79.57±0.19 0.718±0.006 0.418±0.006 0.244±0.002 0.141±0.002 0.048±0.002 0.215±0.002 0.430±0.004 0.0358±0.001 0.0167±0.0005
95.3±0.1 94.7±0.1 0.263±0.003 0.199±0.010 0.089±0.001 0.124±0.002 0.0058±0.0009 0.065±0.008 0.130±0.016 0.058±0.0025 0.023±0.0018
85.17±0.16 80.50±0.12 0.692±0.007 0.403±0.004 0.234±0.002 0.140±0.004 0.047±0.001 0.206±0.002 0.411±0.004 0.037±0.001 0.017±0.001
85.10±0.29 80.97±0.41 0.682±0.010 0.406±0.007 0.232±0.004 0.133±0.003 0.041±0.002 0.195±0.005 0.390±0.010 0.0368±0.001 0.0158±0.001
On CIFAR-10, the advantage of the proposed approach becomes more evident. The Surrogate Ensemble achieves the best Top-1 accuracy and NLL, while maintaining competitive diversity. This indicates that the method effectively balances individual model quality and ensemble diversity.
12
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
On CIFAR-100, the most challenging dataset, the Surrogate Ensemble achieves the highest Top-1 accuracy despite slightly lower average individual model accuracy compared to DeepEns. This highlights the benefit of selecting complementary architectures rather than optimizing models independently. In contrast, Random Search exhibits higher diversity but suffers from weaker individual models, leading to inferior ensemble performance. Overall, these results demonstrate that explicitly modeling diversity is beneficial for constructing effective ensembles, particularly as dataset complexity increases.
5
Conclusion
In this study, we showed that the proposed surrogate diversity function learns a latent embedding of model architectures in which inter-model distances consistently reflect high levels of predictive diversity across FashionMNIST, CIFAR-10, and CIFAR-100. Building on this observation, we proposed a surrogate-guided ensemble construction framework that jointly optimizes accuracy and diversity, demonstrating consistent improvements over standard baselines. Our results further indicate that the role of diversity becomes increasingly important as dataset complexity grows, highlighting the necessity of principled diversity modeling in challenging settings. A key limitation of the proposed approach is the computational cost associated with training a large pool of base models to construct the surrogate dataset. Future work may mitigate this overhead through more efficient sampling strategies or weight-sharing mechanisms, as well as by replacing discrete similarity measures with continuous information-theoretic metrics to improve the smoothness and expressiveness of diversity estimation.
References
[1] Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993–1001, 1990. https://doi.org/10.1109/34.58871. [2] Ye Ren, Le Zhang, and P. N. Suganthan. Ensemble classification and regression–recent developments, applications and future directions. IEEE Computational Intelligence Magazine, 11(1):41–53, February 2016. ISSN 1556-603X. https://doi.org/10.1109/MCI.2015.2471235. [3] Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. AdaNet: Adaptive structural learning of artificial neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 874–883. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/cortes17a.html. [4] Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris C. Holmes, Frank Hutter, and Yee Whye Teh. Neural ensemble search for uncertainty estimation and dataset shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34, pages 7898–7911, 2021. [5] Zhichao Lu, Ran Cheng, Shihua Huang, Haoming Zhang, Changxiao Qiu, and Fan Yang. Surrogate-assisted multi-objective neural architecture search for real-time semantic segmentation. CoRR, abs/2208.06820, 2022. https: //doi.org/10.48550/ARXIV.2208.06820. [6] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeurIPS, volume 30, pages 6405–6416, 2017. [7] Minghao Chen, Jianlong Fu, and Haibin Ling. One-shot neural ensemble architecture search by diversity-guided search space shrinking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16530–16539, June 2021. [8] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, ECCV, pages 660–676. Springer, 2020. https://doi.org/10.1007/978-3-030-58580-8_39. [9] Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10293–10301, 2021. [10] Yu Xue, Zhenman Zhang, and Ferrante Neri. Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy. Electronic Research Archive, 32(2):1017–1043, 2024. ISSN 2688-1594. https: //doi.org/10.3934/era.2024050.
14
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
[11] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. CoRR, abs/1710.10903, 2017. https://doi.org/10.48550/arxiv.1710.10903. [12] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, pages 815–823, 2015. https://doi.org/10.1109/CVPR.2015.7298682. [13] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. [14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. [15] Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev. NES surrogate architectures, 2026. URL https://huggingface.co/datasets/Demoren/ nes-surrogate-architectures. Revision b6b3f3c. [16] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR. OpenReview.net, 2018. [17] Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, and Liwei Wang. Graphnorm: A principled approach to accelerating graph neural network training. In International Conference on Machine Learning, pages 1204–1215. PMLR, 2021. [18] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In J. Peters and D. Sontag, editors, UAI, volume 124, pages 367–377. PMLR, 2020.
Surrogate assisted diversity estimation in neural ensemble search
A
15
NAS-Bench formats
Fig. 5: Combined normal and reduced cells. The red vertices belong to the reduction cell; the blue vertices belong to the normal cell.
16
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
B
Adversarial attacks
Plots of adversarial attacks shown on Figure 6.
100
FGSM Attack
BIM Attack
PGD Attack
FashionMNIST Accuracy (%)
80 60 40 20 0 100
CIFAR-10 Accuracy (%)
80 60 40 20 0
CIFAR-100 Accuracy (%)
80 60 40 20 0
0.00 0.02 0.04 0.06 0.08 0.10 0.00 0.02 0.04 0.06 0.08 0.10 0.00 0.02 0.04 0.06 0.08 0.10 Surrogate Ensemble
Random Search
DeepEns
Fig. 6: Accuracy of the surrogate ensemble under FGSM, BIM, and PGD attacks across increasing ε for FashionMNIST, CIFAR-10 and CIFAR-100.
Surrogate assisted diversity estimation in neural ensemble search
17
Fig. 7: Conversion of an architecture to the NAS-Bench-101 format.
C
Distributions of models in dataset
The main distributions of trained models you can see on Figure 8 and Figure 9:
18
Alexandr Udeneev, Petr Babkin, and Oleg Bakhteev
Fig. 8: Distribution of model accuracies
D
Fig. 9: Distribution of inter-model diversity
Training curve for surrogate funcitons
You can see the process of training accuracy surrogate function on Figure 11 and simularity surrogate function on Figure 10
Fig. 10: Training curve of the surrogate Fig. 11: Training curve of the surrogate model for diversity model for accuracy