DIFFNAS: Predictor-Free Neural Architecture Search via Diffusion Model and Zero-Cost Evaluation

DIFFNAS: Predictor-Free Neural Architecture Search via Diffusion Model and Zero-Cost Evaluation Yize Sun1,2,∗ Hanzhang Ma3,∗ Zixin Wu4 Xuechun Li1 Volker Tresp1,2 Yunpu Ma1,2

arXiv:2610.41818v1 [cs.LG] 30 Oct 2026

1

Ludwig-Maximilians-Universität München (LMU Munich), Germany 2 Munich Center for Machine Learning (MCML), Germany 3 Paderborn University, Germany 4 Faculty of Computer Science, University of Duisburg-Essen, Germany ∗

Equal contribution. Correspondence to: Yunpu Ma <cognitive.yunpu@gmail.com>.

Abstract Neural Architecture Search (NAS) has advanced the automated design of deep networks, but remains computationally expensive due to reliance on model training or performance predictors. We introduce DIFFNAS, a predictor-free NAS framework that leverages a discrete graph diffusion model guided by zero-cost performance proxies. Unlike prior diffusion-based NAS methods that require learned performance predictors, DIFFNAS conditions the generative process on training-free metrics such as MeCo and SWAP, enabling efficient architecture search with minimal computational overhead. Experiments on NAS-Bench-201 show that DIFFNAS finds the optimal architectures for CIFAR-10 and CIFAR-100 (94.37% and 73.51%) without a pretrained predictor, while reducing search cost by up to 17.5× compared to DiffusionNAG. On the MobileNetV3 search space, its accuracy is lower than that of DiffusionNAG. These results highlight the promise of diffusion models combined with zero-cost proxies for scalable and effective architecture search.

1

Introduction

Designing efficient neural network architectures plays an important role in modern deep learning, but it remains a compute-intensive task [Peng et al., 2022]. Neural Architecture Search (NAS) can automate the process but also typically has high computational cost, with numerous methods needing either thousands of candidate architectures trained or tested [Elsken et al., 2019]. Even predictor-based methods attempt to save this cost by learning performance predictors, with those methods also requiring significant amounts of training and adding new sources of bias and error. The fundamental problem is in the search for performing architectures in a resource-efficient way, independent of costly evaluations or surrogate model training. Existing methods for NAS are broadly classified into predictor-based [Wen et al., 2020, Wei et al., 2022], training-free zero-shot methods [Mellor et al., 2021, Jiang et al., 2023, Peng et al., 2024], and diffusion process-based generative models [An et al., 2024]. Predictor-based methods learn a proxy model that approximates the performance of potential architectures, but this lowers search cost in exchange for an increased burden of training and labeled data. Training-free methods employ handcrafted metrics, such as gradient magnitudes or feature correlation statistics, to estimate architecture quality without requiring training, but their performance is task- and search spacedependent. More recently, diffusion-based methods have demonstrated potential in representing the distribution of top-scoring architectures [An et al., 2024, Asthana et al., 2024]. Nonetheless, those methods usually need to train complicated dataset-informed predictors for conducting the generative process, rendering scalability and generalizability challenging. Preprint.

Figure 1: Illustration of diffusion framework and search pipeline for DIFFNAS. In this work, we propose DIFFNAS, a predictor-free NAS framework that combines discrete graph diffusion with zero-cost performance proxies to efficiently generate high-quality architectures. Instead of relying on learned performance predictors, DIFFNAS conditions the generative process directly on training-free metrics, primarily MeCo [Jiang et al., 2023], with SWAP [Peng et al., 2024] as an alternative, which evaluate candidate architectures using a single forward pass through untrained networks. This enables a scalable and lightweight search strategy that maintains high accuracy without expensive model evaluations or predictor training. Our main contributions are summarized as follows: • We propose DIFFNAS, a predictor-free NAS framework that integrates discrete graph diffusion with zero-cost performance proxies for efficient and scalable architecture generation. • We introduce a conditioning mechanism that incorporates training-free metrics such as MeCo and SWAP into both the denoising and sampling processes. • We enhance the diffusion model with a reward-guided fine-tuning stage using DDPO, aligning generation with task-specific objectives without training any performance predictor. • We demonstrate that DIFFNAS finds the optimal architectures for CIFAR-10 and CIFAR100 on NAS-Bench-201 while reducing search cost by up to 17.5× compared to the baseline (DiffusionNAG), and that its generated architectures are consistently strong, with the highest minimum accuracy among generative methods.

2

Related Work

Neural Architecture Search It is an expensive and unintuitive process for human designers to manually construct neural architectures with strong performance for various downstream tasks. NAS aims to automatically find architectures with strong performance after training [Elsken et al., 2019]. Popular NAS methods can be roughly categorized into Bayesian method-based [Kandasamy et al., 2018, White et al., 2021], reinforcement learning-based [Zoph and Le, 2016, Jaafra et al., 2019], gradient-based [Luo et al., 2018, Liu et al., 2018], evolutionary algorithm-based [Real et al., 2019, Lu et al., 2020], predictor-based [Wen et al., 2020, Wei et al., 2022], transferable [Lee et al., 2021, An et al., 2024], and training-free NAS [Mellor et al., 2021, Wu and Tsai, 2024]. The first four types of methods can be computationally expensive, since candidate architectures need to be trained and validated to evaluate their performance. Therefore, the predictor-based NAS builds a performance estimator that predicts the performance of candidate architectures to reduce computational cost. However, extra training is required to train predictors to predict the model’s performance accurately based on various downstream tasks. To tackle this challenge, the transferable and training-free NAS investigates designing transferable estimators and training-free indicators. Recently, DiffusionNAG [An et al., 2024] proposed an approach that generates architectures satisfying the objective with a guidance scheme of a meta-learned dataset-aware predictor. However, it still needs extra efforts to train a meta-learned dataset-aware predictor as the model-guidance scheme, 2

while our novel NAS framework eliminates the need for extra predictors by leveraging a diffusion model guided by a zero-cost proxy. Zero-Shot Indicators The key component of training-free NAS is defining an indicator or measurement to estimate the performance of candidate architectures instead of training and validating them on a specific dataset. For example, NASWOT [Mellor et al., 2021] evaluates and ranks network architectures’ expressivity at zero training cost by applying the Walsh–Hadamard transform to their untrained activations on random inputs and measuring the resulting correlations, ZiCo [Li et al., 2023] performs zero-shot NAS by measuring the inverse coefficient of variation of gradient norms from untrained networks on random inputs and using more stable gradients as a proxy to rank and select superior architectures, MeCo [Jiang et al., 2023] uses the minimum eigenvalue of the Pearson correlation matrix of feature maps computed from a single random input’s forward pass through the untrained network as a zero-cost proxy to evaluate and rank architectures, and SWAP-NAS [Peng et al., 2024] ranks architectures by using the consistency of untrained networks’ sample-wise activation patterns on input batches as a zero-cost performance proxy. We design a framework incorporating zero-shot indicators as guidance to realize efficient, generalized, and high-performance NAS. Graph Diffusion Model Graph diffusion models apply a noise-to-denoise process on graph structures and have driven advances in graph generation (e.g., GraphDDPM [Liu et al., 2023], GDSS [Jo et al., 2022]) and GNN message passing (e.g., GDC [Gasteiger et al., 2019], ADC [Zhao et al., 2021]). Recently, some works have begun to explore graph diffusion for NAS (e.g., DiffusionNAG [An et al., 2024], Multi-conditioned Graph Diffusion for Neural Architecture Search [Asthana et al., 2024]).

3

Methodology

We illustrate the DIFFNAS framework in Figure 1. The zero-shot evaluator scores each architecture and passes the score as a conditioning signal to the graph denoiser. After training, the model can optionally be fine-tuned using a reward-guided diffusion objective. During the search phase, candidate architectures are sampled from the latent space conditioned on high scores. These candidates are re-evaluated using the zero-shot metric, ranked, and the top-performing architectures are selected. Unlike other conditional diffusion models for graph generation, which need intensive neural network training, we present a method that uses a training-free metric as reward to guide the reverse process instead of training any predictors. Neural Architecture Search as Graph Generation To formally describe the problem, let us assume that a classical neural network is described as a directed acyclic graph (DAG) G = (V, E), where V stands for nodes corresponding to operation candidates and E stands for edges corresponding to connections of operations. We present the operation matrix as XV ∈ RN ×FV and the adjacent matrix as XE ∈ RN ×N ×FE , where N = |V |, FV and FE stand for the number of nodes, operation types and connection types respectively. 3.1

Diffusion Process

Unlike approaches that perturb graphs within a stochastic differential equation (SDE) framework [Jo et al., 2022], we adopt a discrete diffusion model following DiGress [Vignac et al., 2023]. The t diffusion process is defined as a forward noising procedure on a graph Gt = (XVt , XE ), where t n×d t n×n XV ∈ X represents node features and XE ∈ E represents the edge attributes at timestep t. Noise is injected through two transition matrices QV ∈ RFV ×FV and QE ∈ RFE ×FE , which stochastically replace node and edge tokens with others from the respective marginal distributions. Formally, at each timestep, the graph is updated via:
t−1 t q(Gt | Gt−1 ) = XVt−1 QtV , XE QE , and the marginal distribution over the noisy graph at timestep t is given by:
0 q(Gt | G0 ) = XV0 Q̄tV , XE Q̄tE , where Q̄t = Q1 Q2 . . . Qt is the cumulative transition operator. 3

(1)

(2)

The transition matrices Qt are chosen to progressively interpolate between the identity matrix and the marginal distribution. Specifically, we set: Qt = αt I + (1 − αt )1m⊤ ,

(3)

where αt ∈ [0, 1] controls the noise level, 1 is an all-ones column vector, and m is the marginal distribution over node or edge types. The resulting cumulative transition matrix becomes: Q̄t = ᾱt I + (1 − ᾱt )1m⊤ ,

where ᾱt =

t Y

αi .

(4)

i=1

We employ a cosine schedule [Nichol and Dhariwal, 2021] to control the decay of ᾱt , defined as:   π(t/T + s) t 2 ᾱ = cos , (5) 2(1 + s) with s as a small offset constant, following standard practice in diffusion models. Each graph is represented with a unified token matrix XG that encodes both nodes and edges, and a corresponding transition matrix QG integrating QV and QE for noise injection in a shared embedding space is applied. 3.2

Reverse Process

The reverse process aims to recover the original graph structure G0 from a fully corrupted version GT by learning a parameterized denoising model that approximates the backward transitions pθ (Gt−1 | Gt ). Following DiGress [Vignac et al., 2023], we model this process as a Markov chain where the model iteratively predicts less noisy versions of the graph by denoising node and edge tokens. At each timestep t, we estimate the posterior distribution over the previous state Gt−1 = t−1 (XVt−1 , XE ) given the current noisy graph Gt , using a learnable model fθ that takes Gt and t as inputs. This model outputs a categorical distribution over node and edge types for each token position:
t−1 pθ (Gt−1 | Gt ) = pθ (XVt−1 | Gt , t), pθ (XE | Gt , t) . (6) We implement fθ using a transformer-based architecture that encodes the entire graph structure, including both nodes and edges, and uses timestep embeddings to incorporate temporal information. The reverse step is viewed as a masked token prediction task, where the model learns to denoise masked or corrupted tokens based on the current graph context. Training is performed by minimizing the variational bound on the negative log-likelihood of the data, which reduces to a standard cross-entropy loss over the predicted node and edge types:   Ldenoise = Et,G0 ,Gt − log pθ (G0 | Gt , t) . (7) 3.3

Denoiser

t Our denoising model fθ operates on a unified token sequence XG , which jointly represents node and t−1 edge tokens at timestep t, and aims to predict the clean graph components (XVt−1 , XE ) at each reverse diffusion step. t Each token in XG is enriched with three types of embeddings: (i) a learned timestep embedding Embed(t), (ii) structural embeddings that encode graph topology, and (iii) a conditioning signal derived from the evaluator M E C O, namely a binary quality label c(G0 ) obtained by thresholding the MeCo score r(G0 ) of the ground-truth graph G0 .

In the first training stage, the MeCo score r(G0 ) of each training graph is converted into the binary label c(G0 ): the score is multiplied by 1000 and compared with a threshold of 60,000 (see Experimental Setup). Only architectures above the threshold are kept as training data, and the label is used as an additional conditioning vector in the transformer. Specifically, the label is projected through a linear layer and added to the token representations before passing them to the transformer layers. This provides guidance to the model, encouraging it to reconstruct graphs with high MeCo scores. 4

To effectively integrate conditioning information, we adopt an adaptive normalization mechanism inspired by AdaLN-Zero [Ede et al., 2022] and FiLM [Perez et al., 2018]. Unlike standard normalization layers (e.g., LayerNorm), our adaptive normalization layer dynamically adjusts the model’s activations based on the conditioning signal. Specifically, given a residual stream h ∈ Rd and a conditioning input c (e.g., timestep or reward), we apply: h − µ(h) + β(c), (8) AdaLN(h | c) = γ(c) · σ(h) where µ(h), σ(h) are the feature-wise mean and standard deviation, and γ(c), β(c) ∈ Rd are scale and shift vectors predicted from c. This allows the model to modulate its behavior in a context-aware manner at every transformer block. Our transformer backbone consists of alternating self-attention and feed-forward layers. The final output is decoded through two MLP heads for node and edge prediction: t X̂Vt−1 = MLPV (Transformer(XG , c(G0 ))),

t−1 t X̂E = MLPE (Transformer(XG , c(G0 ))).

(9)

The model is trained using a cross-entropy loss between the predicted and clean node and edge tokens from the original graph: h i t−1 0 Ldenoise = Et,G0 ,Gt CE(X̂Vt−1 , XV0 ) + CE(X̂E , XE ) . (10) By adaptive normalization, our denoiser is equipped to reconstruct high-quality graphs that are both structurally valid and aligned with downstream objectives. 3.4

Zero-Shot Indicators

To guide generation toward high-quality structures, we use task-specific evaluators such as MeCo [Jiang et al., 2023] and SWAP [Peng et al., 2024], which provide scalar rewards reflecting the quality of each generated architecture. In the first stage, the scores are thresholded to obtain binary conditioning labels. In the second stage, the continuous scores are used as rewards for DDPO fine-tuning. Unlike learned property predictors or reward models that require training and backpropagation, MeCo and SWAP are training-free: they compute architectural quality based on simple one-shot structural signals using a single forward pass with no gradients. This enables fast, scalable, and modular evaluation. MeCo [Jiang et al., 2023] measures the expressivity of a neural architecture by spectral analysis of its internal feature correlations. For each candidate architecture G, we instantiate a corresponding untrained network fG and forward a single random image x ∼ D through it to obtain the hidden representations at each layer. In layer i, let fi (x; θ) ∈ Rn×d denote the feature map. The MeCo score is then defined as the sum of the minimum eigenvalues of the Pearson correlation matrices: MeCo(G) =

D X

λmin (P (fi (x; θ))) ,

(11)

i=1

where P (·) computes the Pearson correlation matrix across feature dimensions, and λmin extracts the minimum eigenvalue. A higher score indicates more diverse and expressive internal activations. Furthermore, we introduce another zero-shot indicator SWAP in Appendix A. 3.5

Reward-Guided Fine-Tuning with DDPO

While the first-stage diffusion model accurately reconstructs graphs and can be lightly steered by zero-cost scores, its training objective is still maximum likelihood. To align the generator directly with the downstream objective, we adopt Denoising Diffusion Policy Optimization (DDPO) [Black et al., 2024] for a second, reward-guided fine-tuning stage. This second-stage optimization provides an opportunity to align with the task-specific objective by directly optimizing the generation policy for reward, rather than relying solely on supervised reconstruction. We observe that DDPO fine-tuning leads to improvements in search performance across downstream tasks. For more details on DDPO, please refer to Appendix B. 5

Table 1: Comparison of NAS methods evaluated on the NAS-Bench-201 search space across CIFAR10, CIFAR-100, Aircraft, and Oxford-IIIT Pets datasets. Methods are grouped by design: Non-weight sharing, Zero-shot, Pre-trained predictor-based NAS, and Diffusion model-based NAS. “A” indicates multiple architecture evaluations during search; “B” indicates use of a pre-trained predictor. “–” denotes unreported results. Method

Search

CIFAR-10 Acc.

CIFAR-100

Aircraft

Oxford IIIT Pets

Acc.

Acc.

Acc.

-

-

NAS-Bench-201 Non-weight sharing 93.92 71.84 93.70 71.04 93.85 71.71

REA [Real et al., 2019] RS [Bergstra and Bengio, 2012] REINFORCE [Williams, 1992]

-

SynFlow [Tanaka et al., 2020] NASWOT [Mellor et al., 2021] ZiCo [Li et al., 2023] CVU [Gracheva, 2021] AZ-NAS [Lee and Ham, 2024] TE-NAS [Chen et al., 2021] TG-NAS [Qiao et al., 2024] Zen-NAS [Lin et al., 2021]

A A A A A A A A

71.10 69.98 71.10 64.08 70.75 67.87 70.64 68.10

-

-

PRE-NAS [Peng et al., 2022] CAP [Ji et al., 2024] RANK-NOSH [Wang et al., 2021]

Pre-trained Predictor Based NAS A+B 94.04 72.02 A+B 94.34 73.41 A+B 94.26 73.51

-

-

DiffusionNAG [An et al., 2024]

Diffusion Model Based NAS A+B 94.37 73.51

58.83

41.80

DIFFNAS (ours)

A

58.85

41.50

Zero Shot 90.40 92.96 94.00 91.90 93.53 92.30 93.75 90.60

94.37

73.51

* A and B under Search respectively denote the need for multiple evaluations of architectures and the use of a pre-trained predictor. Results of other methods are taken from the original publications, except for the non-weight-sharing methods [Lee et al., 2021], SynFlow and Zen-NAS (as reported by Li et al. [2023]), and TE-NAS (as re-evaluated by Lee and Ham [2024]).

4

Experiment

The experiments are designed to evaluate the effectiveness of the proposed method for NAS and to compare its performance with existing approaches. 4.1

Experimental Setup

In our experimental evaluation, we detail the search spaces, baseline comparisons, key hyperparameters, and computational resources used to rigorously assess the performance of our proposed framework. Search Space To validate the efficacy of our framework, we utilize the NAS-Bench-201 (NB201) benchmark search space [Dong and Yang, 2020], which comprises 15,625 unique architectures. Following prior work [Lee et al., 2021, An et al., 2024], we report the test accuracy recorded in the benchmark. We also utilize MobileNetV3 search space, which can generate a total of about 1019 possible sub-architectures by combining hyperparameters such as convolutional kernel sizes (3, 5, 7), expansion ratios (3, 4, 6), and network depths (2, 3, 4). Baseline We compare our method against several generative NAS approaches: DiffusionNAG [An et al., 2024]: A diffusion-based NAS method guided by a pre-trained predictor that steers the model toward generating high-performance architectures. MetaD2A [Lee et al., 2021] A non-conditional generative model that does not rely on predictors during the architecture generation phase. However, it requires training the generated architectures to obtain their performance metrics, which incurs 6

significant computational overhead. During testing, MetaD2A first generates architectures in an objective and unconditional manner, and subsequently evaluates the top-performing architectures using its predictor. Hyperparameters Our approach employs a two-stage training paradigm. In the first stage, a transformer model is trained to learn the characteristics of high-quality architectures. The second stage leverages reward-guided reinforcement learning to steer the framework toward generating architectures with superior performance. The first phase consists of 5000 epochs, while the second phase involves 200 epochs. During the second stage, a reward of -50,000 is applied if the model generates invalid architectures. The framework generates 1000 samples on four datasets: CIFAR10 [Krizhevsky, 2009], CIFAR-100 [Krizhevsky, 2009], Aircraft [Maji et al., 2013], and Oxford-IIIT Pets [Parkhi et al., 2012]. We incorporate the zero-shot NAS method, MeCo [Jiang et al., 2023], to assign binary labels to neural network structures. To preserve MeCo’s accuracy during scaling, we adjust its grading metric by multiplying it by a factor of 1000. A threshold of 60,000 is defined to distinguish between good and poor neural network architectures. Computational Resources Experiments were run on the Hawk system (nodes with NVIDIA A100 40 GB GPUs and AMD EPYC 7742 CPUs) and the Hunter system (AMD Instinct MI300A) at HLRS. Architecture search used a single GPU. Each training (5000 epochs) took 3 GPU-hours on a single GPU with 40 GB VRAM. DDPO fine-tuning (200 epochs) took 10 GPU-hours per run. 4.2

Results on NB201 Search Space

Table 1 presents the performance of various NAS algorithms on NAS-Bench-201. On CIFAR-10, DIFFNAS achieves 94.37%, matching the best-performing method, DiffusionNAG (94.37%), while avoiding the need for a pretrained predictor. Among zero-shot baselines, ZiCo reaches 94.00%, NASWOT achieves 92.96%, and SynFlow obtains 90.40%, highlighting a consistent performance gap. On CIFAR-100, DIFFNAS attains 73.51%, outperforming all zero-shot methods (best: ZiCo and SynFlow, 71.10%) and matching the best pre-trained predictor-based method, RANK-NOSH (73.51%; CAP reaches 73.41%). DiffusionNAG reports the same accuracy (73.51%), but, like RANKNOSH and CAP, it relies on a learned predictor. In contrast, DIFFNAS combines diffusion-based architecture generation with the MeCo zero-cost proxy, offering competitive performance without the complexity of surrogate modeling. Table 2 reports the performance, efficiency, and cost of various NAS methods on NAS-Bench201 across four datasets. Metrics include model size, search time (in GPU hours), monetary cost (USD), and test accuracy. DIFFNAS achieves competitive accuracy across the four datasets while reducing search cost by up to 17.5x compared to baseline. On CIFAR-10 and CIFAR-100, it matches DiffusionNAG’s performance (94.37% and 73.51%, respectively) while requiring 17.5× less search time (0.004 vs. 0.07 GPU hours), and without the need for a pretrained predictor. Compared to PC-DARTS, which incurs 722.5× and 1385× longer search times on the two datasets, DIFFNAS achieves higher accuracy at a fraction of the computational and monetary cost. On Aircraft, DIFFNAS achieves 58.85%, marginally outperforming DiffusionNAG (58.83%) while using 17.5× less compute. It also surpasses RSPS and SETN by more than 15 percentage points, despite their 1000× higher compute cost. On Oxford-IIIT-Pet, DIFFNAS reaches 41.50%, closely matching DiffusionNAG (41.80%) with the same 17.5× cost reduction and no predictor overhead. These results show that DIFFNAS keeps strong accuracy while greatly reducing search cost, which makes it practical for NAS under limited compute budgets. In Table 3, we can find the best architectures for both CIFAR10 and CIFAR100 datasets, corresponding to the accuracy of 94.37% and 73.51%. On CIFAR-10, our mean and min values are the highest among the generative methods. On CIFAR-100, our min value is the highest, but our mean is slightly lower than that of DiffusionNAG. Notably, the structure corresponding to our min value is significantly higher than the structure corresponding to the other methods, including DiffusionNAG. This indicates that the architectures generated by our method are more consistently good. 4.3

Results on MobileNetV3 Search Space

Table 5 in Appendix C shows a comparison between DiffusionNAG and our method, DIFFNAS, on CIFAR-10 using the MobileNetV3 search space. While DIFFNAS achieves a lower accuracy (89.50%) 7

Table 2: Comparison of NAS methods on NAS-Bench-201 across CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets. Reported metrics include model size (in millions of parameters), search time (GPU hours), monetary cost (in USD), and accuracy (%). “–” denotes missing or unreported values. NAS Method

Params (M)

Search Time (GPU h)

Speed Up

Search Cost ($)

Accuracy (%)

NASWOT TE-NAS TG-NAS DiffusionNAG PC-DARTS DIFFNAS (Ours)

0.615 1.317 0.802 1.17 –

0.09 0.43 0.01 0.07 2.89 0.004

22.5× 107.5× 2.5× 17.5× 722.5× 1×

0.22 1.07 0.02 0.18 7.22 0.009

92.96 93.78 93.75 94.37 93.66 94.37

NASWOT TE-NAS TG-NAS DiffusionNAG PC-DARTS DIFFNAS (Ours)

0.615 1.317 0.802 – 0.26 –

0.09 0.43 0.01 0.07 5.54 0.004

22.5× 107.5× 2.5× 17.5× 1385× 1×

0.22 1.07 0.02 0.18 13.85 0.009

69.70 70.44 70.64 73.51 66.64 73.51

Aircraft

RSPS SETN DiffusionNAG PC-DARTS DIFFNAS (Ours)

0.22 0.44 – 0.32 –

5.19 5.16 0.07 0.98 0.004

1297.5× 1290× 17.5× 245× 1×

12.98 12.89 0.18 2.45 0.009

42.19 44.84 58.83 26.33 58.85

Oxford-IIIT Pets

RSPS SETN DiffusionNAG PC-DARTS DIFFNAS (Ours)

0.32 0.32 – 0.44 –

0.93 2.40 0.07 0.79 0.004

232.5× 600× 17.5× 197.5× 1×

2.33 5.99 0.18 1.97 0.009

22.91 25.17 41.80 25.31 41.50

Target Dataset

CIFAR-10

CIFAR-100

* RSPS [Li and Talwalkar, 2020], SETN [Dong and Yang, 2019], and PC-DARTS [Xu et al., 2019] are NAS methods included for comparison. Values for NASWOT, TE-NAS and TG-NAS are taken from Qiao et al. [2024]; values for RSPS, SETN and PC-DARTS are taken from Lee et al. [2021] and An et al. [2024], with search times converted from GPU seconds to GPU hours. Accuracies can therefore differ from Table 1, which uses other sources. Search cost assumes $2.50 per GPU hour; speed-up is relative to DIFFNAS.

Table 3: Performance comparison on CIFAR10 and CIFAR100 datasets in NB201 search space. 1000 architectures are generated for computing statistical metrics for each method. Target Methods Stats. Dataset Oracle Random MetaD2A Diff. Ours CIFAR10

Max Mean Min

94.37 93.50 93.18

94.37 87.12 10.00

94.37 91.52 10.00

94.37 93.13 86.44

94.37 93.41 92.08

CIFAR100

Max Mean Min

73.51 70.62 69.91

72.74 61.59 1.00

73.51 67.14 1.00

73.51 70.34 58.09

73.51 69.91 64.39

than DiffusionNAG (97.39%), we attribute this discrepancy primarily to a mismatch between the MeCo zero-shot evaluator and the final training objective in the MobileNetV3 space. MeCo evaluates untrained architectures based on spectral analysis of internal feature correlations, which performs well in broader search spaces like NAS-Bench-201. However, in the MobileNetV3 space, where architectures are highly refined and performance differences stem from subtle interactions between operations and training dynamics, such zero-cost proxies may not effectively capture the fine-grained distinctions that impact downstream accuracy. This misalignment between the proxy signal and true task performance may have limited the effectiveness of MeCo-guided architecture generation in this case. 8

Table 4: Summary of ablation studies. 1000 architectures are generated for computing statistical metrics for each method. Target Methods Stats. Dataset Oracle w/o w/o Full Top-1,000 MeCo scaling SWAP Frame CIFAR10

Max Mean Min

94.37 93.50 93.18

94.36 87.70 10.00

94.36 92.33 86.64

94.37 86.57 10.00

94.37 93.41 92.08

In addition, the MobileNetV3 search space is known to be sensitive to small architectural changes, making it inherently more difficult for automated methods to discover top-performing designs. Furthermore, due to the absence of publicly available training configurations for DiffusionNAG, we evaluated the architectures discovered by DIFFNAS using self-defined hyperparameters, which may have resulted in suboptimal training and contributed to the performance gap. While DIFFNAS underperforms on the MobileNetV3 search space, its strong results on other benchmarks suggest that this gap may stem from the unique challenges of MobileNetV3 rather than general weaknesses of the approach. 4.4

Ablation Studies

In order to verify the contribution of each module in the framework to the final performance, we designed a series of ablation experiments to remove or replace key components and record their impact on model performance. All experiments are performed in the search space NAS-Bench-201 on the same hardware configuration (NVIDIA A100 GPU) and dataset CIFAR-10. We conducted comparative experiments by removing or replacing the following components, using the Full Framework as a baseline: • w/o MeCo: Removes the MECO-based reward evaluator; the architecture generation proceeds without binary labeling. • MeCo replaced by SWAP: Substitutes MECO with SWAP, a different predictor-free NAS metric, to assess the impact of MECO-guided labeling. • w/o Reward-Guided Phase: Removes Stage 2 training (DDPO) and trains only with token-level supervision. • w/o Diffusion Transformer: Removes the diffusion-based transformer and retains only baseline components to assess its contribution. Table 4 presents the results of ablation studies assessing the contribution of key components in our framework. The Full Framework consistently achieves strong performance, with a maximum accuracy of 94.37%, a mean accuracy of 93.41%, and a minimum accuracy of 92.08%, closely approaching the Oracle Top-1,000 benchmark, which serves as an empirical upper bound. Removing the MeCo-based binary labeling strategy leads to a notable drop in performance, with mean accuracy decreasing to 87.70% and minimum accuracy falling to 10.00%. This highlights MeCo’s importance in stabilizing and guiding architecture generation. Replacing MeCo with SWAP results in even lower performance, with a mean of 86.57% and a minimum of 10.00%, indicating that MeCo provides more effective and reliable zero-shot guidance. Eliminating the reward-guided scaling phase also degrades performance, reducing mean accuracy to 92.33% and minimum accuracy to 86.64%. Notably, the maximum accuracy remains high across all ablation settings, suggesting that while strong architectures can still be discovered, the quality of the overall distribution deteriorates when key components are removed. Finally, when the diffusion transformer is removed entirely, the model fails to generate any valid architectures, highlighting its essential role in structured generation. These results show that the MeCo evaluator, the reward-guided scaling phase and the diffusion transformer are all needed for consistent, high-quality architecture generation. 9

5

Conclusion

We introduce DIFFNAS, a predictor-free Neural Architecture Search framework that integrates discrete graph diffusion with zero-cost performance proxies. By conditioning the generative process on training-free metrics such as MeCo and SWAP, DIFFNAS enables efficient and scalable architecture generation without the need for learned performance predictors. Experiments on NAS-Bench-201 show that DIFFNAS finds the optimal architectures for CIFAR-10 and CIFAR-100 while substantially reducing search cost compared to existing NAS methods. While performance on MobileNetV3 was lower than prior work, we attribute this to a misalignment between zero-shot proxy signals and true task performance in this more complex search space. Ablation studies show that conditioning, the diffusion-based transformer and reward-guided fine-tuning each contribute to robustness. DIFFNAS thus provides a basis for future work on predictor-free, diffusion-based architecture generation.

Acknowledgements The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre. eu) for funding this project by providing computing time on the GCS supercomputers Hawk and Hunter at the High-Performance Computing Center Stuttgart (HLRS).

References Sohyun An, Hayeon Lee, Jaehyeong Jo, Seanie Lee, and Sung Ju Hwang. DiffusionNAG: Predictorguided neural architecture generation with diffusion models. In International Conference on Learning Representations, 2024. Rohan Asthana, Joschua Conrad, Youssef Dawoud, Maurits Ortmanns, and Vasileios Belagiannis. Multi-conditioned graph diffusion for neural architecture search. arXiv preprint arXiv:2403.06020, 2024. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The journal of machine learning research, 13(1):281–305, 2012. Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations (ICLR), 2024. Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020. Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535, 2021. Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3681–3690, 2019. Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326, 2020. Jason Ede, Emiel Hoogeboom, and Efstratios Gavves. Adaptive normalization for conditional diffusion models. arXiv preprint arXiv:2212.09748, 2022. URL https://arxiv.org/abs/ 2212.09748. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019. Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. Advances in neural information processing systems, 32, 2019. Ekaterina Gracheva. Trainless model performance estimation based on random weights initialisations for neural architecture search. Array, 12:100082, 2021. 10

Yesmina Jaafra, Jean Luc Laurent, Aline Deruyver, and Mohamed Saber Naceur. Reinforcement learning for neural architecture search: A review. Image and Vision Computing, 89:57–66, 2019. Han Ji, Yuqi Feng, and Yanan Sun. Cap: A context-aware neural predictor for nas. arXiv preprint arXiv:2406.02056, 2024. Tangyu Jiang, Haodi Wang, and Rongfang Bie. Meco: zero-shot nas with one data and single forward pass via minimum eigenvalue of correlation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 61020–61047, 2023. Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International conference on machine learning, pages 10362–10383. PMLR, 2022. Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. Advances in neural information processing systems, 31, 2018. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf. Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. arXiv preprint arXiv:2107.00860, 2021. Junghyup Lee and Bumsub Ham. Az-nas: Assembling zero-cost proxies for network architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5893–5903, 2024. Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. In International Conference on Learning Representations (ICLR), 2023. Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pages 367–377. PMLR, 2020. Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 347–356, 2021. Chengyi Liu, Wenqi Fan, Yunqing Liu, Jiatong Li, Hang Li, Hui Liu, Jiliang Tang, and Qing Li. Generative diffusion models on graphs: Methods and applications. arXiv preprint arXiv:2302.02591, 2023. Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018. Zhichao Lu, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 35–51. Springer, 2020. Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. Advances in neural information processing systems, 31, 2018. Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International conference on machine learning, pages 7588–7598. PMLR, 2021. Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR, 2021. 11

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. Yameng Peng, Andy Song, Vic Ciesielski, Haytham M Fayek, and Xiaojun Chang. Pre-nas: predictorassisted evolutionary neural architecture search. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1066–1074, 2022. Yameng Peng, Andy Song, Haytham M Fayek, Vic Ciesielski, and Xiaojun Chang. Swap-nas: Sample-wise activation patterns for ultra-fast nas. In ICLR, 2024. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Ye Qiao, Haocheng Xu, and Sitao Huang. Tg-nas: Leveraging zero-cost proxies with transformer and graph convolution networks for efficient neural architecture search. arXiv preprint arXiv:2404.00271, 2024. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780–4789, 2019. Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377–6389, 2020. Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2209.14734. Ruochen Wang, Xiangning Chen, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Rank-nosh: Efficient predictor-based architecture search via non-uniform successive halving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10377–10386, 2021. Chen Wei, Chuang Niu, Yiping Tang, Yue Wang, Haihong Hu, and Jimin Liang. Npenas: Neural predictor guided evolution for neural architecture search. IEEE transactions on neural networks and learning systems, 34(11):8441–8455, 2022. Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European conference on computer vision, pages 660–676. Springer, 2020. Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10293–10301, 2021. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. Meng-Ting Wu and Chun-Wei Tsai. Training-free neural architecture search: A review. ICT Express, 10(1):213–231, 2024. Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737, 2019. Jialin Zhao, Yuxiao Dong, Ming Ding, Evgeny Kharlamov, and Jie Tang. Adaptive diffusion in graph neural networks. Advances in neural information processing systems, 34:23321–23333, 2021. Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.

12

A

Zero-Shot Indicator SWAP

SWAP Peng et al. [2024] quantifies the expressivity of an architecture by measuring the diversity of its activation patterns across different input samples. For a given network fθ , we forward a batch (v) of S inputs {x1 , . . . , xS } and record the post-activation outputs ps at each intermediate unit v for each sample s. These activations are binarized using an indicator function I[·], and the SWAP score is computed as:  h i S V ΨN,θ = ÂN,θ , where ÂN,θ = p(v) : p(v) = I p(v) . (12) s s=1

v=1

S

Here, p(v) ∈ {0, 1} is the binary activation pattern of unit v across the batch, and ÂN,θ counts the number of unique such patterns. A higher SWAP score indicates that the architecture exhibits more varied responses to different inputs, reflecting greater discriminative capacity. These training-free evaluators are efficient, require no learned components, and reflect important aspects of architectural capacity, making them ideal for use in generative and search-based pipelines.

B

DDPO

Following Black et al. [2024], each reverse denoising step is viewed as an action in a T -step Markov decision process (MDP). Given a trained diffusion model pθ , we generate candidate samples {Gi } and score them using the reward function r(G) based on zero-shot indicators. We use the continuous MeCo score as the reward. When generated samples are invalid, they are assigned a negative penalty reward. Rewards are normalized across each batch of samples before the policy-gradient update. DDPO then fine-tunes the model to maximize the expected reward:   J(θ) = Eτ ∼πθ r(G) , (13) which we optimise with the score-function (REINFORCE) gradient: " T # X
∇θ J(θ) = Eτ ∼πθ ∇θ log pθ (Gt−1 | Gt , c) r(G0 ) ,

(14)

t=1

C

MobileNetV3 results

Table 5: Comparison of neural architecture search methods evaluated on the MobileNetV3 search space on CIFAR-10. Algorithm Acc. (CIFAR-10) DiffusionNAG DIFFNAS (Ours)

D

97.39 89.50

External Asset Licenses

To ensure proper credit and compliance, we document below all external assets used in this work, including their versions, licenses, and source URLs. Table 6: External assets, versions, licenses, and source URLs. Asset

Version

License

Source URL

NAS-Bench-201 MobileNetV3 MeCo SWAP PyTorch Lightning

v2.0 0.11 0d830dd2 0853fc83 2.5.1

MIT Apache-2.0 MIT AFL-3.0 license Apache-2.0

https://github.com/D-X-Y/NAS-Bench-201 https://github.com/mit-han-lab/once-for-all https://github.com/HamsterMimi/MeCo https://github.com/pym1024/SWAP https://github.com/Lightning-AI/pytorch-lightning

13

Related documents

• Official Review · ICLR 2025#787197
• Coverage You Can Steer: Online Conformal Calibration for RL-Driven Hardware-Aware NAS#1187805
• Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models#147156
• ZAPS: Zero-Cost Active Proxy Search for Neural Architecture Search#1083646
• Promise and Limits of Cross-Architecture Knowledge Distillation for Lightweight Medical Vision Transformers (LiteMed-ViT)#1456343