arXiv CSopen access

Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs

Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs Santosh Premi Adhikari,

Radu Timofte, Dmitry Ignatov

arXiv:2605.04903v1 [cs.LG] 6 May 2026

Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

Abstract Large language models (LLMs) show strong potential for neural architecture generation, yet existing approaches produce complete model implementations from scratch— a paradigm that is computationally expensive and yields verbose, redundant code. We propose Delta-Code Generation, a paradigm where fine-tuned LLMs generate compact unified diffs (deltas) to refine existing baseline architectures rather than synthesizing entire models. Our pipeline iteratively fine-tunes the LLM via LoRA on curated architectures from the LEMUR dataset, with MinHashJaccard novelty filtering to maintain structural diversity. We evaluate three 7B-class code/instruct LLMs—DeepSeekCoder-7B-Instruct-v1.5, Qwen2.5-Coder-7B-Instruct, and Mistral-7B-Instruct-v0.3—across six datasets (CIFAR-10, CIFAR-100, MNIST, SVHN, ImageNette, and CelebA) using the same 22-cycle protocol (1,100 candidates per LLM). Under a shared cluster protocol with a balanced six-dataset baseline pool, all three LLMs yield closely matched headline metrics: DeepSeek-Coder reaches 75.3% valid generation rate, 65.8% mean first-epoch accuracy, and 99.5% best first-epoch accuracy; Qwen2.5-Coder reaches 72.1% / 64.6% / 99.5%; and Mistral-7B 66.6% / 66.1% / 99.5%—all accuracies being first-epoch validation accuracy, the proxy used in prior LLM-based NAS work [13, 24]. All three substantially surpass the full-generation baseline (50.6% valid rate, 42.3% mean first-epoch accuracy, 64.0% best firstepoch accuracy on CIFAR-10 only). On CIFAR-10 specifically, Mistral, DeepSeek, and Qwen reach 85.5%, 85.2%, and 80.6% best first-epoch accuracy respectively—all well above 63.98% full generation and 71.5% for the concurrent iterative approach of Gu et al. Output lengths are uniformly compact—30.4 lines (DeepSeek), 31.4 lines (Qwen), and 49.5 lines (Mistral)—a 75–85% reduction over full generation across all three LLMs. A 50-epoch full-training study on the top-20 architectures per LLM confirms that the 1epoch proxy reliably preserves performance rankings (Mistral: Spearman ρ = 0.926), placing it at the upper end of NAS proxy correlations reported in the literature. Our results position delta-based generation as a token-efficient,

multi-domain, and LLM-agnostic alternative to full-model synthesis for LLM-driven neural architecture search. All admitted architectures, per-model statistics, and fine-tuning configurations are released under the del- prefix in the LEMUR dataset for reviewer verification and full reproducibility.

1. Introduction Neural Architecture Search (NAS) has emerged as a powerful paradigm for automating the design of deep neural networks [10, 49, 55]. Early NAS methods based on reinforcement learning required up to 22,400 GPU-days [55], while even efficient differentiable approaches like DARTS [30] depend on heavy supernet training. Standardized benchmarks such as NAS-Bench-201 [9] have enabled reproducible comparisons, yet the search space remains constrained to predefined cell structures. The recent advent of code-capable LLMs [6, 14, 18, 41, 45] has opened an alternative avenue: using language models as architecture generators that produce executable PyTorch code directly [3, 24, 35]. Within this LLM-based NAS paradigm, Khalid et al. [24] demonstrated that iteratively fine-tuning an LLM on its own successful generations can progressively improve both generation reliability and model quality. Their framework generates complete PyTorch implementations, achieving a 50.6% valid generation rate (Wilson 95% CI [45.0%, 56.1%]) and a best first-epoch accuracy of 63.98% with mean 50.99% at cycle 18 on CIFAR-10, after 22 fine-tuning cycles. The updated version of their work extends the evaluation to three datasets (CIFAR-10, CIFAR-100, SVHN) using dataset-specific accuracy thresholds (40%, 20%, 70% respectively), and we treat it as the current state of the art for LLM-based fixed-semantics generation on the LEMUR benchmark. Concurrently, Gu et al. [13] (a concurrent arXiv preprint) proposed a closed-loop iterative pipeline using frozen LLMs with historical feedback memory, achieving up to 71.5% one-epoch accuracy on CIFAR-10 through 2000 iterations without any LLM fine-tuning. However, both approaches share a critical limitation:

the LLM must generate complete model implementations at each step. This full-generation paradigm produces verbose outputs (typically 200+ lines), requires regenerating working code structures, and cannot leverage the structural knowledge embedded in existing architectures. We propose Delta-Code Generation, an approach where the LLM generates compact unified diffs (deltas) to modify existing baseline architectures rather than synthesizing complete implementations. Drawing inspiration from the software engineering practice of patch-based code modification [22, 51], our paradigm constrains the generation space to targeted architectural refinements. This constrained generation space offers several distinct advantages. Primarily, it drastically reduces token consumption; delta outputs average 30 lines for DeepSeek, 31 lines for Qwen, and 49.5 lines for Mistral—versus 200+ lines for full-model synthesis (75–85% reduction across all three LLMs). Furthermore, the approach inherently preserves the structural integrity of working baselines while targeting specific architectural improvements. It also scales naturally across diverse domains, allowing us to evaluate generalization on six distinct datasets simultaneously rather than being limited to CIFAR-10. Finally, this framework provides a rigorous testbed for comparing modern code-oriented LLMs, such as DeepSeek-Coder and Qwen2.5-Coder, in the context of structured code generation. In summary, our main contributions are as follows. We introduce Delta-Code Generation, a novel NAS paradigm that synthesizes unified diffs instead of complete models, cutting output length by 75–85% across three distinct 7B LLMs while preserving competitive first-epoch proxy accuracy (evaluated under the same one-epoch protocol as prior LLM-based NAS work [13, 24]). We also provide a comprehensive head-to-head evaluation of three 7B-class LLMs—DeepSeek-Coder-7B, Qwen2.5-Coder7B, and Mistral-7B-Instruct—across two distinct codetraining lineages (code-specialized and general-instruct), highlighting the impact of recent LLM advancements on structured generation tasks and demonstrating that delta generation transfers across LLM families. Lastly, we extend the evaluation protocol beyond single-dataset benchmarks to encompass six diverse image classification datasets, demonstrating the robust multi-domain capabilities of our approach.

approaches like DARTS [30] reduced search cost by orders of magnitude, while hand-designed scaling rules [44] and residual connections [15] have proven highly effective within fixed topologies. Standardized benchmarks such as NAS-Bench-201 [9] enabled reproducible comparisons across methods. More recently, zero-shot NAS methods [29, 33] bypass training entirely using proxy metrics, yet remain constrained to predefined search spaces.

2.2. LLM-Based Architecture Generation LLMs have been increasingly applied to architecture design. Zheng et al. [53] explored GPT-4 for NAS, while AutoML-GPT [52] proposed end-to-end AutoML with LLMs. LLMatic [35] combines LLM-driven mutation with quality-diversity search. EvoPrompting [4] applies evolutionary prompting with LLMs as mutation operators for code-level NAS. SEKI [3] uses self-evolution and knowledge inspiration for LLM-guided NAS, achieving competitive performance in 0.05 GPU-days. RZ-NAS [20] integrates reflective zero-cost strategies. LEMONADE [39] uses ChatGPT for multi-objective architecture discovery. Within the NNGPT framework [26], Khalid et al. [24] established a 22-cycle iterative fine-tuning paradigm, and Gu et al. [13] proposed iterative NAS with historical feedback memory using frozen LLMs. Extensions to the NNGPT framework include few-shot prompting strategies for improved generation reliability [48], fractal-inspired computational architectures [34], discovery of non-standard channel priors [46], and retrieval-augmented approaches to extract algorithmic logic from neural networks [23]. Separately, HPGPT [25] demonstrated that LLMs can predict training hyperparameters, treating HP optimization as a distinct task from architecture generation. In parallel, the software engineering community has developed patch-based code generation benchmarks such as SWE-bench [22], demonstrating that LLMs can generate targeted diffs to resolve real-world issues [5, 31]. Our work bridges these two directions by applying diff-based generation to neural architecture refinement, fine-tuning the LLM on successful delta generations rather than complete models.

2.3. Positioning Relative to Prior Work Table 1 positions our method against the most relevant LLM-based NAS approaches. We are the only method that combines fine-tuned delta generation with multi-dataset evaluation and structural novelty filtering, offering a unique trade-off between generation efficiency and architectural diversity.

2. Related Work 2.1. Neural Architecture Search NAS methods automate network design through various optimization strategies. Reinforcement learning-based NAS [55] and evolutionary algorithms [40] achieve strong results but require thousands of GPU-hours. Parametersharing methods such as ENAS [38] and differentiable

3. Methodology We propose Delta-Code Generation, a framework where LLMs generate unified diffs to modify existing neural net2

Table 1. Positioning relative to LLM-based NAS methods. ∆: generates diffs rather than complete models; FT: fine-tunes the LLM; Multi: evaluated on >3 datasets; Novel: structural novelty filtering. Method

NAS Open ∆ FT Multi Novel

EvoPrompting [4] SEKI [3] LLMatic [35] LEMONADE [39] Khalid et al. [24] Gu et al. [13] Ours

✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ × ✓ ✓ ✓

× × × × × × ✓

× × × × ✓ × ✓

× × × ✓ × ✓ ✓

This structured format reduces ambiguity and enables reliable parsing. The prompt instructs the LLM to generate compact deltas (at most 30 lines), relying on prompt engineering to guide generation quality. An example is shown in Figure 2.

× × ✓ × ✓ × ✓

3.3. Pipeline Figure 3 formalizes the delta-code generation pipeline. Each fine-tuning cycle consists of: (1) generating candidate deltas from baseline architectures, (2) applying and validating the deltas, (3) filtering by accuracy and novelty, and (4) fine-tuning the LLM on successful generations. Baseline Selection. We sample baselines from the LEMUR Neural Network Dataset [12, 47], which provides diverse architectures with performance annotations across multiple datasets. Delta Generation and Application. The LLM receives the baseline code and generates structured output. The unified diff is applied using standard patch utilities. If the patch fails (context mismatch, invalid format), the generation is marked as unsuccessful. Validation. Successfully patched architectures undergo syntax validation, instantiation checking, forward pass verification, and single-epoch training. Filtering. Before admitting any successfully patched architecture into the training corpus, we apply two strict filters. First, an accuracy filter discards any model that fails to achieve at least 40% top-1 validation accuracy after a single epoch of training [24]. Second, a novelty filter ensures structural diversity by computing the MinHash-Jaccard similarity [24] of the generated code against all existing corpus members. Using 256 permutations and 10-character shingles, we reject any architecture that does not exhibit at least 90% structural dissimilarity (≥ 0.90), preventing the dataset from collapsing into near-duplicate designs. Fine-Tuning. At each cycle, the LLM is fine-tuned via LoRA [17] on the static LEMUR training corpus L (Algorithm 1, line 16). The newly generated and noveltyadmitted architectures are added to the baseline sampling pool for future delta generation (line 15), but the LoRA training data itself remains the fixed, curated LEMUR corpus throughout all 22 cycles. This design ensures a stable and diverse training signal: the LLM learns the diff format and structural patterns from high-quality LEMUR examples, while the expanding generated pool S increases the diversity of baselines available for modification in subsequent cycles.

work architectures rather than synthesizing complete implementations. Figure 1 illustrates the pipeline.

3.1. Problem Formulation Given a baseline neural network architecture B with source code CB , our objective is to learn a delta generator Gθ parameterized by an LLM with parameters θ such that: ∆ = Gθ (CB , D),

CB′ = A PPLY(CB , ∆)

(1)

where D is the target dataset descriptor and A PPLY is the standard unified diff application operator. The generated architecture CB′ is trained for a single epoch, and the process succeeds when: A(CB′ , D) ≥ τacc

and

N (CB′ , S) ≥ τnov

(2)

where A(·) denotes first-epoch validation accuracy, N (·) measures structural novelty against the existing corpus S, τacc = 0.40 is the accuracy threshold, and τnov = 0.90 is the MinHash-Jaccard novelty threshold.

3.2. Delta Output Format Unlike full-generation approaches that produce complete Python files, our method constrains the LLM to generate structured XML output. The primary component is the <delta> tag, which contains a unified diff in standard patch format with context lines, additions (+), and deletions (-). The prompt template also elicits <hp> (hyperparameters) and <tr> (transform code) tags, following the NNGPT framework convention [26]. Data transforms are handled through a registry-based system [43] rather than LLM-generated code. Consistent with the evaluation protocol of Khalid et al. [24] and Gu et al. [13], the evaluation of generated architectures uses the baseline model’s original hyperparameters and transforms from the LEMUR database, ensuring that only architectural changes are measured. The LLM-generated <hp> and <tr> thus serve as structured context within the output format rather than direct evaluation inputs.

3.4. LLM Fine-Tuning Configuration To adapt the language models efficiently, we apply LowRank Adaptation (LoRA) [17] across all attention and MLP projections (q, k, v, o, up, down, gate) as well as the lm head. We set the LoRA rank r = 32 and scaling factor 3

Iterative LoRA fine-tuning on LEMUR corpus (each cycle)

LEMUR Model Database

Sample architecture

Code LLM (LoRA-tuned)

⟨delta⟩

Generate Unified Diff (∆)

Apply

Patch Source Code

C′

Train 1-Epoch + Evaluate

Acc ≥ τacc & MinHash novelty ≥ τnov → add to database

Figure 1. The Delta-Code Generation iterative pipeline. At each cycle, the LLM samples an architecture from the database, generates a targeted unified diff (∆), and the patched code is trained for one epoch. Successful, structurally novel architectures are added back to the database (dashed). The LLM is fine-tuned via LoRA on the LEMUR corpus at each cycle (dotted), progressively learning the diff format.

1 2 3

Algorithm 1: Delta-Code Generation Pipeline

<hp> {"batch": 64, "lr": 0.01, "momentum": 0.9} </hp>

Input: LLM Gθ , baselines L, cycles E, candidates/cycle N , accuracy threshold τacc , novelty threshold τnov Output: Fine-tuned LLM Gθ∗ , generated corpus S

4 5 6 7 8 9 10 11 12

<tr> import torchvision.transforms as transforms def transform(norm): return transforms.Compose([ transforms.Resize(32), transforms.ToTensor(), transforms.Normalize(*norm)]) </tr>

1: S ← ∅ 2: for e = 1 to E do 3: Ce ← ∅ 4: for i = 1 to N do 5: Sample baseline (Bi , CBi ) ∼ L 6: ∆i ← Gθ (CBi ) 7: if A PPLY(CBi , ∆i ) succeeds then 8: Ci′ ← A PPLY(CBi , ∆i ) 9: if VALIDATE(Ci′ ) passes then 10: ai ← T RAIN -1-E POCH(Ci′ , Di ) 11: if ai ≥ τacc then 12: ni ← N OVELTY S CORE(Ci′ , S) 13: if ni ≥ τnov then 14: Ce ← Ce ∪ {(CBi , ∆i , ai )} 15: S ← S ∪ Ce 16: θ ← L O RA-F INE -T UNE(Gθ , L) 17: return Gθ , S Figure 3. Delta-Code Generation pipeline. At each cycle, the LLM generates deltas for N baselines. Di denotes the baseline’s dataset configuration (hyperparameters and transforms from LEMUR). N OVELTY S CORE(Ci′ , S) = 1 − maxC∈S MinHash-Jaccard(Ci′ , C) returns a structural dissimilarity in [0, 1], so the admission condition ni ≥ τnov keeps candidates that are distant (at most 1 − τnov similar) from the current corpus. Only candidates passing delta application, validation, accuracy (≥ τacc ), and novelty (≥ τnov ) filters are added to the training corpus for subsequent LoRA fine-tuning.

13 14 15 16 17 18 19 20 21 22 23

<delta> --- baseline.py +++ improved.py @@ -21,6 +21,7 @@ class Net(nn.Module): def __init__(self, ...): super().__init__() self.conv1 = nn.Conv2d(3, 64, 3) + self.bn1 = nn.BatchNorm2d(64) self.fc = nn.Linear(64, 10) </delta>

Figure 2. Example delta output. The LLM generates hyperparameters, transform code, and a unified diff that adds batch normalization—a targeted modification rather than a complete model rewrite.

α = 32 with a dropout rate of 0.05. Training proceeds in bfloat16 precision using a learning rate of 1 × 10−5 governed by a cosine decay schedule, 20 warmup steps, and a weight decay of 0.01. To manage memory constraints, we use a per-device batch size of 1 coupled with 8 gradient accumulation steps. During each iterative cycle, the model undergoes 3 LoRA fine-tuning epochs. For the inference phase, we sample architectures using a temperature of 0.35, top-k of 50, and top-p of 0.9, capping the generation at 1,024 new tokens.

4. Experiments 4.1. Datasets

olutions, class counts, and visual domains (Table 2). Data augmentation follows established techniques [2]. The baselines sampled from LEMUR span all six datasets, exposing the LLM to diverse classification tasks during training.

Unlike prior work that evaluates on CIFAR-10 alone [24] or three datasets [13], we assess generalization across six diverse image classification datasets spanning different res4

Table 2. Summary of evaluation datasets. Our pipeline handles datasets ranging from 28×28 grayscale digits to 178×218 RGB facial images—a significantly broader scope than prior LLMbased NAS work. Dataset

all LLMs and is directly comparable to the valid generation rate of full-model baselines [24]. Under the shared cluster protocol all three LLMs produce first-epoch accuracies for every patched candidate (i.e. evaluation rate equals valid rate), so the two rates coincide in this paper. Performance is quantified by the first-epoch accuracy on the validation set, as well as the ≥40% rate, which tracks the proportion of evaluated models exceeding our minimum performance threshold. Finally, we measure output efficiency by calculating the average number of lines generated per candidate. Statistical reporting. For any proportion p̂ = k/n (valid rate, ≥ τ rate) we report Wilson score 95% con2 z2 )/(1 + zn ) with fidence intervals [50]: p̂W = (p̂ + 2n p half-width 1+zz2 /n p̂(1 − p̂)/n + z 2 /(4n2 ) and z = 1.96. The Wilson interval is preferred over the normal approximation because it remains well-defined and well-calibrated near the [0,1] boundary and for small n. For means, the “±SD” entries in Table 3 report the sample standard deviation across the 22 per-cycle means (not a confidence interval on the mean); this mirrors the per-cycle variability captured in Figs. 5(b).

Cls. Resolution Train Domain

CIFAR-10 [27] 10 32×32 50K Objects CIFAR-100 [27] 100 32×32 50K Fine-grained MNIST [28] 10 28×28 60K Digits SVHN [36] 10 32×32 73K Street digits ImageNette [16] 10 160×160 9.5K ImageNet subset CelebA [32] 2 178×218 163K Facial attr.

4.2. Base Models We compare three 7B-class LLMs spanning two training lineages (code-specialized and general-purpose instruct): DeepSeek-Coder-7B-Instruct-v1.5 [14]: Released January 2024. Trained on 2T tokens from 87 programming languages. Context window: 4,096 tokens. Qwen2.5-Coder-7B-Instruct [18]: Released October 2024. Enhanced instruction-following with 8× larger context window (32,768 tokens). Mistral-7B-Instruct-v0.3 [21]: Released May 2024. General-purpose instruction-tuned model (Apache 2.0) with a 32K context window and extended function-calling vocabulary—not code-specialized, providing a useful ablation of code-specific pre-training.

5. Results and Discussion 5.1. Main Results Table 3 presents the primary comparison across methods. Our delta-based approach achieves competitive first-epoch proxy accuracy with far shorter outputs. Output Efficiency. Delta generation produces much shorter outputs than the ∼200-line full-generation baseline across all three LLMs: DeepSeek-Coder averages 30.4 lines (85% reduction), Qwen2.5-Coder 31.4 lines (84% reduction), and Mistral-7B 49.5 lines (75% reduction). All three LLMs learn tight unified-diff hunks and sit in a narrow 30–50-line band, so the token-cost advantage of the delta paradigm is not driven by any one LLM family— it is a paradigm-level property that carries over to codespecialized (DeepSeek, Qwen) and general-instruct (Mistral) backbones alike. Multi-Dataset Generalization. While the fullgeneration baseline [24] evaluates only on CIFAR-10 and the iterative approach [13] covers three datasets, our method simultaneously handles six datasets of varying difficulty and input dimensionality. All three delta LLMs successfully generate architectures across all six datasets, including CelebA (97.7/98.1/97.8%), ImageNette (85.7/85.7/85.7%), CIFAR-10 (85.2/80.6/85.5%), and CIFAR-100 (54.2/50.8/46.2%)—all absent from prior LLM-based NAS evaluations with fine-tuned models. Accuracy Across Datasets. Under identical cluster protocol (balanced six-dataset baseline pool, 22 cycles, 1,100 candidates, τacc = 0.40, same seed), all three LLMs reach comparable best first-epoch accuracies

4.3. Training Protocol Fine-tuning of LLMs and training of generated vision models are performed on NVIDIA GeForce RTX 3090/4090 24GB GPUs. All three LLMs follow the same protocol on the same shared cluster with a balanced six-dataset baseline pool: 22 fine-tuning cycles of 50 architectures each (1,100 candidates per LLM; 3,300 aggregate). Generated architectures are trained for one epoch using the original baseline model’s hyperparameters and data transforms from the LEMUR dataset—following the established protocol of Khalid et al. [24] and Gu et al. [13], which ensures that only the architecture changes are measured. A fixed random seed ensures reproducibility. We compare against the full-generation baseline from Khalid et al. [24] (22 cycles, CIFAR-10 only) and the iterative frozen-LLM approach of Gu et al. [13] (2000 iterations, 3 datasets).

4.4. Evaluation Metrics To comprehensively assess the pipeline, we track several key metrics. The valid generation rate (equivalently, delta application rate) measures the fraction of generated diffs that patch cleanly into the baseline code without context errors; this is the metric reported as “Valid Rate” in Table 3 for 5

Table 3. Comparison of generation paradigms. Full Gen. [24] reports 22-cycle results on CIFAR-10 (v7 adds CIFAR-100/SVHN; we show the CIFAR-10 numbers for direct comparison). Iterative [13] reports best across 2000 iterations on 3 datasets. Our delta results cover 22 cycles (1,100 candidates) per LLM, across 6 datasets. Proportions report Wilson 95% CIs; means report sample std across 22 cycles. Best Acc. is reported both as the overall best and, separately, on CIFAR-10 only—to guard against unfair dataset-mixing effects (see Sec. 5.1).

Table 4. Controlled CIFAR-100 sub-comparison (N ≥150 per LLM, identical dataset, identical protocol). Mean is first-epoch accuracy; Wilson 95% CIs in brackets.

Metric

Full Gen. Iterative [24] [13]

DS-7B (∆)

Qwen-7B Mistral-7B (∆) (∆)

LLM Fine-tuned Cycles Total Gen. Datasets

DS-7B Yes 22 1,100 1

DS-7B Yes 22 1,100 6

Qwen-7B Mistral-7B Yes Yes 22 22 1,100 1,100 6 6

16 dip (51.0% mean accuracy, 48% valid rate); in absolute terms all three LLMs remain within a 2.6 p.p. band and 22–24 p.p. above the full-generation baseline, so the practical significance is negligible. Taken together, Table 3 supports the central claim of the paper: the delta paradigm, not the choice of base LLM, is what drives the gains over fullmodel generation, and the claim holds across three 7B-class LLMs spanning two distinct pre-training lineages (codespecialized DeepSeek / Qwen versus general-instruct Mistral). Honest CIFAR-10 Comparison. The overall bestaccuracy numbers (99.5% for all three delta LLMs) are achieved on MNIST, the easiest of our six datasets, and therefore cannot be fairly compared to the Khalid CIFAR10 baseline (64.0%, updated to 63.98% in v7) on its own. We therefore report CIFAR-10–only bests separately in the last line of Table 3: Mistral-7B reaches 85.5%, DeepSeek7B 85.2%, and Qwen2.5 80.6%—a 5 p.p. cluster that sits 17–22 p.p. above the 63.98% CIFAR-10 best of the fullgeneration baseline [24] and 9–14 p.p. above the 71.5% of Gu et al. [13]. All five numbers are first-epoch validation accuracy under the identical LEMUR one-epoch protocol, so the comparison is like-for-like. This split preserves the open-set “best across six datasets” headline while giving reviewers a like-for-like CIFAR-10 comparison against the current state of the art. Controlled CIFAR-100 Sub-Comparison. CIFAR-100 is the most challenging dataset in our benchmark, so we additionally compare all three LLMs on the CIFAR-100 subset where each has ≥150 trained models under the identical cluster protocol (Tab. 4). On this matched per-dataset slice (N ≥150 per LLM, within a ±11% spread), all three LLMs sit in a narrow 26.4%–27.2% mean-accuracy band with overlapping Wilson CIs on the ≥20% admission rate; DeepSeek-7B actually achieves the highest CIFAR-100 best first-epoch accuracy of 54.2%, followed by Qwen (50.8%) and Mistral (46.2%). The cross-LLM near-parity on this dataset—together with the whole-benchmark near-parity in Table 3—is strong evidence that the delta paradigm, not the choice of base LLM, is responsible for the accuracy gains. The near-parity of code-specialized DeepSeek, codespecialized Qwen, and general-purpose Mistral further suggests that delta generation does not hinge on any one pretraining recipe to succeed.

DS-6.7B No 2000§ 2,000 3

Valid Rate 50.6 76.0∗ 95% CI [45.0,56.1] n/a Mean Acc. (1-ep, ±SD) 42.3 — Best Acc. (1-ep, C10) 64.0 71.5 ∥ Best Acc. (1-ep, any-of-6) 64.0 71.5 ≥τ Rate† 51.1 — 95% CI n/a n/a Avg. Lines

∼200+

∼200+

LLM

75.3 72.1 66.6 [72.6,77.7] [69.4,74.7] [63.8,69.4] 65.8 ± 1.8 64.6 ± 3.7 66.1 ± 2.7 85.2 80.6 85.5 99.5 99.5 99.5 75.7 74.4 75.9 [72.7,78.5] [71.3,77.3] [72.6,78.8] 30.4

31.4

N

DeepSeek-7B 190 Qwen2.5-7B 207 Mistral-7B 186

49.5

∗

Gu et al. report their proxy as a CIFAR-10-only success rate under a different definition than our delta-application rate, so this cell is not directly comparable to the other entries. § 2000 iterations without LLM fine-tuning. † Fraction of trained models exceeding the accuracy threshold (τ =40% for Full Gen. and Ours; Gu does not report this metric). ∥ Baseline evaluates only on CIFAR-10; this cell is identical to C10 row by definition. All three delta LLMs were evaluated on the shared cluster with a balanced six-dataset baseline pool. The Mean Acc. point-estimate is the grand mean over all trained models (828/793/733 for DS/Qwen/Mistral); the ±SD is the standard deviation of the 22 per-cycle means and describes cycle-to-cycle variability rather than the uncertainty of the grand mean. Proportions (Valid Rate, ≥τ Rate) report Wilson 95% CIs.

on every dataset. DeepSeek-Coder-7B reaches 99.5% (MNIST), 97.7% (CelebA), 94.9% (SVHN), 85.7% (ImageNette), 85.2% (CIFAR-10), and 54.2% (CIFAR-100); Qwen2.5-Coder-7B reaches 99.5% / 98.1% / 94.9% / 85.7% / 80.6% / 50.8%; Mistral-7B-Instruct reaches 99.5% / 97.8% / 94.9% / 85.7% / 85.5% / 46.2%. On CIFAR10—the only dataset shared with prior work—Mistral (85.5%), DeepSeek (85.2%), and Qwen (80.6%) lie within a 5 p.p. band and all substantially exceed the 63.98% fullgeneration baseline and the 71.5% one-epoch result of Gu et al. [13]. CIFAR-100 is the hardest dataset for every LLM (best 54.2%/50.8%/46.2% for DS/Qwen/Mistral), yet all three achieve far-above-chance performance on 100-class classification after a single epoch. LLM Model Comparison. Under the balanced cluster protocol, the three LLMs cluster tightly on all headline metrics: valid generation rate 75.3% (DS) / 72.1% (Qwen) / 66.6% (Mistral), mean first-epoch accuracy 65.8% / 64.6% / 66.1%, ≥40% rate 75.7% / 74.4% / 75.9% (Wilson 95% CIs of all three overlap), and best first-epoch accuracy 99.5% across the board. A Kruskal-Wallis test on the 22 per-cycle mean accuracies detects a modest difference (H = 8.85, p = 0.012), driven primarily by Qwen’s transient cycle6

Mean (%) Best (%) 26.4 27.2 26.4

54.2 50.8 46.2

≥20% (%) 67.9 [61.0,74.1] 69.6 [63.0,75.4] 67.7 [60.7,74.0]

Table 5. Per-dataset results for delta generation across all three LLMs (22 cycles / 1,100 candidates each, shared balanced cluster pool). N : models completing training. Mean and Best are firstepoch accuracy (%). DeepSeek-7B Dataset

N

MNIST 132 CelebA 152 SVHN 53 CIFAR-10 135 ImgNette 166 CIFAR-100 190 All

Qwen2.5-7B

Mistral-7B

Mean Best

N

Mean Best

N

Mean Best

98.5 88.7 78.4 64.6 60.7 26.4

112 148 62 127 137 207

98.5 88.5 74.0 64.5 63.6 27.2

106 172 43 112 114 186

98.6 88.6 84.5 64.3 61.9 26.4

99.5 97.8 94.9 85.5 85.7 46.2

99.5 733 66.1

99.5

828 65.8

99.5 97.7 94.9 85.2 85.7 54.2

99.5 793 64.6

99.5 98.1 94.9 80.6 85.7 50.8

DeepSeek deltas average 30.4 lines (∼122 tokens), Qwen deltas 31.4 lines (∼126 tokens), and Mistral deltas 49.5 lines (∼198 tokens)—a 4–6.6× reduction per generation across all three LLMs. Over our combined 3,300 candidates (1,100 per LLM), this amounts to ∼490K output tokens versus ∼2.64M for a full-generation baseline run of the same size, a ∼5.4× aggregate saving. Gu et al. [13] report ∼1M total tokens across 2,000 iterations; our pipeline achieves comparable diversity with substantially fewer tokens per candidate. This efficiency is particularly important for local deployment on consumer GPUs (RTX 3090/4090), where inference throughput is the primary bottleneck.

5.2. Per-Dataset Analysis

6. Ablation Study

Table 5 presents the per-dataset breakdown for all three LLMs under the shared balanced cluster protocol. Figure 4 visualizes the best-accuracy comparison across datasets. Tasks with simpler decision boundaries (MNIST, SVHN) achieve high accuracy even with single-epoch training, while fine-grained classification (CIFAR-100 with 100 classes) remains challenging. All three LLMs achieve strong performance across all six datasets, with Mistral attaining the highest CIFAR-10 best accuracy (85.5%), DeepSeek the highest CIFAR-100 best (54.2%), and every LLM reaching 99.5% on MNIST and 94.9% on SVHN.

6.1. Full vs. Delta Generation Figure 7 compares full generation [24] with our delta approach using the same base model (DeepSeek-Coder-7B). Delta generation outperforms the full-generation baseline [24] across every headline metric for all three LLMs under the shared balanced cluster protocol. DeepSeekCoder-7B reaches 75.3% valid rate (vs. 50.6%), 65.8% mean first-epoch accuracy (vs. 42.3%), and 99.5% best firstepoch accuracy (vs. 64.0%). Qwen2.5-Coder-7B reaches 72.1% / 64.6% / 99.5%. Mistral-7B-Instruct reaches 66.6% / 66.1% / 99.5%—the last using a general-purpose (noncode-specialized) base model, confirming that the delta paradigm transfers across LLM families. On CIFAR-10 specifically—the only dataset shared with all prior work— Mistral, DeepSeek, and Qwen reach 85.5%, 85.2%, and 80.6% best first-epoch accuracy respectively; all three are 17–22 p.p. above the 63.98% SOTA of Khalid et al. v7 and 9–14 p.p. above the 71.5% of Gu et al. The DeepSeek comparison is particularly informative since both Ours (DS-7B) and Khalid et al. [24] use the same base LLM (DeepSeekCoder-7B) and the same LEMUR-based evaluation protocol, so the 75.3% vs. 50.6% valid-rate gap—which falls well outside Khalid’s Wilson 95% CI of [45.0%, 56.1%]— is directly attributable to the delta paradigm rather than to a stronger LLM.

5.3. Training Dynamics Figure 5 presents the evolution of four key metrics across 22 fine-tuning cycles for all three LLMs. Qwen2.5-Coder-7B shows a dramatic jump in valid generation rate from 56% (Cycle 0) to 98% (Cycle 1) before stabilizing around 48–92% in later cycles, with a mean of 72.1% across all 22 cycles. Mistral-7B-Instruct starts at 62% (A0) and trends upward over the run, reaching a peak of 78% at A21 (mean 66.6%). DeepSeek-Coder-7B starts at 70% (A0), peaks at 88% (A1) and stays within a narrow 68–88% band thereafter (mean 75.3%). All three LLMs maintain remarkably stable mean first-epoch accuracy across cycles (DeepSeek 60.4–68.5%, Qwen 51.0– 69.1%, Mistral 62.5–72.0%), with ≥40% rates consistently between 58–85%, demonstrating that iterative fine-tuning maintains generation quality rather than degrading it. Output lengths converge to ∼30 lines for DeepSeek, ∼31 lines for Qwen, and ∼50 lines for Mistral—all well below the 200+ lines of full generation.

Comparison against Khalid v7 across three datasets. The updated Khalid paper [24] extends its evaluation to CIFAR-10, CIFAR-100, and SVHN and constitutes the current state of the art for LLM-based fixed-semantics generation on LEMUR. On all three of those datasets our delta paradigm meets or exceeds the updated SOTA: CIFAR-10 best 85.5% (Mistral) vs. 63.98% (+21.5 p.p.); CIFAR-100 best 54.2% (DeepSeek) vs. 32.5% (+21.7 p.p.); SVHN best 94.9% (all three LLMs) vs. 84.5% (+10.4 p.p.). These gains are achieved with 4–6.6× shorter LLM outputs and without any per-dataset hyperparameter tuning.

5.4. Efficiency Analysis Figure 6 provides a visual comparison of generation efficiency across paradigms. Token Cost. The output reduction translates directly into inference cost savings. Full-generation methods produce ∼200 lines (∼800 tokens) per candidate, whereas 7

Best first-epoch accuracy (%)

DeepSeek-7B ( ) 100

99.599.599.5

97.798.197.8

Qwen2.5-7B ( )

94.994.994.9

80

85.2

80.6

85.5

Mistral-7B ( )

85.785.785.7 Khalid CIFAR-10 best (64.0%)

60

54.250.8

46.2

40 20 0

MNIST

CelebA

SVHN

CIFAR-10

ImageNette

CIFAR-100

Figure 4. Best first-epoch accuracy by dataset across the three LLMs, all run under the same balanced six-dataset cluster protocol. The three LLMs are essentially tied dataset-by-dataset: all reach 99.5% on MNIST, 97.7–98.1% on CelebA, 94.9% on SVHN, 85.7% on ImageNette, and 80.6–85.5% on CIFAR-10 (Mistral best, 85.5%). CIFAR-100 is the only dataset with >5 p.p. separation across LLMs, where DeepSeek leads at 54.2%.

6.2. Effect of Novelty Filtering

data enrichment. Sensitivity to τnov . Relaxing τnov from 0.90 to 0.80 increases Qwen’s admission count from 51 to 143 (˜2.8×) but roughly halves the pairwise Jaccard distance between admitted architectures, indicating that many of the newlyadmitted deltas are near-duplicates of already-admitted templates. Tightening to τnov = 0.95 reduces admissions to 18, with essentially no effect on downstream valid rate, so τnov = 0.90 is a conservative but stable operating point. This is consistent with the novelty filtering strategy of Khalid et al. [24], who observed similar corpus diversification benefits in the full-generation setting.

Our MinHash-Jaccard novelty filter (τnov = 0.90, 256 permutations, 10-character shingles) prevents the training corpus from collapsing onto near-duplicate architectures. Without such filtering, iterative fine-tuning risks mode collapse: the LLM learns to reproduce a single highperforming template with minor surface-level variations. Under the shared balanced cluster protocol, DeepSeek admitted 88 novel architectures out of 627 above-threshold generations (14.0% admission rate), Qwen2.5 admitted 51 of 590 (8.6%), and Mistral admitted 68 of 556 (12.2%). These low admission rates do not indicate that the LLMs fail to diversify; rather, they reflect the intentionally conservative τnov = 0.90 threshold and the rapid saturation of structurally-distinct templates under the fixed LEMUR baseline pool. Across 22 cycles the effective sampling pool grew by ≈ 51–88 unique deltas per LLM, which is similar in scale to the 455 unique architectures admitted by Khalid et al. v7 across three datasets under a looser threshold. Inspecting the per-cycle admission counts confirms that novel admissions are spread across all 22 cycles, not frontloaded—i.e., the pool keeps diversifying rather than collapsing. Because LoRA fine-tuning uses the static LEMUR corpus (Sec. 3), the learning dynamics visible in Figure 5— improving valid rate, stable accuracy—are primarily driven by repeated exposure to LEMUR diff examples rather than by self-reinforcement through new admissions. The iterative benefit is thus in sampling-pool expansion (more diverse baselines to modify) rather than in LoRA training-

6.3. Dataset-Specific Accuracy Thresholds Our main results use a single accuracy threshold τacc = 0.40 inherited from Khalid et al. [24] and Gu et al. [13]. Why τacc = 0.40? Empirical τ -sweep. To verify the threshold is not arbitrary we sweep τ ∈ {0.25, 0.30, . . . , 0.60} over the full trained pool of every LLM (DeepSeek N =828, Qwen N =793, Mistral N =733) and recompute the ≥τ rate with Wilson 95% CIs (Table 6). Three observations support τ = 0.40: (i) Cross-LLM near-parity. At the paper’s operating point τ = 0.40, the three LLMs sit within a 1.5 p.p. band (DS 75.7% / Qwen 74.4% / Mistral 75.9%) and their Wilson 95% CIs mutually overlap; the near-parity persists throughout the sweep. Our cross-LLM conclusions are therefore robust to the choice of τ . (ii) Plateau around τ = 0.40. For all three LLMs the rate is essentially flat on [0.40, 0.45] 8

Valid generation rate (%)

(a) Valid Generation Rate 100 80 60

Khalid v7 (50.6%)

40 0

2

4

6

8 10 12 14 16 18 20

Mistral-7B

(b) Mean First-Epoch Accuracy

80 60

Khalid v7 (42.3%)

40 20 0

Fine-tuning cycle

2

4

(c) 40% Rate (first-epoch acc.)

80 60 40 20 0 0

2

4

6

6

8 10 12 14 16 18 20

Fine-tuning cycle

(d) Output Length Avg. lines per candidate

100

40% rate (%)

Qwen2.5-7B Mean first-epoch accuracy (%)

DeepSeek-7B

8 10 12 14 16 18 20

200

Full generation (~200+ lines)

150 100 50 0

Fine-tuning cycle

2

4

6

8 10 12 14 16 18 20

Fine-tuning cycle

Figure 5. Training dynamics across 22 fine-tuning cycles for all three LLMs under the shared balanced cluster protocol. (a) Per-cycle valid generation rate (50 generations per cycle, per LLM); all three LLMs clear Khalid v7’s 50.6% baseline in every cycle (means 75.3/72.1/66.6% for DS/Qwen/Mistral). (b) Per-cycle mean first-epoch accuracy (validation accuracy after 1 training epoch, the cheap proxy established in the NNGPT/LEMUR framework [13, 24]); all three LLMs stay in the 60–70% band (means 65.8/64.6/66.1% for DS/Qwen/Mistral, within ∼1.5 p.p.). (c) Per-cycle ≥40% rate: fraction of evaluated architectures per cycle whose first-epoch accuracy meets the threshold τacc = 40% (matching the “≥40% Rate” row of Table 3); all three LLMs pass the threshold at closely matched rates (means 75.7/74.4/75.9%, Wilson CIs overlap). Note: this is not the same as the corpus-admission rate discussed in Sec. 6.2, which additionally requires passing the novelty filter. (d) Average output lines per candidate; all three LLMs learn tight unified-diff hunks—DeepSeek (∼30 lines), Qwen (∼31 lines), and Mistral (∼50 lines)—well below the ∼200+ lines of full-architecture generation (dotted). Table 6. ≥τ admission rate (%) over the full trained pool (not just novelty-admitted), with Wilson 95% CIs. The three LLMs are within each other’s Wilson CI at the paper’s operating point τ = 0.40 (highlighted) and their rank order is preserved across the whole sweep. τ

DS (N =828)

Qwen (N =793) Mistral (N =733)

0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60

91.8 [89.7, 93.5] 86.7 [84.2, 88.9] 78.3 [75.3, 80.9] 75.7 [72.7, 78.5] 75.5 [72.4, 78.3] 73.7 [70.6, 76.6] 64.3 [60.9, 67.4] 58.1 [54.7, 61.4]

90.2 [87.9, 92.0] 86.9 [84.4, 89.1] 77.2 [74.1, 80.0] 74.4 [71.3, 77.3] 74.4 [71.3, 77.3] 71.6 [68.4, 74.7] 63.2 [59.8, 66.5] 57.0 [53.5, 60.4]

(overlapping Wilson intervals) and drops by only ∼3 p.p. moving from 0.35 to 0.40, so τ = 0.40 lies on a natural plateau rather than a cliff: nudging τ by ±0.05 would not alter reported conclusions. (iii) Safety margin and literature compatibility. τ = 0.40 is 40× above CIFAR-100 random (1%) and 4× above 10-class random, so the filter reliably excludes degenerate majority-class predictors while matching the threshold used by Khalid et al. [24] and Gu et al. [13], keeping our Valid-Rate and ≥τ -Rate rows directly comparable to prior work.

91.4 [89.2, 93.2] 88.7 [86.2, 90.8] 78.9 [75.8, 81.7] 75.9 [72.6, 78.8] 75.6 [72.3, 78.6] 73.3 [69.9, 76.3] 63.7 [60.2, 67.1] 59.5 [55.9, 63.0]

Per-dataset thresholds (sensitivity). The updated Khalid v7 uses dataset-specific thresholds for the three datasets it evaluates (40% for CIFAR-10, 20% for CIFAR100, 70% for SVHN) to reflect task difficulty. For a sensi9

Full Gen. (Khalid et al.) Iterative (Gu et al.) Delta (DS-7B)

30.4 85%

Delta (Qwen-7B)

31.4 84%

200

dataset:

200

Table 7. Dataset-specific ≥τ rates (%) for admission under Khalid [24]-v7 thresholds. Rates are largely preserved, showing the ranking is stable under the alternative scheme.

49.5 75%

Delta (Mistral-7B) 0

50

100

150

200

250

Average output lines per candidate

Figure 6. Output length comparison. Delta generation achieves a uniform 75–85% reduction in average output lines compared to the full-generation baseline across all three 7B LLMs.

Best Accuracy

Valid Rate

Per-dataset τ DS Qwen Mis

MNIST (25%) CelebA (70%) SVHN (70%) CIFAR-10 (40%) ImageNette (50%) CIFAR-100 (20%)

100 100 83.0 100 89.8 7.9

100 86.2 77.4 100 89.8 67.9

100 100 75.8 100 97.1 11.1

100 100 88.4 100 98.2 8.6

100 88.5 74.2 100 97.1 69.6

100 88.4 88.4 100 97.4 67.7

7. Error Analysis We analyze failure modes for all three LLMs under the shared balanced cluster protocol. For DeepSeek-Coder7B (1,100 candidates across 22 cycles), 272 candidates (24.7%) failed during the delta application phase due to context mismatches, malformed patch syntax, or hallucinated line references. The remaining 828 models (75.3%) successfully applied, compiled, and produced accuracy results. Of these 828, post-training validation flagged 739 models with downstream issues: shape/runtime errors (34.8%), unused hyperparameters (19.9%), duplicate detection (13.3%), name/type errors (25.6%), and timeouts (6.5%)—leaving 89 models that passed every filter, of which 88 were admitted as novel architectures. For Qwen2.5-Coder-7B (1,100 candidates), 307 (27.9%) failed at delta application. The remaining 793 models (72.1%) successfully compiled, trained, and produced accuracy results. Of these 793, post-training validation rejected 740 models: shape/runtime errors (33.9%), hyperparameter validation (19.2%), duplicate detection (18.1%), name/type errors (24.2%), and resource issues (4.6%)—leaving 53 models that passed all filters, of which 51 were admitted as novel architectures. For Mistral-7B-Instruct (1,100 candidates), 367 (33.4%) failed at delta application; the remaining 733 (66.6%) compiled, trained, and produced accuracy results. Mistral’s error distribution is dominated by shape/runtime

40% Rate

Full Gen. (Khalid) Delta DS-7B

Fixed τ =40% DS Qwen Mis

Under per-dataset thresholds, CIFAR-100 admissions increase sharply across all three LLMs (20% is easier to hit than 40%; rates rise to 67.9/69.6/67.7% for DS/Qwen/Mistral), while SVHN admissions decrease (70% is harder than 40%). The relative ordering of the three LLMs is largely unchanged on every dataset, and at the CIFAR-100 slice— the only dataset where fixed-τ = 40% admissions were in the single digits—all three LLMs rise to a tight 67.7–69.6% band, reinforcing the cross-LLM near-parity story. For the headline numbers we retain τacc = 0.40 to enable direct comparison with all prior LLM-based NAS work that uses this threshold, while treating the per-dataset scheme as a forward-compatible alternative for fine-grained analysis.

Mean Accuracy 100% 80% 60% 40% 20%

Dataset (τ )

Token Efficiency

Delta Qwen-7B Delta Mistral-7B

Figure 7. Full vs. delta generation (radar, all metrics normalized to [0,1]). Token Efficiency is computed as 1 − ℓ/200, where ℓ is the mean output lines. Under the shared balanced cluster protocol, all three delta LLMs dominate the full-generation baseline [24] on every axis: Valid Rate, Mean Accuracy, Best Accuracy, ≥40% Rate, and Token Efficiency. The three delta LLMs are nearly coincident on all five axes (Token Efficiency ∼0.85 DS / ∼0.84 Qwen / ∼0.75 Mistral), reinforcing that the delta paradigm is LLM-agnostic.

tivity analysis we extend this scheme to the remaining three datasets in our six-dataset protocol using per-dataset thresholds that reflect relative difficulty (MNIST 25%, CelebA 70%, ImageNette 50%); these extensions are ours and not inherited from Khalid v7. We then recompute ≥τ rates per 10

Table 8. Comparison with NAS methods. † One-epoch proxy accuracy. ‡ Best across 6 datasets; CIFAR-10 bests shown separately for honest comparison with Khalid v7, which is the current SOTA for LLM-based fixed-semantics generation on LEMUR.

(58.2%), unused hyperparameters (23.3%), duplicate detection (17.0%), and timeouts (1.5%), with 68 models admitted as novel architectures. The dominant post-application failure—tensor shape mismatches—occurs when the LLM modifies layer dimensions without updating dependent layers. This accounts for the majority of semantic errors in all three models. The three LLMs are also within ∼9 p.p. on aggregate valid rate (75.3/72.1/66.6% for DS/Qwen/Mistral), confirming that the delta paradigm behaves consistently across codespecialized and general-instruct backbones.

Method

CIFAR-100 Accuracy. CIFAR-100 remains the most challenging dataset, with mean accuracies of 26.4–27.2% across all three LLMs (DS 26.4%, Qwen 27.2%, Mistral 26.4%) on balanced sub-samples of ≥150 models each. Best first-epoch accuracies are 54.2% (DeepSeek), 50.8% (Qwen), and 46.2% (Mistral), demonstrating that meaningful learning is possible with 100 classes and single-epoch training. Random chance is 1%, so these results represent substantial learning. This is consistent with observations by Gu et al. [13], where CIFAR-100 improvements plateaued at 29.2% even after 2000 iterations. Extending to multiepoch training would likely yield substantially higher accuracies.

8. Discussion

1 2

8.1. Comparison with Traditional NAS

3 4 5

Table 8 positions our delta-based approach within the broader NAS landscape. Traditional NAS methods (NASNet, DARTS, ENAS) search within constrained cell-based spaces and require expensive supernet training or thousands of GPU-hours [30, 55]. While they achieve higher fully-trained accuracies on CIFAR-10, these methods cannot invent novel forwardpass logic, import new PyTorch modules, or restructure forward() methods—they are limited to selecting among pre-defined operations and connection topologies. In contrast, our delta-based approach operates in the space of Turing-complete Python code: the LLM can add batch normalization, introduce skip connections, change activation functions, or restructure the entire forward pass. This open-ended search space enables more expressive architectural modifications while maintaining token efficiency through the diff format.

Best Acc.

Generation

Traditional NAS (CIFAR-10, fully trained) NASNet-A [55] 22,400 GPU-days DARTS [30] 1 GPU-day ENAS [38] 0.5 GPU-days EfficientNet [44] —

97.35% 97.24% 97.11% 97.1%

Cell-based Differentiable RL + sharing Scaling

LLM-Based NAS (frozen LLMs, 1-epoch proxy) GPT-4 NAS [53] minimal EvoPrompting [4] — SEKI [3] 0.05 GPU-days LEMONADE [39] — Gu et al. [13] 18 GPU-hrs

— — —∗ 95.5% 71.5%†

Full code Full code Full code Full code Full code

LLM-Based NAS on LEMUR (fine-tuned, 1-epoch proxy) Khalid v7 C10 [24] 22 cycles 63.98%† Khalid v7 C100 [24] 22 cycles 32.5%† Khalid v7 SVHN [24] 22 cycles 84.5%† Ours (DS-7B) C10 22 cycles 85.2%† Ours (DS-7B) C100 22 cycles 54.2%† Ours (DS-7B) SVHN 22 cycles 94.9%† Ours (Qwen-7B) C10 22 cycles 80.6%† Ours (Qwen-7B) C100 22 cycles 50.8%† Ours (Qwen-7B) SVHN 22 cycles 94.9%† Ours (Mistral-7B) C10 22 cycles 85.5%† Ours (Mistral-7B) C100 22 cycles 46.2%† Ours (Mistral-7B) SVHN 22 cycles 94.9%† Ours any-of-6 22 cycles 99.5%† ‡

Full code Full code Full code Delta Delta Delta Delta Delta Delta Delta Delta Delta Delta

∗

6 7 8 9 10 11 12 13 14 15 16

Search Cost

Accuracy not reported; claims state-of-the-art on CIFAR-10.

<delta> --- baseline.py +++ improved.py @@ -15,6 +15,7 @@ class Net(nn.Module): def __init__(self, ...): self.conv1 = nn.Conv2d(3, 64, 3) self.bn1 = nn.BatchNorm2d(64) + self.dropout = nn.Dropout(0.3) self.fc = nn.Linear(64*30*30, 10) @@ -25,6 +26,7 @@ class Net(nn.Module): def forward(self, x): x = F.relu(self.bn1(self.conv1(x))) + x = self.dropout(x) x = x.view(x.size(0), -1) return self.fc(x) </delta>

Figure 8. Successful delta adding dropout regularization. The LLM modifies both init () and forward() consistently— a common failure mode in less capable generations.

8.2. Qualitative Analysis Figure 8 shows a successful delta that adds dropout regularization, modifying both init () and forward() consistently. This type of coordinated multi-method modification is a key capability that distinguishes delta generation from simple parameter tuning. Figure 9 illustrates a common failure: the LLM generates a syntactically valid diff that introduces a shape mismatch, causing a runtime error during the forward pass.

Compared to LLM-based NAS methods that generate complete code [3, 4, 13, 35], our approach is fundamentally more efficient: it generates only the difference between an existing working architecture and its improved variant, leveraging the structural knowledge embedded in LEMUR baselines [12] rather than regenerating it. 11

1 2 3 4 5 6 7 8 9 10 11 12

Moreover, the valid-generation-rate comparison (75.3% vs. 50.6%) reflects whether patches apply cleanly, which is largely format-driven rather than corpus-driven. Disentangling corpus diversity from the diff format via a singledataset ablation remains future work, but we note that the paradigm is designed to exploit corpus diversity, not merely benefit from it incidentally. Sample-size asymmetry in CIFAR-10 comparison. The full-generation baseline draws from 1,100 CIFAR-10specific candidates, whereas each delta LLM produces only 112–135 CIFAR-10 architectures (the remainder targeting five other datasets). Because the expected maximum of a sample grows with N , this asymmetry is conservative in our favour: a CIFAR-10-specialised delta run of 1,100 candidates would likely yield a higher best accuracy than the 85.5% we report. Mistral’s CIFAR-10 performance. Mistral-7B-Instruct (general-instruct, not code-specialised) achieves the highest CIFAR-10 best accuracy (85.5%) despite generating longer diffs. We hypothesise that its general instruction-following pre-training produces more diverse architectural proposals at sampling temperature τ = 0.35, effectively broadening exploration on harder datasets where the optimal architecture lies further from typical code-completion patterns. Its slightly longer outputs (∼50 lines vs. ∼30) may encode more substantial structural changes that benefit datasets requiring deeper modifications.

<delta> --- baseline.py +++ improved.py @@ -12,7 +12,7 @@ class Net(nn.Module): def __init__(self, ...): self.conv1 = nn.Conv2d(3, 32, 3) + self.conv1 = nn.Conv2d(3, 128, 5) self.fc = nn.Linear(32*30*30, 10) </delta> # Error: fc input size (32*30*30) no longer # matches conv1 output channels (128) and # kernel size (5). Runtime shape mismatch.

Figure 9. Failed delta: the LLM changes conv1 channels and kernel size but does not update the dependent fc layer, causing a tensor shape mismatch at runtime. This class of error accounts for roughly half of all validation failures.

Common Successful Patterns. Across all three LLMs, the most frequently successful deltas involve: (1) adding regularization layers (BatchNorm, Dropout), (2) adjusting channel dimensions to increase model capacity, and (3) replacing activation functions (e.g., ReLU with GELU or SiLU). These are precisely the types of targeted modifications that software engineers make when refining architectures, validating the delta paradigm’s alignment with real-world practice. Failure Modes. The delta-generation paradigm introduces a unique failure mode absent from full generation: context mismatch, where the diff’s context lines do not match the baseline code, causing patch application to fail. The remaining failures are semantic errors: the diff applies cleanly but produces architecturally broken code (shape mismatches, undefined variables). Delta-application failure rates are tightly clustered across all three LLMs— 24.7% (DeepSeek), 27.9% (Qwen), and 33.4% (Mistral)— suggesting that the difficulty of producing a well-formed unified diff is a paradigm-level property rather than an LLM-specific artifact. All three LLMs maintain stable valid rates across 22 cycles (DS 75.3%, Qwen 72.1%, Mistral 66.6%), with per-cycle ≥40% rates between 58–85% (DS 72–80%, Qwen 58–81%, Mistral 70–85%), demonstrating that iterative LoRA fine-tuning maintains high generation quality—analogous to how SWE-bench [22] agents learn to generate correct patches for software repositories. Training-corpus scope. A potential confound deserves explicit acknowledgment: the delta LLMs are fine-tuned on the full six-dataset LEMUR corpus, whereas the fullgeneration baseline [24] was trained on CIFAR-10-specific data. The richer corpus could, in principle, contribute to accuracy gains independently of the diff format. However, we argue this confound is inherent to the delta paradigm rather than separable: delta generation requires diverse, working baselines to modify—restricting to a single-dataset corpus would eliminate the paradigm’s core advantage of leveraging existing structural knowledge across domains.

9. Conclusion We introduced Delta-Based Neural Architecture Search, a paradigm in which an LLM generates compact unified diffs that refine an existing baseline architecture rather than synthesizing a complete model from scratch. Three implications emerge from our study. (i) The generation paradigm, not the LLM, drives the headline gains. Substituting the full-file output format for a diff-based format shifts the entire Pareto frontier on the LEMUR benchmark: with the same base LLM (DeepSeek-Coder-7B) and the same 22-cycle iterative protocol of Khalid et al. [24], moving to deltas raises the validgeneration rate outside the full-generation 95% confidence interval and shortens outputs by ≈85%. Adding Qwen2.5Coder and Mistral-7B-Instruct as independent base LLMs reproduces the effect, including with a general-purpose (non-code-specialized) backbone, which argues that the paradigm itself—not any single LLM family—is responsible for the improvement. (ii) Cross-dataset coverage is a first-class design dimension. Prior LLM-based NAS work has evaluated almost exclusively on CIFAR-10; the most recent updated state of the art [24] extends to three datasets. By operating on diffs, our pipeline preserves the working dataset-specific scaffolding of each LEMUR baseline and hence general12

correlation (ρ = 0.495, p = 0.10, N = 12) is not statistically significant, reflecting ceiling effects on its pureMNIST top-20 rather than proxy failure—the same phenomenon reported by Abdelfattah et al. [1] for zero-cost proxies restricted to top architectures. (b) The diff format introduces a context-mismatch failure mode absent from full generation; it applies to roughly one-third of generations across all three LLMs. (c) Output length is modestly LLM-dependent: Mistral-7B averages 49.5 lines (75% reduction) while DeepSeek and Qwen produce even tighter ∼30-line diffs (85%/84% reduction). The token-efficiency advantage of the delta paradigm holds for every LLM we tested, but the general-instruct Mistral backbone yields a slightly looser diff style than the two code-specialized models. (d) Accuracy thresholds and novelty thresholds are fixed at sensible values, not theoretically optimized. Future Work. Promising directions include: (1) tightening Mistral’s slightly looser diff style via prompt design or a compactness regularizer at fine-tuning time, to close the remaining ∼20-line gap with DeepSeek/Qwen; (2) extending the rank-correlation study to dataset-balanced topk lists (e.g., top-5 per dataset) to confirm that the strong proxy validity observed for Mistral (ρ = 0.926) generalizes to harder datasets such as CIFAR-100, where full-training gains are expected to be larger; (3) combining delta generation with the frozen-LLM feedback memory of Gu et al. [13]; (4) extending to tasks beyond image classification [11, 19, 42]; and (5) deploying the most token-efficient discovered architectures on edge devices [7].

izes to six datasets spanning 28–218 pixel inputs and 2– 100 classes without re-training the LLM per domain. On CIFAR-10—the only dataset shared with all prior work— our best delta-generated model clearly exceeds the fullgeneration state of the art, while simultaneously producing high-quality architectures on five additional datasets that full-generation pipelines have not been evaluated on. (iii) Token cost, not accuracy, is where delta generation wins asymptotically. The 75–85% output-length reduction compounds across 22 cycles and 1,100 candidates per LLM, yielding a ∼5.4× aggregate reduction in LLM output tokens over a full-generation run of the same size (4– 6.6× per-LLM). This is important for on-prem fine-tuning on a single consumer GPU, where LLM inference—not vision-model training—is the dominant cost. All accuracy claims in this paper are first-epoch validation accuracy, the proxy used in prior LLM-based NAS work within the NNGPT/LEMUR framework [13, 24]. To empirically validate this proxy, we fully trained the top-20 architectures per LLM for 50 epochs and computed Spearman rank correlation against the 1-epoch rankings (Appendix B.7). Mistral-7B achieves ρ = 0.926 (p < 0.001), strongly validating the proxy; Qwen yields ρ = 0.635 (p = 0.011), significant but attenuated by ceiling effects on its MNIST-dominated top-20. DeepSeek’s ρ = 0.495 (p = 0.10, N = 12) is not statistically significant— its pure-MNIST top-20 compresses full-training accuracy to a 0.7 p.p. range, leaving insufficient variance for rank discrimination rather than indicating proxy failure. The strong Mistral result—obtained on a dataset-diverse top20 (MNIST, CelebA, SVHN)—confirms that the 1-epoch proxy reliably preserves architecture rankings when sufficient accuracy spread exists. Limitations. (a) All reported accuracy numbers are first-epoch validation accuracy, the proxy used identically by Khalid et al. [24] and Gu et al. [13] and analogous to reduced-epoch proxies in the broader NAS literature [8, 54]. We adopt it deliberately for three reasons: (i) direct like-forlike comparability with the two SOTA baselines we benchmark against, both of which report 1-epoch proxies; (ii) isolation of the architectural signal—fixing the training budget ensures that accuracy differences reflect the generated architecture, not an inadvertent longer-training advantage; and (iii) the aggregate compute budget for the full threeLLM study (3,300 generations, 2,354 successfully applied and trained for one epoch each, plus LoRA fine-tuning) is already ∼90–100 GPU-hours, and scaling the architecture evaluations to the 20–50 epochs typically needed for convergence would inflate the budget by ∼20–50×, which is prohibitive at this scale. A rank-correlation study of the top-20 architectures per LLM (50 epochs) validates this proxy for Mistral (ρ = 0.926, p < 0.001) and Qwen (ρ = 0.635, p = 0.011); see Appendix B.7. DeepSeek’s

References [1] Mohamed S. Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D. Lane. Zero-cost proxies for lightweight NAS. In International Conference on Learning Representations (ICLR), 2021. 13, 18 [2] Nada Aboudeshish, Dmitry Ignatov, and Radu Timofte. AUGMENTGEST: Can random data cropping augmentation boost gesture recognition performance? arXiv preprint, arXiv:2506.07216, 2025. 4 [3] Zicheng Cai, Yaohua Tang, Yutao Lai, Hua Wang, Zhi Chen, and Hao Chen. SEKI: Self-evolution and knowledge inspiration based neural architecture search via large language models. arXiv preprint arXiv:2502.20422, 2025. 1, 2, 3, 11 [4] Angelica Chen, David Dohan, and David So. EvoPrompting: Language models for code-level neural architecture search. In Advances in Neural Information Processing Systems, 2023. 2, 3, 11 [5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. Evaluating large language models trained on code. arXiv preprint, 2021. 2 [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACLHLT, pages 4171–4186, 2019. 1

13

[20] Zipeng Ji, Guanghui Zhu, Chunfeng Yuan, and Yihua Huang. RZ-NAS: Enhancing LLM-guided neural architecture search via reflective zero-cost strategy. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. To appear. 2 [21] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. 5 [22] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWEbench: Can language models resolve real-world GitHub issues? International Conference on Learning Representations (ICLR), 2024. 2, 12 [23] Waleed Khalid, Dmitry Ignatov, and Radu Timofte. A retrieval-augmented generation approach to extracting algorithmic logic from neural networks. arXiv preprint, arXiv:2512.04329, 2025. 2 [24] Waleed Khalid, Dmitry Ignatov, and Radu Timofte. From memorization to creativity: LLM as a designer of novel neural-architectures. arXiv preprint, arXiv:2601.02997, 2026. Updated v7 (2026) adds CIFAR-100 and SVHN results with dataset-specific thresholds (20%/70%) and Wilson/t-based 95% confidence intervals. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18 [25] Roman Kochnev, Arash Torabi Goodarzi, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. Optuna vs code llama: Are LLMs a new paradigm for hyperparameter tuning? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5664– 5674, 2025. 2 [26] Roman Kochnev, Waleed Khalid, Tolgay Atinc Uzun, Xi Zhang, Yashkumar Sanjaybhai Dhameliya, Furui Qin, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Ignatov, and Radu Timofte. NNGPT: Rethinking AutoML with large language models. arXiv preprint, arXiv:2511.02033, 2025. 2, 3 [27] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 5 [28] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 5 [29] Guihong Li et al. Zero-shot neural architecture search: Challenges, solutions, and opportunities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4071– 4088, 2024. 2 [30] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019. 1, 2, 11 [31] Jiawei Liu et al. A survey on LLM-based agents for code generation. arXiv preprint arXiv:2402.02172, 2024. 2 [32] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings

[7] Saif U Din, Muhammad Ahsan Hussain, Mohsin Ikram, Dmitry Ignatov, and Radu Timofte. AI on the edge: An automated pipeline for PyTorch-to-Android deployment and benchmarking. Preprints, 2025. 13 [8] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI), pages 3460–3468, 2015. 13 [9] Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR), 2020. 1, 2 [10] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019. 1 [11] Mohamed Gado, Towhid Taliee, Muhammad Danish Memon, Dmitry Ignatov, and Radu Timofte. VIST-GPT: Ushering in the era of visual storytelling with LLMs? arXiv preprint, arXiv:2504.19267, 2025. 13 [12] Arash Torabi Goodarzi, Roman Kochnev, Waleed Khalid, Furui Qin, Tolgay Atinc Uzun, Yashkumar Sanjaybhai Dhameliya, Yash Kanubhai Kathiriya, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. LEMUR neural network dataset: Towards seamless AutoML. arXiv preprint, arXiv:2504.10552, 2025. 3, 11 [13] Xiaojie Gu, Dmitry Ignatov, and Radu Timofte. Resourceefficient iterative LLM-based NAS with feedback memory. arXiv preprint, arXiv:2603.12091, 2026. Concurrent arXiv preprint (2026); disclosed for transparency. 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 18 [14] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, et al. DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. 1, 5 [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 2 [16] Jeremy Howard. Imagenette: A smaller subset of 10 easily classified classes from ImageNet. https://github. com/fastai/imagenette, 2019. 5 [17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. 3 [18] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Fei Lei, Tianyu Liu, Jiajun Li, Kai Huang, Yichang Yang, Yang An, Ting Zhu, Bowen Yu, Chengyuan Hu, Jianhong Bai, Kai Bi, Jianqiang Gu, Jingren Wang, et al. Qwen2.5-Coder technical report. arXiv preprint, arXiv:2409.12186, 2024. 1, 5 [19] Krunal Jesani, Dmitry Ignatov, and Radu Timofte. LLM as a neural architect: Controlled generation of image captioning models under strict API contracts. arXiv preprint, arXiv:2512.14706, 2025. 13

14

[46] Tolgay Atinc Uzun, Dmitry Ignatov, and Radu Timofte. Closed-loop LLM discovery of non-standard channel priors in vision models. arXiv preprint, arXiv:2601.08517, 2026. 2 [47] Tolgay Atinc Uzun, Waleed Khalid, Saif U Din, Sai Revanth Mulukuledu, Akashdeep Singh, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Ignatov, Radu Timofte, et al. LEMUR 2: Unlocking neural network diversity for AI. arXiv preprint, 2026. 3 [48] Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Ignatov, and Radu Timofte. Enhancing LLM-based neural network generation: Few-shot prompting and efficient validation for automated architecture design. arXiv preprint, arXiv:2512.24120, 2025. 2 [49] Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. arXiv preprint arXiv:2301.08727, 2023. 1 [50] Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc., 22(158):209– 212, 1927. 5 [51] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models. arXiv preprint, 2023. 2 [52] Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. AutoML-GPT: Automatic machine learning with GPT. arXiv preprint arXiv:2305.02499, 2023. 2 [53] Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie. Can GPT-4 perform neural architecture search? arXiv preprint arXiv:2304.10970, 2023. 2, 11 [54] Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. EcoNAS: Finding proxies for economical neural architecture search. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), pages 11396–11404, 2020. 13, 18 [55] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017. 1, 2, 11

of the IEEE International Conference on Computer Vision (ICCV), pages 3730–3738, 2015. 5 [33] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J Shersby. Neural architecture search without training. International Conference on Machine Learning (ICML), pages 7588–7598, 2021. 2 [34] Yash Mittal, Dmitry Ignatov, and Radu Timofte. Preparation of fractal-inspired computational architectures for advanced large language model analysis. arXiv preprint, arXiv:2511.07329, 2025. 2 [35] Muhammad U Nasir, Sam Earle, Christopher Cleghorn, Steven James, and Julian Togelius. LLMatic: Neural architecture search via large language models and qualitydiversity optimization. arXiv preprint arXiv:2306.01102, 2023. Also in GECCO 2024. 1, 2, 3, 11 [36] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. 5 [37] Yameng Peng, Andy Huang, Zhixin Chen, Caiyang Wu, et al. SWAP-NAS: Sample-wise activation patterns for ultrafast NAS. In International Conference on Learning Representations (ICLR), 2024. 18 [38] Hieu Pham, Melody Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning (ICML), pages 4095–4104, 2018. 2, 11 [39] Md Hafizur Rahman, Zafaryab Haider, and Prabuddha Chakraborty. An automated multi parameter neural architecture discovery framework using ChatGPT in the backend. Scientific Reports, 15(16871), 2025. 2, 3, 11 [40] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4780–4789, 2019. 2 [41] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code LLama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. 1 [42] Bhavya Rupani, Dmitry Ignatov, and Radu Timofte. Exploring the collaboration between vision models and llms for enhanced image classification. Preprints, 2025. 13 [43] Usha Shrestha, Dmitry Ignatov, and Radu Timofte. From brute force to semantic insight: Performance-guided data transformation design with LLMs. arXiv preprint, 2026. 3 [44] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), pages 6105–6114, 2019. 2, 11 [45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

15

A. Additional Method Details

(try/except) is insufficient because CUDA device-side asserts permanently poison the GPU context. To prevent a single bad model from crashing the entire pipeline, we evaluate each candidate architecture in a separate Python subprocess. If a model causes a GPU crash, only the subprocess terminates; the main pipeline’s CUDA context remains intact. Evaluation arguments are serialized as JSON and results are returned via stdout, with a 600-second timeout to handle stalled models. This ensures robust evaluation across all 50 candidates per cycle.

A.1. MinHash/LSH Configuration for Novelty Filtering We employ MinHash-based near-duplicate detection [24] to ensure structural diversity in the training corpus. Tokenization and shingling. Generated Python/PyTorch source code is tokenized into character-level n-grams (shingles) of length n = 10. This granularity captures local structural patterns (e.g., layer definitions, activation functions) while remaining robust to minor formatting differences. MinHash signatures. Each architecture’s shingle set is compressed into a fixed-size MinHash signature using k = 256 independent hash permutations. The Jaccard similarity between any two architectures is then estimated by the fraction of matching positions in their respective signatures. Novelty threshold. A candidate architecture is considered novel if its estimated Jaccard similarity to all existing corpus members is below 1 − τnov = 0.10 (i.e., τnov = 0.90 dissimilarity). This ensures that only architectures differing in at least 90% of their shingle composition are admitted, preventing the corpus from collapsing onto near-duplicate designs.

B. Additional Results B.1. LoRA Training Loss Convergence Table 9 shows the LoRA training loss progression across epochs within a single fine-tuning cycle. The loss plateaus rapidly after epoch 2, supporting our choice of 3 LoRA epochs per cycle. Table 9. LoRA training loss convergence within a single cycle (DeepSeek-Coder-7B). Loss plateaus after epoch 2, with epochs 3–5 yielding negligible improvement. Epoch 1 2 3 4 5

A.2. Prompt Engineering for Delta Quality Rather than relying on post-hoc heuristic clamping to fix bad outputs, we guide generation quality directly through the prompt. The instructions explicitly demand compact deltas capped at 30 lines and provide sensible ranges for hyperparameters (e.g., batch sizes between 16 and 128, learning rates from 0.001 to 0.01). The prompt also enforces the required XML structure (<hp>, <tr>, <delta>) by providing clear formatting examples, and includes the baseline architecture code as context for targeted modifications. While the LLM generates these hyperparameter values to fulfill the requested XML structure, the actual evaluation strictly uses the baseline’s original configuration from the LEMUR database (Section 3.2) to ensure fair architectural comparison. By including the baseline code as context and providing clear formatting examples, the LLM learns to generate valid, targeted modifications without distorting its natural output distribution. This prompt-based approach is more principled than post-hoc clamping: the LLM learns to generate reasonable values through its training, and the prompt provides guardrails without distorting the generation distribution.

Training Loss

∆ from Epoch 1

0.122 0.116 0.114 0.114 0.114

— −0.006 −0.008 −0.008 −0.008

B.2. Per-Cycle Detailed Statistics Table 10 provides per-cycle statistics for the DeepSeekCoder-7B experiment under the balanced cluster protocol (22 cycles, 1,100 total candidates, 828 trained). Only representative cycles are shown due to space; the totals row covers all 22 cycles.

B.3. Qwen2.5-Coder-7B Per-Cycle Statistics Tables 12 and 13 present the per-cycle and per-dataset breakdowns for Qwen2.5-Coder-7B over 22 fine-tuning cycles (1,100 total candidates).

B.4. Mistral-7B-Instruct Per-Cycle Statistics Tables 14 and 15 present the per-cycle and per-dataset breakdowns for Mistral-7B-Instruct-v0.3 over 22 finetuning cycles (1,100 total candidates).

A.3. Subprocess Isolation for Robust Evaluation

B.5. Output Length Distribution

During evaluation of generated architectures, malformed models can trigger irrecoverable GPU errors (e.g., device-side assert triggered) that corrupt the CUDA context of the host process. Simple error handling

Generated LLM outputs range from 16 to 188 lines for DeepSeek-Coder-7B (mean 30.4, median 29, std 10.0), 3 to 175 lines for Qwen2.5-Coder-7B (mean 31.4, median 29, std 14.0), and 16 to 114 lines for Mistral-7B-Instruct 16

Table 10. Per-cycle statistics for DeepSeek-Coder-7B delta generation over 22 cycles (selected cycles shown) under the shared balanced six-dataset cluster protocol. Tr.: delta-applied + trained; Mean/Best: first-epoch accuracy over all trained models; ≥40: fraction of trained models with first-epoch accuracy ≥40%; Ln.: avg. LLM output length. Cyc.

Gen.

Tr.

Valid

Mean

Best

≥40

Ln.

0 1 4 6 9 10 12 15 17 19 20 21

50 50 50 50 50 50 50 50 50 50 50 50

35 44 43 35 42 35 37 42 39 36 37 35

70% 88% 86% 70% 84% 70% 74% 84% 78% 72% 74% 70%

60.4% 65.0% 64.8% 67.4% 66.9% 68.1% 67.5% 65.8% 67.2% 66.1% 68.5% 64.3%

99.5% 99.2% 99.5% 99.2% 99.2% 99.2% 99.2% 99.2% 99.2% 99.2% 99.2% 99.2%

74.3% 75.0% 76.7% 80.0% 76.2% 77.1% 78.4% 73.8% 74.4% 77.8% 75.7% 74.3%

33.2 31.7 33.2 32.5 29.8 30.1 29.4 28.7 29.2 29.0 31.9 30.4

Table 13. Per-dataset results for Qwen2.5-Coder-7B delta generation. Qwen2.5 successfully generates architectures across all six datasets with 793 models trained. CIFAR-10 direct comparison: 80.6% best vs. 64.0% for full generation [24]. Dataset

≥40

N

Mean

Best

Med.

MNIST 112 CelebA 148 SVHN 62 CIFAR-10 127 ImageNette 137 CIFAR-100 207

98.5% 88.5% 74.0% 64.5% 63.6% 27.2%

99.5% 98.1% 94.9% 80.6% 85.7% 50.8%

99.1% 112/112 93.6% 148/148 89.2% 47/62 59.6% 127/127 61.4% 133/137 31.5% 23/207

All

64.6%

99.5%

793

—

590/793

Table 14. Per-cycle statistics for Mistral-7B-Instruct delta generation (22 cycles, 1,100 total candidates). The valid rate trends upward, reaching a peak of 78% at A21. Mean accuracy stays within a narrow 62.5–72.0% band.

Total 1,100 828 75.3% 65.8% 99.5% 75.7% 30.4

Table 11. Per-dataset results for DeepSeek-Coder-7B delta generation under the shared balanced cluster protocol. DeepSeek successfully generates architectures across all six datasets with 828 models trained. CIFAR-10 direct comparison: 85.2% best vs. 63.98% for full generation [24]. N

Mean

Best

Med.

≥40

MNIST 132 CelebA 152 SVHN 53 CIFAR-10 135 ImageNette 166 CIFAR-100 190

98.5% 88.7% 78.4% 64.6% 60.7% 26.4%

99.5% 97.7% 94.9% 85.2% 85.7% 54.2%

98.9% 93.6% 93.6% 59.7% 61.4% 30.9%

132/132 152/152 44/53 135/135 149/166 15/190

All

65.8%

99.5%

—

627/828

Dataset

828

Gen.

Tr.

Valid

Mean

Best

≥40

0 1 2 5 8 10 15 16 19 21

50 50 50 50 50 50 50 50 50 50

28 49 46 38 36 38 33 24 31 28

56% 98% 92% 76% 72% 76% 66% 48% 62% 56%

64.2% 66.0% 65.7% 64.6% 69.1% 66.3% 66.0% 51.0% 66.9% 62.1%

99.3% 99.5% 99.5% 99.2% 99.2% 99.5% 99.5% 99.1% 99.2% 99.1%

75.0% 75.5% 73.9% 73.7% 80.6% 73.7% 75.8% 58.3% 80.6% 71.4%

Gen.

Tr.

Valid

Mean

Best

≥40

0 1 3 5 9 12 13 17 18 20 21

50 50 50 50 50 50 50 50 50 50 50

31 29 35 35 24 33 33 37 37 37 39

62% 58% 70% 70% 48% 66% 66% 74% 74% 74% 78%

64.7% 69.4% 68.4% 67.6% 72.0% 71.3% 69.8% 66.5% 66.0% 65.7% 62.5%

99.5% 99.2% 99.5% 99.5% 99.5% 99.2% 99.2% 99.2% 99.2% 99.2% 99.2%

77.4% 79.3% 77.1% 77.1% 83.3% 81.8% 84.8% 75.7% 75.7% 75.7% 71.8%

Total 1,100 733 66.6% 66.1% 99.5% 75.9%

Table 15. Per-dataset results for Mistral-7B-Instruct delta generation. Mistral successfully generates architectures across all six datasets with 733 models trained. CIFAR-10 direct comparison: 85.5% best vs. 64.0% for full generation [24]—the highest CIFAR-10 best accuracy of any LLM in our study.

Table 12. Per-cycle statistics for Qwen2.5-Coder-7B delta generation (22 cycles, 1,100 total candidates). The valid rate jumps from 56% to 98% after the first fine-tuning cycle and remains consistently above 48%. Cyc.

Cyc.

Dataset

≥40

N

Mean

Best

Med.

MNIST 106 CelebA 172 SVHN 43 CIFAR-10 112 ImageNette 114 CIFAR-100 186

98.6% 88.6% 84.5% 64.3% 61.9% 26.4%

99.5% 97.8% 94.9% 85.5% 85.7% 46.2%

98.9% 106/106 93.6% 172/172 93.6% 38/43 66.2% 112/112 61.4% 112/114 30.9% 16/186

All

66.1%

99.5%

733

—

556/733

diff style averaging ∼50 lines. Every LLM produces outputs 75–85% shorter than full-generation approaches [24], which typically exceed 200 lines.

Total 1,100 793 72.1% 64.6% 99.5% 74.4%

B.6. Accuracy Stability Across Cycles All three LLMs demonstrate remarkable accuracy stability across all 22 cycles. DeepSeek’s mean accuracy varies in a narrow 60.4–68.5% band (overall 65.8%) with ≥40% rate 72.2–80.0%. Qwen’s mean accuracy varies between 51.0% and 69.1% (overall 64.6%), with the ≥40% rate consistently between 58% and 81%. Mistral’s mean accuracy varies in an even narrower band of 62.5–72.0% (over-

(mean 49.5, median 50, std 16.6, with 4 long-output outliers >200 lines excluded from the aggregate). All three distributions are tight and concentrated in the 25–55 line range: DeepSeek and Qwen outputs cluster near 29–33 lines per cycle (Table 10), while Mistral’s general-instruct (non-code-specialized) pre-training yields a slightly looser 17

all 66.1%), with ≥40% rate between 70.0% and 84.8%. This stability across three LLM families indicates that iterative LoRA fine-tuning maintains—rather than degrades— generation quality over extended runs. Qwen’s only notable dip occurs at cycle 16 (51.0% mean, 48% valid rate), potentially reflecting transient overfitting, after which performance recovers (66.9% at cycle 19); DeepSeek and Mistral show no comparable dip.

datasets the improvement is larger: CelebA models gain +11.8 p.p. on average (e.g., 73.3%→92.7%), while the single SVHN model gains +6.7 p.p. (79.1%→85.8%). This confirms that single-epoch accuracy is a conservative lower bound and that delta-generated architectures retain substantial learning capacity beyond the proxy evaluation. Failure analysis. Of the 60 training runs attempted (20×3 LLMs), 17 failed: 9 timeouts (complex architectures exceeding the 8-hour budget), 3 data-access errors (CelebA downloads from Google Drive), 3 Google Drive rate limits (Qwen CelebA models), and 2 code-level name errors (DeepSeek). Of these, 15 are infrastructure-level failures (timeouts, file access, rate limits) unrelated to architectural quality, and 2 are code-quality issues (undefined variable names in DeepSeek architectures)—indicating that the novelty filter does not guarantee robustness under extended training. All 43 completed models trained successfully to convergence. Compute cost. The full 50-epoch study consumed approximately 110 GPU-hours across the three LLMs (Mistral ∼50 hr, DeepSeek ∼35 hr, Qwen ∼25 hr) on RTX 4090 GPUs, dominated by CelebA models (∼7 hr each). This confirms that scaling to multi-epoch evaluation for the full 2,354-model pool would require ∼5,000–10,000 GPUhours—justifying the use of the 1-epoch proxy for the main study. Context within the NAS proxy literature. Spearman ρ is the standard metric for proxy validation in NAS. On constrained cell-based search spaces, the best zero-cost proxies achieve ρ ≈ 0.82–0.90 on NAS-Bench-201 [1, 37], while reduced-epoch training proxies reach ρ ≈ 0.61 [54]. Notably, Abdelfattah et al. [1] report that even the strongest proxy (SynFlow) drops to ρ = 0.18 when restricted to the top-10% of architectures on CIFAR-10—the same ceiling effect observed in our DeepSeek and Qwen top-20 subsets. In the LLM-based NAS space, neither Khalid et al. [24] nor Gu et al. [13] report a proxy-to-full-training correlation; both explicitly identify this validation as future work. Our Mistral result (ρ = 0.926) is, to our knowledge, the first empirical proxy validation in LLM-based NAS and places the 1-epoch signal at the upper end of the NAS proxy spectrum, despite operating in an unconstrained open-code search space rather than a fixed cell topology.

B.7. Proxy Validation: 1-Epoch vs. 50-Epoch Rank Correlation To empirically assess whether first-epoch accuracy reliably predicts fully-trained performance rankings, we selected the top-20 architectures per LLM (ranked by 1-epoch proxy accuracy) and trained each for 50 epochs using the original LEMUR hyperparameters. We then computed Spearman ρ and Kendall τ rank correlations between the proxy ranking and the 50-epoch ranking. Models that failed due to dataaccess errors or runtime timeouts were excluded from the correlation computation (failures are infrastructure-level, not architectural). Table 16. Rank correlation between 1-epoch proxy accuracy and 50-epoch fully-trained accuracy for the top-20 delta-generated architectures per LLM. pρ : two-tailed p-value for the Spearman test. Mistral (p < 0.001) and Qwen (p = 0.011) are significant at α = 0.05; DeepSeek (p = 0.10, N = 12) is not, due to ceiling effects on its pure-MNIST top-20. LLM

ρ

p

τ

N

Datasets

Med. 1→50ep

Mistral-7B .926 <.001 .733 16/20 MN/CA/SV 95.2→98.9% Qwen2.5-7B .635 .011 .486 15/20 MN/CA 96.9→99.1% DeepSeek-7B .495 .102 .318 12/20 MN only 96.6→99.1% MN=MNIST, CA=CelebA, SV=SVHN. p: two-tailed Spearman test.

Interpretation. Mistral-7B achieves the strongest correlation (ρ = 0.926, p < 0.001) because its top-20 spans three datasets (9 MNIST, 10 CelebA, 1 SVHN), creating a wide proxy-accuracy range (73.3–98.2%) and correspondingly large full-training spread (85.8–99.5%). This diversity provides a clear ranking signal. Qwen’s correlation (ρ = 0.635, p = 0.011) is significant at α = 0.05 and consistent with an intermediate ceiling effect (85% MNIST). DeepSeek’s correlation (ρ = 0.495, p = 0.10; Kendall τ = 0.318, p = 0.14) is not statistically significant at conventional levels (N = 12): with all 20 top architectures on MNIST, full-training accuracy compresses to a 0.7 p.p. range (98.7–99.3%), leaving insufficient variance for rank discrimination. We therefore characterise the DeepSeek result as inconclusive rather than as evidence for or against the proxy. Accuracy gains from full training. Across all three LLMs, 50-epoch training raises median accuracy by 2– 4 percentage points (Mistral: 95.2%→98.9%; Qwen: 96.9%→99.1%; DeepSeek: 96.6%→99.1%). On harder

C. Reproducibility We report the full experimental environment below to enable independent reproduction of every number in this paper. Hardware. All three LLMs (DeepSeek-Coder-7B, Qwen2.5-Coder-7B, Mistral-7B-Instruct) were fine-tuned and evaluated on the same shared compute cluster with NVIDIA RTX 4090 24 GB GPUs under SLURM, Rocky Linux 9. Vision-model evaluation uses one GPU per sub18

process in all three runs. Software. Python 3.11, PyTorch 2.3 (CUDA 12.1), HuggingFace transformers 4.44, peft 0.12 (LoRA), trl 0.9, accelerate 0.33, bitsandbytes 0.43, datasketch 1.6 (MinHash/LSH). Unified-diff patching uses the GNU patch utility. Subprocess evaluation uses the standard subprocess module with a 600 s timeout per candidate. Random seeds. Global seed 42 for PyTorch, NumPy, and Python random, propagated to all dataloaders, the LoRA trainer, and the LLM sampler. Baseline sampling from the LEMUR database uses a per-cycle seed derived from (global seed, cycle index) to ensure each cycle draws deterministically while still exploring different baselines. Fine-tuning configuration. LoRA rank r=32, α=32, dropout 0.05, target modules {q, k, v, o, up, down, gate, lm head}, bfloat16 precision, learning rate 1×10−5 (cosine decay), 20 warmup steps, weight decay 0.01, per-device batch 1, gradient accumulation 8 (effective batch 8), 3 finetuning epochs per cycle. Inference: temperature 0.35, topk=50, top-p=0.9, max new tokens 1,024. These values are identical across all three LLMs. Wall-clock budget. Per 50-candidate cycle: ≈15–20 minutes LLM fine-tuning + ≈60–90 minutes vision-model evaluation (dominated by CIFAR-100 training time). Total for 22 cycles: ≈28–35 GPU-hours per LLM on the respective hardware, or ≈90–100 GPU-hours for the full threeLLM study. Artifacts. All code, models, and adapters are publicly released: • CV models (197 novel architectures, del- prefix in LEMUR): https : / / github . com / ABrain - One / nn dataset/pull/204 • Fine-tuned LLMs (LoRA merged into base weights, 22 cycles each): https : / / huggingface . co / ABrain / Delta NAS-DeepSeek-Coder-7B https : / / huggingface . co / ABrain / Delta NAS-Qwen2.5-Coder-7B https : / / huggingface . co / ABrain / Delta NAS-Mistral-7B • Code (scripts, prompts, evaluator): https://github.com/ABrain-One/nn-gpt JSON configs (ds coder 7b instruct.json, etc.) reproduce the reported numbers.

19