Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search Yashkumar R Lukhi*, Harsh Rameshbhai Moradiya,
Radu Timofte, Dmitry Ignatov
arXiv:2606.23739v1 [cs.LG] 21 Jun 2026
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Abstract
convolutional and transformer-based MoE designs [23, 27], suggesting that routing inputs through structurally diverse expert networks provides complementary inductive biases that a single backbone cannot capture. The LEMUR neural network dataset [8, 26] provides a structured collection of trained neural network models designed for benchmarking, AutoML research, and LLMbased architecture generation [15, 16]. While prior work within LEMUR has manually designed and evaluated a small number of MoE configurations [20], the space of possible heterogeneous expert combinations across LEMUR’s 29 base architecture families is combinatorially large— 29 = 23,751 possible 4-family quartets—and cannot be 4 explored through hand design alone. This work addresses this gap with a fully automated MoE4 search pipeline built on three contributions. 1. Deterministic code-assembly generator. AlterHeteroMoE4.py programmatically assembles any combination of four base model families from the LEMUR database into a syntactically valid forward-pass-verified MoE4 model, reusing the HeterogeneousGate design, mixup enhancement, and cosine-annealed AdamW training from the reference model [15], without requiring any LLM call in the generation loop. 2. Multi-stage validation pipeline. Each candidate model passes three sequential gates before GPU evaluation: (i) Python compile() syntax check, (ii) CPU forwardpass probe at the actual training resolution (256×256), and (iii) database deduplication via MD5 checksum. This pipeline reduced wasted GPU evaluations substantially and prevented broken or duplicate models from entering the LEMUR database. 3. Campaign automation with fault tolerance. CampaignMoE4.py orchestrates repeated generateevaluate cycles with persistent JSON state, per-batch isolation, and SSH-safe resume semantics, enabling a 28-day unattended campaign targeting 1,000 successful model evaluations. Beyond engineering contributions, the campaign produced a notable scientific finding: deterministic enumera-
We present an automated large-scale search pipeline for heterogeneous 4-Expert Mixture-of-Experts (MoE4) architectures within the LEMUR neural network dataset ecosystem. Building on a hand-crafted heterogeneous MoE reference model, we replace manual design with a deterministic code-assembly generator that systematically combines base architecture families drawn from the LEMUR database into MoE4 ensembles, each governed by a convolutional gating network with temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling. Over a 28-day campaign on an NVIDIA RTX 4090, the pipeline generated 4,463 candidate models across 197 batches, of which 1,021 were evaluated successfully. A critical finding emerged from the campaign: due to alphabetical enumeration via itertools.combinations, the entire explored search space (4.8% of the theoretical 23,751 possible 4-family combinations) is anchored to a single family, AirNet. We characterise this coverage bias precisely, identify the root cause in the generator, and propose a stratified random sampling fix. Within the AirNetanchored scope, ShuffleNet and MobileNetV3 consistently co-produce the highest-accuracy ensembles (mean accuracy up to 0.632), while FractalNet and MNASNet are identified as low-yield families warranting exclusion in future campaigns. The pipeline, analysis artefacts, and corrected generator are released as part of the open-source NNGPT project at https://github.com/ABrain-One/nn-gpt.
1. Introduction Mixture-of-Experts (MoE) architectures have gained renewed attention following the success of large-scale models such as DeepSeek-V2 [5], which demonstrated that sparse expert activation can match or exceed dense model performance at substantially reduced per-token compute. In vision, analogous benefits have been observed across both * Corresponding authors: {yashkumar-rajeshbhai.lukhi,harshrameshbhai.moradiya}@stud-mail.uni-wuerzburg.de
1
tion via sorted itertools.combinations caused the entire explored search space to be anchored to AirNet—the first family alphabetically—covering only 4.8% of the full combination space and zero non-AirNet combinations. We characterise this coverage bias, identify its root cause precisely, and propose a stratified random sampling strategy to eliminate it in future runs. This finding is broadly relevant to any large-scale neural architecture search campaign that uses sorted deterministic enumeration with a throughputbased stop condition. Within the AirNet-anchored scope, we conduct a familylevel accuracy analysis across 1,021 successful models, identifying ShuffleNet and MobileNetV3 as high-yield expert families (mean accuracy 0.632 and 0.621, respectively) and FractalNet and MNASNet as families that consistently degrade ensemble quality or fail to train within available GPU memory. The best single model—AirNet + AlexNet + DPN68 + ResNet—achieves 68.0% Top-1 accuracy on CIFAR-10 in a single training epoch, demonstrating that automated assembly of heterogeneous experts can produce competitive candidates without any manual tuning. Building on the growing body of work applying LLMs to neural architecture tasks [13, 15] and on the accumulated architectural synthesis experience within the NNGPT framework [2, 3, 7, 10, 11, 14, 20, 24, 25], we construct our MoE4 search pipeline atop the NNGPT framework [16], drawing on the LEMUR dataset’s diverse collection of high-capacity and edge-optimised model families [6, 8, 12, 17, 26] as the expert pool. The remainder of this paper is organised as follows. Section 2 reviews related work on MoE architectures and neural architecture search. Section 3 describes the pipeline design and implementation. Section 4 presents experimental setup and campaign statistics. Section 5 reports accuracy results and family-level analysis. Section 6 concludes with lessons learned and directions for future work.
ViTs on ImageNet at comparable cost. Soft MoE formulations [21] simplify training by replacing hard routing with differentiable token-to-expert assignment. Heterogeneous MoE designs—where experts differ in architecture rather than just weights—remain less explored. Abbas and Andreopoulos [1] used experts of varying complexity for adaptive inference under resource constraints, and Ahmed et al. [4] combined different CNN families in an MoE setting, demonstrating the viability of architectural diversity among experts. Our work extends this direction by systematically assembling heterogeneous 4-expert MoE models from 29 distinct base architecture families at scale.
2.2. Neural Architecture Search Neural Architecture Search (NAS) automates the discovery of high-performing network designs, traditionally through reinforcement learning [29], evolutionary algorithms [22], or differentiable search [19]. Brute-force and random search strategies have been shown to be competitive baselines in constrained search spaces [18], and are particularly tractable when individual candidate evaluations are cheap. Our campaign adopts a deterministic combinatorial enumeration strategy analogous to brute-force NAS, where the search space is the set of all 29 = 23,751 4-family ex4 pert combinations, and the stop condition is a throughput target rather than a performance threshold. A key finding of our work—that sorted enumeration induces severe coverage bias—is directly relevant to any NAS campaign using deterministic iteration with an early stop condition.
2.3. Automated Model Generation and Dataset Ecosystems The LEMUR neural network dataset [8, 26] provides a curated collection of trained models for benchmarking and AutoML research, with standardized interfaces for training, evaluation, and database integration. The nn-gpt project [16] extends LEMUR with LLM-driven architecture generation, using the dataset as both a source of expert building blocks and a target repository for newly discovered models. Prior work within nn-gpt used LLM-based generation for 2-expert MoE assembly, achieving limited success (0/36 attempts) before switching to deterministic code assembly [20]. Our pipeline inherits this deterministic assembly approach, extends it to 4-expert configurations, and adds a multi-stage validation layer and campaign automation that were absent in prior nn-gpt generators.
2. Related Work 2.1. MoE Architectures in Vision Mixture-of-Experts models route inputs through a subset of specialized expert networks via a learned gating mechanism, enabling conditional computation and improved model capacity without proportional increases in inference cost. Early CNN-based MoE work demonstrated that routing inputs to structurally distinct expert branches improves classification performance [4, 9]. Layer-level MoE variants such as CondConv [28] condition convolutional filters on input data, while DeepMoE [27] integrates sparsely activated expert layers into ResNet blocks, yielding 3–4% gains on CIFAR-100 without additional compute. In Vision Transformers, V-MoE [23] applies sparse expert selection to transformer MLP blocks and outperforms dense
3. Methodology 3.1. Overview The MoE4 pipeline consists of four sequential stages: (i) expert pool construction from the LEMUR database, (ii) deterministic code assembly of candidate MoE4 mod2
gradient clipping (max-norm = 1.0), a linear warm-up over 5 epochs, and cosine annealing with Tmax = 50.
els, (iii) multi-stage validation before GPU evaluation, and (iv) campaign automation for fault-tolerant long-running execution. All models share a common architectural template derived from the hand-crafted reference model MoE-hetero4-Alex-Dense-Air-Bag, which established the gating design and training configuration used throughout this work.
3.4. Multi-Stage Validation Each assembled model passes three sequential gates before being written to disk and submitted for GPU evaluation: (1) Syntax check. Python’s built-in compile() verifies syntactic correctness. This is fast (<1 ms) and catches import errors or assembly mistakes. (2) CPU forward-pass probe. The model is instantiated on CPU and a dummy batch of shape (2, 3, 256, 256) — matching the default norm 256 flip evaluation transform — is passed through. The output shape is asserted to be (2, 10). This step, taking 50–500 ms per model, catches resolution-dependent dimension mismatches (e.g. spatial collapse in large-kernel experts) that compile() cannot detect. The probe resolution was deliberately matched to the training resolution after an earlier mismatch caused 3 out of 5 initial models to fail during GPU evaluation despite passing a 32×32 probe. (3) Database deduplication. An MD5 checksum of the whitespace-stripped source code is computed via uuid4() and checked against all existing entries in the LEMUR database. Duplicate models are skipped silently, preventing wasted GPU evaluations and database corruption from repeated runs.
3.2. Expert Pool Construction At runtime, AlterHeteroMoE4.py queries the LEMUR database via ab.nn.api.data() to retrieve all base architecture models trained on CIFAR-10 image classification. Models are filtered to retain only established base architectures: UUID-variant models (LLM-generated mutations identified by hexadecimal suffixes), MoE-family models (to avoid circular composition), and lowercaseprefixed generated models (ga-*, ga-mut-*) are all excluded. This yields a pool of 29 base architecture families, including AlexNet, AirNet, ResNet, ShuffleNet, MobileNetV3, DenseNet, FractalNet, and others. The combination space over this pool is 29 4 = 23,751 possible 4-family quartets.
3.3. Code Assembly For each 4-family combination (F1 , F2 , F3 , F4 ), the generator assembles a complete, self-contained PyTorch model file through the following steps: (i) each expert’s source code is retrieved from the database and transformed via transform expert(), which renames class Net to class {Family}Expert, strips import statements and supported hyperparameters(), and applies name sanitisation for hyphenated family names (e.g. InceptionV3-1 → InceptionV3 1Expert); (ii) imports from all four experts are merged and deduplicated; (iii) all prm[’key’] references across experts are collected to build a dynamic supported hyperparameters() function; (iv) the MOE4 WRAPPER template is instantiated with the four expert class names and hyperparameter defaults. The wrapper template implements the HeterogeneousGate gating network (Conv–BN– ReLU–Conv–BN–ReLU–AvgPool–Linear–ReLU– Dropout–Linear) with a learned temperature parameter clamped to [0.5, 5.0] and training-time Gaussian noise on gate logits. Expert outputs are combined as: y=
4 X
3.5. Campaign Automation CampaignMoE4.py orchestrates the full generateevaluate loop targeting a configurable number of successful model evaluations. Each iteration generates a fixed-size batch of new models, evaluates them via NNEval.main() using keyword arguments (bypassing a pre-existing CLI argument ordering bug in NNEval.py), and scans the resulting batch directory for eval info.json (success) and error.txt (failure) artefacts. Campaign state — including cumulative success count, batch index, and per-batch metadata — is persisted to campaign state.json after each batch, enabling transparent resume after SSH disconnects or process interruptions without duplicating completed work. Each batch uses a timestamped unique prefix for database entries, avoiding naming collisions across runs.
4. Experiments 4.1. Dataset and Task
gi (x) · fi (x),
(1)
All experiments were conducted on CIFAR-10, a 10-class image classification benchmark with 50,000 training and 10,000 test images across 10 object categories. Images were preprocessed using the norm 256 flip transform, which resizes inputs to 256×256 and applies per-channel normalisation and random horizontal flipping. Mixup augmentation
i=1
where gi (x) are temperature-scaled softmax gate weights and fi (x) are per-expert output logits. Training uses AdamW with differential learning rates for experts and gate, mixup augmentation (α = 0.2), label smoothing (ϵ = 0.1), 3
Table 1. MoE4 campaign summary statistics.
(α = 0.2) was applied at the MoE4 wrapper level during training.
Metric
4.2. Evaluation Protocol
Total batches completed Total models generated Models evaluated successfully Campaign target Target achieved Models failed (CUDA OOM) Models failed (other) Models pending Blacklisted checksums Campaign duration
Each candidate model was trained for 1 epoch with batch size 32 and learning rate 0.01 on an NVIDIA GeForce RTX 4090 (24 GB VRAM). A single epoch was chosen to maximise throughput across the large candidate pool; the goal of the campaign was not to find a fully converged model but to identify high-potential expert combinations for downstream fine-tuning and database integration. Evaluation was performed by the existing NNEval pipeline, which trains each model, records top-1 accuracy in eval info.json on success, or writes a categorised error.txt on failure. Successful models were automatically registered in the LEMUR database via copy to lemur(). We report top-1 accuracy as the primary metric.
Value 197 4,463 1,021 1,000 ✓ (+2.1%) 3,420 (99.7%) 9 (0.3%) 13 102 28 days
combination is subject to three pre-evaluation filters: class name collision detection (skips combinations where two experts define identically named helper classes), forward-pass validation, and database deduplication. Table 1 summarises the overall campaign statistics.
5. Results and Discussion
4.3. Campaign Configuration
5.1. Campaign Overview
The campaign ran 28 days, organised into batches of 25 models each. Key hyperparameters were fixed across all models: AdamW optimiser, lrexperts = 10−3 , lrgate = 5×10−4 , weight decay 10−2 , label smoothing ϵ = 0.1, gradient clipping max-norm = 1.0, linear warm-up over 5 epochs, and cosine annealing with Tmax = 50. A parameter threshold of 250M was applied mid-campaign (from batch 82 onward) to proactively skip combinations likely to exceed GPU memory. Known-failure checksums were blacklisted (102 entries) to avoid re-evaluating previously failed combinations.
The campaign exceeded its delivery target, producing 1,021 successfully evaluated models across 197 batches (Table 1). Early batches (1–5) achieved near-perfect success rates (84–100%), after which success rates degraded sharply as the generator moved into combinations involving larger expert families. The median per-batch delta-success is 0, meaning more than half of all batches produced zero new successful models, confirming that the failure pattern is structural rather than random.
5.2. Failure Analysis
4.4. Baselines We compare MoE4 ensemble results against the individual base model families used as experts, as reported in the LEMUR database. The best-performing individual families on CIFAR-10 serve as single-model reference points: ShuffleNet, MobileNetV3, and ResNet each achieve strong single-model accuracy and are among the most frequently appearing experts in high-accuracy MoE4 combinations. We do not compare against the previous semester’s manually designed MoE-hetero4-Alex-Dense-Air-Bag model directly, as that model was trained for 200 epochs whereas all campaign models are evaluated at 1 epoch; the comparison would not be meaningful under these conditions.
CUDA out-of-memory errors (OOM) account for 99.7% of all 3,429 failures. The RTX 4090’s 24 GB VRAM is insufficient for many 4-expert combinations at 256×256 resolution with batch size 32, particularly those involving large families such as DenseNet, DPN, or VisionTransformer. The 250M-parameter threshold filter introduced at batch 82 reduced the OOM rate marginally but could not eliminate it, since parameter count alone does not predict peak activation memory during training. This confirms that OOM is a structural consequence of combining four independently parameterised full networks at high resolution, not an incidental failure mode. A resolution reduction or per-expert adaptive pooling layer would be the most effective mitigations for future campaigns.
4.5. Search Space Analysis
5.3. Critical Finding: Search Space Coverage Bias
The 29
theoretical search space contains A critical finding emerged from post-campaign analysis = 23,751 distinct 4-family combina(Table 2). The generator uses: 4 tions. The generator enumerates these via itertools.combinations(sorted(model names), itertools.combinations(sorted(model names), 4) 4), producing combinations in lexicographic order. Each (2) 4
Table 2. Search space coverage statistics. Metric
Value
Architecture families in pool Theoretical combinations 29 4 Combinations attempted Search space coverage Combinations containing AirNet Combinations not containing AirNet
29 23,751 1,146 4.8% 1,146 0
Table 3. Accuracy statistics across 1,021 successful MoE4 models (AirNet-anchored, 1 training epoch). Metric
Value
Mean accuracy Median accuracy 25th percentile 75th percentile Maximum accuracy Minimum accuracy
0.5221 0.5388 0.4891 0.6059 0.6801 0.1250
Figure 1. Accuracy distribution across 1,021 successful MoE4 models (1 training epoch, batch=32, lr=0.01). Red line: mean (0.5221), dashed orange: median (0.5388), dotted green: maximum (0.6801). Table 5. Expert family accuracy profile (families with ≥40 appearances in successful models).
Table 4. Top-5 MoE4 models by CIFAR-10 Top-1 accuracy. Acc.
Expert Combination
0.6801 0.6795 0.6794 0.6756 0.6738
AirNet + AlexNet + DPN68 + ResNet AirNet + AirNext + Diffuser + ShuffleNet AirNet + BagNet + DenseNet + MobileNetV3 AirNet + BagNet + MaxVit + MobileNetV3 AirNet + DPN107 + GoogLeNet + ShuffleNet
sorted() places AirNet first alphabetically. itertools.combinations then generates all 28 = 3,276 AirNet-anchored combinations before 3 advancing to the next anchor family. The campaign stopped at 1,146 combinations — still entirely within the AirNet-anchored slice — having never reached any combination that does not include AirNet. As a result, the entire explored space covers only 4.8% of the theoretical search space, and all results are conditioned on AirNet being present as one of the four experts.
Family
Mean Acc
Success Rate
Apps.
ShuffleNet MobileNetV3 MobileNetV2 InceptionV3-1 ResNet EfficientNet DenseNet SwinTransformer FractalNet MNASNet
0.632 0.621 0.586 0.582 0.578 0.556 0.555 0.493 — 0.218
96.7% 96.4% — 27.2% 96.8% — — — 0.7% 70.1%
92 84 72 320 93 81 79 91 3,273 134
5.4. Accuracy Distribution Across the 1,021 successful models, top-1 accuracy ranges from 0.125 to 0.6801 with a mean of 0.5221 and median of 0.5388 (Table 3, Fig. 1). The distribution is left-skewed, with a sharp low-accuracy cluster corresponding to combinations containing MNASNet (mean accuracy 0.218) and a broad high-accuracy region above 0.55. The best model — AirNet + AlexNet + DPN68 + ResNet — achieves 68.0% top-1 accuracy in a single training epoch (Table 4). All top-5 models share ShuffleNet or MobileNetV3 as a coexpert, consistent with the family-level analysis below.
This finding is not a failure but a precise diagnosis of a coverage blind spot common in large-scale combinatorial search campaigns that use sorted deterministic enumeration with a throughput-based stop condition. The proposed fix is straightforward:
5.5. Family-Level Analysis Table 5 and Fig. 2 show the family-level accuracy profile across successful models. Four key observations emerge: ShuffleNet and MobileNetV3 are the strongest expert families, achieving mean accuracies of 0.632 and 0.621 respectively with near-perfect success rates (>96%). Their lightweight architectures fit comfortably within GPU memory even when combined with larger experts, and their strong feature representations appear highly complementary to AirNet. Future campaigns should seed combinations preferentially with these families. FractalNet is a practical dead end. Despite being the second most frequently attempted family (3,273 appear-
combos = list(itertools.combinations( model names, 4)) random.shuffle(combos) (3) Under uniform random sampling, AirNet would appear in approximately 4/29 ≈ 13.8% of combinations rather than 100%, enabling unbiased exploration of the full search space. 5
6. Conclusion This work presented an automated pipeline for large-scale systematic exploration of heterogeneous 4-Expert Mixtureof-Experts architectures within the LEMUR neural network dataset ecosystem. Over a 28-day campaign on an NVIDIA RTX 4090, the pipeline generated and evaluated 1,021 successful MoE4 models across 197 batches, exceeding the delivery target of 1,000 models and providing a substantial new collection of expert-combination candidates for the LEMUR database. Figure 2. Mean and median accuracy per expert family in successful MoE4 models (families with ≥40 appearances; bars = mean, diamonds = median, whiskers = ±1 SD).
Three engineering contributions enabled this scale: a deterministic code-assembly generator that produces syntactically valid, forward-pass verified MoE4 models from any combination of LEMUR base architecture families; a multistage validation pipeline (syntax check, CPU forward-pass probe at training resolution, database deduplication) that eliminated wasted GPU evaluations from broken or duplicate candidates; and a fault-tolerant campaign orchestrator with persistent state and SSH-safe resume semantics that sustained 28 days of unattended execution.
ances across generated models), FractalNet achieved a success rate of only 0.7% (23 successful evaluations). Its recursive architecture causes frequent OOM failures and shape mismatches at 256×256 resolution. Excluding FractalNet from the expert pool would recover a substantial fraction of wasted compute. MNASNet is a consistent accuracy degrader. Combinations containing MNASNet produced a mean accuracy of 0.218 — above the random baseline of 0.10 for 10-class classification, but far below the campaign median of 0.539. This suggests that MNASNet’s mobile-optimised architecture, designed for latency rather than accuracy, does not contribute useful features in a 4-expert ensemble context. High success rate does not imply high accuracy. SqueezeNet achieves a 96.8% success rate but only 0.514 mean accuracy, whereas ResNet achieves the same success rate with 0.578 mean accuracy. ResNet is therefore the most efficient family in this campaign — reliable to train and high in accuracy — making it a strong default inclusion for future runs.
The campaign’s most significant finding is a search space coverage bias induced by sorted deterministic enumeration: all 1,146 explored combinations were anchored to AirNet, covering only 4.8% of the 23,751 possible 4family quartets and leaving 95.2% of the search space entirely unexplored. This bias is not specific to our pipeline — it is a general risk in any large-scale NAS campaign that combines sorted iteration with a throughput-based stop condition. The proposed fix, stratified random shuffling of the combination list before iteration, is a one-line code change that eliminates the bias entirely. Within the AirNet-anchored scope, family-level analysis across 1,021 models identified ShuffleNet and MobileNetV3 as the highest-yield expert families (mean accuracies 0.632 and 0.621, respectively, with >96% training success rates), and FractalNet and MNASNet as families that consistently waste computation or degrade ensemble quality. The best single combination — AirNet + AlexNet + DPN68 + ResNet — achieved 68.0% Top-1 accuracy in CIFAR-10 in a single training epoch, demonstrating that automated heterogeneous expert assembly can produce competitive models without any manual architecture design.
5.6. General Discussion All results in this section are valid within a well-defined scope: AirNet-anchored MoE4 ensembles on CIFAR-10, evaluated after 1 training epoch. Claims about family quality (e.g. ShuffleNet outperforms SwinTransformer) cannot be fully decoupled from AirNet’s mandatory co-presence until the coverage bias is corrected in a future campaign. Nevertheless, the consistency of family rankings across hundreds of independent combinations — each with different co-experts — provides strong evidence that the observed patterns are genuine and not artefacts of a specific pairing. The proposed stratified sampling fix and expert pool pruning (remove FractalNet and MNASNet) together constitute a concrete, low-effort improvement that should substantially increase the quality and coverage of the next campaign.
Future work will apply the stratified sampling fix and pruned expert pool to conduct an unbiased campaign across the full combination space, extend evaluation beyond single-epoch screening to identify genuinely highperforming ensembles, and investigate memory-aware prefiltering to reduce the dominant CUDA OOM failure rate. 6
References
[14] Waleed Khalid, Dmitry Ignatov, and Radu Timofte. From memorization to creativity: LLM as a designer of novel neural architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3252–3261, 2026. [15] Roman Kochnev, Arash Torabi Goodarzi, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. Optuna vs code llama: Are LLMs a new paradigm for hyperparameter tuning? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5664– 5674, 2025. [16] Roman Kochnev, Waleed Khalid, Tolgay Atinc Uzun, Xi Zhang, Yashkumar Sanjaybhai Dhameliya, Furui Qin, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Ignatov, and Radu Timofte. NNGPT: Rethinking AutoML with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 5664–5674, 2026. [17] Arun Kumar, Aswathy Baiju, Radu Timofte, and Dmitry Ignatov. MobileAgeNet: Lightweight facial age estimation for mobile deployment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3810–3818, 2026. [18] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pages 367–377, 2020. [19] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. [20] Yash Mittal, Dmitry Ignatov, and Radu Timofte. Preparation of fractal-inspired computational architectures for advanced large language model analysis. arXiv preprint, arXiv:2511.07329, 2025. [21] Joan Puigcerver, Carlos Riquelme, and Neil Houlsby. Soft moe: Differentiable sparse mixture of experts. arXiv preprint arXiv:2306.09603, 2023. [22] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4780–4789, 2019. [23] Carlos Riquelme, Joan Puigcerver, Alexander Kolesnikov, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems (NeurIPS), 2021. [24] Usha Shrestha, Dmitry Ignatov, and Radu Timofte. From brute force to semantic insight: Performance-guided data transformation design with LLMs. arXiv preprint, arXiv:2601.03808, 2026. [25] Tolgay Atinc Uzun, Dmitry Ignatov, and Radu Timofte. Closed-loop LLM discovery of non-standard channel priors in vision models. In Proceedings of the International Conference on Pattern Recognition (ICPR), 2026. to appear. [26] Tolgay Atinc Uzun, Waleed Khalid, Saif U Din, Sai Revanth Mulukuledu, Akashdeep Singh, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Yashkumar Rajeshbhai Lukhi, Ahsan Hussain, Krunal Jesani, Usha Shrestha, Yash Mittal, Roman Kochnev, Pritam Kadam, Mohsin Ikram,
[1] Taimoor Abbas and Yiannis Andreopoulos. Biased mixture of experts for efficient inference of deep neural networks. IEEE Transactions on Image Processing, 29:7402– 7417, 2020. [2] Santosh Premi Adhikari, Radu Timofte, and Dmitry Ignatov. Convergence theory for iterative llm-based neural architecture search: A parametric cross-entropy framework with closed-form proxy reliability. arXiv preprint, arXiv:2605.30103, 2026. [3] Santosh Premi Adhikari, Radu Timofte, and Dmitry Ignatov. Delta-based neural architecture search: LLM fine-tuning via code diffs. arXiv preprint, arXiv:2605.04903, 2026. [4] Faisal Ahmed and Lorenzo Torresani. Network of experts for large-scale image categorization. In European Conference on Computer Vision (ECCV), pages 516–532. Springer, 2016. [5] DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2401.06066, 2024. [6] Saif U Din, Muhammad Ahsan Hussain, Mohsin Ikram, Dmitry Ignatov, and Radu Timofte. AI on the edge: An automated pipeline for PyTorch-to-Android deployment and benchmarking. Preprints, 2025. [7] Raghuvir Duvvuri, Chandini Vysyaraju, Avi Goyal, Dmitry Ignatov, and Radu Timofte. Enhancing LLM-based neural network generation: Few-shot prompting and efficient validation for automated architecture design. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3242–3251, 2026. [8] Arash Torabi Goodarzi, Roman Kochnev, Waleed Khalid, Furui Qin, Tolgay Atinc Uzun, Yashkumar Sanjaybhai Dhameliya, Yash Kanubhai Kathiriya, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. LEMUR neural network dataset: Towards seamless AutoML. arXiv preprint, arXiv:2504.10552, 2025. [9] Sam Gross, Michael Wilber, and Serge Belongie. Hard mixture of experts for large scale weakly supervised vision. In European Conference on Computer Vision (ECCV) Workshops, 2017. [10] Xiaojie Gu, Dmitry Ignatov, and Radu Timofte. Resourceefficient iterative LLM-based NAS with feedback memory. arXiv preprint, arXiv:2603.12091, 2026. [11] Krunal Jesani, Dmitry Ignatov, and Radu Timofte. LLM as a neural architect: Controlled generation of image captioning models under strict API contracts. arXiv preprint, arXiv:2512.14706, 2025. [12] Faraz Kayani, Sarmad Kayani, Asad Ahmed, Radu Timofte, and Dmitry Ignatov. Real image denoising with knowledge distillation for high-performance mobile NPUs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3792– 3800, 2026. [13] Waleed Khalid, Dmitry Ignatov, and Radu Timofte. A retrieval-augmented generation approach to extracting algorithmic logic from neural networks. arXiv preprint, arXiv:2512.04329, 2025.
7
Harsh Rameshbhai Moradiya, Alice Arslanian, Dmitry Ignatov, and Radu Timofte. LEMUR 2: Unlocking neural network diversity for AI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3291–3300, 2026. [27] Guolong Wang, Tianlong Wang, Pengtao Xie, and Philip S Yu. Deep mixture of experts via shallow embedding. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2020. [28] Brandon Yang, Gabriel Bender, Quoc V. Le, and J Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. In Advances in Neural Information Processing Systems (NeurIPS), 2019. [29] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2017.
8