From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT Mahmoud Hanouneh, Radu Timofte, Dmitry Ignatov
arXiv:2605.03686v1 [cs.LG] 5 May 2026
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Abstract Automated Machine Learning (AutoML) frameworks increasingly leverage Large Language Models (LLMs) for tasks such as hyperparameter optimization and neural architecture code generation. However, current LLM-based approaches focus on generative outputs and evaluate them by training the produced artifacts. Whether LLMs can learn to reason about neural network performance across datasets remains underexplored. We present a classification task integrated into the NNGPT framework, in which a fine-tuned LLM predicts which of two image classification datasets a given neural network architecture achieves higher accuracy on. The task is built on the LEMUR dataset, which provides standardized PyTorch implementations with reproducible performance metrics. Three prompt configurations of increasing difficulty are evaluated: a normalized-accuracy baseline (trivially reaching 100%), a metadata-enriched prompt replacing accuracies with dataset properties, and a code-only prompt presenting only architecture source code and dataset names. Using DeepSeek-Coder-7B-Instruct fine-tuned with LoRA, the code-only prompt reaches 80% peak accuracy over 15 epochs, while the metadata prompt peaks at 70%. Perdataset analysis reveals complementary strengths: metadata excels for datasets with distinctive properties (CelebAGender at 90.9%) but degrades for overlapping characteristics, whereas the code-only prompt shows more balanced performance. A comparison with DeepSeek-Coder1.3B confirms that model capacity affects this form of architectural reasoning. The results establish that LLMs can be fine-tuned to predict cross-dataset suitability from neural network code, suggesting that architecture source code contains richer discriminative signal than dataset metadata alone.
1. Introduction The rapid progress of deep learning has produced increasingly powerful neural network architectures for computer vision; however, designing, tuning, and benchmarking these
models remains a slow, resource-intensive process that depends heavily on expert knowledge [4]. The choice of architecture and hyperparameters directly determines model performance, yet traditional approaches rely on manual experimentation or computationally expensive search procedures [6]. Automated Machine Learning (AutoML) has emerged to address these challenges by automating the design and optimization of ML pipelines in a data-driven way, aiming to free experts for higher-level tasks and enable novice users to train well-performing models [3]. Classical AutoML techniques—grid search, random search, Bayesian optimization, and evolutionary strategies—explore large configuration spaces but require numerous training iterations, leading to massive computational overhead for complex deep learning models [6]. Large Language Models (LLMs) have recently demonstrated the ability to capture complex patterns across a variety of tasks and, with fine-tuning, can acquire new capabilities to solve previously unseen problems [4, 8]. Their application to AutoML delivers a paradigm shift: instead of iteratively searching for optimal configurations, LLMs can learn from historical experimental data and generate solutions in a single forward pass, significantly reducing computational cost [7]. The NNGPT framework exemplifies this approach by integrating LLMs into the machine learning pipeline to generate or optimize neural network architectures, leveraging learned representations of past experiments from the LEMUR Neural Network Dataset [4]. Unlike traditional datasets that focus solely on input data, LEMUR treats the neural network code itself as data, providing a unified collection of PyTorch implementations across multiple tasks with consistent evaluation pipelines and structured logging of results [4]. This structured representation enables LLMs to learn relationships between architectures, hyperparameters, and performance outcomes. Despite these advances, a gap remains in how LLMs are evaluated within AutoML frameworks. Existing LLMbased approaches focus on generative tasks—producing neural network code or predicting numerical hyperparameter values—and their evaluation relies on training the generated outputs and measuring downstream accuracy [6, 7].
This paradigm does not address a fundamentally different question: whether an LLM can learn to reason about neural network performance across datasets, classifying which of two datasets a given architecture performs better on without generating or training any model. Prior to this work, the NNGPT pipeline supported only code-generation output types and had no mechanism for classification tasks where the LLM’s output is a discrete label rather than executable code. Similarly, benchmarking studies such as AMLB [3] compare AutoML frameworks across tasks and data characteristics but do not test an LLM’s capacity for meta-level reasoning about model–dataset compatibility, and agentbased approaches such as AgentHPO [8] focus on iteratively refining hyperparameters rather than predicting crossdataset performance differences. This leaves open the question of whether LLMs, when fine-tuned on structured performance data from repositories such as LEMUR, can develop a meaningful understanding of the relationship between neural network architectures and dataset properties. This work addresses this gap by presenting the design, implementation, and evaluation of a classification task integrated into the NNGPT framework. Specifically, given a neural network’s source code and two candidate dataset names, the fine-tuned LLM must predict which dataset the network achieves higher accuracy on. The contributions of this work are threefold: (1) a novel classification task formulation within the NNGPT framework that extends LLM-based AutoML beyond generative outputs to reasoning about architecture–dataset relationships; (2) a structured ablation study across three prompt variants that progressively remove explicit performance signals, revealing that architecture source code carries richer discriminative signal than dataset metadata; and (3) empirical evidence that fine-tuned LLMs can predict cross-dataset suitability from source code alone, achieving 80% classification accuracy with DeepSeek-Coder-7B [1].
2. Related Work ‘
2.1. LLM-Based Hyperparameter Optimization Recent work has explored LLMs as alternatives to traditional hyperparameter optimization methods. Kochnev et al. [6] demonstrated that fine-tuned LLMs can match or outperform Optuna on hyperparameter tuning tasks while requiring fewer evaluations. AgentHPO [8] introduces a multi-agent framework in which a Creator agent generates hyperparameter configurations and an Executor agent trains and evaluates them iteratively, achieving competitive results across standard benchmarks. These approaches optimize parameters but do not address whether LLMs can predict cross-dataset performance differences.
2.2. LLM-Driven Neural Architecture Generation NNGPT [7] represents a paradigm shift in neural architecture search by fine-tuning LLMs on structured architecture– performance data from the LEMUR dataset to generate executable PyTorch code in a single forward pass. The framework employs a closed-loop LangGraph-based agent orchestration where generated architectures are trained, evaluated, and fed back into LoRA fine-tuning. Khalid et al. [5] extend this direction by incorporating novelty detection to encourage architecturally diverse generations. While these systems generate architectures, they do not classify or predict performance across datasets—the task addressed in this work.
2.3. Benchmarking and Datasets for AutoML Standardized benchmarking is essential for reproducible AutoML research. The AMLB benchmark [3] provides a unified evaluation framework for comparing AutoML systems, while AutoMLBench [2] systematically evaluates the impact of design decisions such as search space size and meta-learning. The LEMUR dataset [4, 12] addresses a complementary need by treating neural network code as data: it provides a unified collection of PyTorch implementations across multiple tasks with consistent evaluation pipelines and structured logging of experimental results. This structured representation is particularly important for LLM-based approaches, as it allows models to learn relationships between architectures, hyperparameters, and performance outcomes. LEMUR provides the data foundation that makes the classification task presented in this work possible.
3. Methodology 3.1. Task Formulation The classification task is formulated as a constrained text generation problem. Given a neural network architecture (represented as truncated PyTorch source code) and the names of two image classification datasets, the fine-tuned LLM must generate a single token: the name of the dataset on which the architecture achieves higher accuracy. The model receives a structured prompt as plain text and outputs a dataset name as a free-form string, evaluated against the ground truth via cascaded string matching (exact, substring, and normalized substring). The ground-truth label for each pair is derived from normalized accuracy values computed over the LEMUR corpus. For a dataset d, the normalized accuracy of model m is defined as: norm acc(m, d) =
acc(m, d, e) maxm′ acc(m′ , d, e)
(1)
where the maximum is taken over all models evaluated at
the fixed reference epoch e=5. For a pair (d1 , d2 ), the ground-truth label is the dataset with the higher normalized accuracy. This normalization ensures that cross-dataset comparisons reflect relative model suitability rather than absolute dataset difficulty [3, 4]. The pairwise formulation yields n2 distinct dataset pairs per architecture, providing substantially more training signal than a single multi-class label. Training samples are constructed via SQL self-joins with an ordering constraint (d1 .id < d2 .id) that eliminates duplicate and reflexive pairs [7].
Prompt
Model
N
Peak
Ep.
Run
Norm. Acc. Code-Only Metadata Code-Only
7B 7B 7B 1.3B
10 30 30 30
100.0% 80.0% 70.0% 70.0%
9 15 8, 13 22, 23
12 16 15 29
Table 1. Peak classification accuracy across prompt variants and model sizes. Peak = peak accuracy; Ep. = epoch of peak; Run = total epochs.
4. Experiments 4.1. Experimental Setup
3.2. Prompt Design Three prompt configurations form a structured ablation that progressively removes explicit performance signals: V1 – Normalized-accuracy baseline (N=10): provides the neural network source code, both dataset names, and both normalized accuracy values. This serves as a pipeline validation since the LLM need only compare two numbers. V2 – Metadata-enriched (N=30): replaces accuracy values with dataset properties (number of training images, image size, number of channels, number of classes). The LLM must infer relative performance from the interplay between architectural features and dataset properties. V3 – Code-only (N=30): presents only the neural network source code and dataset names, with no accuracy or metadata information. The model must predict the better-suited dataset based solely on its understanding of the architecture’s structure. All configurations share: 20-token genoutput type=classification, eration limit, and 2,000-character code truncation. The ground-truth label better dataset is excluded from input fields and injected only on the output side of training pairs to prevent data leakage.
3.3. Pipeline Integration The classification task reuses NNGPT’s LangGraph-based multi-agent orchestration (Fine-tuner → Generator → Evaluator) [7], but routes outputs through a dedicated classification evaluator rather than the neural network training branch. This preserves the closed-loop self-improvement property where execution logs feed back into LoRA finetuning. The same LoRA configuration (rank=32, α=32, dropout=0.05), cosine learning rate scheduler, and three inner training epochs apply to classification as to code generation, ensuring consistency. For the DeepSeek-1.3B experiments, ONNX Runtime [11] acceleration is used with a separate tuning script.
All experiments were conducted on the Computer Vision Lab Kubernetes cluster at the University of Würzburg using NVIDIA GeForce RTX 4090 GPUs (24 GB VRAM), via the AI Linux docker image1 . The primary LLM was DeepSeek-Coder-7B-Instruct-v1.5, fine-tuned with LoRA (rank=32, α=32, dropout=0.05) using a cosine learning rate scheduler and three inner training epochs per outer finetuning cycle [6]. An additional experiment used DeepSeekCoder-1.3B-Instruct with ONNX Runtime acceleration for the code-only variant. Classification accuracy was computed at each epoch as the proportion of correctly predicted dataset names out of total test samples.
4.2. Results Table 1 summarizes peak accuracy across all configurations. The normalized-accuracy baseline saturates at 100%, confirming that the task is trivially solvable when explicit performance values are provided. The code-only prompt achieves the highest non-trivial accuracy at 80%. 4.2.1. Code-Only Prompt. With only neural network source code and dataset names, DeepSeek-7B started at 20% accuracy at epoch 0—notably below the 50% random baseline, indicating a systematic bias in the pre-trained model’s prior probability over dataset name tokens. Despite this unfavorable starting point, the model showed a clear learning trajectory, reaching 80% at epoch 15 (24/30 correct). Table 2 shows the epoch-byepoch progression. The same configuration evaluated with DeepSeek-1.3B using ONNX Runtime showed substantially more volatile behavior, oscillating between 23% and 70% and peaking at 70% at epochs 22–23. Unlike the 7B model’s steady upward trajectory, the 1.3B model showed no continuous improvement trend in its first 13 epochs. 4.2.2. Metadata-Enriched Prompt. The metadata prompt achieved 70% peak accuracy at epochs 8 and 13 with accuracy oscillating between 40% and 1 AI Linux: https://hub.docker.com/r/abrainone/ai-linux
Ep.
Corr.
Tot.
Acc.
0 2 4 6 9 10 12 13 15
6 14 15 17 20 21 23 23 24
30 30 30 30 30 30 30 30 30
20.0% 46.7% 50.0% 56.7% 66.7% 70.0% 76.7% 76.7% 80.0%
Table 2. Selected epochs for the code-only prompt (DeepSeek7B, N=30). Accuracy rises from 20% to 80% over 15 fine-tuning epochs. Figure 3. Metadata prompt: accuracy vs. training loss. Loss decreases steadily while accuracy oscillates, indicating consistent weight-level learning but a noisy classification signal.
Dataset
Code (7B)
Meta (7B)
Code (1.3B)
CelebA-Gender CIFAR-10 CIFAR-100 ImageNette MNIST Places365 SVHN
72.5 51.9 37.0 52.4 53.2 12.5 43.1
90.9 31.2 40.5 27.5 50.0 45.5 75.0
39.2 63.8 49.1 57.1 53.2 10.0 36.1
Table 3. Per-dataset aggregated accuracy (%) by prompt variant and model. Bold indicates the best result per dataset. Figure 1. Classification accuracy over epochs for all three prompt variants (DeepSeek-7B). The normalized-accuracy baseline saturates at 100%, while code-only and metadata prompts show gradual learning above the 50% random baseline.
inherently more ambiguous than the code-only variant for certain dataset pairs.
4.3. Per-Dataset Analysis Table 3 reveals that the code-only and metadata prompts offer complementary strengths. Metadata excels when dataset properties are distinctive: CelebA-Gender reaches 90.9% because its metadata (64×64, 2 classes, 162K images) provides an unambiguous fingerprint [9]. Similarly, SVHN improves to 75.0% (from 43.1% code-only) as its metadata is sufficiently unique [10]. Conversely, CIFAR-100 collapses to 40.5% and ImageNette to 27.5% with metadata, both sharing overlapping characteristics with other corpus datasets. The code-only prompt provides more balanced performance across datasets, with CelebA-Gender at 72.5% and most datasets clustering around 50%. Figure 2. Code-only prompt: DeepSeek-7B (peak 80%, epoch 15) vs. DeepSeek-1.3B with ONNX (peak 70%, epochs 22–23). The larger model converges faster and more stably.
70%. Training loss decreased steadily from 0.87 to 0.14, indicating consistent weight-level learning, but classification accuracy remained noisy (Fig. 3). This suggests that the mapping from metadata plus code to the correct dataset is
4.4. Ablation Study The three prompt variants constitute a structured ablation over input information. Removing accuracy values (V1→V2) forces the model to reason from dataset properties rather than numerical comparison, reducing accuracy from 100% to 70%. Removing metadata (V2→V3) forces pure code-based reasoning, yet accuracy increases to 80%. This counterintuitive result suggests that metadata
datasets remains untested.
6. Conclusion
Figure 4. Per-dataset aggregated accuracy across three configurations. Metadata excels for distinctive datasets (CelebA-Gender, SVHN); code-only is more balanced but struggles with Places365.
may introduce noise for datasets with overlapping properties, while the architecture source code contains richer discriminative signal. The model-size ablation (7B vs. 1.3B on code-only) shows that the larger model reaches higher peak accuracy (80% vs. 70%) with faster convergence (epoch 15 vs. 22–23) and substantially lower volatility, confirming that model capacity is a relevant factor for architectural reasoning.
5. Discussion The results demonstrate that LLMs can be fine-tuned to predict cross-dataset performance from neural network source code. Both the code-only (80%) and metadata-enriched (70%) configurations exceed the 50% random baseline by a substantial margin, indicating that fine-tuned LLMs extract meaningful discriminative signals. The finding that code-only outperforms metadata is noteworthy. We hypothesize that architecture source code encodes implicit information about a model’s inductive biases—such as receptive field size, depth, and normalization patterns—that correlate with dataset suitability in ways not captured by tabular metadata. Metadata, while informative for distinctive datasets, introduces ambiguity when multiple datasets share similar properties (e.g., CIFAR-10 and SVHN both being 32×32 RGB with 10 classes). Several limitations should be noted. The evaluation set (N=30) is small; the 10-percentage-point gap between codeonly and metadata corresponds to only 3 additional correct predictions. All experiments used a single random seed, so variance across initializations is unknown. The pairwise framing creates statistically dependent samples not controlled for in accuracy computation. The 2,000-character code truncation discards portions of longer architectures, and its effect has not been systematically analyzed. Finally, all experiments are restricted to image classification datasets within LEMUR; generalization to other tasks and
We presented a classification task integrated into the NNGPT framework that extends LLM-based AutoML beyond generative outputs to predicting cross-dataset performance from neural network source code. Using three prompt variants as a structured ablation, we showed that DeepSeek-Coder-7B fine-tuned with LoRA achieves 80% peak accuracy on the code-only configuration, outperforming the metadata-enriched variant (70%). Per-dataset analysis revealed complementary strengths: metadata excels for distinctive datasets, while code provides more balanced performance. The model-size comparison (7B vs. 1.3B) confirmed that capacity matters for architectural reasoning. Future work includes evaluating additional LLM families (e.g., CodeLlama, OlympicCoder), held-out generalization splits on unseen architectures and datasets, combining code and metadata in a single prompt, extending to segmentation and detection tasks in LEMUR [4], and multi-seed runs for statistical robustness.
References [1] DeepSeek-AI. Deepseek-v3 technical report, 2024. 2 [2] Hassan Eldeeb, Mohamed Maher, Radwa Elshawi, and Sherif Sakr. AutoMLBench: A comprehensive experimental evaluation of automated machine learning frameworks. Expert Systems with Applications, 243:122877, 2024. 2 [3] Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: an AutoML benchmark. Journal of Machine Learning Research, 25:1–65, 2024. 1, 2, 3 [4] Arash Torabi Goodarzi, Roman Kochnev, Waleed Khalid, Furui Qin, Tolgay Atinc Uzun, Yashkumar Sanjaybhai Dhameliya, Yash Kanubhai Kathiriya, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. LEMUR neural network dataset: Towards seamless AutoML. arXiv preprint, arXiv:2504.10552, 2025. 1, 2, 3, 5 [5] Waleed Khalid, Dmitry Ignatov, and Radu Timofte. From memorization to creativity: LLM as a designer of novel neural architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2026. to appear. 2 [6] Roman Kochnev, Arash Torabi Goodarzi, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. Optuna vs code llama: Are LLMs a new paradigm for hyperparameter tuning? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5664– 5674, 2025. 1, 2, 3 [7] Roman Kochnev, Waleed Khalid, Tolgay Atinc Uzun, Xi Zhang, Yashkumar Sanjaybhai Dhameliya, Furui Qin, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Ignatov, and Radu Timofte. NNGPT: Rethinking AutoML with large language models. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2026. to appear. 1, 2, 3 [8] Siyi Liu, Chen Gao, and Yong Li. AgentHPO: Large language model agent for hyper-parameter optimization. arXiv preprint, arXiv:2402.01881, 2025. Second Conference on Parsimony and Learning (CPAL 2025). 1, 2 [9] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. CelebFaces Attributes (CelebA) Dataset. http : / / mmlab . ie . cuhk . edu . hk / projects / CelebA . html, 2015. Accessed: 2026-04-22. 4 [10] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. 4 [11] ONNX Community. ONNX: Open neural network exchange. https://github.com/onnx/onnx, 2019. 3 [12] Tolgay Atinc Uzun, Waleed Khalid, Saif U Din, Sai Revanth Mulukuledu, Akashdeep Singh, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Yashkumar Rajeshbhai Lukhi, Ahsan Hussain, Krunal Jesani, Usha Shrestha, Yash Mittal, Roman Kochnev, Pritam Kadam, Mohsin Ikram, Harsh Rameshbhai Moradiya, Alice Arslanian, Dmitry Ignatov, and Radu Timofte. LEMUR 2: Unlocking neural network diversity for AI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2026. to appear. 2