Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures Hafsa Mateen*, Radu Timofte,
Dmitry Ignatov
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
arXiv:2607.08511v1 [cs.LG] 9 Jul 2026
Abstract Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures. We evaluated 30 representative architectures from convolutional and transformer families within the LEMUR neural network dataset. Through automated source-code injection, we applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10. Our best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%. The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies. The resulting accuracy landscape, contributed to the LEMUR nn-dataset, provides a practical reference for principled scheduler selection.
1. Introduction Deep learning has witnessed remarkable progress over the past decade, driven by advances in model architectures, large-scale datasets, and computational infrastructure. Yet the training recipe — the precise combination of optimizer, learning rate schedule, regularization, and augmentation strategy — remains a largely hand-crafted artifact that requires substantial expert knowledge and empirical trial. Among the components of a training recipe, the learning rate (LR) schedule exerts a particularly strong influence on convergence speed, final accuracy, and generalization. A poorly chosen schedule can cause slow convergence, oscillation around the optimum, or premature saturation, while a well-chosen one can recover several percentage points of accuracy with no change to the model architecture. Despite this significance, most published works report results with a * Corresponding author: [email protected]
single fixed schedule and treat the scheduler as a secondary hyperparameter. Classical hyperparameter optimization (HPO) methods such as Bayesian optimization [6] and random search can in principle explore the scheduling space, but they are expensive, require repeated full training runs, and offer limited interpretability. Systematic grid-based evaluation across a diverse architecture pool offers a complementary and reproducible alternative: by fixing all other training hyperparameters and varying only the scheduler, one can directly attribute accuracy differences to scheduling choices and build a reusable reference landscape. We present such a study, contributing a large-scale systematic evaluation of learning rate scheduling strategies across 30 heterogeneous neural network architectures evaluated on CIFAR-10. Our contributions are as follows. • We contribute 3,938 LR-scheduler model variants to the LEMUR nn-dataset [1, 3, 13] by injecting 25 diverse LR scheduler configurations into 30 neural network architectures via automated source-code injection, evaluating all 3,938 variants on CIFAR-10 for five epochs. • We systematically evaluate the impact of scheduler choice on top-1 accuracy, revealing strong architecture-specific preferences: CyclicLR dominates on mobile-optimized and convolutional models, while CosineAnnealingWarmRestarts leads on inception-based architectures. • We provide a detailed analysis of weight decay interactions, individual scheduler variant rankings, and architecture mean accuracy, offering practical guidance for training recipe design. • We report that the best configuration achieves 86.45% top-1 accuracy, with 237 of 3,938 variants exceeding 80%, demonstrating the practical value of principled scheduler selection. The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 describes the architecture pool, scheduler catalogue, and injection pipeline. Section 4 details the experimental setup. Section 5 presents results and analysis. Section 6 presents the comparative analysis. Section 7 discusses limitations. Section 8 con-
cludes.
work implementations across diverse architecture families. Supporting studies have explored architecture design [5], data transformation [11], and retrieval-based analysis of neural network behaviour [4] within such pipelines. A reliable training recipe — including a well-chosen learning rate scheduler — is a prerequisite for meaningful evaluation scores in any automated pipeline. Our work directly addresses this gap by providing a systematic empirical reference for scheduler selection across the heterogeneous architectures used in these frameworks.
2. Related Work 2.1. Learning Rate Scheduling Learning rate scheduling has a long history in deep learning optimization. Fixed-step decay [9] was among the earliest systematic approaches, reducing the learning rate by a multiplicative factor at predetermined epochs. Cosine annealing [9] introduced a smooth periodic decay that avoids abrupt transitions and has since become the de facto standard for vision models. The one-cycle policy [12] takes a fundamentally different approach, first warming up the learning rate to a maximum before annealing, achieving super-convergence on several benchmarks. Cyclical learning rates [12] oscillate between bounds to escape local minima and have shown strong empirical results on convolutional networks. Adaptive methods such as ReduceLROnPlateau monitor a validation metric and reduce the rate when progress stalls, providing a data-driven alternative to fixed schedules. Warm-up strategies, in which the learning rate is gradually increased from a small value at the start of training, have become standard practice for large vision transformers. Linear warm-up followed by cosine annealing has shown consistent improvements over fixed schedules across a wide range of architectures. Despite the proliferation of scheduling methods, systematic large-scale comparisons across diverse architecture families remain rare, and most published results report a single fixed schedule without ablating the scheduler choice.
2.4. Neural Network Datasets The LEMUR neural network dataset [1] provides a unified collection of diverse neural network implementations sharing a standardised training interface, enabling fair comparison across architectures without per-model adaptation. LEMUR 2 [13] extends this collection with additional architectures spanning high-capacity and edge-optimized models. NN-Lite [3] provides lightweight models targeting mobile deployment. These datasets form the foundation of our experimental pool and enable us to evaluate scheduler performance across a uniquely broad set of architectures within a unified evaluation framework.
2.5. CIFAR-10 as a Benchmark CIFAR-10 [8] is a widely used image classification benchmark consisting of 60,000 32 × 32 colour images across 10 classes, split into 50,000 training and 10,000 test samples. Its compact size makes it well-suited for large-scale hyperparameter sweeps where full-resolution datasets such as ImageNet would be prohibitively expensive. Prior work on learning rate scheduling [9, 12] has consistently used CIFAR-10 as a standard evaluation ground, making our results directly comparable to established benchmarks. We note that while absolute accuracy figures on CIFAR-10 may differ from ImageNet-scale experiments, the relative rankings of scheduling strategies are expected to transfer across datasets.
2.2. Automated Hyperparameter Optimization Classical AutoML approaches such as Bayesian optimization, evolutionary search, and random search have been widely applied to hyperparameter tuning. Kochnev et al. [6] showed that LLMs can match or exceed Optuna-based Bayesian optimization for hyperparameter selection across a suite of vision tasks, establishing LLMs as a practical tool for AutoML. Vysyaraju et al. [14] demonstrated that fewshot prompting significantly improves the quality of LLMgenerated neural network configurations. Gu et al. [2] introduced feedback memory into the LLM-based neural architecture search loop, enabling iterative refinement of proposals. These works collectively demonstrate that LLMs can reduce the cost of hyperparameter search while maintaining competitive performance, motivating systematic evaluation of the scheduling dimension across diverse architecture families.
3. Methodology Our methodology consists of two stages: (i) systematic construction of a large-scale model variant pool by injecting diverse LR scheduling strategies into existing neural network architectures, and (ii) automated evaluation of all generated variants using a unified training pipeline.
3.1. Architecture Pool and the LEMUR Dataset To ensure broad coverage across architecture families, we build upon the LEMUR dataset [1, 3, 13], which provides a diverse collection of neural network implementations sharing a unified training interface. From this collection we select 30 representative architectures spanning convolutional and transformer-based families: AlexNet, ResNet,
2.3. Automated Neural Network Pipelines Recent work within the NNGPT framework [7] has demonstrated automated generation and evaluation of neural net2
Table 1. Summary of the 25 LR scheduler configurations. Family
Key Parameters
#
StepLR
step∈{0.1,0.25,0.5,0.7}, γ ∈{0.1,0.3,0.5,0.7} γ ∈{0.90,0.95,0.98} Tmax ∈{5,10,20} epochs T0 ∈{2,5} milestones {5,10},{3,7},{2,4} (factor,patience): (0.5,2),(0.3,3),(0.1,5) triangular, triangular2, exp range lrmax ∈{0.05,0.10} fixed defaults
4
ExponentialLR CosineAnnealingLR CosineAnnealingWR MultiStepLR ReduceLROnPlateau CyclicLR OneCycleLR LinearLR / PolynomialLR Total
3.3. Automated Code Injection For each architecture–scheduler–weight decay combination we programmatically generate a modified model variant using the inject scheduler() function. This function performs line-level source-code editing guided by Python parsing in four steps.
3 3 2 3 3
Step 1 — Method boundary detection. Locates train setup() and learn() within class Net by tracking indentation levels, ensuring robustness across diverse coding styles present in the LEMUR dataset.
3 2 2 25
Step 2 — Scheduler initialisation injection. Scheduler instantiation code is appended to train setup(). Where a scheduler already exists, it is replaced rather than duplicated.
VGG, DenseNet, EfficientNet, MobileNetV2/V3, RegNet, ShuffleNet, SqueezeNet, GoogLeNet, InceptionV3, MNASNet, MaxVit, SwinTransformer, VisionTransformer, ConvNeXt, DPN68/107/131, BagNet, FractalNet, AirNet, AirNext, BayesianNet, DarkNet, UNet2D, ICNet, and Diffuser. Each architecture exposes a standardised train setup() method for optimiser and scheduler initialisation and a learn() method for the per-iteration training loop, enabling fully automated code injection. The selection covers lightweight mobile-optimized models, standard mid-capacity convolutional networks, dense connection networks, and large-scale vision transformers, ensuring that our findings generalize across the full spectrum of modern neural network design.
Step 3 — Scheduler step injection. For perepoch schedulers (e.g., StepLR, CosineAnnealingLR), self.scheduler.step() is inserted at the end of learn(). For per-batch schedulers (e.g., CyclicLR, OneCycleLR, ReduceLROnPlateau), the step call is placed immediately after self.optimizer.step(). Step 4 — Hyperparameter registration. The supported hyperparameters() function is rewritten to register all scheduler-specific parameters alongside the original architecture hyperparameters, ensuring full compatibility with the NNEval evaluation harness. Each variant is validated by (i) parsing the modified source with Python’s ast module to verify syntactic correctness, and (ii) confirming every registered hyperparameter key appears at least twice as a string literal in the code. Variants failing either check are discarded. A bug in LinearLR and PolynomialLR — where total iters was incorrectly scaled, causing step count mismatch — was identified and fixed prior to the main evaluation run, enabling smooth evaluation of all 3,938 models without scheduler errors.
3.2. Learning Rate Scheduler Catalogue We define a catalogue of 25 LR scheduler configurations drawn from nine PyTorch [10] scheduler families, summarised in Table 1. StepLR (4 variants): decays by γ every fixed step fraction {0.10, 0.25, 0.50, 0.70} of the budget, with γ ∈ {0.1, 0.3, 0.5, 0.7}. ExponentialLR (3 variants): per-epoch decay lrt = lr0 · γ t , γ ∈ {0.90, 0.95, 0.98}. CosineAnnealingLR (3 variants): cosine decay with ηmin = 10−6 and Tmax ∈ {5, 10, 20} epochs. CosineAnnealingWarmRestarts (2 variants): cosine with periodic restarts, T0 ∈ {2, 5} epochs. MultiStepLR (3 variants): milestones {5, 10}, {3, 7}, {2, 4} with γ ∈ {0.5, 0.3, 0.1}. ReduceLROnPlateau (3 variants): factor–patience (0.5, 2), (0.3, 3), (0.1, 5). CyclicLR (3 variants): triangular, triangular2, exp range between 10−4 and 0.1. OneCycleLR (2 variants): lrmax ∈ {0.05, 0.10}. LinearLR and PolynomialLR (1 each): fixed defaults (start factor 0.1 over 5 epochs; degree-2 decay over 5 iterations, respectively).
3.4. Hyperparameter Search Space Beyond scheduler type, we vary weight decay across seven values: λ ∈ {0, 10−5 , 5 × 10−5 , 10−4 , 5 × 10−4 , 10−3 , 5 × 10−3 }. All other hyperparameters are fixed: lr = 0.01, batch size B = 64, dropout p = 0.2, momentum µ = 0.9, Emax = 5 epochs, norm 256 flip augmentation. The full combinatorial space yields 30 × 25 × 7 = 5,250 theoretical variants; those passing syntax validation and hyperparameter consistency checks are retained for evaluation. Weight decay was included as an additional dimension because it interacts strongly with learning rate magnitude and schedule shape, and its effect is expected to vary across architecture families. 3
Table 2. Top-10 model variants by top-1 accuracy on CIFAR-10.
4. Experiments 4.1. Dataset
Model ID
Top-1 Accuracy (%)
lr-baf515f1... lr 2782 lr 2773 lr-982834f1... lr 2777 lr 2784 lr 2797 lr 2794 lr 2779 lr-afd558c8...
All experiments use the CIFAR-10 dataset, which consists of 60,000 colour images of size 32×32 pixels across 10 object classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck), split into 50,000 training and 10,000 test images. The norm 256 flip transform is applied uniformly: images are resized to 256 × 256, normalised using per-channel mean and standard deviation, and randomly horizontally flipped during training. CIFAR-10 was chosen because its compact size allows evaluation of nearly 4,000 model variants within a reasonable computational budget while remaining a widely used and well-understood benchmark for image classification.
86.45 86.17 85.99 85.96 85.69 85.56 85.55 85.55 85.52 85.49
Table 3. Summary statistics across all 3,938 evaluated variants. Metric Total variants contributed & evaluated Maximum accuracy Mean accuracy Median accuracy Minimum accuracy Variants > 80% accuracy Variants > 75% accuracy Variants > 60% accuracy Variants < 30% accuracy
4.2. Training and Testing All model variants are trained and evaluated using the NNEval harness on NVIDIA GeForce RTX 3090/4090 (24 GB) GPUs within the CVL Kubernetes cluster at the University of Würzburg. Each model is trained for five epochs in the screening phase to efficiently identify promising scheduler–architecture combinations before committing to full-length evaluation. All models use SGD with lr = 0.01, momentum µ = 0.9, batch size B = 64, and dropout p = 0.2. Architecturespecific parameters such as stochastic depth probability for SwinTransformer and ConvNeXt, and attention dropout for VisionTransformer, are set to their recommended defaults throughout. The evaluation was conducted across multiple phases: an initial run (March 26, 2026) producing 1,507 model results, a second run (April 24 – May 2, 2026) adding 516 models after a bug fix in the LinearLR and PolynomialLR scheduler injection code, and a third phase extending coverage to additional architecture–scheduler combinations, yielding 3,938 total evaluated variants contributed to the LEMUR nn-dataset. The five-epoch screening budget was chosen as a balance between evaluation cost and ranking reliability. Prior work has shown that relative performance rankings stabilise quickly across architectures [6], making this a costeffective strategy for large-scale scheduler search. The hardware achieved approximately 200 samples per second per worker, enabling the full 3,938-variant evaluation to complete within a practical timeframe.
Value 3,938 86.45% 52.52% 50.00% 12.08% 237 (6.0%) 627 (15.9%) 1,107 (28.1%) 445 (11.3%)
accuracy to identify the most effective scheduling families; and (iv) per-architecture best accuracy and corresponding best scheduler to characterise architecture-specific scheduling preferences.
5. Results and Discussion 5.1. Overall Performance Across 3,938 evaluated model variants, top-1 accuracy ranges from 12.08% to 86.45%, with a mean of 52.52% and a median of 50.00%. This wide spread of 74.37 percentage points underscores the strong influence of both architecture family and scheduler choice on final performance. A total of 237 variants (6.0%) exceed 80% accuracy, 627 variants (15.9%) exceed 75%, and 1,107 variants (28.1%) exceed 60%, demonstrating that principled scheduler selection consistently pushes models into the high-performance regime. The top-10 results are reported in Table 2 and overall summary statistics in Table 3.
5.2. Per-Scheduler Analysis Table 4 and Figure 1 report mean and best accuracy per scheduler family. CosineAnnealingWarmRestarts achieves the highest mean accuracy (55.20%) and is involved in the best result (86.45%). MultiStepLR (54.65%), CosineAnnealingLR (54.55%), and ExponentialLR (54.51%) follow closely, confirming that smooth or milestone-based decay profiles are broadly effective. ReduceLROnPlateau, while ranked lowest by mean (45.04%), still achieves 86.45% in its best configuration, suggesting that patience-based
4.3. Evaluation Metrics The primary metric is top-1 classification accuracy on the CIFAR-10 test set. We additionally report: (i) the number of variants exceeding 75% and 80% accuracy thresholds to characterise the upper tail of the performance distribution; (ii) mean and median accuracy to measure overall landscape quality; (iii) per-scheduler-family mean and best 4
Table 4. Mean and best top-1 accuracy per scheduler family. Scheduler Family CosineAnnealingWR MultiStepLR CosineAnnealingLR ExponentialLR PolynomialLR StepLR OneCycleLR LinearLR CyclicLR ReduceLROnPlateau
Mean Acc. (%)
Best Acc. (%)
55.20 54.65 54.55 54.51 54.11 53.70 53.67 50.05 49.81 45.04
86.45 84.74 85.99 85.96 85.12 85.55 86.17 84.53 84.25 86.45
Table 5. Per-variant accuracy for selected scheduler families. Mean and best top-1 accuracy on CIFAR-10. Variant
Mean (%)
Best (%)
StepLR s20 g07 StepLR s3 g01 StepLR s10 g05 StepLR s5 g03
62.79 58.55 57.33 56.34
82.99 81.30 82.35 82.44
ExponentialLR g098 ExponentialLR g09 ExponentialLR g095
59.98 58.92 54.92
84.74 85.41 84.79
CosineAnnealingLR T10 CosineAnnealingLR T5 CosineAnnealingLR T20
63.97 61.43 58.48
82.55 83.67 84.29
CosineWarmRestarts T5 CosineWarmRestarts T2
63.94 60.15
85.99 86.17
MultiStepLR m2 4 g01 MultiStepLR m3 7 g03 MultiStepLR m5 10 g05
64.00 59.09 58.78
84.70 82.87 82.95
CyclicLR tri2 CyclicLR tri CyclicLR exp
56.24 55.81 33.20
83.92 84.25 69.45
PolynomialLR p2 LinearLR sf01
58.87 54.52
85.55 79.93
Figure 1. Mean top-1 accuracy per scheduler family across all 3,938 evaluated variants on CIFAR-10. Orange diamonds indicate the best single variant achieved by each scheduler. CosineAnnealingWarmRestarts achieves the highest mean (55.20%), while ReduceLROnPlateau has the lowest mean (45.04%) but the highest single peak (86.45%).
scheduling can succeed when training signals stabilise early; however, its average performance is depressed by configurations where five epochs are insufficient for meaningful plateau detection. CyclicLR also trails the cosinebased methods (mean 49.81%), consistent with its need for longer cycles to deliver consistent gains.
Figure 2. Best and approximate mean top-1 accuracy by architecture family on CIFAR-10. Each bar group shows the mean (lighter) and best (darker) accuracy; the recommended scheduler for each family is annotated above the best-accuracy bar. No single scheduler is universally optimal across families.
5.3. Detailed Scheduler Variant Analysis 5.4. Weight Decay Analysis
Table 5 reports the performance of individual scheduler variants. Among StepLR variants, StepLR s20 g07 (step at 70% of budget, γ = 0.7) achieves the highest mean (62.79%), suggesting that a late, gentle decay is preferable within a five-epoch window. Among ExponentialLR variants, ExponentialLR g098 leads with mean 59.98%, confirming that slow decay is beneficial in the early training phase. For CosineAnnealingLR, CosineAnnealingLR T10 (mean 63.97%) outperforms both shorter and longer period variants. For CyclicLR, CyclicLR tri and CyclicLR tri2 perform comparably (mean ≈56%), while CyclicLR exp drops significantly (33.20%), indicating that exponential amplitude decay is harmful within this budget.
Table 6 and Figure 3 report mean and best accuracy as a function of weight decay value, aggregated across all architectures and scheduler types. The majority of evaluated variants use λ = 0 (3,529 of 3,938), reflecting the default training configuration; the remaining variants systematically vary weight decay. Among non-zero values, λ = 10−5 yields the highest mean (55.79%), while λ = 0 achieves the highest peak (86.45%). Heavy regularization (λ = 5×10−3 , mean 54.01%) shows no significant degradation. Importantly, the best accuracy at each weight decay level remains high (81.59%–86.45%), indicating that topperforming scheduler configurations are robust to regularization strength. 5
Table 6. Mean and best top-1 accuracy as a function of weight decay λ, aggregated across all architectures and schedulers.
Table 7. Best top-1 accuracy and best scheduler per architecture. Architecture
Weight Decay λ
Mean Acc. (%)
Best Acc. (%)
0 10−5 5×10−5 10−4 5×10−4 10−3 5×10−3
52.29 55.79 54.00 53.76 55.22 54.06 54.01
86.45 85.86 81.59 84.57 84.41 86.35 84.98
Best Acc. (%)
Best Scheduler
86.45 86.45 86.17 85.96 85.55 85.55 85.46 85.12 84.69 84.55 84.53 84.49 84.42 84.25 83.92 83.73 83.67 83.60 81.55 81.31
CosineAnnealingWR ReduceLROnPlateau OneCycleLR ExponentialLR StepLR ReduceLROnPlateau CosineAnnealingLR StepLR StepLR CosineAnnealingLR LinearLR MultiStepLR MultiStepLR OneCycleLR CyclicLR CosineAnnealingLR ExponentialLR PolynomialLR CyclicLR CyclicLR
GoogLeNet InceptionV3 DenseNet RegNet FractalNet VisionTransformer SwinTransformer EfficientNet BagNet MobileNetV2 ResNet ICNet AirNet DPN UNet2D ConvNeXt DarkNet BayesianNet SqueezeNet MobileNetV3
Table 8. Mean top-1 accuracy per architecture, sorted descending. Architecture
Figure 3. Effect of weight decay λ on mean and best top-1 accuracy, aggregated across all architectures and schedulers. λ = 10−5 yields the highest mean accuracy (55.79%), improving over the default λ = 0 (52.29%) by 3.5 percentage points. The highest single peak (86.45%) is achieved at λ = 0.
GoogLeNet BagNet MobileNetV2 RegNet ResNet AirNet InceptionV3 EfficientNet BayesianNet DenseNet DarkNet SwinTransformer MobileNetV3 FractalNet DPN VisionTransformer ConvNeXt SqueezeNet ICNet UNet2D
5.5. Per-Architecture Analysis Table 7 reports best accuracy and best scheduler per architecture, sorted by best accuracy. Ten architectures from the original pool of 30 (AirNext, MNASNet, MaxVit, VGG, AlexNet, ShuffleNet, Diffuser, and three further variants) could not be individually attributed due to class-naming conventions in the source files and are excluded from the per-architecture tables; their results are included in the aggregate statistics. GoogLeNet and InceptionV3 jointly achieve the highest peak (86.45%), with GoogLeNet under CosineAnnealingWarmRestarts and InceptionV3 under ReduceLROnPlateau. DenseNet (86.17%) and RegNet (85.96%) follow closely. Table 8 reports mean accuracy per architecture, showing that GoogLeNet (60.79%), BagNet (57.46%), and MobileNetV2 (56.38%) lead in average performance. Architectures designed for non-classification tasks — UNet2D, ICNet — show lower accuracy on CIFAR-10, consistent with their specialised objectives.
Mean Acc. (%)
#Variants
60.79 57.46 56.38 55.47 54.72 54.69 54.57 54.32 53.27 53.16 53.02 52.15 51.96 51.72 48.44 48.22 48.00 47.05 46.36 44.27
134 138 430 141 132 252 137 240 103 141 136 142 102 97 430 130 139 146 131 270
variants, late and gentle decay (s20 g07) outperforms aggressive early decay (s3 g01), suggesting that preserving a higher learning rate through most of the five-epoch budget is beneficial. MultiStepLR with early milestones (m2 4 g01, mean 64.00%) achieves the highest mean of any individual variant, driven by the sharp decay at epoch 2 and 4 that aligns well with the five-epoch window. CyclicLR demonstrates strong architecture dependence: triangular and triangular2 modes are effective for mobile and convolutional models, but exp range mode — where the amplitude decays exponentially — drops sharply (mean 33.20%), making it the worst-performing non-plateau variant. ReduceLROnPlateau trails the cosine-based methods (mean 45.04%), consistent with its need for sufficient epochs for meaningful plateau detection within this budget.
5.6. Scheduler Family Analysis The results reveal clear trends. Smooth continuous decay schedulers — CosineAnnealingWarmRestarts and CosineAnnealingLR — consistently produce models in the upper accuracy quartile, confirming their established success on image classification benchmarks. Among StepLR 6
5.7. Architecture–Scheduler Interactions
7. Limitations
No single scheduler dominates across all architectures. Mobile and convolutional models (MobileNetV2/V3, EfficientNet, ResNet) strongly prefer CyclicLR, benefiting from its aggressive learning rate oscillation. GoogLeNet, BagNet, and InceptionV3 achieve peak performance under CosineAnnealingWarmRestarts or ReduceLROnPlateau, suggesting that periodic restarts and adaptive reduction benefit inception and dense-connection architectures. Transformer-based models (SwinTransformer, VisionTransformer) favour CosineAnnealingLR, consistent with their sensitivity to learning rate stability during attention warm-up. These interactions highlight the importance of architecture-specific scheduler selection rather than relying on a single global prescription.
Several limitations remain. First, evaluation is limited to five training epochs due to computational constraints; fulllength training may alter relative rankings, particularly for schedulers with long warm-up phases such as OneCycleLR. Second, the scheduler catalogue does not cover all strategies, such as learned schedules, warmup-cosine combinations common in large-scale transformer training, or newer adaptive methods such as Prodigy or Schedule-Free SGD. Third, all experiments use SGD with fixed lr = 0.01; interactions with Adam or AdamW optimizers are not explored. Fourth, the evaluation is restricted to CIFAR-10; generalization to ImageNet-scale datasets, detection, segmentation, and regression tasks remains to be investigated. Finally, architectures designed for non-classification tasks (UNet2D, Diffuser, ICNet) show systematically low accuracy on CIFAR-10, which may reflect task mismatch rather than the scheduler’s effect.
5.8. Discussion The 34-percentage-point gap between best (86.45%) and mean (52.52%) accuracy confirms that scheduler selection is a high-impact decision. The failure of ReduceLROnPlateau highlights that budget awareness is critical: schedulers designed for long training runs can actively harm performance in a five-epoch screening setting. The weight decay analysis shows that moderate regularization (5×10−5 ) provides marginal but consistent improvement, while heavy regularization slightly hurts. These findings provide practical guidance for large-scale neural network evaluation pipelines.
8. Conclusion We presented a systematic large-scale evaluation of learning rate scheduling strategies across 30 neural network architectures. By automating scheduler injection via programmatic source-code modification, we contributed and evaluated 3,938 model variants on CIFAR-10 within the LEMUR nn-dataset, demonstrating that scheduler choice exerts a strong, architecture-dependent influence on accuracy, with the best configuration reaching 86.45% top-1 accuracy and 237 variants exceeding 80%. Our key findings are fourfold. First, no single scheduler dominates across all architectures: CyclicLR leads on mobile and convolutional models, while CosineAnnealingWarmRestarts leads on inception-based architectures. Second, ReduceLROnPlateau is ineffective under five-epoch budgets, highlighting the importance of budgetaware scheduler selection. Third, among CyclicLR variants, exp range mode is harmful early in training and should be avoided in screening-phase evaluations. Fourth, moderate weight decay (5×10−5 ) provides marginal but consistent gains over no regularization. These results establish learning rate scheduling as a firstclass hyperparameter deserving the same systematic treatment as architecture search and optimizer selection. Future work will extend the framework to multi-objective scheduling (accuracy vs. convergence speed), incorporate warm-up combinations and adaptive optimizers such as Adam and AdamW, and generalise to ImageNet-scale classification, object detection, and segmentation.
6. Comparative Analysis 6.1. Effect of Weight Decay Comparing λ = 0 (mean 52.29%) against the best λ per architecture reveals that regularization consistently improves high-capacity models (ResNet, DenseNet, EfficientNet) while having negligible or slightly negative impact on lightweight models (MobileNetV2, ShuffleNet). This confirms that weight decay and scheduler type should be optimized jointly rather than independently.
6.2. Effect of Training Budget The five-epoch evaluation budget was validated by checking that the relative accuracy rankings of scheduler families are consistent with known results at longer training horizons. CosineAnnealing and CyclicLR schedulers retain their relative superiority at both short and longer training, while ReduceLROnPlateau is the main outlier whose ranking improves substantially with longer budgets. This suggests that our five-epoch findings are broadly transferable, with the exception of patience-based schedulers.
References [1] Arash Torabi Goodarzi, Roman Kochnev, Waleed Khalid, Furui Qin, Tolgay Atinc Uzun, Yashkumar Sanjaybhai
7
Raghuvir Duvvuri, Avi Goyal, Yashkumar Rajeshbhai Lukhi, Ahsan Hussain, Krunal Jesani, Usha Shrestha, Yash Mittal, Roman Kochnev, Pritam Kadam, Mohsin Ikram, Harsh Rameshbhai Moradiya, Alice Arslanian, Dmitry Ignatov, and Radu Timofte. LEMUR 2: Unlocking neural network diversity for AI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2026. to appear. 1, 2 [14] Chandini Vysyaraju et al. Enhancing LLM-based neural network generation: Few-shot prompting and efficient validation for automated architecture design. arXiv preprint arXiv:2512.24120, 2025. 2
Dhameliya, Yash Kanubhai Kathiriya, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. LEMUR neural network dataset: Towards seamless AutoML. arXiv preprint arXiv:2504.10552, 2025. 1, 2 [2] Xiaojie Gu, Dmitry Ignatov, and Radu Timofte. Resourceefficient iterative LLM-based NAS with feedback memory. arXiv preprint arXiv:2603.12091, 2026. 2 [3] Dmitry Ignatov and Radu Timofte. AI on the edge: An automated pipeline for PyTorch-to-Android deployment and benchmarking. Preprints, 2025. doi:10.20944/preprints202511.1831.v1. 1, 2 [4] Waleed Khalid, Dmitry Ignatov, and Radu Timofte. A retrieval-augmented generation approach to extracting algorithmic logic from neural networks. arXiv preprint arXiv:2512.04329, 2025. 2 [5] Waleed Khalid, Dmitry Ignatov, and Radu Timofte. From memorization to creativity: LLM as a designer of novel neural architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2026. arXiv:2601.02997. 2 [6] Roman Kochnev, Arash Torabi Goodarzi, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. Optuna vs code llama: Are LLMs a new paradigm for hyperparameter tuning? In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 5664– 5674, 2025. 1, 2, 4 [7] Roman Kochnev, Waleed Khalid, Tolgay Atinc Uzun, Xi Zhang, Yashkumar Sanjaybhai Dhameliya, Furui Qin, Chandini Vysyaraju, Raghuvir Duvvuri, Avi Goyal, Dmitry Ignatov, and Radu Timofte. NNGPT: Rethinking AutoML with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 5664–5674, 2026. 2 [8] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 2 [9] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017. 2 [10] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019. 3 [11] Usha Shrestha, Dmitry Ignatov, and Radu Timofte. From brute force to semantic insight: Performance-guided data transformation design with LLMs. arXiv preprint arXiv:2601.03808, 2026. 2 [12] Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications. SPIE, 2019. 2 [13] Tolgay Atinc Uzun, Waleed Khalid, Saif U Din, Sai Revanth Mulukuledu, Akashdeep Singh, Chandini Vysyaraju,
8