arXiv:2605.03722v1 [cs.LG] 5 May 2026
Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics 1st Xiang Meng
2nd Yan Pei
School of Computer Science and Engineering University of Aizu Aizuwakamatsu, Japan [email protected]
School of Computer Science and Engineering University of Aizu Aizuwakamatsu, Japan [email protected]
Abstract—We propose Evolutionary Dynamic Loss (EDL), a framework that learns a transferable classification loss in probability space using unlimited synthetic prediction–label pairs, without accessing real samples during the main loss pretraining stage. EDL parameterizes the loss as a lightweight network and is trained with a semantics-free ranking-consistency objective that enforces larger penalties for more erroneous predictions. To robustly search loss shapes, we optimize EDL via an evolutionary strategy and introduce chaotic mutation to improve exploration under noisy fitness evaluation. Experiments on CIFAR-10 with ResNet backbones show that EDL can serve as a drop-in replacement for cross-entropy and achieves competitive or improved accuracy, while ablations confirm that chaotic mutation yields faster convergence and better synthetic-pretraining metrics than normal Gaussian mutation. Index Terms—dynamic loss function, evolutionary computation, chaotic, adaptive optimization.
I. I NTRODUCTION Loss functions play a central role in supervised classification: they convert predicted probabilities into learning signals and thereby shape optimization dynamics, training stability, and generalization. Despite their broad success, widely used objectives such as cross-entropy and its variants are fixed analytic forms. When training conditions change—e.g., the hardness distribution, sampling bias, model capacity, label noise, or class imbalance—a fixed loss may no longer provide well-calibrated gradients or robust behavior. This raises a fundamental question: can we learn a loss function that does not depend on any particular dataset, yet remains scalable and transferable across downstream settings? Most existing approaches to learned losses or meta-learned objectives still rely on real task data and training trajectories as outer-loop supervision. Consequently, these approaches face three limitations: supervision is bounded by dataset size and privacy constraints, dataset-specific biases can be absorbed into the learned objective and harm cross-task transfer, and learning a loss is closer to searching for a function shape than to fitting conventional parameters, making the optimization process noise-sensitive and prone to undesirable shapes that may destabilize downstream training. In this work, we propose a loss-learning framework centered on two principles: distribution-free synthetic supervision in the probability space and evolutionary search for robust loss
shapes. Our key observation is that, for classification, a loss depends on the relationship between the predictive distribution and on the label rather than the raw input modality. Therefore, we learn a parametric Evolutionary Dynamic Loss (EDL) directly in probability space by generating unlimited synthetic pairs (p, y) without accessing any real samples. Figure 1 summarizes the overall pipeline. To obtain meaningful supervision without semantics, we impose a simple monotonicity principle: harder predictions should incur larger penalties. Concretely, we sample synthetic pairs, define a deterministic hardness proxy (e.g., based on the true-class probability), and train EDL using a smooth pairwise rankingconsistency objective that encourages EDL to preserve the same ordering. Moreover, we adopt a controllable mixture sampling scheme on the probability simplex to emphasize critical regimes that dominate training dynamics, including extreme-confidence cases and boundary-confusing predictions, which helps constrain the loss shape across the full confidence spectrum. Optimizing such a shape-search objective can be brittle with purely gradient-based methods under noisy pairwise estimates. We therefore employ an evolutionary strategy (ES) to explore EDL parameters globally. In addition, we introduce Logisticmap-driven chaotic mutation to modulate mutation amplitudes in a bounded yet non-periodic manner, improving population diversity and reducing premature convergence in early search stages. The outcome of this stage is a transferable loss prior learned without real data; for practical use, the learned loss can be plugged into standard classifier training as a dropin objective, optionally followed by a lightweight validationbased selection step to better match a target dataset. Contributions. (i) We propose a distribution-free loss-learning formulation that learns a parametric loss directly from synthetic prediction–label pairs in probability space via a rankingconsistency objective. (ii) We develop an evolutionary training strategy for loss-shape search, including Logistic-map-driven chaotic mutation, to improve global exploration and robustness under noisy pairwise supervision. (iii) We demonstrate that the resulting loss prior can be plugged into standard training pipelines and yields stable and competitive generalization across downstream settings. We will make our code and data publicly available upon
Classifier
label Data Generator:
…
predict𝑖 ~random(0,1) ቊ true𝑖 ~{0,1}
Train System
… … …… Loss Network
EDL
Loss1
Best
…
predict1
Train system
…
𝑙𝑎𝑏𝑒𝑙
1
Best
Same EDL 𝑙𝑎𝑏𝑒𝑙
2
predict2
EDL
Loss2
Compare Accuracy
Fig. 1. Overview of the proposed EDL pipeline. We learn a transferable loss prior from unlimited synthetic prediction–label pairs (p, y) in probability space using a ranking-consistency signal and evolutionary search with chaotic mutation. The learned loss can then be used as a drop-in objective for standard downstream classifier training, optionally with a lightweight validation-based selection or calibration step on the target dataset.
acceptance. II. R ELATED W ORK In supervised classification, performance can be improved by optimizing multiple components of the learning pipeline, including initialization [1], learning-rate schedules [2], activation functions [3], data augmentation [4] and network architectures. Among these factors, the loss function is particularly influential: it defines the training signal and the error geometry, thereby shaping optimization dynamics, training stability, and generalization. Accordingly, recent work has increasingly focused on learning and adapting loss functions to better cope with task diversity and distributional shifts [5]. A. Static Loss Functions Static loss functions are specified prior to training and remain fixed throughout optimization. Classical objectives such as cross-entropy and hinge loss are widely adopted due to their simplicity and interpretability [6], [7]. A substantial line of research refines static losses by reshaping penalty profiles or introducing auxiliary terms to improve robustness and to enhance discriminative learning. However, because their functional forms do not change, static losses can be suboptimal when training conditions vary across stages or when the data distribution shifts, as they cannot dynamically reallocate learning pressure across different error regimes. B. Dynamic and Learnable Loss Functions To address the limited adaptability of static objectives, dynamic and learnable loss functions aim to adjust error evaluation during training based on the data distribution or the model
state. Representative approaches [8] include learning adaptive coefficient structures that modulate the relationship between predictions and labels, meta-models [9] that stochastically select and combine handcrafted losses, and meta-networks [10] that generate transformation parameters to update a loss network. Hai [11] further propose an LSTM-based metaoptimizer to update a loss network online, enabling stagewise adaptation during training. In addition, margin-based Softmax losses have been studied under unified formulations, where random search and policy-gradient methods [12] are used to tune key hyperparameters for improved discriminative learning. Overall, these methods highlight the promise of learning the optimization objective itself; however, they typically rely on real datasets and task-specific training trajectories to provide outer-loop supervision.
a) Evolutionary and gradient-free loss optimization.: Complementary to trajectory-driven, gradient-based metalearning, another line of work searches loss shapes using gradient-free optimization. In particular, evolutionary computation can parameterize the loss and optimize it via mutation and selection, enabling global exploration without differentiability requirements. Representative examples include Genetic Loss-function Optimization (GLO) [5] and its continuous parameterizations such as TaylorGLO [13], as well as evolutionary loss-function frameworks such as ELF [14]. These perspectives motivate our use of evolutionary search to learn a transferable loss prior from probability-space supervision.
Training loss: ∆𝐿 ⋅ ∆ 𝐷 > 0
∆𝐿 = ℓ1 − ℓ2
𝑑1
Data Generator: 𝑢 ∼ 𝑈 0,1 , 𝑝 = 𝑢, 1 − 𝑢 ∈ Δ1, 𝑦 = (1,0)
ℓ1
𝑦1
∆𝐷 = 𝑑1 − 𝑑2
𝑑2
ℓ2
EDL
𝑝1
𝑦2 𝑝2
Fig. 2. Synthetic supervision for pretraining EDL via ranking consistency in probability space. Two synthetic prediction–label pairs (p1 , y1 ) and (p2 , y2 ) are sampled on the probability simplex. EDL outputs scalar losses ℓ1 and ℓ2 , and we form ∆L = ℓ1 − ℓ2 . A proxy hardness difference ∆D is computed from the same pairs, and pretraining encourages the ordering agreement ∆L ∆D > 0.
III. P ROPOSED M ETHOD We propose to learn a parametric loss function directly in probability space. Specifically, we model the loss as a evolutionary dynamic Loss that takes a classifier’s predictive distribution and the ground-truth label as input and outputs a non-negative scalar loss. To make the learned loss scalable and dataset-agnostic, we construct synthetic supervision by sampling prediction–label pairs on the probability simplex, and optimize EDL parameters via an evolutionary strategy under a ranking-consistency objective. During evolution, we adopt chaotic mutation driven by a Logistic map to diversify the population and improve global exploration. After training, EDL can be plugged into standard training pipelines as the optimization objective for downstream classifiers. Fig. 2 visualizes the core supervision signal used to train EDL without real data. Note that the condition ∆L · ∆D > 0 is not optimized as a discrete constraint; in practice we minimize a smooth pairwise ranking surrogate (e.g., softplus on −∆L · ∆D) to obtain stable gradients. The proxy hardness is computed deterministically from (p, y) (e.g., using the true-class probability or a distance between p and y), and serves only to define a relative difficulty ordering between two synthetic samples. This construction provides scalable supervision for shaping a monotone loss behavior across diverse confidence regimes, while remaining agnostic to any specific dataset. A. Evolutionary Dynamic Loss Network Let z denote the logits produced by a classifier, and p = softmax(z) ∈ ∆C−1 be the corresponding predictive distribution over C classes. Given a label y ∈ {1, . . . , C} with one-hot encoding ey ∈ {0, 1}C , EDL defines a learned loss Lϕ (p, y) = EDLϕ [p, ey ] , (1)
Fig. 3. Histogram of synthetic probabilities used for EDL training. We group samples by label and plot the distribution of predicted probabilities. The mixture sampling intentionally covers both near-extreme regions (close to 0 or 1) and intermediate-confidence regions, providing diverse supervision for learning a ranking-consistent loss shape in probability space.
where [p, ey ] denotes concatenation and ϕ are EDL parameters. We enforce non-negativity by applying a Softplus output layer, ensuring Lϕ (p, y) ≥ 0.
B. Synthetic Data Construction in Probability Space
To train EDL without using real samples, we generate synthetic training pairs (p, y). We sample labels uniformly, y ∼ Unif({1, . . . , C}), and draw p from a controllable mixture on ∆C−1 to cover diverse confidence regimes. In particular, we strengthen two critical regions that frequently dominate training dynamics: (i) extreme-confidence predictions with probabilities close to 0 or 1, and (ii) boundary-confusing predictions where the true-class probability is close to that of a competing class. The resulting synthetic probabilities exhibit concentrated mass near the extremes while still maintaining broad coverage over intermediate-confidence ranges (see Fig. 3), providing scalable supervision for shaping the loss behavior across the full spectrum of prediction patterns.
Algorithm 1 Training for EDL Require: Population size K, elite size Ke , generations G, batches per evaluation B, mutation scales (σhigh , σlow ), threshold τ , max attempts A Ensure: Best EDL parameters ϕ⋆ 1: Initialize population P = {ϕ(k) }K k=1 ; sample x0 ∼ Unif(0, 1); set t ← 0 2: for g = 1 to G do b (k) ) for all k using B synthetic batches 3: Evaluate F(ϕ 4: Select elites E ⊂ P as the Ke candidates with lowest b ϕbest = arg minϕ∈E F(ϕ) b F; 5: σ ← σhigh if Acc(ϕbest ) < τ else σlow 6: P′ ← E 7: while |P ′ | < K do 8: Sample parent ϕ ∼ Unif(E) 9: for a = 1 to A do 10: xt+1 ← 4xt (1 − xt ); t ← t + 1 11: Sample d ∈ {−1, +1}|ϕ| and ϵ ∼ N (0, I) 12: ϕ′ ← ϕ + σ xt+1 (d ⊙ ϵ) ▷ Chaotic mutation b ′ ) ≤ F(ϕ) b 13: if F(ϕ then break 14: end if 15: end for 16: Add ϕ′ to P ′ 17: end while 18: P ← P′ 19: end for b 20: return ϕ⋆ = arg minϕ∈P F(ϕ) C. Ranking-Consistency Objective and Fitness Our supervision is based on a monotonicity principle: more incorrect predictions should incur larger loss values. For a synthetic sample (p, y), we define an error-severity (hardness) score D(p, y) = 1 − py ,
(2)
where py is the predicted probability assigned to the true class. We then sample pairs (pi , yi ) and (pj , yj ) and compare the ordering induced by D and by EDL outputs. Let ∆Dij = D(pi , yi ) − D(pj , yj ),
sij = sign(∆Dij ),
∆Lij = Lϕ (pi , yi ) − Lϕ (pj , yj ).
(3)
The desired ranking consistency is s · ∆L > 0. To obtain a stable and continuous training signal, we define the fitness as a smooth pairwise ranking loss: h i F(ϕ) = E(i,j) softplus − s · ∆L , (4) where the expectation is over randomly sampled pairs. Minimizing F (ϕ) encourages EDL to assign larger loss to examples with larger error severity. For monitoring, we also report a trend-consistency accuracy: Acc(ϕ) = P(i,j) s · ∆L > 0 .
(5)
D. Evolutionary Optimization with Chaotic Mutation We optimize EDL parameters ϕ using a population-based evolutionary strategy over candidates P = {ϕ(k) }K k=1 . Since F(ϕ) in Eq. 4 is defined as an expectation over randomly sampled pairs, we evaluate each candidate by a Monte Carlo estimate computed on multiple synthetic batches: B 1 X 1 Fb ϕ(k) = B |Sb | b=1
X
(k) softplus − sij ∆Lij ,
(i,j)∈Sb
(6)
(k)
∆Lij = Lϕ(k) (pi , yi ) − Lϕ(k) (pj , yj ). At each generation, we keep the top Ke candidates with the lowest Fb as elites and generate the remaining candidates by chaotic mutation. a) Logistic-map chaos.: We use the Logistic map to generate a chaotic coefficient: xt+1 = r xt (1 − xt ),
r = 4.0, xt ∈ (0, 1).
(7)
b) Chaotic mutation operator.: Let θk denote the k-th learnable parameter in ϕ. Given a mutation scale σ, we mutate each parameter by θk′ = θk + σ dk xt+1 ϵk ,
dk ∈ {−1, +1}, ϵk ∼ N (0, 1), (8) where dk controls mutation direction and the chaotic coefficient xt+1 adaptively modulates the step size in a bounded yet non-linear manner. Compared with a fixed-scale Gaussian mutation, chaotic mutation yields richer perturbation patterns and improves early-stage exploration of diverse loss shapes. We use an adaptive mutation scale (switching between σhigh and σlow based on the best Acc) to balance exploration and refinement. The overall procedure is summarized in Algorithm 1. E. Algorithm Explanation Algorithm 1 repeats evaluation–selection–chaotic mutation: b keeps the best Ke elites, and it evaluates candidates by F, refills the population using Logistic-map-driven chaotic mutation with an adaptive σ. The final output is the EDL with the lowest estimated fitness. F. Theoretical Analysis a) Contrastive supervision in probability space: EDL is trained in probability space using synthetic pairs (p, y), which is sufficient because a classification loss depends only on the relation between predictive probabilities and labels. We reuse the hardness score D(p, y) in Eq.(2), the pairwise ordering variables (∆Dij , sij , ∆Lij ) in Eq.(3), and the ranking-consistency objective F(ϕ) in Eq.(4). Minimizing Eq.(4) enforces the monotonicity principle that harder predictions (larger D) should incur larger loss values (larger Lϕ ). Moreover, if there exists a strictly increasing g(·) such that L⋆ (p, y) = g(D(p, y)), then sign(∆L⋆ij ) = sign(∆Dij ) for any (i, j); hence the objective in Eq.(4) directly promotes a monotone loss shape with a positive ordering margin.
b) Extreme coverage via mixture sampling: Uniform sampling on the simplex may under-represent extremeconfidence regimes (py ≈ 0 or 1), which are often the most sensitive regions for gradient magnitudes and training dynamics. Our synthetic construction therefore draws (p, y) from a controllable mixture distribution (see Fig. 3), which increases the fraction of pairwise constraints involving nearextreme samples. This reduces extrapolation uncertainty at the boundaries and improves ordering consistency across the full confidence spectrum induced by D(p, y). c) Stability under stochastic fitness evaluation: Because F(ϕ) is an expectation over randomly sampled pairs, the ES b fitness estimate F(ϕ) in Eq.(6) is noisy. Averaging over B independent batches reduces estimator variance approximately inversely with B: B X
h i h i 1 b Var F(ϕ) ≈ Var Fb1 (ϕ) . B b=1 (9) Elitism further protects the current best candidate, and the optional non-degradation acceptance rule 1 b F(ϕ) = B
Fbb (ϕ),
b ′ ) ≤ F(ϕ) b F(ϕ
(10)
reduces random drift caused by unlucky perturbations under noisy evaluation. d) Why chaotic mutation helps (bounded, intermittent steps): Normal mutation uses fixed-scale Gaussian perturbations, while our chaotic mutation in Eq.(8) introduces a bounded, time-varying scale factor generated by the Logistic map in Eq.(7). Equivalently, each mutation step can be viewed as ∆θ = σ x ϵ, x ∈ (0, 1), ϵ ∼ N (0, 1), (11) where x follows the chaotic dynamics induced by Eq.(7). In the canonical chaotic regime, x is ergodic on (0, 1) with an invariant distribution that places substantial mass near both 0 and 1, yielding an intermittent step pattern: most mutations are strongly down-scaled (small steps) while a nonnegligible fraction are near unscaled (large steps). This creates a natural “many-small, few-large” exploration behavior: small steps stabilize selection by reducing sensitivity to noisy fitness estimates, whereas rare large steps help escape poor local loss shapes. Combined with elitist selection and the acceptance rule in Eq.(10), chaotic mutation provides a simple mechanism to balance exploitation and exploration during loss-shape search. IV. E XPERIMENTAL R ESULTS A. Experimental Setup a) Datasets: We evaluate EDL on CIFAR-10 [15], which contains 50000 training images and 10000 test images of size 32 × 32. b) Implementation details: Our method contains two learning components: (1) synthetic pretraining of the parametric loss in probability space, and (2) downstream classifier training using the learned loss. Table I summarizes the shared architectures and hyper-parameters. For the classifier, we use
TABLE I I MPLEMENTATION DETAILS (EDL). S TAGE 1 SYNTHETIC PRETRAINING IS THE FOCUS . Downstream (CIFAR-10) Backbone / epochs Optimiser / batch LR schedule Loss
ResNet / 200 SGD (momentum 0.9, wd 5 × 10−4 ) / 128 init 10−2 , step ×0.1 at {120, 160} CE or EDL
Stage 1: Synthetic pretraining (probability space) Classes / input Loss network Ranking supervision Synthetic budget
C = 10; input: predicted prob. vector + one-hot label MLP (2C-10-20-20-1) + Softplus Pairwise ranking on synthetic samples (Softplus ranking loss) A = 8192 samples, B = 4096 pairs per generation
Population / elites Generations Mutation (Normal) Mutation (Chaotic) Noise schedule
K = 6, Ke = 2 80 Gaussian perturbation Logistic-modulated perturbation (chaos factor x0 ∼ U (0, 1)) threshold τ = 0.95; σhigh = 0.20, σlow = 0.01
Optimisation (ES)
TABLE II R ESULTS ON DATASETS CIFAR-10 [15] FOR THE CLASSIFICATION TASK . A LL EXPERIMENTS ARE IMPLEMENTED WITH THE SAME SETTINGS . T HE BEST RESULTS ARE HIGHLIGHTED IN BOLD .
Method CE Smooth [16] L-M Softmax [17] L2T-DLF [8] GLO [5] TaylorGLO [13] ARLF [18] SLF [9] ALA [19] L2T-DLN [11] ELF [14] EDL-GD EDL-ES-Chaotic
ResNet8 87.6 87.9 88.7 89.2 87.7 89.5 89.8 90.7 90.4 90.6 ± 0.18 91.1 ± 0.14
CIFAR-10 [15] ResNet20 ResNet32 91.3 92.5 91.5 92.6 92.0 93.0 92.4 93.1 91.3 91.6 91.5 92.2 93.0 93.9 93.2 93.4 93.8 92.9 93.0 91.9 ± 0.08 92.3 ± 0.24 93.4 ± 0.17 93.9 ± 0.12
the standard ResNet backbone. For EDL, we use a lightweight MLP that maps the concatenated input [p; ey ] to a non-negative scalar via a Softplus head, following the same loss-network architecture as [14]. In the evolutionary setting, we maintain a population of candidate losses, keep elites, and generate offspring by mutation; chaotic mutation uses a Logistic map to modulate the mutation amplitude. Experiments run on a single NVIDIA GPU (RTX3060). c) Evaluation metric: Downstream performance is measured by Top-1 accuracy (%) on the CIFAR-10 test set. For the synthetic pretraining process, we report the best (lowest) fitness Fb and the trend-consistency accuracy Acc. B. Results a) Synthetic pretraining behaviour.: We first examine whether synthetic supervision can shape a meaningful loss prior. Across runs, EDL trained with evolutionary strategy exhibits steadily decreasing fitness and increasing trendconsistency accuracy, indicating that the learned loss preserves the desired monotonic ordering (harder predictions incur larger
loss). Compared with Gaussian mutation, chaotic mutation improves population diversity and typically finds better loss candidates with more stable convergence. b) Downstream CIFAR-10 classification.: We then plug the pretrained EDL into a standard ResNet training pipeline on CIFAR-10. Table II summarizes the test accuracy. EDLES-Chaos yields the best accuracy among learned-loss variants and improves over EDL-ES-Normal, demonstrating the benefit of chaotic exploration during loss pretraining. In contrast, EDL-GD, while achieving high synthetic ranking consistency, transfers less effectively to CIFAR-10, suggesting that directly optimizing EDL by gradient descent on synthetic pairs may lead to less robust loss shapes for real-data optimization. TABLE III N ORMAL VS . CHAOTIC MUTATION IN ES- BASED EDL SYNTHETIC PRETRAINING ( FITNESS ↓, BEST ACC ↑).
Mutation
Final best ↓
Mean fit ↓
Max acc ↑
Mean acc ↑
Std acc ↓
Epoch@ best ↓
Normal Chaotic
0.02810 0.01994
0.25742 0.15528
100.00 100.00
99.18 99.60
1.70 1.44
80 78
Gain
−29.05% −39.68% +0.00% +0.43 −15.32%
−2
C. Chaotic vs. Normal Mutation ablation experiments To isolate the effect of the mutation operator in synthetic EDL pretraining, we compare chaotic mutation (Eq. 7–Eq. 8) with normal Gaussian mutation under identical evolutionary settings. We keep the ES population size, elite selection, evaluation budget (generations and Monte Carlo batches), acceptance rule, and the adaptive mutation-scale schedule fixed; the only difference is whether the mutation amplitude is modulated by a chaotic coefficient (Chaotic) or uses the standard Gaussian perturbation with the same scale (Normal). Fig. 4 shows the trajectory of global best fit during synthetic pretraining, where global best fit denotes the best-so-far ranking-consistency fitness Fb found up to each epoch (lower is better); curves are reported as mean ± std across seeds. b confirming that probabilityBoth variants steadily reduce F, space supervision can shape a meaningful loss prior. Notably, chaotic mutation attains lower fitness earlier in the search (early-to-mid epochs), while both methods become close as they converge. This early advantage translates into a lower average fitness over epochs and a better best-so-far solution overall. Table III quantitatively corroborates this observation. Compared with normal mutation, chaotic mutation achieves a substantially lower final global-best fitness (0.01994 vs. 0.02810) and a much lower mean fitness across epochs, while maintaining saturated ranking accuracy (max best acc = 100%) with improved stability (lower std. best acc) and slightly faster attainment of the best solution.
Fig. 4. Ablation on the mutation operator in ES-based EDL pretraining. We plot global best fit (best-so-far synthetic ranking fitness; lower is better) over epochs, shown as mean ± std across seeds. Compared with normal Gaussian mutation, chaotic mutation reduces the fitness faster in the early-to-mid stage and achieves a slightly better final global-best fitness under the same ES budget, suggesting improved exploration and more robust loss-shape search.
V. C ONCLUSION We presented Evolutionary Dynamic Loss (EDL), a distribution-free loss pretraining framework that learns a transferable loss prior from unlimited synthetic prediction–label pairs (p, y) in probability space, without accessing real samples during the main pretraining stage. The loss is identified via a semantics-free ranking-consistency objective that enforces a monotone penalty ordering with respect to prediction hardness, which is directly reflected by steadily improved synthetic ranking fitness and consistency. To make the shape search robust under stochastic fitness evaluation, we optimize EDL with an evolutionary strategy and a Logistic-map-driven chaotic mutation that accelerates early-to-mid progress and improves aggregated pretraining metrics under the same ES budget; the resulting loss can then be plugged into standard CIFAR-10 training as a drop-in replacement and achieves competitive or improved Top-1 accuracy compared with strong learned-loss baselines. R EFERENCES [1] L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington, “Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks,” in International Conference on Machine Learning. PMLR, 2018, pp. 5393–5402. [2] L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in Artificial intelligence and machine learning for multi-domain operations applications, vol. 11006. SPIE, 2019, pp. 369–386. [3] P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for activation functions,” arXiv preprint arXiv:1710.05941, 2017. [4] Q. Wen, L. Sun, F. Yang, X. Song, J. Gao, X. Wang, and H. Xu, “Time series data augmentation for deep learning: A survey,” arXiv preprint arXiv:2002.12478, 2020. [5] S. Gonzalez and R. Miikkulainen, “Improved training speed, accuracy, and data utilization through loss function optimization,” in 2020 IEEE congress on evolutionary computation (CEC). IEEE, 2020, pp. 1–8. [6] C. Gentile and M. K. K. Warmuth, “Linear hinge loss and average margin,” in Advances in Neural Information Processing Systems, M. Kearns, S. Solla, and D. Cohn, Eds., vol. 11. MIT Press, 1998. [7] T. Zhang, “Statistical behavior and consistency of classification methods based on convex risk minimization,” The Annals of Statistics, vol. 32, no. 1, pp. 56–85, 2004.
[8] L. Wu, F. Tian, Y. Xia, Y. Fan, T. Qin, L. Jian-Huang, and T.-Y. Liu, “Learning to teach with dynamic loss functions,” Advances in neural information processing systems, vol. 31, 2018. [9] Q. Liu and J. Lai, “Stochastic loss function,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, 2020, pp. 4884–4891. [10] S. Baik, J. Choi, H. Kim, D. Cho, J. Min, and K. M. Lee, “Meta-learning with task-adaptive loss function for few-shot learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9465–9474. [11] Z. Hai, L. Pan, X. Liu, Z. Liu, and M. Yunita, “L2t-dln: Learning to teach with dynamic loss network,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023. [12] F. Wang, J. Cheng, W. Liu, and H. Liu, “Additive margin softmax for face verification,” IEEE Signal Processing Letters, vol. 25, no. 7, pp. 926–930, 2018. [13] S. Gonzalez and R. Miikkulainen, “Improved training speed, accuracy, and data utilization through loss function optimization,” in 2020 IEEE congress on evolutionary computation (CEC). IEEE, 2020, pp. 1–8. [14] X. Meng, Z. Hai, X. Liu, and Y. Pei, “Optimization design of adaptive loss function using evolutionary neural networks,” in International Conference on Neural Information Processing. Springer, 2025, pp. 321–335. [15] A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009. [16] T. Nguyen and S. Sanner, “Algorithms for direct 0–1 loss optimization in binary classification,” in International conference on machine learning. PMLR, 2013, pp. 1085–1093. [17] W. Liu, Y. Wen, Z. Yu, and M. Yang, “Large-margin softmax loss for convolutional neural networks,” arXiv preprint arXiv:1612.02295, 2016. [18] J. T. Barron, “A general and adaptive robust loss function,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4331–4339. [19] C. Huang, S. Zhai, W. Talbott, M. B. Martin, S.-Y. Sun, C. Guestrin, and J. Susskind, “Addressing the loss-metric mismatch with adaptive loss alignment,” in International conference on machine learning. PMLR, 2019, pp. 2891–2900.