arXiv CSopen access

Which Optimizer, At What Budget? A Tournament of Optimizers for Search-Based SE

Which Optimizer, At What Budget? A Tournament of Optimizers for Search-Based SE Kishan Kumar Ganguly and Tim Menzies

arXiv:2607.11705v1 [cs.SE] 13 Jul 2026

North Carolina State University [email protected], [email protected]

Abstract—Configuring and tuning modern software is unavoidable, expensive, and error-prone: a single system can expose hundreds of interacting options, and scoring one setting can mean a full build or test run. The standard response is automated optimization, but the number of available optimizers is large and growing. And some of the guidance for selecting among them is misleading: NSGA-II, for example, is widely recommended, yet other algorithms reach the same results using only 1/20th as many evaluations. To help practitioners make better choices about tools to configure their systems, we cluster 20 optimizers, based on six assumptions about the data. Next, we run a tournament across those optimizers, using 106 SE optimization tasks at four labeling budgets (taking 14,000+ CPU hours). We find that no optimizer wins outright. The best one migrates with the budget (from a geometric active learner when labels are scarce to differential evolution when labels are plentiful) so a winner “crowned” at one budget is wrong at another on up to half our tasks. Running such a tournament for every new domain is impractical due to its CPU cost. Fortunately, we find that those 14,000 hours can be replaced by a table lookup over two cheap-to-obtain task attributes (plus the labeling budget). Predictions from this table tie or beat a hindsight oracle on ≈ 75% of held-out tasks. To support open science, our tournament and replication package are open-sourced for SBSE researchers and practitioners at https://github.com/KKGanguly/OptimizerTournament. Index Terms—Search-based SE, algorithm selection, empirical study

I. I NTRODUCTION Configuring and tuning modern software is unavoidable and expensive. A modern system stacks languages, libraries, compilers, and deployment pipelines, each exposing its own flags and defaults: one open-source database used in this study carries 460 binary options, creating a configuration space of 2460 , larger than the number of stars in the sky [1]. Exhaustive search is impractical at this scale, so the field turns to searchbased software engineering (SBSE) to formulate this as an optimization problem. Under this paradigm,a fitness function scores candidate configurations and a search algorithm walks the space looking for good ones [10], [13]. We study black-box optimization: reasoning about a system from inputs and outputs alone. It is the workhorse of SE configuration, used to tune compilers and databases [46], [47], configure product lines and cloud systems [48], [49], and generate adversarial tests [50]. Over 100 such tasks are documented in the MOOT repository (Table I), representing more than a decade of SBSE work [2].

A practitioner who reaches for these tools confronts an overwhelming set of choices. Table II lists 20 optimizers drawn from the recent SE literature, and the No-Free-Lunch theorem guarantees that none of them wins everywhere [51], [52]. Which optimizer suits which task is, today, largely guesswork. The labeling budget compounds the problem. A single SE fitness evaluation can be costly: it may run an entire test suite for program repair [24], execute thousands of unit tests to score LLM-generated code [53], or profile system-level time and energy. Agentic and LLM-driven workflows push this cost higher. Still, An engineer may afford only dozens of evaluations, and the best optimizer at thirty evaluations need not be the best at two hundred. This exposes a gap: The gap. SBSE has no systematic way to organize optimizers by the assumptions they encode, and no cheap, budget-aware rule for choosing one on a given task. We close this gap with an assumption-indexed tournament. Reading the SBSE literature, we map each dominant search assumption to one representative optimizer, then run 20 blackbox optimizers as bets on six of our seven assumptions (Table II) over 106 MOOT tasks (Table I) at four labeling budgets, repeated 20 times: roughly 14,000 CPU hours (the largest such SBSE study we are aware of). We organize the results around four questions: • RQ1: Is optimization worth doing? Yes, even a few dozen labels close 31–75% of the gap between an uninformed choice and the best setting. • RQ2: Does the budget change which optimizer wins? Yes, the winner migrates from a geometric active learner (EZR [33]) when labels are scarce to differential evolution (DE [30]) when they are plentiful. And an optimizer selected at one budget falls outside the target budget’s top Scott-Knott tier on up to 50% of tasks. • RQ3: Do these optimizers really differ? Yes: singleobjective search beats multi-objective methods at equal budget. NSGA-II needs 1000 samples to reach what EZR reaches in 200. • RQ4: Can we cheaply predict the winner? Yes, a guide based on two cheap-to-obtain task attributes plus the budget ties or beats a hindsight oracle on ≈ 75% of held-out tasks, far above the ≈ 45% ceiling of expensive instance clustering features. This paper contributes:

TABLE II I NVENTORY OF THE OPTIMIZERS PLACED IN THE TOURNAMENT. F OR EACH WE GIVE THE ASSUMPTION IT BETS ON , ITS ROLE , THE ORIGINATING REFERENCE , REPRESENTATIVE software-engineering USES , AND THE VENUE OF THOSE USES . T HIS INVENTORY IS AN ARTIFACT OF THE LITERATURE REVIEW (S ECTION III) AND THE MENU THE TOURNAMENT OF S ECTION IV.

TABLE I 106 MOOT SE TASKS USED HERE , GROUPED BY DOMAIN . “ X / Y ” IS INPUTS / OUTPUTS ( DECISIONS / GOALS ). TASKS WITH y ≥ 2 GOALS ARE MULTI - OBJECTIVE . T HE SET IS THE SE CORE OF MOOT; NON -SE TASKS ( SALES , FINANCE , GENERIC ML) ARE EXCLUDED . Example tasks

x/y

106 Total

# rows

Optimizer

Bet

Role

Orig.

SE usage

Venue of SE uses

0.2k–86k

Hill Climbing Simulated Annealing (1+1)-ES Iterated Local Search Tabu Search Genetic Algorithm EDA PSO DE SMAC TPE LINE (kpp) SWAY EZR DODGE Random Search NSGA-II SPEA2 SMS-EMOA MOEA/D

A1 A1 A1 A2 A2 A3 A3 A3 A3 A5 A5 A7 A7 A7 A7 floor A4 A4 A4 A4

greedy trajectory annealed trajectory self-adaptive step trajectory restart escape memory escape crossover population distribution model velocity swarm difference-vector search RF surrogate BO density estimation centroid sampling distance bisection proximity active learning ϵ-pruning blind probing dominance sort strength archive hypervolume select decomposition

[8] [12] [15] [17] [19] [23] [26] [28] [30] [32] [34] [33] [36] [1] [22] [38] [39] [42] [43] [45]

[9], [10], [11] [13], [14] [16] [18] [20], [21], [22] [24], [25] [27] [29] [31] [33], [3] [35], [33] [33], [1] [36], [37] [1], [33] [22] [36], [1], [33], [3] [40], [41] [41] [44] [40]

TSE, ACM CSUR, ICSE FOSE, ASE IST GECCO Comput. & OR, IST, TSE TSE, TOSEM, ICSE-W GECCO ICSE ESEC/FSE FSE, ICSE ICSE, FSE FSE, JSS TSE, TSE JSS, FSE TSE TSE, JSS, FSE, ICSE TOSEM, JSERD JSERD JSS TOSEM

0.9k–167k

0.2k–4.7k

10k 1k–100k 93–20k

5.2k

spanning 7 SE domains; 25 single- and 81 multi-objective

Sources: [1], [2], [3], [4], [5], [6], [7].

A catalog of black-box optimizers organized by seven assumptions, read from the literature and used as an organizing device (Section III); • An assumption-indexed, budget-indexed tournament over 20 optimizers and 106 SE tasks, a reusable rig for SBSE experimentation (Section IV); • An easy-to-use, validated, budget-aware selection guide keyed on two cheap-to-obtain attributes (plus the labeling budget) (Section VI-D). • An open source reproduction package at https://github. com/KKGanguly/OptimizerTournament.

•

II. M OTIVATION Optimization is very effective for SE tasks, even when only a few examples can be labeled. The space of possible configurations is enormous: one system we studied has 460 binary flags, a space of 2460 options (which means there are

more configurations than there are stars in the observable universe [1]). An engineer cannot try them all, and does not have to. Take the honest baseline (pick one configuration at random), then spend just 30 labels (30 builds, or 30 benchmark runs) and let an optimizer choose what to try. Across our 106 tasks that small amount of data closes 31% of the gap between the random guess and the best setting. With 200 labels, still a vanishing fraction of spaces as large as 2460 , the methods of this paper close 75% of that gap (Figure 1, left panel). Labeling very few examples works for even very large problems. Returning to that example with 2460 options, with just 50 labels (0.5% of the data) the methods used in this paper can reach 80% of optimal [1]. Hence we say: The open question in SE optimization is never whether to optimize, only which optimizer earns those few dozen labels. But there is a catch. The right panel of Figure 1 warns

80 70

Best optimizer drifts as budgets diverge

74.5% the champion closes three-quarters of the gap to optimal by 200 evals

60

Evaluation budget

Regret over random removed (%)

Optimization pays, and pays more with budget

52.0%

50

42.3%

40

30.7%

30 20

30

—

28%

47%

50%

50

28%

—

38%

42%

100

47%

38%

—

25%

200

50%

42%

25%

—

50

100

200

40

30

10 0

EZR

30

EZR

50

DE

100

Evaluation budget (labels)

DE

200

50

30

20

10

% of 106 tasks whose best differs

# Domain / type

25 Specific software SS-A . . . SS-X, 3–88 / 2–3 config. billing10k 12 Performance 7z, BDBC, LLVM, 9–35 / 1 (systems) PostgreSQL, x264, redis . . . 11 Cloud / system tun- Apache, SQL, 3–39 / 1 ing HSMGP, rs/sol/wc . . . 35 Project health Health-{Issues, 5 / 2–3 PRs, Commits} 11 Feature models Scrum{1,10,100}k, 124–1044 / 3 FFM-*, FM-* 10 Process / cost models nasa93dem, 9–27 / 3–5 coc1000, POM3a--d, XOMO 2 Testing test120, test600 9/1

0

Evaluation budget 58% of tasks change their best optimizer at least once

Fig. 1. Optimization can be very effective for SE tasks. Left panel: In 106 MOOT SE tasks, reasoning on just 30 labels already closes 31% of the gap between a random guess and the best setting; and 200 labels does even better (reaches to 75% of the optimum). Right panel: the best optimizer changes with budget (58% of tasks switch at least once so no single choice is stable across all budgets). For this reason, this paper builds a selection rule that can guide practitioners on when to use which optimizer.

that deciding which optimizer to select is a nuanced task. That figure compares, for each pair of budgets x and y, the percentage of tasks whose best optimizer at x differs from their best at y. Note that, very often, the “best” optimizer changed when we can access more labels. For this reason, this paper builds a selection rule that can guide practitioners on when to use which optimizer. The rest of this section offers other motivation for this exploration of optimization for SE tasks.

sense if that surface is locally smooth, so small steps yield small, informative changes [8]. Other methods bet on decomposability, on a cheap surrogate standing in for evaluation, and so on (Section III reads one such assumption out of each optimizer’s originating paper). A benchmark tests the tool, but what a practitioner needs to know is whether the assumption holds for their task and budget. This is why a study that merely ranked N algorithms would be incomplete: it would name a winner without explaining

A. Why Evaluate Optimizers on SE Data?

SE needs its own optimizers. SE optimization is budgeted, rugged, and black-box, so methods proven on smooth ML losses may not work in our domain. Choosing the optimizer is itself the problem, and No-FreeLunch guarantees no single choice is always right.

TREE 1 — Single-Objective

Hill Climbing A. Trajectory A1/A2

M1

SA

(1+1)-ES (L,M) SA (F) M2

(M,F)

(1+1)-ES

SF-A

Tabu Search M3

TS

B. Population A3

ILS

(L)

Local (SF-AB)

EDA

PSO (L) DE (M,F)

M1

GA

(M,F)

SF-B

PSO (L)

M2

DE

PSO (L) DE (M,F)

EZR (L) DE (M,F)

SMAC

SF-C

TPE

SMAC Rep. (SF-CD)

D. Samplers A7

M1

EZR (L) SMAC (M,F)

KPP (LINE) SF-D (F)

(L, M)

SWAY M2

DODGE

EZR

GRAND FINAL EZR (L) DE (M,F)

Random

TREE 2 — Multi-Objective NSGA-II E. Multi-Obj. A4

Most SE optimizer studies end in a leaderboard which states that some tool A beat tools B and C on a handful of benchmarks, so A is reported the winner. But a leaderboard entry is a fact about a tool under one setup. It does not say why the tool won, and the “why” is the only part that carries to a new task. Two problems make the leaderboard, on its own, a weak basis for choosing an optimizer. First, the comparison is often unfair: a new method is matched against recent rivals, not the simplest baseline. Only one in twenty LLM-for-SE papers used a simpler approach [57], and a plain sampler has matched far heavier search whenever it was finally tested [36], [7], [1]. With the baseline absent, a win is unreadable: the method may have won because it suited the problem, or merely because its opponents were weak. Second, and more fundamental, even a fair win is condition-specific: across our 106 tasks no family of optimizer leads on more than 37, and the leader at B=30 is usually not the leader at B=200 (§VI-B). What does transfer is the reason behind the win. Every optimizer rests on a core assumption about the domain; e.g. as studies in this paper, assumptions about the shape of the fitness landscape, the surface mapping each configuration to its measured quality. Hill climbing, for instance, only makes

T1-SO Champ

(M,F)

EZR

B. Why we Compare Assumptions, not Tools?

Winner annotations green tick: match winner Green text: stage winner L: low budget (≤ 50) M: medium budget (100) F: full budget (200)

(L)

C. Models A5

SE optimization has certain features which require special kinds of optimizers [54]. Every “if” statement divides the internal state space of a program into different regions, so there is no single gradient to follow. This means that one cannot (e.g.) differentiate a for loop. The search space is marked by discrete options (e.g. boolean configuration options) and is interaction-heavy rather than smooth [55], [56]. Limits are hard constraints, not soft penalties (one byte over a memory cap is a failure, not a gradient). And evaluating any candidate means building and running it [13]. Together, these make SE a budgeted, complex, black-box problem, the kind of problem where the No-Free-Lunch bites hardest [51]: the optimizer that wins in ML may not win here [54].

M1

MOEA/D

T2-MO Champ

SMS-EMOA M2

SPEA2

NSGA-II (L) SPEA2 (M,F)

Fig. 2. The assumption-indexed tournament tree (20 optimizers). Leaves are black-box optimizers, colored by family; each internal node is a head-tohead match isolating one landscape assumption. TREE-1 ranks the singleobjective methods through four assumption branches: Trajectory (A1/A2, with the continuity chain Hill Climbing → SA → (1+1)-ES feeding the Afinal against the Tabu/ILS diversification winner), Population (A3, including DE), Models (A5, comparing SMAC and TPE), and Samplers (A7). Their champions meet in two semifinals, Local and Representation, to give the T1SO champion. TREE-2 ranks the multi-objective methods (A4) into the T2MO champion, and the Grand Final compares the two. Green labels give each match’s budget-tagged winner (L≤50, M=100, F=200). The single-objective champion migrates from EZR at tight budgets to DE once the budget grows. For more details on A1–A7, see Section III-A.

what about the domain made it win, and so could not guide anyone facing a different domain. We therefore organize the experiment around the binary tree of Figure 2. Each branch of the tree is a separate experiment with two optimizers that differ in essentially one assumption. The results from these experiments then reads as evidence about which assumption is useful, and under which budget (knowledge that carries to a new task) rather than as one more brand ranking. Figure 2 summarizes domain assumptions using the notation described in Section III-A. The green check marks show winner results represented later in this paper, From tools to assumptions. A leaderboard says which tool won, not why. So, the conclusion may not transfer. We compare assumptions instead: each match isolates one, across budgets, against a random floor. The byproduct is a cheap, budget-aware selection guide. C. Why Evaluate with Constrained Budgets? Another special feature of this evaluation is that we constrain evaluations budgets to just a few dozen. Why? In SE, enumerating candidate choices is often cheap; but understanding their consequences is not. A single evaluation can be very expensive e.g. scoring a build can mean a recompile and a full test-suite run, and labeling one row of a project-health table can cost an expert an hour or more [1]. It is possible to ask human experts for labels but that process is slow and often error-prone [58]. For example, labeling just one row of a project-health table can cost an expert an hour or more, so even a few dozen of cases can take up a week [6], [59]. Some researchers mine historical logs to deliver labels in bulk. The quality of such labels is highly variable. One study found that 90% of supposed technicaldebt “false positives” were themselves mislabeled [60]. The same pattern recurs across security [61], static analysis [62], and defect datasets [63]. As to using other methods to autolabel examples: regex-based heuristics can be very cheap but coarse-grained [64]. LLMs can assist but cannot serve as the final word [65]. This labeling cost is why it is prudent to score optimizers by the number of evaluations they require. It also explains why a realistic evaluation budget may be dozens, not the thousands a generic hyperparameter optimization benchmark assumes [66]. We note that the budget B is not a constant to fix and forget: Figure 1 (right panel) puts numbers on the catch—the single best optimizer differs between B=30 and B=200 on 50% of tasks, and 58% of tasks change their best optimizer at least once across budgets. A comparison that fixes one budget therefore risks conclusions that do not hold at other budgets. Budget is a first-class variable. Evaluations cost CPUyears and expert hours. Engineer may be able to afford only dozens of evaluations, and which optimizer wins moves as that budget increases. An honest selection guide must be indexed by the budget, not stated for a single one.

TABLE III L ANDSCAPE ASSUMPTIONS BEHIND SBSE OPTIMIZERS : READ FROM EACH METHOD ’ S ORIGINATING PAPER , WITH THE LIMITATION THAT BREAKS EACH AND AN SE TASK WHERE IT MATTERS . ID

Assumption

A1

Local continuity

A2 A3 A4 A5 A6 A7

Limitation & SE example

Fails on rugged landscapes. Compiler-flag tuning [46]. Memory / diversification Fails on deceptive neighborhoods. Test-suite minimization [20]. Building-block decomposability Crossover destroys coupled structure. Program repair [24]. Pareto incomparability Wasteful under clear preferences. Software remodularization [40]. Surrogate feasible Surrogate assumptions fail under few labels. DBMS Knob tuning [70]. Sequential reward signal Fails under heavy delay. Cloud DB tuning [49]. Low intrinsic dimensionality Fails when data fills the space. Highdim. feature models [48].

III. R ELATED W ORK AND O PTIMIZER S ELECTION The no-free-lunch theorem makes exhaustive comparison pointless [51]: rather than enumerate the vast SBSE and hyperparameter-optimization menu [1], [13], we categorize it and sample one representative per category. Reading the originating paper of each optimizer family, we asked one question: what must be true of the fitness landscape for this method to make sense?. This clustered the answers into seven assumption families (Table III), each of which is a testable bet paired with the limitation that breaks it (and an SE example where it matters). From each family we take representatives that are canonical for some assumption (a basic version, not a tuned hybrid, so a win is attributable to the assumption), have a clear specification that we follow as published (a public implementation where one exists, otherwise the originating paper’s algorithm implemented faithfully), and carry recent SE relevance. This yields the 20 black-box optimizers of Table II, each mapped to its assumption, role, origin, and SE uses. These cover six of the seven assumptions; the seventh (A6, reinforcement learning) is described below for completeness but excluded from the tournament, since it needs thousands of environment steps and so cannot operate at the few-dozen-label budgets we study. We leave to future work methods that require resources our setting cannot provide: high-dimensional Bayesian [67], [68], causal [3], [69], knob [70], [47], [49], reinforcementlearning [71], LLM-based [72], [73] optimizers. A. Related work and Assumption Families Trajectory and local search (A1, A2). These methods keep a single candidate and repeatedly replace it with a better neighbor, in effect walking step by step across the landscape rather than maintaining a population. Hill climbing is the textbook embodiment of the continuity bet: it always steps to the best nearby point and halts when none improves [8]. Simulated annealing adds a cooling schedule that occasionally accepts a worse move (more readily early on) so the search can escape local optima [12]. It has been used for

search-based refactoring [40]. The (1+1) Evolution Strategy keeps the same single-incumbent form but replaces the fixed neighborhood with a self-adaptive Gaussian step that tunes its own size (the 1/5 rule) [15], and has been used for test-data generation [16]. A diversification family answers ruggedness by avoiding places already seen: tabu search forbids revisiting recent points, with an aspiration rule that lifts the ban for an exceptional gain [19], and appears in configuration transfer and structural testing [21], [20]. Iterated local search kicks the incumbent into a new region and re-optimizes [17], [18]. These methods are cheap per step and warm up fast. Population and evolutionary search (A3). These methods keep a population of candidates and breed new ones from the fittest, so good partial solutions can spread and combine. Genetic algorithms recombine two parents under the buildingblock bet that good solutions are assembled from good parts [23], extended to programs by genetic programming and automated repair [74], [24]. Differential evolution forms a new candidate by adding the scaled difference of two population members to a third, keeping the child only if it beats its parent [30]. Particle swarm nudges each candidate along its own best and the swarm’s best directions [28]; estimation-ofdistribution algorithms replace recombination with a probability model fitted to the best-so-far and then sample from it [26]. Multi-objective and Pareto search (A4). When objectives conflict there is no single best configuration, only trade-offs: the Pareto frontier are solutions where no objective can be improved without worsening another. These methods evolve a population to approximate that frontier. NSGA-II sorts candidates into nondomination layers and breaks ties by crowding, favouring sparser regions so the frontier stays diverse [39]. SPEA2 keeps an external archive in which each solution is scored by how many others it dominates [42]. MOEA/D splits the problem into many scalar subproblems and solves them together, each pulling toward a different part of the frontier [45], while reference-point methods use fixed directions to scale to many objectives [75]. SBSE has long recommended full Pareto search over weighted or aggregated methods for multiobjective problems [76]. This has only recently been questioned. For instance, meta-multi-objectivization, can reshape the problem so that fewer objectives suffice [77]. Surrogate and model-based search (A5). These methods fit a cheap surrogate model of the response from the points labelled so far, then use it to decide what to evaluate next, spending real evaluations only where the model expects a payoff. Bayesian optimization fits a probabilistic surrogate and picks the next point with an acquisition function that trades expected gain against uncertainty [78]. SMAC uses a Random Forest surrogate, letting it handle the categorical and noncontinuous spaces that Gaussian processes handle poorly [32]. TPE instead models the densities of good and of ordinary configurations and samples where the ratio favors good [34]. The configuration tuning community has produced many taskspecific tuners: knob tuners [70], [47], sequential model-based configuration [7], causal approaches [69], [3], configurationspace reduction [79], drift adaptation [80], co-evolutionary

tuning [81], and Bayesian compiler autotuning [35], [82]. Crucially, recent work shows surrogate’s predictive accuracy is a poor proxy for quality of the configurations a method ultimately selects [37], [83], which motivates judging each optimizer by the configurations it delivers rather than by any internal model’s loss. Reinforcement learning (A6). Reinforcement learning learns a policy by trial and error from a reward that may arrive only after a sequence of actions [84], [85]. In database tuning this yields end-to-end tuners that treat one full benchmark run as a single environment step and its measured quality as the reward [71], [49]. Data-light and geometric search (A7). These methods bet that the data has low intrinsic dimensionality (i.e., it collapses into a few regions), so a few dozen well-chosen labels capture most of its structure, and often they read the geometry of sampled points rather than fit a model [1]. LINE picks centroids covering high-variance regions [33]. EZR is a distance-based active learner that labels points near the boundary between the current best and the rest [1], [33]. DODGE discards configurations that fall in the same ϵ-bin and excels when intrinsic dimensionality is low [22]. SWAY recursively bisects the space by distance to sample a few representatives [36]. The baseline, Random probing labels a random budget and returns the best [38], and can often yield surprisingly strong results [38]. Algorithm selection and instance-space analysis. The dominant alternative to running a tournament is to predict the winner from cheap problem features. Instance-space analysis projects each problem into a low-dimensional space of landscape features and draws “footprints” marking where each algorithm wins [86]; it has been applied to fitness landscapes generally [56], [55] and to search-based test generation specifically [87]. We use the same feature family in RQ4, but we test the predictive claim rather than assume it, and we add the variable this literature omits: the evaluation budget. What we exclude, and why. No study can explore all algorithms since there are so many of them. We elect to exclude causal tuners such as PromiseTune [3] and Unicorn [69]: these are hybrids that build a causal model on top of their search, whereas our inventory takes basic optimizers whose result traces to a single assumption. We also exclude multi-fidelity methods such as BOHB [88] and DEHB [89], which need a fidelity ladder the tabular tasks do not provide. Further, we do not study high-dimensional Bayesian methods such as TuRBO [68], whose machinery targets hundred-plusdimensional spaces far larger than ours. Nor do we study LLM-driven optimizers [90], [91], [72], since these have such a large computational cost it would be prohibitively expensive (to say the least) to run this study on those kinds of algorithms. IV. M ETHOD : A SSUMPTION -I NDEXED T OURNAMENTS A. Evaluating a configuration MOOT tasks are represented as lookup tables, not models. To evaluate new solutions, the simplest approach would be to use dependent values from the nearest neighbor of the

TABLE IV H YPERPARAMETERS , SET TO EACH METHOD ’ S ORIGINATING - PAPER DEFAULTS TO AVOID A TUNING - THE - TUNER CONFOUND . Optimizer

Key settings (defaults)

Hill Climbing Simulated Annealing Iterated Local Sea. Tabu Search Genetic Algorithm (1+1)-ES EDA PSO SMAC DE TPE LINE (kpp) SWAY EZR DODGE Random Search NSGA-II SPEA2 SMS-EMOA MOEA/D

greedy step; full neighbourhood geometric cooling 0.95; random neighbour perturbation on stagnation; accept-better tabu tenure 7 pop. 10; uniform crossover 0.9; mut. 1/n single parent; Gaussian mut.; 1/5 rule pop. 10; univariate model; top-50% pop. 10; w=0.7, c1 =c2 =1.49 RF surrogate; EI acquisition; init 10 pop. 10; F =0.8, CR=0.9; rand/1/bin γ=0.25; startup 10 k from budget; centroid init recursive median-distance bisection distance acquisition; init labels 4 ϵ=0.2; ϵ-bin tabu uniform sampling (floor) pop. 10; SBX; polynomial mutation pop. 10; archive 10 pop. 10; hypervolume contribution pop. 10; 5 neighbours; Tchebycheff

proposed new solution. While this evaluation approach has some precedence in the literature (see Pfisterer and Zela et al. [92], [93]), it tends to reward solutions that are close to existing solutions. Hence, it may not be a fair oracle from membership-query methods like SMAC that propose points between rows. An alternate approach, taken by Eggensperger and Hutter and Nair et al. [94], [95], [32], [7] is to build a random forest ensemble using all the training data. This is useful since SE problems contain categorical and numeric variables, which random forests can handle. Also, SE problems are rarely smooth since every “if” statement in a program can divide the internal state space into yet another region with different properties. Random forests are good for multi-branching and building one model per branch. For these reasons, this study uses random forests as the solution oracle. As to other details: to avoid a tuning-the-tuner conflation, we run every method at its originating-paper defaults (Table IV). We sweep four evaluation budgets, B ∈ {30, 50, 100, 200} measured configurations, and repeat every (method, task, budget) cell over 20 random seeds; all reported statistics are over those repeats. The full design is thus 20 optimizers × 106 tasks × 4 budgets × 20 seeds, which, with the multi-objective tree and the all-pairs ranking used as a check, comes to the order of 1.8 × 105 search runs. Each run also pays for fitting and querying the per-task surrogate. Aggregated over the per-run wall-clock times in our logs (summarized in Figure 3), the study consumed roughly 14,000 CPU hours. (Aside: We mention this CPU cost because it is itself a finding. It is why practitioners do not run this tournament themselves, and therefore why the cheap-to-obtain guide (at the end of this paper) that can cheaply reproduce our verdict is of much practical value.)

B. Tasks and scoring Each task is a set of configurations with one or more measured objectives, from the SE datasets of MOOT (Table I). We orient and normalize every objective to [0, 1] so that 0 is best, and score a configuration by its distance to heaven: the normalized Euclidean distance to the ideal point where every objective is at its best, s
2 1 X o(c) ∈ [0, 1], 0 = best, d2h(c) = o(c) , |O| o∈O (1) where O is the objective set and o(c) the oriented, normalized value of objective o. Lower is better and the ideal point scores 0. We use d2h because it collapses single- and many-objective quality onto one axis, letting the same tournament rank a hill climber against an NSGA-II run. It rewards configurations balanced near the ideal rather than extreme on one axis. It follows the data-light SE line we build on [1], [6], [7]. Why best-point, not the frontier. For multi-objective tasks a method may return a set of nondominated configurations, but a team ships one. We therefore score each method by the single returned configuration with the lowest d2h, the point a practitioner would actually deploy. This reduction could in principle flatter scalar methods, so we do not take it on faith: in RQ3 we re-run the multi-objective comparison with frontier-aware metrics (IGD, GD, hypervolume, Section V). The single-objective methods still lead: at equal budget EZR/ DE win on IGD, GD, and hypervolume too. So the best-point reduction is not what drives the single-versus-multi result. V. E XPERIMENTAL R IG Regret reduction metric. Raw d2h values are not comparable across tasks of different difficulty, so in addition to d2h, we report how much of the available regret a method removes. For a task, let d2hrand be the expected quality of random search at the same budget and d2h⋆ the best achievable quality on the task. A method achieving d2hm removes the fraction ρ =

d2hrand − d2hm , d2hrand − d2h⋆

(2)

of the random-to-optimal gap, with ρ = 1 matching the best configuration on the task and ρ = 0 matching blind sampling. We report the median ρ over tasks at each budget. This framing keeps the random floor visible at all times and makes “earned the right to beat random” measurable. Multi-objective sanity metrics. To check that the bestpoint reduction of Section IV-B does not distort the singleversus-many comparison, RQ3 additionally scores the multiobjective methods with three frontier-aware measures: generational distance (GD), the mean distance from the returned set to the reference frontier constructed from the best-known points pooled over all optimizers’ results, inverted generational distance (IGD), the mean distance from the reference frontier to the returned set, which rewards coverage, and hypervolume (HV), the volume of objective space dominated by the returned set relative to the best-known reference set pooled over all

Runtime Distribution Across Budgets

Mean Runtime in Seconds, log scale

Eval Budget 10

2

10

1

10

0

10

−1

10

−2

30 50 100 200

rch

Sea

dom

Ran

TS

PSO

DE

AC

SM

EZ

R

2

GA

NS

SPE

A2

Optimizer

Fig. 3. Search cost is real and it is the reason a free guide matters. Per-run wall-clock distributions for every wining optimizer at each budget, aggregated to the ≈14,000 CPU hours the full study consumed.

optimizers’ results [42], [43], [75]. Lower GD and IGD and higher HV are better. These are reported only as a cross-check; the tournament itself is decided on d2h. Statistics. Every match and every ranking uses the same nonparametric procedure. Over the 20 seeds we apply ScottKnott clustering [96], which recursively bisects the methods into ranked groups and accepts a split only when a bootstrap test finds the groups distinct and the Cliff’s delta effect size between them is non-negligible. Methods that land in the same group are statistically indistinguishable and share a rank. VI. R ESULTS We organize our results around the following questions: • RQ1: Is optimization worth doing? • RQ2: Does the budget change which optimizer wins? • RQ3: Do these optimizers really differ? • RQ4: Can we cheaply predict the winner? Of these, RQ4 is of the most practical importance. Figure 3 shows the runtimes for a sample of the optimizers used in this study. Note that the runtimes increase exponentially with the labeling budget. Exponential evaluation costs mean that most practitioners cannot afford to repeat this paper’s study. They require some way to quickly peek at a problem, then decide what optimizer to use. A. RQ1: Is optimization useful? As argued in Figure 1: even a few dozen labels are enough to find large optimizations. Hence, we report from Figure 1:

RQ2. The budget chooses the optimizer. When data is scarce, B ≤ 50 EZR works best. But at larger budgets B ≥ 100 DE works best. C. RQ3: Do these optimizers really differ? A discussion of all the results across the tree in Figure 2 (explaining every branch, every win, every tie) is its own paper. In this section, we report on three standout features. A few optimizers are significantly better. As reported above EZR and DE do stand out from the pack. But many optimizers tie. Across the branches of Figure 2, the great majority of head-to-head matches end in a ScottKnott tie: at a given budget, both contestants land in the same rank. These methods use very different machinery, yet on SE data, at budgets an engineer can afford, they mostly deliver the same results. This is a match-level echo of the familylevel result discussed in §II-B (no family leads on more than 37 of 106 tasks) and fits the No-Free-Lunch view: once the budget is tight and the data is fixed, many of these methods are interchangeable, so the field is narrower than the literature’s steady stream of new optimizers implies. A few methods still

TABLE V T OURNAMENT WINNERS BY STAGE AND BUDGET (S COTT-K NOTT, 20 SEEDS ). T HE SINGLE - OBJECTIVE CHAMPION IS EZR AT TIGHT BUDGETS , DE ONCE THE BUDGET GROWS . B OTTOM ROWS : MEDIAN REGRET REDUCTION ρ AND CHAMPION MEAN d2h. Evaluation budget B

RQ1. The payoff from optimization are real, even with very few labels. At budgets of just B = 30 and B = 200, our best optimizers improve optimization from 30.7% (at B=30) to 74.5% (at B=200) of the random-to-optimal regret. B. RQ2: Does budget change which optimizer wins? Table V applies our statistical methods at each level of the trees in Figure 2. As can be seen, as the budget increases, the “best” optimizer changes:

Bracket stage

30

50

100

200

SF-A (Trajectory) SF-B (Population) SF-C (Models) SF-D (Samplers) SF-AB (Local) SF-CD (Representation) T1-SO champion T2-MO champion

TS PSO SMAC EZR PSO EZR EZR NSGA2

TS PSO SMAC EZR PSO EZR EZR NSGA2

TS DE SMAC EZR DE SMAC DE SPEA2

TS DE SMAC EZR DE SMAC DE SPEA2

Grand Final

EZR

EZR

DE

DE

Median ρ (%) Champion mean d2h

30.7 0.21

42.3 0.19

52.0 0.17

74.5 0.16

TABLE VI S INGLE - VS . MULTI - OBJECTIVE SEARCH ON 81 MULTI - OBJECTIVE TASKS ( BEST- POINT d2h, IGD, GD, HV; MEDIAN OVER TASKS ). AT EVERY MATCHED BUDGET THE SINGLE - OBJECTIVE METHODS LEAD OR TIE ; MULTI - OBJECTIVE SEARCH NEEDS 5× THE BUDGET TO REACH EZR@200, AND EVEN THEN EZR@1000 STAYS AHEAD . objectives

B

best-pt d2h↓

IGD↓

GD↓

HV↑

B=200 EZR DE NSGA-II SPEA2 MOEA/D SMS-EMOA

single single multi multi multi multi

200 200 200 200 200 200

0.18 0.16 0.22 0.29 0.22 0.40

0.12 0.13 0.16 0.23 0.16 0.31

0.05 0.03 0.08 0.14 0.07 0.15

0.97 0.93 0.91 0.73 0.92 0.48

B≈1000 (5×) EZR DE NSGA-II SPEA2 MOEA/D SMS-EMOA

single single multi multi multi multi

1000 1000 1000 1000 1000 1000

0.15 0.15 0.18 0.25 0.18 0.37

0.10 0.13 0.10 0.21 0.11 0.28

0.03 0.02 0.02 0.08 0.02 0.11

1.01 0.94 1.01 0.80 0.98 0.52

Method

At equal budget the single-objective methods win or tie every metric. Multi-objective search needs about 5× the budget to catch up for d2h. Let d be the best-known front (pooled over all optimizers) and a one optimizer’s returned set. GD is the mean distance from each point in a to its nearest in d; IGD reverses this, from each point in d to its nearest in a (rewarding coverage); HV is the objective-space volume a dominates relative to a worst-case reference point. Lower GD and IGD are better (closer to the best-known front); higher HV is better (more of the space dominated).

separate from the pack, and which one leads shifts with the budget, the structure our guide is built to capture. And a few of the losses are surprising; e.g. single vs. multi-objective. The SE optimization literature holds that multi-objective methods beat methods that collapse many goals into one aggregate. To our knowledge this has not been checked before on a large SE sample over such a tournament structure. Our setup lets us test it directly: we treat each multiobjective task as a single-objective one by aggregating its objectives into a single d2h score (Equation 1, as EZR does), run the single-objective methods on it, and then score the full set of configurations each method evaluated over its search trajectory with frontier-aware metrics (GD, IGD, HV), defined in Table VI. This puts single- and multi-objective methods on the same frontier-based footing.Note that: • •

lower d2h, GD and IGD are better; higher HV is better.

Table VI compares single- and multi-objective reasoners. Best per column are shown in bold. On all of the measures, multi-objective methods cannot surpass single-objective ones. Even when using 5× more budget, none of the multi-objective optimizers can surpass single-objective ones on equal budgets. And the cost asymmetry is stark, Table VI shows that NSGAII needs 1000 samples to match what EZR reaches in 200.

RQ3. Yes, but less than the literature suggests. Most matches tie: at SE budgets, very different optimizers deliver the same result, and only a few separate themselves from the pack. Overall, EZR and DE lead but which optimizer is best depends on the task and budget and no single method wins everywhere. Even the single- vs. multi-objective divide collapses: at least one single-objective search matches or beats multi-objective methods at equal budget.

Fig. 4. Static instance clustering features cam not find the right optimizer A two-dimensional projection of the tasks in landscape-feature space, colored by the budget-200 winning family. If cheap features predicted the winner, the colors would separate into clean footprints. They do not: the regions overlap heavily, which is the visual companion to the 44.2% accuracy.

D. RQ4: Can we predict the best optimizer? In summary, so far we have said that prior work was incomplete on important context variables (the labeling budget) and misleading on others (the relative merits of single vs. multiple objective optimization). If prior advice cannot be trusted, then practitioners face a new domain with no reliable way to pick an optimizer. Hence we must ask, based on this study: What guidance can be offered for selecting optimizers for new domains? This section explores this question in two parts. First we report on a failed experiment with building a guidance tool based on instance clustering metrics. Secondly, we report a second experiment with building a guidance tool that is far more effective (at predicting the right optimizer) and uses domain features that are very cheap to collect. 1) Guidance via instance clustering metrics (does not work): Prior work [97], [86], [87], [98] suggests optimization difficulty can be modeled as a function of measurable properties of the problem instance or sampled search surface. Following that advice, we collected metrics describing the relationship between the configuration values and the objective scores (fitness-distance correlation [99], dispersion [100], nearest-better clustering [101], meta-model R2 , skewness, kurtosis, and PCA-based geometry features [97], [98]) from a random sample of 100 rows of a data set. This was used to train a feature-based predictor (using Random Forests) [86], [101] of what family of optimizers was most useful for a

particular MOOT data set (for a list of those family names, recall Figure 2). Since the family distribution was imbalanced in our results, we used a 10 times repeated stratified 10-fold cross validation [102]. The resulting classifier only reached 44.2% accuracy for predicting what family of optimizer works best for that data set (for knowledge of what optimizer works best, we used our RQ2 results). As a stricter check, we also trained the same feature-based model to predict the exact optimizer, which reached only 34% accuracy. Thus, even when the featurebased approach is given the easier family-level target, it does not provide reliable guidance in our setting. Figure 4 explains why instance clustering metrics worked so poorly: projected into feature space our tasks do not separate into clean per-winner footprints (observe the region overlap). Hence, we need to look beyond instance clustering metrics. 2) Guidance via heatmap (works better): Given the failure of instance clustering metrics, we tried other approaches. Since the goal was an easy-to-used guide, we asked “what attributes could an engineer read off a MOOT data table at no cost, with no inference”. We used two such task attributes, plus the labeling budget. • Whether the task has conflicting objectives (optimizing one thing hurts another goal; e.g. developing cheaper software with fewer bugs); • The shape of its input space: binary/SAT (i.e. the columns values are true,false), small-numeric, or large-numeric. Each task is assigned to one of six heatmap cells using two inexpensive structural attributes computed directly from its input table. The first distinguishes single-objective tasks (one optimization objective) from multi-objective tasks (two or more objectives). The second characterizes the input space. Let |Xi | denote the number of distinct values of decision variable i. Since MOOT does not contain the real-world range and constraints of the decision variables, we approximate |Xi | with the number of distinct decision variable values observedPin MOOT. Hence, we estimate the search-space size as: i log2 (|Xi |), which is the base-2 logarithm of the Cartesian product of all decision-variable domains. A task is classified as: • Binary/SAT: at least 80% of decision variables are binary (|Xi | = 2); P • Large-numeric: not Binary/SAT and i log2 (|Xi |) ≥ 40; • Small-numeric: all remaining tasks. The boundary 40 is empirically defined. As a sensitivity check, we repeated the guide evaluation with 10, 20, 30, 40, 50, and 60. The general accuracy remained similar for 10 − 40 and dropped minimally (≈ 6% on average) only for higher values. Figure 5 shows the resulting guide, a recommended optimizer per attribute cell per budget. EZR owns the singleobjective small-numeric cells at tight budgets; DE and SMAC take over as the input space grows or the budget increases, exactly the migration RQ2 exposed.

B=30

B=50

B=100

multi | binary/SAT

DE

DE

DE

B=200 DE

multi | large-numeric

LINE

LINE

LINE

LINE

multi | small-numeric

EZR

SMAC

SMAC

SMAC

single | binary/SAT

TPE

TPE

DE

SMAC

single | large-numeric

EZR

EZR

SMAC

SMAC

single | small-numeric

EZR

EZR

EZR

EZR

Evolution

Family Surrogate

Geometric

Fig. 5. The free, budget-aware selection guide (RQ4). Recommended optimizer per objective structure, input-space shape, and budget, read off a task table at zero cost. EZR wins single-objective, small-numeric cells at tight budgets; DE and SMAC take over as input space or budget grows. Ties or beats a hindsight oracle on 74.2% of held-out tasks (Table VII). TABLE VII W IN RATE OF THE BUDGET- AWARE GUIDE UNDER 10- SEED STRATIFIED 10- FOLD CROSS - VALIDATION (200 TRAIN / TEST PARTITIONS ). Evaluation budget B

Win rate vs. oracle (%)

30 50 100 200

81.0 ± 9.4 72.0 ± 7.5 71.0 ± 11.4 73.0 ± 7.8

Overall

74.2 ± 5.6

The decisive test is whether this cheap-to-obtain guide generalizes. We evaluate it under 10 repeated shuffled 10-fold cross-validation stratified over aforementioned task classes. In each fold, we run the tournament (recall Fig. 2 and produce a guide similar to Figure 5. For each budget levels, The guide is then applied to the held-out tasks in that fold and compared against a hindsight oracle, defined as the tournament champion computed on those held out tasks at the same budget. The oracle is a deliberately hard baseline: it has seen the test tasks. Table VII reports the result. The cheap-to-obtain guide ties or beats this hindsight oracle on 74.2% of held-out tasks overall with zero probes. The instance features requiring 100 evaluations predicts the winning family only 44.2% of the time and even lower in terms of exact prediction, while this no-additional-probe guide, combined with the budget, recover the tournament winner about three times in four. RQ4. The best optimizer for a new SE task can be predicted cheaply. A guide keyed on objective conflict, input-space shape, and budget ties or beats a hindsight oracle on 74.2% of held-out tasks, at zero probe cost. Instance clustering metrics failed to do so. A practitioner facing a new SE task needs only to read two attributes from their data table and consult Figure 5. E. Threats to Validity Construct validity. Our quality metric is d2h, and reducing a multi-objective result to its best single point could in principle favor scalar methods. This is why we further evaluate the multi-objective tasks with frontier-aware metrics (IGD, GD, HV) RQ3.

A second construct concern might be that the response surface is a fitted surrogate rather than ground truth. we tuned the surrogate and used one tuned surrogate per task, held fixed across all optimizers, so any bias is shared by every method and cannot change their relative ranking, and we judge methods by delivered d2h rather than by surrogate accuracy. External validity. All tasks come from MOOT and there might be other data sets for which these results do not hold. This is a problem of any empirical study since no study can exercise all data sets. To mitigate this, all we can do is: Make our case study space as large as possible. The 106 tasks span seven SE domains from configuration and performance tuning to project health and defect prediction (Table I), and the algorithms we selected cover five families and six of the seven assumptions an SE practitioner would plausibly try (Table II), given a reading of the current SE optimization literature. • When we exclude algorithms, we take care to justify why they are excluded (see section III-A); • We publish all our scripts and data on-line so other researchers can apply our methods to their data.

•

The opposite of using too little test data is using too much. A related threat is that the guide could be overfit to these tasks. Mitigation: the guide proposed in this paper is validated under stratified cross-validation against a hindsight oracle on heldout tasks (Table VII), not on the data it was fit to. Conclusion validity. Differences between methods could be noise. To mitigate this, every cell is repeated over 20 seeds, all rankings use Scott-Knott clustering gated by a bootstrap test and a Cliff’s delta effect-size threshold (Section V), so reported wins survive both significance and effect-size screens. VII. C ONCLUSION Search-based software engineering offers practitioners an overwhelming menu of black-box optimizers and almost no budgeted rule for choosing among them. Recasting that menu as a set of bets on algorithm assumptions, we organized 20 optimizers into an assumption-indexed tournament and ran it over 106 SE tasks at four budgets, roughly 14,000 CPU hours. The tournament yields no single optimizer. Instead it exposes a boundary: the winner migrates from a fastwarming geometric active learner at tight budgets to modelfree differential evolution once the budget grows, and the single best optimizer differs from its own B=30 choice on up to 50% of tasks by B=200, with 58% switching at least once across the four budgets we tested. The standard way to sidestep such a study, which is predicting the winner from instance clustering metrics, fails at SE budgets (44.2% accuracy). Hence, we propose another guide based on some cheap-toobtain attributes (objective conflict, input-space shape, labeling budget). This second guide scores as well as a hindsight oracle on 74.2% of held-out tasks at near zero probe cost. Our takeaway for practitioners:

Treat optimizer selection in SE as a budget-dependent matrix: anchor the choice on cheap structural signals, and leave expensive instance clustering methods behind. The research message is larger. The boundary reframes optimizer selection as a scheduling problem rather than a oneshot pick. The migration we measured (a geometric learner early, model-free evolution late) suggests a warm handoff : spend the first evaluations with EZR to locate good regions cheaply, then hand its best points to DE as the budget accrues. This is adjacent to multi-fidelity methods such as Hyperband, BOHB, and DEHB [103], [88], [89], but the axis differs: those schedule fidelity within one algorithm, whereas our boundary schedules the algorithm, and with it the landscape assumption. For future work, we suggest four directions: 1) Build the meta-scheduler the boundary implies: a controller that switches assumptions at the measured crossover rather than fidelities within a fixed assumption. 2) Test whether that boundary survives richer methods, such as causal, multi-fidelity, high-dimensional Bayesian, and LLM-driven optimizers [3], [88], [68], [90], each encoding an assumption our seven omit, and each may move the budget crossover or leave it in place. 3) Sharpen the boundary itself: with only four budget points the EZR→DE crossover is real but fuzzy, so a continuous-budget characterization, or a blend rule near the crossover could be the next refinement. 4) Extend the vocabulary of cheap-to-obtain attributes. Beyond these, our guide has a natural role in LLM-driven and agentic search. As a prior, it offers a cheap, auditable first choice. As a baseline, it sets a bar any added sophistication must clear: matching a hindsight oracle on three of four tasks from two cheap-to-obtain attributes is the number to beat. R EFERENCES [1] A. Rayegan and T. Menzies, “Minimal data, maximum clarity: A heuristic for explaining optimization,” Journal of Systems and Software, p. 112897, 2026. [2] T. Menzies, T. Chen, Y. Ye, K. K. Ganguly, A. Rayegan, S. Srinivasan, and A. Lustosa, “Moot: a repository of many multi-objective optimization tasks,” in Proc. 22nd International Conference on Mining Software Repositories, Data and Tool Showcase Track, 2026, to appear. [3] P. Chen and T. Chen, “Promisetune: Unveiling causally promising and explainable configuration tuning,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026, to appear. [4] L. Senthilkumar and T. Menzies, “Can large language models improve se active learning via warm-starts?” ACM Transactions on Software Engineering and Methodology, 2024. [5] A. Lustosa and T. Menzies, “Less noise, more signal: Drr for better optimizations of se tasks,” arXiv preprint arXiv:2503.21086, 2025. [6] ——, “Learning from very little data: On the value of landscape analysis for predicting software project health,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 3, pp. 1–22, 2024. [7] V. Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,” IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 794–811, 2018. [8] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4/E. Pearson, 2021. [9] M. Harman and P. McMinn, “A theoretical and empirical study of search-based testing: Local, global, and hybrid search,” IEEE Transactions on Software Engineering, vol. 36, no. 2, pp. 226–247, 2009.

[10] M. Harman, S. A. Mansouri, and Y. Zhang, “Search-based software engineering: Trends, techniques and applications,” ACM Computing Surveys (CSUR), vol. 45, no. 1, pp. 1–61, 2012. [11] X. He, L. Xu, X. Zhang, R. Hao, Y. Feng, and B. Xu, “Pyart: Python api recommendation in real-time,” in IEEE/ACM 43rd International Conference on Software Engineering. IEEE, 2021, pp. 1634–1645. [12] S. Kirkpatrick, C. D. Gelatt Jr, and M. P. Vecchi, “Optimization by simulated annealing,” science, vol. 220, no. 4598, pp. 671–680, 1983. [13] M. Harman, “The current state and future of search based software engineering,” in FOSE’07. IEEE, 2007, pp. 342–357. [14] S. Wang, D. Lo, L. Jiang, H. C. Lau et al., “Search-based fault localization,” in 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011). IEEE, 2011, pp. 556–559. [15] I. Rechenberg, “Evolutionsstrategie,” Optimierung technischer Systeme nach Prinzipien derbiologischen Evolution, 1973. [16] A. Arcuri, “Test suite generation with the many independent objective (mio) algorithm,” Inf. Softw. Technol., vol. 104, pp. 195–206, 2018. [17] H. R. Lourenço, O. C. Martin, and T. Stützle, “Iterated local search,” in Handbook of metaheuristics. Springer, 2003, pp. 320–353. [18] N. R. Sabar, A. Turky, and A. Song, “A genetic programming based iterated local search for software project scheduling,” in Proc. Genetic and Evolutionary Computation Conference, 2018, pp. 1364–1370. [19] F. Glover, “Tabu search—part i,” ORSA Journal on computing, vol. 1, no. 3, pp. 190–206, 1989. [20] E. Dı́az, J. Tuya, R. Blanco, and J. J. Dolado, “A tabu search algorithm for structural software testing,” Computers & Operations Research, vol. 35, no. 10, pp. 3052–3072, 2008. [21] Y. Ma, G. Luo, X. Zeng, and A. Chen, “Transfer learning for crosscompany software defect prediction,” Information and Software Technology, vol. 54, no. 3, pp. 248–256, 2012. [22] A. Agrawal, W. Fu, D. Chen, X. Shen, and T. Menzies, “How to “dodge” complex software analytics,” IEEE Transactions on Software Engineering, vol. 47, no. 10, pp. 2182–2194, 2019. [23] J. H. Holland, Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT press, 1992. [24] C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,” IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 54–72, 2011. [25] Y. Zhang, M. Harman, G. Ochoa, G. Ruhe, and S. Brinkkemper, “An empirical study of meta-and hyper-heuristic search for multi-objective release planning,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 27, no. 1, pp. 1–32, 2018. [26] H. Mühlenbein and G. Paass, “From recombination of genes to the estimation of distributions i. binary parameters,” in International conference on parallel problem solving from nature. Springer, 1996, pp. 178–187. [27] C. Wei, X. Yao, D. Gong, and H. Liu, “Test data generation for mutation testing based on markov chain usage model and estimation of distribution algorithm,” IEEE Transactions on Software Engineering, vol. 50, no. 3, pp. 551–573, 2024. [28] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proceedings of ICNN’95-international conference on neural networks, vol. 4. ieee, 1995, pp. 1942–1948. [29] M. Lee, S. Cha, and H. Oh, “Learning seed-adaptive mutation strategies for greybox fuzzing,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 384–396. [30] R. Storn and K. Price, “Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces,” Journal of global optimization, vol. 11, no. 4, pp. 341–359, 1997. [31] W. Fu and T. Menzies, “Easy over hard: A case study on deep learning,” in Proceedings of the 2017 11th joint meeting on foundations of software engineering, 2017, pp. 49–60. [32] F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” in LION. Springer, 2011, pp. 507–523. [33] K. K. Ganguly and T. Menzies, “How low can you go? the data-light SE challenge,” in Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE), 2026, to appear. [34] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” NeurIPS, vol. 24, 2011. [35] J. Chen, N. Xu, P. Chen, and H. Zhang, “Efficient compiler autotuning via bayesian optimization,” in IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1198–1209.

[36] J. Chen, V. Nair, R. Krishna, and T. Menzies, ““sampling” as a baseline optimizer for search-based software engineering,” IEEE Transactions on Software Engineering, vol. 45, no. 6, pp. 597–614, 2018. [37] P. Chen, J. Gong, and T. Chen, “Accuracy can lie: On the impact of surrogate model in configuration tuning,” IEEE Transactions on Software Engineering, vol. 51, no. 2, pp. 548–580, 2025. [38] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization.” Journal of machine learning research, vol. 13, no. 2, 2012. [39] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: Nsga-ii,” IEEE transactions on evolutionary computation, vol. 6, no. 2, pp. 182–197, 2002. [40] W. Mkaouer, M. Kessentini, A. Shaout, P. Koligheu, S. Bechikh, K. Deb, and A. Ouni, “Many-objective software remodularization using nsga-iii,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 24, no. 3, pp. 1–45, 2015. [41] R. A. Matnei Filho and S. R. Vergilio, “A multi-objective test data generation approach for mutation testing of feature models,” J. softw. eng. res. dev., vol. 4, no. 1, p. 4, 2016. [42] E. Zitzler, M. Laumanns, and L. Thiele, “Spea2: Improving the strength pareto evolutionary algorithm,” TIK report, vol. 103, 2001. [43] N. Beume, B. Naujoks, and M. Emmerich, “Sms-emoa: Multiobjective selection based on dominated hypervolume,” European journal of operational research, vol. 181, no. 3, pp. 1653–1669, 2007. [44] C. Ni, X. Chen, F. Wu, Y. Shen, and Q. Gu, “An empirical study on pareto based multi-objective feature selection for software defect prediction,” J. Syst. Softw., vol. 152, pp. 215–238, 2019. [45] Q. Zhang and H. Li, “Moea/d: A multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on evolutionary computation, vol. 11, no. 6, pp. 712–731, 2007. [46] M. Zhu and D. Hao, “Compiler auto-tuning via critical flag selection,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 1000–1011. [47] D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang, “Automatic database management system tuning through large-scale machine learning,” in Proceedings of the 2017 ACM international conference on management of data, 2017, pp. 1009–1024. [48] A. S. Sayyad, T. Menzies, and H. Ammar, “On the value of user preferences in search-based software engineering: A case study in software product lines,” in 2013 35Th international conference on software engineering (ICSE). IEEE, 2013, pp. 492–501. [49] J. Zhang, Y. Liu, K. Zhou, G. Li, Z. Xiao, B. Cheng, J. Xing, Y. Wang, T. Cheng, L. Liu et al., “An end-to-end automatic cloud database tuning system using deep reinforcement learning,” in Proceedings of the 2019 international conference on management of data, 2019, pp. 415–432. [50] L. Huang, W. Sun, and M. Yan, “Iterative generation of adversarial example for deep code models,” in IEEE/ACM 47th International Conference on Software Engineering. IEEE, 2025, pp. 2213–2224. [51] D. H. Wolpert and W. G. Macready, “No free lunch theorems for optimization,” IEEE transactions on evolutionary computation, vol. 1, no. 1, pp. 67–82, 2002. [52] Z. Ma, H. Guo, Y.-J. Gong, J. Zhang, and K. C. Tan, “Toward automated algorithm design: A survey and practical guide to metablack-box-optimization,” IEEE Trans. Evol. Comput., 2025. [53] R. Qiu, W. Zeng, J. Ezick, C. Lott, and H. Tong, “How efficient is llm-generated code? a rigorous & high-standard benchmark,” in International Conference on Learning Representations, vol. 2025, 2025, pp. 2233–2261. [54] A. Agrawal, X. Yang, R. Agrawal, R. Yedida, X. Shen, and T. Menzies, “Simpler hyperparameter optimization for software analytics: Why, how, when?” IEEE Transactions on Software Engineering, vol. 48, no. 8, pp. 2939–2954, 2021. [55] A. Aleti, I. Moser, and L. Grunske, “Analysing the fitness landscape of search-based software testing problems,” Automated Software Engineering, vol. 24, no. 3, pp. 603–621, 2017. [56] N. Albunian, G. Fraser, and D. Sudholt, “Causes and effects of fitness landscapes in unit test generation,” in Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020, pp. 1204–1212. [57] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024. [58] M. Easterby-Smith, “The design, analysis and interpretation of repertory grids,” Int. J. Man-Mach. Stud., vol. 13, no. 1, pp. 3–24, 1980. [59] R. Valerdi, “Heuristics for systems engineering cost estimation,” IEEE Systems Journal, vol. 5, no. 1, pp. 91–98, 2010.

[60] Z. Yu, F. M. Fahid, H. Tu, and T. Menzies, “Identifying self-admitted technical debts with jitterbug: A two-step approach,” IEEE Transactions on Software Engineering, vol. 48, no. 5, pp. 1676–1691, 2020. [61] X. Wu, W. Zheng, X. Xia, and D. Lo, “Data quality matters: A case study on data label correctness for security bug report prediction,” IEEE Trans. Softw. Eng., vol. 48, no. 7, pp. 2541–2556, 2021. [62] H. J. Kang, K. L. Aw, and D. Lo, “Detecting false alarms from automatic static analysis tools: How far are we?” in Proc. of the 44th International Conference on Software Engineering, 2022, pp. 698–709. [63] M. Shepperd, Q. Song, Z. Sun, and C. Mair, “Data quality: Some comments on the nasa software defect datasets,” IEEE Transactions on software engineering, vol. 39, no. 9, pp. 1208–1215, 2013. [64] Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,” IEEE Trans. Softw. Eng., vol. 39, no. 6, pp. 757–773, 2012. [65] T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can llms replace manual annotation of software engineering artifacts?” in 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 2025, pp. 526–538. [66] K. Eggensperger, P. Müller, N. Mallik, M. Feurer, R. Sass, A. Klein, N. Awad, M. Lindauer, and F. Hutter, “Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo,” in Thirtyfifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. [67] A. I. Cowen-Rivers, W. Lyu, R. Tutunov, Z. Wang, A. Grosnit, R. R. Griffiths, A. M. Maraval, H. Jianye, J. Wang, J. Peters et al., “Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation,” J. Artif. Intell. Res., vol. 74, pp. 1269–1349, 2022. [68] D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, and M. Poloczek, “Scalable global optimization via local bayesian optimization,” Advances in neural information processing systems, vol. 32, 2019. [69] M. S. Iqbal, R. Krishna, M. A. Javidian, B. Ray, and P. Jamshidi, “Unicorn: Reasoning about configurable system performance through the lens of causality,” in Proceedings of the Seventeenth European Conference on Computer Systems, 2022, pp. 199–217. [70] K. Kanellis, C. Ding, B. Kroth, A. Müller, C. Curino, and S. Venkataraman, “Llamatune: sample-efficient dbms configuration tuning,” Proc. VLDB Endow., vol. 15, no. 11, p. 2953–2965, Jul. 2022. [Online]. Available: https://doi.org/10.14778/3551793.3551844 [71] G. Li, X. Zhou, S. Li, and B. Gao, “Qtune: A query-aware database tuning system with deep reinforcement learning,” Proceedings of the VLDB Endowment, vol. 12, no. 12, pp. 2118–2130, 2019. [72] B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi et al., “Mathematical discoveries from program search with large language models,” Nature, vol. 625, no. 7995, pp. 468–475, 2024. [73] H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song, “Reevo: Large language models as hyper-heuristics with reflective evolution,” NeurIPS, vol. 37, pp. 43 571–43 608, 2024. [74] J. R. Koza, “Genetic programming as a means for programming computers by natural selection,” Statistics and computing, vol. 4, no. 2, pp. 87–112, 1994. [75] K. Deb and H. Jain, “An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints,” IEEE transactions on evolutionary computation, vol. 18, no. 4, pp. 577–601, 2013. [76] T. Chen and M. Li, “The weights can be harmful: Pareto search versus weighted search in multi-objective search-based software engineering,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 1, pp. 1–40, 2023. [77] P. Chen, T. Chen, and M. Li, “Mmo: meta multi-objectivization for software configuration tuning,” IEEE Transactions on Software Engineering, vol. 50, no. 6, pp. 1478–1504, 2024. [78] J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” Advances in neural information processing systems, vol. 25, 2012. [79] R. Cao, L. Bao, K. Zhao, and P. Zhangsun, “Etune: Efficient configuration tuning for big-data software systems via configuration space reduction,” Journal of Systems and Software, vol. 209, p. 111936, 2024. [80] Z. Xiang, J. Gong, and T. Chen, “Dually hierarchical drift adaptation for online configuration performance learning,” in Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026, to appear. [81] G. Xiong and T. Chen, “Cotune: Co-evolutionary configuration tuning,” in 2025 40th IEEE/ACM International Conference on Automated

Software Engineering (ASE). IEEE Press, 2025, p. 1490–1502. [Online]. Available: https://doi.org/10.1109/ASE63991.2025.00126 [82] M. Zhu, D. Hao, and J. Chen, “Compiler autotuning through multiplephase learning,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 4, pp. 1–38, 2024. [83] P. Chen, H. Liang, and T. Chen, “Unveiling many faces of surrogate models for configuration tuning: A fitness landscape analysis perspective,” arXiv preprint arXiv:2509.21945, 2025. [84] B. J. Kröse, “Learning from delayed rewards,” Robotics and Autonomous Systems, vol. 15, no. 4, pp. 233–235, 1995. [85] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015. [86] K. Smith-Miles and M. A. Muñoz, “Instance space analysis for algorithm testing: Methodology and software tools,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–31, 2023. [87] N. Neelofar, K. Smith-Miles, M. A. Muñoz, and A. Aleti, “Instance space analysis of search-based software testing,” IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2642–2660, 2022. [88] S. Falkner, A. Klein, and F. Hutter, “Bohb: Robust and efficient hyperparameter optimization at scale,” in International conference on machine learning. PMLR, 2018, pp. 1437–1446. [89] N. Awad, N. Mallik, and F. Hutter, “DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Z. Zhou, Ed. ijcai.org, 2021, pp. 2147–2153. [90] M. R. Zhang, N. Desai, J. Bae, J. Lorraine, and J. Ba, “Using large language models for hyperparameter optimization,” arXiv preprint arXiv:2312.04528, 2023. [91] E. Meyerson, M. J. Nelson, H. Bradley, A. Gaier, A. Moradi, A. K. Hoover, and J. Lehman, “Language model crossover: Variation through few-shot prompting,” ACM Transactions on Evolutionary Learning, vol. 4, no. 4, pp. 1–40, 2024. [92] F. Pfisterer, L. Schneider, J. Moosbauer, M. Binder, and B. Bischl, “Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization,” in International Conference on Automated Machine Learning. PMLR, 2022, pp. 3–1. [93] A. Zela, J. N. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter, “Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks,” in International Conference on Learning Representations, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:248177810 [94] K. Eggensperger, F. Hutter, H. Hoos, and K. Leyton-Brown, “Efficient benchmarking of hyperparameter optimizers via surrogates,” in Proc. of the AAAI conference on artificial intelligence, vol. 29, no. 1, 2015. [95] K. Eggensperger, M. Lindauer, H. H. Hoos, F. Hutter, and K. LeytonBrown, “Efficient benchmarking of algorithm configurators via modelbased surrogates,” Machine Learning, vol. 107, no. 1, pp. 15–41, 2018. [96] A. J. Scott and M. Knott, “A cluster analysis method for grouping means in the analysis of variance,” Biometrics, pp. 507–512, 1974. [97] O. Mersmann, B. Bischl, H. Trautmann, M. Preuss, C. Weihs, and G. Rudolph, “Exploratory landscape analysis,” in Proc. 13th annual GECCO, 2011, pp. 829–836. [98] P. Kerschke, M. Preuss, S. Wessing, and H. Trautmann, “Detecting funnel structures by means of exploratory landscape analysis,” in Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, 2015, pp. 265–272. [99] T. Jones, S. Forrest et al., “Fitness distance correlation as a measure of problem difficulty for genetic algorithms.” in ICGA, vol. 95, 1995, pp. 184–192. [100] M. Lunacek and D. Whitley, “The dispersion metric and the cma evolution strategy,” in Proceedings of the 8th annual conference on Genetic and evolutionary computation, 2006, pp. 477–484. [101] P. Kerschke, H. H. Hoos, F. Neumann, and H. Trautmann, “Automated algorithm selection: Survey and perspectives,” Evolutionary computation, vol. 27, no. 1, pp. 3–45, 2019. [102] R. Kohavi et al., “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Ijcai, vol. 14, no. 2. Montreal, Canada, 1995, pp. 1137–1145. [103] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” Journal of machine learning research, vol. 18, no. 185, pp. 1–52, 2018.