arXiv CS open access

Less Is More: Tuning Configurable Systems with Imperfect Fidelity

2026 · arxiv_cs

arXiv CS · Papers · 2026

software-architecture software-engineering testing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

arXiv:2608.00759v1 [cs.SE] 1 Aug 2026

Less Is More: Tuning Configurable Systems with Imperfect Fidelity Yulong Ye

Miqing Li

Tao Chen∗

IDEAS Lab University of Birmingham Birmingham, United Kingdom YXY382@student.bham.ac.uk

University of Birmingham Birmingham, United Kingdom m.li.8@bham.ac.uk

IDEAS Lab University of Birmingham Birmingham, United Kingdom t.chen@bham.ac.uk

Abstract

Configuration tuning is essential for optimizing the performance of highly configurable systems, e.g., throughput or runtime, under a given environment. Yet, this is a challenging process as there can be many options to tune, and configuration measurement is often highly expensive. In this paper, we demonstrate the phenomenon of “less can be more”: system configuration tuning can be greatly improved with much superior budget utilization by partially tuning under the imperfect-fidelity—an environment that is similar, but cheaper to measure, compared with the concerned perfect-fidelity of environment under which the system should be tuned. We codify a conceptual framework of fidelity for configurable systems, drawing on which allows us to propose MFTune, a tuner that proactively explores in the space of > 104 possible imperfect-fidelity settings to approximate a useful one, which strikes for the wideness of tuning. This creates high-quality seeds for the perfect-fidelity, which in turn ensures the tuning depth. Experiment results against 10 stateof-the-art tuners, obtained from running diverse real-world systems for 19 months 24×7, show that MFTune performs considerably better on 83.33% cases with up to 19.34% improvement while achieving hours of budget saving in general.

Modern software systems are often highly configurable, exposing numerous tunable configuration options [37, 53]. The goal thereof is intuitive: by allowing flexible configuration, the system can achieve greater applicability across diverse domains and cater to varying performance requirements, e.g., runtime and throughput [21, 22, 61]. Yet, excessive configurability comes with its own costs: it has been shown that globally 59% of the performance issues, where performance requirements were severely violated, are attributed to poorly chosen configurations rather than code [38]. With proper configuration, systems could unlock their full performance potential [47]. Theoretically, the optimal configuration could be identified by exhaustively profiling the system across all possible configurations under a certain environment, e.g., a setting of anticipated workload or job. This, however, is impractical because measuring even a single configuration can be rather costly, taking considerable time and computational resources (e.g., minutes to hours) [20, 24, 25]. The tuning efficiency is further exacerbated by the fact that the configuration space is typically high-dimensional and grows exponentially with the number of configuration options [23, 62, 65]. For example, MySQL—a database system—offers dozens of tunable configuration options, making exhaustive profiling infeasible [54, 63, 66]. Existing system tuners often focus on advanced algorithm designs with smart heuristics (e.g, BestConfig [70]), hoping that the tuner would find the promising configurations soon; or leverage a surrogate model to predict configuration performance (e.g., PromiseTune [18]), which can be unreliable at times. The key unaddressed question is: how to efficiently utilize the budget in tuning? In this paper, we take a different perspective to explicitly tackle this challenge: we view tuning a configurable system under different environments as exhibiting a certain degree of exactness to each other, resembling the notion of fidelity. Therefore, we hypothesize that proactively exploring good configurations found from an imperfect, cheaper environment under which the system is tuned could be beneficial to tuning it for the perfect, more expensive target environment. For example, configurations found by tuning PostgreSQL under a workload of 100 requests per second might expedite and benefit the tuning under that with 10, 000 requests per second—a concept we borrowed from the paradigm known as multi-fidelity optimization [44, 48, 50]. Noteworthily, while similar in spirit, multifidelity optimization differs from prior work on knowledge transfer of environments for configuration tuning [10, 67, 68]: the former proactively explores and exploits unknown fidelity settings of environments for tuning, whereas the latter relies on reusing what is

CCS Concepts • Software and its engineering → Software performance; Search-based software engineering.

Keywords Configurable systems, configuration performance tuning, hyperparameter optimization, multi-fidelity optimization ACM Reference Format: Yulong Ye, Miqing Li, and Tao Chen. 2026. Less Is More: Tuning Configurable Systems with Imperfect Fidelity. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3832783.3834376

∗ Corresponding author.

This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3834376

Introduction

ASE ’26, October 12–16, 2026, Munich, Germany

Yulong Ye, Miqing Li, and Tao Chen

Tuning in fidelity-A test for fidelity-A Tuning in fidelity-B test for fidelity-B Tuning in fidelity-B test for fidelity-A

Tuning in fidelity-A test for fidelity-A Tuning in fidelity-C test for fidelity-C Tuning in fidelity-C test for fidelity-A

500

500 took 67 hours

took 67 hours 450 Throughput (tps)

Throughput (tps)

450 400 350 300

took 20 hours 400 350 300

took 31 hours 250

250 1

200 400 600 # measurements

(a) Helpful fidelity

800

200 400 600 # measurements

800

(b) Harmful fidelity

Figure 1: Tuning PostgreSQL under different environments/fidelity settings via a random search tuner. fidelity-A (time=180s; size=5000k) is the “target perfection”; fidelity-B (time=30s; size=5000k) and fidelity-C (time=60s; size=1050k) are two imperfect fidelity settings. “Tuning in fidelity-A test for fidelity-B” means that the tuning is guided by configurations measured under fidelity-A and the best one found therein is progressively tested under fidelity-B.

available from historically explored and structurally similar environments, limiting their ability to adapt to a completely unforeseen environment. Figure 1 illustrates the idea where the fidelity of environment is set by the limits of execution time and table size in Sysbench [7]. Suppose our “target perfection” is to tune configuration under fidelity-A (hence we are only interested in testing therein), Figure 1a shows that tuning under fidelity-B produces almost similarlyperforming configurations (solid cycle line), when being tested in fidelity-A, to those produced by tuning/testing under fidelity-A directly (solid square line), but is ≈ 36 hours faster to complete—it might well exceed the result of tuning/testing under fidelity-A if given the same wall-clock time. However, this is not straightforward as not all fidelity settings can be useful for fidelity-A, especially given the complex dimensions related to environments and fidelity settings for configurable systems: in Figure 1b, even though the time saving remains significant, the tuning guided in fidelity-C has severely misled the results when the configurations found therein are tested under fidelity-A (solid cycle line vs. solid square line). This is a core issue that has yet been well-addressed in current multi-fidelity optimization. Indeed, multi-fidelity optimization is an established paradigm [11, 33, 48, 50, 51], particularly for hyperparameter optimization (HPO) [43, 45, 60]. For example, when tuning the hyperparameters for deep neural network training, one can reduce the number of epochs in the training to achieve a low-fidelity measurement, which provides a cheaper yet informative approximation of the final model performance under a perfect, often much higher, epoch setting of fidelity [52]. Yet, despite those advances, adopting this paradigm for general system configuration tuning is not easy, because: 1) Mismatched Problem Formulation: Unlike HPO, system configuration tuning lacks a general multi-fidelity problem formulation that defines how fidelity should be modeled and controlled: in HPO, since machine learning models follow similar pipeline, it is

well-known that fidelity is related to factors that are highly influential to training time (i.e., the cost in this context), such as training epochs [30] and training sample size [44, 50]. Yet, given the high variety of system domains, there is no known general definition of fidelity for system configuration tuning. 2) Incompatible Tuner Assumption: Most existing multifidelity tuners assume a monotonic relationship between fidelity perfection and cost, i.e., more costly measurements always yield a higher exactness approximation to the target perfect environment [14, 31]. While such an assumption is reasonable in HPO, it does not necessarily hold for configurable systems. For instance, training with more epochs generally yields measurements that more closely approximate the performance under the perfect full epochs; in contrast, we found that when tuning the database system PostgreSQL, extending the benchmarking duration does not always yield measurements that better approximate those obtained under a higher, perfect duration runs (see §3.3). Moreover, the multifidelity tuners in HPO often rely on a restricted single factor to determine fidelity setting (e.g., the epoch) [33, 44, 51, 52]. Such a design overlooks the possibility that more cost-effective fidelity settings can exist in the multi-dimensional fidelity spaces exhibited in configurable systems (see §3.3). To fill the above gap and fully exploit the benefits of fidelity for system configuration tuning, we first present a tailored conceptual framework to codify the definition of fidelity for configurable systems. Drawing on this, we propose MFTune, a fidelity-aware configuration tuning tool specifically for systems. What makes MFTune unique is that it proactively exploits/extracts the multidimensional fidelity space of more than ten thousands fidelity settings, finding a fair one (i.e., a conceptual “knee point” that represents a desirable trade-off between cost and approximation of the perfect-fidelity1 ) that can accelerate the tuning under the perfect-fidelity setting via a shared archive, hence striking for both the “wideness” and “depth” of configuration tuning with improved budget utilization. In a nutshell, our contributions are: • We propose a unified and extensible conceptual framework to define the notions of fidelity for configurable systems, leading to a new problem formulation (§3). • Deriving from the above framework, we present an active fidelity discovery strategy in MFTune to quantify and extract cheaper, yet useful fidelity settings with respect to the perfect-fidelity setting (§4.1). • MFTune embeds a mechanism that synergizes tuning between a selected imperfect- and the perfect-fidelity settings with diversity preservation: tuning under the imperfect-fidelity setting, which is less costly yet with a good level of perfection, aims to cover a wide, promising area of the configuration space with good diversity, finding high-quality archived seeds for the perfect-fidelity setting. In contrast, tuning under the perfect-fidelity setting seeks to fully leverage the rich seeds discovered under the imperfect-fidelity setting, encouraging a deep investigation of the tuning directions implied (§4.2–§4.3).

1We use the term perfect-fidelity to denote the target fidelity setting of environment

under which one prefers to tune a configurable system.

Less Is More: Tuning Configurable Systems with Imperfect Fidelity

ASE ’26, October 12–16, 2026, Munich, Germany

• We assess MFTune on six real-world configurable systems against 10 state-of-the-art tuners scaling up to 11, 200 fidelity settings (§5–§6). The results suggest that MFTune considerably outperforms the others, ranking the best on the majority of the systems (5/6) with up to 19.34% improvement while generally saving hours of budget. All source code and data are available at: https://github.com/ideaslabo/mftune.

Budget

Workload

Dataset

Time

Scale

Size

Execution volume

Type

Concurrency

Complexity

Distribution

…

Figure 2: Taxonomy of fidelity factors.

2 Preliminaries 2.1 System Configuration Tuning A configurable system often comprises a set of configuration options, each taking categorical or numerical values. The objective is to identify a configuration that optimizes a specific performance metric (e.g., minimizing runtime or maximizing throughput) of the system under a target environment. This can be formulated as: arg min 𝑓 (𝒙) or arg max 𝑓 (𝒙), ∑︁ s.t. 𝜏 (𝒙) ≤ B,

illustrated in Figure 2, the fidelity factors can be assigned into three broad types based on their role and impact on the tuning: • Budget-related: Factors that control resource allocation, e.g., the duration, which controls how long each load-test runs, for tuning Tomcat. • Workload-related: Factors that shape workload characteristics, e.g., the max-func of programs, which sets the maximum number of functions, for tuning Gcc. • Dataset-related: Factors that define the dataset composition, e.g., table-size in the benchmark that tunes MySQL.

(1)

𝒙 ∈𝒳

where 𝒙 = (𝑥 1, 𝑥 2, . . . , 𝑥𝑑 ) is a configuration with the values of 𝑑 options in configuration space 𝒳 . 𝑓 is the concerned performance metric. 𝜏 (𝒙) denotes the cost of measuring a configuration 𝒙 under the given environment. B is the tuning budget, e.g., wall-clock time.

2.2

In general, a fidelity factor can be categorical, numerical or ordinal, depending on the system/environment. Those fidelity factors can often be found in the benchmark used to tune a configurable system, e.g., Sysbench [7] and Wrk [9].

Multi-Fidelity Optimization 3.2

Problem Formulation

Typically, multi-fidelity optimization reduces the cost of expensive measurements by exploiting cheaper approximates of the objective. Here, fidelity is modeled and controlled via a single factor, such as the number of training epochs or the proportion of training data, while extensions to multiple factors remain rare [48]. Formally, the objective of a solution 𝒙 ∈ 𝒳 measured with budget 𝒓 can be denoted by 𝑓 (𝒙, 𝒓), where larger budgets incur higher cost 𝑐 (𝒓) but are assumed to provide more accurate approximations of the measurements in the perfect-fidelity setting at 𝒓 ∗ . This one-dimensional and cost-oriented view of fidelity underpins most multi-fidelity optimization algorithms in, e.g., HPO [11, 33, 51, 52].

Drawing on the definition of fidelity factors for configurable systems, we extend the problem from Equation 1 as a fidelity-aware one. Suppose 𝒛 ∈ 𝒵 denotes a fidelity setting, where 𝒛 = (𝑧 1, 𝑧 2, . . . , 𝑧𝑘 ) is a vector defined by 𝑘 (𝑘 ≥ 1) fidelity factor(s), and 𝒵 stands for the entire fidelity space of settings. The objective function under the fidelity setting 𝒛 is 𝑓𝒛 (𝒙), i.e., the performance of configuration 𝒙 measured at 𝒛. We denote the target, perfect-fidelity setting by 𝒛 ∗ (𝒛 ∗ ∈ 𝒵), and its objective function by 𝑓𝒛 ∗ (𝒙) (i.e., 𝑓𝒛 ∗ (𝒙) = 𝑓 (𝒙)). We denote the cost under the fidelity setting 𝒛 by 𝜏𝒛 (𝒙). Now, the multi-fidelity configuration tuning problem becomes:

3 Codifying Fidelity for Configurable Systems 3.1 Fidelity Factors

arg min 𝑓𝒛 ∗ (𝒙) or arg max 𝑓𝒛 ∗ (𝒙), ∑︁ s.t. 𝜏𝒛 (𝒙) ≤ B.

Fidelity factors are control variables for the fidelity settings of environments. In HPO [32], the fidelity factor is commonly related to “resource types”, such as training epochs, which are general across any models. However, in configuration tuning, the fidelity factors are typically implicit, environment/system-specific, and lack standardized identification criteria. To formalize this notion, in this work, we define a fidelity factor for configurable systems as: “A variable that modulates the environment under which a configuration is measured but is exogenous to the configurable system.” To specify this, we provide a framework of categories grounded in the above definition. Instead of an exhaustive classification, our goal is to offer a structured lens through which potential fidelity factors can be defined, compared, and applied across systems. As

(2)

𝒙 ∈𝒳 ; 𝒛 ∈𝒵

The goal is to find a configuration whose performance value under the perfect-fidelity setting is optimized, through exploring the measurements across the entire fidelity space, including both imperfect-fidelity settings (𝒵/𝒛 ∗ ) and the perfect-fidelity setting (𝒛 ∗ ), subject to a budget B. From the above, the 𝑖th fidelity setting 𝒛𝑖 for configurable systems might influence two aspects: • Cost: The cost of an (imperfect-)fidelity setting refers to the average wall-clock time consumed for profiling the system on 𝑢 given configurations under that setting to obtain their performance measurements2 : 2 The time taken for the deployment of the environment is also included.

ASE ’26, October 12–16, 2026, Munich, Germany

𝜏𝒛𝑖 (𝒙 𝑗 ).

(3)

3.3

0 0

Throughput (tps)

#D2

#D1

−1

#D2

#D1

(a) Good perfection

(b) Bad perfection 200 Varying single factor Varying multiple factors

0.8

(4)

With the cost and fidelity perfection, we anticipate that most of the time, if not all, the tuning is guided by configurations measured under a selected imperfect-fidelity setting 𝑓𝒛𝑖 (𝒙), which should often be cheaper yet provides a good perfection.

1 −1

−1

150

0.6

100 0.4

50 0.2

150

100

50 0 0

𝒛1 𝒛2 𝒛3 𝒛4 𝒛5 𝒛6 𝒛7 𝒛8 𝒛 𝒛19

whereby 𝒮 = {𝒙 1, . . . , 𝒙𝑢 } means a set of 𝑢 configurations. ℱ𝒛𝑖 = [𝑓𝒛𝑖 (𝒙 1 ), . . . , 𝑓𝒛𝑖 (𝒙𝑢 )] and ℱ𝒛 ∗ = [𝑓𝒛 ∗ (𝒙 1 ), . . . , 𝑓𝒛 ∗ (𝒙𝑢 )] denote the performance of all configurations 𝒙 ∈ 𝒮 measured under the fidelity setting 𝒛𝑖 and the perfect-fidelity setting 𝒛 ∗ , respectively. ℛ(·) returns performance ranks; cov(·, ·) is their covariance while 𝜎ℛ ( ℱ𝒛𝑖 ) and 𝜎ℛ ( ℱ𝒛 ∗ ) are the corresponding standard deviations. A higher 𝜌 𝒛𝑖 means the setting is closer to the perfect-fidelity setting.

400

0 0 1

Cost (s)

cov ℛ(ℱ𝒛𝑖 ), ℛ(ℱ𝒛 ∗ ) , 𝜎ℛ ( ℱ𝒛𝑖 ) 𝜎ℛ ( ℱ𝒛 ∗ )

1 −1

𝜌 𝒛𝑖 =

400

500

Cost (s)

• Fidelity Perfection: We use fidelity perfection to denote the extent to which an (imperfect-)fidelity setting can approximate the target perfect-fidelity setting, under which the configurable system should be tuned. To quantify such, in this work we use Spearman correlation [39], which captures the relative nonlinear ranking of the configurations’ performance measured under the 𝑖th (imperfect-)fidelity setting (𝒛𝑖 ) and the perfect-fidelity setting (𝒛 ∗ ):

500

Fidelity perfection

𝑢 𝑗=1

Top configurations at perfect-fidelity 𝒛 ∗ Top configurations at imperfect-fidelity 𝒛 2 Overlapping configurations across fidelities

Top configurations at perfect-fidelity 𝒛 ∗ Top configurations at imperfect-fidelity 𝒛 1 Overlapping configurations across fidelities

Throughput (tps)

𝜏 𝒛𝑖 =

𝑢 1 ∑︁

Yulong Ye, Miqing Li, and Tao Chen

Fidelity setting

0.2

0.4

0.6

Fidelity perfection

(d) Interaction of factors

Figure 3: (a) and (b) visualize the landscapes (processed by MDS [28]) of top-50 performing configurations under perfect-fidelity and two distinct imperfect-fidelity settings; (c) presents the cost and fidelity perfection of 10 imperfectfidelity settings; and (d) shows the selected imperfect-fidelity settings with and without considering factor interactions.

Characteristics of Systems Fidelity

To better showcase the characteristics of fidelity in configurable systems, we conduct an exploratory study on PostgreSQL using the Sysbench [7] benchmark as an example. We tune 20 widely used configuration options and examine 5 commonly mentioned fidelity factors, leading to a configuration space of 8.70 × 10122 and fidelity space of 11,200, respectively3 . Here, we aim to tune for better throughput while reducing the time taken for profiling configurations (the cost). Specifically, we generate 1,000 configurations using Latin Hypercube Sampling (LHS) [57] and measure each under a given perfect-fidelity setting and 10 randomly selected imperfect-fidelity settings. We obtain several observations: Observation 1: Top-performing configurations in the imperfectand perfect-fidelity settings are never completely identical; they can, however, be quite similar or discrepant. Figures 3a and 3b visualize the top-50 high-performing configurations under the perfect-fidelity setting, which requires about 250s per measurement, together with those under two representative imperfect-fidelity settings, requiring roughly 60s and 75s per measurement, respectively. We see that the top-performing configurations measured under different fidelity settings can rarely be identical, but the level of perfection differs: as shown in Figure 3a, there is a strong overlap of the promising configurations between imperfect- and perfect-fidelity settings (the connected points), i.e., the configurations that perform very well under both 3 The details can be found at Tables 1 and 2.

fidelity settings are identical, evidencing the feasibility of leveraging imperfect-fidelity proxies to guide efficient tuning. However, as illustrated in Figure 3b, an opposite pattern emerges, where the top-performing configurations under imperfect- and perfectfidelity settings differ significantly. This highlights the importance of discovering reliable imperfect-fidelity settings. As mentioned, HPO assumes a monotonic relation between fidelity perfection and cost (more cost means better perfection), but what we found for configurable systems is that: Observation 2: Cost and fidelity perfection are not always monotonically correlated, making cost variation an unreliable/uncertain indicator of the change to perfection level. Figure 3c plots the fidelity perfection (via Equation 4) of each imperfect-fidelity setting against its measurement cost. The results show that cost and fidelity perfection are not always monotonically related, e.g., some highly costly settings yield a lower perfection than the cheaper ones. This challenges the common assumption in many multi-fidelity optimization cases that higher cost inherently brings better exactness to the perfect-fidelity [14, 33, 44, 51, 52]. To further understand the unique complexity of fidelity space for configurable systems beyond what was assumed for HPO, within the 10 imperfect-fidelity settings, we randomly choose five that differ only on a single factor, e.g., (time=180s) vs. (time=30s), against the other five that differ on multiple factors, e.g., (time=180s; tables=50) vs. (time=30s;tables=20), based on all of which we measure the performance of 1, 000 configurations. We found that:

Less Is More: Tuning Configurable Systems with Imperfect Fidelity

Observation 3: Considering interaction between multiple fidelity factors can yield “fair” imperfect-fidelity settingsa that would otherwise be difficult to find.

ASE ’26, October 12–16, 2026, Munich, Germany

Algorithm 1: MFTune Input: Budget B ; perfect-fidelity setting 𝒛 ∗ ; parameter 𝛼 Output: The best configuration found under perfect-fidelity: 𝒙𝑏𝑒𝑠𝑡 𝒜 , 𝒛 𝑓 𝑎𝑖𝑟 ←ImperfectFidelityDiscovering( B , 𝒛 ∗ ) 𝒜 ← ImperfectFidelitySeeding( B , 𝒛 𝑓 𝑎𝑖𝑟 , 𝒛 ∗ , 𝒜, 𝛼 ) 𝒙𝑏𝑒𝑠𝑡 ← SeededPerfectFidelityTuning( B , 𝒜, 𝒛 ∗ ) return 𝒙𝑏𝑒𝑠𝑡

a Not perfect, but fair enough to be useful given its cost/perfection level.

2 3

As in Figure 3d, the imperfect-fidelity settings where multiple factors are varied reflect much better cost and fidelity perfection than their single-factor varying counterparts. This implies that assuming imperfect-fidelity settings with only a single factor, as in the classic HPO, would leave their full potential untapped. The above, together with the proposed conceptual framework of fidelity, motivates our idea on proactively exploiting imperfectfidelity settings to accelerate tuning under the perfect-fidelity setting. Yet, this raises three key challenges: • Challenge 1: How can we identify the fair imperfect-fidelity settings in multi-dimensional fidelity space? • Challenge 2: How to explore useful information under the imperfect-fidelity setting(s)? • Challenge 3: How to enable effective exploitation of the imperfect-fidelity setting to benefit tuning under the perfectfidelity setting? Those are what we address via MFTune.

The MFTune Framework

Figure 4 and Algorithm 1 illustrate the workflow of MFTune. Given a perfect-fidelity setting, the key idea is to first explore the fidelity space to identify a “fair” imperfect-fidelity setting—one that offers good fidelity perfection while incurring a much lower cost, hence improving budget utilization. We then conduct a specifically designed diversity-preserving tuning under the selected fair imperfect-fidelity setting to achieve “wide” tuning, after which the resulting archive seeds the subsequent “deep” tuning under the perfect-fidelity setting. To that end, MFTune comprises three phases: Imperfect-Fidelity Discovering (line 1) formulates an active, multi-objective fidelity discovery problem (maximizing level of perfection while minimizing cost) and applies NSGA-II [29]— a common multi-objective genetic algorithm—to evolve a “fair” imperfect-fidelity setting, together with some initially archived seeds of configurations (Challenge 1). Imperfect-Fidelity Seeding (line 2) embeds the discovered imperfect-fidelity setting into the tuning, together with a two-stage tuning strategy, which covers a wide area of the space and generates high-quality configurations to improve archived seeds for tuning under the perfect-fidelity setting (Challenges 2 and 3). Perfect-Fidelity Assuring (line 3) initializes the tuning by filtering the archive of seeded configurations, hence fully exploiting the benefits from the tuning thereof for deeply exploring along the tuning directions implied and returns the best configuration found under the perfect-fidelity setting (Challenge 3). We equally split the budget into the above: tuning under the fair imperfect-fidelity setting for seeds has half the budget (since it has two-stage tuning), while the other two share a quarter of the budget each. All phases are connected by an archive 𝒜, which contains configurations progressively measured under the perfect-fidelity setting; hence, the tuning can stop whenever needed.

Perfect-fidelity measuring configurations under each explored imperfect-fidelity setting and setting the perfect-fidelity setting

ImperfectFidelity Discovering Fidelity setting space

The fair 2 imperfect-/ perfect-fidelity

Archived configurations

measuring configurations under the perfect-fidelity setting System measuring configurations under the fair imperfect-fidelity setting and the perfect-fidelity setting

ImperfectFidelity Seeding

3 Perfect-fidelity

Archived configurations

PerfectFidelity Assuring

Configuraiton space

Figure 4: Workflow and architecture overview of MFTune.

4.1

Discovering Fair Imperfect-Fidelity

At the beginning, MFTune aims to identify a fair imperfect-fidelity that has lower cost while exhibiting a good level of fidelity perfection to the target perfect-fidelity. As such, this forms a typical multi-objective search problem, for which we adopt the NSGAII [29] algorithm to explore the interactions between fidelity factors, together with their nonlinear implications to cost and fidelity perfection (Observations 2 and 3). Specifically, it runs in the following step with a quarter budget (also shown in Figure 5 and Algorithm 2): (1) Generate an archive 𝒜 = {𝒙 1, . . . , 𝒙𝑙 } with 𝑙 diverse configurations via Latin Hypercube Sampling (LHS) [57] to provide the basis for fidelity perfection ( 1 ). (2) Initialize a fidelity population 𝒬 = {𝒛 1, . . . , 𝒛𝑚 } by randomly sampling 𝑚 imperfect-fidelity settings ( 2 ). (3) Measure the configurations in 𝒜 under every fidelity setting in 𝒬 and the perfect-fidelity setting 𝒛 ∗ . These allow us to measure the cost (using Equation 3) and fidelity perfection (using Equation 4) for each of the imperfect-fidelity settings in 𝒬 ( 1 – 2 ). (4) Reproduce 𝑚 new imperfect-fidelity settings based on 𝒬 via boundary mutation and uniformed crossover operators4 in NSGA-II and measure the cost and fidelity perfection under newly explored imperfect-fidelity settings, if any, using the configurations in 𝒜 ( 3 – 5 ). (5) Keep the top 𝑚 imperfect-fidelity settings among the old and new ones as the new 𝒬 via the non-dominated sorting5 in NSGA-II on both the cost and fidelity perfection ( 6 ). 4 Boundary mutation perturbs a decision variable by resampling it within its feasible

range, while uniform crossover exchanges variables between two parent solutions. 5 Non-dominated sorting ranks solutions into Pareto fronts based on dominance relations, where a solution is non-dominated if no other solution is better in all objectives, followed by crowding distance [29].

ASE ’26, October 12–16, 2026, Munich, Germany

Yulong Ye, Miqing Li, and Tao Chen

Algorithm 2: ImperfectFidelityDiscovering

Algorithm 3: ImperfectFidelitySeeding

Input: Budget B ; perfect-fidelity setting 𝒛 ∗ ; consumed budget 𝜏 = 0 Output: The archive 𝒜 and the fair imperfect-fidelity setting 𝒛 𝑓 𝑎𝑖𝑟 1 𝒜 ← generate 𝑙 configurations using LHS 2 𝒬 ← randomly generate 𝑚 imperfect-fidelity settings and measure their cost and fidelity perfection using 𝒜 (via Equations 3 and 4) 3 𝜏 ← update total time taken

Input: Budget B ; fair imperfect-fidelity setting 𝒛 𝑓 𝑎𝑖𝑟 ; perfect-fidelity setting 𝒛 ∗ ; the archive 𝒜; the probability 𝛼 ; consumed budget 𝜏 = 0 Output: The updated archive 𝒜 /* Progressive Diversity Sampling */ 1

7 8

end

return 𝒜 , 𝒛 𝑓 𝑎𝑖𝑟 ← the imperfect-fidelity setting from 𝒬 with the biggest 𝜏 𝑖

𝜌𝑧 𝑧𝑖

Configurations

Measurement

Archived configuration

thread timeout …… 10000

……

1000

……

Current fidelity setting Newly explored fidelity setting

Sampled configurations

- Cost - Fidelity perfection

A fidelity setting (if it has not been measured)

Archive

The above consolidates the archive for more high-quality seeds that are either sub-optimal or optimal as measured under 𝒛 𝑓 𝑎𝑖𝑟 , since they could both be beneficial for tuning under 𝒛 ∗ .

4.3

Assuring under Perfect-Fidelity with Seeds

Tuning under the perfect-fidelity setting 𝒛 ∗ resembles a singlefidelity tuning (Figure 7 and Algorithm 4); we again use GA with boundary mutation/uniform crossover ( 3 – 7 ). Unlike classic GA, the archive 𝒜 accumulated so far serve as good starting points. MFTune firstly measures all configurations in 𝒜, which have not been measured, under 𝒛 ∗ . The configurations in 𝒜 would be filtered, such that only the top 𝑛 therein can be used as seeds, enabling a warm start ( 1 – 2 ). The tuning terminates and returns the best configuration found under 𝒛 ∗ when the quarter budget runs out. This final phase plays a pivotal role for the “depth” of tuning in MFTune, offering extra assurance to the imperfection/uncertainty of the fair imperfect-fidelity setting.

Tuning Guided by Perfect-Fidelity 5

Crossover

size time …… 100K 180 ……

Any unmeasured configuration 2

thread timeout ……

35 10000 …… 70 1000 …… …… …… ……

10000 ……

1000 ……

Mutation 70

10000 ……

System 35

1000 ……

10000 ……

15000 ……

6 Top configurations on performance under the perfectfidelity setting Configurations

Figure 7: Seeded tuning under perfect-fidelity setting.

5 • The initial set of configurations are the top 𝑛 ones obtained (under 𝒛 𝑓 𝑎𝑖𝑟 ) from the progressive diversity sampling. We also ensure that all configurations in 𝒜 so far have their performance values under 𝒛 ∗ (lines 11-15). • At each iteration, the best configuration measured under 𝒛 𝑓 𝑎𝑖𝑟 so far is also pushed into the archive 𝒜 and measured under 𝒛 ∗ (lines 16-25). This maintains local diversity by tracking the progress of tuning, hence further mitigating the possible “imperfection” of 𝒛 𝑓 𝑎𝑖𝑟 to 𝒛 ∗ . • At the end, all the persevered but non-archived configurations would be pushed to 𝒜 and measured under 𝒛 ∗ (line 26)— this seeks to align with Observation 1 that well-performing configurations under different fidelity settings might also be consistent.

Measurement Perfectfidelity setting

Performance (max)

Progressive Diversity Sampling

Sampling

ASE ’26, October 12–16, 2026, Munich, Germany

Experimental Setup

To evaluate MFTune , we ask four research questions (RQs): • RQ1: How beneficial are the imperfect-fidelity settings? • RQ2: How does MFTune perform compared with the stateof-the-art tuners? • RQ3: How does the imperfect-fidelity discovery and its twostage tuning individually contribute? • RQ4: What is the sensitivity of MFTune to 𝛼? All experiments are independently performed on Ubuntu 22.04.4 LTS running on a cluster of 20 servers, each of which is an Intel NUC with 16 CPU cores and 64GB RAM. In total, the full experiment takes over 19 months of CPU time 24×7.

5.1

Systems, Options, and Fidelity

Systems: As in Table 1, we select six widely used systems spanning diverse application domains and performance objectives while covering configuration space sizes between 1016 and 10724 . Options: Each system features a distinct set of options that may influence performance, including categorical options (e.g., enumerated and binary) and numerical options (e.g., integers), as have been used in prior work [16, 40, 49, 66, 67]. Fidelity: To generate fidelity settings, we use: • Sysbench [7]: A transactional database benchmarking tool, generating stressed workloads controlled by various fidelity factors (e.g., table-size and thread). • Wrk [9]: A high-performance HTTP load generator for web servers, controlling request load and traffic intensity via fidelity factors such as connections and duration. • Csmith [2]: A 𝐶 program generator that produces programs of diverse scale and structural complexity for compilers, controlled by, e.g., max-funcs and max-block-size.

ASE ’26, October 12–16, 2026, Munich, Germany

Yulong Ye, Miqing Li, and Tao Chen

Table 1: Subject systems studied. 𝒳 is the size of configuration space; (|𝒞|/|𝒩 |) are the number of categorical/numerical options. tps/rps means transactions/requests per second. System

Domain

Performance

|𝒞|/|𝒩 | 𝒳

MySQL [5] Database Throughput (tps) 2/18 PostgreSQL [6] Database Throughput (tps) 0/20 Tomcat [8] Web server Throughput (rps) 3/17 Httpd [4] Web server Throughput (rps) 7/13 Gcc [3] Compiler Runtime (ms) 20/0 Clang [1] Compiler Runtime (ms) 20/0

Cost (𝒛 ∗ ) Cost (𝒛𝑖 )

4.24 ×10724 ≈250s 8.70 ×10122 ≈250s 4.47 ×10118 ≈180s 4.76 ×1078 ≈180s 7.21 ×1016 ≈43s 7.21 ×1017 ≈40s

39s-90s 40s-76s 10s-60s 10s-101s 2s-6s 2s-4s

Table 2: Details of fidelity generators. “Type” refers to the type of fidelity factors from §3.1. 𝒛 ∗ and 𝒵 denote the perfectfidelity setting used and fidelity space, respectively. Fidelity Factors

Values

Type Budget Dataset Budget Workload Dataset Workload Budget Budget Workload Workload Dataset Workload Workload Workload

time tables threads r/w ratio table-size post threads duration connections max-funcs max-block-size max-block-depth inline-function-prob max-array-len-per-dim

{30, 35, ..., 180} {20, 25, ..., 50} {4, 6, ..., 10} {0.5, 0.6, ..., 0.9} {10𝐾, 15𝐾, ..., 100𝐾 } {𝑡𝑟𝑢𝑒, 𝑓 𝑎𝑙𝑠𝑒} {1, 2, ..., 10} {10, 15, ..., 180} {10, 15, ..., 50} {5, 10, ..., 50} {1, 2, 3, 4} {1, 2, 3, 6} {10, 50, ..., 90} {5, 10, ..., 50}

Benchmark

Sysbench

Wrk

Csmith

𝒵

11200

1296

5880

𝒛∗ 180 50 4 0.5 100𝐾 𝑡𝑟𝑢𝑒 8 180 50 50 4 6 10 50

The fidelity used is summarized in Table 2. In practice, the perfectfidelity setting is typically designated by developers based on business needs (often expensive), serving as the ground truth for assessing configuration quality. To stress test the tuners, we set the setting that incurs the highest costs as the perfect-fidelity setting under which we seek to tune the configuration.

Table 3: Summary of the compared tuners. Tuner

Fidelity

Strategy

Domain

Year

PromiseTune [18] HEBO [27] GA [59] FLASH+ [58] BestConfig [70] SMAC [45]

Single-fidelity Single-fidelity Single-fidelity Single-fidelity Single-fidelity Single-fidelity

Model-based Model-based Model-free Model-based Model-free Model-free

Configuration General General Configuration Configuration General

2026 2022 2020 2018 2017 2011

PriorBand [55] DEHB [11] BOHB [33] Hyperband [51]

Multi-fidelity Multi-fidelity Multi-fidelity Multi-fidelity

Model-free Model-free Model-based Model-free

HPO HPO HPO HPO

2023 2021 2018 2018

measure the relative efficiency and budget utilization of MFTune via ΔB: the budget saving (or overspend) for MFTune to reach the best performance of a counterpart tuner, if applicable, as ΔB = B𝑐 − B𝑚 , where B𝑐 is the earliest time the counterpart tuner reaches its best performance (average over runs) while B𝑚 is the time when MFTune achieves the same. As such, ΔB > 0 indicates the amount of budget saved by MFTune; or otherwise it means MFTune has no budget saving (ΔB = 0) or could even overspend (ΔB < 0).

5.4

5.5 5.2

Parameter Settings

State-of-the-Art Tuners

We compare MFTune against diverse types of tuners6 , spanning across different domains, as summarized in Table 3. • Single-fidelity tuners only conduct tuning under the perfectfidelity setting. • Multi-fidelity tuners exploit resource-allocation strategies across different fidelity settings. Since they work on a single fidelity factor and assume a monotonic relationship between cost and fidelity perfection, we preserve their original design by choosing the factor that is the most influential on cost. Regardless of the fidelity, a tuner can be either model-based or model-free: the former learns surrogate models (e.g., Gaussian Process) to approximate the configuration–performance landscape, steering tuning for the promising regions; the latter relies solely on system measurements to guide the tuning.

5.3

Tuning Budget

In this work, we set the tuning budget B as the wall-clock time allowed to tune the real system. To ensure realism of our experiment and consider the diverse measurement costs of different systems domains, we set B = 24 hours for MySQL/PostgreSQL, B = 12 hours for Tomcat/Httpd, and B = 4 hours for Gcc/Clang. These time budgets correspond to ≈ 250–350 measurements under the perfectfidelity setting, which aligns with most of the prior studies [20]. Note that we also examine the trajectories of tuning therein.

Metrics

The evaluation metrics would clearly be the performance metric that a system is of concerned, i.e., throughput or runtime. We also 6 FLASH uses exhaustive sampling at each iteration for acquisition evaluation, which

does not work on systems with an intractable space; we extend this, namely FLASH+, by randomly sampling 1000 configurations.

The initial sample size for model-based tuners (e.g., SMAC, FLASH+, and HEBO) is set to 30, following common practice in prior studies [19, 58]. For other parameters, such as the configuration population size (𝑛 = 20) of GA, we adopt either the default setting or those reported in the literature to ensure fair comparison [11, 18, 33, 51, 55]. For MFTune, the initial configuration sample size for discovering imperfect-fidelity settings 𝑙 and fidelity population size 𝑚 are both set to 10. We use a smaller fidelity population (𝑚 < 𝑛) because quantifying a fidelity setting is more expensive, which requires measuring a batch of 𝑙 configurations to estimate its perfection level. In particular, we set 𝛼 = 0.5—a generally best value (see §6.4). To avoid bias, we repeat 10 runs for each experiment.

5.6

Test for Statistical Significance

To ensure statistical significance when comparing the performance of multiple tuners, we employ the Scott-Knott ESD test [41]. In a nutshell, it begins by ranking tuners according to their mean performance, then recursively partitions this ordered list into statistically different subgroups. For instance, given three tuners 𝐴, 𝐵, and 𝐶, Scott-Knott ESD might partition them into two subgroups: {𝐴, 𝐵} at rank 𝑟 = 1 and {𝐶} at rank 𝑟 = 2. This indicates that 𝐴 and 𝐵 cannot be distinguished statistically, yet both significantly outperform 𝐶.

Less Is More: Tuning Configurable Systems with Imperfect Fidelity

ASE ’26, October 12–16, 2026, Munich, Germany

Table 4: Comparing MFTune with state-of-the-art tuners over 10 runs. ↑ and ↓ denote maximized (throughput) and minimized (runtime) performance, respectively. 𝑟 denotes Scott-Knott ESD rank on performance. ΔB ≥ 0 and ΔB < 0 refers to the budget saving and overspend of MFTune compared with a counterpart, respectively. ✗ indicates that MFTune fails to achieve the best of a counterpart when budget runs out. The best ranked tuner(s) for each system is highlighted in green . MySQL (↑) 𝑟 Mean (Std)

Hyperband BOHB DEHB PriorBand SMAC BestConfig FLASH+ GA HEBO PromiseTune MFTune

3 3 2 1 3 2 1 4 1 2 1

360.12 (20.61) 9.7 hours 7 498.71 (33.27) 358.35 (20.24) 9.9 hours 7 503.46 (34.03) 365.50 (30.26) 7.5 hours 6 514.26 (68.39) 378.02 (22.10) 0.1 hours 6 516.47 (37.93) 361.56 (31.48) 9.5 hours 2 556.77 (34.14) 365.06 (19.58) 7.1 hours 6 519.42 (35.35) 374.72 (14.79) − 0.5 hours 3 541.14 (30.30) 350.90 (30.00) 6.3 hours 3 542.72 (42.63) 378.03 (18.34) 0.0 hours 4 537.17 (22.20) 365.24 (23.10) 5.0 hours 5 532.60 (14.34) 378.14 (21.75) 0.0 hours 1 570.54 (37.20)

GA MFTune

350 ΔB = 6.3 hours

300

550

500

ΔB = 16.7 hours

(a) MySQL

3,700

ΔB = 6.6 hours

2,800

1 2

12 16 20 24

100

ΔB = 5.2 hours

3,400 3,100 2,800

Budget (hours)

(b) PostgreSQL GA MFTune Runtime (ms)

3,700

GA MFTune

Budget (hours)

4,000

2,500 1 4

12 16 20 24

3028.27 (303.15) 2826.15 (62.30) 2849.06 (80.17) 3049.98 (578.41) 2969.58 (297.48) 2817.72 (55.43) 3010.90 (268.23) 2877.19 (73.24) 2920.09 (118.56) 3259.97 (532.48) 3359.75 (710.07)

3,100

GA MFTune

100 Runtime (ms)

1 4

2 6 5 2 2 6 2 4 3 1 1

3,400

450

250

Throughput (rps)

13.1 hours 23.6 hours 21.7 hours 17.9 hours 4.2 hours 5.2 hours 16.6 hours 16.7 hours 22.8 hours 22.8 hours 0.0 hours

GA MFTune

600 Throughput (tps)

400 Throughput (tps)

PostgreSQL (↑) Httpd (↑) ΔB 𝑟 Mean (Std) ΔB 𝑟 Mean (Std)

Throughput (rps)

Tuner

ΔB = 0.5 hour

80 70

Tomcat (↑) ΔB 𝑟 Mean (Std) 3.4 hours 6.7 hours 6.7 hours 3.0 hours 5.0 hours 7.2 hours 3.6 hours 5.2 hours 4.9 hours 2.6 hours 0.0 hours

4 5 7 4 3 6 1 4 2 2 1

1 2

Budget (hours)

(d) Httpd

7.4 hours 6.6 hours 6.9 hours 6.8 hours 6.0 hours 6.4 hours 1.9 hours 6.6 hours 5.7 hours 4.2 hours 0.0 hours

3 5 4 4 2 4 2 2 5 3 1

68.01 (1.98) 69.85 (2.50) 69.65 (2.65) 69.62 (2.82) 67.26 (1.84) 69.29 (3.10) 67.92 (2.30) 67.78 (2.91) 70.14 (2.34) 69.12 (1.84) 67.01 (2.93)

Clang (↓) ΔB 𝑟 Mean (Std)

0.6 hours 0.2 hours 0.8 hours 0.2 hours − 0.9 hours 0.8 hours 0.6 hours 0.5 hours 0.7 hours 0.9 hours 0.0 hours

5 1 3 6 2 3 4 4 4 6 3

69.09 (3.03) 67.41 (2.83) 68.04 (3.40) 71.27 (2.76) 67.63 (2.60) 68.32 (2.23) 69.02 (1.90) 68.69 (2.41) 69.07 (2.97) 70.26 (1.40) 68.65 (4.43)

ΔB

0.1 hours ✗ ✗ 0.4 hours ✗ ✗ 0.0 hours − 1.4 hours − 0.1 hours 0.3 hours 0.0 hours

Notably, at the early stage, MFTune shows slightly slower convergence since part of its budget is devoted to discover the fair imperfect-fidelity setting. However, this initial investment quickly pays off as the subsequent tuning proceeds more efficiently. Once an appropriate fair imperfect-fidelity setting is found, the tuners benefits from it to accelerate convergence toward promising configurations under the perfect-fidelity setting. As a result, MFTune generally leads to significant budget utilization, saving up to ΔB = 24 − 7.3 = 16.7 hours (on PostgreSQL). Thus, we conclude that: RQ1: Considering imperfect fidelity settings in MFTune is greatly beneficial, yielding up to 16.77% performance improvement and 16.7 hours budget saving.

90 ΔB = −1.4 hours

80 70

6.2 2,500

2890.92 (67.34) 2880.84 (87.21) 2784.64 (50.90) 2899.83 (60.98) 2926.93 (133.76) 2820.93 (69.37) 3309.26 (730.02) 2898.54 (87.09) 3072.72 (613.00) 3145.03 (659.50) 3323.13 (843.83)

Gcc (↓) ΔB 𝑟 Mean (Std)

RQ2: Effectiveness and Efficiency

60 0

Budget (hours)

(e) Gcc

Budget (hours)

(f) Clang

Figure 8: Tuning trajectories with and without multi-fidelity over 10 runs.

Scott-Knott ESD is chosen because, unlike the Kruskal-Wallis test, it overcomes the confounding factor of overlapping groups and eliminates the dependence on post-hoc correlation [56].

6 Results and Analysis 6.1 RQ1: Benefits of Imperfect-Fidelity 6.1.1 Method. For RQ1, we compare MFTune with a classic GA, which is essentially the single fidelity version of MFTune that tunes under the perfect-fidelity setting only; all other designs are identical. We plot the trajectory of the best configuration found under the perfect-fidelity setting (the ones in archive for MFTune). 6.1.2 Results. As shown in Figure 8, MFTune consistently outperforms GA across all six systems, demonstrating superior performance and competitive convergence speed. In efficacy, MFTune achieves up to 16.77% performance improvement (on Httpd), suggesting that integrating the imperfect-fidelity setting enables more effective exploration of high-quality configurations for the perfectfidelity setting.

6.2.1 Method. For RQ2, we compare MFTune with 10 state-of-theart tuners on all systems. To ensure statistical significance, we use the Scott-Knott ESD test to rank the tuners over 10 runs. 6.2.2 Results. As from Table 4, MFTune achieves remarkable results: it is ranked the best in 83.33% (5/6) of the cases. This considerably outperforms the generally second-best tuner, FLASH+, which is ranked the best for 33.33% (2/6) cases only. Among single-fidelity tuners, model-based approaches (e.g., SMAC, FLASH+, HEBO, and PromiseTune) generally outperform model-free ones (e.g., GA and BestConfig), as their surrogate models better balance exploration and exploitation, steering the tuning towards higher-quality configurations. Notably, multi-fidelity HPO tuners (e.g., DEHB) perform poorly. This is because they assume a monotonic relationship between evaluation cost and fidelity perfection, which is commonly not held for configurable systems (as in §3.3). In contrast, MFTune’s handling of the interaction between multiple fidelity factors, together with the tuning for “wideness” and “depth”, has resulted in up to 19.34% performance improvement (on Tomcat). MFTune also shows clear practical advantages in budget utilization: it saves more budget than its counterparts in 50 out of 60 comparisons, with up to 23.6 hours. Therefore: RQ2: MFTune leads to considerably better tuning quality than others: it ranks first in 83.33% of the cases with up to 19.34% performance improvement, achieves better budget utilization in 50 comparisons, and saves up to 23.6 hours.

System

MFTune-I 𝑟 Mean (Std)

MFTune-II ΔB 𝑟 Mean (Std)

MFTune ΔB 𝑟 Mean (Std)

MySQL 1 375.53 (17.12) 1.7 hours 2 338.17 (34.69) 13.1 hours ✗ 3 552.67 (44.61) 13.3 hours PostgreSQL 1 601.17 (30.57) Tomcat 2 2977.38 (271.63) 7.3 hours 3 2810.84 (63.48) 6.6 hours Httpd 2 2761.12 (73.68) 6.4 hours 3 2730.95 (58.04) 7.2 hours 3 70.79 (2.48) 1.7 hours 2 70.17 (2.78) 0.4 hours Gcc Clang 2 68.91 (3.58) 0.0 hours 2 69.57 (2.20) 0.2 hours

6.3

1 378.14 (21.75) 2 570.54 (37.20) 1 3323.13 (843.83) 1 3359.75 (710.07) 1 67.01 (2.93) 1 68.65 (4.43)

RQ3: Ablation Study

0 0.1

0.3

0.5 0.7 𝛼 value

0.9

1 0.8 0.6 0.4 0.2 0 0.1

0.3

0.5 0.7 𝛼 value

0.9

(b) Average performance

(a) Scott-Knott rank

Figure 9: The sensitivity of MFTune to 𝛼 values over all systems/runs (smaller normalized performance is preferred).

6.3.1

6.3.2 Results. As summarized in Table 5, MFTune obtains the best rank in the majority of the cases (5/6), suggesting the necessity of both. Indeed, omitting imperfect-fidelity discovery might lead to a rather harmful imperfect-fidelity setting being used; simplifying two-stage tuning under imperfect-fidelity might cause the tuning to be prone to local-optimal, since without the global diversity encouragement, the “wideness” of tuning would be restricted. In light of the above, we can conclude: RQ3: Discovering imperfect-fidelity and its two-stage tuning are both essential to the superiority of MFTune.

6.4

40 0.6

0.7 0.8 0.9 1 Fidelity perfection

(a) Iteration 𝑘 + 1

70 Cost (s)

Method. For RQ3, we ablate MFTune with two variants: • MFTune-I removes the first phase of imperfect-fidelity discovering and randomly select the fair imperfect-fidelity. • MFTune-II simplifies the two-stage tuning under the fair imperfect-fidelity setting, leaving only one stage without the progressive diversity sampling. Again, we apply the Scott-Knott test on all comparisons.

Average rank

Table 5: Comparing MFTune with its two variants over 10 runs. The format is the same as Table 4.

Normalized avg. performance

Yulong Ye, Miqing Li, and Tao Chen

# cases ranked the best

ASE ’26, October 12–16, 2026, Munich, Germany

fair imperfectfidelity setting

60 50

0.7 0.8 0.9 1 Fidelity perfection

(b) Iteration 𝑘 + 2

40 0.6

0.7

0.8

0.9

Fidelity perfection

Figure 10: The changes within imperfect-fidelity discovering on MySQL. “★” and “◦” denote the imperfect-fidelity settings and the nondominated imperfect-fidelity settings, respectively, where the perfect-fidelity setting has a cost ≈ 250s.

that MFTune progressively reveals candidate imperfect-fidelity settings that yield better fidelity perfection at lower costs, enabling MFTune with a good start for both “wideness” and “depth” of tuning.

RQ4: Sensitivity to 𝛼

6.4.1 Method. To study the sensitivity of MFTune to the 𝛼—the probability of including diverse configurations under the fair imperfectfidelity—in RQ4, we study five values: {0.1, 0.3, 0.5, 0.7, 0.9}. Scott Knott ESD is used to ensure statistical significance over 10 runs. 6.4.2 Results. As shown in Figure 9, the performance of MFTune is sensitive to 𝛼, but generally 𝛼 = 0.5 achieves the best outcomes on both rank and average performance—neither too small nor too large 𝛼 is ideal. This makes sense, because a too small 𝛼 would cause too few diverse configurations to be archived, which may lose some configurations that perform well under the perfect-fidelity setting but are sub-optimal under the fair imperfect-fidelity setting. In contrast, when 𝛼 is too large, too many diverse configurations might be archived, hence missing the chance to find those that perform well under both settings. Overall, we show that: RQ4: MFTune is sensitives to 𝛼, but 𝛼 = 0.5 is generally the best.

7 Discussion 7.1 How Imperfect-Fidelity Discovery Helps? To understand how MFTune benefits from the imperfect-fidelity discovery, Figure 10 illustrates how the explored imperfect-fidelity settings evolve across iterations when tuning MySQL. We can see

7.2

Why Tuning for Imperfect-Fidelity Work?

To disclose why tuning under a fair imperfect-fidelity setting 𝒛 𝑓 𝑎𝑖𝑟 works, Figure 11 plots an example of the tuning trajectory under the fair imperfect-fidelity setting 𝒛 𝑓 𝑎𝑖𝑟 and the changing top 𝑛 configurations in the archive 𝒜 when tested under the perfect-fidelity setting 𝒛 ∗ . We see that the progressive diversity sampling increases the performance measured under both 𝒛 𝑓 𝑎𝑖𝑟 and 𝒛 ∗ at a slow but steady pace. At around the 25th iteration, the large improvement of the archive under 𝒛 ∗ is due to the top configurations under 𝒛 𝑓 𝑎𝑖𝑟 being pushed in. Both trace increases more steeply afterwards, especially when the overall top-performing configurations for 𝒛 𝑓 𝑎𝑖𝑟 are pushed to the archive again at the 40th iteration. Notably, the wall-clock time required for tuning/exploring the configuration under 𝒛 𝑓 𝑎𝑖𝑟 is only ≈ 10 hours, compared with the ≈ 55 hours if we were to tune/explore the same configurations under 𝒛 ∗ . This means that, if given the same wall-clock time (e.g., 10 hours), the number of configurations that can be explored under 𝒛 𝑓 𝑎𝑖𝑟 would be much larger than that under 𝒛 ∗ (782 vs. 142), leading to superior results measured under 𝒛 ∗ even though 𝒛 𝑓 𝑎𝑖𝑟 is imperfect. The above evidence supports the rationale of MFTune’s superiority: finding the right imperfect-fidelity setting and properly tuning configurations can greatly help the tuning under the perfect-fidelity setting—a reflection of our hypothesis mentioned in §1.

Less Is More: Tuning Configurable Systems with Imperfect Fidelity Progressive Diversity Sampling

ASE ’26, October 12–16, 2026, Munich, Germany

Diversity-Preserved Tuning

• Option pruning (e.g., OpAdviser [68]) learns prior data under certain environments/workloads to identify key options/ranges for tuning under a new workload.

Throughput (tps)

450

400 Measuring under fair imperfect-fidelity setting (averaging all configurations in the sample set/population) Measuring under perfect-fidelity setting (averaging the top 𝑛 configurations in the archive 𝒜)

350

MFTune fundamentally differs from them in several aspects: • They assume a reactive situation where a tuner can only reuse historically explored and structurally similar environments/workloads, limiting their ability to adapt to a completely unforeseen environment; while MFTune is a proactive tuner that actively explores unknown fidelity settings. • They assume dozens of environments/workloads; MFTune scales to ten thousand environments/fidelity settings. • They typically reuse the knowledge of other environments/workloads for updating the surrogate model or dimensionality reduction, while MFTune leverages them for consolidating the tuning process directly.

300

250 1

20 30 Iteration in MFTune

Figure 11: Tuning trajectory of MySQL under the fair imperfect-fidelity and testing the corresponding configurations found thereof under the perfect-fidelity setting.

7.3

Threats to Validity

Internal threats to validity could be raised from the parameter setting of 𝛼, which controls the probability of triggering full-fidelity evaluations during low-fidelity guided tuning. In our study, we set 𝛼 = 0.5, following empirical evidence from our sensitivity analysis (RQ3), where this value offered a favorable balance between exploration cost and evaluation reliability. Similarly, parameter 𝑙 controls the number of paired configurations for estimating fidelity perfection. We set 𝑙 = 10 to strike a trade-off between estimation reliability and discovery cost: a large 𝑙 may provide a more reliable estimate but also incurs higher cost for fidelity discovery. Nevertheless, we recognize that the best value of these parameters may vary across systems, and exploring alternative settings could further enhance robustness.

Related Work

Tuning with or without models. Model-free tuners address configuration tuning by guiding the search process exclusively through direct system measurements without models [13, 15, 64]. For example, GA has inspired many tuners that work on a population of configurations [13, 59]. In contrast, model-based tuners learn a surrogate model from measured configuration-performance, paired with other heuristics, to accelerate the tuning [10, 16, 17, 26, 35, 36, 69]. However, all of the above are single-fidelity tuners, where each configuration is measured at the perfect-fidelity setting, guaranteeing accurate measurements at the expense of high costs. MFTune, in contrast, explicitly leverages the imperfect-fidelity setting to improve the tuning and budget utilization. Multi-workloads tuners. Recent works have transferred knowledge and reused information from past tuning tasks/environments/workloads, which might be similar to the fidelity: • Workload mapping (e.g., OtterTune [10]) identifies the most similar historical environment/workload and reuses the data to build a surrogate model. • Model ensemble (e.g., ResTune [67]) combines multiple surrogate models trained on prior environments/workloads and generalizes such to a new workload for tuning.

Multi-fidelity optimization. Multiple fidelity settings have been considered primarily in the context of HPO [34, 44, 50]. A representative solution is Hyperband [51], which samples a set of random solutions and progressively allocates more resources to promising ones via successive halving [46]. Variants such as BOHB [33] and DEHB [11] extend Hyperband by incorporating Bayesian Optimization and Differential Evolution, respectively, to enhance optimization efficiency. However, as mentioned, they mostly consider a limited dimension of the fidelity factors and have also assumed a monotonic correlation between fidelity perfection and cost [14, 31, 42], which, as we have shown, is not the case for configurable systems. This work formulates a conceptual framework specifically for system configuration tuning while presenting MFTune as the tuner that works effectively without the above constraints/assumptions.

Conclusion

This paper presents MFTune, a tuner that tackles system configuration tuning by proactively exploring and exploiting the imperfectfidelity setting. Drawing on a newly formulated conceptual framework of multi-fidelity for configurable systems, MFTune explore in the scale of more than ten thousand imperfect-fidelity settings, finding the most fair one for tuning, which then seeds the tuning under the perfect-fidelity setting for improved budget utilization and superior results. Experiments against 10 state-of-the-art tuners and under six real-world systems show that MFTune achieves better results on 83.33% cases with up to 19.34% improvements, while doing so by generally saving hours’ budget. Looking forward, we envision that this work can inspire new research directions that integrate multi-fidelity with diverse optimization paradigms, paving the way for more advanced configuration tuning for configurable systems.

Acknowledgments This work was supported by an NSFC Grant (62372084) and a UKRI Grant (10054084).

Data Availability Statement All source code and data are available at: https://github.com/ideaslabo/mftune and https://zenodo.org/records/21364961.

ASE ’26, October 12–16, 2026, Munich, Germany

References [1] 2024. Clang. https://releases.llvm.org/download.html. Version: 17.0.6. Accessed: 2024-02-20. [2] 2024. Csmith. https://github.com/csmith-project/csmith. Accessed: 2024-02-20. [3] 2024. Gcc. https://gcc.gnu.org/pub/gcc/releases/gcc-14.2.0/. Version: 14.2. Accessed: 2024-02-20. [4] 2024. Httpd. https://httpd.apache.org/download.cgi. Version: 2.4.57. Accessed: 2024-02-20. [5] 2024. MySQL. https://dev.mysql.com/doc/relnotes/mysql/5.7/en/. Version: 5.7.19. Accessed: 2024-02-20. [6] 2024. PostgreSQL. https://www.postgresql.org/docs/release/12.7/. Version: 12.7. Accessed: 2024-02-20. [7] 2024. Sysbench. https://github.com/akopytov/sysbench. Accessed: 2024-02-20. [8] 2024. Tomcat. https://tomcat.apache.org/download-10.cgi. Version: 10.1.34. Accessed: 2024-02-20. [9] 2024. Wrk. https://github.com/wg/wrk. Accessed: 2024-02-20. [10] Dana Van Aken, Andrew Pavlo, Geoffrey J. Gordon, and Bohan Zhang. 2017. Automatic Database Management System Tuning Through Large-scale Machine Learning. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD Conference 2017, Chicago, IL, USA, May 14-19, 2017, Semih Salihoglu, Wenchao Zhou, Rada Chirkova, Jun Yang, and Dan Suciu (Eds.). ACM, 1009–1024. doi:10.1145/3035918.3064029 [11] Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. 2021. DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, Zhi-Hua Zhou (Ed.). ijcai.org, 2147–2153. doi:10.24963/IJCAI.2021/296 [12] Thomas Bäck and Hans-Paul Schwefel. 1993. An Overview of Evolutionary Algorithms for Parameter Optimization. Evol. Comput. 1, 1 (1993), 1–23. doi:10. 1162/EVCO.1993.1.1.1 [13] Babak Behzad, Huong Vu Thanh Luu, Joseph Huchette, Surendra Byna, Prabhat, Ruth A. Aydt, Quincey Koziol, and Marc Snir. 2013. Taming parallel I/O complexity with auto-tuning. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC’13, Denver, CO, USA - November 17 - 21, 2013, William Gropp and Satoshi Matsuoka (Eds.). ACM, 68:1–68:12. doi:10.1145/ 2503210.2503278 [14] Timur Carstensen, Neeratyoy Mallik, Frank Hutter, and Martin Rapp. 2025. Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimization. In Proceedings of the Fourth International Conference on Automated Machine Learning (Proceedings of Machine Learning Research, Vol. 293), Leman Akoglu, Carola Doerr, Jan N. van Rijn, Roman Garnett, and Jacob R. Gardner (Eds.). PMLR, 4/1–24. https://proceedings.mlr.press/v293/carstensen25a.html [15] Jianfeng Chen, Vivek Nair, Rahul Krishna, and Tim Menzies. 2019. "Sampling" as a Baseline Optimizer for Search-Based Software Engineering. IEEE Trans. Software Eng. 45, 6 (2019), 597–614. doi:10.1109/TSE.2018.2790925 [16] Junjie Chen, Ningxin Xu, Peiqi Chen, and Hongyu Zhang. 2021. Efficient Compiler Autotuning via Bayesian Optimization. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021. IEEE, 1198– 1209. doi:10.1109/ICSE43902.2021.00110 [17] Pengzhou Chen and Tao Chen. 2026. CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG. In Proceedings of the 35th International Joint Conference on Artificial Intelligence, IJCAI 2026, Bremen, Germany, 15-21 August 2026. ijcai.org. [18] Pengzhou Chen and Tao Chen. 2026. PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE). [19] Pengzhou Chen, Tao Chen, and Miqing Li. 2024. MMO: Meta MultiObjectivization for Software Configuration Tuning. IEEE Trans. Software Eng. 50, 6 (2024), 1478–1504. doi:10.1109/TSE.2024.3388910 [20] Pengzhou Chen, Jingzhi Gong, and Tao Chen. 2025. Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning. IEEE Trans. Software Eng. 51, 2 (2025), 548–580. doi:10.1109/TSE.2025.3525955 [21] Tao Chen and Rami Bahsoon. 2015. Toward a Smarter Cloud: Self-Aware Autoscaling of Cloud Configurations and Resources. Computer 48, 9 (2015), 93–96. doi:10.1109/MC.2015.278 [22] Tao Chen, Rami Bahsoon, and Xin Yao. 2018. A Survey and Taxonomy of SelfAware and Self-Adaptive Cloud Autoscaling Systems. ACM Comput. Surv. 51, 3 (2018), 61:1–61:40. doi:10.1145/3190507 [23] Tao Chen, Ke Li, Rami Bahsoon, and Xin Yao. 2018. FEMOSAA: Feature-guided and knee-driven multi-objective optimization for self-adaptive software. ACM Transactions on Software Engineering and Methodology (TOSEM) 27, 2 (2018), 1–50. [24] Tao Chen and Miqing Li. 2021. Multi-objectivizing software configuration tuning. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). ACM, 453–465. doi:10.1145/3468264.3468555

Yulong Ye, Miqing Li, and Tao Chen

[25] Tao Chen and Miqing Li. 2024. Adapting Multi-objectivized Software Configuration Tuning. Proc. ACM Softw. Eng. 1, FSE (2024), 539–561. doi:10.1145/3643751 [26] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Q. Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, Carlsbad, CA, USA, October 8-10, 2018, Andrea C. Arpaci-Dusseau and Geoff Voelker (Eds.). USENIX Association, 578–594. https://www.usenix.org/conference/osdi18/presentation/chen [27] Alexander I Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. 2022. Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation. Journal of Artificial Intelligence Research 74 (2022), 1269–1349. [28] Trevor F Cox and Michael AA Cox. 2000. Multidimensional scaling. CRC press. [29] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 2 (2002), 182–197. doi:10.1109/4235.996017 [30] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015. Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, Qiang Yang and Michael J. Wooldridge (Eds.). AAAI Press, 3460–3468. http://ijcai.org/Abstract/15/487 [31] Judith Echevarrieta, Etor Arza, and Aritz Pérez. 2025. Speeding-Up Evolutionary Algorithms to Solve Black-Box Optimization Problems. IEEE Trans. Evol. Comput. 29, 1 (2025), 117–131. doi:10.1109/TEVC.2024.3352450 [32] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor H. Awad, Marius Lindauer, and Frank Hutter. 2021. HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, Joaquin Vanschoren and Sai-Kit Yeung (Eds.). https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ 93db85ed909c13838ff95ccfa94cebd9-Abstract-round2.html [33] Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 (Proceedings of Machine Learning Research, Vol. 80), Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 1436–1445. http: //proceedings.mlr.press/v80/falkner18a.html [34] Alexander IJ Forrester, András Sóbester, and Andy J Keane. 2007. Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences 463, 2088 (2007), 3251–3269. [35] Jingzhi Gong and Tao Chen. 2023. Predicting Software Performance with Divideand-Learn. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023, Satish Chandra, Kelly Blincoe, and Paolo Tonella (Eds.). ACM, 858–870. doi:10.1145/3611643.3616334 [36] Jingzhi Gong and Tao Chen. 2024. Predicting Configuration Performance in Multiple Environments with Sequential Meta-Learning. Proc. ACM Softw. Eng. 1, FSE (2024), 359–382. doi:10.1145/3643743 [37] Jingzhi Gong, Tao Chen, and Rami Bahsoon. 2025. Dividable Configuration Performance Learning. IEEE Trans. Software Eng. 51, 1 (2025), 106–134. doi:10. 1109/TSE.2024.3491945 [38] Xue Han and Tingting Yu. 2016. An Empirical Study on Performance Bugs for Highly Configurable Software Systems. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2016, Ciudad Real, Spain, September 8-9, 2016. ACM, 23:1–23:10. doi:10. 1145/2961111.2962602 [39] Jan Hauke and Tomasz Kossowski. 2011. Comparison of values of Pearson’s and Spearman’s correlation coefficients on the same sets of data. Quaestiones geographicae 30, 2 (2011), 87–93. [40] Haochen He, Zhouyang Jia, Shanshan Li, Yue Yu, Chenglong Zhou, Qing Liao, Ji Wang, and Xiangke Liao. 2022. Multi-Intention-Aware Configuration Selection for Performance Tuning. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022. ACM, 1431–1442. doi:10.1145/3510003.3510094 [41] Steffen Herbold. 2017. Comments on ScottKnottESD in Response to "An Empirical Comparison of Model Validation Techniques for Defect Prediction Models". IEEE Trans. Software Eng. 43, 11 (2017), 1091–1094. doi:10.1109/TSE.2017.2748129 [42] Md. Abir Hossen, Mohammad Ali Javidian, Vignesh Narayanan, Jason M. O’Kane, and Pooyan Jamshidi. 2026. Multi-Objective Multi-Fidelity Bayesian Optimization with Causal Priors. CoRR abs/2602.00788 (2026). arXiv:2602.00788 doi:10.48550/ ARXIV.2602.00788 [43] Qinghao Hu, Zhisheng Ye, Meng Zhang, Qiaoling Chen, Peng Sun, Yonggang Wen, and Tianwei Zhang. 2023. Hydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters. In 17th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2023, Boston, MA, USA, July 10-12, 2023, Roxana Geambasu and Ed Nightingale (Eds.). USENIX Association, 757–777.

Less Is More: Tuning Configurable Systems with Imperfect Fidelity

https://www.usenix.org/conference/osdi23/presentation/hu [44] Yi-Qi Hu, Yang Yu, Wei-Wei Tu, Qiang Yang, Yuqiang Chen, and Wenyuan Dai. 2019. Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. AAAI Press, 3846–3853. doi:10.1609/AAAI.V33I01.33013846 [45] Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2011. Sequential ModelBased Optimization for General Algorithm Configuration. In Learning and Intelligent Optimization - 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers (Lecture Notes in Computer Science, Vol. 6683), Carlos A. Coello Coello (Ed.). Springer, 507–523. doi:10.1007/978-3-642-25566-3_40 [46] Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic Best Arm Identification and Hyperparameter Optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016 (JMLR Workshop and Conference Proceedings, Vol. 51), Arthur Gretton and Christian C. Robert (Eds.). JMLR.org, 240–248. http: //proceedings.mlr.press/v51/jamieson16.html [47] Pooyan Jamshidi and Giuliano Casale. 2016. An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems. In 24th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2016, London, United Kingdom, September 19-21, 2016. IEEE Computer Society, 39–48. doi:10.1109/MASCOTS.2016.17 [48] Kirthevasan Kandasamy, Gautam Dasarathy, Jeff G. Schneider, and Barnabás Póczos. 2017. Multi-fidelity Bayesian Optimisation with Continuous Approximations. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1799–1808. http://proceedings.mlr.press/v70/kandasamy17a.html [49] Konstantinos Kanellis, Cong Ding, Brian Kroth, Andreas Müller, Carlo Curino, and Shivaram Venkataraman. 2022. LlamaTune: Sample-Efficient DBMS Configuration Tuning. Proc. VLDB Endow. 15, 11 (2022), 2953–2965. doi:10.14778/3551793. 3551844 [50] Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. 2017. Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA (Proceedings of Machine Learning Research, Vol. 54), Aarti Singh and Xiaojin (Jerry) Zhu (Eds.). PMLR, 528–536. http://proceedings.mlr.press/v54/klein17a.html [51] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. J. Mach. Learn. Res. 18 (2017), 185:1–185:52. https://jmlr.org/papers/ v18/16-558.html [52] Yang Li, Yu Shen, Jiawei Jiang, Jinyang Gao, Ce Zhang, and Bin Cui. 2021. MFESHB: Efficient Hyperband with Multi-Fidelity Quality Measurements. In ThirtyFifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. AAAI Press, 8491–8500. doi:10.1609/AAAI.V35I10.17031 [53] Hongyuan Liang, Yue Huang, and Tao Chen. 2025. The Same Only Different: On Information Modality for Configuration Performance Analysis. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 2522–2534. doi:10.1109/ICSE55347.2025.00212 [54] Youpeng Ma, Tao Chen, and Ke Li. 2025. Faster Configuration Performance Bug Testing with Neural Dual-Level Prioritization. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering. IEEE Press, 988–1000. https://doi.org/10.1109/ICSE55347.2025.00201 [55] Neeratyoy Mallik, Edward Bergman, Carl Hvarfner, Danny Stoll, Maciej Janowski, Marius Lindauer, Luigi Nardi, and Frank Hutter. 2023. PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/ 1704fe7aaff33a54802b83a016050ab8-Abstract-Conference.html [56] Mary L McHugh. 2011. Multiple comparison analysis testing in ANOVA. Biochemia medica 21, 3 (2011), 203–209.

ASE ’26, October 12–16, 2026, Munich, Germany

[57] Michael D. McKay. 1992. Latin Hypercube Sampling as a Tool in Uncertainty Analysis of Computer Models. In Proceedings of the 24th Winter Simulation Conference, Arlington, VA, USA, December 13-16, 1992, Robert C. Crain (Ed.). ACM Press, 557–564. doi:10.1145/167293.167637 [58] Vivek Nair, Zhe Yu, Tim Menzies, Norbert Siegmund, and Sven Apel. 2020. Finding Faster Configurations Using FLASH. IEEE Trans. Software Eng. 46, 7 (2020), 794– 811. doi:10.1109/TSE.2018.2870895 [59] Arman Shahbazian, Suhrid Karthik, Yuriy Brun, and Nenad Medvidovic. 2020. eQual: informing early design decisions. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1039–1051. doi:10.1145/3368089. 3409749 [60] Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger (Eds.). 2960–2968. https://proceedings.neurips.cc/paper/2012/hash/ 05311655a15b75fab86956663e1819cd-Abstract.html [61] Zezhen Xiang, Jingzhi Gong, and Tao Chen. 2026. Dually Hierarchical Drift Adaptation for Online Configuration Performance Learning. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE). [62] Gangda Xiong and Tao Chen. 2025. CoTune: Co-evolutionary Configuration Tuning. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 1490–1502. doi:10.1109/ASE63991.2025.00126 [63] Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker. 2015. Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015, Elisabetta Di Nitto, Mark Harman, and Patrick Heymans (Eds.). ACM, 307–319. doi:10.1145/2786805.2786852 [64] Yulong Ye, Tao Chen, and Miqing Li. 2025. Distilled Lifelong Self-Adaptation for Configurable Systems. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1333– 1345. doi:10.1109/ICSE55347.2025.00094 [65] Yulong Ye, Hongyuan Liang, Chao Jiang, Miqing Li, and Tao Chen. 2026. Revealing Domain-Spatiality Patterns for Configuration Tuning: Domain Knowledge Meets Fitness Landscapes. ACM Trans. Softw. Eng. Methodol. (March 2026). Just Accepted. doi:10.1145/3803859 [66] Xinyi Zhang, Zhuo Chang, Yang Li, Hong Wu, Jian Tan, Feifei Li, and Bin Cui. 2022. Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation. Proc. VLDB Endow. 15, 9 (2022), 1808– 1821. doi:10.14778/3538598.3538604 [67] Xinyi Zhang, Hong Wu, Zhuo Chang, Shuowei Jin, Jian Tan, Feifei Li, Tieying Zhang, and Bin Cui. 2021. ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases. In SIGMOD ’21: International Conference on Management of Data, Virtual Event, China, June 20-25, 2021, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 2102–2114. doi:10.1145/ 3448016.3457291 [68] Xinyi Zhang, Hong Wu, Yang Li, Zhengju Tang, Jian Tan, Feifei Li, and Bin Cui. 2023. An Efficient Transfer Learning Based Configuration Adviser for Database Tuning. Proc. VLDB Endow. 17, 3 (2023), 539–552. doi:10.14778/3632093.3632114 [69] Mingxuan Zhu and Dan Hao. 2023. Compiler Auto-Tuning via Critical Flag Selection. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 1000–1011. doi:10.1109/ASE56229.2023.00209 [70] Yuqing Zhu, Jianxun Liu, Mengying Guo, Yungang Bao, Wenlong Ma, Zhuoyue Liu, Kunpeng Song, and Yingchun Yang. 2017. BestConfig: tapping the performance potential of systems via automatic configuration tuning. In Proceedings of the 2017 Symposium on Cloud Computing, SoCC 2017, Santa Clara, CA, USA, September 24-27, 2017. ACM, 338–350. doi:10.1145/3127479.3128605

Received 2026-03-26; accepted 2026-06-18

Less Is More: Tuning Configurable Systems with Imperfect Fidelity

Analyze & explore related research in Conceptio

Related documents