Finding Performance Issues in Database Systems by Exploiting Dormant Code Paths
arXiv:2605.22992v1 [cs.SE] 21 May 2026
Jinsheng Ba ETH Zurich Switzerland
Zhendong Su ETH Zurich Switzerland
Abstract
Automatically finding performance issues is challenging due to the complexity of modern DBMSs. First, determining whether executing a query on a database results in unexpectedly suboptimal performance is difficult, as there is typically no ground truth to define what constitutes optimal performance. This challenge is further compounded by the use of various heuristics and cost models during query optimization, which often involve trade-offs that prioritize optimizing certain types of queries at the expense of others. Second, complex optimization strategies in DBMSs make it challenging to systematically test performance. For a given query, DBMSs may apply join optimization to determine the optimal join order, index optimization to select the appropriate indexes, and so on. These optimizations collectively influence performance. Relying solely on input queries, however, cannot comprehensively explore the full range of optimization strategies and their interactions. Various black-box methods have been proposed to derive “optimal” performance for finding performance issues, but they focus on a specific category of issues and cannot systematically explore optimization strategies. APOLLO [16] examines whether a DBMS has worse performance than an old version of the same DBMS. However, APOLLO can only find regression issues. AMOEBA [21] expects that the execution time of pairs of semantically equivalent queries is similar. However, it only finds the issues of rewriting queries [21]. CERT [1] finds performance issues by finding inconsistent cardinality estimation, which is typically deemed as the most critical component for query optimization [17]. However, CERT can only find performance issues related to incorrect cardinality estimation. PUPPY [40] assumes that the execution time of queries under the default optimization configuration should be no longer than under alternative configurations. However, this approach is limited to detecting configuration-related performance issues. These methods focus on manipulating input queries, which are inefficient for exploring optimization strategies. In this paper, we propose Branch Flip Analysis (BFA ), a novel, general white-box method to systematically find per-
Performance is a critical characteristic of fundamental systems, such as Database Management Systems (DBMSs). Both academia and industry have invested decades in exploring efficient optimization algorithms. Despite these efforts, DBMSs are prone to performance issues, which incur suboptimal performance. Finding such issues is a longstanding challenge as no ground-truth performance is available. Existing work adopts black-box methods to examine performance consistency across executions, but cannot systematically test optimizations. In this work, we propose a novel, general white-box methodology, Branch Flip Analysis (BFA), to systematically and effectively uncover performance issues. BFA flips code branches to enforce or disable an optimization, and the performance is expected to be not significantly better. Otherwise, a performance issue exists. BFA provides a new perspective to finding performance issues and testing optimization logics in a fine-grained manner. We realized BFA in a prototype system QueryZen, and evaluated it on four widely-used and mature DBMSs: PostgreSQL, MySQL, CockroachDB, and MariaDB. QueryZen found 21 previously unknown and unique performance issues with the workload of the extensively used benchmarks TPC-H and TPC-DS. The core concept of BFA is simple and broadly applicable, and can be adapted to analyze the performance of other software systems.
1
Introduction
Database Management Systems (DBMSs) serve as the backbone for data manipulation and processing across various applications. Ensuring optimal DBMS performance is critical for maintaining the efficiency, scalability, and reliability of the systems they support. Over the past decades, researchers and practitioners have explored various algorithms and architectures to optimize performance, including join order search space exploration [6, 7, 24] and hardware acceleration [9, 25, 39]. However, performance issues that incur suboptimal performance remain a persistent challenge. 1
formance issues by manipulating the DBMS source code. Although a DBMS’s default optimization strategy is not guaranteed to be optimal, it should still deliver near-optimal performance. Our idea is to selectively disable or enforce specific optimizations through code manipulation to construct a reference DBMS. This reference version should never significantly outperform the original. Otherwise, a performance issue exists in the original DBMS, as it should achieve at least comparable performance to the reference DBMS. We propose a simple and general code manipulation operation that flips code branches of IF statements, which we observed are commonly used for optimization decisions. Formally, we define BFA as follows. For a DBMS’s source code, B = {b1 , b2 , ..., bn } is the set of all conditional branches of the IF statements that enable or disable optimization strategies. For a branch bi ∈ B, let bi,d and bi,o denote, respectively, the default and the alternative execution path. Given a specific workload, P(B) represents the performance of the DBMS with B. For a specific branch b j ∈ B, we flip its branch and derive B′ such that ∀i ̸= j (bi = bi,d )&b j = b j,o , i.e., all branches except b j execute their default paths bi,d , while b j executes the alternative path b j,o . If P(B) << P(B′ ), a performance issue related to the optimization controlled by b j is found. Thus, in a novel manner, BFA solves the aforementioned challenges faced by existing work—it derives expected performance by constructing a reference DBMS. Code manipulation also allows us to explore various optimization strategies in a systematic and fine-grained manner. In contrast, existing work is constrained by external inputs or configurations, which often fail to control all the available optimizations. To manipulate code, we propose flipping branches, which is motivated by our observation that 8 of 10 historical performance issues from APOLLO , AMOEBA , CERT , and PUPPY could be found by flipping branches. Branch flipping is straightforward to realize and general, so it can be widely applied to test DBMSs without domain expertise. Additionally, it highlights the code segment relevant to the performance issue, thus facilitating issue analysis. Listing 1 shows a running example of BFA . Lines 1–3 include the query to identify the performance issue, lines 4–12 show the code that BFA manipulates, and lines 14–19 show the query plans, the concrete execution steps of this query in a tree structure, before and after the code manipulation. To facilitate the presentation, we only show relevant parts. This query includes three-table joining. SUPPLIER is joined with PARTSUPP by HashJoin, as shown in the left part of line 18. Their result is joined with PART by HashJoin again, as shown in the left part of line 15. The IF statement in line 10 checks whether to ignore PARALLEL optimization, which is an optimization strategy in PostgreSQL that executes an operation in multiple CPU cores for efficiency. For this query, this optimization only applies to the scanning of PART in line 16. We flipped the IF branch, so the PARALLEL optimization is applied to other operations HashJoin and Hash. We found that PostgreSQL has better per-
Listing 1: A performance issue found by BFA in PostgreSQL. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
SELECT ... FROM PART , SUPPLIER , PARTSUPP ... WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey ...; -- ---------------- Code Manipulation -------------------- a/ src / backend / optimizer / util / clauses .c +++ b/ src / backend / optimizer / util / clauses .c @@ -803 ,8 +803 ,6 @@ max_parallel_hazard_test (...) Assert ( context -> max_hazard != PROPARALLEL_UNSAFE ); context -> max_hazard = proparallel ; /* done if we are not expecting any unsafe functions */ - if ( context -> max_interesting == proparallel ) return true; break ; -- ----------- Query Plans and Performance ------------... ... -> HashJoin ( cost =66839) -> Parallel HashJoin ( cost =41019) -> Parallel Scan part -> Parallel Scan part -> Hash -> Parallel Hash -> HashJoin -> HashJoin ... ... Time: 266 ms Time: 130 ms
formance without breaking functionality. Note that the code patch used by BFA is an overfix as it enforces PARALLEL for all operations. To properly fix this issue, we require developers’ expertise to determine which operations use PARALLEL. When we reported this issue to developers, they replied that “It was a pretty exciting case” and planned to solve it by enabling PARALLEL for these operations shortly. This issue can not be found by previous methods because PARALLEL cannot be enabled for Hash by any version of PostgreSQL (i.e., regression issues), any configuration, or any estimated cardinality. We designed and implemented QueryZen, a prototype of BFA . Technically, to avoid the heavy overhead of recompilation for each code manipulation, we instrument all code manipulations into the code with a single compilation and enable them one by one via external environmental variables. We evaluated QueryZen on PostgreSQL, MySQL, MariaDB, and CockroachDB, which are widely used in previous performance testing works [1, 16, 40]. In total, QueryZen found 21 unique and previously unknown performance issues with the workloads from standard benchmarks (i.e., TPC-H [23] and TPC-DS [33]). These findings resulted in an average performance improvement of 26.2×, and up to 374.9×. QueryZen is necessary to find these issues as most of them cannot be found by APOLLO , CERT , PUPPY , and AMOEBA . Furthermore, these issues already existed before the publication of prior methods. For testing efficiency, we avoid the recompilation time of an average of 5 minutes for each execution. On average of both benchmarks, QueryZen also excludes 75.8% unnecessary code manipulations and increases branch coverage by 8.6%. Overall, we make the following contributions: • Conceptually, we propose a novel methodology BFA to find performance issues by flipping code branches. • Technically, we designed and implemented a novel ar2
APOLLO [16], AMOEBA [21], CERT [1], and PUPPY [40]. APOLLO examines performance regression across different versions of DBMSs; AMOEBA expects semantic-equivalent queries to have similar performance; CERT checks specifically incorrect estimated cardinalities, which refers to the estimated number of rows returned by a query plan; PUPPY compares performance under the default configuration and other configurations. In total, we identified and reproduced 10 confirmed or fixed performance issues. APOLLO has 3 confirmed performance issues. Although it does not publicize the full list of found issues, we found these issues in DBMS bug trackers. However, due to the environment (e.g., database download link) in the issue reports being invalid now, we successfully reproduced 2 issues in our environment. AMOEBA has 6 confirmed issues. Similar to APOLLO , we reproduced 5 issues. CERT has 11 confirmed issues. Unlike other methods, CERT specifically detects inconsistent estimated cardinalities, instead of execution time. In Listing 7 of CERT paper, authors provide two cases to show the execution time gap between pairs of queries due to incorrect cardinalities found, and we evaluated both cases. PUPPY has 54 confirmed issues, but does not publicize the full list of its found issues. By searching the bug trackers of DBMSs, we found a report, which we believe is similar to the issues described in PUPPY paper.
Table 1: Historical performance issues. Method
DBMS
Issue ID
Branch Flipping
APOLLO APOLLO AMOEBA AMOEBA AMOEBA AMOEBA AMOEBA CERT CERT PUPPY
PostgreSQL PostgreSQL CockroachDB CockroachDB CockroachDB CockroachDB CockroachDB CockroachDB CockroachDB MySQL
1 3 51820 57330 57566 58283 58284 88455 89161 115676
✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Sum: 8/10
chitecture QueryZen to avoid recompilation and unnecessary code manipulations for efficient testing. • Practically, we evaluated QueryZen on four widely-used DBMSs, and found 21 previously unknown and unique performance issues.
2
Background
Query optimization. Given a query Q in a declarative language, a DBMS parses and translates it into a concrete execution plan P, also known as a query plan. To optimize P, DBMSs typically use heuristic rules to transform Q to an equivalent Q′ , expecting that the execution cost of P′ , the plan of Q′ , is lower than that of P [5, 11, 13]. For instance, if Q contains a projection operator π that selects specific columns A, pushing π down can eliminate unnecessary columns early: Q : πA (R ▷◁ S) ⇒ Q′ : πA (πA (R) ▷◁ πA (S)), where ▷◁ represents joining tables R and S. However, this rule may introduce potential performance issues. If Q′ includes the operators (e.g., GROUP BY) that require the columns removed by π, recomputation may be necessary incurring degraded performance. Additionally, modern DBMSs employ cost-based optimization, deriving P to n semantically equivalent query plans {P1 , P2 , ..., Pn }, and estimating the execution cost of each Pi by the cost function C(Pi ) [17]. DBMSs choose and execute the optimal plan Po = arg minPi ∈P C(Pi ). Incorrect C() brings potential performance issues.
3
Results. Table 1 shows the 10 performance issues from APOLLO , CERT , PUPPY , and AMOEBA . Overall, we can observe the unexpected performance of 8 issues by the same code manipulation that flips code branches of IF statements once. Specifically, we changed if (condition) to if (!condition). Both performance issues in APOLLO are due to the redundant initialization of hash tables and incorrect parallel computing. Flipping branches disables both, so we can observe the unexpected performance. Both performance issues in CERT affect the estimated cardinalities incurring an inefficient usage of HASH JOIN. Flipping branches changes the strategies of cardinality estimation to find both issues. PUPPY affects optimization strategies by configurations, whose values are checked in IF statements, so flipping branches can achieve the same effect as configurations. For AMOEBA , we cannot use code manipulation to find #57330 and #58284, because both performance issues are due to unimplemented optimizations, such as simplifying JOIN ON TRUE to JOIN. Overall, this simple code manipulation of flipping branches has proven effective in identifying historical performance issues found by various testing methods, demonstrating its potential for finding performance issues through code manipulation.
Motivating Study
To investigate the potential of finding performance issues by code manipulation, we studied whether we can manipulate code to observe the unexpected performance of historical issues found by prior methods.
Example. Listing 2 shows the issue #115676 in MySQL found by PUPPY . Lines 1–9 include the SQL statements to create the database, lines 11–12 show the queries to identify the performance issue, and lines 14–20 show MySQL source code. The issue is due to improper usage of Batched Key
Issue collection. We considered the performance issues found by the four existing performance-testing methods: 3
4 validates whether DBMS performance optimization, step ⃝ and DBMS′ produce semantically consistent results. Finally, 5 evaluates the performance of DBMS and DBMS′ . step ⃝ If DBMS′ demonstrates superior performance, it shows a potential performance issue in the original DBMS.
Listing 2: Historical performance issue 115676 in MySQL. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
SET optimizer_switch = ’ batched_key_access = on ’; CREATE TABLE t0 ( c1 DECIMAL ZEROFILL , c2 FLOAT UNIQUE); INSERT DELAYED IGNORE INTO t0 (c2 , c1 ) VALUES...; CREATE TABLE t2 ( c1 FLOAT) ; INSERT LOW_PRIORITY IGNORE INTO t2 ( c1 ) VALUES("6Y^c !3 Sq ") ...; CREATE TABLE t3 ( c0 TINYINT UNIQUE KEY, c2 INT(252) ); CREATE TABLE IF NOT EXISTS t4 ( c0 DECIMAL, c2 FLOAT, c10 DECIMAL) ; CREATE TABLE t5 LIKE t4 ; INSERT LOW_PRIORITY IGNORE INTO t4 ( c0 ) VALUES(NULL) ...;
4.1
In principle, BFA can test target DBMSs with any workload. In DBMSs, a workload refers to the collection of tasks or operations that DBMSs need to execute over a given period. In this paper, we use workload to represent databases and queries specifically. In Figure 1, we used the TPC-H benchmark [23] to populate the database with multiple tables including PART, SUPPLIER, and PARTSUPP, and the query is provided by the benchmark. To facilitate the presentation, we simplified the query in our example to highlight only the clauses relevant to the performance issue.
SELECT DISTINCT t0 . c1 FROM t4 , t2 , t0 , t3 NATURAL JOIN t5 ; -- 28 ms SELECT /* + BKA () */ DISTINCT t0 . c1 FROM t4 , t2 , t0 , t3 NATURAL JOIN t5 ; -- 22 ms -- ---------------- Code Manipulation -------------------- a/ sql / sql_optimizer . cc +++ b/ sql / sql_optimizer . cc -3511 ,7 +3511 ,7 static bool setup_join_buffering ( JOIN_TAB *tab , JOIN *join, return false; } - if (!( bnl_on || bka_on )) goto no_join_cache ; + if (( bnl_on || bka_on )) goto no_join_cache ;
4.2
2 Execution on Original DBMS (⃝)
We execute the target DBMS with the workload obtained in 1 Before executing the query, the DBMS translates it step ⃝. into a query plan, which is then executed to produce results. During this process, we record both the estimated cost from the query plan and the execution time, which are later uti5 In Figure 1, the query involves joining the lized in step ⃝. tables PART, SUPPLIER, and PARTSUPP. The query plan employs the HASH JOIN operation to perform joining and Seq Scan operation to read data from tables. PostgreSQL, as the target DBMS, leverages an optimization strategy that utilizes parallel CPU cores to enhance performance. Specifically, for this query, PostgreSQL executes the table scan of PART in parallel using the Parallel Seq Scan on part operation.
Access (BKA) optimization, which decides whether to use indexes for joining. The second query in line 12 uses the query hint /*+ BKA()*/ to enforce BKA incurring a shorter execution time than the first query in line 11, which PUPPY finds as a performance issue. In line 19, the IF statement checks the variable bnl_on, which stores the value of the configuration BKA. In default, the value is false, so goto no_join_cache is not executed. We flipped this branch to execute goto no_join_cache, and it is equivalent to enabling BKA. Therefore, manipulating code can observe the same unexpected performance found by changing configurations. 8 of 10 historical performance issues can be found by flipping the code branch of an IF statement.
4.3 4
1 Workload Preparation (⃝)
Approach
3 Execution on Derived DBMS′ (⃝)
In this step, we manipulate DBMS’s source code to derive DBMS′ and execute DBMS′ on the same workload as DBMS. Considering testing efficiency, our code manipulations focus on the query optimization component. DBMSs, such as PostgreSQL, are highly complex systems; for instance, PostgreSQL version 17 consists of over 2 million lines of code. Among these, query optimization is a critical area directly tied to performance improvement. As shown in Figure 1, the query optimization code in PostgreSQL is located within the folder src/backend/optimizer, which we specifically targeted for our manipulations. Note that focusing on a particular component is not compulsory but for efficiency. Followed by our study in Section 3, we propose a simple strategy to manipulate code by flipping the branches of IF statements. For both branches of an IF statement, if DBMS executes one, we request DBMS′ to execute the other one. Suppose DBMS applies an optimization in the executed branch
We propose BFA , a novel approach to identify performance issues by manipulating program code. The core intuition behind BFA is that the default optimization strategy should ideally have optimal performance. If removing or enforcing an optimization brings in a performance improvement, it indicates a potential performance issue in the target DBMS. System overview. Figure 1 provides an overview of BFA , illustrated using the running example in Listing 1. The process 1 where we prepare a database and a query begins in step ⃝, 2 we execute the target to serve as the workload. In step ⃝, 3 we manipulate DBMS on the workload. Next, in step ⃝, the program code to derive an alternative version, referred to as DBMS′ , and execute DBMS′ on the same workload. To ensure that the manipulated code in DBMS′ only affect 4
1 Workload Preparation
Query Plan
2 Execution on Original DBMS
... src/backend/optimizer/util/clauses.c: ->Hash Join (cost=66839) bool max_parallel_hazard_test(...) { ->Parallel Seq Scan on part … ->Hash if (context->max_interesting == proparallel) ->Hash Join ... return true;
SELECT ... FROM PART, SUPPLIER, PARTSUPP … WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey ...;
...
Query Plan’
5 Performance Examination Estimated cost: 66839 Execution time: 266ms
≤
Estimated cost: 41049 Execution time: 130ms
>
...
3 Execution on Derived DBMS' NATION REGION
PART SUPPLIER
src/backend/optimizer/util/clauses.c: bool max_parallel_hazard_test(...) { ... ... if (context->max_interesting == proparallel) return true; ...
->Parallel Hash Join (cost=41019) ->Parallel Seq Scan on part ->Parallel Hash ->Hash Join
4 Functionality Testing
Figure 1: Overview of BFA . of an IF statement, DBMS′ disables this optimization by executing the other branch, and vice versa. We only manipulate an IF statement per execution. We believe this strategy is simple yet efficient for exploring optimization space in code. In Figure 1, DBMS executes the true branch of if (context->max_interesting == proparallel), so we disable the true branch and enforce DBMS′ to execute the false branch.
4.4
operation, so its child operation must be Sort. Such a rule can be used to automatically check whether the query plans produced by DBMS′ are correct.
4.5
5 Performance Examination (⃝)
Lastly, for the DBMS′ that are functionally correct, we compare the performance of DBMS and DBMS′ obtained from 2 and ⃝. 3 In addition to execution time, we also comsteps ⃝ pare the estimated cost for testing efficiency. Measuring execution time may take several minutes per query. For high testing efficiency, before executing a query to measure its ex2 and ⃝, 3 we obtain the query plan and ecution time in steps ⃝ compare the estimated cost, which can be obtained by executing the query with the prefix EXPLAIN in milliseconds. Only when DBMS′ has a smaller estimated cost than DBMS, do we execute the query to evaluate the execution time. Measuring estimated cost significantly improves the testing efficiency but may miss the cases that DBMS′ reports a higher estimated cost but a shorter execution time compared to DBMS. Therefore, we leave it optional for BFA . When finding a potential performance issue, our method outputs the workload, a code patch equivalent to the code manipulation, both query plans, and execution time. In Figure 1, DBMS′ has a significantly smaller estimated cost and execution time than DBMS, so it indicates a potential performance issue, which was quickly confirmed by PostgreSQL developers.
4 Functionality Testing (⃝)
To make sure the flipped branches only enable or disable optimization strategies with no side effects, we check whether DBMS′ is still functionally correct. We propose two solutions for DBMS users and developers, respectively. As DBMS users lack expert knowledge of the DBMS′ s internal code, we adopted a differential testing approach to evaluate functionality in practice. Differential testing involves comparing the results of executing the same inputs across different systems; inconsistencies in the results may indicate potential bugs. Specifically, we validate whether DBMS and DBMS′ consistently produce identical results. If they do, we assume that DBMS′ is functionally correct. The trustworthiness of this method increases with the number of test cases evaluated. In this paper, we collected 10 thousand randomly generated small workloads from SQLancer [27, 28], which is a popular DBMS testing tool. As shown in our evaluation (Section 6), 96% of the DBMS′ s that pass the above validation do not affect program functionalities. For DBMS developers, we can use specification-based validation approaches. Since developers have expert knowledge of the code, we expect developers to provide a specification of the query plans, which can be used for checking validity. For example, in PostgreSQL’s query plans, the operation GroupAggregate aggregates pre-sorted rows for a GROUP BY
5
Implementation
We implemented BFA in the QueryZen prototype. We discuss the key implementation details in this section. 5
5.1
1 Workload Selection for ⃝
manipulating them is inefficient. It is especially wasteful for large-scale DBMS source code, which includes thousands IF statements to flip. To further enhance testing efficiency, we record the IF statements covered during execution and manipulate only those that are executed. To achieve this, we add a logging function to the instrumented IF statement as follows: if((condition)^ (log(ID)&& get_env("DROP")== ID)). The function log(ID) logs the ID of the statement to a log file, allowing us to track which IF statements were executed. This information helps us selectively decide which IF statements to manipulate. Importantly, the log(ID) function always returns true, ensuring it does not alter the semantics of the instrumented IF statement.
We used the standard benchmarks TPC-H [23] and TPCDS [33] as workloads. While exploring diverse workloads can reveal more performance issues, it is more practical to focus on commonly used workloads. Unlike crashes, where any issue in any workload can have significant consequences, performance issues are more impactful when they affect widely used workloads. Developers generally consider an optimization beneficial if it improves performance on commonly used workloads, even if it causes regressions on rarely used ones [38]. For this reason, we adopted standard benchmarks TPC-H and TPC-DS. TPC-H is a decision support benchmark simulating a wholesale supplier’s sales scenario, featuring 22 queries, while TPC-DS is a more advanced benchmark representing a large-scale retail business scenario, comprising 99 queries. Both benchmarks are widely used for performance evaluation [4, 12], and we believe their workloads are representative of commonly used scenarios.
5.2
Algorithm 1 BFA implementation in QueryZen Input: program: DBMS, program: DBMS′ , query: Q 1: cost = explain(DBMS, Q) 2: time, i f _list = execute(DBMS, Q) 3: for ID in i f _list do 4: SET DROP = ID 5: cost ′ = explain(DBMS′ , Q) 6: if cost > cost ′ then 7: time = execute(DBMS′ , Q) 8: compare(time,time′ ) 9: end if 10: end for
3 Code Manipulation for ⃝
To efficiently manipulate code, which is the critical step for BFA , we did two optimizations to eliminate recompilation and unnecessary code manipulations. Eliminating recompilation. For each code manipulation, the recompilation of the target DBMS often demands significant computational resources, especially for complex DBMSs. This poses a substantial limitation to the efficiency of BFA . To address this challenge, we instrumented code manipulation operations into the program upfront and enabled each one dynamically using external environment variables. Specifically, for an IF statement if (condition), we modified it to if (condition ^ get_env("DROP")== ID). Here, the function get_env("DROP") retrieves the value of the environment variable DROP, and ID is a unique identifier for each IF statement. The “^” symbol denotes the bitwise XOR operator, which evaluates to true when both operands are not equal. If the value of DROP matches ID, the IF condition is negated, causing the other branch to execute. For other values of DROP, the IF statement remains unaffected. Consequently, we can control the behavior of each IF statement by setting the environment variable "DROP" appropriately, eliminating the need for recompilation during testing. Although it introduces additional performance overhead, the overhead is tiny. For example, our instrumentation introduces less than 0.1 second additional execution time for PostgreSQL when running the TPC-H benchmark. More importantly, the performance overhead is added to both DBMS and DBMS′ , so it does not affect performance comparison.
Algorithm 1 shows the implementation algorithm of BFA in QueryZen that eliminates recompilation and unnecessary code manipulations. First, we obtain the estimated cost of Q on DBMS in line 1, and execute Q to obtain execution time and the list of executed IF statements in line 2. Then, we enumerate the list and enable each corresponding code manipulation by setting the environmental variable DROP in lines 3 and 4. After that, if the estimated cost of executing Q on DBMS′ is smaller than that on DBMS, we execute Q on DBMS′ to check whether Q has a shorter execution time. If so, a potential performance issue is found. This algorithm ensures that we only manipulate executed code without recompilation, significantly improving testing efficiency.
6
Evaluation
To evaluate the effectiveness of QueryZen in finding performance issues, we sought to answer the following questions: Q.1 Effectiveness. Can QueryZen find previously unknown performance issues? Q.2 Necessity. Whether QueryZen is necessary to find these performance issues?
Eliminating unnecessary code manipulation. When executing a query, many IF statements are not executed, and
Q.3 Sensitivity. To what extent QueryZen’s components contribute to the finding of these performance issues? 6
Tested DBMSs. We tested PostgreSQL, MySQL, CockroachDB, and MariaDB. PostgreSQL is one of the most established DBMSs. MySQL is the most popular opensource DBMS [37]. MariaDB is another popular DBMS that was forked from MySQL. CockroachDB is a widely used enterprise-class DBMS, and its open version on GitHub has been starred more than 30.1k times. These DBMSs have been extensively tested by previous works about finding performance issues [1, 16, 20, 21, 40]. Recall that QueryZen is not specific to any programming language, so we can test both C (PostgreSQL, MySQL, MaraiDB) and Go (CockroachDB) programming languages. We used the latest available development versions of tested DBMSs (PostgreSQL: 17.0, MySQL: 9.0, MariaDB: 11.7, CockroachDB: dcb0d27). We manipulated the code in specific folders (PostgreSQL: /app/postgres/src/backend/optimizer, MySQL: /app/mysqlserver/sql, CockroachDB: /app/cockroach/pkg/sql/opt, MariaDB: /app/mariadb-server/sql). The folders of PostgreSQL and CockroachDB include only the code of query optimizer, and the folders of MySQL and MariaDB include all SQLrelated code.
Table 2: Previously unknown and unique performance issues found by QueryZen. DBMS
Query
Issue ID
Perf. ↑
Status
PostgreSQL PostgreSQL PostgreSQL PostgreSQL PostgreSQL MySQL MySQL MySQL MySQL MySQL MySQL MySQL MySQL MySQL CockroachDB CockroachDB CockroachDB MariaDB MariaDB MariaDB MariaDB
TPC-H 2 TPC-H 2 TPC-H 10 TPC-DS 4 TPC-DS 60 TPC-H 8 TPC-H 11 TPC-H 16 TPC-H 16 TPC-DS 7 TPC-DS 15 TPC-DS 16 TPC-DS 19 TPC-DS 47 TPC-H 2 TPC-H 7 TPC-DS 63 TPC-H 2 TPC-H 4 TPC-H 7 TPC-H 18
1 2 3 4 5 116456 116484 116309 116534 116773 116774 116775 116776 116777 134803 135001 136350 35280 35332 35331 35333
2.0× 3.1× 1.5× 374.9× 1.5× 1.1× 3.1× 3.5× 2.1× 12.1× 2.4× 1.7× 1.9× 1.6× 24.0× 2.9× 2.5× 102.2× 2.6× 1.6× 1.9×
Confirmed Confirmed Confirmed Confirmed Fixed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Confirmed Analyzing Confirmed Analyzing
Num 2 6 8 11 8 9 9 8 7 12 24 22 9 9 6 5 12 23 18 11 4
Sum: 21
Benchmark preprocess. We adapted TPC-H and TPC-DS benchmarks for the four tested DBMSs and removed incompatible queries. TPC-DS benchmark includes 99 test cases. Each test case includes one or multiple queries, and we only use the first query in the evaluation for simplicity. Additionally, due to incompatible SQL grammar, we removed 3, 7, 14, and 12 cases in the TPC-DS benchmark for PostgreSQL, MySQL, CockroachDB, and MariaDB, respectively. TPC-H includes 22 test cases, which can be directly executed in the four tested DBMSs. We excluded case 15 because it includes creating views, while QueryZen aims to evaluate queries. We used 1 GB of data for both benchmarks, considering it to be a balanced trade-off between execution time and testing efficiency. This choice aligns with prior work [12], which employed the same data size.
10 minutes to validate and deduplicate performance issues before reporting to developers. Finding overview. Table 2 lists the performance issues QueryZen found. Perf. ↑ represents the performance gap due to the code manipulation. Developers typically respond directly regarding the status of the issue: Confirmed denotes that the issue has been identified as a performance issue; Fixed denotes that a fix has been proposed or pushed; Analyzing denotes that developers are still investigating and have not yet replied to us. Num indicates the number of other TPC-H and TPC-DS queries affected by the same issue, with at least a 10% performance gap. In total, QueryZen found 21 unique and previously unknown performance issues. Within them, 19 issues had been confirmed, and one issue had been fixed by developers. These confirmed issues support our assumption that flipping branches effectively disables or enables optimization strategies to disclose performance issues. The remaining two issues in MariaDB were still under analysis, which may be due to the difficulty of analyzing performance issues. To avoid burdening developers’ workload, we did not continue to report more issues in MariaDB. QueryZen found only three issues in CockroachDB, and a possible reason is that many optimizations are implemented in its domain-specific language, which QueryZen does not instrument. On average, these performance issues show a 26.2× performance gap compared to executing original queries. Especially, issue #4 in PostgreSQL shows a 374.9× performance gap. Furthermore, these performance issues have a broader impact as these issues affect 10 other queries on average.
Experimental infrastructure. We conducted all experiments on an AMD Ryzen Threadripper 3990X processor that has 64 physical and 128 logical cores clocked at 2.2 GHz. Our test machine uses Ubuntu 24.04 with 256 GB of RAM. We limit our maximum utilization to 20 physical cores to avoid resource competition affecting performance measurement.
Q.1 Effectiveness We ran QueryZen to find performance issues. For each potential performance issue, to exclude false alarms due to environmental factors, we recompiled the DBMS with the manipulated code in a separate environment and ran the same query ten times to check whether we could observe the same performance gap. After that, we reported our workload and code patch to the developers. On average, each query requires 7
is challenging because we lack the information on the joint column distribution.
Listing 3: Issue 4 of cost model in PostgreSQL. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
SELECT ... FROM CUSTOMER , WEB_SALES , DATE_DIM ... WHERE ... t_w_secyear . sale_type = ’w ’ AND t_s_firstyear . dyear = 2001 AND t_s_secyear . dyear = 2001+1 AND t_c_firstyear . dyear = 2001 AND t_c_secyear . dyear = 2001+1 AND t_w_firstyear . dyear = 2001...; -- ---------------- Code Manipulation -------------------- a/ src / backend / optimizer / util / pathnode .c +++ b/ src / backend / optimizer / util / pathnode .c @@ -555 ,16 +555 ,6 @@ add_path (...) case COSTS_BETTER2 : if ( keyscmp != PATHKEYS_BETTER1 ) + if ( keyscmp == PATHKEYS_BETTER1 ) { ... accept_new = false; /* old dominates new */ } break ; -- ----------- Query Plans and Performance ------------... ... -> NestLoop ( cost =794037) -> HashJoin ( cost =794032) ... ... Time: 50 m :24 s :486 ms Time: 8s :179 ms
False positive analysis. Two types of false positives exist in the performance issues found by QueryZen. We consider a reported issue a true positive only if it is confirmed by the developers; otherwise, it is treated as a false positive. One source of false positives in QueryZen is misalignment with developers’ intended optimization trade-offs. A false positive QueryZen found in PostgreSQL involves the inline optimization for Common Table Expressions (CTEs), which determines whether to execute the CTE and the outer query together or separately. Although QueryZen observed a 64× performance improvement when forcing inline optimization, PostgreSQL developers did not consider it a true issue. They explained that enabling inline improves performance for some workloads but degrades it for others. Since there is currently no algorithm to distinguish between these cases, the optimization remains off by default. This type of trade-off-based false positive occurred only twice, suggesting that it is rare for QueryZen. Another potential source of false positives comes from performance fluctuation. We observed that DBMS performance can fluctuate due to environmental factors such as system load or cache state. In our initial experiments, such fluctuation incurred around 30% false performance issues. To address this, we strictly controlled system usage and only reported issues when the observed performance gap exceeded 10%. In this way, we did not observe any false positives caused by performance fluctuation, as confirmed by DBMS developers.
Root-reason analysis. We analyzed the root causes of these performance issues. We focused on PostgreSQL, whose developers were the most responsive in acknowledging and analyzing our reported issues. For the other three systems, detailed analysis was not available, likely due to their internal development policies or cycles. Within five confirmed issues in PostgreSQL, three are due to an underperforming executor, and the others are due to an inaccurate cost model. The executor-related issues stem from suboptimal implementation of certain execution strategies, while the cost model issues arise from inaccurate cost estimations that lead the optimizer to select suboptimal query plans. Listing 1 presents an example of an executor-related issue, where the HashJoin operator is expected to run in parallel but fails to do so. We show another example of a cost model issue as follows.
QueryZen found 21 previously unknown performance issues on widely used benchmarks TPC-H and TPC-DS.
Q.2 Necessity
Example: an issue of cost model. Listing 3 shows a performance issue QueryZen found in PostgreSQL with query 4 in the TPC-DS benchmark. This issue exposes the most significant performance gap (51 minutes to 8 seconds). Similarly, we show relevant clauses of the query in lines 1–7 and relevant code in lines 9-18. This query includes multi-table joining, so multiple possible query plans are available for various joining order and algorithms. In line 13, the IF statement checks whether to exclude additional query plans for evaluation. For this query, the true branch in lines 15–18 is executed, and the chosen query plan uses NestLoop to join tables. We manipulate the code to disable the true branch, and more potential query plans are evaluated. Therefore, a much more efficient query plan that uses HashJoin to join tables is evaluated and chosen. Developers quickly confirmed this issue after we reported. After analysis, developers believe the root cause is the accuracy of selectivity estimation for the multiple predicates in the WHERE clause. Fixing this performance issue
We evaluated whether QueryZen is necessary to find these performance issues. Specifically, we examined 1) whether the performance issues found by QueryZen can be potentially found by previous performance-testing methods, and 2) whether these performance issues existed when prior methods were proposed. We considered the same performancetesting methods in Section 3: APOLLO [16], AMOEBA [21], CERT [1], and PUPPY [40]. Issue reproducing. We evaluated whether the performance issues found by QueryZen can be potentially found by prior performance-testing methods APOLLO , AMOEBA , CERT , and PUPPY . Since these methods target different aspects of performance, such as cardinality estimation or configuration sensitivity, and QueryZen focuses on source code manipulation, the sets of bugs they uncover are not directly comparable. 8
commits.
Table 3: The reproducibility of previous methods on the issues found by QueryZen. DBMS
Issue ID
PostgreSQL 1 PostgreSQL 2 PostgreSQL 3 PostgreSQL 4 PostgreSQL 5 MySQL 116456 MySQL 116484 MySQL 116309 MySQL 116534 MySQL 116773 MySQL 116774 MySQL 116775 MySQL 116776 MySQL 116777 CockorachDB 134803 CockorachDB 135001 CockorachDB 136350 MariaDB 35280 MariaDB 35332 MariaDB 35331 MariaDB 35333 Sum (✓):
APOLLO
AMOEBA
CERT
Issue existence date. Additionally, we evaluated whether these performance issues existed when prior performancetesting methods APOLLO , AMOEBA , CERT , and PUPPY were proposed. If so, it demonstrates that these performance issues were not found by prior methods with an extensive testing effort. We retrieved the last commit date of the code manipulated QueryZen, and we assumed the performance issues existed no later than the date.
PUPPY
✓ ✓
✓ ✓
Results. Figure 2 shows the last commit date associated with the 21 performance issues found by QueryZen and the publication date of the prior methods. Issues #116484, #116309, #116534, and #116773 share the same commit, as do issues #116776 and #116777. It is because both commits focus on tidying up code style, resulting in extensive modifications, including the manipulation of code across multiple issues. In total, 14, 19, 21, and 21 of 21 issues existed before the publication date of APOLLO , AMOEBA , CERT , and PUPPY . A large portion of performance issues already existed for a long time, while prior methods have not found them in practice.
✓ ✓
✓ ✓ ✓ ✓
✓
✓ 8
✓ 0
4
1
To address this, we conducted a qualitative analysis, a common approach in prior work [2, 27, 28]. Specifically, given a performance issue and both execution paths of a DBMS before and after QueryZen’s code manipulation, 1) if the first lines of discrepancy execution paths are lastly updated in different commits, we assumed the issue can be found by APOLLO ; 2) if the instrumented IF condition is directly controlled by a SQL clause, we assumed there exists a pair of semantic-equivalent queries to execute both paths so that the issue can be found by AMOEBA ; 3) if the first branches in discrepancy execution paths directly manipulate estimated cardinalities, we assumed the issue can be found by CERT ; 4) if the instrumented IF condition is directly controlled by an external configuration, we assumed the issue can be found by PUPPY . These assumptions do not guarantee that prior methods can find these issues. For example, although the first lines of both execution paths are updated in different commits, it is not necessary that both versions of the DBMSs show the performance gap.
Most issues found by QueryZen cannot be found by APOLLO , AMOEBA , CERT , and PUPPY , and existed before the publication of prior methods.
Q.3 Sensitivity The soundness of code manipulations. QueryZen manipulates all IF branches and ensures that the flipped branches impact only performance, not functionalities, through differential testing. To evaluate its soundness, we manually examined the branches that passed the functionality testing. Since the total number of branches (e.g., in MySQL and MariaDB) is too large for exhaustive analysis, we randomly sampled 100 flipped branches and reviewed their associated code and comments, which typically describe the purpose of each branch or function. Our manual inspection showed that none of the flipped branches affect the correctness of the results. For PostgreSQL, MySQL, CockroachDB, and MariaDB, 100%, 92%, 100%, and 93% of the sampled branches, respectively, control the optimization decisions, where modifications are expected to affect performance only. The remaining branches involved non-query operations such as table locking, file writing, and authentication. These operations do not influence query results and thus cannot be filtered out by differential testing. Because we instrumented all SQL-related code in MySQL and MariaDB, some non-optimization branches were included. However, as these non-query operations typically do not impact query results, they do not compromise the soundness of our code manipulation for finding performance issues.
Results. Table 3 shows whether prior methods can find the performance issues found by QueryZen. Overall, 8, 0, 4, and 1 issues have the potential to be found by APOLLO , AMOEBA , CERT , and PUPPY , respectively. It demonstrates the necessity of QueryZen for finding these performance issues, and black-box methods are insufficient to explore optimization strategies for finding performance issues. The number of APOLLO is much higher than others, because developers typically submit a minimal code block in each commit, making it more likely to observe the code update in different 9
Figure 2: The last commit associated with the issues found by QueryZen and the publication date of prior methods. 50.00 45.00
Table 4: Average number of executed and all instrumented code manipulations by QueryZen. DBMS
TPC-H
TPC-DS
All
1,202 (30.65%) 2,285 (18.42%) 2,384 (32.09%) 3,833 (12.60%)
1,362 (34.73%) 2,404 (19.38%) 2,448 (32.95%) 3,947 (12.97%)
3,922 12,407 7,429 30,428
PostgreSQL
40.00 35.00 30.00
Baseline QueryZen
25.00 20.00 15.00 10.00
PostgreSQL MySQL CockroachDB MariaDB Avg:
23.44%
5.00 0.00
1
2
3
4
5
6
7
8
9
10 11 12 13 14 16 17 18 19 20 21 22 Sum
MySQL
15.00 10.00
Baseline
5.00 0.00
1
25.01%
2
3
4
5
6
7
8
9
10 11 12 13 14 16 17 18 19 20 21 22 Sum
15.00
MariaDB
10.00
Baseline
5.00
QueryZen
0.00
The efficiency contribution of eliminating unnecessary code manipulations. For each query, QueryZen records the executed IF statements and only manipulates them, reducing the workload associated with unnecessary code manipulations. We evaluated the extent to which unnecessary code manipulations can be excluded. Table 4 presents the average number of executed and all instrumented code manipulations by QueryZen. On average, QueryZen executes only 23.44% and 25.01% of all instrumented code manipulations in the TPC-H and TPC-DS benchmarks, respectively. It significantly improves the testing efficiency by excluding the overhead of executing the remaining 76.56% and 74.99% unnecessary code manipulations.
QueryZen
1
2
3
4
6
7
8
9
10
11
12
13
14
16
17
18
19
20
21
22 Sum
Figure 3: Branch coverage for TPC-H benchmark before and after manipulating code.
Branch coverage increased by QueryZen. We evaluated the extent of optimization space explored. Specifically, we compared the branch coverage before and after code manipulation. A larger branch coverage gap indicates that QueryZen ’s manipulations enable the exploration of more optimization opportunities. Our analysis focused on the query optimizer, which is the component we instrumented. We identified 35,522 branches in PostgreSQL, 531,247 branches in MySQL, and 564,585 branches in MariaDB query optimizers. The higher branch counts for MySQL and MariaDB are due to QueryZen instrumenting all SQL-related code. We did not evaluate CockroachDB, which does not support branch coverage measurement for end-to-end tests [36].
The efficiency contribution of eliminating recompilation. QueryZen eliminates the need for recompilation by instrumenting the code to read environmental variables that control execution flow. To evaluate the impact of this instrumentation on testing efficiency, we specifically assessed whether the time required for recompilation represents a significant bottleneck. Our experiments show that even with full utilization of 112 CPU cores, compiling PostgreSQL, MySQL, CockroachDB, and MariaDB once takes 49.0, 655.0, 309.2, and 202.0 seconds, respectively. Given that our testing campaign involves a large number of executions, the overhead of recompiling for each execution becomes impractical. By leveraging QueryZen ’s design, we eliminate recompilation overhead, thereby significantly enhancing testing efficiency.
Figure 3 shows the branch coverage for the TPC-H benchmark before and after manipulating code. In the x-axis, each number represents the query ID, and right-column Sum represents the accumulated coverage of all queries. The number in the y-axis represents the percentage of branch coverage. QueryZen cumulatively covers 16.7%, 3.6%, and 5.5% more branches than executing original queries in PostgreSQL, MySQL, and MariaDB. On average of all queries, QueryZen covers 3,339, 11,049, and 21,115 additional branches in the 10
optimizers of PostgreSQL, MySQL, and MariaDB, accounting for 9.40%, 2.08%, and 3.74% of all measured branches. The percentage is lower in MySQL and MariaDB because their total number of measured branches is significantly larger than PostgreSQL’s. We observed a similar result for the TPCDS benchmark. QueryZen cumulatively covers 19.1%, 4.6%, and 8.1% more branches than executing original queries in PostgreSQL, MySQL, and MariaDB. Notably, QueryZen covers 59.9% of the PostgreSQL optimizer’s branches, demonstrating that a substantial portion of optimization strategies has been tested. On average of all queries, QueryZen covers 3,311, 8,818, and 22,922 additional branches in PostgreSQL, MySQL, and MariaDB, accounting for 9.32%, 1.66%, and 4.06% of all measured branches. Note that, during the branch coverage evaluation of MySQL, unexpected behaviors caused by manipulated code occasionally blocked data collection, resulting in incomplete results. The results show QueryZen substantially explores optimization space. Using more diverse workloads and code manipulations may further expand coverage.
7
300 lines of Shell script. Our implementation is not DBMSspecific, so we do not require additional implementation effort to test other DBMSs. From an integration perspective, BFA can be paired with any workload. We chose TPC-H and TPC-DS benchmarks because they are representative of performance workloads. From an applicability perspective, BFA can test any open-sourced DBMSs as BFA is based on source code. Considering these features of BFA , we argue that BFA can be widely adopted. Fixing performance issues. BFA highlights the relevant code segment for the performance issue, facilitating analysis but fixing them still requires non-trivial effort. One reason is that fixing performance issues requires comprehensive consideration, which usually consumes much time. The code manipulated by BFA shows performance issues based on a single workload without considering all workloads, so more effort is required to propose a fixing patch that does not lower the performance for other workloads. Another reason is that performance issues might have lower priority than other issues, such as correctness and crash bugs.
Discussion Limitations. While BFA demonstrates effectiveness in identifying previously unknown performance issues, it has several limitations. First, although we apply functionality checks using differential testing and validate against thousands of test cases, semantic equivalence is inferred rather than guaranteed. Subtle functional divergences may go undetected without formal specifications, which would require manual effort from developers. Second, the effectiveness of BFA depends on the quality of the underlying workload. While we use TPC-H and TPC-DS to represent common scenarios, other types of workloads might trigger different optimizations. Finally, our evaluation was conducted under a fixed system configuration. Although we observed consistent performance gaps across repeated trials, different hardware or memory settings may affect reproducibility.
Methodology. We propose a white-box testing methodology for identifying performance issues by directly manipulating program code. Unlike existing approaches [1, 21, 40], which focus on manipulating external inputs, BFA operates on the program code itself. This approach offers several advantages. First, it enables systematic and fine-grained testing of program logic, overcoming the limitations of input-based methods that can only trigger behaviors in a constrained manner. As a result, our methodology uncovered 21 unique and previously unknown performance issues. Second, the code manipulated by BFA facilitates issue debugging and analysis. Prior works compare the performance of executing input pairs, but it often remains unclear which part of the code incurs the performance difference. In contrast, our approach directly highlights the relevant code segments responsible for performance variations, streamlining the analysis process. Finally, BFA does not require domain-specific knowledge. Traditional white-box testing methods, such as unit testing, often rely on domain expertise to establish appropriate testing entry points. In contrast, BFA flips general code branches, making it readily applicable to target systems without requiring specialized knowledge.
8
Related Work
Testing performance issues. The most related strand of research is on finding performance issues in DBMSs automatically. Ba et al. proposed CERT [1] to find performance issues by testing specifically cardinality estimation. Jung et al. proposed APOLLO [16], which compares the execution times of a query on two versions of a database system to find performance regression bugs. Liu et al. proposed AMOEBA [21], which compares the execution time of a semantically-equivalent pair of queries to identify an unexpected slowdown. Wu et al. proposed PUPPY [40], which examines whether performance improves after adjusting default configurations. In addition, several fuzzing-based approaches explore performance issues in general software.
Path to adoption. We believe that a simple testing approach has the potential for wide adoption. From a conceptual perspective, BFA is a general method to manipulate program code to find performance issues, which is easy to understand. From an implementation perspective, BFA is easy to implement as we implemented the code manipulation script in around 200 lines of Python code and testing logic in around 11
SlowFuzz [26] searches for slow executions by maximizing total execution path length, while PerfFuzz [18] uses multidimensional feedback to independently maximize execution counts of individual program locations, helping reveal code regions prone to excessive execution time. Unlike the above methods that examine optimization logic by executing different workloads and configurations, BFA enforces the optimization logic by code manipulation for the same workload and configurations, achieving a fine-grained testing of optimization logic at the source code level.
and TLP [28] detect logic bugs in SELECT statement implementations, while DQE [31] targets logic errors in UPDATE and INSERT statements. Similar to testing approaches that find bugs, our core contribution is a novel testing methodology that finds performance issues by manipulating program code.
9
Conclusion
In this paper, we have proposed Branch Flip Analysis (BFA ), a general white-box method to find performance issues by flipping branches of IF statements in source code. As the default optimization strategies should ideally have optimal performance, we disable or enforce an optimization by flipping code branches to derive a reference DBMS, whose performance is expected to be no better than the original DBMS. BFA provides a new angle to find performance issues and can test target DBMSs in a systematic and fine-grained manner, facilitating issue analysis. Technically, we improved the testing efficiency by eliminating recompilation and unnecessary code manipulations. We implemented BFA into the prototype QueryZen and demonstrated its effectiveness by evaluating it on four mature and widely used DBMSs, PostgreSQL, MySQL, CockroachDB, and MariaDB. QueryZen found 21 unique and previously unknown performance issues with standard benchmarks TPC-H and TPC-DS. These findings resulted in an average performance improvement of 26.2×, and up to 374.9×. We believe such a straightforward methodology helps improve DMBSs’ performance in practice and opens a promising research avenue to analyze program behaviors by manipulating source code, instead of manipulating external inputs so that target systems can be systematically examined.
Performance optimization. Performance optimization is a longstanding research problem in the system and database research community. Ruan et al. [30] uses disaggregated memory to improve DBMS performance. Bindschaedler et al. [3] disaggregated computation and storage in DBMSs to improve performance. Tsalapatis et al. proposed Memsnap [32] to optimize the DBMS storage layer. Liu et al. [22] proposed Juno to optimize the search algorithm in vector DBMSs. Apart from optimization, several benchmarks are used to compare performance across DBMSs. The TPC-H [23] and TPC-DS [33] benchmarks are widely recognized as industry standards. Leis et al. introduced the Join Order Benchmark (JOB) [17], which incorporates complex join orders. Instead of proposing new algorithms and system designs to improve the performance, BFA aims to expose the unexpected suboptimal performance due to implementation issues. Configuration-based performance optimization. Systems typically expose configuration parameters that control execution, and searching for optimal configurations is a common approach to improving performance. Violet [10] applies selective symbolic execution to detect specious configurations that unexpectedly degrade performance. LearnConf [19] investigates configuration-related performance bugs and develops a static analysis tool that identifies which configurations influence specific performance dimensions and in what ways. Jin et al. [15] conduct empirical studies and propose detection frameworks for performance bugs in real systems. These works primarily focus on misconfigurations, whereas BFA targets a broader class of performance issues in a more systematic way.
References [1] Jinsheng Ba and Manuel Rigger. Cert: Finding performance issues in database systems through the lens of cardinality estimation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024. [2] Jinsheng Ba and Manuel Rigger. Keep it simple: Testing databases via differential query plans. Proceedings of the ACM on Management of Data, 2(3):1–26, 2024.
DBMS testing. Beyond performance testing, automated methods have been developed to uncover various DBMS bugs. Many fuzzing tools target security-critical issues, such as memory errors. Tools like SQLSmith [35], Griffin [8], DynSQL [14], and ADUSA [21] employ grammar-based techniques to generate test cases aimed at detecting memoryrelated vulnerabilities. Inspired by grey-box fuzzers like AFL [34], Squirrel [41] uses code coverage feedback to identify memory issues. Other approaches focus on discovering logic bugs in DBMSs. Oracles such as PQS [29], NoREC [27],
[3] Laurent Bindschaedler, Ashvin Goel, and Willy Zwaenepoel. Hailstorm: Disaggregated compute and storage for distributed lsm-based databases. In ASPLOS ’20: Architectural Support for Programming Languages and Operating Systems, Lausanne, Switzerland, March 16-20, 2020, pages 301–316, 2020. [4] Peter A. Boncz, Thomas Neumann, and Orri Erling. TPC-H analyzed: Hidden messages and lessons learned 12
from an influential benchmark. In Raghunath Nambiar and Meikel Poess, editors, Performance Characterization and Benchmarking - 5th TPC Technology Conference, TPCTC 2013, Trento, Italy, August 26, 2013, Revised Selected Papers, volume 8391 of Lecture Notes in Computer Science, pages 61–76. Springer, 2013.
[15] Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. Understanding and detecting realworld performance bugs. In Jan Vitek, Haibo Lin, and Frank Tip, editors, ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, Beijing, China - June 11 - 16, 2012, pages 77–88. ACM, 2012.
[5] Surajit Chaudhuri. An overview of query optimization in relational systems. In Alberto O. Mendelzon and Jan Paredaens, editors, Proceedings of the Seventeenth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, June 1-3, 1998, Seattle, Washington, USA, pages 34–43. ACM Press, 1998.
[16] Jinho Jung, Hong Hu, Joy Arulraj, Taesoo Kim, and Woon-Hak Kang. APOLLO: automatic detection and diagnosis of performance regressions in database systems. Proc. VLDB Endow., 13(1):57–70, 2019. [17] Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter A. Boncz, Alfons Kemper, and Thomas Neumann. How good are query optimizers, really? Proc. VLDB Endow., 9(3):204–215, 2015.
[6] Pit Fender and Guido Moerkotte. Counter strike: Generic top-down join enumeration for hypergraphs. Proc. VLDB Endow., 6(14):1822–1833, 2013. [7] Pit Fender, Guido Moerkotte, Thomas Neumann, and Viktor Leis. Effective and robust pruning for top-down join enumeration algorithms. In Anastasios Kementsietsidis and Marcos Antonio Vaz Salles, editors, IEEE 28th International Conference on Data Engineering (ICDE 2012), Washington, DC, USA (Arlington, Virginia), 1-5 April, 2012, pages 414–425. IEEE Computer Society, 2012.
[18] Caroline Lemieux, Rohan Padhye, Koushik Sen, and Dawn Song. Perffuzz: Automatically generating pathological inputs. In Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, pages 254–265, 2018. [19] Chi Li, Shu Wang, Henry Hoffmann, and Shan Lu. Statically inferring performance properties of software configurations. In Proceedings of the Fifteenth European Conference on Computer Systems, pages 1–16, 2020.
[8] Jingzhou Fu, Jie Liang, Zhiyong Wu, Mingzhe Wang, and Yu Jiang. Griffin : Grammar-free DBMS fuzzing. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, pages 49:1–49:12. ACM, 2022.
[20] Jie Liang, Zhiyong Wu, Jingzhou Fu, Mingzhe Wang, Chengnian Sun, and Yu Jiang. Mozi: Discovering dbms bugs via configuration-based equivalent transformation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–12, 2024.
[9] Jana Giceva, Gustavo Alonso, Timothy Roscoe, and Tim Harris. Deployment of query plans on multicores. Proc. VLDB Endow., 8(3):233–244, 2014.
[21] Xinyu Liu, Qi Zhou, Joy Arulraj, and Alessandro Orso. Automatic detection of performance bugs in database systems using equivalent queries. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pages 225–236. ACM, 2022.
[10] Yigong Hu, Gongqi Huang, and Peng Huang. Automated reasoning and detection of specious configuration in large systems with symbolic execution. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 719–734, 2020.
[22] Zihan Liu, Wentao Ni, Jingwen Leng, Yu Feng, Cong Guo, Quan Chen, Chao Li, Minyi Guo, and Yuhao Zhu. JUNO: optimizing high-dimensional approximate nearest neighbour search with sparsity-aware algorithm and ray-tracing core mapping. In Rajiv Gupta, Nael B. AbuGhazaleh, Madan Musuvathi, and Dan Tsafrir, editors, Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- 1 May 2024, pages 549– 565. ACM, 2024.
[11] Yannis E Ioannidis. Query optimization. ACM Computing Surveys (CSUR), 28(1):121–123, 1996. [12] Oleg Ivanov and Sergey Bartunov. Adaptive cardinality estimation. arXiv preprint arXiv:1711.08330, 2017. [13] Matthias Jarke and Jurgen Koch. Query optimization in database systems. ACM Computing surveys (CsUR), 16(2):111–152, 1984. [14] Zu-Ming Jiang, Jia-Ju Bai, and Zhendong Su. Dynsql: Stateful fuzzing for database management systems with complex and valid sql query generation. In 32st USENIX Security Symposium (USENIX Security 23), 2023.
[23] Raghunath Othayoth Nambiar, Meikel Poess, Andrew Masland, H. Reza Taheri, Matthew Emmerton, Forrest Carman, and Michael Majdalany. TPC benchmark 13
roadmap 2012. In Raghunath Othayoth Nambiar and Meikel Poess, editors, Selected Topics in Performance Evaluation and Benchmarking - 4th TPC Technology Conference, TPCTC 2012, Istanbul, Turkey, August 27, 2012, Revised Selected Papers, volume 7755 of Lecture Notes in Computer Science, pages 1–20. Springer, 2012.
LOS 2023, Vancouver, BC, Canada, March 25-29, 2023, pages 498–512. ACM, 2023. [31] Jiansen Song, Wensheng Dou, Ziyu Cui, Qianwang Dai, Wei Wang, Jun Wei, Hua Zhong, and Tao Huang. Testing database systems via differential query execution. In Proceedings of IEEE/ACM International Conference on Software Engineering (ICSE), 2023.
[24] Thomas Neumann. Query simplification: graceful degradation for join-order optimization. In Ugur Çetintemel, Stanley B. Zdonik, Donald Kossmann, and Nesime Tatbul, editors, Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2009, Providence, Rhode Island, USA, June 29 - July 2, 2009, pages 403–414. ACM, 2009.
[32] Emil Tsalapatis, Ryan Hancock, Rakeeb Hossain, and Ali José Mashtizadeh. Memsnap µcheckpoints: A data single level store for fearless persistence. In Rajiv Gupta, Nael B. Abu-Ghazaleh, Madan Musuvathi, and Dan Tsafrir, editors, Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- 1 May 2024, pages 622–638. ACM, 2024.
[25] Johns Paul, Jiong He, and Bingsheng He. GPL: A gpubased pipelined query processing engine. In Fatma Özcan, Georgia Koutrika, and Sam Madden, editors, Proceedings of the 2016 International Conference on Management oData, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, pages 1935–1950. ACM, 2016.
[33] Website. Tpc-ds benchmark. https://www.tpc.org/ tpcds/, 1988. Accessed: 2022-11-15. American fuzzy lop (afl) fuzzer. [34] Website. http://lcamtuf.coredump.cx/afl/technical_ details.txt, 2013. Accessed: 2022-11-15.
[26] Theofilos Petsios, Jason Zhao, Angelos D Keromytis, and Suman Jana. Slowfuzz: Automated domainindependent detection of algorithmic complexity vulnerabilities. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 2155–2168, 2017.
[35] Website. Sqlsmith. https://github.com/anse1/ sqlsmith, 2015. Accessed: 2022-11-15. [36] Website. Cockroachdb code coverage measurement. https://cockroachlabs.atlassian.net/wiki/ spaces/CRDB/pages/73171260/Code+coverage, 2022. Accessed: 2025-05-15.
[27] Manuel Rigger and Zhendong Su. Detecting optimization bugs in database engines via non-optimizing reference engine construction. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann, editors, ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pages 1140–1152. ACM, 2020.
Database systems ranking. https: [37] Website. //db-engines.com/en/ranking, 2022. Accessed: 2025-05-15. Performance on widely used work[38] Website. loads. https://www.postgresql.org/ message-id/CAMkU%3D1xoU06eW4CrEZyDDn% 2BfnJaCe3b04rE3mdVu4Gsxmj9KFA%40mail.gmail. com, 2025. Accessed: 2025-11-15.
[28] Manuel Rigger and Zhendong Su. Finding bugs in database systems via query partitioning. Proc. ACM Program. Lang., 4(OOPSLA):211:1–211:30, 2020.
[39] Lisa Wu, Andrea Lottarini, Timothy K. Paine, Martha A. Kim, and Kenneth A. Ross. Q100: the architecture and design of a database processing unit. In Rajeev Balasubramonian, Al Davis, and Sarita V. Adve, editors, Architectural Support for Programming Languages and Operating Systems, ASPLOS 2014, Salt Lake City, UT, USA, March 1-5, 2014, pages 255–268. ACM, 2014.
[29] Manuel Rigger and Zhendong Su. Testing database engines via pivoted query synthesis. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4-6, 2020, pages 667–682. USENIX Association, 2020. [30] Chaoyi Ruan, Yingqiang Zhang, Chao Bi, Xiaosong Ma, Hao Chen, Feifei Li, Xinjun Yang, Cheng Li, Ashraf Aboulnaga, and Yinlong Xu. Persistent memory disaggregation for cloud-native relational databases. In Tor M. Aamodt, Natalie D. Enright Jerger, and Michael M. Swift, editors, Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASP-
[40] Zhiyong Wu, Jie Liang, Jingzhou Fu, Mingzhe Wang, and Yu Jiang. Puppy: Finding performance degradation bugs in dbmss via limited-optimization plan construction. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 560–571. IEEE Computer Society, 2024. 14
[41] Rui Zhong, Yongheng Chen, Hong Hu, Hangfan Zhang, Wenke Lee, and Dinghao Wu. SQUIRREL: testing database management systems with language validity and coverage feedback. In Jay Ligatti, Xinming Ou, Jonathan Katz, and Giovanni Vigna, editors, CCS ’20: 2020 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, USA, November 9-13, 2020, pages 955–970. ACM, 2020.
15