arXiv:2605.15232v1 [cs.SE] 13 May 2026
Method-level Change-proneness: A Better Metric for Black-box Test Suite Minimization Md. Siam
Kazi Sakib
Institute of Information Technology University of Dhaka Dhaka, Bangladesh [email protected]
Institute of Information Technology University of Dhaka Dhaka, Bangladesh [email protected]
Abstract—Test Suite Minimization (TSM) reduces the size of test suites while preserving their fault detection capability. In black-box TSM, reduction is performed without analyzing production code. While several black-box TSM approaches have explored metrics like test logs or test similarity, those often suffer from scalability and efficiency issues. On the other hand, change-proneness (CP), recently emerged as an efficient and scalable alternative metric, has only been applied at class level. To accurately identify fault-revealing test cases, we propose CP at finer-grained method-level and implement Method-level Changeproneness based Test-suite Minimization (MCTM). MCTM first calculates CP for each method from version control metadata, then determines the dependency between test cases and methods by analyzing the test-code call-graph. Next, it scores the association between test cases and their invoked methods using statistical measures such as Average, Geometric Mean etc. Finally, test cases with the highest scores are selected to form the reduced suite. Evaluation on 15 open-source Java projects with 635 buggy versions shows MCTM achieves 0.93 accuracy and 0.94 fault detection rate on average, significantly outperforming class-level CP and similarity-based approaches while maintaining superior efficiency. Index Terms—Test suite minimization, Test suite reduction, Change-proneness, Black-box, Method-level granularity
I. I NTRODUCTION Test Suite Minimization (TSM) aims to reduce the number of test cases while preserving the test suite’s ability to detect faults [1], [2]. When TSM is performed without utilizing production code, it is referred to as black-box [3], [4]. Various metrics such as test logs and test code similarity have been explored to estimate a test case’s fault detection potential, but these metrics often suffer from poor scalability, inefficiency in terms of execution time, or high resource demands [5]. Recently, change-proneness (CP) has been explored as a more scalable and efficient metric alternative. [6]. However, the approach leveraging CP operates on class-level [6], which is coarse-grained and may overlook important details such as the specific impact of changes within individual methods which is essential for effective TSM [7], [8]. Prior studies in software maintenance and defect prediction demonstrate the benefits of analyzing software artifacts at finer granularity. Metrics computed at the method level capture localized modifications and context-specific characteristics that are strongly associated with defects such as complexity, coupling, and churn [9]. Compared to class or file-level metrics,
method-level features have been shown to achieve higher accuracy in identifying fault-prone components and localizing buggy regions [7], [10]. Such fine-grained analysis also enables robust performance on large systems and enhances test case prioritization in evolving codebases [11], [12]. These findings suggest that extending CP analysis to the method level could provide more precise guidance for selecting test cases in black-box TSM. Achieving this requires quantifying CP at the method level and scoring test cases based on their association with change-prone methods. To be viable for black-box TSM, the approach must be effective in fault detection while remaining efficient and lightweight. Several approaches have been proposed for TSM in blackbox. Philip et al. [3] introduced a black-box technique that leverages test code, commit complexity, historical test logs and version control (VC) metadata to train classification models for reducing test suites. Cruciani et al. [13] proposed a clusteringbased method using test code diversity as the core metric. Building on this, Pan et al. developed AST-based (ATM) [4] and Large Language Model (LLM) - based (LTM) [5] approaches, both relying on test case similarity and evolutionary search. Most recently, Siam et al. [6] introduced CP as a lightweight alternative metric for black-box TSM. Their approach leverages VC metadata to calculate CP values at the class level. These values are then used to score and prioritize test cases. While this work demonstrated the effectiveness of CP-based test selection, it operated exclusively at class-level granularity, which may miss fine-grained change patterns that occur at the method level within classes. To the best of our knowledge, no prior work has investigated the application of CP at method-level granularity for black-box TSM. In this study, we investigate method-level CP as a metric for black-box TSM and integrate it into an approach called MCTM (Method-level Change-proneness based Test-suite Minimization). MCTM begins by extracting change metrics for each method from the project’s Git history. For each buggy version, method-level changes are analyzed from project inception to the current version. CP is then calculated using two metrics: Change Frequency (ChgFrq), which measures how often a method changes, and Change Extent (ChgExt), which captures the magnitude of those changes [14]. Once high-CP methods are identified, MCTM determines the association of the test-
cases with the change-prone methods. To do this, it builds a call graph of the test code and analyzes it to map test cases to the methods they invoke. Each test case is scored based on its association with change-prone methods using various statistical measures, including Average, Geometric Mean etc. Finally, test cases with the highest association scores are selected to form the reduced test suite. We evaluate MCTM on 15 Java projects comprising 635 buggy versions, testing 8 configurations across three minimization budgets (retaining 25%, 50%, and 75% of the original test suite). Our evaluation addresses the following research questions: • RQ1: How effective is method-level CP in guiding black-box TSM? At a 50% minimization budget, MCTM demonstrated strong effectiveness, achieving an average fault detection rate of 0.94 and accuracy of 0.93, with the best results obtained using the Change Frequency metric and Average aggregation. It also maintained high efficiency, requiring only 0.98 minutes per version across all configurations. Results were consistent for other minimization budgets (25% and 75%). • RQ2: How does method-level CP perform compared to class-level in terms of effectiveness and efficiency? The best configuration of MCTM significantly outperformed the best configuration of the class-level CP-based approach, CTM, in fault detection capability. It achieved +0.23 accuracy gain and +0.19 FDR gain across all projects and budgets. While introducing finer granularity, MCTM preserved practical efficiency, running with only a modest overhead (mean runtime of 0.98 minutes vs 0.44 minutes per version). • RQ3: How does MCTM perform against similarity-based approaches (ATM and LTM)? Compared to similaritybased methods, MCTM achieved significant gains in both effectiveness and efficiency. It improved accuracy by 0.26 over ATM and 0.22 over LTM, with an FDR improvement of 0.10 over LTM, while reducing execution time by 2.65x. These findings demonstrate that method-level CP enables more effective black-box TSM than existing class-level CP and similarity-based approaches while being practically efficient. II. R ELATED W ORK Test Suite Minimization (TSM) focuses on improving the efficiency of software testing by removing redundant test cases from a test suite while preserving its fault detection capability [1], [2], [15]. By reducing the size of test suites, TSM helps lower testing time and resource consumption without compromising the quality of testing outcomes. Extensive research has been conducted on TSM, leading to a variety of techniques in the literature [1], [2], [13]. Broadly, these techniques can be categorized into two categories: whitebox and black-box approaches. White-box TSM relies on production code, code coverage data, or requirement models to guide test case selection [2]. In contrast, black-box TSM avoids dependence on production code and instead leverages
information available from test code, version control metadata, or commit history to identify valuable test cases [3], [13]. Existing test suite minimization (TSM) techniques predominantly employ white-box strategies. These include (a) greedy heuristics, which iteratively select test cases based on their coverage of code elements like statements or branches [16]–[18], (b) search-based techniques, which use genetic algorithms or other evolutionary strategies to find optimal subsets of test cases that balance coverage and size [19], [20], (c) clustering-based methods, which group test cases by the similarity of their coverage profiles and select representatives from each cluster [21], [22], (d) model-based approaches, which analyze system models such as UML diagrams or state machines to evaluate test case relevance [23], and (e) requirement-driven strategies [24], [25], which prioritize test cases based on their traceability to functional or non-functional requirements. Although these methods have shown effectiveness in controlled experiments, their adoption in practice faces significant barriers. In industrial settings, production codebases are often very large and complex, making the analysis and instrumentation required for white-box TSM impractical at scale [13]. Additionally, the process of collecting and analyzing code coverage data can introduce considerable overhead, with studies reporting up to 30% additional testing time [4], [13]. Such constraints limit the scalability and practical applicability of white-box TSM in fast-paced development environments. To address these challenges, researchers have explored black-box TSM, which eliminates the dependency on production code by utilizing only test code or external metadata. Black-box approaches have gained attention for their ease of integration and lower computational overhead [4]–[6], [13]. Philip et al. [3] proposed the first black-box TSM approach, FastLane, using commit risk, version control metadata and test logs in a logistic regression model. They leverage the regression model to determine whether to run or skip the execution of a test case. FastLane achieved an 18.04% reduction in test suite size while retaining 99.99% of faults. However, it depends on historical test data, which is not always available [6]. Cruciani et al. [13] introduced FAST-R, a black-box approach that relies solely on the source code of test cases. FAST-R transforms test code into vectors using a term frequency model [26] and applies random projection [27] to reduce their dimensionality. Clustering algorithms are then used on these vectors, with cluster centroids selected to form the minimized test suite. Their results showed that FASTR was significantly more efficient than white-box techniques in terms of minimization time. However, although FAST-R performed well for C projects, it showed noticeably lower fault detection capability in Java. This makes it less suitable for such contexts. To address this limitation, Pan et al. [4] proposed ATM, a TSM approach based on syntactic similarity. They removed irrelevant code elements, i.e. comments and test oracles, transformed the remaining code into Abstract Syntax Trees (ASTs).
Then they computed pairwise similarity of ASTs to guide test selection using evolutionary search techniques (GA and NSGA-II). While effective in fault detection, ATM struggled with scalability. To overcome this, LTM [5] was introduced, which captures semantic similarity using large language models such as UniXcoder and CodeBERT. It generates vector embeddings for test cases and computes similarity using cosine similarity and Euclidean distance. Although LTM improved scalability, it requires GPU support for efficient execution. Siam et al. [6] addressed this issue by introducing change proneness (CP) as a metric for TSM. Their black-box approach leverages version control metadata to identify change-prone classes and measures each test case’s association with these classes. The technique demonstrated significant efficiency gains over ATM while maintaining comparable fault detection capability. However, analyzing change proneness at the class level remains coarse-grained and may miss critical details such as the distribution and intensity of changes within classes, which are important for effective test suite reduction [7], [8]. Prior work highlights the advantages of analyzing software artifacts at finer granularities, with method-level change metrics often outperforming coarser file or class-level metrics in fault prediction and localization [28]. Method-level metrics capture localized modifications and contextual information, enabling more accurate bug predictions and effective fault localization [10], [29]. Their effectiveness largely stems from the ability to represent factors such as complexity, coupling, and churn, which are strongly linked to defects [9]. Luo et al. [11] demonstrated that models leveraging over 40 methodlevel metrics perform robustly even on large systems. More recently, combining fault-proneness estimations with methodlevel change data has been shown to enhance test case prioritization [12], [30]. Change metrics, when calculated at methodlevel, have been proven effective for bug identification [7], [28], [31]. These suggest that CP at method-level granularity can serve as a better metric for black-box TSM. However, to the best of our knowledge, its impact has not yet been explored. III. M ETHODOLOGY In this study, we evaluate the efficacy of CP as a metric for TSM by enhancing its granularity to method-level. Our goal is to investigate whether finer-grained change insights lead to more precise and effective minimization outcomes. Figure 1 illustrates the methodology of our approach, MCTM. MCTM begins by mining method-level change histories from Git and computes CP scores based on the frequency and extent of changes. Next, a call graph is generated to uncover method-level dependencies for each test case. These dependencies are then used to score and rank test cases based on their association with change-prone methods. Finally, test suites are minimized under different budget levels to assess the impact of this finer-grained analysis. A. Method Level Change Information Extraction Changes in code are strongly associated with fault introduction, as frequent modifications often indicate instability and
increased complexity in the affected components [32], [33]. Identifying such frequently changed regions can help focus testing efforts on fault-prone areas and improve the effectiveness of test suite minimization. To capture this phenomenon more precisely, we analyze change histories at the method level, moving beyond coarser class or file-level granularity to gain finer insights into localized changes. Changes in code are often linked to fault introduction, making frequently modified methods strong indicators of faultprone areas [32], [33]. To capture this, we extract three method-level change metrics: (a) the number of commits in which a method was modified, reflecting how often it has changed; (b) the total number of commits since the method was added, which provides context about its lifetime and exposure to change; and (c) the number of lines added, modified, and deleted, indicating the scale of modifications. These metrics, adapted from prior work [6], have been widely applied in fault prediction studies [34], [35] and in analyzing class-level change trends [14]. By applying them at the method level, we aim to achieve finer-grained insights into change-prone regions. To capture a holistic view of historical changes, we compute these metrics from the project’s inception to the current commit. At the end of this step, we obtain a list of all modified methods along with their corresponding change metrics, forming a detailed historical profile of changes across the commit range. B. Change Metrics Aggregation Due to Java’s support for polymorphism, we treat methods with the same name but different parameters as distinct functions. Each method is uniquely identified by its name, parameter types, and parameter identifiers. As a result, certain refactoring actions such as parameter renaming, are interpreted as the creation of a new method rather than a modification of an existing one. For instance, in TABLE I, the method ExtendedBufferedReader::read appears twice due to the renaming of the parameter from buf to buffer. Although semantically identical, they are treated as separate methods with distinct histories. To handle such cases, we aggregate the duplicate method entries in the following manner: • We normalize method signatures by removing parameter identifiers to match methods that differ only in variable identifiers (e.g., buf vs. buffer). • We add the change metrics i.e. number of changes, insertions, deletions, and modifications to capture the complete change history across all versions, resulting in the aggregated entry shown in the grey row of TABLE I. • We take the maximum total commit count among the duplicates to preserve the full commit span since the method’s creation (TABLE I, Aggregated). C. Change-proneness Calculation Change proneness (CP) refers to the likelihood that a software entity will undergo modifications in the future [36]. Since past changes often indicate areas likely to change again,
Figure 1: Approach Overview TABLE I: Commit Statistics for ExtendedBufferedReader::read() Method
Change Commits
Total Commits
Insertions
Deletions
Remarks
4 2 6
573 21 573
91 32 123
49 33 82
Original Renamed Aggregated
ExtendedBufferedReader::read(char [ ] buf , int off , int len) ExtendedBufferedReader::read(char [ ] buffer , int off , int len) ExtendedBufferedReader::read(char [], int, int)
prior change history serves as a strong predictor of future evolution [32], [37], [38]. While Siam et al. [6] applied CP at the class level, we advance this idea to the method level to capture localized change patterns with greater precision. This shift allows us to identify fault-prone regions that class-level analysis might miss. We define change frequency (ChgFreq) as the proportion of commits where a method was modified relative to the total number of commits since its introduction (Eq. 1). This metric reflects how often a method experiences modifications over its lifetime, highlighting areas of code that exhibit recurrent instability. ChgFreqM =
Number of ChangesM Total CommitsM
(1)
Here, M denotes the method under consideration. To capture the extent of changes, we first compute code churn, which is the total number of lines added, deleted, and modified (Eq. 2). We then define change extent (ChgExt) as the ratio of code churn to total commits, as shown in Eq. 3. This formulation reflects how substantial the modifications are in each commit on average, distinguishing methods that undergo minor edits from those experiencing large, disruptive changes. CodeChurnM = AddM + DelM + ModM ChgExtM =
CodeChurnM Total CommitsM
(2) (3)
D. Call-graph Generation A call graph is a directed graph that represents method invocations within a program. Each node corresponds to a method, and each edge indicates a call from one method to another [39]. It highlights the dependencies between methods and helps in understanding how changes in one part of the code may affect others. For our analysis, we generate call graphs from the test code (orange-marked portion of Figure 2) in each project to capture
Figure 2: Test case - Method Dependency Mapping Example
method-level dependencies. The test code is first compiled into JAR files, which serve as input for constructing the call graphs. We opt for static call graphs because they are efficient and suitable for a black-box setup. Unlike dynamic call graphs, which require executing the program and monitoring runtime behavior, static call graphs can be derived directly from compiled bytecode. This avoids the need for analyzing source code or system internals, aligning with our goal of minimizing test suites in a production-agnostic manner. To ensure compatibility with our representation of changed methods, we further normalize the raw call graphs. This involves (1) replacing calls to <init> with their respective constructor names and (2) reformatting inner class method invocations by substituting ’$’ with ’::’. These steps produce fully processed call graphs, ready for method-level dependency analysis across all subject projects.
E. Test case - Method Dependency Mapping In this step, we analyze the processed call graph to identify the method dependencies of each test case. Starting from each test method as the root node, we apply a depth-first search (DFS) to collect all directly and transitively invoked methods. This captures both immediate and hierarchical dependencies. For example, as shown in Figure 2, method LangUtilsTest::testLang865() demonstrates multiple levels of method invocation in a hierarchical manner. The test method directly invokes LocaleUtilsTest::assertValidToLocale() and LocaleUtils::toLocale(). Subsequently, methods Locale::getVariant(), LangParser::isEmpty() and Locale::getLanguage() are invoked through LocaleUtilsTest::assertValidToLocale(). Therefore we get a dependency list as {LocaleUtilsTest::assertValidToLocale(), LocaleUtils::toLocale(), Locale::getVariant(), Locale::isEmpty(), Locale::getLanguage()}. We also include the test method itself in the dependency list. This accounts for recently added tests that may still detect faults even if their dependent methods remain unchanged [40]. As a result, we obtain a multi-level hierarchical mapping of each test case to its associated method dependencies. Thus the final dependency list we get becomes {LocaleUtilsTest::testLang865(), LocaleUtilsTest::assertValidToLocale(), LocaleUtils::toLocale(), Locale::getVariant(), Locale::isEmpty(), Locale::getLanguage()}. Since we only packaged the test code into JARs, our call graphs capture only the method invocations originating from test regions (highlighted in orange and grey in Figure 2). However, subsequent hierarchical calls into production code (highlighted in red in Figure2) are not captured, as these would require source code access. F. Test case Scoring and Ranking A test case can depend on multiple methods, each with its own change-proneness (CP) value. To score test cases based on their association with change-prone methods, we extend the statistical measures introduced by [6] from class-level to method-level granularity. Originally, eight statistical measures: minimum, maximum, sum, mean, geometric mean, harmonic mean, standard deviation, and median—were proposed to summarize CP distributions. However, measures such as minimum and maximum are highly sensitive to outliers, which can distort test case prioritization by giving disproportionate weight to extreme CP values [41], [42]. Similarly, standard deviation reflects the spread of CP values but can amplify the influence of outliers and is less informative for skewed distributions [42]. Moreover, using the sum as an aggregation metric can bias the score in favor of test cases with a large number of dependencies, potentially overshadowing test cases with a small number of dependencies in highly fault-prone areas. To address these concerns and achieve robust scoring, we considered only four statistical measures: arithmetic mean (Avg), geometric mean (GMean), harmonic mean (HMean), and median. These metrics balance sensitivity to variation with
robustness to outliers and skewness, offering a more stable representation of method-level CP distributions for guiding test case selection. Finally, we select test cases with the highest association scores under each measure. To reflect realistic developer expectations, we apply test suite minimization under three recommended budgets: 25%, 50%, and 75% [4]–[6]. IV. E XPERIMENTAL S ETUP AND R ESULT A NALYSIS In this section, we evaluate MCTM through a series of experiments. We begin by outlining the research questions that guide our study, followed by a description of the experimental setup, including the subject systems and evaluation metrics. We then present and analyze the results obtained from applying MCTM to the test suites. A. Research Questions RQ1: How effective is MCTM to detect faults under different configurations? MCTM utilizes 8 configurations, combining two CP measures (ChgFrq and ChgExt) and four statistical measures (Subsection III-F) used to score a test case’s association with change-prone methods. This question explores how different configurations influence minimization effectiveness under three budget levels: 25%, 50%, and 75%. RQ2: How does method-level change-proneness perform compared to class-level in terms of effectiveness and efficiency? In this question, we compare MCTM (method-level) with CTM (class-level) to assess whether the finer-grained methodlevel analysis in MCTM leads to improved fault detection while remaining efficient enough to be practical for black-box. RQ3: How does MCTM compare with state-of-the-art TSM techniques that use other metrics? This question investigates whether method-level changeproneness can serve as a strong alternative to similaritybased methods. We evaluate MCTM against two state-of-theart black-box approaches: ATM [4], which reduces test suite based on syntactic similarity, and LTM [5], which leverages large language models to detect and remove redundant tests. TABLE II: Subject Projects Statistics Project
Cli Codec Compress Csv Gson JacksonCore JacksonDatabind JacksonXml Jsoup JxPath Lang Math Mockito Time
Project # of Tests Avg # of Avg Size versions Size test cases Commits (KLoC) (faults) (KLoC) per version per version 2 9 45 2 9 31 74 6 14 20 30 71 21 30
39 18 47 16 18 26 112 6 93 22 64 106 38 26
4 15 29 7 20 45 72 10 13 6 61 73 36 56
256 413 404 193 984 356 1,814 152 494 250 1,796 2,078 1,182 3,918
486 928 1,393 828 1,226 902 3,009 660 785 374 2,398 2,932 1,880 1,600
B. Experimental Setup We evaluated the performance of MCTM under the eight configurations outlined in RQ1. All experiments were conducted on a machine with an Intel Core i5 9400F processor (6 cores, 2.9 GHz), 16 GB RAM, running Ubuntu 20.04. Since LTM [5] relies on GPU-based inference, we equipped the machine with an additional GTX 1050 Ti GPU to support its evaluation. This represents an important distinction in computational requirements, as MCTM, CTM, and ATM do not require GPU support. Both MCTM and CTM produce deterministic results. In contrast, ATM and LTM involve stochastic processes due to their use of genetic algorithms (GA) and NSGA-II, resulting in non-deterministic outputs. To account for this variability, each approach was executed 10 times, and the reported results represent the average over these runs. All experiments were performed on the same machine to ensure consistency. 1) Configurations: A configuration refers to a specific combination of metrics and scoring methods used within an approach to prioritize and select test cases. Each technique we evaluated supports multiple such configurations, reflecting alternative ways of leveraging its underlying mechanisms. For MCTM, we explored eight configurations derived from two change metrics combined with four statistical aggregation measures (2x4=8). CTM, which works at the class level, includes 16 configurations based on two CP measures and eight statistical strategies (2x8=16). ATM offers six configurations, with four using similarity measurements paired with a genetic algorithm (GA) and two others employing NSGAII. Likewise, LTM provides 10 configurations by combining five large language models with two similarity measurements. To ensure a fair evaluation, we identified the best-performing configuration from each approach. This enables us to compare their performance effectively and gain insights into how different design choices influence fault detection capability. 2) Minimization Budgets: The minimization budget defines the proportion of test cases to retain after reduction. It represents the target size of the minimized test suite relative to the original suite. Similar to [4], [5], we evaluated our approach under three budgets: 25%, 50%, and 75%. These levels reflect commonly adopted practices in industry, where varying degrees of test suite reduction are applied to balance between fault detection capability and testing cost [4]. C. Dataset We evaluated our approach using projects from Defects4J1 (v2.0.1), a widely adopted benchmark dataset in software testing research [43]. Defects4J provides real and reproducible faults from open-source Java projects, along with the necessary test suites and fault-fixing commits. It is well-suited for test suite minimization (TSM) experiments, where linking test failures to system faults is essential [4], [5]. Our dataset initially included all 16 projects from Defects4J v2.0.1, totaling 661 buggy versions. However, we excluded
the Chart project due to unresolvable Git commit hashes that prevented successful metric extraction. This refinement resulted in 15 projects comprising 635 buggy versions for our experiments. Each buggy version contains a single real fault that triggers failures in one or more test cases. Table II summarizes the key characteristics of the projects. The system sizes range from 2 KLoC to 74 KLoC, while the corresponding test suite sizes span 4 KLoC to 73 KLoC. The number of buggy versions per project varies between 4 and 174, with test suites containing an average of 152 to 3,919 test cases per version. These figures were determined by analyzing the latest version of each project using the CLOC tool.2 . By utilizing this diverse set of projects, we ensure a comprehensive evaluation across small to large systems, enabling robust assessment of both effectiveness and scalability. D. Implementation Details Similar to prior studies [5], [6], we implemented our approach in Python to ensure portability and ease of integration with existing analysis tools. Several components of our method build upon established libraries and utilities. To extract method-level change metrics, we used PyDriller3 , a lightweight Python framework designed for mining software repositories [44]. PyDriller provides a convenient API to access commit histories, analyze code churn, and identify finegrained modifications, making it well-suited for processing version control metadata. Although our approach is language-agnostic by design, this study focuses on Java projects. For call graph generation, we employed the java-callgraph4 tool, a lightweight utility capable of producing both static and dynamic call graphs from Java bytecode [45]. Given the constraints of our black-box setup where production code and runtime execution are unavailable, we opted for static call graphs. This enabled us to efficiently capture method-level dependencies within test code without requiring program execution. E. Evaluation Metrics To evaluate the performance of MCTM, we considered three metrics: Accuracy, Fault Detection Rate (FDR), and Execution Time. Accuracy. The primary goal of test suite minimization is to reduce the number of test cases while preserving fault detection capability. Accuracy measures how well this capability is maintained after minimization. Formally, as defined in Equation (4), accuracy is calculated as: Accuracy =
|F ′ | |F |
where |F ′ | denotes the number of fault-revealing test cases retained in the minimized test suite, and |F | is the total number of fault-revealing test cases in the original suite. 2 https://github.com/AlDanial/cloc 3 https://github.com/ishepard/pydriller/
1 https://github.com/rjust/defects4j
(4)
4 https://github.com/gousiosg/java-callgraph
Fault Detection Rate (FDR). While accuracy reflects the proportion of fault-revealing tests retained, FDR provides a more relaxed perspective by focusing on whether any fault is detected after minimization. Specifically, FDR measures the percentage of buggy program versions for which the minimized test suite contains at least one fault-revealing test case [4], [5]. Equation (5) defines FDR as: Pn FDR =
i=1 fi
n
(5)
where n is the total number of buggy versions, and fi = 1 if the minimized suite for version i includes at least one faultrevealing test case, and fi = 0 otherwise. For example, in the Codec project, which has 18 faulty versions, if the minimized test suites detect faults in 15 versions but fail in 3, the FDR is computed as 15/18, yielding an FDR of 0.83. Execution Time. To assess the efficiency of MCTM, we measure the total execution time, which includes the time required for change metric extraction, call-graph generation, association analysis and scoring. This time is then compared with the total time required for CTM, ATM and LTM. Fisher’s exact test We used Fisher’s exact test [46] to assess whether the differences in fault detection proportions between MCTM, CTM, ATM, and LTM are statistically significant. As a non-parametric test, it does not assume any underlying distribution and works well with small data samples, making it suitable for our data. A p-value below 0.05 indicates a significant difference in performance between the configurations. Odds ratio We computed the odds ratio [47] to quantify the strength of differences in fault detection between configurations. The odds ratio indicates how much more likely one configuration is to outperform another in terms of detected faults. An odds ratio greater than 1 suggests higher odds of better performance, while a value below 1 indicates the opposite. This measure complements Fisher’s exact test by providing an effect size for the observed differences. F. Results We applied our methodology using three minimization budgets: 25%, 50%, and 75%. Consistent with prior studies [4]– [6], we center our analysis on the 50% budget, as results across the other budgets (25% and 75%) exhibited similar trends and led to the same conclusions. The data, codes and results of our experiments are available in our replication package [48]. 1) RQ1 Results: Table III, IV, and V present the projectwise accuracy, fault detection rate (FDR), and execution time, respectively, of the eight MCTM configurations under a 50% minimization budget. Bold values highlight the bestperforming configurations for each statistic. All configurations share the same underlying method-level change metric extraction, call graph generation, and dependency analysis. Since the time differences among configurations arise only from scoring, which is computationally inexpensive, we report execution time of MCTM as a single column. On average,
MCTM completed processing in 58.40 seconds per version, with a median of 38.16 seconds. Among the eight configurations, the combination of methodlevel ChgFrq with Average aggregation consistently achieved the strongest results. It reached a mean accuracy of 0.93 and FDR of 0.94, ranking highest across most subject projects. This suggests that aggregating change frequencies evenly across all associated methods effectively captures their collective evolution patterns, enabling better prioritization of faultrevealing test cases. In contrast, using Median aggregation produced the weakest results (mean accuracy: 0.82, mean FDR: 0.84). This likely stems from the Median masking extreme values in the associated methods, which are critical for identifying highly changeprone components. By scoring test cases based on a middle value, it underrepresents cases where a few highly volatile methods dominate a test’s behavior, leading to suboptimal prioritization. Configurations employing ChgExt exhibited slightly lower performance compared to ChgFrq. While ChgExt/Average was the strongest within this group, it still was outperformed by ChgFrq/Average. This may be attributed to ChgExt’s broader numerical range, which amplifies variance during aggregation. Interestingly, aggregation methods like Geometric Mean, which dampen outliers, showed more stable results across both metrics, highlighting their potential in situations where extreme values are common. Fisher’s exact test further supports these findings. The best configuration (ChgFrq/Average) significantly outperformed all other configurations except ChgExt/Average at the α < 0.05 level. While ChgFrq/Average achieved higher accuracy and FDR in seven projects (JxPath, Codec, JacksonXml, JacksonDatabind, Mockito, Cli, Csv), the observed differences against ChgExt/Average were not statistically significant. As observed in Table III and IV, the lowest accuracy and FDR for the best MCTM configuration (0.77) occurred in the JXPath project. A manual inspection revealed that this drop was due to limitations of static analysis in our blackbox setup. Specifically, inherited methods in test code were not fully captured because static call graphs were generated only from compiled test code. Inheritance plays a key role in JXPath, leading to incomplete dependency mapping and reduced selection effectiveness. Despite this, the overall consistently high accuracy and FDR across all projects affirm the robustness of method-level change metrics for black-box test suite minimization. 2) RQ2 Results: We compare the effectiveness and efficiency of method-level (MCTM) and class-level (CTM) change-proneness. As shown in TABLE III and IV, MCTM consistently outperforms CTM across all statistical measures. On average, MCTM achieved 0.93 accuracy and 0.94 FDR, while CTM reached only 0.70 and 0.75, respectively. Fisher’s exact test further confirms the significance of this improvement (p = 0.004, Odd’s Ratio = 23.4), indicating that MCTM is over 23 times more likely to achieve superior fault detection. The
TABLE III: Results and descriptive statistics of Accuracy of MCTM, CTM, ATM and LTM across projects for the 50% minimization budget. The highest Accuracy results are highlighted in bold.
Avg 1.00 0.77 0.84 1.00 0.97 1.00 0.86 0.96 0.91 0.94 0.93 1.00 0.99 0.94
MCTM Change Frequency Change Extent HMean GMean Median Avg HMean GMean Median 0.90 1.00 0.84 1.00 0.73 1.00 0.75 0.77 0.77 0.73 0.75 0.73 0.77 0.68 0.81 0.84 0.83 0.84 0.76 0.83 0.77 0.98 0.96 0.97 1.00 0.96 0.95 0.98 0.86 0.96 0.82 0.96 0.72 0.94 0.80 0.83 1.00 0.89 1.00 0.81 1.00 0.83 0.78 0.84 0.76 0.87 0.71 0.83 0.68 0.81 0.96 0.73 0.96 0.79 0.97 0.71 0.76 0.89 0.68 0.94 0.73 0.91 0.65 0.92 0.93 0.93 0.92 0.85 0.93 0.89 0.84 0.87 0.85 0.89 0.73 0.81 0.75 0.85 0.92 0.85 0.94 0.71 0.88 0.58 0.76 0.95 0.72 0.99 0.75 0.87 0.67 0.91 0.90 0.81 0.94 0.78 0.92 0.84
0.77 0.91 0.93 0.95 0.99 1.00
0.76 0.79 0.85 0.84 0.90 0.98
Technique Project Time JxPath Codec JacksonXml JacksonDatabind Gson Math Jsoup Compress Mockito Cli Csv JacksonCore Lang Statistics Min 25% Quartile Mean Median 75% Quartile Max
0.77 0.87 0.92 0.93 0.96 0.96
0.68 0.77 0.82 0.83 0.85 0.97
0.75 0.89 0.93 0.94 0.98 1.00
0.69 0.73 0.77 0.74 0.79 0.96
0.77 0.85 0.90 0.92 0.96 0.95
0.56 0.68 0.76 0.74 0.83 0.98
CTM ATM LTM Change Extent Tree Edit Distance Cosine Similarity GMean Genetic Algorithm UnixCoder 0.89 0.68 0.69 0.55 0.67 0.75 0.84 0.65 0.65 0.89 0.37 0.59 0.68 0.63 0.77 0.73 0.66 0.68 0.55 0.74 0.69 0.53 0.61 0.71 0.79 0.84 0.76 0.77 0.55 0.71 0.80 0.74 0.73 0.72 0.91 0.83 0.86 0.53 0.53 0.72 0.72 0.70 0.43 0.58 0.70 0.73 0.80 0.89
0.37 0.62 0.67 0.66 0.74 0.91
0.53 0.65 0.71 0.70 0.74 0.83
TABLE IV: Results and descriptive statistics of FDR of MCTM, CTM, ATM and LTM across projects for the 50% minimization budget. The highest FDR results are highlighted in bold.
Avg 1.00 0.77 0.88 1.00 0.97 1.00 0.88 0.99 0.91 0.97 0.95 1.00 0.96 0.93
MCTM Change Frequency Change Extent HMean GMean Median Avg HMean GMean Median 0.96 1.00 0.84 1.00 0.76 1.00 0.80 0.77 0.77 0.73 0.77 0.73 0.77 0.68 0.81 0.88 0.88 0.88 0.81 0.88 0.81 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.88 0.96 0.84 0.97 0.75 0.94 0.82 0.83 1.00 0.89 1.00 0.83 1.00 0.83 0.80 0.86 0.80 0.89 0.75 0.88 0.72 0.86 0.98 0.77 0.98 0.83 0.99 0.78 0.77 0.89 0.70 0.94 0.74 0.91 0.66 0.97 0.97 0.97 0.97 0.95 0.97 0.97 0.89 0.89 0.89 0.92 0.76 0.84 0.79 0.94 0.94 0.88 0.94 0.69 0.88 0.56 0.77 0.96 0.73 1.00 0.77 0.88 0.69 0.95 0.93 0.87 0.98 0.84 0.97 0.89
0.77 0.92 0.95 0.97 1.00 1.00
0.77 0.80 0.87 0.87 0.95 1.00
Technique Project Time JxPath Codec JacksonXml JacksonDatabind Gson Math Jsoup Compress Mockito Cli Csv JacksonCore Lang Statistics Min 25% Quartile Mean Median 75% Quartile Max
0.77 0.89 0.93 0.95 0.98 1.00
0.70 0.78 0.84 0.86 0.89 1.00
0.77 0.92 0.94 0.97 1.00 1.00
0.69 0.75 0.80 0.77 0.83 1.00
0.77 0.88 0.92 0.93 0.99 1.00
0.56 0.70 0.79 0.79 0.83 1.00
CTM ATM LTM Change Extent Tree Edit Distance Cosine Similarity GMean Genetic Algorithm UnixCoder 0.72 0.92 0.80 0.59 0.86 0.95 0.88 0.83 0.83 1.00 0.50 0.83 0.85 0.67 0.80 0.72 0.78 0.78 0.59 0.84 0.80 0.47 0.71 0.86 0.87 0.91 0.85 0.87 0.74 0.89 0.84 0.90 0.85 0.75 1.00 0.94 0.73 0.65 0.69 0.57 0.81 0.80 0.47 0.62 0.75 0.74 0.86 1.00
0.50 0.72 0.80 0.83 0.88 1.00
0.69 0.80 0.83 0.82 0.85 0.95
TABLE V: Time Comparison (in minutes) of Best Performing Configurations of MCTM, CTM, ATM and LTM Method Project Cli Codec Compress Csv Gson JacksonCore JacksonDatabind JacksonXml Jsoup JxPath Lang Math Mockito Time Statistics Min 25% Quartile Mean Median 75% Quartile Max
MCTM ChgFrq Avg 0.17 0.49 0.82 0.31 0.64 0.63 2.66 0.17 0.42 0.36 1.90 2.54 0.89 1.65
CTM ChgExt GMean 0.14 0.30 0.45 0.26 0.36 0.27 0.90 0.19 0.22 0.13 0.76 0.99 0.68 0.52
ATM Tree Edit Genetic A. 1.06 2.74 3.55 0.63 17.05 2.18 46.97 0.39 4.68 0.93 66.25 140.73 33.68 320.40
LTM CosineSim. UnixCoder 0.43 0.41 0.58 0.26 1.48 0.47 3.26 0.16 0.44 0.24 5.12 12.75 1.67 7.97
0.17 0.38 0.98 0.63 1.46 2.66
0.13 0.23 0.44 0.33 0.64 0.99
0.39 1.34 45.80 4.12 43.65 320.40
0.16 0.38 2.60 0.53 2.93 12.75
difference is especially notable in project Jsoup (0.96 vs 0.53 accuracy), where classes often follow a Chain of Responsibility pattern [49]. Here, method-level granularity uncovers finegrained dependencies missed by static black-box class-level analysis [6]. These findings highlight the value of methodlevel metrics in guiding fault-revealing test selection. In terms of efficiency (TABLE V), MCTM incurs a modest overhead, averaging 0.98 minutes per minimization run compared to 0.48 minutes for CTM. This increase stems from the deeper method-level analysis but remains within practical limits. The highest observed time was 2.66 minutes for JacksonDatabind, the largest project in our dataset (74 KLoC, 3000+ commits, 1814 tests per version). Since test suite minimization is performed infrequently in real-world workflows [50], this additional cost is acceptable given the substantial improvements in fault detection. 3) RQ3 Results: To answer RQ3, we compared MCTM with the best configurations of two state-of-the-art blackbox TSM techniques: ATM and LTM. Table III, IV and V summarizes their performance in terms of accuracy, FDR, and execution time respectively under a 50% minimization budget. MCTM achieved a mean accuracy of 0.93 and FDR of 0.94, significantly outperforming all baseline methods. In comparison, ATM and LTM achieved 0.67/0.81 and 0.71/0.84 for accuracy and FDR, respectively. This substantial gain highlights the strength of method-level change-proneness in guiding test case selection. While similarity-based approaches like ATM and LTM aim to minimize overall redundancy, they
often exclude fault-revealing tests when similar test cases exist. This inter-test dependency reduces their fault detection potential. Such case occurs in project JacksonXml (0.37 accuracy with ATM and 0.50 with LTM), where occurrence of test cases are prominent which are very similar to the bug revealing ones. In contrast, MCTM evaluates each test case independently, scoring them based on their association with highly change-prone methods. This avoids such dependencies and ensures a higher likelihood of retaining fault-revealing tests. The superiority of MCTM over ATM and LTM is statistically validated using Fisher’s exact test. Against ATM, MCTM achieved an odds ratio of 78.0 (p = 0.000069). The comparison with LTM yields an even stronger odds ratio of 169.0 (p = 0.000010). Both results are highly significant (p < 0.001) and supported by tight confidence intervals, confirming the strength of these findings. In terms of execution time, MCTM averaged 0.98 minutes per version, compared to ATM’s 45.80 minutes and LTM’s 2.60 minutes. Even though LTM benefits from GPU acceleration (GTX 1050 Ti), it is still 2.65x slower than MCTM, which runs entirely on CPU. The high runtime of ATM and LTM arises from their computationally intensive all-pair similarity calculations and evolutionary search heuristics. In contrast, as discussed below in Subsection V-A, MCTM employs a lightweight, per-test analysis of change-proneness and dependencies, resulting in a linear-time process that scales efficiently with project size. These findings collectively demonstrate that method-level change-proneness is not only a more effective metric for blackbox TSM but also enables highly efficient implementations suitable for large-scale systems. V. D ISCUSSION A. Time Complexity The efficiency of our approach stems from its reliance on lightweight analysis of version control metadata. Both CTM and MCTM achieve a time complexity of O(n · m), where n represents the total number of commits and m the number of modified methods or classes on a commit. This enables them to scale well even for projects with extensive commit histories and large codebases. In contrast, similarity-based techniques such as ATM and LTM involve computing pairwise similarities across all test cases (O(t2 · d1 · d2 ), where t is the number of test cases and d1 , d2 are the AST depths of the respective test cases). These similarity computations are further coupled with evolutionary search algorithms (GA and NSGA-II), which operate over large populations and multiple generations. As a result, their overall complexity is exponential, leading to significantly higher execution times. This distinction is particularly relevant for large industrial systems where test suites may contain thousands of test cases, making scalability a critical factor.
B. Accuracy and FDR Trends A closer look at fault detection performance reveals interesting patterns across the different approaches. MCTM demonstrates comparable accuracy and FDR (0.93 mean accuracy and 0.95 mean FDR for MCTM). This similarity arises because both methods prioritize test cases associated with highly change-prone components. In the Defects4J dataset, multiple fault-revealing test cases often target the same change-prone regions. Consequently, once these regions are identified and prioritized, most relevant tests are naturally included in the reduced suite. This leads to high coverage of fault-detecting tests, resulting in stable accuracy and FDR across different minimization budgets. On the other hand, ATM and LTM focus on reducing redundancy by minimizing the overall similarity among selected test cases. While this strategy promotes diversity and increases the likelihood of capturing at least one fault-revealing test (thus boosting FDR), it can also deprioritize similar tests targeting the same fault. As a result, accuracy may suffer because not all fault-revealing test cases are retained. For example, TABLE III shows the mean accuracy of ATM is 0.67, while the mean FDR is 0.80 (TABLE IV) This trade-off between diversity and thoroughness is inherent in similarity-based approaches and highlights a key difference in their behavior compared to change-proneness-based techniques.
robustness in heterogeneous test suites, it comes at the cost of higher execution times and, in the case of LTM, dependency on GPU hardware. Their computational complexity limits scalability in large industrial systems where thousands of test cases exist. Overall, MCTM demonstrates a strong balance by achieving fine-grained fault localization through method-level CP analysis while retaining practical efficiency. This balance makes it a promising choice for real-world applications, where both effectiveness and scalability are critical. VI. T HREATS TO VALIDITY A. Construct Validity
It is also important to consider the computational requirements of these techniques. LTM relies on GPU-based inference to process large language model embeddings, requiring hardware support beyond standard CPU resources. In contrast, MCTM, CTM, and ATM operate entirely on CPU, making them more accessible for environments with limited hardware. Despite LTM benefiting from GPU acceleration (via a GTX 1050 Ti), our experiments show that MCTM consistently achieves lower execution times. This highlights its superior efficiency even under less favorable computational conditions and reinforces its suitability for practical adoption in resourceconstrained environments.
Construct validity examines whether the chosen metrics and methods accurately capture the effectiveness of TSM. We operationalize TSM effectiveness as the ability to retain tests that are likely to reveal faults while minimizing test suite size and execution overhead. Our central construct is that method-level change proneness serves as a valid proxy for fault proneness. Our approach relies on method-level change proneness metrics derived from version control metadata. While version control histories may not fully reflect semantic changes such as refactoring or formatting changes that do not alter program behavior, they have been widely used as proxies for code evolution in prior fault prediction and TSM studies [6], [33], [38]. We mitigate such noise by considering changes over the entire commit history, rather than relying on a small set of recent commits that may amplify transient fluctuations. Additionally, we utilize static call graphs to analyze test case dependencies. While static analysis may miss certain dynamic behaviors (e.g., inheritance, polymorphism resolved at runtime), it offers a practical and scalable means to approximate method interactions without requiring program execution. This is particularly important in our black-box setup, where we intentionally avoid reliance on production code or runtime environments. The effectiveness of static call graphs in similar contexts has been demonstrated in prior work [51], further supporting their use here.
D. Summary of Trade-offs
B. Internal Validity
The comparative analysis of MCTM, CTM, ATM, and LTM reveals distinct trade-offs between accuracy, fault detection robustness, computational efficiency, and hardware requirements. Change-proneness-based techniques (MCTM, CTM) excel in efficiency and scalability due to their lightweight reliance on version control metadata and static analysis. Their ability to prioritize tests covering highly change-prone regions leads to consistently high accuracy and fault detection rates, making them particularly effective for large projects with extensive test suites. However, this focus can potentially overlook tests unrelated to change-prone areas, which might be necessary for detecting less frequent, isolated faults. Similarity-based methods (ATM, LTM) emphasize test diversity, improving the likelihood of detecting faults across varied code paths. While this strategy enhances fault detection
Internal validity relates to whether observed effects are genuinely due to our methods rather than confounding factors. One notable concern is hardware disparity: LTM requires GPU-based inference to support large language models, whereas MCTM, CTM, and ATM run entirely on CPU. This difference could bias efficiency comparisons in favor of LTM. However, despite LTM benefiting from GPU acceleration (GTX 1050 Ti), our results show that MCTM consistently outperforms it in execution time, highlighting MCTM’s superior efficiency even under less favorable hardware conditions. We also recognize that ATM and LTM involve stochastic optimization techniques (GA and NSGA-II), which may produce non-deterministic results across runs. To address this variability, we executed each non-deterministic configuration 10 times and reported the average results. In contrast, MCTM
C. Hardware Considerations
and CTM are deterministic, ensuring stable and reproducible outputs. Additionally, all experiments were conducted on the same hardware platform to avoid variability introduced by differing system environments. C. External Validity External validity considers the generalizability of our findings beyond the studied dataset and context. Our experiments are based on 15 Java projects from Defects4J v2.0.1, covering 635 buggy versions. While these projects are open-source and may differ from industrial systems, Defects4J remains the most widely adopted benchmark in TSM and fault prediction research due to its collection of real-world faults and reproducible test suites [43]. This ensures our findings are relevant to the broader software engineering community. We also note that Defects4J versions contain a single fault per buggy version, whereas real-world systems often exhibit multiple interacting faults. While this is a limitation of all studies using Defects4J, it does not diminish the value of our work, as single-fault scenarios provide a controlled environment for evaluating fault detection capability. Future work can extend our evaluation to multi-fault datasets as they become available. Although our focus on Java projects may raise questions about applicability to other languages, the core principles of change-proneness and test code analysis are languageagnostic. Our approach can generalize to other ecosystems with minor adaptations. D. Conclusion Validity Conclusion validity pertains to whether the statistical analyses applied lead to accurate and trustworthy inferences. We employed Fisher’s exact test, a non-parametric test, suited to small sample sizes and categorical data, to assess whether differences in fault detection rates are statistically significant. Additionally, we reported odds ratios to quantify the strength of these differences. While multiple pairwise comparisons can increase the risk of Type I errors, our consistent findings across configurations and projects provide strong evidence of the robustness of our results. VII. C ONCLUSION AND F UTURE W ORK This study investigates the use of method-level CP as a metric for black-box TSM and incorporates it into an approach, MCTM. To select test cases for the reduced suite, MCTM extracts method level change-metrics to calculate CP, analyzes test case’s association with change-prone methods and scores test cases using statistical measures (average, median etc.). We evaluated MCTM on a large dataset under 8 different configurations and identified the optimal MCTM configuration (ChgFrq with Average aggregation). Comparative analysis with state-of-the-art TSM approaches demonstrates that MCTM achieves substantially higher effectiveness with 0.93 accuracy and 0.94 FDR while remaining efficient and lightweight. The findings demonstrate that, as a metric, method-level CP can guide for effective black-box TSM.
Future work can examine how results vary across different datasets, as well as the impact of statement-level CP and adaptation to other widely-used programming languages. R EFERENCES [1] S. Yoo and M. Harman, “Regression testing minimization, selection and prioritization: a survey,” Softw. Test. Verification Reliab., vol. 22, no. 2, pp. 67–120, 2012. [Online]. Available: https://doi.org/10.1002/stv.430 [2] S. U. R. Khan, S. P. Lee, N. Javaid, and W. Abdul, “A systematic review on test suite reduction: Approaches, experiment’s quality evaluation, and guidelines,” IEEE Access, vol. 6, pp. 11 816–11 841, 2018. [Online]. Available: https://doi.org/10.1109/ACCESS.2018.2809600 [3] A. A. Philip, R. Bhagwan, R. Kumar, C. S. Maddila, and N. Nagappan, “Fastlane: test minimization for rapidly deployed large-scale online services,” in Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, J. M. Atlee, T. Bultan, and J. Whittle, Eds. IEEE / ACM, 2019, pp. 408–418. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00054 [4] R. Pan, T. A. Ghaleb, and L. C. Briand, “ATM: black-box test case minimization based on test code similarity and evolutionary search,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 1700–1711. [Online]. Available: https://doi.org/10.1109/ICSE48619. 2023.00146 [5] ——, “Ltm: Scalable and black-box similarity-based test suite minimization based on language models,” IEEE Transactions on Software Engineering, pp. 1–19, 2024. [6] M. Siam, M. N. Fuad, and K. Sakib, “An exploratory study on the impact of change-proneness as a metric in black-box test suite minimization,” in IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Montreal, Canada, March 4-7, 2025, 2025, pp. 855–860. [7] H. Hata, O. Mizuno, and T. Kikuno, “Reconstructing fine-grained versioning repositories with git for method-level bug prediction,” IWESEP ‘10, pp. 27–32, 2010. [8] ——, “Bug prediction based on fine-grained module histories,” in 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland, M. Glinz, G. C. Murphy, and M. Pezzè, Eds. IEEE Computer Society, 2012, pp. 200–210. [Online]. Available: https://doi.org/10.1109/ICSE.2012.6227193 [9] C. Catal and B. Diri, “Investigating the effect of dataset size, metrics sets, and feature selection techniques on software fault prediction problem,” Information Sciences, vol. 179, no. 8, pp. 1040–1058, 2009. [10] E. Giger, M. Pinzger, and H. C. Gall, “Comparing fine-grained source code changes and code churn for bug prediction,” in Proceedings of the 8th International Working Conference on Mining Software Repositories, MSR 2011 (Co-located with ICSE), Waikiki, Honolulu, HI, USA, May 21-28, 2011, Proceedings, A. van Deursen, T. Xie, and T. Zimmermann, Eds. ACM, 2011, pp. 83–92. [Online]. Available: https://doi.org/10.1145/1985441.1985456 [11] Y. Luo, K. Ben, and L. Mi, “Software metrics reduction for faultproneness prediction of software modules,” in IFIP International Conference on Network and Parallel Computing. Springer, 2010, pp. 432– 441. [12] M. Mahdieh, S.-H. Mirian-Hosseinabadi, and M. Mahdieh, “Test case prioritization using test case diversification and fault-proneness estimations,” Automated Software Engineering, vol. 29, no. 2, p. 50, 2022. [13] E. Cruciani, B. Miranda, R. Verdecchia, and A. Bertolino, “Scalable approaches for test suite reduction,” in Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, J. M. Atlee, T. Bultan, and J. Whittle, Eds. IEEE / ACM, 2019, pp. 419–429. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00055 [14] E. Arvanitou, A. Ampatzoglou, A. Chatzigeorgiou, and P. Avgeriou, “A method for assessing class change proneness,” in Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE 2017, Karlskrona, Sweden, June 15-16, 2017, E. Mendes, S. Counsell, and K. Petersen, Eds. ACM, 2017, pp. 186–195. [Online]. Available: https://doi.org/10.1145/3084226.3084239
[15] S. U. R. Khan, S. P. Lee, R. W. Ahmad, A. Akhunzada, and V. Chang, “A survey on test suite reduction frameworks and tools,” Int. J. Inf. Manag., vol. 36, no. 6, pp. 963–975, 2016. [Online]. Available: https://doi.org/10.1016/j.ijinfomgt.2016.05.025 [16] B. Miranda and A. Bertolino, “Scope-aided test prioritization, selection and minimization for software reuse,” Journal of Systems and Software, vol. 131, pp. 528–549, 2017. [17] R. Noemmer and R. Haas, “An evaluation of test suite minimization techniques,” in International Conference on Software Quality. Springer, 2019, pp. 51–66. [18] S. Mohanty, S. K. Mohapatra, and S. F. Meko, “Ant colony optimization (aco-min) algorithm for test suite minimization,” in Progress in Computing, Analytics and Networking: Proceedings of ICCAN 2019. Springer, 2020, pp. 55–63. [19] H. Hemmati, A. Arcuri, and L. Briand, “Achieving scalable modelbased testing through test case diversity,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 22, no. 1, pp. 1–42, 2013. [20] M. Zhang, S. Ali, and T. Yue, “Uncertainty-wise test case generation and minimization for cyber-physical systems,” Journal of Systems and Software, vol. 153, pp. 1–21, 2019. [21] Y. Liu, K. Wang, W. Wei, B. Zhang, and H. Zhong, “User-sessionbased test cases optimization method based on agglutinate hierarchy clustering,” in 2011 International Conference on Internet of Things and 4th International Conference on Cyber, Physical and Social Computing. IEEE, 2011, pp. 413–418. [22] C. Coviello, S. Romano, G. Scanniello, A. Marchetto, G. Antoniol, and A. Corazza, “Clustering support for inadequate test suite reduction,” in 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2018, pp. 95–105. [23] A. Arrieta, S. Wang, U. Markiegi, A. Arruabarrena, L. Etxeberria, and G. Sagardui, “Pareto efficient multi-objective black-box test case selection for simulation-based testing,” Information and Software Technology, vol. 114, pp. 137–154, 2019. [24] C.-T. Lin, K.-W. Tang, and G. M. Kapfhammer, “Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests,” Information and Software Technology, vol. 56, no. 10, pp. 1322–1344, 2014. [25] C.-T. Lin, K.-W. Tang, C.-D. Chen, and G. M. Kapfhammer, “Reducing the cost of regression testing by identifying irreplaceable test cases,” in 2012 Sixth International Conference on Genetic and Evolutionary Computing. IEEE, 2012, pp. 257–260. [26] W. B. Johnson, J. Lindenstrauss et al., “Extensions of lipschitz mappings into a hilbert space,” Contemporary mathematics, vol. 26, no. 189-206, p. 1, 1984. [27] P. D. Turney and P. Pantel, “From frequency to meaning: Vector space models of semantics,” Journal of artificial intelligence research, vol. 37, pp. 141–188, 2010. [28] H. Sugimori and S. Hayashi, “Revisiting method-level change prediction: A comparative evaluation at different granularities,” CoRR, vol. abs/2502.17908, 2025. [Online]. Available: https://doi.org/10. 48550/arXiv.2502.17908 [29] J. Sohn and S. Yoo, “Empirical evaluation of fault localisation using code and change metrics,” IEEE Transactions on Software Engineering, vol. 47, no. 8, pp. 1605–1625, 2019. [30] M. Wen, R. Wu, and S.-C. Cheung, “How well do change sequences predict defects? sequence learning from software changes,” IEEE Transactions on Software Engineering, vol. 46, no. 11, pp. 1155–1175, 2018. [31] R. Abbas and F. A. Albalooshi, “A sequential comparative analysis of software change proneness prediction using machine learning,” Int. J. Softw. Innov., vol. 10, no. 1, pp. 1–16, 2022. [Online]. Available: https://doi.org/10.4018/ijsi.297993 [32] R. Koçi, X. Franch, P. Jovanovic, and A. Abelló, “Web API changeproneness prediction,” in IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2024, Rovaniemi, Finland, March 12-15, 2024. IEEE, 2024, pp. 429–434. [Online]. Available: https://doi.org/10.1109/SANER60148.2024.00050 [33] G. Rodrı́guez-Pérez, G. Robles, A. Serebrenik, A. Zaidman, D. M. Germán, and J. M. González-Barahona, “How bugs are born: a model to identify how bugs are introduced in software components,” Empir. Softw. Eng., vol. 25, no. 2, pp. 1294–1340, 2020. [Online]. Available: https://doi.org/10.1007/s10664-019-09781-y [34] W. Rhmann, B. Pandey, G. A. Ansari, and D. K. Pandey, “Software fault prediction based on change metrics using hybrid algorithms: An empirical study,” J. King Saud Univ. Comput.
Inf. Sci., vol. 32, no. 4, pp. 419–424, 2020. [Online]. Available: https://doi.org/10.1016/j.jksuci.2019.03.006 [35] F. Rahman and P. Devanbu, “Ownership, experience and defects: a finegrained study of authorship,” in Proceedings of the 33rd international conference on software engineering, 2011, pp. 491–500. [36] L. Kumar, S. Lal, A. Goyal, and N. L. B. Murthy, “Change-proneness of object-oriented software using combination of feature selection techniques and ensemble learning techniques,” in Proceedings of the 12th Innovations on Software Engineering Conference (formerly known as India Software Engineering Conference), ISEC 2019, Pune, India, February 14-16, 2019, R. Naik, S. Sarkar, T. T. Hildebrandt, A. Kumar, and R. Sharma, Eds. ACM, 2019, pp. 8:1–8:11. [Online]. Available: https://doi.org/10.1145/3299771.3299778 [37] E. Arisholm, L. C. Briand, and A. Føyen, “Dynamic coupling measurement for object-oriented software,” IEEE Trans. Software Eng., vol. 30, no. 8, pp. 491–506, 2004. [Online]. Available: https://doi.org/10.1109/TSE.2004.41 [38] A. G. Koru and J. Tian, “Comparing high-change modules and modules with the highest measurement values in two large-scale open-source products,” IEEE Trans. Software Eng., vol. 31, no. 8, pp. 625–642, 2005. [Online]. Available: https://doi.org/10.1109/TSE.2005.89 [39] M. Keshani, G. Gousios, and S. Proksch, “Frankenstein: fast and lightweight call graph generation for software builds,” Empir. Softw. Eng., vol. 29, no. 1, p. 1, 2024. [Online]. Available: https://doi.org/10.1007/s10664-023-10388-7 [40] R. Haas, R. Nömmer, E. Juergens, and S. Apel, “Optimization of automated and manual software tests in industrial practice: A survey and historical analysis,” IEEE Trans. Software Eng., vol. 50, no. 8, pp. 2005–2020, 2024. [Online]. Available: https: //doi.org/10.1109/TSE.2024.3418191 [41] E. Ahmadi, M. Elsanhoury, K. Selvan, P. Välisuo, and H. Kuusniemi, “Robust multi-sensor fusion positioning based on gnss/imu using factor graph optimization,” in 2025 IEEE/ION Position, Location and Navigation Symposium (PLANS). IEEE, 2025, pp. 1247–1256. [42] P. J. Rousseeuw and M. Hubert, “Robust statistics for outlier detection,” Wiley interdisciplinary reviews: Data mining and knowledge discovery, vol. 1, no. 1, pp. 73–79, 2011. [43] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440. [44] D. Spadini, M. Aniche, and A. Bacchelli, “Pydriller: Python framework for mining software repositories,” in Proceedings of the 2018 26th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2018, pp. 908– 911. [45] J. Jász, I. Siket, E. Pengo, Z. Ságodi, and R. Ferenc, “Systematic comparison of six open-source java call graph construction tools,” in Proceedings of the 14th International Conference on Software Technologies, ICSOFT 2019, Prague, Czech Republic, July 26-28, 2019, M. van Sinderen and L. A. Maciaszek, Eds. SciTePress, 2019, pp. 117– 128. [Online]. Available: https://doi.org/10.5220/0007929201170128 [46] M. Raymond and F. Rousset, “An exact test for population differentiation,” Evolution, pp. 1280–1283, 1995. [47] A. Arcuri and L. Briand, “A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering,” Software Testing, Verification and Reliability, vol. 24, no. 3, pp. 219–250, 2014. [48] Anonymous, “Mctm replication package,” 2025, accessed: 2025-05-31. [Online]. Available: https://figshare.com/s/8276e5a92bdf39b08d93 [49] E. Freeman and E. Freeman, Head first design patterns - your brain on design patterns. O’Reilly, 2004. [Online]. Available: http://www.oreilly.de/catalog/hfdesignpat/index.html [50] R. Noemmer and R. Haas, “An evaluation of test suite minimization techniques,” in Software Quality: Quality Intelligence in Software and Systems Engineering - 12th International Conference, SWQD 2020, Vienna, Austria, January 14-17, 2020, Proceedings, ser. Lecture Notes in Business Information Processing, D. Winkler, S. Biffl, D. Méndez, and J. Bergsmann, Eds., vol. 371. Springer, 2020, pp. 51–66. [Online]. Available: https://doi.org/10.1007/978-3-030-35510-4 4 [51] Q. Luo, K. Moran, and D. Poshyvanyk, “A large-scale empirical comparison of static and dynamic test case prioritization techniques,” in Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, 2016, pp. 559–570.