Temporal Modeling of Change History for Black-Box Test Suite Minimization Kamruzzaman Asif, Md. Siam and Kazi Sakib
arXiv:2605.25441v1 [cs.SE] 25 May 2026
Institute of Information Technology, University of Dhaka, Bangladesh {bsse1217, bsse1104, sakib}@iit.du.ac.bd
Keywords:
Test Suite Minimization, Black-Box Testing, Change History, Temporal Modeling.
Abstract:
Test Suite Minimization (TSM) reduces the size of test suites while preserving their fault detection capability. In black-box TSM, reduction is performed without relying on production-code instrumentation. While several black-box TSM approaches have explored metrics like test logs or test similarity, these often suffer from scalability and efficiency issues. Recently, change history has been explored as a lightweight and scalable indicator for guiding black-box TSM. However, existing approaches treat historical modifications uniformly, ignoring the temporal dynamics of software evolution where recently modified code tends to be more fault-prone. To address this limitation, we introduce temporal modeling into black-box TSM and propose Temporal Riskdriven Test Suite Minimization (TRTM). TRTM extracts modification history from version-control metadata and applies exponential temporal attenuation to weight changes based on recency, producing time-weighted class-level risk scores that reflect fault-proneness. Next, it determines dependencies between test cases and production classes by constructing static call graphs derived solely from test code, preserving the black-box setting. The risk scores of the classes exercised by each test case are then aggregated using statistical measures such as Average and Geometric Mean to compute a risk score for the test case. Finally, test cases with the highest risk scores are selected to construct the reduced suite. Evaluation on a large dataset containing 14 projects with 631 versions shows that TRTM consistently outperforms the state-of-the-art baseline, achieving a mean Accuracy of 0.72 (vs. 0.66) and Fault Detection Rate (FDR) of 0.75 (vs. 0.69), while also reducing execution time.
1
INTRODUCTION
Modern software systems evolve continuously through frequent commits, refactorings, and feature additions. In large-scale and continuously integrated environments, executing the entire test suite for every version is often impractical due to time and resource constraints (Khan et al., 2018). As a result, test suite minimization (TSM) has emerged as an important technique to reduce testing effort by removing redundant test cases while preserving fault detection effectiveness (Yoo and Harman, 2012; Khan et al., 2018). In recent years, black-box test suite minimization approaches have gained attention because, unlike white-box techniques, they do not require productioncode analysis or structural coverage instrumentation, making them applicable to industrial systems where instrumentation is costly or unavailable (Philip et al., 2019; Cruciani et al., 2019; Pan et al., 2023; Pan et al., 2024). Existing work on black-box TSM explores diverse surrogate indicators to approximate a
test case’s fault-detection potential. These include execution-history information such as commit risk and test outcome correlations (Philip et al., 2019), and similarity-based diversity measures derived from test code representations (Cruciani et al., 2019; Pan et al., 2023; Pan et al., 2024). While these approaches avoid structural instrumentation, they are limited by factors such as data availability, computational cost, or representation complexity. More recently, software change history has been explored as a lightweight indicator for guiding blackbox test suite minimization within the Changeproneness based Test suite Minimization (CTM) framework (Siam et al., 2025). However, it treats historical modifications uniformly, assigning equal importance to both recent and older changes, thereby overlooking the inherently temporal nature of software evolution. Prior studies in defect prediction and software reliability have shown that software risk exhibits temporal dynamics, where recently modified components are more likely to be fault-prone compared to components
whose changes have stabilized over time (Graves et al., 2002; Nagappan and Ball, 2005; Hassan and Holt, 2005). Ignoring temporal recency may therefore attenuate the predictive resolution of change-based risk estimation and obscure short-term instability patterns. To address these limitations, we introduce Temporal Risk-driven Test Suite Minimization (TRTM), an approach that integrates temporal modeling as an explicit dimension within change history-based blackbox test suite minimization. TRTM first extracts change history from version-control repositories to compute class-level measures using two change metrics—change frequency and change extent—to capture how often and how substantially a class has evolved (Arvanitou et al., 2017). It then applies exponential temporal attenuation to these historical modifications, allowing their influence to decay as a function of age. This temporally weighted formulation emphasizes recent changes and produces time-weighted risk scores that reflect the likelihood of a class being fault-prone. Next, TRTM constructs test–class dependency mappings using static call-graph analysis derived solely from the test code. Finally, these time-weighted risks are aggregated using statistical operators (e.g., Average, Geometric Mean, Harmonic Mean, and Median) to compute a risk score for each test case, and the highest-scoring tests are selected to form the minimized test suite. We systematically evaluate TRTM across a range of temporal horizons, change metrics, and aggregation operators. The evaluation is conducted on a large dataset containing 14 projects with 631 versions from the Defects4J benchmark. We assess fault detection effectiveness using Accuracy and Fault Detection Rate (FDR), and measure efficiency in terms of execution time. To examine the impact of temporal modeling, TRTM is compared against the state-of-the-art static change history-based baseline, CTM (Siam et al., 2025), under identical minimization settings. The results show that incorporating temporal modeling consistently improves fault detection effectiveness over the static baseline. Across the explored configurations, TRTM achieves strong performance, and at the 50% minimization budget under a representative setting, attains a mean Accuracy of 0.72 and FDR of 0.75, compared to 0.66 and 0.69 for CTM, with statistically significant improvements for both metrics. In addition, TRTM achieves lower execution time, with a mean of 0.82 minutes per version compared to 1.04 minutes for CTM. Overall, these findings demonstrate that temporal recency not only enhances the effectiveness of change history-based minimization but also delivers practical efficiency gains, making it an effective modeling dimension for black-box TSM.
2
RELATED WORK
Test Suite Minimization (TSM) aims to reduce testing effort by removing redundant test cases while preserving fault detection effectiveness (Yoo and Harman, 2012; Khan et al., 2018). Over the years, extensive research has proposed various strategies to identify representative subsets of test suites. Broadly, these techniques fall into two categories: white-box and black-box approaches. White-box TSM relies on production code analysis, structural coverage information, or requirement traceability to guide test case selection. Representative techniques include greedy coverage-based heuristics (Miranda and Bertolino, 2017; Noemmer and Haas, 2019; Mohanty et al., 2020), search-based evolutionary algorithms (Hemmati et al., 2013; Zhang et al., 2019), clustering methods based on coverage similarity (Liu et al., 2011; Coviello et al., 2018), and model-driven or requirement-driven prioritization strategies (Arrieta et al., 2019). While effective in controlled environments, such approaches require code instrumentation and coverage collection, which can introduce significant computational overhead and scalability challenges in large industrial systems (Khan et al., 2018). In rapidly evolving continuous integration environments, these constraints limit the practical adoption of whitebox minimization techniques. To address these limitations, black-box TSM approaches eliminate reliance on structural coverage and instead leverage surrogate information from test code or development history. FastLane (Philip et al., 2019), the first black-box TSM technique, uses commit risk indicators, version control metadata, and historical test logs within a statistical learning framework to predict test relevance. Although FastLane achieves substantial reduction while preserving fault detection effectiveness, it depends on historical execution data, which may not always be available. Subsequent work explored similarity-based blackbox reduction. FAST-R (Cruciani et al., 2019) represents test cases using term-frequency vectors from test code and applies clustering to select representative tests. ATM (AST-based Test case Minimizer) (Pan et al., 2023) further refines this approach using syntactic representations derived from abstract syntax trees and evolutionary multi-objective optimization. To improve semantic expressiveness and scalability, LTM (Language model-based Test suite Minimization) (Pan et al., 2024) leverages large language model embeddings to capture deeper test similarity relationships. While these approaches demonstrate strong effectiveness, they require substantial computational resources for similarity computation or embedding generation.
Figure 1: TRTM approach overview.
More recently, CTM (Change-proneness based Test suite Minimization) (Siam et al., 2025) explores software change history as a lightweight and scalable source of information for black-box TSM. CTM leverages historical modifications to guide test case selection, prioritizing tests associated with frequently or substantially modified components. By relying solely on change history, it provides a practical alternative to similarity- or execution-log-based approaches. Despite these advances, existing black-box TSM techniques—whether execution-driven or change history-based—generally treat historical information as temporally uniform. However, research in defect prediction and software reliability has consistently shown that software risk is inherently timesensitive, with recently modified components exhibiting higher fault-proneness compared to components whose changes have stabilized over time (Graves et al., 2002; Nagappan and Ball, 2005). This gap motivates the need to explicitly incorporate temporal sensitivity into change history-based black-box test suite minimization.
logical change history from version-control metadata. These records capture both the frequency and magnitude of historical modifications. TRTM then applies exponential temporal attenuation to these modification events, weighting changes according to their recency. Consequently, classes with substantial recent modification activity receive higher risk scores, guiding which test cases are retained in the minimized suite. As illustrated in Figure 1, the TRTM pipeline consists of the following stages. First, chronological modification records of production classes are extracted from version-control history. Second, class-level temporal risk is computed by applying exponential attenuation to these modification events. Third, static analysis is performed to map each test case to the production classes it exercises. Fourth, class-level risk scores of the classes exercised by each test case are aggregated using statistical operators to assign a risk score for that test case. Finally, the highest-risk test cases are selected to construct the minimized test suite. The following subsections describe each stage in detail.
3.1
3
METHODOLOGY
In this study, we propose Temporal Risk-driven Test Suite Minimization (TRTM), a black-box TSM approach designed to retain a subset of test cases while preserving fault detection effectiveness. Software faults frequently arise in parts of the system that undergo structural modifications during development. TRTM therefore estimates the likelihood of each production class being fault-prone by analyzing its chrono-
Extract Change History
We begin by extracting the historical modification records of classes from the project’s version-control repository. This yields a chronological view of each class’s evolution, achieved by analyzing the Git commit history. For each commit, the modified classes are identified and associated change metadata is collected. Specifically, for each class C, modification events are recorded with: (i) the commit timestamp, and (ii) the number of lines added, deleted, or modified in that commit. These commit-level records capture two key
aspects of structural evolution: how frequently a class changes (change frequency) and how extensively it evolves (change extent). Such change metrics have been widely used in prior software engineering studies as indicators of software evolution and predictors of fault-proneness (Rhmann et al., 2020). Because Git records modifications at the file-path level, the same logical class may appear under different paths over time due to file renaming or relocation. To preserve class identity across such changes, we perform a consolidation step during preprocessing. We first extract change histories for all unique file paths in the repository and parse each path to determine the corresponding logical class name. When multiple file paths correspond to the same class, their records are merged by taking the union of events and ordering them chronologically by timestamp. After this consolidation process, each production class is represented by a unified, time-ordered sequence of modification events. These chronological histories form the basis for the temporal risk modeling described in the next subsection.
3.2
Using the modification history extracted in the previous step, we calculate the risk (i.e., the likelihood of being fault-prone) associated with each production class by incorporating the temporal recency of its historical modifications. The intuition is that recent development activity may indicate unstable code regions and therefore deserves greater influence when calculating class-level risk. Let a production class C have a sequence of n historical modification events. Each event i occurred at time ti . For a target version evaluated at time t, we define the age of modification event i as (1)
where ∆ti represents the elapsed time since the modification occurred, measured in days. To account for temporal recency, the contribution of each modification event is attenuated as a function of its age. Specifically, we estimate the temporal risk of class C as n
Rα (C) = ∑ wi e−α∆ti ,
This yields the temporally weighted change frequency: n
Freq
Rα (C) = ∑ e−α∆ti .
(2)
i=1
where wi denotes the magnitude of modification event i, and α > 0 is a temporal attenuation parameter controlling how rapidly the influence of older modifications decays over time. This formulation assigns greater weight to recent modifications while gradually
(4)
i=1
Change Extent. Here, the contribution of each modification event is weighted by the magnitude of code churn introduced in that commit. We define the churn of modification event i as Churni = Addi + Deli + Modi ,
Temporal Risk Modeling
∆ti = t − ti ,
reducing the impact of older events. Similar exponential attenuation mechanisms have been employed in prior defect prediction and reliability studies to model the diminishing influence of historical development activity (Graves et al., 2002; Hassan and Holt, 2005). The event weight wi depends on the change metric used to characterize the modification. In this study, we consider two complementary change metrics commonly used in software evolution analysis. Change Frequency. In this formulation, each modification event contributes equally to the risk estimation: wi = 1. (3)
(5)
where Addi , Deli , and Modi denote lines added, deleted, and modified in event i, respectively. To mitigate the heavy-tailed distribution typically observed in such metrics, we apply logarithmic normalization: wi = ln (1 + Churni ) .
(6)
The resulting temporally weighted change extent is given by n
−α∆ti RExt . α (C) = ∑ ln (1 + Churni ) e
(7)
i=1
The attenuation parameter α determines how strongly the model prioritizes recent modifications over older ones. Rather than fixing a single value, we systematically evaluate a range of decay configurations spanning multiple temporal scales to analyze the impact of temporal sensitivity on minimization effectiveness. For interpretability, α is parameterized using a half-life formulation ln(2) , (8) T where T denotes the temporal half-life of a modification’s risk. Under this formulation, the influence of a modification decreases to 50% of its original value after T days. Varying T allows the model to systematically examine how different temporal sensitivities influence risk estimation, from short-term recency to long-term historical effects. α=
3.4
Aggregate Class Risk to Test Cases
The dependency mappings defined in Equation 9 identify the production classes exercised by each test case. Using these mappings, class-level temporal risk scores are aggregated for each test case. Let T denote a test case and Deps(T ) = {C1 ,C2 , . . . ,Cm } denote the set of production classes reachable from T through the static call graph. The score of test case T is computed by aggregating the temporal risk scores of all classes in Deps(T ): Score(T ) = A({Rα (C) | C ∈ Deps(T )}),
Figure 2: Test-Class Dependencies.
3.3
Extract Test–Class Dependencies
To associate class-level temporal risk with test cases, we identify which production classes are exercised by each test case. This step establishes the structural relationships required to associate test cases with the classes whose risk they potentially expose. Consistent with the black-box setting, we derive this mapping using static analysis of the test code without instrumenting or modifying the production system. For our analysis, we generate call graphs (graymarked portion of Figure 2) strictly from the test suite of each project using the java-callgraph tool. The test code is first compiled into JAR files, which serve as a direct input to construct the call graphs. We opt for static call graphs because they are computationally efficient and highly suitable for a black-box setup. Unlike dynamic call graphs, which require executing the program and monitoring runtime behavior, static call graphs can be derived directly from compiled bytecode. This avoids the need for deep source code analysis or system instrumentation, aligning with our goal of minimizing test suites in a production-agnostic manner. Treating each test method as an entry point in the call graph, we perform a Depth-First Search (DFS) to traverse the invocation edges and collect all reachable associated classes. This DFS traversal ensures we systematically capture both direct invocations and deep transitive dependencies between the test suite and the production classes. Formally, for a test case T , we define its static dependency set as: Deps(T ) = {C1 ,C2 , . . . ,Cm },
(9)
where each C j denotes a production class that is directly or indirectly reachable from T via the static call graph. The resulting test–class dependency mappings provide the structural foundation required to aggregate our temporally weighted class risks to individual test cases in the subsequent step.
(10)
where Rα (C) denotes the temporally weighted risk score of class C computed in Section 3.2, and A(·) represents an aggregation operator applied to the set of class-level risk scores. Several statistical operators can be used to summarize class-level risk distributions, including Minimum, Maximum, Sum, Average, Geometric Mean, Harmonic Mean, Median, and Standard Deviation. However, operators such as Minimum and Maximum are highly sensitive to outliers and may assign disproportionate influence to extreme class risks (Ahmadi et al., 2025; Rousseeuw and Hubert, 2011). Similarly, Standard Deviation primarily captures variability rather than central tendency and can be unstable for skewed distributions (Rousseeuw and Hubert, 2011). The Sum operator may also bias the score toward test cases with a large number of dependencies, potentially overshadowing smaller dependency sets that contain highly fault-prone classes. To obtain stable and robust test-level risk estimates, four aggregation operators are considered: Arithmetic Mean (Avg), Geometric Mean (GMean), Harmonic Mean (HMean), and Median. These operators provide complementary perspectives on central tendency while remaining less sensitive to extreme values in the classlevel risk distribution.
3.5
Risk-Based Selection
The aggregation process described in Section 3.4 produces a test-level risk score for each test case. These scores are used to guide budget-constrained test suite minimization. For each project, we compute the testlevel risk scores for all test cases. We then select the subset of test cases with the highest score values under a predefined retention budget. Consistent with prior black-box TSM studies, we evaluate three commonly adopted budget levels—25%, 50%, and 75% of the original test suite (Pan et al., 2023; Pan et al., 2024; Siam et al., 2025). The selected subset constitutes the minimized test suite.
4
EXPERIMENTAL SETUP AND RESULT ANALYSIS
In this section, we evaluate TRTM through a series of empirical experiments. We begin by outlining the research questions that guide our study, followed by a description of the experimental design, including the subject systems, configurations, and evaluation metrics. We then present and analyze the results obtained from applying TRTM to the test suites.
4.1
Research Questions
To systematically examine the role of temporal modeling in test suite minimization, our study is guided by the following research questions: RQ1: How does temporal modeling influence the effectiveness of test suite minimization under different configurations? This research question investigates the sensitivity of TRTM to its configuration parameters, examining how variations in temporal decay horizons, change metrics, and aggregation operators influence fault detection effectiveness. Since different configurations may lead to different outcomes, we evaluate TRTM across these dimensions to understand how temporal sensitivity and risk aggregation affect the preservation of fault-revealing test cases and to identify stable and effective configuration settings. RQ2: Does temporal modeling improve the effectiveness and efficiency of test suite minimization compared with the state-of-the-art static approach? This research question evaluates whether incorporating temporal modeling improves fault detection effectiveness and computational efficiency. To answer this, we take the representative TRTM configuration identified in RQ1 and compare it directly against the state-of-the-art baseline, CTM (Siam et al., 2025), under identical budgets and experimental settings.
4.2
Experimental Design
To evaluate TRTM, we conduct a systematic comparison against CTM, the state-of-the-art static change history-based approach, across multiple configurations. All experiments were executed on a machine equipped with an Intel Core i7-9750H processor (6 cores, 12 threads; base frequency 2.60 GHz with turbo up to 4.50 GHz), 30 GB RAM, running Ubuntu 24.04.4 LTS. The implementation is deterministic, yielding identical results across runs. 1) Configurations: A configuration is defined by a combination of change metric, temporal horizon, and aggregation operator used to compute test case risk
scores. We consider two class-level change metrics: Change Frequency and Change Extent. These metrics are extended using temporal attenuation as described in Section 3.2. To systematically analyze the influence of temporal recency, we evaluate the temporal attenuation parameter across a base-2 exponential progression of temporal horizons ranging from 20 to 29 days (1–512 days). Such logarithmic parameter sweeps are commonly used to explore sensitivity across multiple orders of magnitude while avoiding bias toward specific ranges (Hutter et al., 2011; Bergstra and Bengio, 2012). The upper bound of 512 days deliberately exceeds the empirically validated predictive windows reported in defect prediction literature (Graves et al., 2002), ensuring that the full relevant temporal range is covered without imposing arbitrary horizon choices. For each test case, class-level risk scores of the exercised classes are aggregated using the statistical operators defined in Section 3.4. In total, this results in 2 × 10 × 4 = 80 TRTM configurations (2 change metrics, 10 temporal horizons, and 4 aggregation operators) evaluated across all project versions. 2) Minimization Budgets: The minimization budget defines the proportion of test cases retained after reduction relative to the original test suite. Consistent with prior black-box TSM studies, we evaluate three retention levels: 25%, 50%, and 75% (Pan et al., 2023). For each configuration and budget level, test cases with the highest association scores are selected until the specified retention percentage is reached.
4.3
Dataset
We evaluate TRTM using Defects4J1 (v2.0.1), a widely adopted benchmark in software testing research (Just et al., 2014). Defects4J provides real, reproducible faults from open-source Java projects along with their corresponding test suites and fault-fixing commits. Each buggy version contains a single real fault that triggers failures in one or more test cases. The initial dataset includes 16 projects with 661 buggy versions. We exclude Chart due to unreliable version-control history that prevented reliable extraction of historical change information, and Collections due to version inconsistencies and its limited size of only 4 buggy versions, which limits meaningful empirical evaluation. The final dataset consists of 14 projects and 631 buggy versions. For each buggy version, change history is extracted only from preceding commits to avoid using future information. Table 1 summarizes the characteristics of the subject systems. Project sizes range from 2–74 KLoC, 1 https://github.com/rjust/defects4j
Table 1: Subject projects statistics (Defects4J v2.0.1). Project Cli Codec Compress Csv Gson JacksonCore JacksonDatabind JacksonXml Jsoup JxPath Lang Math Mockito Time
2 9 45 2 9 31 74 6 14 20 30 71 21 30
39 18 47 16 18 26 112 6 93 22 64 106 38 26
4 15 29 7 20 45 72 10 13 6 61 73 36 56
256 413 404 193 984 356 1,814 152 494 250 1,796 2,078 1,182 3,918
486 928 1,393 828 1,226 902 3,009 660 785 374 2,398 2,932 1,880 1,600
with test suites spanning 4–73 KLoC. The number of buggy versions per project varies between 6 and 112, and test suites contain between 152 and 3,918 test cases per version on average. Statistics are derived from repository metadata and CLOC analysis. By using the same benchmark adopted in prior black-box TSM studies (Pan et al., 2023; Pan et al., 2024; Siam et al., 2025), the evaluation ensures compatibility with existing work while covering diverse system sizes and evolution histories.
4.4
(Pan et al., 2023). It is defined as:
Size # Versions Test Size Avg # Tests Avg # Commits (KLoC) (faults) (KLoC) per version per version
Evaluation Metrics and Statistical Analysis
To evaluate the performance of TRTM, we consider three metrics: Accuracy, Fault Detection Rate (FDR), and Execution Time. 1) Accuracy: The primary objective of test suite minimization is to reduce the number of test cases while maintaining the fault detection capability. Accuracy measures the proportion of fault-revealing test cases retained in the minimized test suite relative to the original suite. Formally, accuracy is defined as: Accuracy =
|F ′ | , |F|
(11)
where |F ′ | denotes the number of fault-revealing test cases retained in the minimized test suite, and |F| represents the total number of fault-revealing test cases in the original test suite for a given buggy version. Accuracy therefore quantifies how well the minimized suite preserves the defect-exposing capacity of the original test suite at the test-case level. 2) Fault Detection Rate (FDR): While accuracy reflects proportional preservation, Fault Detection Rate (FDR) provides a more relaxed perspective by evaluating whether at least one fault-revealing test case is retained after minimization. FDR measures the percentage of buggy versions for which the minimized test suite contains at least one fault-revealing test case
FDR =
1 n ∑ fi , n i=1
(12)
where n denotes the total number of buggy versions and 1, if the minimized suite for version i retains at least one fault-revealing test fi = case, 0, otherwise. (13) 3) Execution Time: To assess the efficiency of TRTM, we measure the total execution time, which includes the time required for change-history extraction, callgraph generation, test–class dependency analysis, and risk scoring. Statistical Analysis: To assess whether TRTM leads to statistically significant improvements over the baseline approach, we perform paired comparisons at the project-version level. For continuous effectiveness measures, we apply the Wilcoxon signed-rank test, a non-parametric test suitable for paired samples. For the binary fault detection outcome, we use Fisher’s exact test to compare the proportion of detected faults. Statistical tests are conducted between TRTM and the baseline under identical minimization budgets (25%, 50%, and 75%). To control for multiple comparisons, Bonferroni correction is applied to adjust the significance threshold. Cliff’s delta (δ) is reported to provide an estimate of effect size.
4.5
Result Analysis
Analysis focuses on results obtained at the 50% minimization budget, as trends observed at the 25% and 75% budgets are consistent and are reported in the replication package (Asif et al., 2026). The first part of the analysis examines how effectiveness varies across different temporal decay horizons, change metrics, and aggregation operators, capturing the sensitivity of TRTM to its configuration choices (RQ1). The second part compares the best-performing TRTM configuration against the state-of-the-art static baseline approach (RQ2). 4.5.1
RQ1: Influence of Temporal Modeling
Figure 3 summarizes the performance of TRTM across temporal decay horizons, aggregation operators, and change metrics at the 50% minimization budget. The figure provides an overview of how minimization effectiveness varies as temporal sensitivity changes across the evaluated configurations.
Accuracy x Change Extent
0.69
0.68
0.73
1d
0.71
0.69
0.69
0.71
2d
0.71
0.70
0.69
0.72
2d
0.70
0.71
0.68
0.71
4d
0.71
0.71
0.67
0.71
4d
0.70
0.72
0.67
0.70
8d
0.71
0.70
0.68
0.70
8d
0.69
0.71
0.69
0.71
16d
0.69
0.70
0.65
0.70
16d
0.70
0.71
0.67
0.71
32d
0.68
0.70
0.67
0.71
32d
0.70
0.72
0.68
0.71
64d
0.68
0.71
0.70
0.71
64d
0.70
0.72
0.70
0.71
128d
0.67
0.71
0.71
0.68
128d
0.67
0.72
0.72
0.70
256d
0.65
0.70
0.70
0.67
256d
0.64
0.70
0.71
0.67
512d
0.62 Avg
0.65
0.65
0.64
512d
ean
ean
n dia
0.63 Avg
HM
Strategy
Me
0.65 n dia
HM
Strategy
0.72
0.71
0.76
1d
0.74
0.72
0.73
0.74
2d
0.74
0.73
0.72
0.74
2d
0.73
0.74
0.71
0.74
4d
0.74
0.74
0.71
0.74
4d
0.73
0.75
0.71
0.73
8d
0.73
0.73
0.72
0.73
8d
0.73
0.74
0.73
0.74
16d
0.72
0.73
0.68
0.73
16d
0.73
0.74
0.70
0.75
32d
0.71
0.72
0.70
0.74
32d
0.72
0.75
0.71
0.74
64d
0.71
0.74
0.72
0.74
128d
0.71
0.74
0.74
0.72
256d
0.69
0.73
0.72
512d
0.65 Avg
0.68 ean
0.68 ean
HM
Strategy
0.68 0.66 0.64 0.62
FDR x Change Extent
0.74
GM
0.70
Me
1d
Temporal Horizon
Temporal Horizon
FDR x Change Frequency
G
0.69 ean
0.72
0.76 0.74
Mean FDR
GM
0.65 an Me
0.74
Mean Accuracy
0.71
Temporal Horizon
Temporal Horizon
Accuracy x Change Frequency 1d
0.72 0.70
64d
0.73
0.75
0.73
0.73
128d
0.71
0.75
0.75
0.73
0.71
256d
0.68
0.72
0.74
0.71
0.66
0.67 ian
512d
0.66 Avg
0.69 ean
0.72 ean
0.69 ian
0.64
d Me
GM
HM
Strategy
d Me
0.68
Figure 3: Four-panel heatmap of TRTM performance across temporal horizons at the 50% minimization budget. Rows correspond to temporal horizons (1–512 days at base-2 intervals); columns correspond to aggregation operators (Avg, GMean, HMean, Median). Cell values report mean Accuracy (top panels) and FDR (bottom panels) across all project versions for Change Frequency (left) and Change Extent (right) metrics.
A consistent trend is visible across both change metrics and all aggregation operators. Configurations with shorter temporal horizons generally achieve higher effectiveness, while performance gradually declines as the horizon increases. In particular, results remain strongest within the lower portion of the evaluated range (approximately 1–128 days) and decrease toward the longest horizons. For example, under the GMean/ChangeExtent configuration, mean Accuracy reaches approximately 0.72 at intermediate horizons but drops to 0.65 at the 512-day horizon. This pattern indicates that temporal modeling is most beneficial when recent modifications receive stronger emphasis, whereas very long horizons increasingly resemble static change aggregation. Although the horizon effect is visible across all operators, their behaviors differ. The Avg operator shows a steady performance decline as the horizon increases. Under ChangeExtent, mean Accuracy decreases from approximately 0.71 at the 1-day horizon to around 0.63 at the 512-day horizon. This pattern reflects the arith-
metic mean’s tendency to dilute the influence of recent modifications as older changes accumulate. In contrast, the HMean operator exhibits unstable behavior with irregular drops at specific horizons. For example, under ChangeFrequency, mean Accuracy falls to about 0.65 at the 16-day horizon while adjacent horizons remain near 0.67–0.68. This instability occurs because the harmonic mean is highly sensitive to near-zero values; when temporal decay produces sparse risk distributions, even a single low-risk class can substantially suppress the aggregated score. The remaining operators, GMean and Median, exhibit more consistent behavior. Median achieves the highest peak Accuracy (0.73) at the 1-day horizon under ChangeFrequency, but this performance declines steadily as the horizon increases. The strong performance at very short horizons likely reflects the close temporal proximity between code modifications and fault introduction in the Defects4J dataset, where buggy versions often occur shortly after the changes that introduced the fault. In contrast, GMean main-
tains stable Accuracy between approximately 0.71 and 0.72 across a wide range of horizons (2–128 days), indicating a more stable aggregation behavior. Table 2 further depicts the stability of these two operators across all project versions. Although Median achieves slightly higher mean values (Accuracy 0.73 and FDR 0.76), its minimum Accuracy drops to 0.48, indicating larger variability across projects. In contrast, GMean maintains a higher minimum Accuracy of 0.53 while achieving comparable mean performance (Accuracy 0.72 and FDR 0.75), suggesting a more stable performance floor. This variability can also be observed in individual projects. For example, in JacksonCore, Median/ChangeFrequency (1-day horizon) yields an Accuracy of 0.48, whereas GMean/ChangeExtent (32day horizon) achieves a higher Accuracy of 0.69. Such differences arise because Median is sensitive to the underlying distribution of values, while GMean captures multiplicative interactions across contributing factors, often resulting in more balanced aggregation behavior (Huber and Ronchetti, 2009; Fleming and Wallace, 1986). Detailed per-project results are available in the replication package (Asif et al., 2026). Based on these observations, GMean is selected as the aggregation operator for subsequent analysis. Table 2: Descriptive statistics for GMean/ChangeExtent (32day horizon) and Median/ChangeFrequency (1-day horizon) across all project versions at the 50% minimization budget.
Accuracy
FDR
Statistics
GMean
Median
GMean
Median
Min 25% (Q1) Mean Median 75% (Q3) Max
0.53 0.64 0.72 0.73 0.77 1.00
0.48 0.64 0.73 0.71 0.81 1.00
0.59 0.65 0.75 0.75 0.82 1.00
0.52 0.67 0.76 0.75 0.85 1.00
Across the two change metrics, ChangeExtent consistently produces slightly stronger and more stable results when combined with GMean. Figure 4 shows how performance varies with the temporal horizon for this configuration. Both Accuracy and FDR increase slightly from very short horizons, remain stable across an intermediate range (approximately 4–128 days), and decline beyond this region. Selecting a configuration from within such a stable performance plateau is a principled approach in empirical parameter analysis, as configurations within a flat region yield equivalent and reliable performance (Bergstra and Bengio, 2012; Tantithamthavorn et al., 2016). Based on this behavior, a 32-day horizon is selected for the subsequent analysis, as it lies within the stable performance plateau while representing a practically meaningful develop-
Figure 4: Mean Accuracy and FDR of TRTM (GMean/Change Extent) as a function of temporal horizon at the 50% minimization budget.
ment window that captures recent modification activity across the subject systems. Answer to RQ1: Incorporating temporal modeling into test suite minimization meaningfully influences fault detection effectiveness, with the degree of impact shaped by both the decay horizon and the choice of aggregation operator. Across configurations, shorter temporal horizons consistently achieve higher fault-detection effectiveness, while very long horizons degrade performance as they increasingly resemble static change aggregation. Among the evaluated aggregation operators, GMean provides the most stable behavior across projects, and ChangeExtent yields slightly stronger results than ChangeFrequency (achieving a mean Accuracy of 0.72 and an FDR of 0.75 at the 32-day horizon). 4.5.2
RQ2: Comparison with Baseline Approach
We compare the effectiveness and efficiency of TRTM with the current state-of-the-art static baseline approach CTM, using the best TRTM configuration identified in RQ1. Effectiveness. Table 3 presents the per-project Accuracy and FDR results at the 50% minimization budget. TRTM consistently outperforms CTM across the project set, achieving a mean Accuracy of 0.72 and mean FDR of 0.75, compared to 0.66 and 0.69 for CTM. The improvement is broad-based: TRTM achieves higher or equal Accuracy in 12 of 14 projects and higher or equal FDR in 13 of 14 projects. The minimum Accuracy also improves from 0.42 to 0.53, indicating that temporal modeling reduces worst-case minimization failures across the project set. Several projects exhibit particularly notable improvements, revealing a consistent pattern in how temporal modeling enhances effectiveness. For example, JacksonXml achieves perfect Accuracy and FDR of 1.00, compared to 0.80 for CTM. Similarly, Jsoup and Csv show substantial gains over CTM (e.g.,
Jsoup: 0.42→0.53 Accuracy and 0.48→0.60 FDR; Csv: 0.53→0.69 Accuracy). Similar improvements are observed in Mockito, where FDR increases from 0.79 to 0.86. These improvements indicate that temporal modeling more accurately captures the relationship between recent code changes and fault-proneness, enabling more effective identification of fault-relevant test cases compared to the static baseline. TRTM yields marginally lower Accuracy than CTM only for Cli (0.77 vs. 0.78) and JacksonCore (0.67 vs. 0.71), both of which reflect the limitation of a fixed global horizon — for instance, our extended analysis shows that Cli achieves Accuracy of 0.80 at shorter horizons of 2 and 4 days, suggesting that per-project horizon calibration could further improve performance. Table 3: Accuracy, FDR, and Execution Time of TRTM (GMean/Change Extent, 32-day horizon) and CTM (GMean/Change Extent) across projects at the 50% minimization budget. Bold indicates the higher value per metric per project (for Time, lower is better). Accuracy FDR Time (min) TRTM CTM TRTM CTM TRTM CTM Time 0.73 0.66 0.75 0.71 1.19 1.35 JxPath 0.55 0.55 0.59 0.59 0.35 0.34 Codec 0.77 0.72 0.81 0.75 0.55 0.67 JacksonXml 1.00 0.80 1.00 0.80 0.27 0.44 JacksonDatabind 0.84 0.84 0.86 0.85 1.68 2.06 Gson 0.75 0.68 0.75 0.75 0.73 0.93 Math 0.63 0.53 0.65 0.57 1.83 2.46 Jsoup 0.53 0.42 0.60 0.48 0.44 0.55 Compress 0.78 0.75 0.79 0.77 0.91 1.05 Mockito 0.74 0.68 0.86 0.79 1.18 1.49 Cli 0.77 0.78 0.83 0.83 0.22 0.35 Csv 0.69 0.53 0.69 0.56 0.43 0.58 JacksonCore 0.67 0.71 0.67 0.71 0.45 0.61 Lang 0.61 0.53 0.63 0.57 1.20 1.64 Statistics Min 0.53 0.42 0.59 0.48 0.22 0.34 25% Quartile 0.64 0.54 0.65 0.58 0.43 0.56 Mean 0.72 0.66 0.75 0.69 0.82 1.04 Median 0.73 0.68 0.75 0.73 0.64 0.80 75% Quartile 0.77 0.74 0.82 0.78 1.19 1.45 Max 1.00 0.84 1.00 0.85 1.83 2.46 Project
Statistical analysis at the 50% minimization budget confirms that TRTM yields significant improvements over CTM on both effectiveness metrics. Fisher’s exact test shows a statistically significant improvement in FDR (p < 0.001, OR = 7.27), indicating that TRTM is substantially more likely to retain at least one fault-detecting test case in versions where CTM fails. The Wilcoxon signed-rank test further confirms a statistically significant improvement in Accuracy after Bonferroni correction (p = 0.004, Cliff’s δ = 0.063). While the effect size for Accuracy is small, the improvement is consistent across 631 project versions, indicating a systematic benefit of temporal modeling. Results at 25% and 75% budgets follow consistent
trends and are available in the replication package. Efficiency. TRTM achieves efficient execution while improving effectiveness over CTM. The mean execution time per version is 0.82 minutes for TRTM compared to 1.04 minutes for CTM (median: 0.64 vs. 0.80 minutes), indicating an overall reduction in execution time across projects. These results indicate that incorporating temporal attenuation—a lightweight weighting operation—preserves efficient execution while delivering statistically significant gains in test suite minimization effectiveness. Answer to RQ2: TRTM consistently improves fault detection effectiveness over the state-of-the-art static CTM baseline. At the 50% minimization budget, TRTM achieves a mean Accuracy of 0.72 (vs. 0.66) and an FDR of 0.75 (vs. 0.69), while elevating the minimum Accuracy floor to 0.53 (vs. 0.42) across the project set. These improvements are statistically significant for both metrics after Bonferroni correction (Accuracy: p = 0.004; FDR: p < 0.001, OR = 7.27). In addition, TRTM maintains efficient execution, achieving a lower mean execution time of 0.82 minutes per version compared to 1.04 minutes for CTM.
5
THREATS TO VALIDITY
Construct Validity. Construct validity concerns whether the evaluation metrics capture the intended notion of effectiveness. We use Accuracy and FDR, both widely adopted in prior TSM research (Pan et al., 2023; Pan et al., 2024; Siam et al., 2025), where Accuracy measures the proportion of fault-revealing test cases preserved and FDR captures whether at least one such test case is retained. Our approach relies on change history-based indicators (change frequency and change extent) to estimate class-level risk. These indicators have been extensively used in software evolution and defect prediction studies as proxies for fault-proneness (Arvanitou et al., 2017; Graves et al., 2002). By combining these indicators with temporal modeling and evaluating across multiple configurations and budgets, the assessment captures meaningful variations in effectiveness. Internal Validity. Internal validity concerns whether the observed differences are attributable to the proposed approach rather than experimental artifacts. To ensure a fair comparison, all approaches were evaluated under identical settings, utilizing the same datasets, dependency extraction pipeline, and minimization budgets. While our analysis relies on static test–class dependency extraction—which may miss dynamic behaviors such as reflection or runtime poly-
morphism—this limitation applies uniformly across all evaluated configurations, thereby preserving the validity of the comparative results. Furthermore, TRTM is fully deterministic and does not rely on stochastic optimization components; this architectural choice, combined with a uniform execution environment, guarantees consistent and reproducible results across all experimental runs. Conclusion Validity. Conclusion validity concerns the statistical soundness of the reported results. To support reliable conclusions, we employ non-parametric statistical tests: the Wilcoxon signedrank test for paired comparisons of Accuracy, and Fisher’s exact test for fault detection outcomes (FDR). Effect sizes are reported using Cliff’s δ for Accuracy and odds ratios for FDR to quantify the magnitude of observed differences. These measures help ensure that the reported improvements are statistically supported and not due to random variation. External Validity. External validity concerns the generalizability of the findings beyond the studied context. The evaluation is conducted on 14 Java projects from the Defects4J benchmark, comprising 631 buggy versions. While these projects are open-source and may differ from industrial systems, Defects4J is one of the most widely used benchmarks in software testing research due to its collection of real faults and reproducible test suites (Just et al., 2014), supporting the relevance of our findings to the broader community. Defects4J versions typically contain a single fault per buggy version, whereas real-world systems may involve multiple interacting faults. While this is a limitation, it provides a controlled setting for evaluating fault detection capability and has been widely adopted in prior TSM studies. Additionally, buggy versions in Defects4J are often closely associated with recent changes, reflecting the bursty nature of software development and the temporal proximity between code modifications and fault introduction (Graves et al., 2002; Rahman et al., 2013). This aligns with established software evolution principles, where recently modified components are more likely to be fault-prone. A single global temporal horizon is adopted to enable consistent large-scale analysis, although the observed variation across projects suggests that temporal sensitivity may be project-dependent. Future work can explore adaptive, project-specific horizon selection. Finally, although the study focuses on Java systems, the underlying principles of change history analysis and temporal risk modeling are not language-specific and can be extended to other ecosystems with appropriate tooling, and future work should evaluate the approach on more diverse datasets, including systems
with different development patterns and multi-fault scenarios.
6
CONCLUSIONS
In this study, we addressed a key limitation in change history-based black-box test suite minimization: the uniform, time-agnostic treatment of historical modifications. We introduced Temporal Risk-driven Test Suite Minimization (TRTM), the first approach to incorporate temporal modeling into change historybased TSM to capture the recency of software changes when estimating class-level risk. TRTM utilizes two change history-based indicators, change frequency and change extent, and applies temporal attenuation over configurable time horizons to compute time-weighted class-level risks, which are then aggregated through statistical operators to guide test suite minimization. We evaluated TRTM across 2 × 10 × 4 = 80 configurations, combining two change metrics, ten temporal horizons (1–512 days), and four aggregation operators, on a large dataset consisting of 14 Java projects with 631 buggy versions from Defects4J. The results show that TRTM consistently improves fault detection effectiveness across projects and minimization budgets. At the 50% minimization budget, TRTM achieves a mean Accuracy of 0.72 (vs. 0.66) and FDR of 0.75 (vs. 0.69) compared to the state-of-the-art baseline CTM, with statistically significant improvements while maintaining efficient execution time. Future work can investigate adaptive, projectspecific temporal configurations to further optimize minimization effectiveness. Additionally, while the underlying principles of temporal risk modeling are inherently language-agnostic, our current evaluation focused exclusively on Java systems; therefore, future research should explore the applicability of TRTM across other programming languages. Finally, extending this evaluation to encompass multi-fault datasets and real-world industrial systems remains a crucial next step for establishing broader generalizability.
REFERENCES Ahmadi, E., Elsanhoury, M., Selvan, K., Välisuo, P., and Kuusniemi, H. (2025). Robust multi-sensor fusion positioning based on gnss/imu using factor graph optimization. In 2025 IEEE/ION Position, Location and Navigation Symposium (PLANS), pages 1247–1256. IEEE. Arrieta, A., Wang, S., Markiegi, U., Arruabarrena, A., Etxeberria, L., and Sagardui, G. (2019). Pareto efficient multi-objective black-box test case selection for
simulation-based testing. Information and Software Technology, 114:137–154. Arvanitou, E.-M., Ampatzoglou, A., Chatzigeorgiou, A., and Avgeriou, P. (2017). A method for assessing class change proneness. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, pages 186–195. Asif, K., Siam, M., and Sakib, K. (2026). Temporal modeling of change history for black-box test suite minimization. https://doi.org/10.5281/zenodo.19506050. Bergstra, J. and Bengio, Y. (2012). Random search for hyperparameter optimization. In JMLR. Coviello, C., Romano, S., Scanniello, G., Marchetto, A., Antoniol, G., and Corazza, A. (2018). Clustering support for inadequate test suite reduction. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 95–105. IEEE. Cruciani, E., Miranda, B., Verdecchia, R., and Bertolino, A. (2019). Scalable approaches for test suite reduction. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 419–429. IEEE. Fleming, P. J. and Wallace, J. J. (1986). How not to lie with statistics: The correct way to summarize benchmark results. Communications of the ACM. Graves, T. L., Karr, A. F., Marron, J. S., and Siy, H. (2002). Predicting fault incidence using software change history. IEEE Transactions on software engineering, 26(7):653–661. Hassan, A. E. and Holt, R. (2005). The top ten list: Dynamic fault prediction. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM 2005), pages 263– 272. Hemmati, H., Arcuri, A., and Briand, L. (2013). Achieving scalable model-based testing through test case diversity. ACM Transactions on Software Engineering and Methodology (TOSEM), 22(1):1–42. Huber, P. J. and Ronchetti, E. M. (2009). Robust Statistics. Wiley. Hutter, F., Hoos, H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In LION. Just, R., Jalali, D., and Ernst, M. D. (2014). Defects4j: a database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, page 437–440, New York, NY, USA. Association for Computing Machinery. Khan, S. U. R., Lee, S. P., Javaid, N., and Abdul, W. (2018). A systematic review on test suite reduction: Approaches, experiment’s quality evaluation, and guidelines. IEEE Access, 6:11816–11841. Liu, Y., Wang, K., Wei, W., Zhang, B., and Zhong, H. (2011). User-session-based test cases optimization method based on agglutinate hierarchy clustering. In 2011 International Conference on Internet of Things and 4th International Conference on Cyber, Physical and Social Computing, pages 413–418. IEEE.
Miranda, B. and Bertolino, A. (2017). Scope-aided test prioritization, selection and minimization for software reuse. Journal of Systems and Software, 131:528–549. Mohanty, S., Mohapatra, S. K., and Meko, S. F. (2020). Ant colony optimization (aco-min) algorithm for test suite minimization. In Progress in Computing, Analytics and Networking: Proceedings of ICCAN 2019, pages 55–63. Springer. Nagappan, N. and Ball, T. (2005). Use of relative code churn measures to predict system defect density. In Proceedings of the 27th international conference on Software engineering, pages 284–292. Noemmer, R. and Haas, R. (2019). An evaluation of test suite minimization techniques. In International Conference on Software Quality, pages 51–66. Springer. Pan, R., Ghaleb, T. A., and Briand, L. (2023). Atm: Blackbox test case minimization based on test code similarity and evolutionary search. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1700–1711. IEEE. Pan, R., Ghaleb, T. A., and Briand, L. C. (2024). Ltm: Scalable and black-box similarity-based test suite minimization based on language models. IEEE Transactions on Software Engineering, 50(11):3053–3070. Philip, A. A., Bhagwan, R., Kumar, R., Maddila, C. S., and Nagppan, N. (2019). Fastlane: Test minimization for rapidly deployed large-scale online services. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 408–418. IEEE. Rahman, F., Posnett, D., and Devanbu, P. (2013). How, and why, process metrics are better. In Proceedings of the 2013 International Conference on Software Engineering (ICSE), pages 432–441. IEEE. Rhmann, W., Pandey, B., Ansari, G., and Pandey, D. (2020). Software fault prediction based on change metrics using hybrid algorithms: An empirical study. Journal of King Saud University - Computer and Information Sciences, 32(4):419–424. Emerging Software Systems. Rousseeuw, P. J. and Hubert, M. (2011). Robust statistics for outlier detection. Wiley interdisciplinary reviews: Data mining and knowledge discovery, 1(1):73–79. Siam, M., Mridha, N. F., and Sakib, K. (2025). An exploratory study on the impact of change-proneness as a metric in black-box test suite minimization. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 855–860. IEEE. Tantithamthavorn, C., McIntosh, S., Hassan, A. E., and Matsumoto, K. (2016). Automated parameter optimization of classification techniques for defect prediction models. In Proceedings of the 38th International Conference on Software Engineering (ICSE), pages 321–332. IEEE. Yoo, S. and Harman, M. (2012). Regression testing minimization, selection and prioritization: a survey. Software testing, verification and reliability, 22(2):67–120. Zhang, M., Ali, S., and Yue, T. (2019). Uncertainty-wise test case generation and minimization for cyber-physical systems. Journal of Systems and Software, 153:1–21.