How do Execution Features Improve Statistical Fault Localization?
arXiv:2606.30324v1 [cs.SE] 29 Jun 2026
An Empirical Study Marius Smytzek
Andreas Zeller
CISPA Helmholtz Center for Information Security Saarbrücken, Germany [email protected]
CISPA Helmholtz Center for Information Security Saarbrücken, Germany [email protected]
Abstract—Automated fault localization helps developers find faults in large code bases. Statistical fault localization (SFL) ranks suspicious lines from pass/fail spectra, but line execution alone misses information like data-flow, values, or branch conditions that explain why a failure occurs. This study evaluates whether augmenting SFL with execution features improves localization accuracy and developer-oriented inspection effort. We extract execution features with EFDD for all Tests4Py subjects, train per-subject random forests, map importances to source lines, and combine the resulting weights with established SFL formulas. The evaluation measures referencepatch accuracy, line- and function-level effort, robustness, and feasibility using a confounder-adjusted mixed-effects model, corroborated by paired statistical tests and outcome-neutral quality checks. Index Terms—Fault localization, automated debugging, execution features, diagnosis, dynamic analysis
I. I NTRODUCTION Debugging consumes a significant portion of development effort, requiring developers to identify root causes across thousands of lines of code. Automated fault localization techniques address this by ranking suspicious code locations most likely responsible for failures. Traditional statistical fault localization (SFL) techniques, such as TARANTULA [1], correlate code line execution with test outcomes and have become widely adopted because code locations are directly actionable for developers. However, SFL cannot distinguish among many lines executed in failing tests, passing tests weaken correlations for shared lines, and ranked lists do not explain failure conditions. Our prior work on Execution-Feature-Driven-Debugging (EFDD) empirically demonstrated that execution characteristics beyond line coverage, including definition-use pairs [2], variable value predicates, and scalar-pair relations [3], correlate more strongly with failures than line execution alone. Consider the middle() function (Figure 1), which returns the median of its inputs. When middle(2, 1, 3) fails (returning 1, not 2), its executed lines also appear in passing runs, so SFL ranks Lines 6 and 7 as most suspicious but cannot explain why. Execution features reveal the critical difference: reaching Line 7 while x > y holds occurs only in failing tests, exposing the condition that coverage alone misses. We propose augmenting SFL with execution feature-based weighting. Our method extracts execution features from pass-
■: covered line
1 2 3 4 5 6 7 8 9 10 11 12
def middle(x, y, z): m = z if y < z: if x < y: m = y elif x < z: m = y elif x > y: m = y elif x > z: m = x return m
x y z
3 3 5
1 2 3
3 2 1
5 5 1
2 1 3 1
■ ■ ■ ■ ■ 2 ■ ■ ■ ■ ■ 3 ■ ■ □ □ ■ 4 □ ■ □ □ □ 5 ■ □ □ □ ■ 6 ■ □ □ □ ■ 7 □ □ ■ ■ □ 8 □ □ ■ □ □ 9 □ □ □ ■ □ 10 □ □ □ ■ □ 11 ■ ■ ■ ■ ■ 12 ✔ ✔ ✔ ✔ ✘
Fig. 1. Fault Localization. SFL ranks Lines 6 (correct) and 7 (fault) equally high while missing the condition x > y provided by execution features.
ing and failing test runs, trains a classifier to identify failureindicative features, maps features to source locations, and weights baseline SFL scores by feature importance. The baseline formula remains intact, so the method adds an inspectable weighting signal rather than replacing SFL. It contributes a pre-registered evaluation of whether feature augmentation transfers EFDD’s diagnostic evidence to localization, spanning line- and function-level accuracy, separate reference-patch and test-impact outcomes, and practical utility through diagnostics and quality checks. II. BACKGROUND AND R ELATED W ORK A. Statistical Fault Localization SFL ranks program elements by correlating lines with test outcomes. Classical formulas include Tarantula [4], Ochiai [5], DStar [6], and Naish2 [7], with early evaluations analyzing formula accuracy and inspection effort [8]. SFLKit provides a configurable Python workbench with multiple predicate types, spectra, and formulas [9]. Comparative studies show that relative performance depends on datasets, fault types, and evaluation protocol, with real faults differing from older artificial settings [10], which motivates evaluating feature augmentation across several formulas rather than treating one suspiciousness coefficient as representative.
B. Learning and Reweighting for Fault Localization Prior work improves SFL by reweighting spectra, combining spectra with search or optimization, or learning over program elements. Examples include PageRank-based test differentiation [11], genetic programming and search-based localization [12], constrained feature selection over traces [13], and learned rankings from dynamic, mutation-based, and static features [14]. Our study follows this additive view but differs from prior learned-ranking methods on a critical point. Prior methods learn the ranking itself and effectively replace the suspiciousness formula with a learned model. We keep each baseline formula and only modulate it multiplicatively, use the random forest only for per-subject feature attribution, and keep the added signal inspectable and separable from the baseline. C. Execution Features and Data-Flow Signals Execution features extend line coverage with various dynamic information. For instance, def-use information can strengthen localization by capturing dependencies missed by coverage-only spectra [2]. Predicate-based localization, such as PredFL, shows that branch conditions complement spectrum-based localization [15], and TraPT combines program/test transformations with learning-to-rank [16]. Our prior EFDD work showed that various execution features capture failure conditions that line spectra miss [17], motivating feature weighting that remains compatible with existing SFL pipelines. D. Complementary Debugging Signals Our central claim, that richer execution information improves localization, connects to several established lines. Statistical debugging isolates bugs by correlating instrumented predicates with failures across runs [3], and program slicing narrows attention to statements that dynamically or statically affect a faulty value [18]. Value-based localization similarly reasons over variable states rather than line coverage alone. These approaches change either the granularity (predicates, slices, values) or the inference (correlation, dependence). In contrast, we keep the established SFL ranking intact and add an inspectable feature-importance weight on top of it. Beyond spectra and execution predicates, specification-assisted localization introduces violated specifications [19], while learningbased repair connects localization with patch generation, ranking, and validation [20], reinforcing the need to evaluate localization signals independently before combining them with downstream repair machinery. E. Evaluation Practice and Practical Utility Fault-localization evaluation can overstate improvements through single-fault simplification, idealized oracles, and effort models [21], [22]. Ranked suspiciousness alone does not always help developers [23]. A critical limitation is singlepatch bias, where evaluating only against reference edits can miss broader failure-relevant regions. We therefore complement standard top-k and EXAM, kept for comparability, with function-level aggregation, test-impact coverage, diagnostics,
and outcome-neutral checks, so conclusions do not depend on one reference patch alone. F. Positioning of This Study Existing studies show that reweighting and learning improve rankings and that execution features provide richer signals than line execution alone, yet two gaps remain: whether featureimportance weighting yields robust gains across multiple SFL baselines on real faults, and whether conclusions hold beyond reference-patch-only evaluation. This study addresses both for Tests4Py. Because a positive result is not guaranteed (mapping is heuristic, importances may be unstable, and sparse failing tests may weaken learning), we separate patch-line, function-level, test-impact, and diagnostic outcomes to distinguish true gains from longer functions, frequent execution, or benchmark-patch idiosyncrasies. We scope the study to a controlled paired delta: execution-feature evidence as an additive signal over established SFL, holding code, tests, formula, and metrics fixed. LLM-based localization is a complementary future direction [24], but as a baseline it would confound this delta with prompt design, model priors, and reranking, so measuring the delta is a prerequisite for, not a substitute for, later LLM-integrated localization. III. R ESEARCH Q UESTIONS AND H YPOTHESES A. General Hypotheses HACC (Accuracy Hypothesis). Augmenting SFL with execution-feature weights improves fault localization effectiveness over baseline SFL, because execution features capture failure-relevant behavior that line spectra alone cannot represent. HROB (Robustness Hypothesis). The improvement is robust across different SFL formulas, because execution-feature signals are orthogonal to the suspiciousness formula used by the baseline. B. Research Questions RQ1: Patch-Line Accuracy. Does execution-feature augmentation improve top-k localization metrics (top-1, top-5, top-10) for reference-patch lines in Tests4Py compared with each baseline SFL technique? RQ2: Developer Effort. Does execution-feature augmentation improve or maintain line- and function-level inspection-effort metrics compared with each baseline SFL technique? RQ3: Broader Ground Truth. Do augmented rankings discover failure-relevant locations beyond the reference patch, as measured by dynamically discovered testimpact sets? RQ4: Robustness. Is the improvement direction consistent across all five baseline techniques (TARANTULA, OCHIAI, DSTAR, NAISH2, and GP13) and, exploratorily, across the implemented ranking variants?
IV. VARIABLES , DATASETS , AND S AMPLING A. Independent Variables We vary two factors: localization method and baseline formula (TARANTULA, OCHIAI, DSTAR, NAISH2, and GP13). The localization methods are exactly the unmodified SFL baselines and the primary random-forest feature weighting, plus a clearly separated, exploratory family of ranking variants defined in Section V. B. Dependent Variables Patch-line outcomes (RQ1 for top-k, RQ2 for effort) are top-1, top-5, and top-10 accuracy, EXAM score, and wasted effort (best, average, worst case) against referencepatch lines. Function-level outcomes (RQ2) aggregate rankings at function granularity to reflect developer workflows: function-weighted mass@1/3/5, faulty-function probability mass, function-weighted inspection effort, and practical effort until reaching a faulty function. Test-impact outcomes (RQ3) evaluate against both reference patch hunks and dynamically discovered test-impact sets: impact-found@1/5/10, minimum impact rank, and the total number of impact locations ranked. Exploratory ranking-variant outcomes (RQ4, exploratory) apply the same patch-line and test-impact metrics to the ranking variants of Section V. Diagnostic metrics are the interpretive quantities enumerated in Section VII (importance concentration, entropy, skewness, dropping, mapping conflicts, and weak-signal counts). We additionally report feasibility outcomes for runtime, timeouts, exclusions, and trace/storage failures. Ties in ranking are handled through repeated randomized tie-breaking and averaged per subject. C. Confounding Variables and Controls The confounders we consider are exactly bug difficulty, project size, executable-line count, test-suite size, number of failing tests, coverage density, and the chosen SFL formula. Our design is within-subject paired: for each bug, the baseline and the augmented method run on identical code, tests, and spectra, so every time-invariant per-bug characteristic in this list takes the same value for both methods and does not shift the average paired contrast. Pairing does not make these factors irrelevant, however; they govern how large the augmentation effect is and where it concentrates, so we model them rather than set them aside. Our primary analysis is therefore the confounder-adjusted mixed-effects model of Section VII, in which the method indicator is the fixed effect of interest, these confounders enter as covariates, the SFL formula enters as a fixed factor, and random effects per project and subject absorb residual between-cluster variation. Line- and function-level inspection effort is additionally normalized by the number of executable lines and functions, respectively, so the covariate reflects difficulty rather than raw size. A perproject random slope for the method effect estimates effect heterogeneity directly, and we report per-project distributions together with the diagnostics in Section IV to show where sparse failing evidence or feature dropping drives weak or negative effects. The distribution-free paired tests retain their
role as assumption-light corroboration of this model. We keep these metric families separate, rather than collapsing them into one score, because a method may improve line-level ranking while degrading effort or broader coverage. D. Dataset, Sampling, and Unit of Analysis We evaluate all 310 reproducible Tests4Py bugs without subsampling because Tests4Py provides executable Python faults with tests and reference fixes, and using the complete set avoids selection bias. Each bug is one analysis unit. E. Operational Definitions Reference faulty lines are the exact lines modified in the official benchmark fix. Patch-hunk impact sets contain the corresponding modified hunk regions. For function-level outcomes, every function containing at least one reference faulty line or patch-hunk line is treated as faulty so that multilocation patches may yield multiple faulty functions. Testimpact sets use observed line spectra. A line is included if at least one failing test observes it and either no passing test observes it or its failing-observation ratio is at least 0.6. If this yields an empty set, we use all failing-observed lines. We choose 0.6 because a line observed in a clear majority of failing tests is more plausibly failure-relevant. At the same time, lower thresholds admit incidental lines and higher ones discard relevant lines exercised by only some failing tests. We vary it over the fixed grid {0.5, 0.6, 0.7, 0.8} as a preregistered sensitivity analysis. We keep patch-hunk and testimpact sets distinct so that an improvement against reference edits is not automatically read as improved coverage of failurerelevant behavior. V. M ETHODOLOGY Our core approach extracts execution features, trains a random forest to identify failure-correlated features, maps feature importances to source locations, and boosts baseline SFL scores with feature-derived weights. A. Feature Extraction A feature is an EFDD execution-derived signal encoded as a binary or tertiary variable. In line with the EFDD feature taxonomy, features are drawn from coverage, value, condition, and exception families, including line/branch/function/loop execution, def-use pairs, variable-value and length properties, scalar-pair relations, condition outcomes, and exception exits [17]. We use EFDD to instrument Python code and generate one labeled feature vector per passing or failing test execution. EFDD aggregates an entire run into a single vector recording the presence and values of these features, matching the run’s pass/fail label. One labeled vector per execution is the natural unit at test-outcome granularity, and a passing and a failing vector suffice to compute feature differences, so no within-run subdivision is required. Unobserved features receive their fixed default value, requiring no distributional assumptions. Constant within-subject features are excluded before primary random-forest training and reported as featuredropping diagnostics.
B. Random Forest Model Training For each subject, we train a 200-estimator scikit-learn random forest (RF) with Gini splits on all available labeled vectors after constant-feature removal. We do not tune hyperparameters per subject. All other random-forest parameters remain at their defaults except balanced class weights to account for pass/fail imbalance when a subject has few failing tests, and random operations use master seed 42 or deterministic seeds derived from it. We use a random forest because execution features are sparse, correlated, and discriminative only in combination: it captures non-linear interactions and yields importance scores that map back to source locations. The RF is a per-subject feature-attribution device, not a predictor of unseen data: we extract failure-associated importances within one known faulty revision and read them as per-line weights modulating an unchanged SFL formula, so classical overfitting to unseen executions is not the central validity concern. We therefore train per subject with no train/test split; standard SFL is itself transductive, computing suspiciousness from the same pass/fail spectra, so a held-out split would evaluate the augmentation on a different footing than its baseline. The design also resists spurious importance: bagging and per-split feature subsampling reduce variance, all features are binary or tertiary so the impurity-importance bias toward high-cardinality features does not apply, and the negative control (Section VIII) would expose any gain that is merely a fitting artifact. The model thus estimates failureinformative weights rather than final ranks, keeping the result an SFL ranking augmented by execution evidence, not a replacement. C. Feature Importance Extraction and Mapping We extract scikit-learn feature importances and map features to lines by semantics, strictly following the mapping of EFDD [17], for instance, line features map to their own line, def-use pairs map to both definition and use lines, or value predicates map to definition lines. These mapping rules are fixed before execution and applied uniformly across all subjects and formulas. If one feature maps to multiple source lines, the same feature importance is assigned to each mapped line before aggregation. Raw feature importances are minmax normalized to [ε, 1] with ε = 10−3 as wf = ε + (1 − ε)(I(f ) − Imin )/(Imax − Imin ). If all importances are equal, we use Imin = 0 and Imax = 1. If multiple features map to a line, we take the maximum normalized weight, which prevents many low-value features from inflating a line while still letting one strong failure-indicative feature affect the ranking. Lines without mapped evidence receive the lower bound ε. We record mapping conflicts and feature dropping as diagnostics and relate conflict rates to performance changes. D. Score Boosting We first compute suspw (l) = susp′ (l)×w(l), where susp′ (l) is the original SFL score and w(l) the feature weight, normalized within each subject and baseline so formula scale does not drive the blend. The primary RF ranking then uses
susp(l) = (1 − λ) norm(susp′ (l)) + λ norm(suspw (l)) with a priori λ = 0.35. Thus 1 − λ = 0.65 keeps baseline SFL the dominant term, while λ is still large enough for feature evidence to reorder lines where the signals agree. A smaller value would make the feature signal inert, and a larger one would override the established formula. As a pre-registered sensitivity analysis, we recompute the primary metric over the fixed grid λ ∈ {0.15, 0.25, 0.35, 0.45, 0.55}. This analysis is exploratory and cannot redefine the primary conclusion. E. Hybrid and Adaptive Ranking Variants Beyond the primary random-forest weighting, we report a small, clearly-exploratory family of ranking variants: a hybrid fusion of the available execution signals, a fail-first variant that prioritizes discriminative failing evidence, and an adaptive variant that reweights samples under class imbalance or sparse failing tests. These variants reuse the same benchmark, mapping, and metric protocol and serve only as interpretation aids. They are corrected for multiplicity separately and cannot substitute for the primary RF result. We defer a fully specified learned-fusion mechanism to future work. VI. E XECUTION P LAN The study runs in four phases: Phases 1–2 produce the rankings for RQ1, RQ2, and RQ4, Phase 3 constructs the ground-truth sets for RQ3, and Phase 4 computes metrics and analyses for all research questions. A. Phase 1: Feature Extraction and Model Training For each Tests4Py subject, we run all tests with EFDD instrumentation using a 120-second timeout per test, convert traces into labeled feature vectors, verify the expected binary or tertiary encoding, train one random forest per subject, and save models and feature vectors so a ranking can be reproduced without rerunning instrumentation. B. Phase 2: Feature Weighting and Source Line Ranking We extract importances, map them to source locations using Section V, aggregate by maximum importance, normalize weights to [ε, 1], and verify every executable line receives a weight. For each subject, we compute TARANTULA, OCHIAI, DSTAR, NAISH2, and GP13 scores, generate feature-augmented rankings, and average metrics over 10,000 randomized tiebreaking simulations seeded from master seed 42, so arbitrary source order does not dominate cases of equal suspiciousness. C. Phase 3: Expanded Ground-Truth Discovery For each subject, we construct reference patch-hunk sets from official patches and test-impact sets using the lineobservation rule from Section IV. Test-impact discovery is recorded separately from the patch-hunk set, allowing the analysis to report when a method succeeds only on the reference edit, only on broader impact locations, or on both.
TABLE I A NALYSIS TIERS . Tier
Analysis / outcome
Primary Primary Primary
Confounder-adjusted mixed-effects model Reference-patch top-1/5/10 accuracy Reference-patch EXAM, wasted effort
RQ1–RQ4 RQ1 RQ2
RQ
Secondary Secondary Secondary Secondary
Distribution-free paired tests Function-level effort Impact-found@k, min. rank Direction across 5 baselines
RQ1–RQ3 RQ2 RQ3 RQ4
Exploratory Exploratory Exploratory
Ranking variants λ/aggregation sensitivity Importance diagnostics
RQ4 n/a n/a
D. Phase 4: Metric Calculation and Analysis We calculate the metrics in Section IV and store them by subject, baseline, method, and metric type, alongside trained models and feature vectors, supporting paired tests, projectlevel analysis, and independent recomputation. VII. A NALYSIS P LAN Directional expectations are positive reference-patch top-k gains for RQ1, no systematic degradation in line- or functionlevel effort for RQ2, at least comparable test-impact coverage for RQ3, and positive direction across most baselines for RQ4. Table I separates the primary analysis from the secondary and exploratory and maps each to its research question. A. Primary and Exploratory Analyses The primary method is random-forest feature weighting, compared against each of the five unmodified SFL formulas. The primary outcomes are the five reference-patch metrics: top-1, top-5, and top-10 accuracy, EXAM score, and averagecase wasted effort, each computed after randomized tie breaking and averaged per subject. For a given baseline formula, we call the augmented method superior if the confounderadjusted model’s method coefficient is an improvement with a non-negligible standardized effect (|d| ≥ 0.2) on at least four of these five metrics and degrades none of them by a non-negligible effect. We interpret HACC as supported only if RF weighting is superior in this sense for the majority of the five baseline formulas. We interpret HROB as supported only if this superiority holds in all five baseline formulas, confirmed by the method×formula interaction and a sign test. The exploratory ranking variants form a family with separate multiplicity control. If the primary comparison is negative or mixed, positive variant results are treated as exploratory evidence about alternative aggregation, not as support for the primary claim. B. Primary Analysis for RQ1, RQ2, and RQ3 The primary inference for every outcome is a single confounder-adjusted (generalized) linear mixed-effects model fit over all subjects, baselines, and paired method conditions. Its fixed effects are the method indicator (augmented vs. baseline, the effect of interest), the SFL formula as a factor, their interaction, and the confounders of Section IV as covariates; random intercepts for project and for subject-within-project
and a per-project random slope for the method effect estimate effect heterogeneity directly rather than averaging it away. Continuous metrics (EXAM, wasted effort, function-level effort) use a Gaussian model; binary per-subject outcomes (top-k, impact-found@k) a binomial logit. We report the method coefficient with 95% confidence intervals and standardize it for the superiority rule, read HROB from the interaction, and use the confounder coefficients and random-slope variance to characterize where and why the effect varies. We apply Benjamini–Hochberg correction to the method coefficients within the reference-patch top-k, effort, and test-impact families, reported alongside improve/tie/degrade counts per RQ. With 310 subjects per formula, the design has about 80% power to detect a standardized method effect of d ≈ 0.16, so smaller effects are not treated as strong practical evidence. To corroborate the model under minimal distributional assumptions, we re-test each contrast with distribution-free paired procedures: paired permutation tests for top-k and impact-found@k proportions with paired mean differences and 95% bootstrap confidence intervals, and Wilcoxon signedrank tests with rank-biserial correlations for EXAM, wasted effort, function-weighted EXAM, faulty-function mass, practical effort, and minimum impact rank. Patch-hunk and testimpact sets are analyzed separately. Hypothesis support rests on the model and the four-of-five superiority rule rather than on any single corrected p-value, and negligible significant effects are treated as weak practical evidence. Exploratory hybrid/adaptive tests are corrected separately and reported after the primary results. C. Robustness Analysis for RQ4 For the primary RF method, we read the per-formula method effects and the method×formula interaction from the model and determine for each baseline formula whether the augmented method is superior under the four-of-five metric rule defined above. We require this superiority in all formulas for HROB and confirm that the positive direction dominates with a sign test. We report improve/tie/degrade direction counts per formula. Exploratory variants are summarized with the same direction counts. D. Practical Feasibility We report medians, interquartile ranges, and per-project distributions of per-phase and total runtime, together with timeouts, exclusions, and trace/storage failures, to judge whether the method is plausible for offline debugging. Subjects affected by feasibility failures are counted explicitly and not silently removed from denominator summaries. E. Exploratory Analyses To interpret why methods improve or degrade, we analyze feature-category contribution, project-level variation, per-baseline sensitivity, constant-feature fraction, importance concentration/entropy/skewness, failing-vector counts, and feature-to-location mapping-conflict rates. We do not perform manual relabeling of Tests4Py, so all diagnostics are computed
automatically from artifacts. We also report the correlation matrix among dependent variables so that redundant metrics are not over-interpreted. The fixed sensitivity checks vary the lower-bound weight ε ∈ {10−4 , 10−3 , 10−2 }, the blend parameter λ ∈ {0.15, 0.25, 0.35, 0.45, 0.55}, and the aggregation operator (mean and sum instead of max). Where at least three passing and three failing vectors exist, repeated seeded and bootstrap refits quantify feature-importance stability and flag subjects whose rankings may reflect noisy per-subject models.
or unstable importance mass. Feature-to-line mapping uses semantic but heuristic rules, so alternative mappings could change rankings, most of all for features spanning multiple locations such as def-use pairs and scalar relations. We mitigate this by documenting the rules, recording conflicts, evaluating function-level outcomes that are less brittle to mapping differences, and running an exploratory alternative mapping that assigns def-use and scalar-relation importance to the use line only.
F. Handling of Edge Cases and Missing Data
B. External Validity
Subjects enter paired analysis if at least one passing and one failing feature vector remain, which is the minimum needed to compute pass-versus-fail feature differences. They are excluded only if the subject cannot be executed, feature extraction fails (e.g., from the 120-second per-test timeout), or one outcome class is absent. We impose no higher failing-test threshold, as that would select against sparse but realistic subjects. We accept partial feature loss, constant-feature removal, and trace-size anomalies unless they make the subject unanalyzable. All exclusions and deviations will be documented. VIII. O UTCOME -N EUTRAL C HECKS A. Data Quality Checks Before interpreting results, we verify that labeled feature vectors, binary/tertiary encodings, executable-line weights, and patch-hunk/test-impact records are evaluable. If a check fails, we record the affected subject and either repair the dataprocessing step or exclude it from paired analysis with an explicit reason, so implementation failures are not interpreted as localization performance. B. Sanity Checks on Results We add a perfect pass/fail feature as a positive control, replace learned importances with random scores as a negative control, retrain 10 randomly selected subjects with the same seed to verify identical metrics, and inspect paired differences for malformed subjects. The positive control checks that the pipeline propagates known-fault evidence to the ranking, and the negative control (our primary guard against spurious importance) checks that improvement is not merely an artifact of perturbing suspiciousness scores.
Tests4Py and EFDD are Python-specific, so other languages require adapted instrumentation. Using all 310 subjects maximizes within-benchmark diversity, but curated faults with reproducible tests and mostly isolated fixes may not transfer to flakiness, partial coverage, or multiple faults. Subjects with few failing executions may provide weak evidence for learning failure-indicative features; we capture this through feature-dropping, weak-signal, and project-stratified analyses that show whether effects are broad or concentrated. The SFL baselines are established but do not represent all learning- or specification-based localization approaches. C. Construct Validity Top-k, EXAM, and wasted effort inherit single-patch bias when used alone, which the separate function-level, testimpact, and diagnostic families mitigate. Official fixes define only one reference solution, and test-impact sets are dynamic approximations that depend on the executed tests, so we read them as complementary rather than definitive evidence. Bug difficulty varies with locality and coupling, which paired within-subject comparisons reduce. X. R EPRODUCIBILITY AND DATA AVAILABILITY We will publish source code, data, trained models, analysis scripts, and documentation under the Apache 2.0 license as a repository and long-term archival artifact (e.g., Zenodo). XI. C ONCLUSION
We use a master seed (42) for reproducible derived seeds, document hyperparameters and constants, rerun to verify identical results, and publish all derived seeds, configurations, and results for every phase with the final manuscript.
This pre-registered study tests whether execution-feature weighting improves statistical fault localization on 310 Tests4Py Python bugs, that is, whether failure-indicative EFDD features transfer from diagnosis to line-level ranking across SFL baselines. Its expected contribution is evidence on when richer execution signals improve localization beyond line spectra, with reproducible artifacts for future debugging research.
IX. T HREATS TO VALIDITY
ACKNOWLEDGMENTS
C. Replicability Checks
A. Internal Validity Using EFDD reduces custom-instrumentation risk but may miss feature types outside its scope, and other learners or hyperparameters might behave differently. We treat feature importances as ranking evidence rather than a causal explanation of the fault, and use diagnostics to expose concentrated
This work is funded by the European Union (ERC S3, 101093186). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
R EFERENCES [1] J. A. Jones, M. J. Harrold, and J. Stasko, “Visualization of test information to assist fault localization,” in Proceedings of the 24th International Conference on Software Engineering. New York, NY, USA: ACM, 2002, pp. 467–477. [Online]. Available: https://doi.org/10.1145/581339.581397 [2] R. Santelices, J. A. Jones, Y. Yu, and M. J. Harrold, “Lightweight fault-localization using multiple coverage types,” in Proceedings of the 31st International Conference on Software Engineering, ser. ICSE ’09. USA: IEEE Computer Society, 2009, p. 56–66. [Online]. Available: https://doi.org/10.1109/ICSE.2009.5070508 [3] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan, “Scalable statistical bug isolation,” SIGPLAN Not., vol. 40, no. 6, p. 15–26, jun 2005. [Online]. Available: https://doi.org/10.1145/1064978.1065014 [4] J. A. Jones and M. J. Harrold, “Empirical evaluation of the Tarantula automatic fault-localization technique,” in Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’05. New York, NY, USA: Association for Computing Machinery, 2005, p. 273–282. [Online]. Available: https://doi.org/10.1145/1101908.1101949 [5] R. Abreu, P. Zoeteweij, and A. J. C. v. Gemund, “An evaluation of similarity coefficients for software fault localization,” in Proceedings of the 12th Pacific Rim International Symposium on Dependable Computing, ser. PRDC ’06. USA: IEEE Computer Society, 2006, p. 39–46. [Online]. Available: https://doi.org/10.1109/PRDC.2006.18 [6] W. E. Wong, V. Debroy, Y. Li, and R. Gao, “Software fault localization using DStar (D*),” in 2012 IEEE Sixth International Conference on Software Security and Reliability, 2012, pp. 21–30. [7] L. Naish, H. J. Lee, and K. Ramamohanarao, “A model for spectra-based software diagnosis,” ACM Trans. Softw. Eng. Methodol., vol. 20, no. 3, aug 2011. [Online]. Available: https://doi.org/10.1145/2000791.2000795 [8] R. Abreu, P. Zoeteweij, and A. J. van Gemund, “On the accuracy of spectrum-based fault localization,” in Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007), 2007, pp. 89–98. [9] M. Smytzek and A. Zeller, “SFLKit: A workbench for statistical fault localization,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 1701–1705. [Online]. Available: https://doi.org/10.1145/3540250.3558915 [10] S. Pearson, J. Campos, R. Just, G. Fraser, R. Abreu, M. D. Ernst, D. Pang, and B. Keller, “Evaluating and improving fault localization,” in Proceedings of the 39th International Conference on Software Engineering, ser. ICSE ’17. IEEE Press, 2017, p. 609–620. [Online]. Available: https://doi.org/10.1109/ICSE.2017.62 [11] M. Zhang, X. Li, L. Zhang, and S. Khurshid, “Boosting spectrumbased fault localization using PageRank,” in Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2017. New York, NY, USA: Association for Computing Machinery, 2017, p. 261–272. [Online]. Available: https://doi.org/10.1145/3092703.3092731 [12] X. Xie, F.-C. Kuo, T. Y. Chen, S. Yoo, and M. Harman, “Provably optimal and human-competitive results in sbse for spectrum based fault localisation,” in Proceedings of the 5th International Symposium on Search Based Software Engineering - Volume 8084, ser. SSBSE 2013. Berlin, Heidelberg: Springer-Verlag, 2013, p. 224–238. [Online]. Available: https://doi.org/10.1007/978-3-642-39742-4_17 [13] T.-D. B. Le, D. Lo, and M. Li, “Constrained feature selection for localizing faults,” in Proceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), ser. ICSME ’15. USA: IEEE Computer Society, 2015, p. 501–505. [Online]. Available: https://doi.org/10.1109/ICSM.2015.7332502 [14] Y. Kim, S. Mun, S. Yoo, and M. Kim, “Precise learn-to-rank fault localization using dynamic and static features of target programs,” ACM Trans. Softw. Eng. Methodol., vol. 28, no. 4, oct 2019. [Online]. Available: https://doi.org/10.1145/3345628 [15] J. Jiang, R. Wang, Y. Xiong, X. Chen, and L. Zhang, “Combining spectrum-based fault localization and statistical debugging: An empirical study,” in Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’19. IEEE Press, 2019, p. 502–514. [Online]. Available: https://doi.org/10.1109/ASE.2019.00054
[16] X. Li and L. Zhang, “Transforming programs and tests in tandem for fault localization,” Proc. ACM Program. Lang., vol. 1, no. OOPSLA, Oct. 2017. [Online]. Available: https://doi.org/10.1145/3133916 [17] M. Smytzek, M. Eberlein, L. Grunske, and A. Zeller, “How execution features relate to failures: An empirical study and diagnosis approach,” ACM Trans. Softw. Eng. Methodol., Dec. 2025, just Accepted. [Online]. Available: https://doi.org/10.1145/3783989 [18] E. Soremekun, L. Kirschner, M. Böhme, and A. Zeller, “Locating faults with program slicing: an empirical analysis,” Empirical Softw. Engg., vol. 26, no. 3, may 2021. [Online]. Available: https://doi.org/10.1007/s10664-020-09931-7 [19] D. Gopinath, R. N. Zaeem, and S. Khurshid, “Improving the effectiveness of spectra-based fault localization using specifications,” in Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’12. New York, NY, USA: Association for Computing Machinery, 2012, p. 40–49. [Online]. Available: https://doi.org/10.1145/2351676.2351683 [20] Q. Zhang, C. Fang, Y. Ma, W. Sun, and Z. Chen, “A survey of learning-based automated program repair,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 2, dec 2023. [Online]. Available: https://doi.org/10.1145/3631974 [21] F. Steimann, M. Frenkel, and R. Abreu, “Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators,” in Proceedings of the 2013 International Symposium on Software Testing and Analysis, ser. ISSTA 2013. New York, NY, USA: Association for Computing Machinery, 2013, p. 314–324. [Online]. Available: https://doi.org/10.1145/2483760.2483767 [22] E. Soremekun, L. Kirschner, M. Böhme, and M. Papadakis, “Evaluating the impact of experimental assumptions in automated fault localization,” in Proceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 159–171. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00025 [23] C. Parnin and A. Orso, “Are automated debugging techniques actually helping programmers?” in Proceedings of the 2011 International Symposium on Software Testing and Analysis, ser. ISSTA ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 199–209. [Online]. Available: https://doi.org/10.1145/2001420.2001445 [24] S. Kang, G. An, and S. Yoo, “A quantitative and qualitative evaluation of llm-based explainable fault localization,” Proc. ACM Softw. Eng., vol. 1, no. FSE, jul 2024. [Online]. Available: https://doi.org/10.1145/3660771