arXiv:2606.11834v1 [cs.SE] 10 Jun 2026
How Requirements Quality Makes (or Breaks) Traceability Link Recovery Tobias Hey∗
Julian Frattini∗
Karlsruhe Institute of Technology (KIT) Karlsruhe, Germany [email protected]
Chalmers University of Technology and University of Gothenburg Gothenburg, Sweden [email protected]
Abstract—Traceability information between requirements and source code greatly benefits the maintenance of a software system. Since manually establishing trace links is cumbersome and errorprone, previous research explored automated traceability link recovery (TLR) approaches to support this task. However, quality defects in requirements impact subsequent activities such as TLR, yet evidence about this remains scarce. Our objective is to contribute empirical evidence on this impact. At the same time, we aim to understand how the performance of TLR approaches varies given these quality defects. To this end, we annotated 28 types of quality defect in 189 use case descriptions from two datasets. Then, we executed five distinct TLR approaches on the dataset and measured their performance in recovering trace links. Finally, we performed statistical tests to quantify the defects’ effect strength on this performance. Our results show that some quality defects harm TLR performance, e.g., sentences that do not start with noun phrases, while others actually benefit performance, e.g., use cases that include implementation details. Moreover, different types of approaches respond differently to these defects. As a consequence, the performance-optimizing choice of a TLR approach depends on the quality of the dataset. Index Terms—traceability link recovery, requirements quality, quality defects, use case descriptions
I. I NTRODUCTION Trace links (TLs) connecting requirements to source code artifacts make explicit where a requirement is implemented. These links greatly support the maintenance of a software system: For example, if a requirement changes, the TLs locate which parts of the implementation need to be changed accordingly [1]. However, establishing TLs requires significant effort, is error-prone, and is rarely done upfront. Hence, automated traceability link recovery approaches have become a popular avenue of research [2]. Recent approaches at automated traceability link recovery (TLR) have suggested that the quality of requirements artifacts may affect the performance of TLR [3], [4]. For example, as illustrated in Figure 1, sentences starting without a noun phrase (i.e., lacking a semantic subject) may negatively influence the TLR by not providing sufficient information on the actor of this activity. The part that is on an inconsistent level of abstraction, however, describes system-internal communication and not user-system interaction. This additional implementation detail might positively influence the TLR despite being ∗ Both authors contributed equally to this study.
Use case name: InsertFeedback Description: Inserts a feedback for the selected site. Participating actor: initialized by Tourist Entry conditions: The tourist card is in a particular site. Flow of events User System: 1. Activate the feature on the issue of feedback. 2. Fill out the form, selecting one vote and inserting a comment, then submit. 3. Confirm the issue of feedback and insert the selected site in the list of sites visited. Exit conditions: The system shall notify the successful combination of feedback to the site. Fig. 1. Use case description adapted from the eTour dataset with quality defects Starts without Noun Phrase and Inconsistent Level of Abstraction
commonly considered a defect [5], as the semantic gap to bridge between the artifacts is smaller. While these assumptions seem plausible, we lack both a confirmation if this effect is true and a quantification of how strong it is. This gap matters for several reasons. From a requirements quality perspective, the lack of confirmation and quantification relegates the decision whether or not to address these alleged quality defects to mere guesswork instead of an evidence-based tradeoff. If it were clear which quality defects have an actual, significant impact on TLR, allocating resources towards removing them was justified. From an automated TLR perspective, the lack of confirmation and quantification obscures the eligibility and potential performance of TLR algorithms. If it were clear which quality defects have an actual, significant impact on TLR, an organization could decide which TLR algorithm fits the current status of their requirements best, what performance to expect, and which steps are necessary to improve it. Reaching these two goals requires empirical evidence about the effect of requirements quality on TLR. We contribute the first step towards these goals via this empirical study answering the following research questions: RQ1: Which factors of requirements quality impact the performance of automated traceability link recovery? RQ2: Do different approaches for automated traceability link recovery respond differently to these factors?
©2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. DOI: (to be added once issued)
To this end, we annotated 28 different types of requirements quality defects in 189 use case descriptions from two different datasets. After executing five automated TLR approaches to recover the TLs on all use case descriptions, we statistically determined whether any of these defects impact the performance of the TLR approaches. The resulting set of statistically significant defects indicates which defects are relevant to TLR performance and whether an approach moderates their effect. Thus, the contributions of this work are as follows: C1: A dataset of requirements quality defects in the use case descriptions of two projects. C2: An observational study on the impact of these quality defects on requirements to code traceability link recovery. II. R ELATED W ORK Our work applies insights from requirements quality research to traceability link recovery between requirements and code, briefly introduced in Sections II-A and II-B. A. Requirements Quality Requirements engineering is a means-to-an-end, i.e., it has the purpose of “support[ing] the stakeholders in whatever activities they’re performing in the project” [6], including the activity of TLR. As means-to-an-end, the quality of a requirements artifact is determined by its impact on subsequent activities [7]. This activity-based perspective on requirements quality properly frames requirements quality [8]: Only if a requirements quality defect has an impact on the attributes of a subsequent activity [9] can it be considered an actual defect. For example, if using passive voice in a natural language requirements sentence produces a significant difference in the completeness of the subsequent modeling activity, then it can be considered a defect [10]. On the other hand, if an alleged quality defect does not have a negative impact on any subsequent activity then this defect is negligible. Recent roadmaps in requirements quality research have called for more empirical evidence on the relationship between requirements quality defects and potentially impacted activities [8], [6]. Since some alleged quality defects have been shown to have a positive impact on some activities [7], recent publications suggested to refer to these with the more neutral term requirements quality factors instead [11], which we adopt throughout this manuscript. B. Traceability Link Recovery Automated traceability link recovery between requirements and code has been studied since the 1990s, with early breakthroughs using information retrieval (IR) techniques. Methods such as vector space models (VSMs) [12], latent semantic indexing (LSI) [13], and latent Dirichlet allocation (LDA) [14] identify links based on textual similarity. To improve performance, researchers have combined multiple IR techniques [15] and incorporated developer feedback with transitive linking, as seen in COMET [16]. Another line of research investigated structural dependencies within source
code, such as call and inheritance relationships, to further improve results [17], [18]. Gao et al. [19], [20] utilize consensual biterms to refine TLR, enriching artifacts with term pairs extracted from both requirements and code. This approach improves performance with models like VSM and LSI. Hey et al. [21] demonstrate that a finer-grained analysis of artifacts can improve TLR. Their approach, FTLR, establishes links at the sentence level for requirements and the method level for code, rather than using entire documents. FTLR captures the semantics of these smaller units through word embeddings. Additionally, they highlight the importance of selecting relevant information in the requirements to enhance performance and show that this selection can be automated using an LLM-based classifier [3]. Despite these advances, traditional techniques struggle with semantically related but textually dissimilar artifacts. Recent approaches leverage machine learning (ML) and large language models (LLMs), employing recurrent neural networks (RNNs) [22], ranking models [23], active learning [24], selfattention mechanisms [25], fine-tuned LLMs [26], graph neural networks [27], and prompting techniques [28], [29], [30], [31] to bridge the semantic gap. However, all but the promptbased approaches require initial links of a project to be available to train the approach. Fuchß et al. [29] show with their LiSSA framework that the combination of embedding-based retrieval as a candidate filter for a following LLM-based classification can be beneficial for requirements to code TLR. Most of these approaches do not consider the influence of requirements quality on TLR or treat it as probabilistic noise [16]. To our knowledge, only the work by Vogelsang et al. [4] pursues a goal similar to ours. They evaluate traceability link recovery performance of two LLMs on five small, handcrafted software systems, each providing both a clean and a defective version of the requirements specification, where the defective version was created by manually introducing five manually seeded defect types. However, the authors note that these small systems (14-25 requirements and 141-220 LOC) may not be representative of real-world software, and that the deliberately selected defect types may not reflect those that occur in requirements specifications. Therefore, they call for further empirical evidence. Our study responds to this call by providing complementary evidence under a different setup designed to address these limitations. III. M ETHOD TLR is valuable yet difficult to establish manually, and hence, benefits from automated support. The quality of requirements artifacts likely influences the performance of these TLR approaches [3]. In this study, we aim to identify which factors of requirements quality affect TLR and how strongly. We identified occurrences of requirements quality factors in use case descriptions and observed their impact on the performance of several automated TLR approaches.
Fig. 2. Overview of the phases of this study
TABLE I OVERVIEW OF THE USED TRACEABILITY LINK RECOVERY DATASETS ILLUSTRATING THE NUMBER OF REQUIREMENTS (R EQ .), EXTRACTED SENTENCES (S ENT.), CODE FILES AND TRACE LINKS (R EQ . TO C ODE ) Number of Artifacts Dataset
Domain
SLOC
Req.
Sent.
Code
TLs
eTour iTrust
Tourism Healthcare
12.4k 14.6k
58 131
266 333
116 226
308 286
189
599
342
594
Total:
Figure 2 visualizes the steps and phases in this study. The group labels correspond to the respective sections and subsections in Section III. A. Data Collection Answering the RQs required a dataset with (1) requirements artifacts in a comparable, textual format, (2) source code artifacts, and (3) manually established trace links between them. The latter served as the gold standard against which we evaluated the performance of a TLR approach. 1) Dataset Selection: We aimed to gather a dataset that maximizes heterogeneity while also remaining reasonably comparable, i.e., in which requirements are specified in a comparable syntax. To this end, we chose two datasets consisting of use case descriptions linked to source code commonly used in TLR research: eTour and iTrust (see Table I). Both were provided by the Center of Excellence for Software & Systems Traceability (CoEST) and map English use case descriptions to Java source code. eTour includes use case descriptions following a template that demands the name, description, preand post-conditions, potential quality requirements, and the actual flow of events. In eTour, entire use case descriptions are linked to the respective source code files that implement them. In iTrust, the individual subflows of a use case description are mapped to the respective source code files. As the dataset of iTrust did not include information on the use case name
the subflow belongs to, we recovered that information from a version of the dataset used by Bencharrada et al. [42]. While these datasets may still not fully represent industrygrade software systems, they are at least several orders of magnitude more realistic in size than the systems previously studied [4]. 2) Variable Selection: Next, we determined the quality of each use case description. We sampled requirements quality factors with an alleged impact on subsequent activities from the requirements quality factor ontology [11], an existing collection of such factors. We selected all quality factors that apply to use cases, requirements, sentences, and phrases, which resulted in 66 potentially relevant factors. Then, we jointly decided for each quality factor whether we hypothesize a potential (positive or negative) impact on the performance of TLR, the response variable of interest in our study. These decisions were based on the authors’ domain expertise in requirements engineering and traceability, as well as discussion among the authors until agreement was reached. The following examples illustrate our decisions: • We included the quality factor Tangled Requirements, i.e., that a use case should not describe several requirements or different functionalities [33]. We assume that tangled requirements make the recovery of all TLs more difficult and, thus, negatively impact the TLR performance. • We included the quality factor Free of UI Design Details, i.e., that a requirement should not impose on the UI design [7]. While a violation of that factor is commonly considered a defect as it imposes on the solution space [43], we assume a positive impact of UI design details, since a TLR approach might recognize these design details easier in the source code artifacts and, therefore, perform better. • We excluded the quality factor Specification Clone i.e., that there should not be a carbon copy of a requirement or a part thereof [44]. While specification clones may have a
TABLE II L IST OF SELECTED QUALITY FACTORS WITH THEIR ORIGIN , DESCRIPTION , LEVEL , IF THEY WERE MEASURED MANUALLY (²) OR AUTOMATICALLY (Ô), ARE DETERMINISTICALLY (D) OR HEURISTICALLY (H) MEASURABLE (C), AND ARE SUFFICIENTLY DISTRIBUTED (¡).
sentence level
use case level
Lvl.
Name
Ref.
Description
²/Ô
C
¡
Functional Duplication Inputs or Outputs not Quantified Use Case Naming Problems
[32] [7] [33]
² ² ²
D D H
p p p
Contains Justifications Inappropriate Scope Incoherent Text Order Mislocated Functional Requirement Free of Actor-Actor Interaction Happy Use Case Large Use Case Coherent
[34] [5] [5] [35]
If a use case includes the same functionality twice The degree to which quantitative inputs or outputs are specified in the use case Situations where the use case’s name bears no relation with the concept described or the same name is used for different concepts Requirement contains justifications for decisions taken A use case should only contain detail relevant to the problem statement A use case should follow a logical path with events in the correct order Functional requirements that are placed in fields addressing quality requirements
² ² ² ²
H H H H
p p p ✓
[35] [32] [33] [5]
² Ô Ô Ô
H D D D
p ✓ ✓ ✓
Meaningless Actor Meaningless Use Case
[32] [32]
Ô Ô
H H
p p
Tangled Requirements
[33]
Ô
H
✓
Scattered Requirements
[33]
Sentence should not contain actor to actor interaction If a use case does not contain a section for an alternative flow Many alternative flows and steps The sentence being written should repeat a noun in the last sentence or a previous sentence, if possible. If an actor is wrongly defined or meaningless to the functionality Wrongly defined or meaningless use case that does not contribute to the requirements specification When a use case contains descriptions of several requirements or different functionalities When the specification of one functionality is not encapsulated in one use case
Ô
H
✓
Contains Alternatives Contains Clarifications Inconsistent Level of Abstraction Free of UI Design Details Coordination Ambiguity
[5] [5] [7] [36]
² ² ² ² ²
D H H H H
✓ ✓ ✓ ✓ p
Anaphora Passive Voice Requirements Length Starts without Noun Phrase
[36] [36] [36] [37]
Ô Ô Ô Ô
D D D D
✓ ✓ ✓ ✓
Optional Negatives Sentence Level Complexity Complete Comparisons
[38] [39] [40] [41]
Alternative paths should be excluded from the main flow. Requirement contains clarifications of described functionalities The use case should be at a consistent level of abstraction The artifact should describe the problem domain instead of the solution domain The use of coordinating conjunctions (e.g., and or or) leads to multiple potential interpretations of a sentence A pronoun refers to a previous part of the text Contains verb in passive voice Number of words in a use case step Requirements shall start with the subject. As passive voice already covers this, we use noun phrases instead The sentence contains an optional part Statements of system capability not to be provided A sentence level complexity metric called NPC-Sentence A comparison is incomplete if there is no value for reference. E.g.: ’The system needs to be faster’.
Ô Ô Ô Ô
H H H H
p ✓ ✓ p
negative impact on other activities (such as maintaining a set of requirements), we do not assume that it influences the performance of a TLR approach. Which factor to consider as relevant was a subjective decision. To mitigate bias, we decided jointly after thorough discussion and documented choices for verification in our replication package [45]. After this step, we considered 36 of the 66 quality factors (54.5%) relevant to TLR. We further analyzed these factors with regard to their operationalization, resulting in the final selection of factors displayed in Table II. First, we determined the measurability of the eligible quality factors [37]. We distinguished between unmeasurable, manually measurable (²), and automatically measurable factors (Ô). For example, we classified the quality factor viability of alternatives as unmeasurable because it would require that “[a]lternatives should be viable and make sense” [5]. Measuring this quality factor would require domain knowledge that was not accessible to us. For each measurable quality factor, we further distinguished if we could measure it deterministically ((C = D)) or heuristically ((C = H)).
Excluding the 8 unmeasurable factors resulted in 28 (13 manually and 15 automatically) measurable requirements quality factors that we considered for the quality assessment of the use cases (see Table II). Where quality factors were ambiguous, we specified the formulation to preserve as much information as possible while ensuring that the factor can be measured with reasonable precision. For example, Ramos et al. consider a Large Use Case when “there are many alternative flows and steps” [33] without quantifying “many”. Instead, we consider the number of steps in a use case as its “size”, thereby preserving more information. Besides, we identified that most additional unnecessary information in the dataset were not “justifications” [34] but “clarifications” such that we newly introduced the quality factor Contains Clarifications. As the quality factor Passive Voice already covers passive phrase structures where the subject is not in the beginning of a sentence, we decided to change the quality factor Starting with Subject to Starts without Noun Phrase. This allows us to measure the impact of non-standard phrase structures on TLR. Additionally, we changed the factor Free of NFRs to
Mislocated Functional Requirements, as the only cases of nonfunctional requirements present in the datasets were quality requirements in the eTour dataset that were explicitly labeled in the use case template structure. However, we identified cases where the statements in the quality requirement field were actually functional requirements, which led us to observe those cases of mislocated functional requirements instead. Of the 28 factors, 10 were originally on sentence level and 18 on use case level according to the publications that proposed them [5]. We projected Contains Clarification, Contains Alternatives, and Inconsistent Level of Abstraction to the sentence that causes the issue despite being originally defined as use case level factors which yields more precise insights. 3) Manual Labeling: To obtain data on the 13 manually measurable quality factors, the two authors manually labeled the two datasets. As five of the 13 manually labeled factors were on sentence level, we labeled the 599 sentences of the datasets (see Table I) on this fine-grained level and the other eight factors on the 189 use case or subflow artifacts, resulting in 4507 data points. To ensure the reliability of this subjective step, we jointly developed an extraction guideline with instructions and examples, to be found in our replication package [45]. After agreeing on an initial version of the guidelines, each author individually labeled the dataset in three successive iterations. After each iteration, disagreements were discussed and the guidelines refined where necessary. Subsequently, each author individually revisited and relabeled all data points affected by the guideline changes to ensure consistency across the dataset. Per iteration, we reached an average Cohen’s kappa of 85%, 93%, and 100% for use case and 83%, 100% and 100% for sentence level factors, confirming a substantial agreement [46]. 4) Automatic Labeling: For the automatically measurable quality factors (Ô), we implemented analyzers using natural language processing (NLP) techniques. The factors Happy Use Case, Large Use Case, Coherent, Anaphora, Passive Voice, Starts without Noun Phrase, and Requirements Length we were able to measure deterministically (C = D) or at least approximately whenever relying on NLP techniques. For the other factors (C = H) we had to resort to heuristics that can only provide indication for the existence of the factor. We used the following heuristics: Meaningless Actor: The actor appears in the actors field but not the main flow of the use case description. Meaningless Use Case: According to the traceability gold standard the use case has no links to the source code. Tangled Requirements: The number of trace links from one requirement to multiple source code elements. Scattered Requirements: The maximum number of incoming trace links of one of the code files linked to the requirement. Note that the latter three heuristics are only operationalizations of their respective construct. We assume meaningfulness, tangled-ness, and scattered-ness exist before any TLs and use the gold-standard TLs as proxies, since direct assessment would require domain knowledge beyond our reach.
The remaining heuristics are independent of the gold standard but rely on NLP and keyword matching techniques, which might not cover all possible cases: Optional: Sentence contains one of the following keywords: “possibly”, “eventually”, “optionally”, “if possible”, “if appropriate”, “if needed”, “if necessary”, “if required”, “if applicable”, “if desired”, and “if applicable” Negatives: Sentence contains negating token (“not”, “n’t”, “wouldn’t”, “never”, “nowhere”, “noone”, “no-one”) and the negation has a dependency to a verb or auxiliary verb. Sentence Level Complexity: The sentence level complexity metric, or NPC-Sentence [40], can be calculated as follows. For each noun phrase (NP) chunk, the occurrence count in a sentence is divided by the total occurrence counts in all sentences. Then, all the frequency distributions of the NP chunks in the sentence are added together to form the final complexity value. Complete Comparisons: We consider a comparative adjective (JJR) or adverb (RBR) to be incomplete, if it is not followed by a comparison marker ("than" or "as") and is not part of a quantified conjunction (such as "four times faster"). 5) Factor Consolidation: Two additional steps were necessary before utilizing the observations of quality defects in the use cases. Since the gold standard TLs connect use cases with source code artifacts, we firstly needed to aggregate all sentence level factors per use case depending on their data type. For Boolean sentence level variables (e.g., whether a sentence contains an Anaphora or not) we calculated the ratio where the variable was true. For numeric sentence level variables (e.g., the length of a step in a use case), we selected the maximum of all sentences per use case, as we deemed the maximum more expressive than the average. Secondly, we excluded factors from the analysis if their natural distribution was strongly skewed. For example, we did not encounter any instance of Functional Duplication in our dataset. Hence, all data points in our analysis had the same value for this variable (Functional Duplication = 0) such that our observational data analysis would have not been able to make any inference about the influence of this factor. We excluded 12 factors that were insufficiently distributed (¡ = p in Table II), documented in our replication package [45]. While this meant discarding a considerable amount of manually gathered information, the distribution of the factors in our data was impossible to anticipate prior to their measurement. 6) Traceability Link Recovery: We operationalized the TLR task using several automatic algorithms tracing requirements to source code, since the different paradigms presented in Section II-B may react differently to quality defects, thus acting as a confounding factor. We discard supervised MLbased approaches, as they require a sufficient number of links per project for training, which may not be feasible given the size of our datasets. Furthermore, requiring existing links of a project limits the possible application scenarios for those approaches to scenarios where developers are available and willing to provide a substantial amount of those initial links [3]. Requiring no initial trace links, so performing
unsupervised TLR, is applicable in all scenarios and, thus, broadens the applicability of the gained insights of our study. The existing unsupervised TLR approaches can be categorized by paradigms and used technologies (see section II-B). Most approaches are considering the TLR as either an information retrieval or classification task. Of the most common IR paradigm-based approaches we chose the two most prevalent classical techniques VSM and LSI that use classical vector representations, and two approaches based on more modern language model-based embeddings FTLR, and LiSSA_IRonly. As representative of the more uncommon classification paradigm we make use of the recent LLM-based LiSSA approach. This selection covers the relevant range of approaches presented in Section II-B. For VSM and LSI we make use of the implementation provided in the replication package of Gao et al. [19]. FTLR [21], [3] presents the state-of-the-art of non-LLM-based approaches. It targets fine-grained (sentence level) information and combines IR-based TLR with neural network-based word embeddings. We make use of FTLR’s best performing configuration without use case template or requirements classification-based filters, as these filters are already targeted towards removing parts of the requirements with certain flaws. This configuration utilizes method comments and call dependencies. For VSM, LSI, and FTLR we report the results with per-project optimized thresholds that were optimized based on F1 -score on the respective gold standard. Thus, the results represent the upper boundary of their performance. LiSSA [29] is a retrieval-augmented generation (RAG)–based framework that uses LLMs to recover trace links across different software artifacts such as requirements, documentation, and code. For requirements to code TLR it outperforms FTLR, presenting the state-of-the-art in unsupervised TLR for this artifact pairing. Again, we use the best configuration reported by the authors, Fuchß et al., using GPT-4o with chain-of-thought (CoT) prompting. Additionally, we can use only the retrieval part of LiSSA, which is basically an embedding-based IR approach with OpenAI’s text-embedding-3-large, Fuchß et al. provided as well, which we call LiSSA_IR-only. As performance metrics we chose precision, recall, F1 score, and F2 -score. Those metrics are commonly used in TLR research [47]. They measure performance in a classification setting, providing insights on whether a link is correct or not. Table III lists the performance of all five investigated automated TLR approaches in terms of the four metrics across the two sampled datasets. To analyze the impact of the quality factors on performance, we computed precision, recall, F1 score, and F2 -score at the use case level instead of the project level. Specifically, for each use case, we determined the associated true positives, false positives, true negatives, and false negatives and derived the corresponding metrics from these counts. B. Data Analysis To answer our RQs, we performed regression analyses where we regress our outcome variables on the requirements
TABLE III P RECISION , R ECALL , AND F- SCORES OF USED TLR APPROACHES eTour
iTrust
Approach
P.
R.
F1
F2
P.
R.
F1
F2
VSM LSI FTLR LiSSA_IR-only LiSSA
.557 .452 .505 .216 .409
.427 .453 .597 .815 .734
.483 .453 .548 .342 .526
.448 .453 .576 .525 .633
.208 .251 .234 .058 .199
.227 .255 .241 .531 .451
.217 .253 .238 .105 .276
.223 .254 .240 .202 .360
quality factors. We followed the Pearlian framework for statistical causal inference [48], [49] as summarized by Siebert [50] using Bayesian data analysis (BDA) methods [51]. While BDA is less established in SE research, it produces more granular inferences from the data, preserves uncertainty instead of reducing complex data to binary results, and makes decisions on causal assumptions and statistical modeling explicit [52], [51]. As we cannot provide a comprehensive, pedagogical introduction to BDA methods in this manuscript, we refer the interested reader to appropriate textbooks [51], examples [52], [53], [54], [55], and our replication package [45]. The framework for statistical causal inference dictates three steps: modeling, identification, and estimation. We apply the same framework for answering both research questions. a) Modeling: First, we created a causal model in the form of a directed, acyclic graph (DAG) [56]. In this causal DAG, every node represents a variable and every edge an assumed causal relationship. More importantly, the absence of any edge between two variables represents a certain lack of a relationship between those variables. Our causal DAG consists of all (measurable) variables with an edge directed to the outcome TLR performance. There is only one interrelation among these variables: if a sentence is written in Passive Voice, we also assume that it Starts without Noun Phrase, which we encode with a directed edge. Additionally, we assume that the dataset itself has a direct influence both on TLR performance, because one dataset might just be easier to trace than the other, and on all quality factors, because the distribution of quality factors varies between the datasets. Similarly, we assume that the TLR approach influences TLR performance, as different paradigms for automated traceability link recovery may handle quality defects differently well. b) Identification: Next we derive a statistical model from the aforementioned causal model [48]. In this identification step, we selected all variables relevant to our investigation, as well as all variables necessary to de-confound the effect of interest [57]. In our case, this requires the inclusion of the dataset variable in our statistical model, as it is a potential common cause of all independent variables (i.e., different datasets may have different distributions of the variables) and the dependent variables (i.e., different datasets may naturally perform better or worse in TLR). c) Estimation: Finally, we estimated the strength of effect that each independent variable has on the response— i.e., dependent—variables through a regression model. Within
the Bayesian data analysis methods, we first selected an appropriate distribution type for each response variable [58]. We selected the distribution types based on the maximum entropy criterion [59] and ontological assumptions. As the values of all metrics are bounded within [0, 1], we chose a distribution from the Beta family [60]. The Beta distribution family deals with the heteroskedasticity of the data at the edges of the [0, 1] range. To account for occurrences of the values 0 and 1 in each metric, we selected the zero-one-inflated Beta distribution [61]. Next, we defined regression formulae for each response variable. These specify each response variable in relationship to all available independent variables. The strength and direction of the effect of each variable—the quantity of interest to answer our research question—is represented by a coefficient in the form of a random variable. We selected uninformative prior distributions for the coefficients of each included predictor and confirmed their eligibility via prior predictive checks [62]. After confirming their eligibility, we trained each model with the collected data. Hamiltonian Monte Carlo Markov Chains (MCMC) [63] update the coefficient distributions based on the empirical data. During this process, the parameters of the coefficient distributions are adjusted to better predict the response variable based on the independent variables [51]. After the training process, we perform posterior predictive checks, which work similarly to the prior predictive check but use the updated posterior coefficient distributions instead of the prior distributions. To evaluate each model, we inspected the distributions of the updated path coefficients. Traditionally, coefficients where the 95% credibility interval (CI) is consistent with 0, i.e., where the CI overlaps with 0, are not considered statistically significant [55]. To avoid the fallacy of reducing complex data down to the binary property of statistical significance, we still investigate all factors whose coefficient shows a notable effect. Therefore, in addition to the statistically significant factors we consider coefficients where the 50% credibility interval is not consistent with 0 to show a weak impact. While the posterior path coefficients answer RQ1, their interpretation is challenging since the regression formulae are wrapped by a logit-link function [51]. This link function scales the summed value of all regressors to the range (0, 1), which is the appropriate range for the estimated main parameter in a Beta distribution. To aid understanding the actual impact of significant quality factors on the response variables we additionally plot selected marginal effects. Marginal effects visualize the change in the response variable when manipulating one isolated regressor while keeping all others at a representative level [64]. Numeric predictors were fixed at their mean, and categorical predictors at their mode. In our case, the marginal effects were generated for FTLR assuming the dataset eTour. As such, they show the actual effect of this variable on the scale of the response variable. To answer RQ2, we performed the same data analysis process but with additional regression formulae. We formulated one regression model for every quality factor, containing an
interaction effect between this factor and the TLR approach. Such an interaction effect discerns whether a quality factor affects the response variable differently for different TLR approaches [65]. We only used the F2 -score as a response instead of all four performance metrics to limit the space of potential models. We plotted the conditional effects to visualize similar and different effects of quality factors across TLR approaches, which provides insight into whether some approaches deal with certain significant factors better or worse than others. IV. R ESULTS In this section, we show the results from the data analysis and set them into context of existing requirements quality work. For conciseness, we report only relevant and meaningful results. Our replication package contains the data, material, and figures of all omitted results [45]. A. Posterior Coefficient Distributions Figure 3 visualizes the posterior distributions of variable coefficients from the data analysis that were at least weakly significant. The dots represent the average effect of the factor on the respective response variable (coded in colors), while the two whiskers represent the single and double standard deviation distance from the average. The further the dot is away from 0, the stronger the effect. The more narrow the whiskers, the more certain the effect. Of the 16 investigated quality factors, 13 show at least a weakly significant effect on at least one of the four outcome metrics. Some factors exhibit narrow CIs, suggesting a very certain estimated effect. Large Use Cases affect only precision and the F1 -score slightly, where Scattered Requirements impede all metrics significantly. Tangled Requirements, on the other hand, have a positive effect on all metrics but recall. The other factors have less certain effects on the response variables. Notable are the negative effects of use cases with a high Requirements Length, and the ratio of sentences that contain Anaphora and Negatives on the performance of TLR approaches. On the other hand, the degree to which a requirement is written on an Inconsistent Level of Abstraction and Contains Alternatives will benefit particularly recall and F2 score. Sentence-level complexity significantly benefits precision at a minor cost to recall, improving F1 - and F2 -scores overall. The maximum Requirements Length (i.e., the highest number of words used in a single use case step), on the other hand, harms precision significantly, as does the existence of anaphora and negatives, which additionally also affect recall. Putting these results into context, some quality factors behave as expected. For example, the fact that a sentence that Starts without a Noun Phrase omits information potentially relevant to TLR and, hence, impedes TLR performance aligns with our expectations. Similarly, the negative effect of Requirements Length on all metrics but recall aligns with literature suggesting that larger use cases are harder to parse [33]. Recommendations against Anaphora [36] and Negatives [39] align with the observed negative effect on automated TLR.
Large Use Case
Metric Precision
Scattered Requirements
Recall F1
Tangled Requirements
F2
Starts without Noun Phrase Inconsistent Level of Abstraction Mislocated Functional Requirement Happy Use Case Coherent Sentence Level Complexity Requirements Length Contains Alternatives Anaphora Negatives -0.5
0.0
0.5
Posterior coefficient distribution
Fig. 3. Posterior coefficient distribution of (at least weakly) significant coefficients from the analysis
However, some quality factors traditionally labeled as “defects” exhibit a contrary effect. This is most notable for Inconsistent Level of Abstraction, Contains Alternatives, and Tangled Requirements. Requirements specified on an inconsistent level of abstraction, i.e., containing system-internal information, are traditionally considered solution-orientated [66]. Such requirements that impose on the solution-space rather than fully specifying the problems a system is supposed to solve have been shown to negatively affect the downstream software development process. They cause requirements engineers to commit to premature design decisions at an early stage of the project that require costly rework later [43]. Hence, literature advocates against an Inconsistent Level of Abstraction and considers it a quality defect [5]. Yet, for automated TLR, this factor shows a slight positive effect, possibly because internal implementation cues aid in recovering links to code artifacts. A similar case holds for Contains Alternatives: while established quality guidelines recommend separating alternative steps from the main scenario of a use case for clarity [5], automated TLR appears to benefit from blended scenario information. Again, labeling this quality factor as a “defect” may not be warranted in the case of automated TLR. The positive effect of Sentence Level Complexity on precision and, consequently, F1 - and F2 -scores is less intuitive to interpret. Prior literature assumes complex sentences have a negative effect, particularly in hindering human comprehension [40]. However, our results suggest this does not translate
to automated TLR approaches. Due to the operationalization of this factor using a suggested compound metric [40], though, we cannot trace this effect to specific linguistic properties. Contrasting the effect of the individual components of the complexity metric might provide further insights in the future. Answer to RQ1 13 out of 16 quality factors show at least a weakly significant effect on at least one outcome metric. Factors such as scattered requirements, requirements length, and the use of negatives impede TLR performance, while factors such as tangled requirements or contains alternatives benefit it. Overall, these results show that requirements quality affects automated TLR approaches, although the strength and direction of the effects vary across factors. B. Marginal Effects Figure 5 visualizes the marginal effects of two selected factors, Anaphora and Tangled Requirements, on the outcome metrics. Due to space limitations, we sampled these two as they show different factor scales (percentage vs. absolute), different effect directions (negative vs. positive effect), and also different effect strengths (moderate vs. strong). These two act as representative visualizations of the translation from posterior coefficient distributions to actual metric outcomes. Other marginal effects can be found in our replication package [45].
Fraction of Steps on an inconsistent Level of Abstraction
Number of outgoing Traceability Links
Total Number of Steps
Average F2 Value
0.5
Approach VSM
0.4
LSI FTLR 0.3
LiSSA LiSSA_IR-only
0.2 0.00
0.25
0.50
0.75
1.00
0.0
2.5
5.0
7.5
10.0
12.50
10
20
30
40
50
Fig. 4. Conditional effects between the approach and significant quality factors
Fraction of Steps containing Anaphora
Number of outgoing Traceability Links
Average Metric Value
0.6 0.5 0.4 0.3 0.2 0.00
0.25
0.50
Metric
0.75
1.00 0.0
Precision
Recall
2.5
5.0 F1
7.5
10.0
12.5
F2
Fig. 5. Marginal effects on the outcome metrics
While the x-axes of both plots show the different levels of the two factors, the two y-axes show the expected average value (as a line) and the 95% CI around it. Both plots show that recall, while highest, is affected the least and, therefore, shows the flattest slope, which is consistent with the coefficient values in Figure 3 where these intersect zero. On the other hand, the significant values of the posterior coefficients for Tangled Requirements in terms of precision, F1 - and F2 -score translate to steep slopes across the range of possible values of the quality factor. On average, increasing the number of outgoing links from a (theoretical) minimum to the maximum entails an increase in precision by about 0.2. Answer to RQ1 The strength of effects that quality factors exhibit on the performance on automated TLR approaches varies. At the extreme, use cases with a high number of outgoing traceability links (12, as observed in our sample) achieve about 20% higher precision than use cases with the minimum number of outgoing traceability links. C. Conditional Effects Figure 4 visualizes selected conditional effects. The x-axes represent the different levels of the quality factors (one percentaged, one real), the y-axis the average expected F2 -score. For each TLR approach, we plot a colored line (mean) and a 50% CI as a transparent ribbon around it (the width of the CI
is arbitrarily chosen and intended only to convey estimation uncertainty without cluttering the figure). For conciseness, we sampled representative effects for this manuscript. The conditional effects of all omitted significant quality factors are available in our replication package [45]. Most regression models suggest that TLR approaches respond similarly to varying levels of a given quality factor. For example, the left subplot visualizes the conditional effect of an increasing degree of Inconsistent Level of Abstraction on the F2 -score per TLR approach. As seen in Figure 3, this quality factor has a slight, yet not significant, positive impact on the F2 -score. All five approaches respond similarly: as use case steps increasingly include design- or implementation-specific details, F2 -scores tend to improve. The different intercepts of the five lines represent the different baseline performances of the five approaches, in line with Table III which identifies LiSSA_IR-only as the automated TLR approach that performs lowest in terms of F2 -score. The slightly different slopes represent the interaction effects of the approaches to the quality factor. Notably, VSM benefits most (i.e., has the steepest slope), likely because it relies entirely on keyword matching, and implementation-specific terminology aids its TLR. Thus, while Inconsistent Level of Abstraction may rightly be labeled a quality defect for downstream design activities [5], [43], it does not hinder but rather aids automated TLR. Only a few quality factors show markedly divergent effects across approaches. The middle subplot of Figure 4 visualizes the conditional effect between Tangled Requirements and the used approach on the F2 -score. While FTLR and LiSSA remain fairly stable, VSM and LSI performance suffers the more outgoing trace links connect a requirement with source code artifacts. On the other hand, LiSSA_IR-only strongly benefits from that same increase in the quality factor. The interpretation of this effect is less straight-forward. Requirements quality literature suggests that a “use case that contains several different functionalities could be hard to understand” [33], which intuitively translates also to a negative impact on automated TLR performance. However, Figure 3 and Figure 5 show a significant positive effect on precision, which also affects the F1 - and F2 -score. The conditional effect offers a finergrained explanation. LiSSA_IR-only responds positively as this approach retrieves the top 20 target artifacts, where 20
is, by design, a generous upper bound considering the two selected datasets. Thus, the requirements that do contain many outgoing trace links to source code artifacts improve the precision of LiSSA_IR-only as a larger portion of the retrieved 20 artifacts are likely to be true positives. LiSSA is stable because it additionally classifies the top 20 potential target artifacts retrieved from the IR approach [29], and thus, enhances its precision independently from the number of outgoing links. VSM and LSI, are threshold-optimized for F1 -score, and as a lower number of outgoing links is more common in the dataset they respond negatively to the quality factor exceeding the derived threshold. FTLR is also threshold optimized [3] but constructs links from the code files, making it less sensitive to this factor. Thus, whether Tangled Requirements constitute a defect for automated TLR depends on the approach: effects vary drastically across methods. The right subplot containing the conditional effect between Large Use Cases and the approaches shows a less pronounced, yet opposite, response. While LiSSA remains stable, the F2 score of FTLR, LSI, and LiSSA_IR-only decreases the more steps a use case contains, while the score of VSM increases. This divergence can be interpreted as follows. LiSSA, powered by GPT-4o, demonstrates the strongest capability to interpret semantic content across multiple steps, enabling it to retrieve relevant links regardless of the number of steps in a use case. VSM’s performance improves as use cases grow longer, likely because its TF-IDF-based matching benefits from increased term frequency and lexical coverage: longer text provides more opportunities for keyword overlap with source code artifacts, even when those artifacts relate to semantically distinct steps. In contrast, the other IR-based methods such as LSI and LiSSA_IR-only exhibit slight performance degradation with increasing step count, as their vector-space embeddings become less precise when aggregating semantically heterogeneous content across multiple steps. Answer to RQ2 For most studied quality factors, all five automated TLR approaches respond similarly. However, the approach notably moderates the effect of some quality factors on the F2 -score. This shows automated TLR approaches respond differently to certain quality factors such as the number of outgoing trace links and total number of steps contained in the use case depending on their design and architecture. V. T HREATS TO VALIDITY Our study is subject to the following threats to validity, categorized according to the types by Wohlin et al. [67]. A. Threats to internal validity We limited our selection of independent variables to those contained in an existing repository of requirements quality factors [11]. While this makes for a transparent selection based on established previous research, it also risks ignoring other
potentially relevant variables not included in the repository. Secondly, we had to exclude several variables as described in Section III-A2 because we lacked the domain knowledge to collect them. The only mitigation strategy within our means was to make our causal assumptions explicit in a causal DAG documented in our replication package [45]. This DAG makes explicit which variables we considered and which we ignored or had to discard. This allows other scholars to scrutinize our decisions and propose a competing causal DAG with revised assumptions as typical in principled statistical causal inference via model comparison [51]. B. Threats to external validity The generalizability of the inferred results is limited by the inclusion of only two datasets and five TLR approaches in our analysis. While further datasets and approaches would increase the external validity of the results, our strict selection criteria mentioned in Sections III-A1 and III-A6 limited the population to sample from. We used all data and working approaches available to us in the study but acknowledge that an extrapolation of the results, especially from only two datasets, has to be considered with caution. C. Threats to conclusion validity We obtained our conclusions via Bayesian methods in a statistical causal inference framework. Every design decision, such as the selection of the distribution type to model the response variables, represents a potential threat to the validity of our conclusion. To minimize this threat, we jointly discussed all these decisions and made them to the best of our knowledge under the guidance of established literature [56], [59], [51] and documented every subjective influence in our replication package for other scholars to assess. Still, improvements to the data analysis via even more complex regression models (e.g., involving hierarchical models) may further improve the conclusion validity. Finally, the interaction effects used to answer RQ2 are known to demand a much larger volume of data in order to converge on a result [68]. To acknowledge this threat, we plot the CIs in the visualization of conditional effects as seen in Figure 4 to retain explicit information about the uncertainty of these effects. D. Threats to construct validity Several quality factors from the repository [11] were described only vaguely and were difficult to measure. Our operationalizations may affect how well we could represent intended concepts. For example, the heuristics for Scattered Requirements, Tangled Requirements, and Meaningless Use Case are based on the TL gold standard of the datasets. Thus, the indicators depend on the quality of the gold standard, as well. For example, if the gold standard is incomplete, more use cases may be considered Meaningless Use Cases than actually are. Additionally, the heuristics for Tangled and Scattered Requirements must be considered with caution, as a high number of trace links does not necessarily mean that these defects actually are present. For example, if many classes are needed
to implement one use case step, that does not necessarily mean that this step has tangled requirements. We operationalized vague quality factors to the best of our knowledge after extensive discussions and documented all decisions to allow iterative improvement of the results’ construct validity. VI. D ISCUSSION Finally, we discuss the obtained results and put them into context with regard to their implications for TLR (Section VI-A) and requirements quality (Section VI-B), and outline potential ways of continuing this research in Section VI-C. A. Implications for TLR Besides contributing evidence to previously postulated hypotheses, our results enable the three use cases mentioned in Section I. Firstly, they allow predicting the fitness of a requirements specification for the activity of automated TLR. An organization planning to recover trace links between use cases and source code artifacts using one of the studied approaches can evaluate their use cases with the list of quality factors identified as significant. This quality assessment allows estimating how well the use cases in their current form will allow automated TLR. Secondly, the results can inform the design of requirements writing guidelines that instruct on producing “high”-quality requirements artifacts. Instead of basing writing guidelines on intuition, anecdotal evidence, or theory alone, our results serve as support for the particular effect of quality factors on automated TLR. An organization aiming to maximize the performance of an automated TLR approach can advise writing use cases in particular ways, e.g., avoiding excessive Requirements Length (for certain TLR approaches) or the use of Negatives. Such writing guidelines are more likely to produce requirements that can be considered of high-quality for the particular task than those not based on such evidence [6], [8]. Finally, the results can inform the design of automatic TLR algorithms. As acknowledged by the authors, the performance of even state-of-the-art approaches (Table III) leaves room for improvement which harms their practical applicability [3], [29]. Considering the impact of requirements quality in the design of new automated TLR approaches may allow further improving their performance. B. Implications for Requirements Quality Our results support the activity-based paradigm that the quality of requirements artifacts impacts the performance of subsequent activities [7], [8], i.e., the automated TLR. As such, this study provides empirical evidence requested by research roadmaps for requirements quality [6], [8], and therefore, contributes to the larger endeavor of evidence-based quality assessment of requirements artifacts. Accumulating sufficient evidence will allow companies to reason about which alleged quality factor to address (e.g., resolving all anaphora) and which to ignore (e.g., keeping alternatives in use case descriptions) for a given activity like automatic TLR. When using requirements artifacts for multiple activities— as usually the case—the quantification of the impact of a
quality factor on each activity allows deciding the tradeoff. For example, when deciding whether or not to consider Inconsistent Levels of Abstraction as a defect worth removing, comparing the benefit that they have on automated TLR with the drawback they have on design specifications [43] allows for an evidence-based decision. C. Future Work The results invite several streams of future work. For one, extrapolating the base assumption that the quality of artifacts impacts the activities in which they are used, an obvious extension of our work would be to also consider how the quality of the source code artifacts to which the requirements are traced impacts the TLR approach. Future studies could extend our approach to find (1) effects of the source code quality and (2) interaction effects between the quality factors of the two artifact types. Furthermore, there are several other types of requirements artifacts as well as traceability tasks involving requirements artifacts, such as inter-requirements TLR or requirements to model TLR. Since use case descriptions are mostly textual, we anticipate that our results may look similar for other textual requirements artifacts. Extending the approach to other TLR tasks will yield a more complete picture of the impact of requirements quality, though. VII. C ONCLUSION The quality of requirements artifacts affects the performance of automated TLR approaches trying to connect them to source code that implements these requirements. In this study, we were able to show that use case descriptions with high degrees of sentences not beginning with noun phrases, long requirements, or scattered requirements will impede the performance of some automated TLR approaches. On the other hand, inconsistent levels of abstraction and tangled requirements might benefit them. We recommend that the results of our study will be used as starting points for context-specific investigations on the impact of any of these factors. Ultimately, once the validity of our conclusions is strengthened, we envision that this avenue of research can inform organizations about how fit for automated traceability link recovery their requirements artifacts are. DATA AVAILABILITY S TATEMENT All protocols, data, source code, analysis scripts, figures, and results are available at https://github.com/JulianFrattini/ rq4tlr and archived in our replication package [45]. ACKNOWLEDGEMENTS This work was funded by Core Informatics at KIT (KiKIT) of the Helmholtz Assoc. (HGF) and supported by the German Research Foundation (DFG) - SFB 1608 - 501798263 and KASTEL Security Research Labs. Additionally, this work was supported by the KKS foundation through the S.E.R.T. Research Profile project at Blekinge Institute of Technology. The authors express their deep gratitude to Michael Unterkalmsteiner for providing feedback on the manuscript.
R EFERENCES [1] J. Cleland-Huang, O. Gotel, A. Zisman et al., Software and Systems Traceability. Springer, 2012, vol. 2. [2] S. Charalampidou, A. Ampatzoglou, E. Karountzos, and P. Avgeriou, “Empirical studies on software traceability: A mapping study,” Journal of Software: Evolution and Process, vol. 33, no. 2, p. e2294, Feb. 2021. [3] T. Hey, J. Keim, and S. Corallo, “Requirements classification for traceability link recovery,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE’24), 2024. [4] A. Vogelsang, A. Korn, G. Broccia, A. Ferrari, J. Fischbach, and C. Arora, “On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability,” in Proceedings of the 2025 ACM/IEEE 45th International Conference on Software Engineering: New Ideas and Emerging Results, 2025. [5] K. T. Phalp, J. Vincent, and K. Cox, “Assessing the quality of use case descriptions,” Software Quality Journal, vol. 15, pp. 69–97, 2007. [6] H. Femmer and A. Vogelsang, “Requirements quality is quality in use,” IEEE Software, vol. 36, no. 3, pp. 83–91, 2018. [7] H. Femmer, J. Mund, and D. M. Fernández, “It’s the activities, stupid! a new perspective on re quality,” in 2015 IEEE/ACM 2nd International Workshop on Requirements Engineering and Testing. IEEE, 2015, pp. 13–19. [8] J. Frattini, L. Montgomery, J. Fischbach, D. Mendez, D. Fucci, and M. Unterkalmsteiner, “Requirements quality research: a harmonized theory, evaluation, and roadmap,” Requirements Engineering, pp. 1–14, 2023. [9] J. Frattini, J. Fischbach, D. Fucci, M. Unterkalmsteiner, and D. Mendez, “Measuring the fitness-for-purpose of requirements: An initial model of activities and attributes,” in 2024 IEEE 30th International Requirements Engineering Conference (RE). IEEE, 2024. [10] H. Femmer, J. Kučera, and A. Vetrò, “On the impact of passive voice requirements on domain modelling,” in Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement, 2014, pp. 1–4. [11] J. Frattini, L. Montgomery, J. Fischbach, M. Unterkalmsteiner, D. Mendez, and D. Fucci, “A live extensible ontology of quality factors for textual requirements,” in 2022 IEEE 30th International Requirements Engineering Conference (RE). IEEE, 2022, pp. 274–280. [12] G. Antoniol, G. Canfora, G. Casazza, A. D. Lucia, and E. Merlo, “Recovering traceability links between code and documentation,” IEEE Transactions on Software Engineering, vol. 28, no. 10, pp. 970–983, Oct. 2002. [13] A. Marcus and J. I. Maletic, “Recovering Documentation-to-source-code Traceability Links Using Latent Semantic Indexing,” in Proceedings of the 25th International Conference on Software Engineering, ser. ICSE ’03. Washington, DC, USA: IEEE Computer Society, 2003, pp. 125– 135. [14] H. U. Asuncion, A. U. Asuncion, and R. N. Taylor, “Software traceability with topic modeling,” in 2010 ACM/IEEE 32nd International Conference on Software Engineering, vol. 1, May 2010, pp. 95–104. [15] M. Gethers, R. Oliveto, D. Poshyvanyk, and A. D. Lucia, “On integrating orthogonal information retrieval methods to improve traceability recovery,” in 2011 27th IEEE International Conference on Software Maintenance (ICSM), Sep. 2011, pp. 133–142. [16] K. Moran, D. N. Palacio, C. Bernal-Cárdenas, D. McCrystal, D. Poshyvanyk, C. Shenefiel, and J. Johnson, “Improving the effectiveness of traceability link recovery using hierarchical bayesian networks,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE ’20. New York, NY, USA: Association for Computing Machinery, Jun. 2020, pp. 873–885. [17] A. Panichella, C. McMillan, E. Moritz, D. Palmieri, R. Oliveto, D. Poshyvanyk, and A. D. Lucia, “When and How Using Structural Information to Improve IR-Based Traceability Recovery,” in 2013 17th European Conference on Software Maintenance and Reengineering, Mar. 2013, pp. 199–208. [18] H. Kuang, P. Mäder, H. Hu, A. Ghabi, L. Huang, J. Lü, and A. Egyed, “Can Method Data Dependencies Support the Assessment of Traceability Between Requirements and Source Code?” J. Softw. Evol. Process, vol. 27, no. 11, pp. 838–866, Nov. 2015. [19] H. Gao, H. Kuang, K. Sun, X. Ma, A. Egyed, P. Mäder, G. Rong, D. Shao, and H. Zhang, “Using Consensual Biterms from Text Structures of Requirements and Code to Improve IR-Based Traceability Recovery,” in Proceedings of the 37th IEEE/ACM International Conference on
Automated Software Engineering, ser. ASE ’22. New York, NY, USA: Association for Computing Machinery, Jan. 2023. [20] H. Gao, H. Kuang, W. K. G. Assunção, C. Mayr-Dorn, G. Rong, H. Zhang, X. Ma, and A. Egyed, “TRIAD: Automated Traceability Recovery based on Biterm-enhanced Deduction of Transitive Links among Artifacts,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, Apr. 2024, pp. 1–13. [21] T. Hey, F. Chen, S. Weigelt, and W. F. Tichy, “Improving Traceability Link Recovery Using Fine-grained Requirements-to-Code Relations,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), Sep. 2021, pp. 12–22. [22] J. Guo, J. Cheng, and J. Cleland-Huang, “Semantically Enhanced Software Traceability Using Deep Learning Techniques,” in Proceedings of the 39th International Conference on Software Engineering, ser. ICSE ’17. Piscataway, NJ, USA: IEEE Press, 2017, pp. 3–14. [23] W. Wang, N. Niu, H. Liu, and Z. Niu, “Enhancing Automated Requirements Traceability by Resolving Polysemy,” in 2018 IEEE 26th International Requirements Engineering Conference (RE), Aug. 2018, pp. 40–51. [24] C. Mills, J. Escobar-Avila, A. Bhattacharya, G. Kondyukov, S. Chakraborty, and S. Haiduc, “Tracing with Less Data: Active Learning for Classification-Based Traceability Link Recovery,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), Sep. 2019, pp. 103–113. [25] M. Zhang, C. Tao, H. Guo, and Z. Huang, “Recovering Semantic Traceability between Requirements and Source Code Using Feature Representation Techniques,” in 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), Dec. 2021, pp. 873– 882. [26] J. Lin, Y. Liu, Q. Zeng, M. Jiang, and J. Cleland-Huang, “Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models,” in Proceedings of the 43rd International Conference on Software Engineering, ser. ICSE ’21. Madrid, Spain: IEEE Press, Nov. 2021, pp. 324–335. [27] B. Wang, Z. Zou, X. Liang, H. Jin, and P. Liang, “HGNNLink: Recovering requirements-code traceability links with text and dependency-aware heterogeneous graph neural networks,” Autom Softw Eng, vol. 32, no. 2, p. 55, May 2025. [28] A. D. Rodriguez, K. R. Dearstyne, and J. Cleland-Huang, “Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability,” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), Sep. 2023, pp. 455–464. [29] D. Fuchß, T. Hey, J. Keim, H. Liu, N. Ewald, T. Thirolf, and A. Koziolek, “LiSSA: Toward Generic Traceability Link Recovery Through RetrievalAugmented Generation,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 1396–1408. [30] T. Hey, D. Fuchß, J. Keim, and A. Koziolek, “Requirements Traceability Link Recovery via Retrieval-Augmented Generation,” in Requirements Engineering: Foundation for Software Quality, A. Hess and A. Susi, Eds. Cham: Springer Nature Switzerland, 2025, pp. 381–397. [31] D. Fuchß, S. Schwedt, J. Keim, and T. Hey, “Beyond Retrieval: A Study of Using LLM Ensembles for Candidate Filtering in Requirements Traceability,” in 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW), 2025, pp. 5–12. [32] A. Rago, P. Frade, M. Ruiva, and C. A. Marcos, “An Approach for Automating Use Case Refactoring,” Electronic Journal of SADIO, vol. vol. 13, Jun. 2014. [33] R. Ramos, J. Castro, F. Alencar, J. Araújo, A. Moreira, C. d. E. da Computacao, and R. Penteado, “Quality improvement for use case model,” in 2009 XXIII Brazilian Symposium on Software Engineering. IEEE, 2009, pp. 187–195. [34] E. Parra, C. Dimou, J. Llorens, V. Moreno, and A. Fraga, “A methodology for the classification of quality of requirements using machine learning techniques,” Inf. Softw. Technol., vol. 67, no. C, pp. 180–195, Nov. 2015. [35] C. Usdadiya, S. Tiwari, and A. Banerjee, “An Empirical Study on Assessing the Quality of Use Case Metrics,” in Proceedings of the 12th Innovations in Software Engineering Conference (Formerly Known as India Software Engineering Conference), ser. ISEC ’19. New York, NY, USA: Association for Computing Machinery, Feb. 2019, pp. 1–11. [36] A. Ferrari, G. Gori, B. Rosadini, I. Trotta, S. Bacherini, A. Fantechi, and S. Gnesi, “Detecting requirements defects with NLP patterns: An
industrial experience in the railway domain,” Empirical Softw. Engg., vol. 23, no. 6, pp. 3684–3733, Dec. 2018. [37] H. Femmer, M. Unterkalmsteiner, and T. Gorschek, “Which Requirements Artifact Quality Defects are Automatically Detectable? A Case Study,” in 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW), Sep. 2017, pp. 400–406. [38] G. Lami, S. Gnesi, F. Fabbrini, M. Fusani, and G. Trentanni, “An Automatic Tool for the Analysis of Natural Language Requirements,” 2004. [39] H. Femmer, D. Méndez Fernández, S. Wagner, and S. Eder, “Rapid quality assurance with Requirements Smells,” Journal of Systems and Software, vol. 123, pp. 190–213, Jan. 2017. [40] C. Y. Din and D. Rine, Requirements content goodness and complexity measurement based on NP chunks. VDM Publishing Saarbrücken, 2008. [41] H. Hasso, H. Geppert, M. Dembach, and D. Toews, “Detection of defective requirements using rule-based scripts,” in International Conference on Requirements Engineering - Foundation for Software Quality (REFSQ) 2019, 2019. [42] E. Ben Charrada, A. Koziolek, and M. Glinz, “Supporting requirements update during software evolution,” J. Softw. Evol. Process, vol. 27, no. 3, pp. 166–194, Mar. 2015. [43] J. Frattini and A. Frattini, “Adopting use case descriptions for requirements specification: an industrial case study,” in 2025 IEEE 31st International Requirements Engineering Conference (RE). IEEE, 2025. [44] E. Juergens, F. Deissenboeck, M. Feilkas, B. Hummel, B. Schaetz, S. Wagner, C. Domann, and J. Streit, “Can clone detection support quality assessments of requirements specifications?” in Proceedings of the 32nd ACM/IEEE International Conference on Software EngineeringVolume 2, 2010, pp. 79–88. [45] T. Hey and J. Frattini, “How Requirements Quality Makes (or Breaks) Traceability Link Recovery - Replication Package,” https://doi.org/10. 5281/zenodo.20448214, 2026. [46] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960. [47] Y. Shin, J. H. Hayes, and J. Cleland-Huang, “Guidelines for Benchmarking Automated Software Traceability Techniques,” in 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability, May 2015, pp. 61–67. [48] J. Pearl, Causality. Cambridge university press, 2009. [49] ——, “Causal inference in statistics: An overview,” Statistical Surveys, 2009. [50] J. Siebert, “Applications of statistical causal inference in software engineering,” Information and Software Technology, vol. 159, p. 107198, 2023. [51] R. McElreath, Statistical rethinking: A Bayesian course with examples in R and Stan. Chapman and Hall/CRC, 2018. [52] C. A. Furia, R. Feldt, and R. Torkar, “Bayesian data analysis in empirical software engineering research,” IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1786–1810, 2019.
[53] C. A. Furia, R. Torkar, and R. Feldt, “Applying bayesian analysis guidelines to empirical software engineering data: The case of programming languages and code quality,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 3, pp. 1–38, 2022. [54] R. Torkar, R. Feldt, and C. A. Furia, “Bayesian data analysis in empirical software engineering: The case of missing data,” Contemporary Empirical Methods in Software Engineering, pp. 289–324, 2020. [55] J. Frattini, D. Fucci, R. Torkar, L. Montgomery, M. Unterkalmsteiner, J. Fischbach, and D. Mendez, “Applying bayesian data analysis for causal inference about requirements quality: a controlled experiment,” Empirical Software Engineering, vol. 30, no. 1, p. 29, 2025. [56] F. Elwert, “Graphical causal models,” in Handbook of causal analysis for social research. Springer, 2013, pp. 245–273. [57] C. Cinelli, A. Forney, and J. Pearl, “A crash course in good and bad controls,” Sociological Methods & Research, vol. 53, no. 3, pp. 1071– 1104, 2024. [58] A. Gelman, A. Vehtari, D. Simpson, C. C. Margossian, B. Carpenter, Y. Yao, L. Kennedy, J. Gabry, P.-C. Bürkner, and M. Modrák, “Bayesian workflow,” arXiv preprint arXiv:2011.01808, 2020. [59] E. T. Jaynes, Probability theory: The logic of science. Cambridge: Cambridge University Press, 2003. [60] P. Paolino, “Maximum likelihood estimation of models with betadistributed dependent variables,” Political Analysis, vol. 9, no. 4, pp. 325–346, 2001. [61] R. Ospina and S. L. Ferrari, “A general class of zero-or-one inflated beta regression models,” Computational Statistics & Data Analysis, vol. 56, no. 6, pp. 1609–1623, 2012. [62] J. S. Wesner and J. P. Pomeranz, “Choosing priors in Bayesian ecological models by simulating from the prior predictive distribution,” Ecosphere, vol. 12, no. 9, p. e03739, 2021. [63] S. Brooks, A. Gelman, G. Jones, and X.-L. Meng, Handbook of Markov Chain Monte Carlo. CRC press, 2011. [64] T. D. Mize, L. Doan, and J. S. Long, “A general framework for comparing predictions and marginal effects across models,” Sociological Methodology, vol. 49, no. 1, pp. 152–189, 2019. [65] B. Denters and R. A. Van Puijenbroek, “Conditional regression analysis: Problems, solutions and an application,” Quality and Quantity, vol. 23, no. 1, pp. 83–108, 1989. [66] D. M. Fernandez, S. Wagner, K. Lochmann, A. Baumann, and H. de Carne, “Field study on requirements engineering: Investigation of artefacts, project parameters, and execution strategies,” Information and Software Technology, vol. 54, no. 2, pp. 162–178, 2012. [67] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, A. Wesslén et al., Experimentation in software engineering. Springer, 2012, vol. 236. [68] A. Gelman, “You need 16 times the sample size to estimate an interaction than to estimate a main effect,” https://statmodeling.stat.columbia.edu/ 2018/03/15/need16/, accessed: 2023-11-24.