Adapt the Object to the Subject: Representative Study Material for Empirical Software Engineering

Adapt the Object to the Subject: Representative Study Material for Empirical Software Engineering Max Unterbusch∗ , Julian Frattini† , Michael Mircea‡ , Alexander Korn∗ , Kurt Schneider‡ , Jonathan Ullrich§ , and Andreas Vogelsang∗ ∗ paluno – The Ruhr Institute for Software Technology, University of Duisburg-Essen, Germany

{max.unterbusch, alexander.korn, andreas.vogelsang}@uni-due.de † Chalmers University of Technology and University of Gothenburg, Sweden

arXiv:2611.07772v1 [cs.SE] 5 Nov 2026

julian.frattini@chalmers.se ‡ Software Engineering Group, Leibniz University Hannover, Germany

{michael.mircea, kurt.schneider}@inf.uni-hannover.de § Fraunhofer IESE, Germany

jonathan.ullrich@iese.fraunhofer.de

Abstract—External validity is a central concern in empirical software engineering, yet the debate has largely focused on whether study participants are representative. How well the study objects they reason about—code, models, or requirements—are representative of each participant’s own work is often overlooked. Practitioners’ own artifacts would be ideal, but are often out of reach, so researchers sample from public datasets or construct objects artificially. Whether this lack of subject–object fit affects participants’ ability to act on the material remains unknown. We take a first step toward answering this question for requirements. We propose an approach that uses an LLM to generate contextadapted synthetic requirements just-in-time during an online survey from a short, participant-supplied context description, optionally injecting requirements smells as controlled stimuli. To validate the approach, we conducted a controlled experiment with 31 industry practitioners in a 2 × 2 design that crosses stimulus generation (static vs. synthetic) with quality (clean vs. smelly). Each participant rated the perceived representativeness and assessability of 20 requirements. Using Bayesian data analysis within a statistical causal-inference framework, we find that perceived representativeness has a strong positive effect on assessability. Synthetic requirements are rated as more representative and—primarily due to this gain—more assessable than static ones sourced from public datasets; smell injection leaves both perceptions unchanged. Our results support treating a lack of study-object representativeness as a first-class validity threat and position context-adapted synthetic study objects as a practical addition to the empirical methodology toolbox. Index Terms—Study objects, external validity, requirements, empirical research, controlled experiment, Bayesian data analysis, causal inference, sensitivity analysis

I. I NTRODUCTION Empirical studies have become a core instrument of software engineering (SE) research. As SE is an applied science, it is important that our empirical results are useful to practice, which makes external validity a central concern. In general, generalizability can be improved by making studies as realistic and representative as possible considering the target population

to which one wants to draw conclusions for [1]. Much of this concern has centered on the study participants, where a long-standing debate asks whether students can adequately represent industry practitioners [2], [3]. However, this concern only addresses one dimension of external validity, the interaction of treatment and selection [4]. The extent to which the study objects—including code, commits, architecture diagrams, and requirements—resemble those that a participant would encounter in their own work is often overlooked [5]. Using practitioners’ own artifacts as study material would have high external validity in the setting dimension, but this is often infeasible: Such artifacts are often confidential, hard to obtain, and sometimes too sparse in the phenomenon under study to be useful. A common approach is therefore to sample study objects from public datasets (e.g., [6]–[9]) or construct them artificially (e.g., [10]–[13]). However, in these settings, the material may be of low representativeness and might not match a participant’s domain and expertise. Currently, we lack evidence on the extent to which the representativeness of study objects affects the assessability by participants. We take a first step toward answering this question for a particular SE study object, requirements. Requirements are core to many streams of research, including research on requirements quality [9], [13]–[15]. Unnoticed defects in requirements can become costly problems in downstream SE tasks and risk project success [16]. For this reason, requirements smells have been defined as indicators for potential quality deficiencies [14]. However, for most smells, we lack empirical evidence for their relevance [15], [17], although first studies have begun to investigate their effects [9], [18], [19]. We believe part of the reason is that requirements quality is inherently difficult to study. Whether a requirement smell is genuinely harmful depends on context factors, such as domain, the process model, or the availability of stakeholders who could resolve it [15]. Experiments in requirements quality

© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

research are only meaningful if participants can relate the presented requirements to a context they know; confronting them with unfamiliar material would undermine the judgments the study relies on. To address this problem, we propose and evaluate an approach that generates context-adaptive synthetic requirements just-in-time and uses them as study objects. At the start of a study, each participant supplies a short description of their working context, which is used to condition a large language model (LLM) that generates requirements tailored to that participant on the fly. To evaluate whether the approach is capable of adding controlled stimuli, an additional LLM-based mutation step injects requirements smells into the generated study objects, yielding both clean and smelly requirements. We conducted a controlled experiment with 31 industry practitioners, following a 2 × 2 within-subjects design that crosses the stimulus generation (synthetic vs. static) with the requirement quality (clean vs. smelly). The static condition serves as the control level: requirements sampled from projects in public datasets without adaptation to the participant. Each participant rates requirements from all four combinations of conditions on two questions, capturing the dimension of perceived representativeness and assessability. Both constructs are first-person judgments that only the practitioners themselves can make in their context. Analyzing the ratings with Bayesian data analysis [20] within a framework for statistical causal inference [21], we find that perceived representativeness has a strong positive effect on participants’ assessability, i.e., their confidence in acting on the requirements. Since representativeness could not be experimentally manipulated, we adjusted for known confounders in our causal model and show in a sensitivity analysis that the observed association could only be spurious if an unknown confounder were implausibly strong. Synthetic requirements, which we experimentally controlled, are rated as more representative and, primarily for this reason, more assessable than static ones sourced from public datasets. The LLMbased smell mutation leaves both perceptions unchanged. In summary, this paper makes the following contributions: Empirical evidence that study-object representativeness has a strong positive effect on practitioners’ perceived ability to act on the material. Demonstrated here for requirements, this supports treating a lack of object representativeness as a first-class validity threat in human-subject experiments and motivates studying it for other types of study objects. • An approach for generating context-adapted study objects just-in-time from a short, participant-supplied context description, here instantiated for requirements with an optional LLM-based mutation step that injects requirements smells. In a controlled experiment with 31 practitioners, synthetic requirements were perceived as more representative and, primarily through this gain, more assessable than static ones sampled from public datasets, while smell injection left both perceptions unchanged. The approach is in principle transferable to other types of SE study objects. •

•

Methodological guidance on when the approach is and is not appropriate, and how to apply it. II. BACKGROUND & R ELATED W ORKS

A. Validity of Experiments The validity of an experiment is a fundamental concern in empirical research. Following Wohlin et al. [4], adequate validity is achieved when the experimental results hold for the target population to which the experimenters intend to generalize. Four core types of validity are typically considered: Conclusion, construct, internal, and external validity. These types of validity usually cannot be maximized simultaneously within a single experiment and must therefore be prioritized with respect to the experiment’s purpose [4]. Specifically important to this work are internal validity, which concerns the ability to draw causal conclusions between the variables of interest, and external validity, which concerns the ability to generalize the results beyond the study setting [4]. While experiments focus on establishing causal evidence with high internal validity, external validity must not be neglected. As Sjøberg et al. [1] argue, an experiment that lacks external validity is of no practical use, since its conclusions hold only within the experimental setting. Hence, they identify realism as a particular concern for generalizability to industrial practice and recommend that experimental tasks should be as representative of industrial practice as possible. It is important to be aware that external validity is a multidimensional concept. Wohlin et al. distinguish three types, defined by the interaction of the treatment with selection, history, and setting [4]. The selection dimension—most strongly associated with external validity for many SE researchers— refers to validity threats that occur because the population of study subjects is not representative of the population to which the experimenters intend to generalize. However, there are also other dimensions of external validity. The history dimension refers to threats that occur because the experiment is preceded by special events that might impact the results. Most important for this work is the setting dimension, the interaction of setting and treatment, which Wohlin et al. define as “the effect of not having the experimental setting or material representative of, for example, industrial practice” [4, p. 138]. Regarding the first-mentioned dimension of selection, a widely discussed validity concern in SE research is the choice of study participants [2], [3]. It is broadly acknowledged that experiments with professional participants are desirable, and there is evidence that practitioners perceive studies conducted with industry professionals as more relevant [22]. In practice, however, this is often difficult to realize as the availability of professionals is limited. The choice of study subjects thus represents a deliberate tradeoff: employing practitioners increases external validity but, compared to the more homogeneous population of students, comes at the cost of control [1]. A less often discussed threat to external validity stems from the study objects [5], which are part of the setting dimension. Context-fit, i.e., how well the study objects fit a study participant’s context, may moderate the effect of a

stimulus on an outcome. To illustrate the importance of using representative study objects, consider the following example from requirements quality research: In an experiment where students reviewed a static set of requirements unfamiliar to them, the use of passive voice (the stimulus, which was supposedly associated with ambiguity/poor quality) greatly affected a subsequent modeling task (the outcome) [23]. However, in a different study conducted at a case company, passive voice turned out to be practically harmless [24]. An important difference between the studies is that in the latter, the contextfit between experiment subjects and objects was much greater as practitioners are more familiar with their requirements. Hence, passive voice was rarely judged problematic in the actual context because the practitioners’ domain knowledge and working context compensated the supposed defect [24]. In the example above, the latter study has more external validity with respect to the setting dimension than the former because it had access to the real requirements from their subject’s working context. In research practice, however, real artifacts are often out of reach—most commonly due to confidentiality, but also when the phenomenon under study occurs too rarely for any single practitioner to supply enough examples, or when the material does not yet exist, as when researchers evaluate a newly proposed SE method. While prior work suggests that domain familiarity aids RE tasks [25], we lack evidence on whether and to what extent the representativeness of study objects affects the assessability by participants. In this confirmatory research, we test and quantify this effect in a controlled experiment and derive implications for the design of experiments. B. Requirements Generation with LLMs LLM-assisted requirements elicitation and documentation has become an active area of requirements engineering (RE) research [26]. Several studies have evaluated the quality of LLM-generated requirements against human baselines. Krishna et al. [27] find that GPT-4-2023-09-25 and CodeLlama34b produce software requirements specifications comparable in completeness and consistency to those of entry-level engineers. They also find that the generated requirements largely avoid ambiguities. Quattrocchi et al. [28] investigate user story generation across multiple LLMs and report similar coverage and stylistic quality to human authors, though with less creativity and diversity regardless of model size. Beyond generation from scratch, Mircea et al. [29] show that LLM revisions of stakeholder-authored requirements improve stakeholder alignment, readability, and unambiguity. A smaller body of work uses LLM-generated requirements for research purposes rather than practitioner support. Fantechi et al. [30] generated conflicting requirements from a seed set using ChatGPT and subsequently asked the model to detect the resulting inconsistencies. El-Hajjami and Salinesi [31] evaluate synthetic requirements as training data for ML classifiers, finding that despite lower diversity, they are valuable for classification tasks. Femmer et al. [32] investigated linguistic properties of requirements generated by gpt-4.1-2025-04-14

under varying amounts of contextual input, finding that richer context improves output quality, yet synthetic requirements remain syntactically simpler than those in public datasets. Our work differs from both strands. The practitioner-facing studies above compare LLMs and humans as competing producers of requirements for the same project, while the research-oriented studies feed synthetic requirements into ML models or other automated tasks. Neither evaluates synthetic requirements as study objects for human participants in empirical studies. The use of LLMs to generate experimental material has also been proposed in other fields, notably psychology [33]. To our knowledge, however, this approach has neither been proposed nor examined for SE study objects. III. S TUDY D ESIGN We report our study design following the guideline for reporting controlled experiments by Jedlitschka et al. [34]. A. Study Objectives, Research Questions & Overview The first objective of this paper is to investigate whether the representativeness of a requirement is itself a determinant of practitioners’ perceived ability to act soundly on the material. We consider a requirement representative when a practitioner could plausibly encounter it in their own work, and assessable when they feel confident performing their typical tasks on it. Establishing such a link is the question on which the importance of all other representativeness-related design choices in human-subject studies depends. We raise the following research question (RQ): RQ1. To what extent does perceived representativeness of a requirement affect its perceived assessability? The second objective of this paper is to investigate the general suitability of LLM-generated requirements to be used as context-adapted study objects in empirical research. We investigate whether practitioners, as we hypothesize, rate the synthetic requirements as more representative (important for external validity in the setting dimension, see Section II-A) and as more assessable (important for the soundness of experiments) than the static baseline. We thus ask: RQ2. To what extent do practitioners perceive synthetic requirements as representative compared to static ones? RQ3. To what extent do practitioners perceive synthetic requirements as assessable compared to static ones? The third objective is to investigate whether synthetically generated requirements can be further mutated to inject requirements smells while preserving their perceived representativeness and assessability. This invariance is a precondition for using the pipeline as a source of study stimuli: most empirical studies must manipulate their study objects to administer a treatment. If the smell injection itself altered perceived representativeness or assessability, the treatment would be confounded with these properties, rendering the approach useless for this purpose. Hence, we also ask: RQ4. Does LLM-based smell mutation affect the perceived representativeness of synthetic requirements?

Quality

Generation Static Clean

Synthetic Clean

Static Smelly

Synthetic Smelly

Fig. 1. Conditions of the 2 × 2 within-subjects design.

RQ5. Does LLM-based smell mutation affect the perceived assessability of synthetic requirements? To address these RQs, we conduct a controlled experiment with 31 industry practitioners. The study follows a 2×2 withinsubjects design that crosses the factor generation (synthetic vs. static) with the factor quality (clean vs. smelly), yielding four conditions under which each participant evaluates five distinct requirements (see Figure 1). Both levels of the generation factor represent a subset of the requirements from a single coherent project, shown to the participant in a randomized order, yet only the synthetic condition is context-adapted. In the synthetic condition, the project is generated on the fly by an LLM from the participant’s self-reported context (application domain and system description). The static condition serves as control level, where the project is randomly selected from a public requirements dataset with no adaptation to the participant, as often done in RE research (e.g., [8], [9]). The staticsmelly condition contains naturally occurring smells, while the synthetic-smelly requirements have been injected by an LLM. We specifically chose a within-subjects design because between-subjects variance among industry practitioners is expected to be large relative to our feasible sample size. Pairing the four conditions within each participant therefore yields substantially more statistical power than a between-subjects design at the same N . Simultaneously, the within-subjects design helps avoid the tendency of participants to respond neutrally to Likert-scale items due to the lack of comparison in a between-subjects design. B. Study Objects 1) Static Requirements: To benchmark context-adapted against static study objects, we draw the static requirements from two publicly available datasets, each covering one popular requirements format: For classical specification-style requirements, we use the PROMISE+ dataset [35] containing 88 projects and for user stories we use the dataset provided by Dalpiaz [36], containing 22 projects. Since our design also compares clean and smelly study objects, we cannot sample arbitrarily but have to ensure that the static condition contains genuine, naturally occurring smells. We focus on five of the nine smell types used by Vogelsang et al. [13], originally drawn from the catalog of Frattini et al. [37]: subjective language, weak verbs, optional parts, negative statements, and passive voice. We excluded three smells that are inherently relational (e.g., contradiction) because we require every smell to be self-contained in a single

Prompt: Requirements Generation Context: You are an expert requirements engineer in [domain]. You value high-quality requirements. Task: Generate 20 requirements for the following system: [systemDescription]. The set of requirements does not need to be complete. Write the requirements in the following style: [requirementsFormat]. Your output MUST be a valid JSON in the format {REQ1: requirement, ..., REQ20: requirement}. Fig. 2. Prompt template used to generate context-adapted synthetic

requirements. Runtime placeholders are shown in [. . . ].

requirement. We later also excluded one smell that occurred only rarely in the two datasets (vague pronoun). We then systematically mined both datasets for projects that contain examples of each of the smells as well as a sufficient number of clean requirements. For all of the 5,357 requirements, we individually prompted an efficient OpenAI model (gpt-5.4-mini-2026-03-17 with temperature 0) to rate the presence of each smell on an ordinal scale from 0 (confident not present) to 4 (confident present). The full prompt is provided in our online appendix1 . A project qualified for our study if it contained (i) at least one requirement with confidence 4 for each of the five smells and (ii) at least five requirements with confidence 0 across all smell categories. Whenever there were multiple candidates per class (e.g., requirement with subjective language smell), we selected the requirement at random. This yielded five qualifying userstory and specification-style projects each. To validate the LLM-based pre-filtering, two authors independently labeled each candidate (smell genuinely present / requirement genuinely clean). Both raters agreed on 94 of 100 cases, corresponding to Cohen’s κ = 0.67 (substantial) and Gwet’s AC1 = 0.93 (very strong). In 13 cases at least one annotator rejected the candidate. For these cases, we drew a replacement requirement from the same project on which both raters agreed. In three cases no suitable replacement within the same project could be found for the originally targeted smell; we substituted a requirement bearing a different smell and balanced these substitutions across subjective language, negative statement, and optional part (one each). 2) Synthetic Requirements: For the synthetic requirements condition, we generate 20 requirements with OpenAI’s flagship model at the time of writing (gpt-5.5-2026-04-23) conditioned on the participant’s context (application domain and system description) just-in-time while our participants are filling the survey. We used a temperature of 0.7 to balance output determinism and creativity in the generated requirements. The prompt is shown in Figure 2. We feed 15 of the 20 generated requirements (positions 2– 4, 6–8, 10–12, 14–16, 18–20) into a second LLM call that mutates them to introduce smells for the synthetic-smelly cell. 1 http://doi.org/10.6084/m9.figshare.32397573

Prompt: Smell Mutation Context: You are an expert requirements engineer in [domain] and you will be given 15 requirements for the following system: [systemDescription]. Task: Modify some of the requirements by adding requirements smells. For each of the following five smells, pick one requirement out of the 15 where you can apply it: subjective language, optional parts, weak verbs, passive voice, and negative phrases (each smell is provided with a definition and two examples). Requirements: [REQ2-REQ4], ..., [REQ18-REQ20] Your output MUST be a valid json, keyed by REQSUBJ, REQOPT, REQWEAK, REQPASS, REQNEG.

Prompt template used to mutate five of the 15 supplied requirements to contain smells. Runtime placeholders are shown in [. . . ]; content abbreviated for space is shown in italics. Fig. 3.

We prompt the same model to produce exactly five mutations, one per smell type, and the model is free to choose which of the 15 candidates to mutate, so that no requirement is forced to carry a smell that would be unnatural for it. The mutation prompt is shown in Figure 3. The remaining five requirements (positions 1, 5, 9, 13, 17, i.e., every fourth requirement) are passed through unchanged and form the synthetic-clean cell. Generating initially only 20 requirements is a deliberate tradeoff to keep total waiting time below roughly 15 seconds. Consequently, the synthetic requirements that the participant eventually sees are also a non-contiguous excerpt of their underlying project, matching the static condition in this regard. C. Study Execution 1) Instrumentation: The study is implemented as an online survey in LimeSurvey, extended with the LLM Survey Plugin that allows us to trigger LLM calls just-in-time between pages based on participant input2 . The survey has four pages: Page 1 – Context elicitation. Participants provide the three inputs that drive the generation step: the application domain (free text, e.g., “consumer mobile app”), a system description of one or two sentences (free text, e.g., “a language-learning app with gamification elements for motivation”), and the requirement format most representative of their work (one of classical specification-style or user story). In addition, participants select the activities they typically perform on the basis of requirements (e.g., writing requirements, writing code, deriving test cases). This last question primes participants for the assessability item on Page 3 and lets us interpret their answers relative to their familiar working tasks. While the participant transitions from Page 1 to Page 2, two things happen in the background. First, the LLM generates 20 requirements (see Section III-B2), which takes approximately 10 seconds, during which the participants see a loading animation. Second, we randomly draw one of the five curated projects of static requirements that match the chosen format. 2 https://git.se.uni-hannover.de/llm-survey-tool/llmsurvey. Version 2.3 of the plugin is also available in our online material1

Page 2 – Task description. Page 2 introduces the rating task and shows the two rating items. On the transition from Page 2 to Page 3, we make the LLM call to mutate the previously generated requirements to contain smells (see Section III-B2), which takes about five seconds. We chose to have this intermediate screen between the generation and mutation steps in order to avoid a single, yet longer waiting screen. Page 3 – Rating task. Each participant evaluates 20 requirements, five per each of the four cells (see Figure 1), presented in randomized order and fully blinded. Participants were told neither that some requirements were synthetically generated nor that some were deliberately mutated to carry a smell. For each stimulus, participants respond to two 5-point Likert items on a scale from strongly disagree (1) to strongly agree (5). The first item, “I could imagine encountering this requirement in my own work,” operationalizes representativeness. The second item, “I feel confident that I could perform my typical tasks based on this requirement,” operationalizes assessability as the self-reported ability to act soundly on the requirement. Page 4 – Closing. Finally, we elicit demographic information to characterize the sample: years of professional experience in software or systems engineering (integer estimate), frequency of working with requirements artifacts (daily, weekly, monthly, rarely, never), and a self-positioning between exclusively academic (1) and exclusively industry (5) on a fivepoint scale. An optional free-text field invites participants to share general comments. Before launch, we piloted the complete survey with three non-author participants. The pilot confirmed that the instrument functioned as intended and, in particular, that the two rating items were interpreted consistently with our intention. 2) Participants & Data Collection: We aimed to recruit 30 industry practitioners with a software or systems engineering background via personal invitations. We consider this sampling strategy appropriate and refrained from openly calling for participants in order to avoid academic peers diluting our sample. The intended target group is practitioners who routinely engage with written requirements artifacts; we therefore filter out respondents who report never working with requirements, as well as those that report having an exclusively academic background. The survey was open from May to June 2026 and participation was entirely voluntary. 3) Requirements Analysis: Previous works showed that LLM-generated requirements may have lower syntactical complexity than real requirements [32]. Such under-complexity would limit the realism of the study objects [1], [5]. To investigate whether our synthetic requirements exhibit the same under-complexity, after data collection, we analyze metrics of the previous study: parse-tree-depth, defined as the longest path from root to leaf in the dependency tree of a sentence [38] and the sentence length measured in the number of lexical words of the requirements. Moreover, to validate that the smell mutation step produced the intended smell types, two authors independently re-inspect each synthetic-smelly requirement.

assumed causal effect exposure

assumed moderation

covariate

outcome

RQ2

generation RQ5

RQ4

RQ3

quality

format

representativeness RQ1

assessability experience

role

Fig. 4. DAG encoding our causal assumptions

D. Data Analysis We utilize Bayesian data analysis (BDA) [20] within a framework for statistical causal inference (SCI) [21] to answer our RQs based on the collected data. We chose BDA over the more common frequentist methods for their fine-grained insights [39] and their ability to properly model data that is not normally distributed [40]. Since our outcome variables are on an ordinal scale and, hence, not normally distributed, BDA is more eligible. The SCI framework allows differentiating causal relationships from spurious associations [41]. We adhere to the framework by Siebert [21] comprising a modeling, identification, and estimation step. 1) Modeling: First, we explicitly specified our causal assumptions in the form of a directed acyclic graph (DAG) [42]. In the resulting DAG, visualized in Figure 4, every node represents a variable and every directed edge an assumed causal relationship. For example, we assumed that the representativeness of a requirement affects its assessability (RQ1). Table I contains a complete list of our variables. In our online material,1 we justified each included edge between two variables, which we omitted from this manuscript for brevity. 2) Identification: From our causal assumptions modeled in a DAG, we identified the adjustment set, i.e., the set of variables necessary to statistically control to avoid bias via confounding [20]. Since the data were collected via a controlled experiment and the two exposure variables were fully controlled, confounders were eliminated by design for RQ2-RQ5 [41]. Still, we statistically adjusted for all covariates affecting any outcome variable to improve the precision of the subsequent estimation [43]. 3) Estimation: Finally, we estimated the strength of the assumed causal relationships as specified in Figure 4. For this, we trained two Bayesian regression models—one per outcome variable—following established procedures for BDA [20]. For every regression model, we choose the outcome distribution that is consistent with what we know about the outcome variable while adding no further assumptions [44]. In our case, both outcome variables are measured on an ordinal scale with five levels. This implies that (1) any valid value must be one of the five levels, (2) the five levels have an inherent order to them, but (3) the distances between adjacent levels are

unknown and not necessarily equal (e.g., the distance from level 1 to 2 need not equal that from 2 to 3). We therefore use a cumulative model [45] (Equation (1)) to represent the outcome variable. The cumulative model assumes that each response results from an underlying, latent variable, which can be understood as the continuous attitude of a respondent towards a statement. In the absence of contrary evidence, we assume that this latent attitude is normally distributed, i.e., it follows the maximum-entropy distribution for a real-valued quantity with a given mean and variance. The location ϕi of this distribution is described by a linear predictor (Equations (2) to (4)), which specifies how the predictors shift a participant’s latent attitude toward representativeness (and, equivalently, toward assessability). The thresholds κk are the values of the latent attitude at which a participant switches from one response level (e.g., “neutral”) to the next (e.g., “agree”). In summary, a respondent selects option k or lower if their latent attitude, which is normally distributed around ϕi , falls below the threshold κk , and an option above k otherwise. Formally, Pr(repi ≤ k) = Φ(κk − ϕi ), where Φ is the cumulative distribution function of the standard normal distribution (a socalled probit link). repi ∼ Cumulative(ϕi , κ) ϕi = αID[i] + βg · geni + βq · qualityi

(1) (2)

+ βg×q · geni · qualityi

(3)

+ βf · formati

(4)

αj ∼ N (0, σ)

for j = 1, . . . , J

σ ∼ Weibull(2, 1) βg , βq ∼ N (0, 0.5)

(6) (7)

βg×q , βf ∼ N (0, 0.2) κk ∼ N (0, 1)

(5)

(8) for k = 1, . . . , K − 1,

κ1 < · · · < κK−1

(9) (10)

The linear predictor contains a participant-specific varying intercept αID[i] (Equations (5) and (6)) and a coefficient for each predictor (e.g., βq for quality). Because every participant provided several ratings, the varying intercept accounts for the resulting non-independence and captures systematic, unmeasured differences between participants, such as a general tendency to agree. The varying intercepts regularize these estimates, which typically improves estimation compared to models that ignore the clustering [46]. The coefficients β are the quantities of interest for the analysis. They are expressed on the latent (probit) scale, i.e., in standard deviations of the latent variable. Given the lack of strong evidence about these effect strengths, we select uninformative prior probability distributions to initialize all coefficients [20]. For example, we initialize βg with a normal distribution centered around 0 and with a standard deviation of 0.5 (Equation (7)). This distribution puts equal probability mass above and below 0, representing our prior belief that the effect of the generation method could

TABLE I R ELEVANT VARIABLES INVOLVED IN THE STUDY AND THEIR ROLE IN THE representativeness (R EP.) AND assessability (A SS .) MODELS (◦ = OUTCOME , • = EXPOSURE , • = COVARIATE ). B OTH MODELS ADDITIONALLY INCLUDE THE INTERACTION G E N E R A T I O N × Q U A L I T Y . Variable

Description

Type

Scale

generation quality

The method with which the requirement was generated Whether or not the requirement contains a smell

categorical categorical

format experience role

The format in which a requirement is specified The number of years of experience of the participant The activities that a participant performs on requirements

representativeness assessability

How representative a participant perceives a requirement to be How assessable a participant perceives a requirement to be

be either negative or positive. We performed prior predictive checks by sampling from these priors directly without updating them to confirm the eligibility of the priors [47]. Once confirmed, we trained the model using Markov Chain Monte Carlo sampling [48]. This process updates the coefficients in light of the collected data to better reflect the observed effects of the predictors on the outcome variables, producing posterior probability distributions. After training, we checked diagnostics [20] and performed posterior predictive checks [47] to confirm that the regression models trained well. At this point, the coefficients represent the likely strength of the effects that the predictors have on the outcomes, under the assumption specified in Figure 4 and our prior beliefs. This allows answering our RQs from the posterior probability distributions. For all RQs, we check whether the 95% credible interval (CI) of the coefficient associated with the factor in question contains 0. If not, then it implies a posterior probability of at least 97.5%3 that the factor affects the respective outcome variable either positively (CI > 0) or negatively (CI < 0). RQ1 requires inferring about the effect of perceived representativeness on perceived assessability. This effect is captured by the path coefficient of representativeness in the assessability model, βrepresent , with generation adjusted as a confounder so that the coefficient admits a causal interpretation (see Figure 4). RQ2 and RQ3 require inferring about the effect of the generation approach on the representativeness (RQ2) and assessability (RQ3) of requirements. These effects are captured by the path coefficient of the generation approach βg in the respective regression model. RQ4 and RQ5 require inferring about how requirements quality moderates the effect of the generation approach on the representativeness (RQ4) and assessability (RQ5) of requirements. Similarly, we investigate the 95% CI of the path coefficient of the interaction effect βg×q . 4) Validity Assessment: The internal validity of the result to RQ1 may be threatened, as this is the only RQ where the exposure (representativeness) was not experimentally manipulated. 3 The equal-tailed 95% CI leaves 2.5% of posterior mass beyond each bound. If the lower bound is above 0, the 95% inside the interval and the 2.5% above it are both positive, so at least 97.5% of posterior mass is ≥ 0.

Rep.

Ass.

{static = 0, synthetic = 1} {clean = 0, smelly = 1}

• •

• •

categorical continuous categorical

{specification, user story} N {write, design, code, ...}

•

• • •

ordinal ordinal

{1, 2, 3, 4, 5} {1, 2, 3, 4, 5}

◦

• ◦

Hence, the effect of representativeness on assessability may be biased by unobserved confounders [49], [50]. The participantspecific intercept αID[i] already absorbs any confounder that is constant within a participant, such as a general tendency to rate leniently. A remaining confounder would therefore have to vary across the requirements that a participant rates, such as the clarity of a requirement. To assess the reliability of the results to RQ1, we perform a sensitivity analysis [51]. We consider a hypothetical confounder U , standardized to unit variance, that is characterized by two parameters: γ, the effect of a one-standard-deviation increase in U on the latent assessability, and δ, the difference in the mean of U between requirements rated lowest and highest in representativeness. Omitting such a confounder from the model biases the estimated effect of representativeness on assessability by approximately γ ·δ [51]. We express the effect of representativeness as ∆, the difference in latent assessability between the lowest and the highest level of representativeness, which combines all polynomial contrasts [45]. For a grid of values of δ and γ, we subtract the bias γ · δ from every posterior draw of ∆ and determine the combinations at which the lower bound of the 95% credible interval reaches zero. These combinations describe how strong a confounder would have to be to render the observed effect compatible with zero. Vice versa, this would be the necessary strength of the confounder in order to suggest the observed effect strength if it were indeed not there. Since absolute values of δ and γ are hard to judge, we compare them to two empirical anchors from our own data [49]. For γ, we use the standard deviation σ of the participantspecific intercepts αID[i] in the assessability model. A confounder with γ = σ would affect assessability as strongly as the entire variation between participants in their baseline assessability ratings. For δ, we use δsyn , the imbalance of the experimentally manipulated factor generation between requirements rated lowest and highest in representativeness, standardized in the same way as U . A confounder with δ = δsyn would be as closely aligned with representativeness as the treatment that was designed to change it. Both anchors represent strong effects. If even a confounder that matches both anchors simultaneously does not render the effect compatible with zero, we consider the answer to RQ1 reliable. Since the bias formula is exact for linear models, its application on the

latent scale of the probit model is an approximation [52]. We therefore interpret the analysis as an assessment of robustness rather than as an exact bound. IV. S TUDY R ESULTS A. Participant Information From a total of 54 responses, 17 participants dropped out on page 1 (context elicitation), two dropped out on page 2 (task description) and four dropped out on page 3 (main rating task). This left 31 complete responses, none of which met our exclusion criteria of never working with requirements or having a purely academic background. Our sample spans a diverse range of professional experience, with a median of 7 years in software engineering (Figure 5a). The reported domains cover e-commerce, transportation, medical technology, automotive, and several others. Fourteen participants selected specification-style requirements as more representative of their work, while 17 selected the user story format. Most participants write, refine, and prioritize requirements (Figure 5b). Moreover, most work with requirements regularly, with only two reporting rare engagement (Figure 5c). Consistent with our recruitment goal, the professional background leans toward industry (Figure 5d). B. Requirements Analysis To gauge the complexity of the synthetic requirements, we compare them to the static requirements used in this study. Synthetic requirements show similar parse-tree depth to the static requirements in our sample for both the user story and the specification-style format (Figure 6a). Sentence length, by contrast, differs more visibly. Synthetic requirements tend to be longer than their static counterparts used in this study in both formats (Figure 6b). Two authors independently vetted all synthetic-smelly requirements to check whether the smell mutation step injected the smells correctly. Both authors unanimously agree that all smells have been correctly included with respect to the five intended types (perfect agreement). C. Results from the Bayesian Data Analysis Representativeness affects Assessability (RQ1). Figure 7 reports the posterior estimates and 95% CIs of the coefficients on which the five RQs hinge. Our assessability model estimates the effect of perceived representativeness on assessability (bottom whisker in Fig. 7). Because representativeness enters the regression as a five-level ordinal predictor, its effect is decomposed by default into four orthogonal polynomial contrasts (linear, quadratic, cubic, and quartic), which together exhaust any function of the five levels. The linear contrast βrepresent.linear has a 95% CI entirely above zero (Figure 7), indicating a strong positive effect. The three higher-order contrasts (quadratic, cubic, and quartic) are more difficult to interpret, thus we excluded them from Figure 7 (full posteriors in our online material1 ). Nonetheless, the cubic contrast also has a 95% CI entirely above zero, indicating that the effect of representativeness on

assessability exhibits some curvature. Since generation, a confounder of the effect of representativeness on assessability, is controlled for in the model, this coefficient admits a causal interpretation under the assumptions specified in Figure 4: requirements that practitioners rated as more representative of their work were also rated as substantially better assessable. The sensitivity analysis in Figure 8 shows that a confounder U with γ = σ ≈ 0.50, i.e., one that affects assessability as strongly as the entire variation between participants, would require δ = 3.61 to render ∆ compatible with zero. This is about 2.2 times the imbalance δsyn = 1.64 of the experimentally manipulated factor generation. Conversely, a confounder with δ = δsyn would require γ ≈ 1.1, about 2.2 times σ. We consider a confounder of this strength implausible, though the effect of representativeness on assessability is supported by this sensitivity analysis rather than identified by randomization. Synthetic Requirements (RQ2 & RQ3). For both representativeness and assessability, the credible interval of βgeneration (first and third rows in Fig. 7) lies entirely above zero. We therefore conclude with at least 97.5% posterior probability that synthetic study objects increase both the perceived representativeness (RQ2) and the perceived assessability (RQ3) of requirements relative to static study objects sampled from public datasets. The marginal effects of generation on both outcomes (figures in our online material1 ) illustrate the size of this shift. For representativeness, the most likely response was strongly disagree in the static condition (about 36% likelihood) and strongly agree in the synthetic condition (about 57% likelihood), indicating a strong effect of generation. For assessability, the effect is much weaker but points in the same direction. The reason the direct effect of generation on assessability is smaller than that on representativeness is the strong indirect path identified for RQ1: most of the assessability gain flows through representativeness. Smell Mutation (RQ4 & RQ5). To assess whether LLMbased smell mutation alters the effect of generation approach, we inspect the interaction coefficients βgeneration×quality of the representativeness and assessability models, respectively (second and fourth rows in Fig. 7). For both outcomes, the 95% CI includes zero near the center of the posterior distribution. Therefore, any assumed moderation of the generation effects by requirement quality could be almost as plausibly negative as positive. Yet either way, the strength of the moderation tends to be small. V. D ISCUSSION A. Threats to Validity We discuss the threats to the validity of our study along the four established validity types [4]. Some threats are inherent to the proposed approach of generating study objects with LLMs, while others are specific to the present study. Internal validity. By design, the study objects are tailored to each participant’s context and therefore differ in content from

Participants

Participants

8 6 4 2 0

5 10 15 20 25 30 35

R Pri efine orit iz W e Ma rite n Re age Est view im De ate sig Tesn Co t Othde er

0

Participants

24 21 18 15 12 9 6 3 0

(a) Years of experience.

(b) Requirements-based activities.

14 12 10 8 6 4 2 0

Participants

10

16 14 12 10 8 6 4 2 0

Ne

1 2 exclusively academic

(c) Requirements frequency.

(d) Professional background.

ver arely nthly eekly Daily R Mo W

3

4

5 exclusively industry

Synthetic Static

User story

30 27 24 21 18 15 12 9 6

Synthetic Static

User story

Spec-style

(a) Parse-tree depth.

Spec-style

(b) Sentence Length.

Fig. 6. Complexity of synthetic vs. static requirements used in this study (synthetic user stories: n = 170; synthetic spec-style: n = 140; static user stories: n = 50; static spec-style: n = 50).

Effect of 1 SD of U on latent assessability

24 21 18 15 12 9 6 3

Lexical word count

Parse-tree depth

Fig. 5. Participant demographics (N = 31).

2.0

imbalance of synthetic 3×

1.5

2×

1.0

1× participant SD

0.5

0.0 0

1

2

3

4

Imbalance of U between rep = 1 and rep = 5 items (SD units)

Adjusted lower 95% bound

Representativeness

−6

generation generation × quality

−4

−2

0

Fig. 8. Result of the sensitivity analysis, showing that the empirical anchor (black point) lies in the region below the red line where the effect survives.

Assessability generation generation × quality represent. linear

0.5

0.0

0.5

1.0

1.5

2.0

Fig. 7. Posterior distribution of the RQ-relevant coefficients of the representativeness model (top) and assessability model (bottom).

one participant to the next. Participant and stimulus set are thus partly confounded, which is a degree of experimental control we deliberately trade for representativeness. One concrete consequence is that thin or ambiguous context descriptions can affect the generated requirements and, in turn, the measured variables. A few participants indeed provided only sparse descriptions, for instance, giving “complete vehicle” as the system, or describing more than one system. Such inputs may have constrained the relevance of the generated requirements; even so, we still observed a clear difference in perceived representativeness between the static and synthetic conditions. A further concern is blinding: because the synthetic study objects are personalized, it is likely that participants recognized some requirements as LLM-generated; some participants

also explicitly noted this in the free-text feedback. We do not know how this knowledge could have influenced the responses. Awareness about the LLM generation could have positively impacted the outcome variables if participants wanted to help us authors achieve “good results”, i.e., social-desirability bias [4]. We tried to mitigate this risk by reminding participants on Page 2 of the survey to respond honestly, that there are no right or wrong answers, and that they should answer with what feels right to them. On the other hand, awareness about the LLM generation could have negatively impacted the outcome variables if participants are generally skeptical or distrusting in LLM-generated content. To further mitigate this risk, participants were not told that any requirements were static, and the presentation order was randomized. The same reasoning applies to the blinding between clean and smelly requirements, which apparently, is less likely to have leaked. External validity. Our findings rest on a sample recruited through personal invitations and thus is not a random sample from the population of software engineering practitioners. In SE research, a truly random sample is usually not possible to achieve. In addition, the synthetic requirements were produced

with a single LLM, and results may differ for other LLMs. Construct validity. We operationalized representativeness and assessability each through a single item, although both are multifaceted: representativeness can concern content or style, and assessability can be judged with a different task in mind (writing, reviewing, coding, or testing). We deliberately elicit one holistic judgment per construct rather than querying each facet separately, which could cognitively overload participants and dilute the overall gut feeling that ultimately governs whether they can engage with a study object. Moreover, both operationalizations are self-reported. For representativeness, we see this as a necessity and a firstperson judgment as the most reasonable operationalization. Assessability, in contrast, is a proxy: it reflects participants’ reported confidence to act, not how soundly they would actually act. By design, we study this proxy first. Confidence is the more immediate question: for representativeness to affect how soundly practitioners act, it must first affect how confidently they act. Measuring behavior directly would instead require fixing a single task and object type which would also entangle the representativeness effect with task design and individual skill. Moreover, such a design assumes an agreed standard of correct action (i.e., ground-truth), which, in the case of context-dependent phenomena, may not be available (e.g., requirements quality). Nonetheless, in future work, task-based designs (see Section V-E) are necessary to support the results from our judgment study with behavioral measures. In addition, the static requirements were already matched to each participant’s preferred requirement format, so format adaptation is present in both conditions. The synthetic condition adds only the application domain and system description on top, meaning that any observed benefit of context adaptation is therefore a conservative estimate and would likely be larger if requirement format were varied as well. Conclusion validity. Due to the non-determinism of LLMs, the synthetic requirements are not perfectly reproducible, which concerns the reliability of the treatment implementation: even an identical context would not yield the exact same requirements. We accept this variability as an unavoidable property of the proposed approach. To support comparability and replication across studies, we report our LLM use following the reporting guidelines of Korn et al. [53].

requirements smells. Our controlled experiment yields positive evidence on both. Participants perceived the synthetic requirements as more representative of their own work and as better assessable for their typical tasks than the static requirements sampled from public datasets (RQ2, RQ3). The assessability gain stems primarily from improved representativeness, with a smaller direct effect of generation mode, plausibly reflecting that LLM generated requirements carry an inherent property that makes them more assessable (e.g., better grammar or lack of foreign domain terms). Injecting artificial smells did not measurably impact either perceived representativeness or assessability (RQ4, RQ5). The free-text feedback reveals that several participants were positive about the contextual fit. One respondent stated: “I was highly impressed by how well the (supposedly) generated requirements matching my application domain present real world use cases.” A recurring critique, however, was that the synthetic requirements came across as too high-level. One participant described them as “too complex and partly too nonspecific,” and another noted that some “lacked enough context for implementation or testing”. While some participants recognized that requirements were LLM-generated, none indicated awareness that others were static; hence, we cannot attribute individual comments to the synthetic or the static condition specifically. Looking at the logs, we see that both the requirements generation and requirements mutation step have been completed for every participant leaving Page 1. Hence, we conclude that no participant dropped out during the two waiting screens, which combined lasted approximately 15 seconds. For the justin-time generation approach to be useful, it is important that the waiting time does not lead to a higher drop out rate. Regarding the smell mutation step, we observe that all smells have been correctly generated and all represent plausible, naturally-sounding cases. However, we also note some homogeneity in the generated smells. This is particularly the case for the weak verb smell, for which the LLM most often injected the verb “may”. While these examples are correct smell representations, they demonstrate little diversity. We assume the reason could be that the LLM adhered closely to the examples provided in the prompt.

B. General Observations

Our observations are consistent with the broader evidence on LLM-generated requirements. Like prior studies reporting high consistency and alignment of LLM-generated requirements [27], [28], we find the synthetic requirements to be stylistically consistent and well aligned with the systems our participants described. A more notable contrast concerns complexity. In a related study, synthetic requirements were found to be syntactically simpler than those in public datasets [32]. We find no such under-complexity here: the synthetic requirements are comparable in parse-tree depth to the static requirements and even tend to be longer (Figure 6), also compared to the other datasets investigated in the previous study. Two factors plausibly explain this shift. First, the previous work

We observe that the representativeness of requirements has a strong positive effect on participants’ perceived confidence to act on the material (RQ1). Thereby, this study provides direct empirical evidence about the assumed link between representativeness and assessability in the case of requirements. We hypothesize that the same holds for other object types that are relevant to SE research, such as code and models. This draws the external validity of empirical studies, which rarely consider the context-fit of their study objects, into question. We also investigated whether LLM-generated requirements are suitable as context-adapted study objects in empirical research, and whether they can additionally be mutated to carry

C. Relation to Existing Evidence

already showed that supplying a detailed project and domain description yields more realistic complexity, which is exactly the kind of input our context elicitation collects. Second, gpt5.5 is a more capable model than the gpt-4.1 used previously. While we cannot disentangle the two, both suggest that the under-complexity reported earlier should not be treated as an inherent property of LLM-generated requirements. D. Recommendations for Research Our approach of context-adapted study objects raises assessability through increasing representativeness. If researchers find other means of supplying context-adapted objects that make them more representative of the participants’ working context (e.g., customization “by hand”), this could likewise raise assessability. Herein, we propose the use of our approach as one way of obtaining more representative objects, and thereby, reducing the threat to external validity from setting. The proposed approach is not specific to requirements, and of course, not specific to requirements smells. In principle, the same scheme—eliciting a short participant-supplied working context, conditioning an LLM on it, and using the output as a study stimulus—could be instantiated for other SE study objects. In fact, LLMs are already heavily used to generate several SE artifacts in practice, including source code, unit tests, and code documentation [54]. Whether such instantiations of the approach would raise perceived representativeness, and whether the representativeness-to-assessability link we observe here would carry over, are open empirical investigations that the present study does not answer. Nonetheless, we have several recommendations for researchers that plan to employ the approach, for which we conjecture that they are valid for other types of study objects as well. When to apply the approach. Adopting context-adapted study objects is a deliberate methodological tradeoff. Fixed study objects are perfectly uniform across participants, which simplifies treatment standardization and allows per-item comparisons across participants. Context-adapted study objects vary across participants by design, but each participant sees material grounded in their own working context, supporting external validity in the setting dimension as discussed in Section II-A. We therefore recommend the approach primarily for context-dependent research questions, where representativeness is particularly important and some loss of stimulus control is acceptable. For studies that depend on holding stimuli perfectly constant, fixed objects remain preferable. Within this scope, the approach applies in principle to humansubject studies regardless of participant background (students or practitioners), provided a meaningful working context can be elicited. We are more cautious about non-human study subjects, specifically LLMs, due to information leakage and selfpreference bias risks (e.g., first injecting and then detecting requirements smells with the same LLM). How to apply the approach. We recommend vetting the generation approach both before and after data collection. A pre-hoc check confirms that the generated material carries the intended properties. For research that requires stimuli (e.g.,

smells) to be more diverse, we assume more variability could be reached by having more diverse examples, different prompt variants, or inviting the LLM to be creative. Other research might require a higher level of control on the generated stimuli, favoring strict prompt-adherence. A post-hoc check, similar to our re-inspection of the mutated requirements, then verifies validity on the material participants actually saw. Such a post-hoc check, in consequence, has impacts on the scalability of the approach. Where the design permits, we further recommend eliciting participants’ own representativeness judgments, since they surface mismatches that researchers may not anticipate. Finally, the context-elicitation instrument deserves the same care as the generation prompt, as thin or ambiguous descriptions might degrade the result. Prompting participants for sufficient detail, e.g., through examples or guided fields, and piloting the instrument are worthwhile. Reporting and ethics. Although LLM non-determinism precludes exact reproduction, studies should report their LLM use. We recommend following prompting reporting guidelines, such as the one of Korn et al. [53] to which we adhered. Finally, while the approach deliberately avoids requesting confidential material, the elicited context descriptions can themselves be sensitive and need to be handled with care. E. Future Work Our study examined a single object type, requirements, and measured assessability through self-reports. Two extensions are therefore particularly important. First and foremost, the study should be replicated with other study objects, such as source code, tests, or architectural diagrams, to test whether the representativeness–assessability link we observe generalizes beyond requirements, as we conjecture. Second, selfreported assessability should be complemented with revealed behavior, by having participants act upon the study objects through specific tasks. Such task-based designs would also allow investigating whether more realistic settings raise participants’ engagement and motivation to take a study seriously, as suggested by Sjøberg et al. [1]. VI. C ONCLUSION External validity in empirical SE has long been debated in terms of who participates in our studies, and much less in terms of the study objects participants work on. In our experiment with 31 practitioners, the perceived representativeness of requirements had a strong positive effect on their assessability, i.e., participants’ confidence in acting on them. This provides support for a threat that Wohlin et al. [4] already list in their catalog of external validity, but for which empirical evidence was missing. We also proposed generating context-adapted study objects just-in-time from a short description of the participant’s working context. Instantiated for requirements, these were perceived as both more representative and more assessable than objects sampled from public datasets, primarily through the representativeness gain. We therefore recommend that researchers consider study-object representativeness when designing experiments, and see context-adapted objects as one

way to achieve it. Whether the representativeness-assessability link carries over to other object types and downstream behavioral effects is an open question and we hope this first step invites the community to pursue it. R EFERENCES [1] D. Sjoberg, B. Anda, E. Arisholm, T. Dyba, M. Jorgensen, A. Karahasanovic, E. Koren, and M. Vokac, “Conducting realistic experiments in software engineering,” in Proceedings International Symposium on Empirical Software Engineering. IEEE, 2002, pp. 17–26. [2] I. Salman, A. T. Misirli, and N. Juristo, “Are students representatives of professionals in software engineering experiments?” in 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol. 1. IEEE, 2015, pp. 666–676. [3] D. Falessi, N. Juristo, C. Wohlin, B. Turhan, J. Münch, A. Jedlitschka, and M. Oivo, “Empirical software engineering experts on the use of students and professionals in experiments,” Empirical Software Engineering, vol. 23, no. 1, pp. 452–489, 2018. [4] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in Software Engineering. Berlin, Heidelberg: Springer Berlin Heidelberg, 2024. [5] J. Frattini, R. Torkar, R. Feldt, and C. Furia, “Towards improving the external validity of software engineering experiments with transportability methods,” in Proceedings of the 30th international conference on evaluation and assessment in software engineering, 2026. [6] S. Scalabrino, M. Linares-Vásquez, R. Oliveto, and D. Poshyvanyk, “A comprehensive model for code readability,” Journal of Software: Evolution and Process, vol. 30, no. 6, p. e1958, 2018. [7] R. P. Buse and W. R. Weimer, “Learning a metric for code readability,” IEEE Transactions on software engineering, vol. 36, no. 4, pp. 546–558, 2009. [8] F. Dalpiaz, I. Van Der Schalk, S. Brinkkemper, F. B. Aydemir, and G. Lucassen, “Detecting terminological ambiguity in user stories: Tool and experimentation,” Information and Software Technology, vol. 110, pp. 3–16, 2019. [9] J. Frattini, D. Fucci, R. Torkar, L. Montgomery, M. Unterkalmsteiner, J. Fischbach, and D. Mendez, “Applying bayesian data analysis for causal inference about requirements quality: A controlled experiment,” Empirical Software Engineering, vol. 30, no. 1, p. 29, 2024. [10] D. Gopstein, J. Iannacone, Y. Yan, L. DeLong, Y. Zhuang, M. K.-C. Yeh, and J. Cappos, “Understanding misunderstandings in source code,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 129–139. [11] V. R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sørumgård, and M. V. Zelkowitz, “The empirical investigation of perspective-based reading,” Empirical software engineering, vol. 1, no. 2, pp. 133–164, 1996. [12] S. Ajami, Y. Woodbridge, and D. G. Feitelson, “Syntax, predicates, idioms—what really affects code complexity?” Empirical Software Engineering, vol. 24, no. 1, pp. 287–328, 2019. [13] A. Vogelsang, A. Korn, G. Broccia, A. Ferrari, J. Fischbach, and C. Arora, “On the impact of requirements smells in prompts: The case of automated traceability,” in 2025 IEEE/ACM 47th International Conference on Software Engineering: New Ideas and Emerging Results, 2025, pp. 51–55. [14] H. Femmer, D. M. Fernández, S. Wagner, and S. Eder, “Rapid quality assurance with Requirements Smells,” Journal of Systems and Software, vol. 123, pp. 190–213, 2017. [15] J. Frattini, “Identifying relevant factors of requirements quality: An industrial case study,” in International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 2024, pp. 20–36. [16] D. M. Fernández, S. Wagner, M. Kalinowski, M. Felderer, P. Mafra, A. Vetrò, T. Conte, M.-T. Christiansson, D. Greer, C. Lassenius, T. Männistö, M. Nayabi, M. Oivo, B. Penzenstadler, D. Pfahl, R. Prikladnicki, G. Ruhe, A. Schekelmann, S. Sen, R. Spinola, A. Tuzcu, J. L. De La Vara, and R. Wieringa, “Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice,” Empirical Software Engineering, vol. 22, no. 5, pp. 2298–2338, 2017. [17] L. Montgomery, D. Fucci, A. Bouraffa, L. Scholz, and W. Maalej, “Empirical research on requirements quality: A systematic mapping study,” Requirements Engineering, vol. 27, no. 2, pp. 183–209, 2022.

[18] E. Gentili and D. Falessi, “Practitioners’ perceptions on requirements smells,” Information and Software Technology, vol. 187, p. 107823, 2025. [19] D. La Prova, E. Gentili, and D. Falessi, “Anticipating bugs: Ticketlevel bug prediction and temporal proximity effects,” Empirical Software Engineering, vol. 31, no. 2, p. 43, 2026. [20] R. McElreath, Statistical rethinking: A Bayesian course with examples in R and Stan. Chapman and Hall/CRC, 2018. [21] J. Siebert, “Applications of statistical causal inference in software engineering,” Inf. Softw. Technol., vol. 159, p. 107198, 2023. [22] X. Franch, D. Mendez, A. Vogelsang, R. Heldal, E. Knauss, M. Oriol, G. H. Travassos, J. C. Carver, and T. Zimmermann, “How do Practitioners Perceive the Relevance of Requirements Engineering Research?” IEEE Transactions on Software Engineering, vol. 48, no. 6, pp. 1947– 1964, 2022. [23] H. Femmer, J. Kučera, and A. Vetrò, “On the impact of passive voice requirements on domain modelling,” in Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement, 2014, pp. 1–4. [24] J. Krisch and F. Houdek, “The myth of bad passive voice and weak words an empirical investigation in the automotive industry,” in 2015 IEEE 23rd international requirements engineering conference (RE). IEEE, 2015, pp. 344–351. [25] I. Hadar, P. Soffer, and K. Kenzi, “The role of domain knowledge in requirements elicitation via interviews: An exploratory study,” Requirements Engineering, vol. 19, no. 2, pp. 143–159, 2014. [26] M. A. Zadenoori, J. Dabrowski, W. Alhoshan, L. Zhao, and A. Ferrari, “Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review,” 2025. [27] M. Krishna, B. Gaur, A. Verma, and P. Jalote, “Using LLMs in Software Requirements Specifications: An Empirical Evaluation,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 475–483. [28] G. Quattrocchi, L. Pasquale, P. Spoletini, and L. Baresi, “Can LLMs Generate User Stories and Assess Their Quality?” IEEE Transactions on Software Engineering, vol. 52, no. 5, pp. 1773–1790, 2026. [29] M. Mircea, E. Gevrek, E. Schmid, and K. Schneider, “Supporting Stakeholder Requirements Expression with LLM Revisions: An Empirical Evaluation,” in International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 2026, pp. 303–319. [30] A. Fantechi, S. Gnesi, L. Passaro, and L. Semini, “Inconsistency Detection in Natural Language Requirements using ChatGPT: A Preliminary Evaluation,” in 2023 IEEE 31st International Requirements Engineering Conference (RE). IEEE, 2023, pp. 335–340. [31] A. El-Hajjami and C. Salinesi, “Synthline: A Product Line Approach for Synthetic Requirements Engineering Data Generation using Large Language Models,” in International Conference on Research Challenges in Information Science. Springer, 2025, pp. 208–225. [32] H. Femmer, F. Houdek, M. Unterbusch, and A. Vogelsang, “Description and Comparative Analysis of QuRE: A New Industrial Requirements Quality Dataset,” in 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW). IEEE, 2025, pp. 23–29. [33] T. S. Behrend and R. N. Landers, “Participant interactions with artificial intelligence: Using large language models to generate research materials for surveys and experiments,” Journal of Business and Psychology, pp. 1–23, 2025. [34] A. Jedlitschka, M. Ciolkowski, and D. Pfahl, “Reporting experiments in software engineering,” in Guide to Advanced Empirical Software Engineering, F. Shull, J. Singer, and D. I. K. Sjøberg, Eds. London: Springer London, 2008, pp. 201–228. [35] B. Carvalho da Silva, R. Nascimento, G. Braz Junior, L. Rivero, L. E. G. Martins, R. Pereira dos Santos, and D. Viana, “Promise+: A dataset of labeled software requirements,” 2024. [Online]. Available: https://zenodo.org/records/12805484 [36] F. Dalpiaz, “Requirements data sets (user stories),” 2018. [Online]. Available: https://zenodo.org/records/13880060 [37] J. Frattini, L. Montgomery, J. Fischbach, M. Unterkalmsteiner, D. Mendez, and D. Fucci, “A Live Extensible Ontology of Quality Factors for Textual Requirements,” in 2022 IEEE 30th International Requirements Engineering Conference (RE), 2022, pp. 274–280. [38] D. Brunato, L. De Mattei, F. Dell’Orletta, B. Iavarone, and G. Venturi, “Is this sentence difficult? do you agree?” in Conference on Empirical

Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2018, pp. 2690–2699. [39] C. A. Furia, R. Feldt, and R. Torkar, “Bayesian data analysis in empirical software engineering research,” IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1786–1810, 2019. [40] L. Gren and R. Berntsson Svensson, “Is it possible to disregard obsolete requirements? a family of experiments in software effort estimation,” Requirements Engineering, vol. 26, no. 3, pp. 459–480, 2021. [41] J. Pearl, “Causal inference in statistics: An overview,” Statistics Surveys, 2009. [42] M. M. Glymour and S. Greenland, “Causal diagrams,” in Modern Epidemiology. Lippincott, 2008, ch. 12, pp. 183–209. [43] C. Cinelli, A. Forney, and J. Pearl, “A crash course in good and bad controls,” Sociological Methods & Research, vol. 53, no. 3, pp. 1071– 1104, 2024. [44] E. T. Jaynes, Probability Theory: The Logic of Science. Cambridge University Press, 2012. [45] P.-C. Bürkner and M. Vuorre, “Ordinal regression models in psychology: A tutorial,” Advances in Methods and Practices in Psychological Science, vol. 2, no. 1, pp. 77–101, 2019. [46] N. A. Ernst, “Bayesian hierarchical modelling for tailoring metric thresholds,” in Proceedings of the 15th international conference on mining software repositories, 2018, pp. 587–591. [47] J. S. Wesner and J. P. Pomeranz, “Choosing priors in Bayesian ecological models by simulating from the prior predictive distribution,” Ecosphere, vol. 12, no. 9, pp. e03 739:1–e03 739:11, 2021. [48] S. Brooks, A. Gelman, G. Jones, and X.-L. Meng, Handbook of Markov Chain Monte Carlo. CRC, 2011. [49] C. Cinelli and C. Hazlett, “Making sense of sensitivity: Extending omitted variable bias,” Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 82, no. 1, pp. 39–67, 2020. [50] C. A. Furia and R. Torkar, “Mitigating omitted variable bias in empirical software engineering,” Empirical Software Engineering, vol. 31, no. 5, p. 123, 2026. [51] D. Y. Lin, B. M. Psaty, and R. A. Kronmal, “Assessing the sensitivity of regression results to unmeasured confounders in observational studies,” Biometrics, pp. 948–963, 1998. [52] C. Mood, “Logistic regression: Why we cannot do what we think we can do, and what we can do about it,” European sociological review, vol. 26, no. 1, pp. 67–82, 2010. [53] A. Korn, L. Zaruchas, C. Arora, A. Metzger, S. Smolka, F. Wang, and A. Vogelsang, “Reporting llm prompting in automated software engineering: A guideline based on current practices and expectations,” in Proceedings of the 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering, 2026, p. 184–194. [54] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024.

Related documents

• How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community#927947
• Object-Oriented Software Engineering#1222808
• How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community#660886
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1345191
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1336059