Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection

Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection Giovanna Brocciaa #

Julian Frattinia #

CNR–ISTI, Pisa, Italy

Chalmers University of Technology, Gothenburg, Sweden

a

corresponding author

arXiv:2608.21298v1 [cs.SE] 21 Aug 2026

a

corresponding author

Chetan Arora

Maurice H. ter Beek

Monash University, Melbourne, Australia

CNR–ISTI, Pisa, Italy

Alessandro Fantechi

Andreas Vogelsang

University of Florence, Florence, Italy

University of Duisburg-Essen

Alessio Ferrari Trinity College Dublin, Dublin, Ireland

Abstract Background. Requirements inspection (RI) is a well-established practice for detecting potential defects in requirements artifacts early in the software lifecycle. Recent advances in large language models (LLMs) have stimulated interest in their potential to support requirements engineering (RE) tasks. However, empirical evidence on the effects of LLMs when used as collaborative assistants in human-performed RI remains scarce. Aims. We aim to investigate the impact of LLM support on human-performed RI, considering inspection effectiveness in terms of smell identification and severity classification (i.e., nocuous vs innocuous), as well as inspection duration. Method. We conducted a controlled crossover design experiment with 34 participants, who inspected textual specifications with and without LLM support, identifying and classifying requirements smells while recording inspection time. We analyzed the data using one Bayesian regression model per outcome variable, accounting for validity threats induced by the crossover design as well as covariates and mediators. Results. Results show that LLM support negatively affects smell detection accuracy but has no significant effect on smell classification or task duration. A learning effect is present across experimental periods, but reduced when RI is first performed with LLM support. Conclusions. Our findings provide empirical evidence that LLM support does not necessarily improve performance and may, instead, hinder it for novice inspectors. Moreover, the results suggest that learning RI with LLM-support from the beginning may slow down the skill acquisition process, implying threats for LLM-supported learning. 2012 ACM Subject Classification Software and its engineering → Requirements analysis; Software and its engineering → Empirical software validation Keywords and phrases Human-AI collaboration, Requirements inspection, Requirements smells, Large Language Models, ChatGPT, Empirical study Digital Object Identifier 10.4230/LIPIcs...

1

Introduction

Software inspection is a well-established quality assurance practice that involves the systematic examination of software artifacts with the objective of identifying defects early in the development lifecycle. Since its formalization in the mid-1980s, inspection has been recognized as an effective and cost-efficient technique for improving software quality, reducing overall development costs [19, 18]. Inspections can be applied before executable artifacts exist, © Giovanna Broccia and Julian Frattini and Chetan Arora and Maurice H. ter Beek and Alessandro Fantechi and Andreas Vogelsang and Alessio Ferrari; licensed under Creative Commons License CC-BY 4.0 Leibniz International Proceedings in Informatics Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Germany

XX:2

Human-AI Collaboration in Requirements Engineering

making them particularly valuable for detecting defects in requirements artifacts during the requirements engineering (RE) phase. Requirements inspection (RI)—a specialized form of inspection focused on these artifacts—plays a critical role in ensuring the quality, as defects introduced at the RE stage tend to propagate downstream and become expensive to fix [44]. In this context, the notion of requirements smells has gained increasing attention. Requirements smells are indicators of potential quality issues—such as ambiguity, vagueness, or unverifiability—that may hinder understanding or lead to incorrect interpretations [23]. Although not all smells correspond to actual defects, they serve as early warning signals of violations of desirable quality properties. Distinguishing between nocuous (i.e., harmful) and innocuous smells is therefore essential, as it allows inspectors to prioritize issues more likely to negatively affect downstream activities [15]. Recent advances in generative AI and large language models (LLMs) have stimulated increasing interest in their use to support RE activities. Prior work has investigated LLM-based approaches for analyzing and transforming requirements artifacts, including applications to code generation [40], test scenario generation [3, 59], and the derivation of UML sequence diagrams from natural-language requirements [24], among others. More recently, Vogelsang et al. [58] investigated the impact of requirements smells on LLM performance in requirements-to-code traceability, further highlighting the sensitivity of LLM-based techniques to requirements quality. However, existing studies primarily evaluate model performance and output quality [62]. In contrast, empirical evidence on how human-LLM collaboration, where LLMs serve as supportive assistants in human-performed inspection, affects inspection performance remains largely unexplored. This gap is particularly critical for novice requirements analysts, who risk overreliance on LLMs, potentially hindering skill development. At the same time, organizations are increasingly adopting AI-assisted onboarding to compensate for the limited availability of senior experts for traditional mentoring1 . To address this gap, we conducted a controlled crossover experiment [57] in which students acting as novice inspectors performed RI tasks with and without ChatGPT (GPT-4o/GPT-4.1 models). Treating students as novice requirements analysts is common in empirical RE studies [6, 5]. ChatGPT was selected among the available LLMs, given preliminary evidence of its potential [22]. Our study focuses on the identification and severity classification of requirements smells (i.e., nocuous vs. innocuous), as well as on inspection duration. Our results indicate that using ChatGPT as a support tool affects requirements smell detection negatively, while it has no significant effect on smell severity classification. Moreover, inspection performance is influenced by a learning effect: participants generally perform better when executing the task for the second time. However, when the inspection task is first performed with ChatGPT support, this learning effect is reduced across periods. We also observe that longer inspection times are unexpectedly associated with poorer performance. Finally, smell detection and classification performance are positively correlated, indicating that participants who identify more smells also tend to classify them more accurately. The main contributions of this work are: A novel empirical investigation of the impact of using LLMs for RI. Evidence of learning and order effects in LLM-supported RI by novice inspectors. Practical insights into how (not) to use LLMs for RI by novice analysts. The remainder of this paper is structured as follows. Section 2 introduces background concepts on RI, requirements smells, and the classification of nocuous and innocuous smells, 1

https://newsletter.getdx.com/p/ai-cuts-developer-onboarding-time-in-half. Visited 22 Jan 2026.

G. Broccia et al.

and reviews related work. Section 3 describes the experimental design. Section 4 presents the results, while Section 5 discusses the findings. Section 6 discusses threats to validity. Finally, Section 7 concludes the paper and outlines directions for future work.

2

Background and Related Work

2.1

Requirements Inspection and Smells

Software inspection comprises a family of structured review techniques aimed at identifying violations of quality attributes in software artefacts [19]. When applied to RE, inspection focuses on systematically examining requirements artifacts to identify defects, inconsistencies, and indicators of poor quality that may compromise downstream development activities. Detecting defects in requirements artifacts early in the software lifecycle is critical to preventing their propagation into later development phases, where they can cause rework, delays, and quality issues in downstream activities, and where their correction becomes increasingly difficult and costly [44, 52]. In this context, requirements smells have been introduced as observable indicators of potential quality issues in requirements artifacts [23]. Requirements smells do not necessarily correspond to defects themselves, but signal situations that may hinder understanding, introduce ambiguity, or increase the likelihood of misinterpretation. As such, smells provide a practical means to guide inspection activities by drawing attention to requirements that warrant closer scrutiny. In 2022, a catalogue of 206 requirements quality factors (i.e., smells) was published [25], derived from a systematic mapping study of 105 relevant papers [45]. Frattini et al. categorized these smells into three main classes: (i) lexical smells, related to words or terms that may be misinterpreted; (ii) syntactic smells, with problematic word(s) or sentence structures; and (iii) semantic smells, which concern the interpretation of requirements within their usage context. Among smells, it is important to distinguish between benign issues and those likely to cause misunderstandings and defects. In the early 2000s, Chantree et al. introduced the distinction between nocuous and innocuous phenomena [15]. Nocuous issues are likely to result in divergent stakeholder interpretations and, consequently, incorrect or inconsistent implementations. In contrast, innocuous issues may exhibit potentially problematic characteristics at a superficial level, yet are typically interpreted consistently and are therefore unlikely to cause misunderstandings. Accordingly, while smells indicate potential quality problems, not all necessarily require corrective action. Some may be easily disambiguated through context and domain knowledge and can therefore be considered innocuous [39]. Others may give rise to multiple plausible interpretations and thus represent a nocuous risk if left unaddressed [27]. Distinguishing between nocuous and innocuous smells allows inspectors to prioritize issues more likely to negatively affect downstream development activities.

2.2

Related Work

Recent research has increasingly investigated the use of LLMs to support requirements quality assurance. A systematic literature review by Wolf et al. synthesizes 26 peer-reviewed studies (2019–2025) on AI-based requirements quality assessment, highlighting a shift since 2023 toward LLM-driven approaches and semantically richer assessments [62]. Part of the works treat LLM as quality defect detectors [21, 42, 50], while others partially take into account also the human perspective, and the implications of using LLM for inspection [9, 7, 51, 41].

XX:3

XX:4

Human-AI Collaboration in Requirements Engineering

LLM as Detectors. In the first group, Fantechi et al. [21] explored the use of GPT-3.5 for detecting inconsistencies in natural-language requirements. Their study evaluated the LLM’s predictions against reference assessments provided by students. The results show that the model can identify certain inconsistencies, providing evidence of feasibility, while also highlighting variability in detection accuracy depending on inconsistency type and requirements formulation. Mahbub et al. [42] evaluated GPT-4’s performance in detecting multiple classes of defects in requirements artifacts, including ambiguities, inconsistencies, and incompleteness. Their study quantitatively assessed the model’s precision across these defect types and different requirements versions. The results indicate that GPT-4 performs better in identifying incomplete requirements than in detecting inconsistencies and ambiguities. The authors concluded that, despite promising results, LLMs cannot replace human analysts in zero-shot settings due to limitations in domain-specific reasoning. Raj et al. [50] proposed a framework that leverages LLMs to identify, categorize, and resolve ambiguities in software requirements artifacts. The evaluation focuses on the LLM’s ability to identify ambiguity types and generate alternative formulations on a small evaluation set, providing preliminary evidence of LLMs’ potential to support ambiguity-related quality improvement. LLM and the Human Perspective. In this group, Lubos et al. [41] investigated the use of an LLM to evaluate software requirements quality according to the ISO/IEC/IEEE 29148 standard. Their approach employs the LLM to assess quality characteristics, explain its judgments, and propose improved versions of requirements. The approach was assessed with the involvement of software engineers, who examined the usefulness of the LLM-supported analysis. The results suggest that LLMs can provide valuable support for requirements quality assessment and improvement, particularly through explanations and suggested refinements. Bashir et al. [7] empirically studied the use of LLMs for ambiguity detection and explanation in industrial requirements. The authors evaluated performance across three industrial datasets and reported a 20.2% average improvement in classification performance when using ten-shot prompting compared to zero-shot settings. In addition, they conducted human evaluations with industry experts, who assessed the generated explanations across dimensions including naturalness, adequacy, usefulness, and relevance, with an average score of 3.84/5. Boukhlif et al. [9] examined the use of LLMs to identify ambiguities, inconsistencies, and gaps in software requirements, with a particular focus on implications for software testing. The study compared LLM-based analysis with conventional SRS analysis techniques using quantitative metrics, as well as qualitative assessment of generated insights. The results indicate that LLMs can improve the accuracy and efficiency of requirements analysis, particularly in detecting ambiguous terms and uncovering inconsistencies that traditional approaches may overlook. Finally, Seifert et al. [51] report a replication of an inspection experiment on requirements to assess whether LLMs can compete with human reviewers. They evaluate GPT-4-Turbo as the LLM reviewer. While the results indicate that LLMs can identify certain issues, the authors note that comparability with the original human inspection setting is constrained by necessary adaptations (e.g., how the LLM is trained and the material provided), and thus the results should be interpreted with care. Contributions. Overall, existing work provides valuable empirical evidence on the potential of LLMs to identify and reason about quality issues in textual requirements. However, prior research largely centers on evaluating LLM outputs—often with human involvement limited to assessing the correctness, usefulness, or plausibility of model-generated results—

G. Broccia et al.

rather than examining their role within human-centered inspection processes. Human-inthe-loop scenarios are, however, closer to realistic industrial practice, where LLMs are increasingly viewed as support tools that assist rather than fully replace human reasoning and decision making in software engineering activities [46, 56]. This is particularly important in mission-critical contexts, where requirements quality and human oversight remain crucial. Furthermore, the use of chat-based LLMs is inherently iterative and dialogic, enabling a collaborative sensemaking process in which humans refine prompts, assess suggestions, and apply domain knowledge to inspection decisions. Unlike prior work, this study adopts this realistic human-in-the-loop perspective by evaluating inspector performance when using an LLM as a support tool during RI, focusing on effectiveness, learning and order effects, and inspection duration. By adopting this perspective, our work complements existing research and addresses an important gap in empirical evidence on human–AI collaboration in RI.

3

Study Design

The experimental design and procedures are described in accordance with the guidelines reported by Jedlitschka et al. [38] and Vegas et al. [57].

3.1

Goal and Research Questions

The overall goal of the experiment is as follows: Understanding the potential of using LLMs to support RI for novice inspectors, and the impact of LLM-use on their learning behavior. To achieve the goal, our aim is to answer the following research questions (RQs): RQ1 What is the impact of LLM support on the time required to perform RI? RQ2 What is the impact of LLM support on the accuracy of requirements smell identification? RQ3 What is the impact of LLM support on the accuracy of classifying identified requirements smells as nocuous?

3.2

Selection of Subjects

Participants were BSc’s students in Computer Engineering at the University of Florence, enrolled in the Industrial Computer Science course. They were opportunistically recruited based on availability. Participation was voluntary and granted a bonus toward the course exam; however, students could achieve high exam scores regardless of participation through the standard examination process. Prior to the experiment, all participants were informed about the study goals and procedures and provided written informed consent. They were explicitly informed that the collected data would be used exclusively for research purposes, analysed in an anonymous and aggregated form, and that no link between their identity and the recorded responses would be retained. Participants were divided into two experimental groups (group 1 and group 2, both containing 17 participants each). When allocating participants to groups, we blocked on the factors proficiency in RI and proficiency in using LLMs, collected during a pre-test questionnaire (cf. Section 3.3) on a five-point ordinal scale. They were assigned to groups using optimal matching based on Mahalanobis distance over these two variables. Ultimately, both experimental groups contained a similiar number of participants (with a maximum difference of 2) for each of the scales’ five levels of both factors. This way, the effect of these factors is evenly dispersed among the groups and does not confound the effect of interest.

XX:5

XX:6

Human-AI Collaboration in Requirements Engineering

Table 1 Types of considered smells Smell categories and types Lexical smells subjective language

optional parts weak verbs Syntactic smells vague pronouns passive voice negative phrases Semantic smells logical inconsistencies numerical discrepancies ambiguities

3.3

Description Words of which the semantics are not objectively defined (e.g., user-friendly, easy to use, cost effective). Sentences expressing personal opinions or feelings. Sentences containing optional parts, e.g., by using words such as possibly, eventually, if possible, if needed, etc. Weak verbs, such as can, could, may, etc. Pronouns that refer back to a previous part of the text for which the reference is unclear. Sentences using passive voice such that it is unclear who is performing a certain action. Sentences containing negative expressions or modifiers (e.g., not). Two requirements, which are connected to the same concepts, contradicting each other. Two requirements connected to the same concepts, containing inconsistent and/or contradicting numerical information. Unclear/imprecise sentence parts that can be misunderstood if read by different people.

Experimental Objects and Tasks

The experiment was conducted over two sessions, during which participants performed RI tasks. To perform the tasks, participants were provided with (i) a requirements document, (ii) a reference document providing definitions and examples of the selected smell types and the nocuous/innocuous distinction, and (iii) a test document to support RI. Participants were asked to inspect each requirement in the provided requirements document and determine whether it exhibited one of the predefined smell types; classify identified smells as nocuous or innocuous; record the start and end times of the inspection task. Additionally, participants were provided with (iv) a pre-test questionnaire before the inspection sessions, and (v) post-test questionnaires after each inspection session. (i) Requirements document. The documents used in the two sessions contained a set of requirements for two distinct video game implementations: the Arkanoid game in the first session and the Snake game in the second session. These documents contain smelly and non-smelly requirements and were designed to be comparable in terms of size and number of smells (i.e., Arkanoid: 40 requirements, 21 smells; Snake: 39 requirements, 19 smells). The smells contained cover the nine smell types from [25] listed in Table 1. We also classified each smell as nocuous or innocuous. The requirements documents were extended versions of two documents previously used in [58]. Starting from these original documents, one of the authors extended the documents and injected additional smells to ensure coverage of all nine selected smell types, with each smell type appearing at least once across the documents and each requirement containing at most one smell2 . The same author also performed an initial nocuous/innocuous classification of the injected smells. Subsequently, two additional authors independently inspected the

2

The choice of constraining each requirement to contain at most one smell, although not fully representative of real-world requirements documents, was made to simplify both the inspection task for participants and the subsequent analysis. In particular, allowing multiple smells per requirement would have increased annotation ambiguity for participants and complicated the computation and interpretation of performance measures (e.g., precision and recall).

G. Broccia et al.

Figure 1 Excerpt of the test document. Red annotations indicate example expected completions and were not shown to participants.

requirements documents, identifying the presence of smells and classifying them as nocuous or innocuous. The three authors then jointly reviewed all identified discrepancies through an iterative, consensus-oriented discussion process until a shared decision was reached for each requirement. Given the inherently subjective and context-dependent nature of requirements smells and their perceived severity [33], we adopted a dialogic reconciliation process rather than relying on formal inter-rater agreement measures. This process was applied consistently to both smell identification and nocuous/innocuous classification. Based on the final agreedupon version of the documents, a reference annotation was established for each requirement, specifying the presence or absence of a smell, the smell type, and its nocuous or innocuous classification. This annotated version of the documents served as the ground truth for the subsequent analysis of participants’ inspection performance. Although the initial annotation from one of the authors could have biased the ground truth, we argue that the multiple iterations and re-evaluations mitigate this possible bias. (ii) Reference document. Participants received a reference document containing the definition of each smell type, along with two illustrative examples and their corresponding explanations. The document also included the definition of the nocuous and innocuous classes, together with two examples for each class. (iii) Test document. Participants received a spreadsheet to support the inspection task (an excerpt of which is shown in Fig. 1). Each row corresponded to a single requirement, while columns represented the nine selected smell types, allowing participants to indicate whether a given requirement exhibited a specific smell. An additional field was provided to classify the identified smell as nocuous or innocuous. The spreadsheet also included dedicated fields for recording the start and end time of the inspection task. (iv) Pre-test questionnaire. Before the inspection task, participants completed a pre-test questionnaire collecting demographic and background information. The questionnaire gathered the identifier, gender, age, and the highest degree completed by each participant. In addition, participants self-reported their proficiency in RI and in using LLMs using a five-point scale (1–5). For participants who reported proficiency in RI greater than 1, the questionnaire also collected information about the types of RI tasks with which they were familiar and the contexts in which they had previously performed such tasks. (v) Post-test questionnaire. After each inspection session, participants completed a post-test questionnaire tailored to the experimental condition they experienced, distinguishing between LLM-supported and unsupported inspections and between period 1 and period 2.

XX:7

XX:8

Human-AI Collaboration in Requirements Engineering

Table 2 List of variables involved in the study Variable

Name

Type

Description

Scale

llm

LLM-Support

exposure

{gpt, nogpt}

duration

Task Duration

outcome

detection

Detection Accuracy

outcome

classification

Classification Accuracy

outcome

period

Experimental Period Treatment Sequence

design

Whether or not an LLM was used during RI Number of minutes taken to complete the task Macro-averaged F1 -score computed across all smell types for smell detection F1 -score of correctly classifying requirements smells as nocuous, computed only on correctly detected smells Time slot of the observation

Participant ID Proficiency using LLMs Proficiency in RI Proficiency in English GPT Usage

design covariate

sequence

subject prof.llm prof.ri prof.english gpt.use

design

covariate covariate covariate

Order in which the participants performed the task with/without LLM-support ID of the participant Self-reported proficiency in using LLMs Self-reported proficiency in performing RI tasks Self-reported proficiency in the English language Self-reported number of requirements for which the LLM was used during the inspection task (when available)

N0 [0; 1]

[0; 1]

{period1, period2} {group1, group2}

1–34 {1, 2, 3, 4, 5} {1, 2, 3, 4, 5} {1, 2, 3, 4, 5} {0, 1–10, 11–20, 21–35, 36–40}

Collectively, across all sessions and experimental conditions, the post-test questionnaires gathered qualitative comments and feedback on the inspection task, participants’ preferences between the two variants (LLM-supported vs. non-supported), and self-reported proficiency in English. When LLM support was used, participants reported the LLM version and the number of requirements inspected with LLM assistance.

3.4

Choice of Design and Variables

To achieve the objectives of the study, we conducted a crossover experiment [57]. The experiment was carried out over two periods (period 1 and period 2), separated by a oneweek washout interval. In period 1, group 1 performed the RI task without the support of ChatGPT, while group 2 performed the same task with ChatGPT support; in period 2, the exposure conditions were reversed. During the study (May 2025), participants used ChatGPT models from the GPT-4o and GPT-4.1 families, with model availability depending on subscription tier and quota limitations. Each period involved the inspection of a different requirements document (cf. Section 3.3). Table 2 lists all variables involved in the study. The Type-column differentiates the main factor of interest (exposure), the three dependent variables (outcome), the factors induced by the crossover-design (design), and additional factors that may affect the outcomes (covariate).

G. Broccia et al.

3.5

Experimental Procedure

3.5.1

Experimental Phases

The experiment began with a training phase in which participants attended an introductory lesson designed to provide all the information necessary to perform the inspection task successfully. The lesson covered the following topics: RI, requirements smells, the distinction between nocuous and innocuous smells, and basic principles of prompt engineering. To increase engagement and motivation, the lesson incorporated gamification elements by framing the activity as a simulated hiring scenario in which participants were employees of a video game company competing for the “employee of the month” distinction. Following the introductory lesson, participants were asked to complete an online pre-test questionnaire no later than three days before the first inspection task session. One week after the introductory lesson, participants entered the inspection phase, which constituted the core phase of the experiment. This phase was conducted over two sessions (period 1 and period 2), separated by a one-week washout interval. The inspection tasks were performed without time constraints. Participants initiated the inspection during an in-class session and were allowed to complete it outside the classroom if necessary, provided that they accurately recorded the start and end times of the task. Although this variability in working conditions reduces control, this choice was made to prioritize data completeness and avoid comparisons between fully and partially completed inspections. Only two participants continued after the in-class session. After each inspection task session, participants uploaded the completed test document to a shared online folder labeled with their participant ID and completed an online post-test questionnaire, which varied according to the experimental period and assigned condition. The pre-test and post-test questionnaires were administered through an online survey platform.

3.5.2

Measurement Procedure

To quantify participants’ performance in identifying requirements smells, we calculated effectiveness metrics by comparing each participant’s annotations with the ground truth. Each participant completed the RI task by marking, for each requirement, the presence or absence of a predefined set of smell types. The comparison with the ground truth was performed at the level of individual smell types. Specifically, for each participant and for each smell type, we computed an F1 -score by treating the identification of that smell across all requirements as a binary classification problem. True positives, false positives, and false negatives were derived by aligning participant annotations and ground-truth labels based on requirement identifiers. Participant-level performance was summarized by computing the macro-averaged F1 -score across smell types. This metric is common in the evaluation of RE tasks [8]. Although Fβ can give a more nuanced and practically relevant viewpoint [8], the computation of the actual value of β would have required a dedicated experimental evaluation, and it is left for future work. Classification performance, again evaluated against the ground truth, was computed only for smells that were correctly identified, as classification presupposes successful detection; including undetected smells would conflate detection errors with classification errors and would therefore not reflect classification ability per se. For each participant, classification effectiveness was computed by treating the assignment of the nocuous label as a binary classification problem over the set of correctly identified smells. Specifically, true positives, false positives, and false negatives were derived by aligning participants’ nocuous classifications with the ground-truth nocuous labels. We focused on the nocuous class because correctly identifying nocuous smells is critical in practice, as these smells are more likely to lead to

XX:9

XX:10

Human-AI Collaboration in Requirements Engineering

Period Sequence

duration

Subject Proficiency LLM

LLM Usage LLM−support

Proficiency English Proficiency ReqInspection

Figure 2 DAG for RQ1

misunderstandings and downstream defects if left unaddressed. Participant-level classification performance was summarized using the F1 -score. The same considerations about Fβ discussed above apply also for this task.

3.6

Data Analysis

To analyze the collected data, we utilized Bayesian data analysis (BDA) within the framework for statistical causal inference (SCI) [53, 48]. BDA techniques are more complex to apply than their frequentist counterpart, but offer richer insights that are more intuitive to interpret and more robust against uncertainty [29, 26]. In our particular analysis, BDA allows us to properly model the outcome variables using a distribution family with maximum entropy [37]. Whereas frequentist methods differentiate only normally and non-normally distributed outcomes, BDA distinguishes many more distribution families [35]. Since none of our outcome variables follow a normal distribution, BDA is the the more appropriate choice [43]. The SCI framework ensures that identified effects are causal and not correlational, within the scope of the collected data [48]. For the overarching SCI framework, we adhered to the three-step process proposed by Siebert [53], which is based on the seminal work by Pearl [48]. Despite SCI principles being particularly effective for analyzing observational data [30], they also support analyzing experimental data by distinguishing total from direct effects [27] and clearly guide the analysis in crossover-design experiments [57, 28]. For BDA, we followed established textbooks by McElreath [43] and Gelman et al. [31].

3.6.1

Modeling

In the first step proposed by Siebert [53] titled modeling, we explicitly encoded our causal assumptions in a directed, acyclic graph (DAG) [34]. In a DAG, nodes represent relevant variables and directed edges assumed causal relationships between these variables. For each RQ, we modeled one DAG centered around the relationship between the exposure (i.e., the use of ChatGPT) and the respective outcome variable (i.e., duration, detection performance, and classification performance). Fig. 2 shows the DAG for RQ1 with the exposure colored red and the outcome cyan. The DAGs for RQ2 and RQ3 differ only (1) by the respective outcome variable, and (2) by including duration as a mediator variable.

3.6.2

Identification

Based on the DAG for each RQ, we performed the second step of the SCI framework [53] named identification. During this step, we identified all variables beyond the main exposure (LLM-support) that we need to statistically control in order to avoid any bias. To this end,

G. Broccia et al.

we applied d-separation [48], which identified the adjustment set of variables necessary to de-confound a causal effect of interest. In the exemplary DAG for RQ1 in Fig. 2, the variables period and sequence confound the effect of LLM-support on duration, and hence, must be included in the adjustment set [48]. The variable LLM-usage acts as a mediator, which we included to isolate the direct effect of interest [43]. Finally, the remaining variables act as simple covariates, which we included to improve the precision of the effect estimation [16].

3.6.3

Estimation

With the adjustment set determined for each RQ, we conducted the final estimation step [53] to estimate the direction and strength of the effect that the statistically controlled variables have on the outcome. For this process, we followed common guidelines for BDA [43, 31]. We started by selecting a distribution family with maximum entropy for each of the three outcome variables [37]: We treated every outcome variable as a random variable and selected the least restrictive distribution family that is consistent with all epistemological and ontological assumptions [43]. This yielded two different distribution families: The outcome variable duration was modelled as a negative binomial (or: gamma Poisson) distribution as the variable represents a discrete count (in number of minutes) with overdispersion (VMR = 7.2 > 1). The outcome variables detection performance and classification performance were both modeled as beta distributions since they are measured as F1 -scores bounded between 0 and 1. Given the existence of 0.0 and 1.0 scores occurring in both variables, we use the zero-one-inflated beta distribution variant specifically. Next, we designed a Bayesian regression model per RQ. We specified the shape-defining parameter of each distribution with a linear model containing all variables from the adjustment set. For example, the regression model for RQ1 is defined as follows: durationi ∼ NegBinom(ni , pi ) logit(pi ) ← α + αID + βllm · llmi + βllm×prof.llm · llmi · prof.llmi + . . . The . . . represent the remaining factors included in the model, i.e., all of the remaining factors listed in Table 2 and shown in Fig. 2. For brevity of the presentation, we obscured them in the manuscript. The regression model logit(pi ) ← . . . consists of several components: A population-level intercept α representing the general difficulty of the experimental task A group-level intercept αID representing the general skill of a participant. This models subject-level variability [57]. Population-level effects βX · Xi , where Xi is the value of variable X (e.g., the use of ChatGPT, labelled llm) in observation i and βX is a probability distribution representing the effect that these values have on the outcome variable. Interaction effects βX×Y Xi · Yi where one variable Y moderates the effect of another variable X on the outcome. Every intercept (α and αID ) and effect coefficient (all of βX and βX×Y ) is a probability distribution of the effect strength of the respective factor on the outcome variable. To each of these, we assign a prior probability distribution, i.e., encode our assumptions of feasible effect strengths prior to observing the data [43]. Given the lack of strong empirical evidence for either of these factors, we use uninformative priors like N (µ = 0, σ = 1) [60], i.e., normal distributions centered around 0 with a wide variance 1. This represents the belief that

XX:11

XX:12

Human-AI Collaboration in Requirements Engineering

nogpt

nogpt

gpt

gpt 40

60

80

100

120

average duration in minutes

(a) Effect on task duration

0.1

0.2

0.3

0.4

average F1 − score

0.5

(b) Effect on detection accuracy

Figure 3 Marginal effects of LLM-support

any factor could have either a positive (µ > 0), negative (µ < 0), or nonexistent (µ ≈ 0) effect on the outcome. To assess the eligibility of the selected priors, we perform prior predictive checks [60]. In this process, we sample from the Bayesian regression model without updating the priors via the collected data. The results of this prior predictive check visually showed that the range of potential outcomes encompass the actual outcomes, confirming the plausibility of the priors. With the priors’ eligibility confirmed, we continued with model training. We implemented the analysis using the R library brms [12] acting as an interface to the statistical modelling language STAN [14]. During the training process, Hamiltonian Monte Carlo Markov Chains (MCMC) updated the coefficient distributions based on the collected data [11]. This way, the parameters of the coefficient distributions (e.g., µ and σ) were adjusted to better reflect the outcome variable based on the predictor variable values. After training, we assessed several diagnostics to ensure the validity of the training process. The Gelman-Rubin convergence statistic showed acceptable values R̂ ∈]0.99, 1.01[ for all coefficients [32]. Similarly, the n proportion of effective sample sizes was above the threshold neff > 0.1 for all coefficients [43]. We observed no divergent transitions, raising concerns about the training process. Finally, we performed posterior predictive checks by drawing samples from the now-updated coefficient distributions, which confirmed that the training process moved the estimates closer and tighter to the observed values. Note that, since our goal is not out-of-sample prediction but rather causal inference, we do not perform train-test splits on the available data. To obtain interpretable results on the outcome space, we calculated marginal and conditional effects and plotted their distribution. Marginal and conditional effects are estimated from the model by varying only one or two factors while holding all others constant at representative levels. This way, the isolated effect of that factor on the outcome variable becomes clear, showing how it affects the outcome variable.

4

Study Results

The following sections present the results obtained from the analysis about whether the use of ChatGPT in RI impacts the duration of the task (RQ1 in Section 4.1), the accuracy of detecting smells (RQ2 in Section 4.2), and the accuracy of correctly classifying correctly detected smells as nocuous (RQ3 in Section 4.3). Due to the page limitation, this section reports only meaningful results. All descriptive statistics, DAGs, analyses, results, and figures can be found in our replication package.

4.1

RQ1: Impact on Task Duration

Fig. 3a shows the marginal effect of LLM-support on the task duration. The mean duration values (points in Fig. 3a) are close, and the 95%-credibility intervals (whiskers in Fig. 3a) overlap, indicating that the use of ChatGPT does not significantly change the task duration.

XX:13

Period Group2 Period 1 Group1

Period 2 0.1

0.2

0.3

0.4

0.5

average F1 − score

Sequence

G. Broccia et al.

0.5 0.4 0.3 0.2 0.1

average F1 − score

(a) Conditional effect of the experimental period and sequence

−1

0

1

2

average duration (standardized by µ=54.2 and σ=19.7)

(b) Marginal effect of the duration

Figure 4 Marginal effects of design variables on detection accuracy

Apart from that, no other predictor in the linear model for task duration showed a significant effect. At most, the period variable shows a slight decrease in duration in Period 2 (by about 5 minutes). This decrease is not significant but expected as an maturation effect observed in crossover-design experiments [57].

4.2

RQ2: Impact on Detection Accuracy

Fig. 3b shows the marginal effect of LLM-support on the accuracy of detecting requirements smells. Observations from the control group (i.e., participants not using ChatGPT) achieved about 8% higher F1 -scores on average. The total uncertainty around these averages, represented by the 95%-credibility intervals, still overlaps. Fig. 4a visualizes the conditional effect of both period and sequence on the detection accuracy. Observations in the second period achieved better F1 -scores by more than 10% in both sequences. This clearly supports a maturation effect typical for crossover-design experiments [57, 28]. Across both periods, group 1 (who performed the task without ChatGPT in period 1) reached slightly higher accuracy, suggesting an optimal sequence [57]. Significantly though, the conditional effect of both period and sequence shows a clear effect. Participants in group 1, who started the inspection task without ChatGPT support, exhibited a greater maturation effect—a ∼ 12% improvement in performance between periods 1 and 2—compared to participants in group 2, who started with ChatGPT support and improved by only ∼ 7%. Fig. 4a shows this through a greater distance between the accuracy average between the two periods in group 1 than group 2. This is a clear indication of a carryover effect possible in crossover-design experiments [57]. Additionally, longer task durations are associated with lower F1 -scores, as seen in Fig. 4b. None of the other covariates show an effect on the detection accuracy.

4.3

RQ3: Impact on Classification Accuracy

Fig. 5a shows that using ChatGPT has virtually no effect on accuracy when classifying correctly detected smells as nocuous. Again, the classification accuracy also improves between the periods as visualized in Fig. 5b, though this effect is less strong. Similarly, there is a notable yet weaker carryover effect where the F1 -score of classifying smells in group 1 (starting without ChatGPT) improves by about 5% between periods and in group 2 only by about 3%. Also, longer task durations produce worse classification accuracy. All other covariates do not show any significant effect.

XX:14

Human-AI Collaboration in Requirements Engineering

Period 2

nogpt

Period 1

gpt 0.4

0.5

0.6

average F1 − score

(a) Marginal effect of LLM-usage

0.7

0.4

0.5

0.6

average F1 − score

0.7

(b) Marginal effect of the experimental period

Figure 5 Marginal effects of design variables on classification accuracy

 Answer to RQs When using LLMs for RI, participants took equally long time (RQ1) but detected smells less precisely with ∆F1 ≈ −8% (RQ2). Notably, performing the task first with and later without LLMs reduced the learning effect (i.e., the maturation-based improvement of F1 -scores between periods) by half (6% instead of 12% improvement). The severity classification accuracy remains unaffected by LLM usage (RQ3).

5

Discussion

Contrary to the common expectation that LLMs may uniformly improve performance in RE tasks raised by prior work studying LLMs in isolation [42, 7, 9], the results highlight a more nuanced picture. In particular, while ChatGPT support did not significantly affect inspection time or smell classification accuracy, it was associated with decrease in smell detection accuracy and reduced learning effect when inspection was first performed with LLM support. Based on these findings, we discuss relevant observations and implications.

5.1

Key Observations

Potential Over-Reliance and Automation Bias. The negative effect of LLM support on smell detection accuracy suggests that, for novice inspectors, the availability of an LLM may interfere with careful, systematic requirements analysis. One plausible explanation is that participants may have delegated part of the inspection effort to the LLM, relying on its responses instead of engaging deeply with the requirements. This aligns with concerns raised in prior research about over-reliance on AI-generated suggestions and automation bias, particularly when users lack strong domain expertise [47, 13]. Similar concerns have been reported in qualitative studies of human–LLM collaboration in other software engineering activities. Hamza et al. observed that, during a hands-on workshop with professional software engineers using ChatGPT, participants sometimes exhibited social loafing and excessive reliance on the AI, reducing intellectual engagement and critical thinking [36]. Distinct Cognitive Processes in Smell Detection and Severity Classification. Interestingly, LLM support did not significantly influence smell severity classification accuracy once smells were correctly identified. This suggests that, when participants successfully detected a smell, they were generally able to reason about its severity (nocuous vs. innocuous) regardless of LLM support. A similar pattern was observed in prior work on RI, where inspectors provided accurate justifications once a violation had been correctly identified, independently of the supporting representation [10]. Classification may therefore rely more on conceptual understanding of quality risks than on the mechanical detection of smell indicators. This distinction underscores the importance of treating detection and classification as two

G. Broccia et al.

cognitively distinct activities in RI. Reduced Skill Development with Early LLM Support. The observed learning effect across experimental periods aligns with established methodological practice in crossover designs, which requires learning effects to be explicitly examined and accounted for [57, 28]. Participants performed better in the second inspection session, indicating that repeated exposure to the task led to improved effectiveness. However, the reduced learning effect observed when the inspection task was first performed with LLM support is particularly relevant. It suggests that early reliance on LLMs may hinder the development of inspection skills, potentially by short-circuiting the reflective processes through which novice inspectors internalize inspection strategies and quality heuristics. Longer Duration Does Not Imply Better Performance. The negative association between longer inspection duration and performance indicates that spending more time on the task did not translate into higher accuracy. Rather than reflecting thoroughness, longer durations may signal uncertainty, lack of strategy, or difficulties in understanding the requirements. These difficulties may have been exacerbated by participants’ self-reported moderate English proficiency (average 3.5 on a 1–5 scale), which could have hindered requirements comprehension. This further supports the interpretation that effective inspection depends more on structured reasoning than on time investment alone.

5.2

Implications for Practice, Research, and Teaching

Implication for Practice. For practitioners, these results caution against uncritical adoption of LLMs as support tools in RI, especially for novice inspectors. While LLMs can generate plausible explanations and suggestions, their use may inadvertently reduce inspectors’ engagement with the requirements and weaken detection performance. Organizations introducing LLMs into inspection workflows should therefore consider restricting their use to specific phases (e.g., post-inspection review or justification refinement) rather than as a primary aid during initial defect detection. Similar concerns about the impact of early generative AI use on skill development have also been reported in software development contexts, where developers expressed worries that novices would become too reliant on generative AI tools, missing opportunities to develop coding and problem-solving skills [54]. The lack of time savings suggests that LLM support does not necessarily lead to efficiency gains in inspection tasks. Practitioners should not assume that LLM-assisted inspection will be faster or more effective without careful evaluation in their specific context. Similar patterns have been observed in other software engineering activities. Stray et al. [55] report that the introduction of GitHub Copilot was not associated with substantial, consistently measurable productivity gains in development tasks, despite developers’ strong perception of increased efficiency. This gap between perceived and observed benefits reinforces our findings that LLM support should not be assumed to yield performance or efficiency improvements without careful, context-specific evaluation. On the other hand, LLMs may still support onboarding novice engineers when positioned as aids for learning and orientation rather than substitutes for human reasoning or task execution. Recent studies suggest that LLMs can help newcomers navigate documentation, reduce information overload, and explore unfamiliar technical material in a self-directed manner. However, effective onboarding still requires sustained human mentoring and supervision [4, 2]. Implications For Research. From a research perspective, this study contributes empirical evidence to the still underexplored area of human–LLM collaboration in RI. The results highlight the need to move beyond evaluating LLM output quality in isolation and toward studying how LLMs shape human behavior, learning, and decision-making processes. Future

XX:15

XX:16

Human-AI Collaboration in Requirements Engineering

research should investigate alternative integration strategies, such as constrained or staged LLM usage, and examine whether different prompting styles, explanations, or transparency mechanisms mitigate the negative effects observed here. Replication studies with professional inspectors and more complex industrial requirements are needed to assess the generalizability of these findings and to understand how expertise moderates the impact of LLM support. Implications for Teaching. The findings also have important implications for teaching RE. Introducing LLMs too early in inspection training may limit students’ opportunities to develop foundational inspection skills and critical reading abilities. Educators should therefore be cautious when integrating LLMs into RE curricula and consider first grounding students in manual inspection techniques before exposing them to AI-based assistance. These implications are consistent with recent calls for human-centered software engineering education in the AI era, which warn that premature or unstructured use of generative AI tools may undermine foundational learning, reflective practice, and skill development if not carefully integrated into curricula [1]. Complementary empirical evidence emerges from studies of novice interaction with LLMs in educational software engineering contexts. Rahe and Maalej report that programming students using ChatGPT frequently delegated substantial portions of problem-solving to the LLM, often entering repetitive cycles of generating, copying, and submitting solutions without engaging in deeper reasoning [49]. Their analysis also shows that increased interaction time did not translate into improved performance. Despite longer time on task and high perceived usefulness, no corresponding learning or effectiveness gains were observed. These findings closely mirror our observations in RI, reinforcing the concern that unstructured LLM support may reduce cognitive engagement and hinder learning in novice users rather than enhancing inspection effectiveness. At the same time, LLMs can still play a valuable pedagogical role if used deliberately, for example, as tools for discussion, reflection, or comparison after an initial manual inspection. Used in this way, LLMs may help students articulate reasoning, understand alternative interpretations, and reflect on quality issues without replacing essential learning processes.

6

Threats to Validity

We discuss the threats to validity that apply to this study following Wohlin et al. [61]. Construct Validity. Several covariates (e.g., proficiency in using LLMs and proficiency in RI) may suffer from limited construct validity. These self-reported Likert-scale variables are coarse proxies for the underlying constructs of interest. Given the difficulty of operationalizing such latent concepts, we resorted to common practice of self-reporting on coarse scales. The absence of significant effects for these covariates on the outcome variables may therefore stem from this limited construct validity, and we refrain from drawing conclusions about their impact on the experimental task. The construction of the ground truth may also threaten construct validity. Although the annotation process involved independent inspections by multiple authors followed by iterative consensus-oriented reconciliation discussions, no formal inter-rater agreement metrics were computed. We mitigated this threat through collective review and discussion until a shared agreement was reached for each requirement. Internal Validity. The data collection did not include explicit measures of participants’ general skill or motivation, both of which are likely to affect all outcome variables. In RQ2 and RQ3, this threatens internal validity, as skill or motivation may confound the relationship between duration and the other outcome variables. Because highly skilled or motivated students are likely to both perform faster and better, the absence of an explicit variable for skill or motivation induces a spurious association between those variables [43].

G. Broccia et al.

Fig. 4b supports this interpretation: longer task durations are counterintuitively associated with lower detection accuracy. We therefore assume this association to be non-causal and confounded by skill or motivation. Since such latent constructs are difficult to operationalize and using proxies such as student grades raises ethical concerns, we chose not to collect this data. Allowing participants to continue the inspection task outside the classroom may also have introduced variability in working conditions and reduced experimental control. However, this choice was made to prioritize data completeness and avoid comparisons between fully and partially completed inspections. In practice, this occurred only for two participants during period 1 (2.9% of the total inspection task executions). External Validity. The obtained results are subject to several threats to external validity, which limit the scope of the conclusions. Specifically, the experiment does not fully reflect real RI practice in three dimensions: the participants (i.e., experimental subjects), the requirements (i.e., experimental objects), and the RI task itself (i.e., the experimental task). Participants were bachelor students in Computer Engineering, without extensive practical experience in RI. While “using students as participants remains a valid simplification of reality needed in laboratory contexts” [20], the observed effects may differ for experienced industrial practitioners with established inspection strategies and domain knowledge. Similarly, the requirements used for inspection were simpler than industrial requirements to remain comprehensible within the scope of the experiment. In addition, each requirement contained at most one smell. While this simplified annotation and performance analysis, industrial requirements may contain multiple interacting smells, limiting generalizability to more complex settings. Finally, the RI task was limited to the detection and classification of requirements smells and relied on a predefined set of smell types rather than fully open-ended identification. Although these smell categories were selected from empirically validated taxonomies grounded in industrial practice [25], this choice may have influenced participants’ inspection behavior and simplified the task compared to real-world RI settings. These simplifications were necessary to make the experiment feasible, but limit the external validity of the results beyond novice inspectors in controlled educational contexts. Conclusion Validity. The conclusion validity of our results is threatened by two aspects of the data analysis. Firstly, the sample size of 34 participants is relatively limited. Secondly, the analysis involves subjective choices, including the selection of a probability distribution family with maximum entropy, linear model parameters, and prior probability distributions. However, Bayesian data analysis mitigates these threats in two ways. First, it naturally preserves uncertainty in the results, which is reflected in the credibility intervals [29]. Second, it makes all analysis steps explicit, including subjective decisions, allowing other researchers to review and revise them if necessary.

7

Conclusions

This paper presented a controlled crossover-design experiment investigating the effects of LLM support on human-performed RI. Focusing on requirements smell identification, nocuous/innocuous classification, and inspection time, the study provides evidence that LLM support does not necessarily improve inspection outcomes for novice inspectors. In particular, the results show that LLM support negatively affects smell detection accuracy, has no significant effect on smell classification or task duration, and is associated with a reduced learning effect when used from the outset. These findings suggest that early reliance on LLMs may hinder, rather than support, the development of systematic inspection skills. Future work should investigate alternative ways of integrating LLMs into RI workflows,

XX:17

XX:18

Human-AI Collaboration in Requirements Engineering

such as constrained, staged, or post-hoc usage, to mitigate the negative effects observed in this study. Replications with professional inspectors and more complex industrial requirements are needed to assess the generalizability of the findings and to understand how expertise moderates the impact of LLM support. Further research should also explore how different interaction designs, prompting strategies, and explanation mechanisms influence both inspection effectiveness and learning, with the goal of identifying configurations in which LLMs can support human inspection skills.

8

Data Availability

The replication package is publicly available at [17]. References 1 2

3

4

5 6 7 8 9

10 11 12 13 14 15 16 17

Silvia Abrahão et al. Software engineering by and for humans in an AI era. ACM Trans. Softw. Eng. Methodol., 34(5):129:1–129:46, 2025. doi:10.1145/3715111. Elijah Kayode Adejumo and Brittany Johnson. Towards leveraging LLMs for reducing open source onboarding information overload. In Proceedings ASE, pages 2210–2214. ACM, 2024. doi:10.1145/3691620.3695286. Chetan Arora, Tomas Herda, and Verena Homm. Generating test scenarios from NL requirements using retrieval-augmented LLMs: An industrial study. In Proceedings RE, pages 240–251. IEEE, 2024. doi:10.1109/re59067.2024.00031. Maider Azanza et al. Can LLMs facilitate onboarding software developers? An ongoing industrial case study. In Proceedings CSEE&T, pages 1–6. IEEE, 2024. doi:10.1109/ CSEET62301.2024.10662989. Muneera Bano et al. Learning from mistakes: An empirical study of elicitation interviews performed by novices. In Proceedings RE, pages 182–193. IEEE, 2018. doi:10.1109/RE.2018.00027. Muneera Bano et al. Inspectors academy: Pedagogical design for requirements inspection training. In Proceedings RE, pages 215–226. IEEE, 2020. doi:10.1109/RE48521.2020.00032. Sarmad Bashir et al. Requirements ambiguity detection and explanation with LLMs: an industrial study. InProceedingsICSME, pages 620–631.IEEE,2025. doi:10.1109/icsme64153.2025.00063. Daniel M. Berry. Empirical evaluation of tools for hairy requirements engineering tasks. Empir. Softw. Eng., 26(6):111:1–11:77, 2021. doi:10.1007/s10664-021-09986-0. Mohamed Boukhlif et al. Using LLMs to analyze software requirements for software testing: A comparative study. In Proceedings ICSBT, volume 1330 of LNNS, pages 413–421. Springer, 2024. doi:10.1007/978-3-031-86698-2_37. Giovanna Broccia, Sira Vegas, and Alessio Ferrari. The role of cognitive abilities in requirements inspection: Comparing UML and textual representations, 2026. arXiv:2601.16009. Steve Brooks et al. Handbook of Markov Chain Monte Carlo. CRC, 2011. doi:10.1201/b10905. Paul-Christian Bürkner. brms: An R package for Bayesian multilevel models using Stan. J. Stat. Softw., 80(1):1–28, 2017. doi:10.18637/jss.v080.i01. Irina Carnat. Human, all too human: accounting for automation bias in generative large language models. Int. Data Priv. Law, 14(4):299–314, 2024. doi:10.1093/idpl/ipae018. Bob Carpenter et al. Stan: A probabilistic programming language. J. Stat. Softw., 76(1):1–32, 2017. doi:10.18637/jss.v076.i01. Francis Chantree et al. Identifying nocuous ambiguities in natural language requirements. In Proceedings RE, pages 56–65. IEEE, 2006. doi:10.1109/re.2006.31. Carlos Cinelli, Andrew Forney, and Judea Pearl. A crash course in good and bad controls. Sociol. Methods Res., 53(3):1071–1104, 2024. doi:10.1177/00491241221099552. Broccia Giovanna et al. An LLM as requirements inspection assistant? A controlled experiment showing negative effects for novice inspectors – replication package, 2026. doi:10.5281/zenodo. 18360108.

G. Broccia et al.

18 19 20

21

22

23 24

25 26

27

28

29 30

31 32 33

34 35

36 37 38

Michael Fagan. A history of software inspections. In Software Pioneers: Contributions to Software Engineering, pages 562–573. Springer, 2011. doi:10.1007/978-3-642-59412-0_34. Michael E. Fagan. Advances in software inspections. IEEE Trans. Softw. Eng., 12(7):744–751, 1986. doi:10.1109/tse.1986.6312976. Davide Falessi et al. Empirical software engineering experts on the use of students and professionals in experiments. Empir. Softw. Eng., 23(1):452–489, 2018. doi:10.1007/ s10664-017-9523-3. Alessandro Fantechi et al. Inconsistency detection in natural language requirements using ChatGPT: a preliminary evaluation. In Proceedings RE, pages 335–340. IEEE, 2023. doi: 10.1109/re57278.2023.00045. Alessandro Fantechi, Stefania Gnesi, and Laura Semini. Rule-based NLP vs ChatGPT in ambiguity detection, a preliminary study. In Proceedings REFSQ-JP, volume 3378 of CEURWS, pages 1:1–1:10, 2023. URL: https://ceur-ws.org/Vol-3378/NLP4RE-paper1.pdf. Henning Femmer et al. Rapid quality assurance with Requirements Smells. J. Syst. Softw., 123:190–213, 2017. doi:10.1016/j.jss.2016.02.047. Alessio Ferrari, Sallam Abualhaijal, and Chetan Arora. Model generation with LLMs: From requirements to UML sequence diagrams. In Proceedings REW, pages 291–300. IEEE, 2024. doi:10.1109/rew61692.2024.00044. Julian Frattini et al. A live extensible ontology of quality factors for textual requirements. In Proceedings RE, pages 274–280. IEEE, 2022. doi:10.1109/re54965.2022.00041. Julian Frattini et al. A second look at the impact of passive voice requirements on domain modeling: Bayesian reanalysis of an experiment. In Proceedings WSESE, pages 27–33. ACM, 2024. doi:10.1145/3643664.3648211. Julian Frattini et al. Applying bayesian data analysis for causal inference about requirements quality: a controlled experiment. Empir. Softw. Eng., 30:29:1–29:50, 2025. doi:10.1007/ s10664-024-10582-1. Julian Frattini, Davide Fucci, and Sira Vegas. Crossover designs in software engineering experiments: Review of the state of analysis. In Proceedings ESEM, pages 482–488. ACM, 2024. doi:10.1145/3674805.3690754. Carlo A. Furia, Robert Feldt, and Richard Torkar. Bayesian data analysis in empirical software engineering research. IEEE Trans. Softw. Eng., 47(9), 2021. doi:10.1109/tse.2019.2935974. Carlo A. Furia, Richard Torkar, and Robert Feldt. Applying Bayesian analysis guidelines to empirical software engineering data: The case of programming languages and code quality. ACM Trans. Softw. Eng. Methodol., 31(3):40:1–40:38, 2022. doi:10.1145/3490953. Andrew Gelman et al. Bayesian Data Analysis. CRC, 2013. doi:10.1201/b16018. Andrew Gelman and Donald B. Rubin. Inference from iterative simulation using multiple sequences. Statist. Sci., 7(4):457–472, 1992. doi:10.1214/ss/1177011136. Vincenzo Gervasi et al. Ambiguity in requirements engineering: Towards a unifying framework. In From Software Engineering to Formal Methods and Tools, and Back, volume 11865 of LNCS, pages 191–210. Springer, 2019. doi:10.1007/978-3-030-30985-5_12. M. Maria Glymour and Sander Greenland. Causal diagrams. In Modern Epidemiology, chapter 12, pages 183–209. Lippincott, 2008. Lucas Gren and Richard Berntsson Svensson. Is it possible to disregard obsolete requirements? a family of experiments in software effort estimation. Requir. Eng., 26(3):459–480, 2021. doi:10.1007/s00766-021-00351-7. Muhammad Hamza et al. Human-AI collaboration in software engineering: Lessons learned from a hands-on workshop. In Proceedings IWSiB, pages 7–14, 2024. doi:10.1145/3643690.3648236. Edwin T. Jaynes. Probability Theory: The Logic of Science. Cambridge University Press, 2012. doi:10.1017/cbO9780511790423. Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. Reporting experiments in software engineering. In Guide to Advanced Empirical Software Engineering, pages 201–228. Springer, 2008. doi:10.1007/978-1-84800-044-5_8.

XX:19

XX:20

Human-AI Collaboration in Requirements Engineering

39

40 41 42

43 44

45 46 47

48 49 50

51

52 53 54

55 56 57

58

59

Jennifer Krisch and Frank Houdek. The myth of bad passive voice and weak words: An empirical investigation in the automotive industry. In Proceedings RE, pages 344–351. IEEE, 2015. doi:10.1109/RE.2015.7320451. Hui Liu et al. Deep learning based program generation from requirements text: Are we there yet? IEEE Trans. Softw. Eng., 48(4):1268–1289, 2022. doi:10.1109/tse.2020.3018481. Sebastian Lubos et al. Leveraging LLMs for the quality assurance of software requirements. In Proceedings RE, pages 389–397. IEEE, 2024. doi:10.1109/re59067.2024.00046. Taslim Mahbub et al. Can GPT-4 aid in detecting ambiguities, inconsistencies, and incompleteness in requirements analysis? A comprehensive case study. IEEE Access, 12:171972–171992, 2024. doi:10.1109/access.2024.3464242. Richard McElreath. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. CRC, 2018. doi:10.1201/9781315372495. Daniel Méndez Fernández et al. Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice. Empir. softw. Eng., 22(5):2298–2338, 2017. doi: 10.1007/s10664-016-9451-7. Lloyd Montgomery et al. Empirical research on requirements quality: a systematic mapping study. Requir. Eng., 27(2):183–209, 2022. doi:10.1007/s00766-021-00367-z. Sriraam Natarajan et al. Human-in-the-loop or AI-in-the-loop? Automate or collaborate? In Proceedings AAAI, volume 39, pages 28594–28600, 2025. doi:10.1609/aaai.v39i27.35083. Samir Passi and Mihaela Vorvoreanu. Overreliance on AI: Literature review. Technical Report MSR-TR-2022-12, Microsoft, June 2022. URL: https://www.microsoft.com/en-us/ research/publication/overreliance-on-ai-literature-review/. Judea Pearl. Causal inference in statistics: An overview. Statist. Surv., 3:96–146, 2009. doi:10.1214/09-ss057. Christian Rahe and Walid Maalej. How do programming students use generative AI? Proc. ACM Softw. Eng., 2(FSE):978–1000, 2025. doi:10.1145/3715762. Ankit Raj et al. Enhancing software requirements quality: Ambiguity detection and resolution using large language models. In Proceedings CSCI, volume 2505 of CCIS, pages 340–355. Springer, 2024. doi:10.1007/978-3-031-95127-5_25. Daniel Seifert et al. Can large language models (LLMs) compete with human requirements reviewers? - Replication of an inspection experiment on requirements documents. In Proceedings PROFES, volume 15452 of LNCS, pages 27–42. Springer, 2024. doi: 10.1007/978-3-031-78386-9_3. Forrest Shull, Ioana Rus, and Victor Basili. How perspective-based reading can improve requirements inspections. IEEE Comput., 33(7):73–79, 2000. doi:10.1109/2.869376. Julien Siebert. Applications of statistical causal inference in software engineering. Inf. Softw. Technol., 159:107198, 2023. doi:10.1016/j.infsof.2023.107198. Viktoria Stray, Astri Barbala, and Viggo Tellefsen Wivestad. Human-AI collaboration in software development: A mixed-methods study of developers’ use of GitHub Copilot and ChatGPT. In Proceedings FSE, pages 1325–1332. ACM, 2025. doi:10.1145/3696630.3730566. Viktoria Stray et al. Developer productivity with and without GitHub Copilot: A longitudinal mixed-methods case study, 2025. arXiv:2509.20353. Wannita Takerngsaksiri et al. Human-in-the-loop software development agents. In Proceedings ICSE-SEIP, pages 342–352. IEEE, 2025. doi:10.1109/ICSE-SEIP66354.2025.00036. Sira Vegas, Cecilia Apa, and Natalia Juristo. Crossover designs in software engineering experiments: Benefits and perils. IEEE Trans. Softw. Eng., 42(2):120–135, 2016. doi: 10.1109/tse.2015.2467378. Andreas Vogelsang et al. On the impact of requirements smells in prompts: The case of automated traceability. In Proceedings ICSE-NIER, pages 51–55. IEEE, 2025. doi: 10.1109/ICSE-NIER66352.2025.00016. Fanyu Wang et al. Requirements-driven automated software testing: A systematic review. ACM Trans. Softw. Eng. Methodol., 2025. doi:10.1145/3767739.

G. Broccia et al.

60

61 62

Jeff S. Wesner and Justin P.F. Pomeranz. Choosing priors in Bayesian ecological models by simulating from the prior predictive distribution. Ecosphere, 12(9):e03739:1–e03739:11, 2021. doi:10.1002/ecs2.3739. Claes Wohlin et al. Experimentation in Software Engineering. Springer, 2024. doi:10.1007/ 978-3-662-69306-3. Elise Wolf, Adam Trendowicz, and Julien Siebert. Quality assessment of software requirements using artificial intelligence methods: A systematic literature review. Inf. Softw. Technol., 191:107979, 2025. doi:10.1016/j.infsof.2025.107979.

XX:21

Related documents

• Two Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and Misses#925587
• Understanding the GDPR from a requirements engineering perspective—a systematic mapping study on regulatory data protection requirements#29946
• Epic-Organized vs. Requirement-Aligned Gherkin: An Empirical Evaluation of LLM-Based Acceptance Criteria Generation#1106006
• A Scenario-Based Model for Analyzing the Ethical Requirements of Intelligent Autonomous Systems#696836
• The Prominent Role of Humanities in the Development and Application of Artificial Intelligence#110593