On the Viability of Requirements Generation From Code: An Experience Report Alexander Korn, Jone Bartel, Max Unterbusch, Andreas Vogelsang
arXiv:2606.25550v1 [cs.SE] 24 Jun 2026
paluno – The Ruhr Institute for Software Technology University of Duisburg-Essen Essen, Germany {firstname}.{lastname}@uni-due.de
Abstract—Empirical research in Requirements Engineering is hampered by a lack of adequate datasets that pair source code with corresponding requirements. A tempting route to addressing this lack is the use of Large Language Models to synthesize requirements from existing code bases. We investigate this question by evaluating an LLM-based and RAG-supported agentic approach that generates requirements from source code, verifies their implementation status relying on a human-in-theloop, and synthetically introduces requirements smells and nonimplemented requirements. Our goal was to create datasets that mimic reality and foster empirical RE research. However, during the study, various problems arose, leading to this experience report. Contrary to our initial hypotheses, LLMs were unable to (i) generate non-implemented requirements reliably, (ii) generate high quality requirements, and (iii) reliably introduce synthetic requirements smells. Furthermore, neither an LLM nor a single human-in-the-loop suffices to detect requirements smells reliably. These findings suggest that the generation of code-torequirements datasets using LLMs is not yet viable and requires human supervision, especially for quality assurance. We critically reflect on our lessons learned and draw relevant conclusions for both researchers and practitioners. Index Terms—Agentic AI, LLMs, Synthetic Datasets, Requirements Smells
I. I NTRODUCTION Empirical research in Requirements Engineering (RE) is based on the quality of the data used. To best support research, data should (i) be FAIR (findable, accessible, interoperable, reusable) [1], (ii) contain a realistic number of artifacts, (iii) contain high-quality requirements, and yet (iv) be realistic in quality [2], so that real-world imperfections can also be researched. For supporting research on the relation between requirements and code [3], datasets should also (v) contain relevant trace links, e.g. requirements-to-code traces. Although there are datasets satisfying some of these dimensions, there is a lack of datasets satisfying all dimensions (cf. Section II). Ideally, industry datasets could be used, which would render (iv) useless. However, industry data can rarely be disclosed for proprietary or legal reasons, leading to a large number of academic or toy datasets used in RE research [4]. Due to significant advances in the performance of Large Language Models (LLMs), researchers started using LLMs to generate or synthesize textual artifacts such as code and requirements [5], [6] or even requirements traceability data [7]–
[9]. However, the generated datasets show deficiencies in quality and realism. Our initial idea was to propose an agentic human-in-theloop approach that synthesizes requirements from existing code bases and link them to the corresponding code locations. Specifically, we wanted to use Retrieval-Augmented Generation (RAG) to ground the requirements in code, presumably leading to high-quality requirements with clear links to code. We operationalize requirements quality by using requirements smells as quantifiable proxies for quality issues in requirements [10], [11]. We hypothesized that the generated requirements would show a low smell rate and are, in fact, implemented in the source code. We further hypothesized that we could extend the approach to generate a control group of non-implemented requirements with similar properties to the implemented requirements. Finally, we planned to extend the approach by an agent that introduces requirements smells synthetically, such that a realistic dataset would be generated with explicit labeling of requirements smells. In this experience report, we describe how we implemented and evaluated this approach and finally had to withdraw most of our hypothesis. In contrast, we observe the following problems: • The supposedly high-quality requirements still exhibited a smell rate of 24.5%. • 73.8% of the smell-free but non-implemented requirements generated were marked as implemented by a human evaluator, showing that the LLM often generates implemented requirements instead. • The inter-rater agreement for smell detection and appropriateness evaluation of generated requirements showed fair agreement at best. • Smell evaluation by the human-in-the-loop was highly subjective and context dependent. In this experience report, we describe our approach, the evaluation, the results, and reflect on these. The key contributions are: • An empirical evaluation of our approach across 2 software projects, examining implementation accuracy, hallucination rate, generation quality, and human-LLM reliability on smell detection. • Concrete lessons learned on the limitations of LLMbased requirements generation, specifically around smell
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
labeling subjectivity, the LLM complying with the given prompts, and codebase-dependency of the results. • Two publicly available requirements-to-code datasets generated and peer-reviewed during the study. DATA AVAILABILITY All data used in this study, including the requirements-tocode datasets generated during the evaluation, the experimental implementation of the presented approach, and the analysis code used to process the study results, are available in the supplementary material1 . II. R ELATED W ORK Related work stems from two areas: requirement-to-codetraceability and the more recent requirement-from-code generation. A. Requirement-to-code Datasets Requirement-and-code datasets provide requirements along with the specific code traces that implement them. Although their applicability is broad, the most common use is by the traceability community interested in creating or recovering traceability links from requirements to source code [3]. Zoogan et al. [12] list all datasets used for requirements traceability up until 2017. A subset of this list is relevant as requirement-and-code datasets. Disregarding publicly unavailable datasets, e.g., non-disclosable industry data sets, and infrequently used datasets, the remaining datasets embody a trend towards standard datasets being re-used as baselines and for better comparability2 . These standard datasets including iTrust, eTour, SMOS, and eANCI, indeed are frequently used in more recent studies [9], [13]–[17]. The code-andrequirement datasets provide a golden set of requirements and their trace links to code that were manually crafted and hence, are usually small in size. Even combined, they do not suffice to provide sufficient amounts of training data for Deep Learning (DL) methods [6], [18]. Additionally, these ideal requirements-and-code datasets deviate from the mixed-quality in real-world requirement datasets [2]. Hence, we hypothesized that datasets are more realistic if we use synthetically introduce requirements smells. B. Requirements Generation from Code Xu et al. [7] show the feasibility of generating requirements from source code. They use existing requirement-to-code datasets to finetune GPT-models and then generate requirements for the application of legacy systems. Dearstyne [8] tested the generation of requirements from source code as one of the four applied problems in requirements traceability. Practitioners find that automatically generated requirements are comparable in quality to their handcrafted requirements. Persson et al. [19] propose Code2Req, which allows users to generate requirements from code. Their approach is based on RAG-supported LLMs and includes existing requirements 1 Supplementary material: https://doi.org/10.6084/m9.figshare.32393787 2 We show a tabular view of this list in our supplementary material.
sets as context information. They also show the feasibility of automatic requirement generation from code using LLMs, although they describe it as not yet practically valuable. Jin et al. [13] focus on generating requirements from code with the aim of facilitating the understanding and verification of code generated by LLMs. These approaches do not consider two major aspects: First, they aim to produce exclusively high-quality requirements, rather than generating a dataset that reflects the characteristics of real-world requirements, which are of mixed quality [2]. Dearstyne’s work tries to correct for that by incorporating existing sets of requirements from the given codebase as context to the LLMs. However, this is not enough since (i) an idealized set of context requirements leads to the initial problem of unrealistic requirements, and (ii) a realistic set of context requirements leads to an inexplicit quality decrease in output requirements, as reported in her work. Without explicitly knowing the location of these qualitative deficiencies, they are of little use to research. Second, their approaches are fully automated without human validation or feedback. In contrast, when producing datasets intended for downstream research or project work, an additional verification and validation step is required to ensure the structural and content correctness of the resulting artifacts. This human feedback can also be used to guide the requirements generation process. III. A PPROACH This section describes the approach used for our study. Figure 1 illustrates the complete architecture. A. Agentic Approach Generator Agent. The generator agent is responsible for producing batches of n requirements (in our experiments: n = 10). To accomplish this, the agent employs RAG to retrieve code chunks from the target project’s code base. For each batch, we invoke the generator separately n times, each time producing a candidate requirement. The system prompt deliberately refrains from requiring the generated requirement to be guaranteed as implemented or valid, as this is the responsibility of the subsequent verifier agent. Instead, the prompt provides guidance on the characteristics of wellformed requirements and conventions for how requirements should be expressed. Furthermore, the LLM is instructed to output a rationale explaining the basis for the candidate requirement, the code files deemed relevant, the names of the functions that supposedly implement the requirement, and a confidence value. The user prompt supplies the LLM with a list of all previously generated requirements, labeled by the human-in-the-loop (cf. Section III-C). The exact prompts are provided in the supplementary material1 . As RAG returns only the most relevant results for a given query, we employ sharding (cf. the work of Shao et al. [20]), which enables systematic retrieval of different regions of the code base. Each chunk is assigned a random shard from 00–99 as metadata. At each generator invocation, a random
Generation Sample random shards Block chunks
Generator agent Found new requirement?
Trace link recovery agent RAG vector store Batch of requirements
trace links
requirement
no
reject decides yes
Human-inthe-loop Verifier agent
accept
adds synthetic smells to requirements
Support requirement?
Smell mutation agent
Label as rejected
Add labels pass shots to generator
Shot pool
Fig. 1. An overview of the complete agentic architecture of our experimental approach.
subset of three candidate chunks is selected, restricting retrieval to chunks belonging to that subset. If the generator cannot identify novel requirements within those chunks, it is instructed to return no requirement. In this case, the shard subset is resampled and the generator re-invoked. If no more requirements can be extracted from a chunk, it is marked as saturated and excluded from future subsets. We sample the shards randomly to allow for a diverse set of requirements to be generated. As a control group for evaluation, the generator is additionally tasked to generate a configurable number of nonimplemented requirements (20 % in our experiments). In this case, the prompt instructs the LLM to reconstruct requirements for which the agent does not see clear evidence about their implementation. Verifier Agent. This agent is tasked with determining whether a previously generated requirement candidate is actually implemented in the code base. Unlike the generator, the verifier is permitted to search the entire code base with RAG, as the most relevant retrieval results are expected to concentrate around the candidate requirement. We prompt the LLM to classify the retrieved evidence as either (a) supporting, (b) contradicting, or (c) insufficient for the candidate requirement. To retrieve a diverse and representative set of evidence chunks, the verifier issues 3 targeted RAG queries: (i) the top-4 chunks retrieved by the candidate requirement text, capturing functional intent, (ii) the top-3 chunks retrieved by the function names identified by the generator, targeting likely implementation sites, and (iii) the top-3 chunks retrieved by the file names identified by the generator. We determined this query constellation to be productive in initial experiments. Beyond its verdict (supporting, contradicting, or insufficient), the verifier is also instructed to return a rationale,
the relevant files and functions, and a confidence value. The system prompt additionally includes explicit evaluation rules to guide the LLM in distinguishing between the verdict categories. In the user prompt, the verifier is provided with the candidate requirement, and the files, functions, and rationale returned by the generator agent. Mutation Agent. Analogously to the generation of nonimplemented requirements, the proportion of smelly requirements in each batch is configurable (20 % in our experiments). Smelly requirements are generated by selecting requirements verified as implemented by the verifier agent and mutating them to synthetically introduce target requirements smells. The system prompt requests preserving the original requirements as much as possible, restricting changes strictly to those parts necessary to introduce the target smell. In the user prompt, we include the requirement to be mutated, the randomly selected target smell, chosen from a list of smell categories (cf. Table I), and a description of the smell category. The agent employs RAG to retrieve requirements similar to the candidate from a curated dataset of smelly requirements, pre-filtered to include only examples exhibiting the selected target smell. These examples then serve as fewshot demonstrations to improve mutation quality. The mutated requirement is added to the batch as a smelly variant of an otherwise well-formed requirement. Trace-Link-Recovery (TLR) Agent. The TLR agent serves the purpose of supporting the human-in-the-loop in assessing the implementation status of a requirement, thus producing verified trace links for all generated requirements. The agent receives a candidate requirement together with the file and function names identified by the preceding agents. It then issues targeted RAG queries to retrieve relevant code chunks: (i) the top-2 chunks retrieved by the requirement text, (ii) the
top-4 chunks retrieved by function names, and (iii) the top-4 chunks retrieved by file names. Again, this constellation was determined through earlier experiments. The agent is prompted to identify the exact lines of code that implement the given requirement. To support this, the system prompt includes explicit tracing guidelines specifying that trace links must be evidence-based. Clear criteria are established for what constitutes a valid trace link, including conditions, assignments, and method signatures, as well as for what must be excluded, such as calls to other methods, comments, and tests. These guidelines closely follow the tracing rules by Vogelsang et al. [21]. B. Retrieval-Augmented Generation (RAG) We employ RAG to enable the agent to efficiently access the complete code base of the target projects. We preprocess the source code of each project prior to indexing. First, we exclude all files irrelevant to the approach, i.e., non-code files and test files. The remaining files are then split into chunks of 100 lines each, with an overlap of 10 lines at the beginning and 10 lines at the end of each chunk. This serves the purpose of balancing retrieval precision against context coverage, as overly large chunks risk introducing excessive irrelevant code into the results. We prefix each chunk with a metadata header containing the file identifier, the complete path within the project, and the line range covered by the chunk. This facilitates file-based retrieval, as the file path can be retrieved directly from the chunk text. In addition, each chunk is assigned a random shard identifier (cf. III-A), which is added to the vector store as metadata. For the TLR agent, each chunk is additionally stored in a second variant with line numbers prepended to every line of code. This allows the TLR agent to return exact line numbers reliably. C. Human-in-the-loop & Multi-shot Prompting We follow a human-in-the-loop approach for two reasons. First, because we generate requirements from source code without pre-existing ground truth, automated verification alone does not guarantee validity. Second, Unterbusch and Vogelsang [22] demonstrate that validated human reasoning in the RE context, incorporated via few-shot learning, enables LLMs to adapt to context-specific quality judgments efficiently. This suggests that human involvement grounds model behavior in ways that purely automated approaches cannot replicate. We feed back all requirements labeled by the human-inthe-loop to the generator agent as few-shot examples. This feedback loop serves the purpose of LLM adaptation as described by Unterbusch et al. Additionally, it also prevents the generation of duplicate requirements by making the generator aware of all previously accepted (or rejected) requirements. IV. S TUDY D ESIGN To report the study design and study results, we follow the guidelines by Jedlitschka and Pfahl [23] on how to report experiments in software engineering.
A. Goals & Research Questions The goal of our human-in-the-loop study is to explore whether the generation of realistic synthetic requirements from source code is viable. We assess this in three areas. First, we investigate whether the requirements generated by our approach are actually implemented in the code base and not hallucinated. Human-verified trace links act as ground truth. Second, we evaluate the quality of the generated requirements by examining whether they are free of requirements smells. Third, we investigate the viability of synthetically introducing requirements smells to produce realistic datasets. This results in the following research questions: • RQ1.1: To what extent are requirements reconstructed from source code by an LLM actually implemented in the code base? (Implementation Accuracy) • RQ1.2: To what extent are the reconstructed requirements hallucinated by an LLM? (Hallucination Rate) • RQ2: To what extent are requirements generated by an LLM free of requirements smells? (Generation Quality) • RQ3: To what extent do human evaluators agree with the LLM on the category of synthetically introduced requirements smells? (Human-LLM Reliability) B. Datasets & Participants We use two software projects as code bases to generate synthetic requirements. The first project, hereafter referred to as SEP, is a student project developed by bachelor’s students during a practical software engineering course at our university. The source code comprises approximately 24,000 lines of code written in Java, TypeScript, HTML, and CSS, and implements a full-stack web application resembling a personal driver hiring platform (like UBER). The second project, Mattermost, is the publicly available source code3 of the desktop client of the Mattermost messaging application. It has approximately 28,000 lines of code written in TypeScript and HTML. The two projects were selected to cover a diverse range of code characteristics, ranging from a well-maintained long-term open-source project to a student project developed under less rigorous review practices within a constrained timeframe. All human evaluators involved are PhD students with experience in Requirements Engineering. C. Experimental Setup To conduct our experiments, we implemented the agentic approach described in Section III in Python. To make the approach accessible to evaluators, we developed a web application using SvelteKit that interfaces with the Python backend. LLM requests were managed using Celery tasks, enabling multiple agent invocations to be processed simultaneously. During the study, evaluators accessed the locally deployed web application to complete their evaluations, with all results stored in a database for subsequent export and analysis. For retrieval-augmented generation, we used ChromaDB as 3 https://github.com/mattermost/desktop (commit adfcd2c on main).
TABLE I S MELL C ATEGORIES U SED T HROUGHOUT THE S TUDY. E XTRACTED F ROM P REVIOUS W ORK BY VOGELSANG ET AL . [21] AND E XTENDED FOR T HIS PAPER BASED ON W ORK BY F RATTINI ET AL . [24]. Smell Category
Description
Subjective Language
Words of which the semantics are not objectively defined, such as user friendly, easy to use, cost effective, etc. OR Sentences expressing personal opinions or feelings. Sentences or sentence parts that are unclear/imprecise and can be misunderstood if read by different people. Sentences containing escape clauses or conditional exceptions that allow the requirement to be circumvented or interpreted as not applicable, e.g., using phrases like ’where possible’, ’as appropriate’, ’unless otherwise agreed’, etc. Terms or phrases that cannot be objectively measured or tested, such as ’sufficient’, ’adequate’, ’reasonable’, ’as needed’, etc.
Ambiguities Loopholes Open-ended, non-verifiable terms Superlatives Comparatives Negative Statements Vague Pronouns Non-atomic Passive Voice Optional Parts Weak Verbs UI/UX Aesthetics Scope Creep
Over strong guarantees using absolute words such as always, never, guarantee, 100 %, cannot fail, etc. Comparative words such as better, more, etc. Sentences containing negative modifiers (e.g., not), negative expressions. Pronouns that refer back to a previous part of the text for which the reference is unclear. Requirements that combine multiple distinct conditions, behaviors, or features into a single statement, making them difficult to test or trace individually. Sentences using passive voice such that it is unclear who is performing a certain action. Sentences containing optional parts, e.g., by using the words possibly, eventually, if possible, if needed, etc. Weak verbs, such as can, could, may, etc. Requirements to UI/UX without measurable criteria, such as modern look, pleasant UI, nice animations, etc. Making too broad assumptions, such as support all languages, work on all devices, integrate with any service, etc.
a local vector store, with embeddings produced by OpenAI’s text-embedding-3-large model. We selected OpenAI’s gpt-5.4-mini-2026-03-17 as the model for all agentic tasks, as it demonstrated suitable performance across all tasks while offering a favorable costperformance tradeoff compared to the full gpt-5.4 model. We did not evaluate any additional models. For each of the two projects, three independent evaluators evaluated the generated requirements, serving as the humanin-the-loop for one round of generation. The evaluators were asked to assess requirements for one hour with respect to the following three criteria: 1) Appropriateness. A subjective 5-point Likert scale score assessing two aspects: (a) whether the requirement conforms to the expected format, and (b) whether it describes plausible system behavior that could be expected from the given system. 2) Requirements Smells. All requirements smells identified by the evaluator in the given requirement, selected from a predefined list of 14 smell types (see Table I). 3) Implementation Status. Based on the traceability information produced by the TLR agent, highlighting the most relevant code locations related to the requirement’s implementation, evaluators classify the requirement as fully implemented, partially implemented, or not implemented. While the initial design aimed for a single evaluation round (only by the human-in-the-loop), we planned additional rounds as a contingency to resolve potential inter-rater disagreements. In a second round, the evaluators were rotated across requirements sets and asked to independently re-rate the appropriateness and smell labels to assess inter-rater agreement. During the rating, they did not have access to the first evaluator’s decisions. In a third round, two authors of this paper reviewed all inter-rater disagreements in the smell labels and resolved them by developing a codebook addressing common sources
of confusion that arose during evaluation. Although the specific conventions were derived from disagreements observed in practice rather than specified in advance, the following codebook was established to ensure consistent labeling across evaluators. • The use of should alone does not constitute the weak verb smell; • the vague pronoun smell applies only when multiple possible references exist for a given pronoun; • when both UI/UX aesthetics and subjective language are applicable, the more specific UI/UX aesthetics smell should be used; • the smells open-ended, non-verifiable terms and subjective language are consolidated into subjective language; • the passive voice smell applies only when the use of passive voice results in an unclear or unidentifiable actor. V. S TUDY R ESULTS A. Human-in-the-loop Datasets The human-in-the-loop study produced a total of 188 requirements across the two source code projects. The generated requirements are labeled either as (i) implemented, i.e., implemented and smell-free by the LLM, (ii) non-implemented, i.e., not-implemented and smell-free, or (iii) smelly, i.e., including a synthetically added requirement smell. We introduce the non-implemented class as a control group. These categories are solely based on the LLM’s perspective. We show the distribution of implemented, non-implemented, and smelly requirements in Table II. B. RQ1.1: Implementation Accuracy This research question aims to assess the implementation accuracy of the LLM-generated requirements by examining the extent to which requirements generated from source code are confirmed as implemented by the human-in-the-loop evaluators, based on the trace link code excerpts presented to
TABLE II R EQUIREMENTS G ENERATED D URING THE H UMAN - IN - THE - LOOP S TUDY
Impl.
# of Requirements Non-impl. Smelly
Total
E1 E2 E3
18 19 21
7 8 7
5 7 6
30 34 34
E1 E2 E3
16 21 19
5 6 8
5 5 5
26 32 32
Project
Evaluator
SEP
Mattermost
TABLE III RQ1.1: I MPLEMENTATION ACCURACY OF THE LLM BASED ON H UMAN -I N -T HE -L OOP L ABELING
Precision
Results Recall
F1 -score
SEP
E1 E2 E3 E1–E3
0.944 0.895 0.857 0.897
0.708 0.739 0.720 0.722
0.810 0.810 0.783 0.800
Mattermost
E1 E2 E3 E1–E3
1.000 0.952 1.000 0.982
0.762 0.769 0.731 0.753
0.865 0.851 0.844 0.853
0.939
0.738
0.826
Project
Evaluator
All
them during the study. We report precision, recall, and F1 scores of comparing the LLM’s generation methods (i.e., whether it was tasked to generate an implemented or nonimplemented requirement) with the labels of the human-inthe-loop evaluators in Table III. To compute these values, we define true positives as labeled implemented by both the LLM and human evaluator, false positives as labeled implemented by the LLM but labeled non-implemented by the human, etc. As can be seen, there is a consistent gap between precision and recall across all evaluators and projects. While the precision is consistently high (0.857–1.000), the recall stays lower (0.708–0.769). When the LLM is tasked to generate an implemented requirement, humans almost always agree. Still, when the LLM is tasked to generate a non-implemented requirement deliberately, humans often find it to be implemented based on the given trace links. Furthermore, the performance for Mattermost is consistently stronger than for SEP. A possible explanation for this may be the higher code quality of a wellmaintained long-term project. C. RQ1.2: Hallucination Rate In Table IV, we report the false positive requirements (i.e., labeled as implemented by the LLM and labeled as non-implemented by the human-in-the-loop), as well as the hallucination rate, which is defined as the false positive rate. The overall hallucination rate of 6.1 % is low, which suggests that the LLM reliably generates requirements that are actually implemented. Still, there is a large difference between the two projects. For Mattermost, we see a rate of only 1.8 %, while for SEP, the rate is 10.3 %. This discrepancy
TABLE IV RQ1.2: R EQUIREMENTS H ALLUCINATED BY THE LLM AND H ALLUCINATION R ATE AS A SSESSED BY THE H UMAN -I N -T HE -L OOP U SING T RACE L INKS Project SEP Mattermost Total
LLM: Impl.
Results FP (Hallucinated)
Hallucination Rate
58 56 114
6 1 7
10.3 % 1.8 % 6.1 %
follows the observation from RQ1.1. It is possible, that the code quality is relevant for this difference. Well-structured code that has been developed over a long time-span with many different reviewers, eases the LLM to ground the generated requirements in clear evidence. However, as Mattermost is also more complex than the student projects, it might have been easier for the evaluators to identify hallucinations as such. Though, as there are only 7 reported hallucinations in total, generalizability is limited. D. RQ2: Generation Quality Comparing the smell labels of the first-round human-in-theloop with the second-round evaluators, we find that Cohen’s κ ranges from −0.083 (Mattermost; E1) to 0.803 (Mattermost; E3), with an overall κ = 0.213 across both projects and all evaluator pairs, indicating fair agreement at best. This disagreement shows the necessity of the third labeling round, in which two authors of this paper resolved disagreements by developing a codebook from common sources of confusion (cf. Section IV-C). The peer-reviewed labels from this third round are used as the basis for the results reported below. We report all results in Table V. The smell rate in requirements labeled as smell-free (implemented and non-implemented combined) by the LLM, over both projects and all evaluators, is 24.5 %. This smell rate is calculated based on the peer-reviewed smells. As observed for RQ1.1 and RQ1.2, both projects exhibit notable differences in variation. The smell rates for SEP range from 7.1 %–40.7 %, showing considerable variance across the 3 evaluators. For Mattermost, the results are much more consistent with a range of 25.9 %–33.3 %. Of all possible smell categories, only 6 categories were labeled to appear in the allegedly smell-free requirements. These categories, in descending order of frequency, are: nonatomic (12), passive voice (11), ambiguities (8), negative statements (6), subjective language (3), and vague pronouns (1). Note, that a single requirement may contain multiple smells. Many of these dominant smells are structural and syntactic, rather than semantic ones, suggesting that the LLM tends to generate requirements that are functionally correct but linguistically imprecise. The absence of smell categories such as superlatives or comparatives together with the functional correctness suggests that the LLM does well in presenting the code behavior based on the given information. A representative example for a non-atomic requirement as generated by the LLM is (Mattermost): ”When a download
TABLE V RQ2: R EQUIREMENTS S MELLS F OUND IN LLM-G ENERATED S MELL -F REE R EQUIREMENTS A FTER P EER -R EVIEW
# of Reqs.
Results # of Smelly Reqs.
Smell Rate
E1 E2 E3 E1–E3
25 27 28 80
3 11 2 16
E1 E2 E3 E1–E3
21 27 27 75 155
Project
Evaluator
SEP
Mattermost
All
TABLE VI RQ3: R EQUIREMENTS S MELLS F OUND IN LLM-G ENERATED S MELLY R EQUIREMENTS A FTER P EER -R EVIEW Any Smell? # %
Exact Smells? # %
5 7 6 18
4 7 5 16
80.0 % 100.0 % 83.3 % 88.9 %
4 5 3 12
80.0 % 71.4 % 50.0 % 66.7 %
E1 E2 Mattermost E3 E1–E3
5 5 5 15
5 5 4 14
100.0 % 100.0 % 80.0 % 93.3 %
2 3 3 8
40.0 % 60.0 % 60.0 % 53.3 %
All
33
30
90.9 %
20
60.6 %
Project
Evaluator
12.0 % 40.7 % 7.1 % 20.0 %
SEP
E1 E2 E3 E1–E3
7 8 7 22
33.3 % 29.6 % 25.9 % 29.3 %
38
24.5 %
item no longer has a valid file location, opening its folder should mark it as deleted and, if a download location is configured, open that download folder instead.” This highlights the LLM trying to include closely related functionality (in code) in a single requirement, leading to long, complicated, and/or non-atomic requirements. Another representative requirement, this time for passive voice, is: ”When the main window is closed on Windows or Linux and tray minimization is enabled, the application hides the window instead of quitting.” It is unclear which actor is responsible for closing the main window. Ambiguous requirements would often allow various interpretations of the intended functionality, e.g., in: ”If a notification sound is configured, the notification is shown silently and the chosen sound is stored for later playback” it is unclear whether the mentioned silent notification is shown directly after configuring a notification sound, or whether this configures a general system property. E. RQ3: Human-LLM Reliability To assess whether synthetically introduced smells are reliably detected by human evaluators, we compare the LLM’s intended smells against the peer-reviewed smells from the final evaluation round. In both projects, 60.6 % of intentionally smelly requirements were labeled with the exact intended smell category, and SEP showed a higher exact-match agreement (66.7 %) than Mattermost (53.3 %). When considering whether any smell was detected, regardless of category, 90.9 % of intentionally smelly requirements received at least one smell label. We report all results in Table VI. Figure 2 shows the per-category agreement: while some smells, e.g., subjective language and UI/UX aesthetics, are detected with perfect agreement, other smell categories such as passive voice and loopholes show zero exact-match agreement. F. Appropriateness Lastly, we evaluated the appropriateness scores. We did not include them as a research question, as they are a highly subjective measure, which is reflected in the inter-rater agreement between the first and second evaluation rounds. Over both projects, Krippendorff’s α = 0.178, with an exact
# of Smelly Reqs.
rbs k ve wea ics t sthe x ae u / i age u ngu ve la eep i t c e subj cope cr s ties igui 2/3 (67%) amb ves rlati 2/3 (67%) supe rts pa nal o i t 3/5 (60%) op mic -ato non ns 2/4 (50%) onou e pr 1/2 (50%) u s g t va men tate s s 1/3 (33%) e ativ arative neg p 0/3 (0%) com oice ive v s s a s 0/1 (0%) p 50% hole loop 0% 25% 50% 75%
1/1 (100%) 2/2 (100%) 4/4 (100%) 1/1 (100%) 1/1 (100%)
100%
Exact Smell Detection Rate (TPR)
Fig. 2. Agreement of human evaluators (after peer-review) with LLM on the categories of synthetically introduced smells.
agreement of 36.2 % (70.2 % when allowing a one-point deviation between ratings), confirms that appropriateness scores are inconsistent across evaluators and should be interpreted accordingly. Comparing appropriateness scores across generation methods, the Kruskal-Wallis test reveals a significant overall effect for Mattermost (p = 0.0001; ε2 = 0.212) and across both projects combined (p = 0.0034; ε2 = 0.061), but not for SEP alone (p = 0.0981; ε2 = 0.048). Pairwise comparisons using the Mann-Whitney U test with Holm–Bonferroni correction show that for Mattermost, positive requirements score significantly higher than both not-implemented (padj = 0.0043, p < 0.01; r = −0.447) and smelly requirements (padj = 0.0005, p < 0.001; r = −0.598). For SEP, we do not detect significant differences between generation methods after Holm–Bonferroni correction. Across both projects combined, only the contrast between positive and smelly requirements is calculated as significant (padj = 0.002, p < 0.01; r = −0.367), while non-implemented requirements do not differ significantly from either group. These results suggest that
human evaluators likely perceive smelly requirements as less appropriate than well-formed ones. Figure 3 shows the mean appropriateness scores per batch between evaluators for both projects. Although the trend line suggests a slight upward trajectory over successive batches, indicating that the multi-shot feedback loop may have an impact, for both projects, this correlation is not statistically significant (Spearman’s rank correlation: ρ = 0.40; p = 0.600), and the small number of batches (n = 4) limits any meaningful interpretation of this trend. We note it as a tentative observation that warrants further investigation with larger batch counts. VI. D ISCUSSION We initially intended to provide a complete approach for generating realistic requirements-to-code datasets. Throughout the study and evaluation, various problems arose, leading to this experience report. While the hallucination rate is low, confirming our initial hypothesis that, through code-grounding, LLMs should be able to generate implemented requirements, this does not hold true for non-implemented ones. Low recall comparing the human judgments to the LLM’s task shows that, even when specifically prompted to generate non-implemented requirements, the LLM cannot reliably perform this task. We agree with previous work [13], [19] that LLMs show promising results in requirements generation from code. However, similarly to what these studies already hint at, we find that human oversight is still necessary. We hypothesized that through clear guidelines provided in the prompt, the LLM would be able to generate high-quality requirements if desired, thus containing a low rate of unintentional requirements smells. Seeing that 24.5 % of requirements are still labeled as smelly by the human evaluators also proves this hypothesis wrong. Femmer and Vogelsang [25] argue that requirements quality is fundamentally quality-inuse: a requirement is only defective if it actually impairs a downstream activity, and whether a given smell does so depends heavily on context. Frattini [26] reinforces this with industrial evidence, showing that the factors that determine whether a smell constitutes an actual defect are numerous and context-specific. Consequently, the 24.5 % smell rate reported in RQ2 should be interpreted as the rate at which generated requirements exhibit linguistic patterns associated with potential quality issues, rather than confirmed defects. Preliminary evidence that smells affect at least some downstream tasks is provided by Vogelsang et al. [21], who show that requirements smells negatively impact automated traceability, suggesting that this concern is not merely theoretical. When designing the original study, we assumed that one human-in-the-loop would be sufficient to label requirements smells with adequate reliability. However, the weak inter-rater agreement of κ = 0.213 made an additional round of evaluation and the development of a codebook necessary, defining conventions to operationalize context-specific interpretations of smells. This is supported by the findings of Unterbusch and Vogelsang [22], showing that smell judgments are contextdependent and not a task that can be objectively completed
even by trained evaluators. We acknowledge that this also possibly affects the actual smell rate for RQ2, regardless of the countermeasures taken (cf. Section VI-B). Furthermore, we assumed that synthetically introduced smells could be reliably labeled, enabling the production of realistic datasets. We see that human evaluators identified a smelly requirements in 90.9 % of cases, showing that these requirements are recognized as problematic. However, since only 60.6 % of smells were exactly identified, the exact smell perceived by humans again appears to be context-dependent and not objectively identifiable. For practitioners, our findings suggest that while a low hallucination rate indicates that the approach is useful for generating requirements from existing codebases, there are other issues that are not immediately obvious. Generated requirements should not be used without human review, as the LLM fails to self-assess quality and to follow the prompted tasks precisely. Critically, the performance gap between Mattermost and SEP indicates that the approach performs better for well-maintained codebases, precisely those most in need of requirements generation in practice. Further investigation of the unintentional smells introduced by LLMs could inform prompt engineering improvements, highlighting the defects to which LLMs are most vulnerable. A. Lessons Learned From the study and its findings, we learn the following lessons. 1) Labeling smells in requirements always requires multiple humans, as smells are highly context-sensitive. Neither an LLM nor a single human evaluator alone is reliable for this task. 2) LLMs can support the generation of requirements from code, but should not be trusted with this task without human supervision as (a) prompts are not always followed precisely (e.g., generating implemented requirements even when explicitly prompted not to do so) and (b) generated requirements can exhibit major quality deficiencies. 3) The reliability of LLM-based requirement generation, even when based on source code, strongly depends on the codebase; practitioners should not expect consistent results across codebases of different maintenance levels. B. Threats to Validity Internal Validity. Prompting must be considered a threat to validity whenever LLMs are employed (cf. Korn et al. [27]), as prompts may miss important details or bias the LLM toward outcomes anticipated by the prompt author. We mitigated this by iteratively refining the prompts throughout the study. The multi-shot feedback loop introduces a further threat, as each evaluator’s later ratings are shaped by their own prior decisions, causing individual biases to accumulate, thus complicating cross-evaluator comparison. Additionally, since smelly requirements are mutations of previously accepted
Mattermost
Sep
Appropriateness score (1 5)
5 4 3 2 1
E1 E2 E3 Mean Trend 1
E1 E2 E3 Mean Trend 2
3
Batch number (per evaluator)
4
1
2
3
Batch number (per evaluator)
4
Fig. 3. Comparison of human-in-the-loop appropriateness ratings over time. Each batch is generated with reviews from last batch as shot samples.
requirements, evaluators may recognize them based on familiarity with the unmodified counterpart. Presenting smell categories as a predefined list may further encourage evaluators to label smells they would not otherwise have noticed, potentially inflating false positive rates. The evaluator pool consisted exclusively of PhD students with varying levels of RE expertise, which may introduce selection bias and noise, particularly for smell detection. External Validity. The approach is evaluated exclusively on general-purpose software projects, excluding domains with domain-specific requirements such as safety-critical or heavily regulated systems. We consider this a limited threat, as the approach relies solely on information retrieved via RAG from the target code base without incorporating domain-specific knowledge. Furthermore, the fixed RAG configuration may require reconfiguration for projects that differ substantially in scale from those used in this study. However, as we only used 2 very different software projects, generalizability of the assumed explanations between Mattermost and SEP is very limited. Construct Validity. Appropriateness ratings are inherently subjective as evaluators may interpret the 5-point scale differently, leading to inconsistent ratings. The smell labels in round three are not purely independent human judgments, as the authors had to resolve disagreements and produce the codebook; the labels therefore may partially reflect the authors’ interpretation. Implementation status is assessed solely based on TLR agent trace links, meaning that retrieval failures may produce mislabeled requirements rather than reflecting true implementation status. We consider this an acceptable tradeoff. Finally, results are based on a single LLM; alternative models may yield different outcomes, although the exact model is reported in full to enable reproduction and comparison. Conclusion Validity. The datasets produced during the study are relatively small, which can reduce statistical power for subgroup analysis. Although the Holm-Bonferroni correction is applied to each family of comparisons, the overall family-wise error rate across all analyzes may not be fully controllable.
VII. C ONCLUSION We present an LLM-based and RAG-supported agentic approach for generating realistic requirements-to-code datasets from existing code bases, and evaluate it in a human-in-theloop study across two software projects. Although we originally intended to offer this approach as a viable solution for generating datasets for downstream RE research, we now offer an experience report, as during the study various problems arose. Although we initially hypothesized that LLMs would be able to reliably generate both implemented and intentionally non-implemented requirements, only the former held true (6.1 % hallucination rate). Furthermore, we hypothesized that through prompting with rigorous guidelines, the LLM would be able to generate requirements free of requirements smells (thus potentially high in quality). In practice, 24.5 % of the generated requirements exhibit requirements smells, showing that quality cannot be self-assessed by the LLM alone. A single human-in-the-loop also proved to be insufficient in detecting smells reliably, confirming that requirements smells are context-dependent and cannot be classified reliably without shared conventions, as reflected in a low initial inter-rater agreement of κ = 0.213. This required multiple rounds of evaluations that were aimed at resolving common misunderstandings. Finally, the reliability of LLM-based requirements generation strongly depends on code quality, such that practitioners should not expect consistent results across code bases of different maintenance levels. These findings suggest that fully autonomous generation of requirements from code is not yet viable, requiring human supervisor at least for quality assurance. We offer our lessons learned from this study and publish two generated requirements-to-code datasets as a concrete contribution to the empirical RE community. A. Future Work Several directions remain for future work. Automated smell detection could be applied as an initial optimization step before human review, reducing cognitive workload. Rather than asking evaluators to identify all smells from scratch,
the human-in-the-loop could confirm or reject automatically detected smell candidates, shifting the task from open-ended labeling to verification. The multi-shot feedback loop shows a tentative upward trend in appropriateness scores across batches; a larger evaluation with more batches and evaluators would allow this effect to be examined more rigorously, and may inform whether the feedback loop can reduce the need for human oversight. Finally, a larger evaluation across more code bases and domains, particularly safety-critical or heavily regulated systems, is necessary to establish the generalizability of these findings. ACKNOWLEDGMENTS We thank the study participants for their time and support throughout the evaluation of the presented approach. Large language models were used to assist in improving the writing of this paper and to support the development of the experimental implementation of the approach. Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project number: 566352773. R EFERENCES [1] A.-L. Lamprecht, L. Garcia, M. Kuzak, C. Martinez, R. Arcila, E. Martin Del Pico, V. Dominguez Del Angel, S. van de Sandt, J. Ison, P. A. Martinez, P. McQuilton, A. Valencia, J. Harrow, F. Psomopoulos, J. L. Gelpi, N. Chue Hong, C. Goble, and S. Capella-Gutierrez, “Towards FAIR principles for research software,” Data Science, vol. 3, no. 1, pp. 37–59, 2019. [2] D. Yang, X. Xie, X. Yang, M. Hu, Y. Huang, Y. Zhang, W. Miao, T. Su, C. Wan, and G. Pu, “Assessing the impact of requirement ambiguity on LLM-based function-level code generation,” arXiv preprint arXiv:2604.21505, 2026. [3] J. L. C. Guo, J.-P. Steghöfer, A. Vogelsang, and J. Cleland-Huang, Natural Language Processing for Requirements Traceability. Springer Nature Switzerland, 2025, pp. 89–116. [4] Q. Motger, C. Catot, and X. Franch, “Characterizing datasets for LLM-based requirements engineering: A systematic mapping study,” in International Conference on Evaluation and Assessment in Software Engineering (EASE). ACM, 2026. [5] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 35, no. 2, pp. 1–72, 2026. [6] A. El-Hajjami and C. Salinesi, “Synthline: A product line approach for synthetic requirements engineering data generation using large language models,” in International Conference on Research Challenges in Information Science. Springer, 2025, pp. 208–225. [7] R. Xu, Z. Xu, G. Li, and V. S. Sheng, “Bridging the gap between source code and requirements using GPT (student abstract),” in AAAI Conference on Artificial Intelligence, vol. 38, no. 21, 2024, pp. 23 686– 23 687. [8] K. R. Dearstyne, “Intelligent traceability to support software maintainability and accountability,” in IEEE International Requirements Engineering Conference (RE). IEEE, 2025, pp. 607–611. [9] Y. Wang, J. Keung, X. Ma, Z. Mao, K. Chen, and Y. Li, “R2Code: A self-reflective LLM framework for requirements-to-code traceability,” arXiv preprint arXiv:2604.22432, 2026. [10] H. Femmer, D. Méndez Fernández, S. Wagner, and S. Eder, “Rapid quality assurance with requirements smells,” Journal of Systems and Software (JSS), vol. 123, pp. 190–213, 2017. [11] E. Gentili and D. Falessi, “Characterizing requirements smells,” in International Conference on Product-Focused Software Process Improvement (PROFES). Springer, 2023, pp. 387–398. [12] W. Zogaan, P. Sharma, M. Mirahkorli, and V. Arnaoudova, “Datasets from fifteen years of automated requirements traceability research: Current state, characteristics, and quality,” in IEEE International Requirements Engineering Conference (RE). IEEE, 2017, pp. 110–121.
[13] D. Jin, Z. Jin, Y. Zhang, Z. Fang, L. Li, Y. He, X. Chen, and W. Sun, “UserTrace: User-level requirements generation and traceability recovery from software project repositories,” 2025. [14] D. Fuchß, T. Hey, J. Keim, H. Liu, N. Ewald, T. Thirolf, and A. Koziolek, “LiSSA: Toward generic traceability link recovery through retrievalaugmented generation,” in IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2025, pp. 1396–1408. [15] K. Moran, D. N. Palacio, C. Bernal-Cárdenas, D. McCrystal, D. Poshyvanyk, C. Shenefiel, and J. Johnson, “Improving the effectiveness of traceability link recovery using hierarchical bayesian networks,” in ACM/IEEE International Conference on Software Engineering (ICSE). ACM, 2020, pp. 873–885. [16] S. J. Ali, V. Naganathan, and D. Bork, “Establishing traceability between natural language requirements and software artifacts by combining RAG and LLMs,” in International Conference on Conceptual Modeling (ER). Springer, 2024, pp. 295–314. [17] T. Hey, F. Chen, S. Weigelt, and W. F. Tichy, “Improving traceability link recovery using fine-grained requirements-to-code relations,” in IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2021, pp. 12–22. [18] J. Lin, Y. Liu, Q. Zeng, M. Jiang, and J. Cleland-Huang, “Traceability transformed: Generating more accurate links with pre-trained BERT models,” in IEEE/ACM International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 324–335. [19] E. Persson, E. Alégroth, and T. Gorschek, “Code2Req: Using generative AI to generate requirements from source code,” 2025. [Online]. Available: http://dx.doi.org/10.2139/ssrn.5845431 [20] R. Shao, J. He, A. Asai, W. Shi, T. Dettmers, S. Min, L. Zettlemoyer, and P. W. Koh, “Scaling retrieval-based language models with a trilliontoken datastore,” Advances in Neural Information Processing Systems, vol. 37, pp. 91 260–91 299, 2024. [21] A. Vogelsang, A. Korn, G. Broccia, A. Ferrari, J. Fischbach, and C. Arora, “On the impact of requirements smells in prompts: The case of automated traceability,” in IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), 2025, pp. 51–55. [22] M. Unterbusch and A. Vogelsang, “Context-adaptive requirements defect prediction through human-LLM collaboration,” in IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), 2026. [23] A. Jedlitschka and D. Pfahl, “Reporting guidelines for controlled experiments in software engineering,” in International Symposium on Empirical Software Engineering (ESEM), 2005. [24] Y. Li, J. Keung, X. Ma, C. Y. Chong, J. Zhang, and Y. Liao, “LLMbased class diagram derivation from user stories with chain-of-thought promptings,” in IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 2024, pp. 45–50. [25] H. Femmer and A. Vogelsang, “Requirements Quality Is Quality in Use,” IEEE Software, vol. 36, no. 3, pp. 83–91, May 2019. [26] J. Frattini, “Identifying Relevant Factors of Requirements Quality: An Industrial Case Study,” in Requirements Engineering: Foundation for Software Quality, D. Mendez and A. Moreira, Eds. Cham: Springer Nature Switzerland, 2024, pp. 20–36. [27] A. Korn, L. Zaruchas, C. Arora, A. Metzger, S. Smolka, F. Wang, and A. Vogelsang, “Reporting LLM prompting in automated software engineering: A guideline based on current practices and expectations,” in ACM International Conference on AI Foundation Models and Software Engineering (FORGE), 2026.