arXiv:2604.15222v1 [cs.SE] 16 Apr 2026
AI-Assisted Requirements Engineering: An Empirical Evaluation Relative to Expert Judgment Oz Levy Faculty of Industrial Engineering and Technology Management, Holon Institute of Technology (HIT) Holon, Israel [email protected]
Ilya Dikman Faculty of Industrial Engineering and Technology Management, Holon Institute of Technology (HIT) Holon, Israel [email protected]
Natan Levy School of Computer Science and Engineering Hebrew University Jerusalem (HUJI) Jerusalem, Israel [email protected]
Michael Winokur Faculty of Industrial Engineering and Technology Management, Holon Institute of Technology (HIT) Holon, Israel [email protected]
Abstract Artificial Intelligence (AI) is increasingly introduced into systems engineering activities, particularly within requirements engineering (RE), where quality assessment and validation remain heavily dependent on expert judgment. While recent AI tools demonstrate promising capabilities in analyzing and generating requirements, their role within formal systems engineering processes—and their alignment with established INCOSE criteria—remains insufficiently understood. This paper investigates the extent to which AI-based tools can support systems engineers in evaluating requirement quality, without replacing professional expertise. The research adopts a structured systems engineering methodology to compare AI-assisted requirement evaluation with human expert assessment. A controlled study was conducted in which system requirements were evaluated against established INCOSE “good requirement” criteria by both experienced systems engineers and an AI-based assessment tool. The evaluation focused on consistency, completeness, clarity,
and testability, examining not only accuracy but also the decision logic underlying each assessment. Results indicate that AI tools can provide consistent and rapid preliminary assessments, particularly for syntactic and structural quality attributes. However, expert judgment remains essential for contextual interpretation, ambiguity resolution, and trade-off reasoning. Rather than positioning AI as a replacement for systems engineers, the findings support its role as a decision-support mechanism within the RE lifecycle. From a systems engineering perspective, this study contributes empirical evidence on how AI can be integrated into RE workflows while preserving traceability, accountability, and engineering consistency. The paper further discusses implications for SE practice, including workload reduction, quality assurance processes, and the boundaries of automation. The results inform both practitioners and researchers seeking to responsibly integrate AI into systems engineering processes.
Keywords Requirements engineering, large language models, requirement quality, functional and non-functional requirements, human-in-the-loop, AI-assisted engineering
Introduction In recent years, AI has revolutionized various aspects of life and industry, becoming a key tool in processes once considered inapplicable to automated mechanisms (AI Index Steering Committee, 2024). Leading organizations like Tesla and NASA have integrated AI into large-scale projects (Tesla, 2025; NASA, 2016). For instance, Baidya (Baidya et al., 2022) explored the opportunities and challenges of using AI in critical projects like robotics and aeronautics, where AI assists with data analysis and provides insights into complex engineering tasks. In systems engineering, AI is expected to enhance the ability to analyze and classify engineering requirements, which form the foundation for product design and development. Existing research has explored AI’s role in classifying requirements in software and quality management systems (Cheligeer et al., 2022 ,Tamai and Anzai, 2018). However, these studies primarily cover specific, categorized tasks in software or quality requirements rather than providing a tool for comprehensive, standardized classification as per the International Council on Systems Engineering (INCOSE) standards (INCOSE, 2023), which classify requirements into categories such as functional and non-functional. With increasing demands for product quality and precision, analyzing requirements based on criteria like necessity, clarity, and verifiability is crucial. AI may enhance this process, offering rapid and accurate analyses that increase systems engineers’ efficiency. However, significant challenges remain due to AI’s tendency to generate misinterpretations, as seen in various applications (Maleki et al., 2024,Martínez et al., 2023). Such inaccuracies could have severe consequences in complex engineering projects (Hadar et al., 2022,Pérez-Cerrolaza et al., 2024). This study, therefore, proposes to evaluate the accuracy and reliability of AI models in such contexts, asking whether AI could match a systems engineer’s judgment or remains merely an auxiliary tool, similar to calculators. This research also examines the feasibility of integrating AI into engineering education, emphasizing how AI-based tools can equip engineers with 21st-century skills while fos-
tering responsible engineering practices to address ethical challenges in critical systems. By bridging the gap between research and practical application, it aims to equip engineers with essential tools for today’s technological landscape.
Background and Related Work Requirements engineering is a cornerstone of successful system development, providing a clear definition of what a system must do and under what conditions (Siddique, 2022b). High-quality requirements serve as the foundation for design and verification, whereas poorly defined requirements can lead to costly rework and project failures . Traditionally, requirements are documented in natural language due to its accessibility and legal familiarity in contracts . However, alternate media (such as models, diagrams, or prototypes) have been explored as complementary ways to express requirements, aiming to improve understanding and reduce ambiguity (Kolligs and Thomas, 2025). In parallel, the use of AI in RE has been growing. AI techniques have a long history in RE, from early knowledge-based systems to today’s data-driven approaches (Dalpiaz and Niu, 2020b). With modern AI breakthroughs – including powerful language models like GPT-4, Anthropic’s Claude (Sonnet series), and Meta’s LLaMA – there is renewed potential to automate and enhance many RE activities.
Engineering Requirements Definition and Role in Systems Engineering In systems engineering a requirement is typically a statement of needed functionality, constraint, or property that a system must satisfy to meet stakeholder objectives (Siddique, 2022a). Requirements engineering involves eliciting these needs and translating them into precise system specifications (Siddique, 2022a). Well-written requirements ensure that development teams and stakeholders share a common understanding of the system’s goals.
2
the operational definition and evaluation criteria described in the Methodology section below, where requirement quality is assessed using seven INCOSEaligned attributes suitable for objective human and AI evaluation (INCOSE, 2023).
Language Models
Figure 1. Requirements Engineering Process: A Flowchart of Core Stages.
Conversely, inadequate requirements are often cited as a primary cause of project issues – “low quality requirements are among the first potential mistakes. . . felt downstream in the form of increased costs and schedule overruns” (Kolligs and Thomas, 2025). Clear and precise requirements make it easier to verify the system and avoid costly late-stage changes (INCOSE, 2023).
Classification of Requirements Requirements can be classified along several dimensions. A common classification is functional vs. nonfunctional requirements. Functional requirements specify the services, behaviors, or functions a system must execute – essentially, what the system should do. Non-functional requirements (NFRs), in contrast, define how the system should perform or the qualities it must have (Siddique, 2022a). These include performance metrics, security levels, usability, reliability, and other quality attributes or constraints. For example, an online service’s functional requirement might be the ability to register a new user, while a non-functional requirement could stipulate that the registration response occurs within 2 seconds (a performance constraint). Both types are vital: functional requirements ensure the system’s features meet user needs, and NFRs ensure the system is usable and trustworthy under real-world conditions (Siddique, 2022a).
Distinguishing Effective from Deficient Requirements The distinction between effective and deficient requirements in this study follows the quality characteristics defined in the INCOSE Guide to Writing Requirements (GtWR) (INCOSE, 2023). Rather than redefining these principles here, the paper adopts
Historical Evolution and Breakthroughs in AI Artificial intelligence has progressed through three main waves, each reshaping how RE activities are automated and supported.
Figure 2. The Evolution of Artificial Intelligence in Requirements Engineering: A Chronological Perspective.
1. Symbolic AI (mid-1950s – early 1990s). The first wave focused on symbolic reasoning, in which expert systems captured domain knowledge with IF–THEN rules and formal logic. During the 1980s, such rule-based engines were already inspecting requirements documents for internal consistency (Dalpiaz and Niu, 2020a). Although effective for narrow domains, these systems struggled with ambiguous natural language and quickly became brittle as project scope grew. 2. Data-Driven Machine Learning (1990s – late 2000s). Rising computational power enabled statistical natural-language processing and classical machine-learning (ML) algorithms for example, support-vector machines and decision trees. Researchers began to prioritize, classify and cluster large sets of requirements, reducing manual workload while highlighting potential defects (Dalpiaz and Niu, 2020a). At the same time, automated detectors of vague or subjective terms emerged, flagging words such as fast or user-friendly that undermine precision in software specifications (Kolligs and Thomas, 2025). 3. Deep Learning and Large Language Models (2010s – present). The third wave was triggered by multilayer neural networks and, above all, the Transformer architecture (Vaswani et 3
al., 2017). Deep models trained on vast corpora achieved state-of-the-art results across computer vision, speech and NLP. Transfer learning allowed researchers to fine-tune pretrained language models (e.g., BERT) on modest RE datasets, yielding substantial gains in tasks such as functional/non-functional classification (Kaur and Kaur, 2024). MartínezFernández(Martínez-Fernández et al., 2022) describes this period as the “rising wave of deep learning,” characterized by increasingly context-aware techniques that can parse entire specification documents, map traceability links and even generate draft requirements. Together, these three waves chart a shift from manually encoded rules to data-centric, context-sensitive analysis transforming RE from labour-intensive bookkeeping into a semi-automated, insight-driven discipline.
Artificial Intelligence in Requirements Engineering
isolated tasks (e.g., FR/NFR classification or requirement generation), are grounded in software-centric datasets, or rely on accuracy metrics without comparing AI behavior to professional systems-engineering judgment. Moreover, while several approaches implicitly reference quality attributes, few explicitly align AI evaluations with the structured criteria defined in the INCOSE Guide to Writing Requirements. As a result, it remains unclear to what extent AI assessments correspond to how experienced systems engineers reason about requirement quality, ambiguity, and acceptability. Another limitation concerns explainability and accountability. Although prompt engineering, expert personas, and structured outputs have been shown to improve AI performance and usability, AI systems still lack access to technical truth, system context, and cross-requirement reasoning. Consequently, the literature consistently emphasizes that AI should function as a decision-support tool rather than an autonomous evaluator, with human experts retaining responsibility for validation and trade-off decisions (Bender et al., 2021, Shneiderman, 2020).
AI has been increasingly applied to support RE, primarily through natural language processing (NLP) and machine learning techniques aimed at reducing manual effort and improving consistency. Early work focused on narrow, task-specific automation such as classifying requirements as functional or non-functional, detecting vague or subjective language, or clustering requirements by topic. Systematic reviews show that, given sufficient training data, machine-learning models can achieve high accuracy on well-defined classification tasks, particularly in software requirements datasets (e.g., Dalpiaz and Niu, 2020b, Kaur and Kaur, 2024).
Taken together, prior work suggests strong potential for AI to support RE, but also reveals a gap between task-level automation and engineering-grade evaluation aligned with INCOSE practice. Addressing this gap requires empirical comparison between AI-based assessments and human expert judgment, using clearly defined quality criteria and controlled experimental design. The following section presents the methodology adopted in this study to evaluate AI performance against experienced systems engineers across both requirement quality assessment and functional/non-functional classification.
Recent advances in large language models (LLMs) have expanded these capabilities. Transformerbased models can process longer contexts and capture semantic intent beyond surface-level keywords, enabling more nuanced classification and annotation of requirements. Systems such as NLP4ReF demonstrate that LLM-based approaches can outperform traditional NLP pipelines in requirement classification and even generate candidate requirements at low cost and high speed (Peer et al., 2024). These results indicate that AI can meaningfully assist with routine RE activities, especially in large-scale or time-constrained projects.
Methodology
However, existing research remains largely fragmented and domain-specific. Most studies focus on
This section explains how the research was designed and carried out. The goal was to evaluate how well AI can understand, classify, and assess engineering requirements compared to human experts. The methodology combines traditional systems engineering practices with modern AI techniques, creating a structured process from data collection to result analysis. Two main datasets were used, one from a real-world case study (Dr. Tools )(Hadar et al., 2022) and another public dataset (PROMISE)(Cleland-Huang et al., 2006) to test AI performance in different contexts. Each step of the process, from requirement review and prompt de4
sign to model execution and human comparison, was carefully planned to ensure reliable and meaningful results. The following subsections describe each stage of the methodology in detail. Figure 3 presents the complete process of the AI research and development journey, from collecting requirements to reporting results. The workflow includes several key stages: acquiring and analyzing data, designing prompts, validating inputs with experts, and performing AI-based classification. Human benchmarks are then collected to compare AI performance with expert evaluations, followed by accuracy, agreement, and robustness checks to ensure reliability. The process concludes with synthesis and reporting, where all findings are integrated into a comprehensive summary of results.
nology, this dataset involves an RFID-enabled inventory management system for operating rooms. It includes 31 stakeholder and 76 optimized system requirements, covering: 1. Functional goals: real-time instrument localization, set verification before incision, and automated alerts. 2. Non-functional goals: reliability, security, and integration with hospital IT systems. The context diagram (Figure 4) illustrates the system’s environment, highlighting interactions among surgical staff, sterile-processing departments, and hospital databases. This rich requirement set serves as a benchmark for evaluating AI classification against expert systems engineering practice.
Figure 3. The Journey of AI Research and Development: From Requirements Acquisition to Findings Reporting Figure 4. Context Diagram of Dr. Tools system.
This adaptation preserves the conceptual integrity of INCOSE while enabling a standardized, AI-friendly evaluation framework. The methodology addresses three research questions: 1. RQ1: To what extent can AI classify engineering requirements according to INCOSE “good requirement” criteria compared to an experienced systems engineer? 2. RQ2: How effectively can AI distinguish between functional and non-functional requirements compared to an experienced systems engineer? 3. RQ3: What are the advantages and limitations of AI compared to human experts in understanding and classifying engineering requirements?
Data Collection Two independent datasets were used: Project A – DR Tool Case Study: Originating from a graduate engineering project at the Holon Institute of Tech-
Project B – PROMISE Dataset: This publicly available corpus contains 969 software requirements (444 functional, 525 non-functional), distributed across twelve non-functional sub-classes (e.g., Security, Usability, Performance). To address class imbalance, SMOTE–Tomek resampling was applied within a stratified ten-fold cross-validation pipeline. Logistic regression achieved 76.16 % ± 2.58 % accuracy, a significant improvement over the 58.31 % ± 2.05 % baseline, underscoring the value of balanced training data. The bar chart in Figure 5, titled “Number of Examples per Class,” presents the distribution of requirement examples across different categories. The data show that Functional Requirements (F) dominate the dataset with 444 examples, significantly more than any other class. Following this, Security (SE) has 125 examples, Usability (US) 85, Operability (O) 77, and Performance (P) 67. The remaining classes such as Look-and-Feel (LF), Availability (A), Maintainability (MN), Scalability (SC), Fault Tolerance (FT), Legal (L), 5
and Portability (PO) contain relatively few examples, ranging from 49 down to 17. This indicates a strong imbalance in the dataset, with functional requirements being the most represented category by a large margin.
Figure 5. Requirement type distribution in the PROMISE dataset.
How the Requirements Were Reviewed The process of reviewing the requirements aimed to ensure that each statement met recognized quality standards and could be effectively analyzed by the proposed AI framework. To achieve this, the study followed the principles outlined in the INCOSE Guide to Writing Requirements (GtWR), which defines the characteristics of a “good requirement.” These principles served as both the foundation for human review and the benchmark for automated evaluation. According to INCOSE, a good requirement must be clear, precise, and feasible. Building on this foundation, the present research examined how Natural Language Processing (NLP) and Machine Learning (ML) techniques can assist in automatically analyzing and classifying requirements. The proposed AI pipeline integrates text classification, clustering, and topic modeling using models such as Support Vector Machines (SVM), Convolutional Neural Networks (CNNs), and Latent Dirichlet Allocation (LDA). To differentiate between functional and nonfunctional requirements, the Quality Requirements Mining and Classification Process was applied, combining Word2Vec and Doc2Vec vectorization methods with CNN architectures. This enables large-scale analysis of Software Requirements Specifications (SRS) in accordance with ISO/IEC 25030 standards. Additionally, the methodology includes AI-assisted test case generation, where LSTM and Transformerbased models extract contextual information to automatically produce preliminary test cases . As part of the review framework, INCOSE’s nine quality traits
were used as the reference point, with terminology adapted to align with ISO/IEC/IEEE 29148:2018, IEEE 830:1998, and the NASA Systems Engineering Handbook (2016). The traits Necessary and Appropriate were replaced with Essential and Independent, emphasizing the need for requirements to be indispensable and solution neutral. The terms Correct and Conforming were excluded from automated assessment, as they require domain-specific and organizational context not yet measurable by AI. Thus, the review process focused on seven key attributes that could be objectively evaluated: Essential, Independent, Unambiguous, Complete, Singular, Feasible, and Verifiable ensuring that each requirement was both human-validated and machine-assessable according to international quality standards. Two characteristics “Correct” and “Conforming” were intentionally omitted from the automated evaluation because: 1. Correct requires domain-specific verification against underlying analysis. 2. Conforming depends on organization-specific style and pattern guidelines. Since these attributes cannot be reliably evaluated through current AI/NLP methods, the research focuses on the seven characteristics that can be assessed objectively: 1. Essential: Defines a necessary capability or constraint. 2. Independent: Specifies what is needed, not how to implement it. 3. Unambiguous: Can be interpreted in only one way. 4. Singular: Expresses a single idea. 5. Feasible: Achievable within project constraints. 6. Verifiable: Can be validated through inspection, analysis, or testing. For the DR Tool, three certified systems engineers independently reviewed each requirement, classifying it as either “good” or “not good.” For requirements deemed not good, they also documented the specific reasons for the quality issues. The reviewers then held a consensus meeting to resolve discrepancies and agree on a final classification. This review and consensus process aligns with the evaluation approach described in Section 3.5. For PROMISE, existing expert labels were validated by a single expert reviewer before use, in accordance with the validation process outlined in Section 3.6.
6
Prompt Engineering
Comparative Analysis
Two separate prompt templates were developed:
Performance metrics were computed in Python 3.11 for each model and dataset:
• Experiment 1 – DR Tool: The prompt instructed the model to evaluate each requirement against the seven INCOSE criteria and explain which were met or violated. The goal was to assess requirement quality. [Appendix B]. • Experiment 2 – PROMISE: The prompt instructed the model to classify each requirement as Functional or Non-Functional and, for the latter, specify the appropriate NF sub-category. Together, these experiments provide complementary insights: quality evaluation (DR Tool) vs. type classification (PROMISE).
Model Execution Model Execution All data were processed in batches of ≤ 3,000 tokens using PromptTag. Each batch was evaluated by ChatGPT-4, Claude Sonnet 3.5, and Meta Llama 3. Raw outputs were stored verbatim for subsequent analysis.
Human Survey (Project A) To benchmark AI performance against human judgment, a Google Forms survey was distributed to over 300 systems engineers worldwide. A set of 20 requirements selected based on model disagreement was used. Participants (n = 21 complete, 2 partial) performed two tasks per requirement: 1. Indicate whether the requirement is acceptable or problematic. 2. Identify which of the seven INCOSE - like criteria are violated. The survey was pilot tested with a small group of participants to identify potential issues and refine question clarity before being distributed to the broader user base. Results from a two-layer human baseline: overall acceptability and specific quality violations [Appendix A].
AI-Only Evaluation (Project B) PROMISE requirements were classified by the three models without human intervention. Each model assigned a label (F/NF) and, where relevant, an NF subclass (e.g., Reliability, Security). Predictions were compared against reference labels.
• Accuracy (quality classification for DR Tool; F/NF for PROMISE) • Inter-model agreement • Precision, recall, and F1 scores (against the human baseline for Project A) Areas of disagreement between AI and humans were analyzed to reveal systematic ambiguity patterns.
Synthesis and Reporting The results from AI evaluations, expert review, and human surveys were triangulated to ensure methodological Validity. A detailed flow diagram outlines inputs, processes, validation gates, and outputs, leading to actionable recommendations for operationalizing AI in requirements engineering.
Results This chapter presents the outcomes of applying the methodology described in Chapter 3. The evaluation consisted of two complementary analyses. First, the ability of a LLM to identify poorly defined system engineering requirements according to the INCOSE Guide to Writing Requirements was compared against survey results provided by professional engineers. Second, the LLM’s capability to classify software requirements into functional and non-functional categories was assessed against the pre-labeled PROMISE_exp dataset.
AI Model Performance in Requirements Classification Section 4.1 presents an in-depth empirical analysis that goes beyond traditional consensus-based evaluation to examine individual engineer assessments across the complete dataset. Rather than simply comparing AI models against a single consensus outcome, we analyze 420 individual assessment points (21 engineers × 20 requirements) to provide a more nuanced understanding of AI model performance in requirements quality assessment. Introduction and Expanded Methodology. The expanded methodology examines how three state-of7
the-art AI language models Claude Sonnet 3.5, GPT4, and Llama 3 perform when compared against each individual engineer’s assessment, revealing patterns of agreement and disagreement that are obscured by consensus-only analysis. This granular approach provides insights into the variability of human expert judgment and the consistency of AI assessments across different evaluation perspectives. Dataset Overview and Expert Panel Statistics. • Total Engineers: 21 systems engineering professionals • Requirements Evaluated: 20 medical equipment tracking system requirements • Individual Assessment Points: 420 (21 × 20) • Valid Responses: 380 (19 engineers × 20 requirements due to some missing responses) • Experience Distribution: 86% with 8+ years, 14% with less than 8 years • AI Models Tested: Claude Sonnet 3.5, GPT-4, Llama 3 Figure 6 presents the distribution of requirements as classified by the engineers who responded to the survey, based on the quality criteria that they fail to meet. Each requirement may be considered noncompliant with more than one criterion, and the percentages represent the proportion of requirements that fail to meet the standards for each criterion. 74% of the requirements were found to be unambiguous , 66% fail the Complete criterion, and 42% Verifiable criterion. Furthermore, 32% Independent criterion, 26% Singular criterion (i.e., they express more than one idea), and when looking at the low percentages, 11% Essential criterion and 9% Feasible criterion (i.e., they are not achievable). This data indicates that clarity is strong, but practicality and testability are limited.
Individual Engineer Assessment Patterns. Analysis of individual engineer assessments reveals significant variability in quality evaluation stringency. While some engineers identified issues in nearly all requirements, others were more lenient in their assessments. This variability provides a crucial context for understanding AI model performance. Engineer agreement varies significantly across requirements, ranging from 10% (RS19.1.1) to 84% (RS1.1.1 and RS11.0.1) finding quality issues. This variation indicates that some requirements have obvious defects while others are subject to interpretation differences among experts. AI Model Performance Against Individual Engineers. When comparing AI models against individual engineer assessments rather than just consensus, a more complex performance picture emerges. We calculate accuracy for each AI model against all 380 individual assessment points, providing a statistically robust evaluation of model performance. Detailed Performance Metrics. Table ?? summarizes how often each model produced the same answer as the engineers. • Claude Sonnet 3.5 achieved the strongest performance, matching human judgments in 85% of the comparisons. • GPT-4 and Llama 3 showed substantially lower agreement, at approximately 45-48%. The final column, Standard Deviation (Std. Deviation), indicates how consistent each model was across different engineers: • A smaller value (e.g., ±12.3%) indicates stable behavior, meaning the model performed similarly with all evaluators. • A larger value (around ±18%) suggests the model’s accuracy varied significantly, agreeing well with some engineers but poorly with others. In summary, Claude Sonnet 3.5 was both the most accurate and the most consistent model. In contrast, GPT-4 and Llama 3 demonstrated lower accuracy and greater variability across individual evaluators.
Figure 6. INCOSE Quality Criteria Violations.
Key Findings and Implications. The key findings reveal strong differentiation in model performance and human consistency. When compared against 380 individual engineer evaluations, Claude Sonnet 3.5 achieved an impressive 85% accuracy, significantly outperforming GPT-4 (45%) and Llama 3 8
Req. ID RS1.1.1 RS2.0.1 RS3.0.1 RS5.1.2 RS6.0.1 RS9.1.1 RS11.0.1 RS14.0.1 RS19.2.3 RS19.2.4 RS8.0.1 RS11.0.2 RS24.1.1 RS18.2.2 RS1.2.1 RS9.1.2 RS15.0.1 RS17.0.1 RS19.1.1 RS19.2.1
Engineers Finding Issues 16 14 13 10 12 15 16 16 11 12 13 14 11 15 14 16 13 8 2 12
Total Responses 19 19 19 19 19 20 19 20 20 20 20 20 20 20 20 20 20 20 20 20
Percentage 84% 74% 68% 53% 63% 75% 84% 80% 55% 60% 65% 70% 55% 75% 70% 80% 65% 40% 10% 60%
Consensus Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Not Good Disputed Good Not Good
Table 1. Engineers findings per requirement
Model Claude Sonnet 3.5 GPT-4o Llama 3
Point Estimate 85.0% 45.0% 47.9%
95% CI Lower 81.2% 40.1% 42.8%
95% CI Upper 88.3% 49.8% 53.1%
Margin of Error 3.55% 4.85% 5.15%
Table 2. Performance Confidence Intervals (95% CI) via Bootstrap Resampling (n = 10, 000)
(47.9%). Claude also showed the highest consistency across evaluators, with a performance range of only 25%, while GPT-4 and Llama 3 fluctuated more widely (range: 35%). These performance gaps are statistically significant, as demonstrated by the non-overlapping 95% confidence intervals derived via bootstrap resampling with 10,000 iterations.
Classification of Functional and Non-Functional Software Requirements (LLM vs. PROMISE_exp Dataset) The PROMISE_exp dataset comprises 969 software requirements, of which 444 are functional and 525 are non-functional. The non-functional requirements are further divided into eleven subcategories: Quality (SE, 125), Usability (US, 85), Interface (O, 77), Performance (PE, 67), Legal and Finance (LF, 49), Availability (A, 31), Maintainability (MN, 24), Security (SC, 22), Fault Tolerance (FT, 18), and Portability
(L, 15 & PO, 12). Three large language models Claude 3.5 Sonnet, GPT-4o, and Llama 3.0 were tasked first with distinguishing functional vs. non-functional requirements and then assigning non-functional items to the correct subcategory. Model outputs were compared against the ground-truth labels to assess per-category accuracy, identify best performers, and quantify performance gaps. NASA and INCOSE do not explicitly prescribe a fixed taxonomy for nonfunctional requirements (NFRs). In practice, several frameworks are used, and in Israel, a common approach is to map PROMISE dataset codes to standard Systems Engineering (SE) NFR categories. Table 3 presents a mapping that enables a unified classification across projects, applied in our working environment as stated above, facilitating comparative analysis and traceability. Table 4 (Original Promise_ext break down), provided fundamental insights into the distribution of requirement types used in the study. This data served as the bedrock for evaluating the performance of 9
SE NFR (Canonical) Functional Security Usability (incl. Look-and-feel) Operability / Supportability Performance Availability Maintainability Scalability Reliability (incl. Fault-Tolerance) Legal / Compliance Portability
PROMISE Types Mapped F SE US, LF O PE A MN SC FT L PO
Table 3. Aligning PROMISE_exp requirements with systems-engineering non-functional taxonomy.
Requirement Type Functional Requirement Availability Legal Look-and-feel Maintainability Operability Performance Scalability Security Usability Fault Tolerance Portability Total
Code F A L LF MN O PE SC SE US FT PO
Count 444 31 15 49 24 77 67 22 125 85 18 12 969
Table 4. Original Promise_ext break down
the artificial intelligence models in classifying software requirements as Functional (FR) versus NonFunctional (NFR). Classification Distribution. Figure 7 illustrates how different models compare to the reference classification by showing the proportion of items each one assigns to the two categories. It highlights the overall tendency of each model and allows a visual comparison of how their distributions align with or diverge from the reference pattern. It summarizes the comparative distribution of functional and nonfunctional classifications across the ground truth and three large language models.
Figure 7. FR vs. NFR Prediction Distribution
As we can observe in figure 7 the original data base exhibits a relatively balanced distribution, with 45.8% of the requirements labeled as Functional and 54.2% as Non-Functional. In contrast, Claude 3.5 displays a strong functional bias, assigning 70.6% of the items to the Functional category and only 29.4% 10
to the Non-Functional category. GPT-4o shows a milder deviation, producing 37.4% Functional and 62.6% Non-Functional classifications, indicating a moderate inclination toward Non-Functional labeling. Llama 3.0 diverges the most from the ground truth, with 29.3% classified as Functional and 70.7% as Non-Functional, reflecting a pronounced nonfunctional bias. Collectively, these discrepancies highlight systematic tendencies within each model, demonstrating that none of them accurately replicates the empirical distribution. Each model exhibits a distinct directional bias—functional overestimation in Claude and non-functional overestimation in GPT-4o and Llama—indicating differing sensitivities to linguistic cues present in the dataset. Model Performance Observations. The following points highlight behavior differences identified between models, addressing recall differences, subtype trends, and category-specific patterns. Recall measures a model’s ability to correctly identify all relevant instances of a given class, defined as the proportion of true positives among all actual positives, reflecting how many relevant instances are successfully detected versus missed. These results are summarized in Table 5. Functional Requirements (n = 444): • Llama 3.0 achieved the highest recall at 86.3%, representing a 41.7-point advantage over Claude. This performance is attributed to effective pattern matching when “shall. . . ” constructs are used in requirement statements. • Claude exhibited low recall (44.6%), indicating a bias toward labeling requirements as nonfunctional. Non-Functional (n = 525):
Requirement
Subcategories
• Claude 3.5 Sonnet dominated all NFR subtypes, with particularly strong performance in Maintainability (91.7%) and Fault Tolerance (88.9%), reflecting its strength in detecting nuanced attribute-oriented language. • The largest performance gaps were observed in less frequent categories (FT, MN, LF), where specialized terminology and smaller sample sizes amplify inter-model differences. • Perfect scores in Portability (L and PO) indicate consistent keyword cues that the model reliably captures.
This study demonstrates that large language models can effectively automate the distinction between functional requirements (FR) and non-functional requirements (NFR); however, performance varies by task formulation and prompting strategy. Specifically: • FR versus NFR classification is best handled by GPT-4o (approximately 85% accuracy) when using a clear, single-label prompt (“Classify as Functional or Non-Functional”). • NFR subcategory assignment benefits from models specialized in attribute-based language; Claude 3.5 Sonnet consistently outperformed other models across quality, performance, security, and reliability categories. • Simple “shall. . . ” statements representing pure functional requirements are reliably recognized by Llama 3.0; however, it tends to underdetect NFRs without additional prompting. Claude 3.5 processed all 969 requirements correctly. GPT-4o and Llama 3.0 produced 12 % duplicate or split entries.
Conclusions and Contributions This research examined the use of Large Language Models (LLMs) as AI “copilots” in requirements engineering, with a focus on their ability to support requirement quality assessment and classification within an INCOSE-aligned framework. The findings demonstrate that AI can assume a portion of routine analytical work, provided its deployment is embedded within a structured human-in-the-loop (HITL) process. Regarding RQ1, the study shows that modern LLMs can classify requirements according to INCOSE “good requirement” criteria with accuracy comparable to experienced systems engineers. High alignment was observed in identifying problematic requirements, particularly for structural and linguistic defects. However, a consistent rationale gap was identified: while AI and human experts often agree on the need for revision, they may differ on the underlying cause. AI emphasizes formal rule violations, whereas human experts prioritize ambiguity, missing context, and conceptual intent. For RQ2, the analysis confirms that AI can distinguish between functional and non-functional requirements by identifying underlying intent rather than relying solely on syntax. Different models ex11
Model Llama 3.0
Task Functional Requirements (FR)
Recall (%) 86.3%
GPT 4.0
Functional Requirements (FR)
44.6%
Non-Functional Requirements (NFR)
>88% (up to 91.7%)
Claude 3.5 Sonnet
Key Observations Highest recall among all models; demonstrates strong pattern matching, especially for explicit “shall”-based requirement formulations. Low recall due to systematic bias toward labeling requirements as non-functional, even when functional intent is explicit. Dominates all NFR subcategories; excels in Maintainability (91.7%) and Fault Tolerance (88.9%) due to sensitivity to nuanced, attribute-oriented language.
Table 5. Comparative recall performance across models for Functional and Non-Functional Requirement classification.
hibit distinct classification biases, indicating that model selection must align with project priorities, especially in safety-critical or performance-driven systems. In addressing RQ3, the results highlight AI’s primary advantages in efficiency, consistency, and repeatability. AI applies evaluation criteria uniformly and does not suffer from cognitive fatigue or subjective drift. Conversely, AI’s limitations remain noticeable at this stage of the technology: it does not provide totally accurate performance with regard to technical truth, physical feasibility, and cross-document engineering logic, and remains susceptible to hallucinations. These gaps between AI capabilities and human expertise directly inform how AI should be operationalized. Rather than viewing AI’s limitations as a barrier, they define a clear boundary for a workflow shift in requirements engineering. By delegating structural and linguistic auditing to AI, systems engineers can redirect cognitive effort toward high-value validation activities, including feasibility assessment, trade-off analysis, and architectural reasoning. To operationalize this shift, we propose a concrete, AI-assisted workflow allocation. Table 6 maps the specific responsibilities of the AI copilot versus the human expert across the seven evaluated INCOSE criteria. Based on this allocation, a recommended three-step workflow emerges for industrial practice: 1. Initial AI Pre-Audit: The LLM autonomously scans draft requirements, resolving Singular/Unambiguous violations and tagging issues related to Verifiability and Independence.
2. Human Review & Reconciliation: The systems engineer reviews AI-flagged items, accepting or modifying the structural recommendations and resolving contextual nuances. 3. Expert Validation: Relieved of basic syntax checking, the engineer focuses deeply on the Essential and Feasible criteria, applying domain expertise and architectural judgment. This approach demonstrates that LLM-based tools can perform structural and linguistic analysis in alignment with INCOSE guidelines, achieving near expert-level agreement for routine tasks. By introducing AI as a standardized pre-audit layer, organizations can reduce early lifecycle review effort and peer-review bottlenecks, while strictly preserving expert authority over system validation and decisionmaking.
Future Applications and Potential Challenges AI presents significant opportunities for RE that can be expected, including automated requirements elicitation from stakeholder interviews, synthesis of requirements from high-level goals, personalization of documentation for diverse audiences, and continuous monitoring for evolving needs. Large language models can already draft candidate requirements, detect ambiguities, and propose refinements and are continuously evolving. Integration of AI with alternative media such as visual models could enhance clarity and stakeholder communication (Kaur and Kaur, 2024). Key challenges remain: ensuring validation and trust in AI outputs, establishing accountability for AI-generated requirements, safe12
INCOSE Criterion
AI Copilot Role (Pre-Audit)
Unambiguous
Flags vague terminology and linguistic inconsistencies. Identifies and splits compound requirements (e.g., detecting "and/or"). Detects implementation-specific language and design bias. Highlights missing parameters, units, or standard constraints. Checks for quantifiable metrics and measurable targets. Limited. Flags potential duplicates or outof-scope keywords. Limited. Flags extreme metric anomalies based on training data.
Singular Independent Complete Verifiable Essential Feasible
Human Expert Role (Validation & Decision) Resolves domain-specific ambiguity and contextual nuances. Reviews and approves the separated requirement statements. Ensures true solution neutrality based on system architecture. Validates conceptual completeness against stakeholder needs. Confirms actual testability within project resources and constraints. Primary Driver. Determines necessity for the system’s core mission. Primary Driver. Assesses technical, budgetary, and schedule realism.
Table 6. Decision Allocation: Mapping AI vs. Human Responsibilities across INCOSE Quality Criteria.
guarding sensitive data, maintaining ethical and regulatory compliance, and integrating AI tools into the inherently social RE process. Training engineers to use AI effectively is also critical, especially in the range of possibilities for automation, but also notably in the limitations as described in this research. There is a need for further empirical evidence on realworld adoption, frameworks for responsible AI use, and multidisciplinary collaboration. With careful governance, AI combined with established RE practices is starting to show a potential to improve efficiency, manage complexity, and enhance requirement quality, ultimately leading to better systems, but as noted, more empirical research is required.
Acknowledgments The conceptual design of the research, its implementation, analysis of results and conclusions are original authors’ work. ChatGPT 5.2 assisted with consistency checking of the structure and flow of the paper, assisting in identifying repetitive paragraphs, and English corrections. It assisted in a limited way in the generation of the paper’s abstract which was extensively revised by the authors.
References AI Index Steering Committee. (2024). AI index 2024 annual report. Baidya, S., Das, S. K., Uddin, M. H., Kosek, C., & Summers, C. (2022). Digital twin in safetycritical robotics applications: Opportunities and challenges. Proceedings of the 2022 IEEE International Performance, Computing, and
Communications Conference (IPCCC), 101– 107. https:// doi.org/ 10.1109/ IPCCC55026. 2022.9894313 Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), 610–623. Cheligeer, C., Huang, J., Wu, G., Bhuiyan, N., Xu, Y., & Zeng, Y. (2022). Machine learning in requirements elicitation: A literature review. AI EDAM, 36, e32. https : / / doi . org / 10 . 1017 / S0890060422000103 Cleland-Huang, J., Settimi, R., Zou, X., & Solc, P. (2006). The detection and classification of non-functional requirements. Proceedings of the 14th IEEE International Requirements Engineering Conference (RE’06), 39–48. https:// doi.org/10.1109/RE.2006.65 Dalpiaz, F., & Niu, N. (2020a). Requirements engineering in the days of artificial intelligence. IEEE Software, 37(4), 7–10. https : / / doi . org / 10.1109/MS.2020.2986047 Dalpiaz, F., & Niu, N. (2020b). Requirements engineering in the days of artificial intelligence. IEEE Software, 38(4), 7–12. https : / / doi . org / 10.1109/MS.2020.2973364 Hadar, A., Levy, N., & Winokur, M. (2022). Management and detection system for medical surgical equipment. https:// arxiv.org/ abs/ 2211. 02351 INCOSE. (2023). Guide to writing requirements (version 4). International Council on Systems Engineering. https : / / www . incose . org / 13
docs / default - source / working - groups / requirements-wg/gtwr/incose_rwg_gtwr_v4_ 040423_final_drafts.pdf?sfvrsn=5c877fc7_2 Kaur, K., & Kaur, P. (2024). The application of AI techniques in requirements classification: A systematic mapping. Artificial Intelligence Review, 57, Article 57. https://doi.org/10.1007/ s10462-023-10667-1 Kolligs, J. W., & Thomas, L. D. W. (2025). Characterizing efficacy of alternative media for requirements expression. Systems, 13(5), 314. https: //doi.org/10.3390/systems13050314 Maleki, N., Soni, P., Padmanabhan, K., & Dutta, K. (2024). AI hallucinations: A misnomer worth clarifying. 2024 IEEE Conference on Artificial Intelligence (CAI), 133–138. https:// doi.org/ 10.1109/CAI58096.2024.00026 Martínez, G., Conde, J., Reviriego, P., MerinoGómez, E., Hernández, J. A., & Lombardi, F. (2023). How many words does ChatGPT know? the answer is ChatWords. arXiv preprint arXiv:2309.16777. https : / / doi . org / 10.48550/arXiv.2309.16777 Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., et al. (2022). Software engineering for AI-based systems: A survey. ACM Transactions on Software Engineering and Methodology, 31(4). https://doi. org/10.1145/3557944 NASA. (2016). Nasa systems engineering handbook, chapter 2.0: Fundamentals of systems engineering [NASA/SP-2016-6105 Rev 2]. National Aeronautics and Space Administration. https: / / www . nasa . gov / reference / 2 - 0 fundamentals-of-systems-engineering/ Peer, J., Mordecai, Y., & Reich, Y. (2024). NLP4ReF: Requirements classification and forecasting: From model-based design to large language models. 2024 IEEE Aerospace Conference, 1– 16. https : / / doi . org / 10 . 1109 / AERO58547. 2024.10521022 Pérez-Cerrolaza, J., Abella, J., Borg, M., Donzella, C., Cerquides, J., Cazorla, F. J., & Flores, J. L. (2024). Artificial intelligence for safetycritical systems in industrial and transportation domains: A survey. ACM Computing Surveys, 56(7), 1–40. https : / / doi . org / 10 . 1145 / 3626314 Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human-Computer Interaction, 36(6), 495–504.
Siddique, I. M. (2022a). Harnessing artificial intelligence for systems engineering: Promises and pitfalls. European Journal of Advances in Engineering and Technology, 9(9), 67–72. https: //doi.org/10.5281/zenodo.11545453 Siddique, I. M. (2022b). Systems engineering in complex systems: Challenges and strategies for success. European Journal of Advances in Engineering and Technology, 9(9), 61–66. https: //doi.org/10.5281/zenodo.11545350 Tamai, T., & Anzai, T. (2018). Quality requirements analysis with machine learning. Proceedings of the 13th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE), 241–248. Tesla. (2025). Artificial intelligence at Tesla. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762
14