arXiv CSopen access

AI-Driven Test Case Generation from Natural Language Requirements: A Survey of Techniques and Research Gaps

AI-Driven Test Case Generation from Natural Language Requirements: A Survey of Techniques and Research Gaps Orimoloye Folorunsho, Hassan Reza School of Electrical Engineering and Computer Science University of North Dakota, Grand Forks, ND 58201, USA [email protected]; [email protected]

Abstract Software testing is critical for verifying that systems meet their specified requirements, yet it remains among the most time-consuming and expensive activities in development. While requirements-based test generation allows test cases to be derived early from requirements artifacts, generating them directly from natural language is challenging due to the inherent ambiguity and imprecision of NLP. Recent advances in AI, natural language processing (NLP), and large language models (LLMs) have made automating this pipeline increasingly feasible, but have also introduced new risks, including hallucination, reduced traceability, and inconsistent evaluation. This survey attempts to address four research questions: what AI and NLP techniques have been proposed for generating test cases from natural language requirements; what tools and frameworks support these approaches; how generated test cases are evaluated; and what research gaps remain. Following the systematic review guidelines of Kitchenham and Charters, we searched major scholarly databases spanning 2000–2025 and, after applying strict inclusion criteria, identified 21 primary studies. The literature is organized into three evolutionary eras, revealing that no existing approach simultaneously satisfies six key quality dimensions: automation, ambiguity handling, domain applicability, traceability, evaluation thoroughness, and hallucination control. The survey makes three main contributions: a three-era evolutionary synthesis of AI-based test generation; a sixcriteria gap analysis showing that no current approach fully addresses all quality dimensions; and four actionable research guidelines targeting hallucination, traceability, complexity sensitivity, and compliance. Together, these position this work as a focused, LLM-era-inclusive synthesis of test-case generation from natural-language software requirements. Keywords - Software testing, natural-language requirements, natural-language processing, large language models, requirements-based testing, automated test generation, AIbased testing. I. INTRODUCTION The demand for higher software quality and better customer experience, together with the growing complexity of modern systems, underscores the critical role of software testing in contemporary engineering practice. This need is especially evident in safety-critical domains such as aviation, automotive, finance, healthcare, and aerospace, where testing workloads are substantial, and defects can have severe consequences. Organizations must therefore ensure not only that software behaves as intended, but also that it does not behave in unintended ways or produce unintended outcomes. Software testing remains one of the most widely used means

of assessing system reliability, quality, and functionality [36]. In a typical software project, approximately 50 percent of elapsed time and more than 50 percent of total development cost are devoted to testing activities [36]; in many projects, more than half of the overall schedule is now allocated to testing [37]. The software testing process is commonly organized into three stages: test case generation, execution, and evaluation [22]. Among these stages, test case generation is widely regarded as the most challenging and labour-intensive [22], [31]. During this phase, testers construct test cases to verify that functional requirements and design specifications are satisfied [31]. Although test cases may be created manually or generated automatically with tool support, manual construction is tedious, error-prone, and often inconsistent across projects and teams [4]. As a result, automating test case generation has become increasingly important in agile and continuous integration environments, where it helps reduce costs and effort while improving consistency [4]. Requirements-based test generation (RBTG) addresses this challenge by deriving functional test cases directly from requirements specifications or design artifacts rather than from source code [22]. Applying RBTG early in the software development life cycle enables testers to detect ambiguities and irregularities in requirements before defects propagate to later stages [1], [22]. However, RBTG also faces inherent limitations. Although natural language is widely used to document software requirements because of its accessibility and expressiveness [1], [39], it is often ambiguous and imprecise [24], [39]. This ambiguity complicates the direct automation of test-case generation from natural-language requirements [22]. In recent years, advances in artificial intelligence (AI) and natural language processing (NLP) have significantly expanded the potential to automate software testing. AI and NLP-based techniques, including syntactic and semantic analysis, information extraction, and learning-based reasoning, enable free-form requirement statements to be transformed into structured testing artifacts. For example, a requirement such as “the system shall display an error message if a user enters an invalid phone number” can be decomposed into test scenarios with explicit actors,

preconditions, actions, and expected results [24]. Prior studies show that AI-driven test generation can accelerate testing, reduce manual effort, detect inconsistencies and anomalies, improve test procedures, and yield more effective test case designs [24], [25]. Likewise, NLP-based approaches can improve communication between developers and testers while supporting earlier detection of ambiguity and inconsistency in requirements [24], [27]. Among these advances, NLP plays a foundational role by providing the mechanisms for transforming free-form requirements into structured, testable representations. Natural-language requirements are among the most accessible yet most ambiguous artifacts in the software development life cycle. Consequently, each stage of the test case-generation pipeline, from identifying actors and actions to extracting conditions, resolving coreferences, and mapping requirement structure to test case fields, depends on recovering linguistic structure from free-form text. NLP provides the operational basis for this recovery. In particular, part-of-speech tagging and dependency parsing identify syntactic relations; semantic role labelling and named-entity recognition expose actoraction-object triples that map naturally to test case fields; and transformer-based contextual embeddings enable systems to generalize beyond fixed templates to the varied phrasings found in industrial requirements. The primary studies examined in this survey advance the field along one or more of these dimensions. For this reason, an NLP-focused perspective, rather than a purely model-driven or formalspecification-driven one, provides the most appropriate organizing framework for this survey. However, the recent shift from traditional NLP pipelines to transformer-based LLMs has changed not only the scale of automation achievable but also the nature of the risks involved. Transformer-based large language models (LLMs) have substantially improved the semantic interpretation of natural-language requirements, but they have also introduced new risks, particularly hallucination, in which generated test cases are fluent yet semantically inconsistent with their source requirements. Li et al. [53] quantified this phenomenon in the HaluEval benchmark, reporting that 977 of 5,000 annotated ChatGPT responses (19.5 percent) contained hallucinations. Arora et al. [15] examined this risk directly through retrievalaugmented generation, providing the clearest primary study evidence to date of an architectural mitigation strategy. This emerging trustworthiness concern motivates Gap G1 (the absence of Hallucination Detection in LLM-Generated Test Cases), which is examined in this survey. More broadly, these developments expose limitations in the existing secondary literature, which has yet to provide a focused synthesis of AI/NLP-driven test case generation from natural-language requirements in the LLM era. Against this background, this survey differs from prior reviews in four main respects. First, in terms of scope, Yang et al. [22] survey the broader requirements-based test generation literature across 267 primary studies, in which formal specification, model-based, and search-based approaches predominate. By contrast, this survey applies strict inclusion criteria (I4: natural-language input; I5: empirical

AI/NLP test case-generation validation) to isolate the AI/NLP-driven subset that is not separately resolved in Yang’s broader taxonomy. Second, in terms of temporal coverage, Boukhlif et al. [24] and Garousi et al. [27] examine only pre-LLM NLP-based testing, whereas Mustafa et al. [23] and Ahsan et al. [40] cover the 2000-2018 and 2005-2014 periods, respectively. This survey extends through October 2025 and, to the best of our knowledge, is the first to integrate primary studies from the LLM and hybrid eras alongside earlier rule-based and statistical-NLP work. Third, in terms of analytical focus, Wang et al. [25] survey LLM-based software testing broadly and treat natural-language input as only one of several input modalities. In contrast, this survey focuses specifically on the natural-language-to-test-case pipeline, where the risk of hallucination and the tension between traceability are especially pronounced. Fourth, in terms of contribution, prior surveys mainly provide taxonomies and classifications. In contrast, this survey presents a six-criterion gap analysis showing that no existing approach satisfies all six dimensions simultaneously and derives four actionable guidelines with explicitly defined deliverable artifacts. Taken together, these distinctions position this survey as the first AI/NLP-focused, LLM-era-inclusive, and gap-driven treatment of test case generation from natural-language requirements. Motivated by these gaps in the existing review literature, this paper presents a focused survey of AI and NLP-based test case generation from natural-language software requirements, spanning early rule-based NLP systems, hybrid model-driven approaches, and modern transformer-based LLM techniques. The main contributions of this work are as follows: (C1) a three-era evolutionary synthesis grounded in twenty-one confirmed primary studies; (C2) a Related Work Summary table comparing approaches, techniques, inputs and outputs, assumptions, and limitations; (C3) a six-criteria comparison and gap-analysis table showing that no existing approach simultaneously satisfies all six quality dimensions; and (C4) four concrete, actionable research guidelines addressing the identified gaps. To support this objective, the remainder of this paper is organized as follows. Section II reviews background on requirements-based testing, NLP in software engineering, and the motivation for integrating AI techniques. Section III presents the research questions and the systematic literature review methodology, including the search strategy, inclusion and exclusion criteria, study selection, and quality assessment. Section IV analyzes traditional NLP-based test case generation approaches, whereas Section V examines AIdriven and LLM-based techniques. Section VI compares tools and frameworks using the Related Work Summary and Six Criteria Comparison tables. Section VII identifies four research gaps (G1–G4) with empirical support from the primary studies. Section VIII proposes four actionable guidelines (R1–R4) that address these gaps. Section IX concludes the paper. II. BACKGROUND Following the motivation, scope, and contribution established in the Introduction, this section presents the

2

conceptual foundation for the subsequent analysis. It first examines requirements-based test case generation as the immediate problem context, then considers the role of natural language processing in software engineering as the enabling technical basis and finally highlights the emergence of artificial intelligence and large language models as a significant shift in the field. Together, these perspectives show the progression from formal, rule-based approaches to contemporary AI-driven methods for generating test cases from natural language requirements. A. Requirements-Based Test-Case Generation Requirements are central to software development because they specify functionality, usability, performance, reliability, security, and design constraints [22]. They serve as a blueprint for both developers and testers, promoting clarity and reducing miscommunication across the development life cycle [22], [23]. Consequently, the quality of software development and testing depends heavily on the quality of the underlying requirements [23]. Requirements-based testing derives test cases from requirements while abstracting the implementation's internal structure [41]. In this context, requirements are categorized as functional, non-functional, or domain specific [40]. Functional requirements describe what the system must do, non-functional requirements define quality attributes and constraints, and domain requirements capture application-specific properties that may be functional or non-functional. Bruel et al. [39] identified five main approaches for expressing software requirements, which are distinguished by the primary notation or style used in specifying the requirements: •

Natural language is a fully informal approach that expresses requirements using English or other human languages.

•

Semi‑formal approaches combine natural language with structured modelling elements, such as the Systems Modelling Language (SysML) [42] and the Unified Requirements Modelling Language (URML) [43].

•

Seamless approaches integrate requirements directly with implementation and design through programming language constructs [39], [44], exemplified by Seamless Object‑Oriented Requirements (SOOR) [44].

•

Automata and graph‑based approaches rely on graph‑theoretical representations [6] to specify requirements, including UML activity and sequence diagrams [45], labelled transition systems [45], and finite‑state machines [46].

•

Mathematical approaches employ formal notations [22] or model‑checking‑based methods [47] to develop specifications and generate test cases, using formal specification languages such as Z [47], [50] and the Structured Object‑Oriented Formal Language (SOFL) [49], [48], the latter exemplified by tool support for SOFL‑driven automatic test case generation and result analysis.

These requirement expression styles differ substantially in their suitability for requirements-based test generation (RBTG). Mathematical, automata-based, and seamless approaches provide formally structured representations that support deterministic test case derivation, but they impose high modelling overhead and require specialized expertise. As

a result, their adoption is largely concentrated in safety-critical or highly regulated domains. By contrast, natural language and semi-formal requirements dominate industrial practice because they are more accessible and flexible, despite their inherent ambiguity and imprecision. This tension has increasingly shifted RBTG research toward bridging the gap between informal or semi-structured natural-language requirements and executable test artifacts, thereby motivating the use of natural language processing, machine learning, and, more recently, large language models to recover structure, intent, and testable semantics from free-form requirement text. B. Natural-Language Processing In Software Engineering. Manual testing is often costly and labor-intensive. To mitigate these challenges, NLP techniques such as language modelling, named-entity recognition, and text embedding enable advanced automated text processing, which can substantially reduce testing effort and cost [51]. More broadly, NLP is grounded in decades of research on text mining, statistical language modelling, and information extraction. Kao and Poteet [38] provide a foundational reference covering these core NLP and text-mining techniques, many of which underpin modern software engineering applications. In recent years, software engineers have increasingly applied NLP to analyze human language and design artifacts and to automate tasks such as test case generation [5], [7], [51]. Zhao et al. [28] systematically mapped 404 studies on NLP for requirements engineering and showed that techniques for requirements analysis, extraction, and classification form a direct technical substrate for the downstream test-casegeneration pipelines examined in this survey. NLP converts free-form text into structured representations through techniques such as tokenization, part-of-speech tagging, syntactic parsing, semantic role labeling, and coreference resolution [24]. For example, Viggiato et al. [51] used tokenization to decompose requirement sentences into word sequences, which were then reduced to their root forms through lemmatization. Similarly, Carvalho et al. [3] and Verma and Beg [2] showed how parse trees and controlled natural languages can be used to map requirement sentences directly to test case structures. Early approaches in this area relied primarily on rule-based parsers. In contrast, more recent work has shifted toward machine-learning pipelines built on models such as conditional random fields, recurrent neural networks, and transformers [20], [24]. This evolution reflects a broader move toward data-driven and representation-learning approaches that can better accommodate the linguistic variability and contextual complexity of natural language. C. Motivation for AI Integration Traditional NLP techniques have long been applied to software engineering tasks, including software testing. Although these methods are effective for capturing syntactic structure and limited semantic information, they are less effective for domain adaptation, contextual reasoning, and implicit requirement semantics. Building on this foundation, machine-learning techniques and, more recently, large language models (LLMs) have substantially expanded these capabilities by learning mappings between natural language requirements and executable test artifacts [20], [25]. ML and LLM-based approaches can automatically identify edge cases, constraints, and negative scenarios, all of which are essential in Agile and DevOps environments [35]. In

3

addition, LLMs can interpret paraphrases, reason over contextual variation, and generate executable code for test scripts [20], [33]. These advances position AI-driven techniques as a natural progression beyond rule-based and feature-engineered NLP pipelines. Tertiary evidence further supports this shift. Amalfitano et al. [29], in a tertiary study of twenty secondary reviews, confirm that the use of artificial intelligence in software testing is now well established. However, they also observe that test case generation remains under-mapped at the secondary study level, thereby reinforcing the need for the focused synthesis undertaken in this survey. Operational precedents further illustrate both the promise and the limitations of current approaches. Sawada et al. [30] demonstrate intelligent requirements-to-test traceability at NASA-JPL using NLP and ML techniques, providing concrete evidence that AI-driven traceability is feasible even in safety-critical settings. At the same time, their work shows that such capabilities are typically realized through customized pipelines rather than end-to-end generative systems. This distinction directly anticipates the traceability challenge later formalized as Gap G2 in this survey. III. SYSTEMATIC REVIEW METHODOLOGY To ensure a comprehensive and unbiased review of the literature, we conducted a systematic literature review (SLR) following the guidelines proposed by Kitchenham and Charters [52]. This section outlines the review protocol, including the research questions, search strategy, study selection criteria, quality assessment, data extraction, and synthesis methods. We developed the SLR protocol for this survey paper by clearly defining the research objectives. We formulated research questions focused on key aspects of AI-driven testcase generation from natural language requirements. We then used a comprehensive search strategy across multiple academic databases and digital libraries. To ensure only relevant, high-quality research articles were included, we applied specific inclusion and exclusion criteria. The studyselection process involved screening titles, abstracts, and full texts, followed by a thorough review using a qualityassessment checklist to evaluate methodological rigor and relevance. For data extraction, we systematically recorded key information from each selected study, including the proposed techniques, the tools and frameworks developed, the evaluation metrics, and the identified gaps, using a standardized form. Next, during the synthesis phase, we organized and summarized the findings. This approach enabled clear comparison and identification of trends across the literature. By following this method, the review ensures transparency, reproducibility, and comprehensive coverage of the topic, ultimately providing a solid foundation for mapping the current state of the art and guiding future research directions. A. Research Goals and Objectives The primary goal of this research is to systematically map and synthesize the current state of the art in generating test cases from natural language software requirements using artificial intelligence (AI) and natural-language processing (NLP) techniques. The objectives are:

• To identify and categorize the various AI and NLP techniques proposed for test case generation from natural language software requirements. • To review and analyze the tools and frameworks available for supporting AI-based test case generation. • To examine how generated test cases are evaluated and the metrics commonly used for their assessment. • To highlight existing research gaps, limitations, and challenges in the field, thereby guiding future research directions. These goals and objectives provide clear direction for the research, ensuring that the systematic review focuses on the most relevant aspects of AI-driven test-case generation from natural language requirements. The review aims to answer the following research questions (RQs): RQ1: What techniques have been proposed for generating test cases from natural language software requirements using AI and NLP? RQ2: What tools and frameworks have been developed to support AI-based test case generation from NLSRs? RQ3: How are the generated test cases evaluated, and what metrics are used? RQ4: What are the main research gaps and challenges in this area? These research questions are operationalized by explicitly mapping them to the survey’s evidence base. RQ1 (techniques) is addressed in Sections IV and V through the corpus of twenty-one primary studies organized across the three evolutionary eras. RQ2 (tools and frameworks) is addressed in Section VI, where Table III summarizes related work and provides a cross-cutting comparison of tools. RQ3 (evaluation) is addressed in Section VI through Table IV, particularly criterion C5 on evaluation thoroughness. RQ4 (gaps and challenges) is addressed in Section VII through four formalized research gaps: G1 (hallucination), G2 (traceability), G3 (complexity sensitivity), and G4 (compliance). Each gap is grounded in specific cell values in Table IV and in external quantitative baselines from [16] and [53] and is mapped one-to-one in Fig. 7 to the four actionable recommendations (R1-R4) developed in Section VIII. This explicit chain from research question to section, table, gap, and recommendation ensures that each finding remains traceable to its evidentiary basis. These questions guide the search, selection, and synthesis of primary studies. B. Search Strategy We conducted a systematic search of the following digital libraries and databases, covering the period from January 2000 to October 2025: IEEE Xplore, Engineering Village, ScienceDirect (Elsevier), ACM Digital Library, SpringerLink, arXiv.org (for recent preprints), and published books and other journals/databases. We constructed search strings by combining keywords related to the core concepts, specifically test-case generation,

4

natural language requirements, and AI/NLP techniques. To ensure broad coverage, we applied Boolean operators and wildcards. The core search query was: ("Test Case Generation" AND "Artificial Intelligence" AND "Natural Language Requirements") AND ("natural language processing" OR "natural language processing systems" OR "software testing" OR "software engineering" OR "requirements engineering" OR "specifications" OR "large language models" OR "natural languages" OR "specification languages" OR "automatic test pattern generation") We further refined the searches using specific technique names such as rule-based, UML, OCL, transformer, GPT, and performed backward and forward snowballing on key papers, including full inspection of the 324-reference list of Yang et al. [22]. C. Inclusion and Exclusion Criteria We included studies that met the following criteria: •

I1: Peer-reviewed journal articles, conference papers, or authoritative arXiv preprints.

•

I2: Published between 2000 and 2025.

•

I3: Written in English.

•

I4: Focus on generating test cases (or test artifacts) from natural language requirements.

•

I5: Use some form of AI or NLP techniques, for example, rule-based systems, statistical ML, deep learning, LLM, or hybrid approaches.

•

I6: Present empirical results or a clear methodological contribution.

due to factors such as a lack of NLP focus, insufficient detail, failure to address test generation jointly under I4 and I5, or venue quality concerns under E6. •

Inclusion: A total of twenty-one primary studies were ultimately included in the review after strict application of I4, I5, and E6. Five of these studies were identified via backward snowballing on the reference list of Yang et al. [22].

Figure 1 presents the four-stage PRISMA flow diagram summarizing the filtration process: 539 records were identified, 136 full-text papers were assessed, and 21 studies were finally included. The left column shows the records retained at each stage; the dashed right column indicates how many records were excluded at that stage and the rationale for the exclusion. The green inclusion box anchors the final twenty-one primary studies that feed the remainder of this survey. E. Quality Assessment

Studies were excluded if: •

E1: They were purely theoretical with no implementation or evaluation.

•

E2: They focused on test execution or test prioritization without generation.

•

E3: They used only formal specifications (e.g., Z, VDM) without natural-language input.

•

E4: They were short papers, posters, or tutorials lacking sufficient detail.

•

E5: They were duplicate publications of the same study (only the most complete version was retained).

•

E6: They were published in venues not indexed in IEEE Xplore, ACM Digital Library, Scopus, Web of Science, or DBLP, or that exhibit characteristics associated with predatory publishing.

D. Study Selection Process The selection process followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. It included four stages: •

Identification: The search returned 539 records before duplicates were removed.

•

Screening: Titles and abstracts were evaluated against the inclusion criteria. 403 records were excluded, leaving 136 potentially relevant papers.

•

Eligibility: The full texts of 136 papers were reviewed for eligibility. One hundred twenty papers were excluded

Figure 1. PRISMA study-selection flow leading to the twenty-one primary studies included in this survey.

•

Each included study was evaluated for quality using the following checklist adapted from [52]:

• •

QA1: Are the research objectives clearly stated? QA2: Is the proposed technique or tool described in sufficient detail? • QA3: Is the evaluation thorough (e.g., use of benchmarks, comparison with baselines, statistical analysis)? • QA4: Are the limitations and threats to validity discussed? We concluded the assessment by rating studies on a threepoint scale: yes = 1, partially = 0.5, and no = 0. The total quality score contextualized the findings without excluding any studies; all twenty-one confirmed primary studies met the minimum quality threshold of 2.0. F. Data Extraction We created a data-extraction form to gather the following information from each primary study: bibliographic details (authors, year, title, venue); research objectives; input requirements type (structured, unstructured, domain-

5

specific); NLP/AI techniques used; semantic representation (if any); output test artifacts (test steps, executable scripts, formal specifications); tooling context (prototype, opensource, commercial); evaluation metrics and results; application domain; and reported limitations and future work. We synthesized the extracted data through thematic analysis and categorized studies by technique era: rule-based, modelbased, ML, and LLM/hybrid. We also grouped them according to taxonomy dimensions developed through the analysis. Using these categories, we identified patterns, trends, and research gaps based on limitations reported in the primary studies. IV. TRADITIONAL NLP-BASED TEST GENERATION This section addresses RQ1 by examining traditional techniques for generating test cases from natural language requirements, including controlled natural languages (CNL), syntactic and semantic parsing, template and boilerplate approaches, and model-based methods grounded in UML and related formalisms. The discussion spans two historical phases: the Formalism Era (2000–2012) and the Statistical/NLP Era (2012–2022). Throughout this section, Figure 2 is used as a conceptual five-phase abstraction of requirements-to-test generation rather than a rigid workflow. Depending on the approach, phases may be emphasized unevenly, collapsed, specialized, revisited iteratively, or only partially realized. The Formalism Era relied primarily on deterministic NLP pipelines, controlled natural languages, and direct translation from requirements into formal specifications. In contrast, the Statistical/NLP Era introduced greater flexibility through template matching, pattern extraction, machine-learningbased classifiers, and structured intermediate models, such as UML and Object Constraint Language (OCL) mappings. Application of the strict inclusion criteria I4 (natural language input) and I5 (empirical validation for test case generation) yields no confirmed primary studies before 2012. This absence is methodologically meaningful rather than a limitation of the search. It reflects the dominant formal specification-first posture of the period, exemplified by approaches based on Z, SCR, VDM, B, and TLA+, which are excluded under I4 because they accept formal rather than natural language input. The encyclopedic model-based testing tradition of this era [32] and subsequent retrospectives on formalism in requirements engineering [39] confirm this orientation, while foundational software testing literature [31], [36] documents widespread reliance on deterministic processing, controlled natural languages, and specification-to-test-case translation pipelines. Requirements-based test suite work during this period operated almost exclusively over formal representations, as illustrated by Vaysburg et al.’s dependence analysis approach to requirements-based test suite reduction [41] and Cartaxo et al.’s UML sequence diagram plus labelled transition system method [45]. Earlier work in the Z with Isabelle tradition [50], as well as later extensions applying Zbased model checking to interlocking systems [47], further exemplify the era’s formalism-first stance. Importantly, this tradition did not disappear after 2012 but was absorbed into hybrid pipelines that accept controlled natural language input while retaining formal specification translation as an intermediate step. NAT2TESTSCR [3],

which uses SCR semantics, and NAT2TESTCPN [8], which targets Coloured Petri Nets, constitute primary study evidence of this bridge between controlled natural language and formal test case derivation. Broader coverage of pre-2012 work on test case generation from natural language lying outside the strict I4 + I5 scope of this survey is provided indirectly by related work mapping studies [23], [27], which reviewed this earlier literature under more inclusive criteria. A. Controlled Natural Languages (CNL) Controlled Natural Languages (CNLs) represent one of the earliest and most influential approaches to reducing linguistic variability in requirements-based test case generation. By constraining natural language input with restricted grammars and vocabularies, CNL-based methods enable deterministic parsing and formal downstream processing. As a unifying conceptual model for requirements-to-test-case generation from natural language specifications, this survey adopts the refined five-phase framework shown in Figure 2: requirement ingestion, linguistic analysis, structured representation, test-case synthesis, and executable tests. CNL-based frameworks illuminate this abstraction especially clearly because they narrow the ingestion space to controlled natural language forms, reduce variability in linguistic analysis, and support explicit formal representations for traceable test derivation and, in some cases, execution-ready test artifacts; however, not every implementation materializes these phases as fully separate stages.

Figure 2. Conceptual five-phase pipeline for natural-language-driven test-case generation, from requirement ingestion to executable tests.

Viewed through the refined five-phase framework, NAT2TESTSCR derives its effectiveness from making the major requirements-to-test transformations explicit: controlled requirement ingestion via SysReq-CNL statements; lightweight linguistic analysis over restricted sentence patterns; structured representation in SCR tables; test-case synthesis via formal analysis with T-VEC; and the eventual production of execution-oriented test vectors. Figure 2 clarifies these recurring transformations and helps explain why CNL approaches achieve strong traceability and determinism at the cost of broader input flexibility. Depending on the approach, however, some phases may be collapsed, specialized, or revisited iteratively rather than implemented as strictly separate sequential steps.

6

Despite these strengths, NAT2TESTSCR requires authors to be trained in writing requirements within the controlled syntax, which limits scalability and adoption in environments where informal natural-language requirements dominate. This trade-off between precision and usability is a defining characteristic of CNL-based approaches and recurs throughout requirements-based test-generation research. Silva et al. [8] extended this line of work with NAT2TESTCPN, published in the same journal in 2019, which translates the same CNL requirements into Coloured Petri Net (CPN) models and drives test case generation through simulation rather than constraint-based analysis. NAT2TESTCPN achieved approximately 80 percent mutation scores on the vending machine, nuclear power plant, and Embraer examples, and approximately 54 percent on the turn indicator example, with a peak mutation score of 85.07 percent on the Embraer case. In that example, NAT2TESTCPN matched NAT2TESTSCR’s performance, while exceeding it on the remaining three systems, demonstrating that alternative formal back-ends can be effectively substituted within a CNL-driven pipeline. Together, NAT2TESTSCR and NAT2TESTCPN provide strong primary-study evidence that controlled natural language approaches can enable highly effective, formally traceable test-case generation. At the same time, their reliance on constrained input languages underscores why subsequent research sought to relax linguistic restrictions through statistical NLP, machine learning, and hybrid approaches, as examined in the following subsections. B. Syntactic and Semantic Parsing Whereas CNL-based methods reduce ambiguity by constraining how requirements are written, a complementary line of research seeks to recover structure directly from unconstrained natural-language requirements through syntactic and semantic parsing. Rather than imposing controlled grammars, these approaches apply linguistic analysis to identify entities, actions, conditions, and relationships in free-form text, thereby expanding applicability while accepting greater interpretive uncertainty. Verma and Beg [2] developed an early method that combines part-of-speech (POS) tagging with parse-tree generation to transform requirement sentences into knowledge graphs representing entities and their relationships. Semantic frames were subsequently used to derive test inputs, conditions, and expected outputs from these graphs. Building on this foundation, later studies applied dependency parsing and semantic role labelling (SRL) to more reliably identify actoraction-object triples, thereby improving extraction accuracy and robustness [10], [12]. These parsing-based pipelines form the basis for many modern hybrid systems that combine linguistic analysis with downstream formal or generative test synthesis mechanisms.

as knowledge graphs, actor-action-object relations, and condition-bearing semantic structures. Their outputs typically support downstream test-case synthesis, but the extent to which synthesis and executable-test generation are integrated varies substantially across approaches. While parsing-based techniques offer greater flexibility and broader applicability than controlled natural languages, their effectiveness depends on the accuracy of linguistic analysis and on the handling of the ambiguity inherent in unconstrained natural language. As a result, these approaches motivated subsequent research into template-based, statistical NLP, and learning-based methods that further relax input assumptions while maintaining testability and traceability. C. Template and Boilerplate Approaches Between unconstrained parsing and fully controlled natural languages lies a more pragmatic design space: template and boilerplate approaches. Rather than enforcing a formal grammar, these methods guide requirements authors toward recurring syntactic patterns that reduce ambiguity while preserving some natural language flexibility. In this way, they trade part of the freedom of free-form text for greater regularity in downstream extraction and test case derivation. Lim et al. [12] proposed a unified boilerplate approach that integrates Rupp’s boilerplates with the Easy Approach to Requirements Syntax (EARS) to support test case information extraction from both positive and negative requirements. Their NLP-based system extracted actors, conditions, and responses from the PURE datasets [54], achieving overall correctness rates of 61.7 percent on Pointis, 50.0 percent on Mdot, and 10.3 percent on Npac. The authors report substantially lower correctness for negative requirements than for positive ones, highlighting a key limitation of templateguided extraction as linguistic complexity increases. While such approaches reduce ambiguity and improve consistency, they also constrain expressive flexibility and require users to learn and adhere to specific authoring conventions. The industrial applicability of template-oriented workflows is demonstrated by SPECMATE [9], which automatically generates acceptance tests from structured acceptance criteria. When evaluated on 961 user stories in collaboration with Allianz Deutschland, SPECMATE automatically generated 56 percent of the 604 manually created test cases. Building on this line of work, Fischbach et al. [11] introduced CiRA (Conditionals in Requirements Artifacts). This RoBERTabased NLP approach extracts conditional structures from informal natural language requirements and maps them to acceptance-test conditions via cause-and-effect graphs. In an industrial study involving Allianz Deutschland, Ericsson, and Leopold Kostal, CiRA automatically generated 71.8 percent of 578 manually created test cases and discovered 80 additional relevant cases that the manual testing process had missed.

Within the refined five-phase abstraction in Figure 2, these methods contribute most directly to the linguistic analysis phase through operations such as tokenization, parsing, semantic role labelling, named-entity recognition, and condition extraction, and to the structured representation phase through the construction of intermediate artifacts such

7

Although template, boilerplate, and parsing-based approaches differ in how they recover structure from requirements, they converge on a common operational logic: linguistic elements in the requirement text are mapped to corresponding fields in the generated test case. Figure 3 visualizes this shared correspondence by showing how actors, actions, conditions, and expected system responses align with test-case fields such as actor role, test steps, preconditions, and expected results. For the example requirement “the system shall display an error message if a user enters an invalid phone number,” the actor “user” maps to the Actor field, the action “enters an invalid phone number” maps to the Action/TestStep field, the conditional clause maps to the Precondition/Trigger field, and the expected response “display an error message” maps to the Expected Result field. This mapping captures the common intermediate logic that systems such as SPECMATE and CiRA make explicit, and that rule-based extraction approaches such as Medeshetty et al. [17] exploit in a more implicit form.

Figure 3. Conceptual correspondence between requirement-level linguistic elements and their mapped test-case fields across templatebased, boilerplate-based, and parsing-based approaches.

Viewed through the refined five-phase abstraction in Figure 2, these approaches are best understood as concentrating on the boundary among requirement ingestion, linguistic analysis, and structured representation, where semi-patterned requirements are normalized, their conditions and responses are extracted, and the resulting intermediate forms support more reliable downstream test-case synthesis. In most cases, the final executable tests phase still depends on additional tooling or project-specific realization. D. Model-Based And UML Approaches Where template, boilerplate, and parsing-based methods infer test case structure by mapping linguistic elements to corresponding test fields, model-based and UML-oriented approaches make that intermediate representation explicit by transforming natural language requirements into behavioral or structural models. Model-Based Testing (MBT), as defined by El-Far and Whittaker [32] in the Wiley Encyclopedia of Software Engineering, is the derivation of test cases from behavioral or structural models of the system under test. In requirements-based settings, these methods typically translate textual requirements into formal or semi-formal models, most Unified Modeling Language (UML) artifacts, before applying systematic test-generation techniques such as traversal, simulation, or constraint solving. Wang et al. [10] proposed the Use Case Modelling for System Test Generation (UMTG) framework, which applies natural

language processing to restricted use case specifications and translates them into UML models augmented with Object Constraint Language (OCL) constraints. Test cases are then generated by constraint-solving over the resulting formal representations. Evaluated in two industrial automotive case studies, UMTG achieved 95 percent correct translation of use case steps into OCL constraints and generated test cases that covered all expert-designed scenarios, as well as additional critical scenarios not previously considered. These results demonstrate the effectiveness of combining NLP-based extraction with formal modelling to achieve both automation and traceability. Other model-based approaches similarly integrate linguistic preprocessing with behavioral modelling. Kamonsantiroj et al. [6] introduced a memorization-based method for generating test cases from concurrent UML activity diagrams, while Allala et al. [7] combined modeldriven engineering (MDE) with NLP to transform natural language user requirements into test cases via model-to-model transformations. Keyword-based NLP techniques have also been applied to functional software requirements specifications to support UML-oriented test case generation [5]. In an industrial aerospace context, Olajubu et al. [4] translated natural language “shall” statements into a domain-specific modeling language, from which executable test cases were generated. Within the refined five-phase abstraction in Figure 2, modelbased and UML-driven approaches operate primarily across the structured representation, test-case synthesis, and executable tests portions of the framework. They use models such as UML, OCL, SCR, domain-specific languages, or other behavioral formalisms as the explicit bridge between naturallanguage requirements and systematic test derivation, while solvers, simulators, traversal engines, or model checkers support the generation and realization of concrete test artifacts. Model-based approaches provide strong guarantees of coverage, determinism, and traceability, making them particularly attractive in safety-critical and regulated domains. However, they impose significant modelling overhead and often depend on restricted requirement formats or domainspecific assumptions. These constraints limit scalability and help explain why subsequent research has increasingly explored statistical NLP, machine learning, and largelanguage-model-based techniques that seek to reduce manual modelling effort while preserving testability. Taken together, the approaches examined in Section IV demonstrate how deterministic NLP pipelines, controlled languages, templates, and model-based representations can recover structure from natural language requirements and support test case generation. However, these techniques typically rely on constrained input formats, predefined linguistic patterns, or explicit modelling effort, which limits their robustness and scalability when dealing with highly variable, informal, or evolving requirements. These limitations motivated a shift toward learning-based approaches that reduce manual authoring effort and operate directly on unrestricted natural language. The next section examines how machine learning and large language modelbased techniques address these challenges by leveraging datadriven semantic understanding, while introducing new tradeoffs in terms of traceability, determinism, and trustworthiness.

8

V. AI-DRIVEN AND LLM-BASED TECHNIQUES Recent advances in artificial intelligence and naturallanguage processing have fundamentally redefined requirements-based automated test generation, shifting the field from rule and feature-engineered pipelines toward datadriven and generative architectures. This section addresses RQ1 and RQ2 by examining AI and NLP techniques proposed for generating test cases from natural language requirements, as well as the tools and frameworks that operationalize these techniques in practice. As in Section IV, Figure 2 is used here as a conceptual five-phase abstraction rather than a rigid workflow; depending on the architecture, phases may be fused within a single model invocation, externalized into retrieval or verification components, revisited iteratively, or only partially implemented. Against that framing, the discussion traces the evolution from pre-LLM machine-learning approaches to transformer-based and large-language-model (LLM) systems, highlighting improvements in coverage, precision, and adaptability when operating on complex, ambiguous requirement texts. By linking techniques to concrete implementations, this section distils recurring strengths, limitations, and emerging architectural patterns. It begins with pre-LLM machinelearning systems, then turns to transformer and LLM-based approaches, before considering domain-specific applications and hybrid symbolic-neural designs that seek to recover some of the control and auditability lost in fully generative settings. Building on this synthesis, Section VI systematically compares the identified tools and frameworks using uniform evaluation criteria, exposing unresolved tradeoffs and motivating the research gaps discussed later in the paper. A. Pre-LLM Machine Learning Pipelines Between 2015 and 2020, AI-based test case generation from natural language requirements was dominated by traditional machine learning approaches that relied on supervised learning to map requirement sentences to structured test case components. Representative methods employed classifiers such as Support Vector Machines (SVMs) and Bidirectional Long Short-Term Memory networks (Bi-LSTMs) to perform sentence or fragment-level classification of requirement text into test artifacts [24], [26]. Compared with rule-based NLP systems, these approaches achieved greater flexibility and accuracy but relied heavily on labelled training data and exhibited limited cross-project generalization. In Figure 2, these systems typically distribute effort across linguistic analysis, structured representation, and test-case synthesis, often via separate learned or feature-engineered components rather than a single end-to-end generator. Typical implementations combined linguistic feature extraction from requirements using part-of-speech tags and dependency relations, supervised classification of requirement fragments into test case elements such as inputs, actions, and expected outcomes, and sequence-oriented generation of structured test templates. While effective within narrowly defined training contexts, reliance on handcrafted features, annotated datasets, and project-specific models constrained scalability and hindered adoption across heterogeneous domains and evolving requirements. These limitations exposed a structural ceiling for featureengineered machine-learning systems. They created the

conditions for transformer-based architectures, which absorbed more of the interpretation and generation burden into shared representations. The next subsection examines how that shift changed both the achievable level of automation and the nature of the associated risks. B. Transformer and LLM Paradigm The 2020s marked the rise of transformer-based large language models (LLMs) capable of few-shot reasoning, significantly expanding the scope of requirements-based automated test generation. Transformer architectures have enabled models to capture long-range dependencies and contextual semantics in free-form requirement texts, allowing test cases to be generated directly from natural language inputs without the feature engineering and supervised classifiers required by earlier approaches. Within the conceptual five-phase framework in Figure 2, transformer-based and LLM approaches often compress much of the linguistic analysis, portions of structured representation, and parts of test-case synthesis into a single generative inference process. At the same time, retrieval, prompting, verification, and test realization are frequently externalized into surrounding components, so these systems are better understood as reconfiguring the framework than as eliminating it. Korraprolu et al. [18] evaluated six publicly available LLMs BARD, ChatGPT-3.5, Claude, Gemini, GPT-4 Omni, and LLaMA 3 for generating test cases from English natural language requirements. Although all six models could produce executable tests, their performance varied considerably across coverage criteria, revealing differences in how effectively each model captured decision logic and conditional structure rather than surface-level syntactic completeness. Figure 4 summarizes Korraprolu et al.’s benchmark across twenty-five requirements using four metrics: decision coverage, condition coverage, Modified Condition/Decision Coverage (MCDC), and execution rate. Gemini and ChatGPT-3.5 achieved the highest average decision and condition coverage, with Gemini reaching 85.2 percent and 82.9 percent, respectively, and ChatGPT-3.5 reaching 83.2 percent and 80.5 percent. Performance declined across all models on MCDC, where Gemini and Claude achieved the highest averages at 50.2 percent and 50.1 percent, respectively. By contrast, execution remained consistently high across all six models, with a median of 100 percent and an average above 96 percent. The central result is the large gap between execution and MCDC performance: the models generally produced runnable tests, but they were substantially less effective at covering interacting logical conditions. This divergence provides quantitative evidence that current general-purpose LLMs remain sensitive to the complexity of requirements, directly motivating Gap G3. This systematic degradation in MCDC performance as conditional interactions increase provides early empirical evidence that LLM-based test generation remains sensitive to requirement complexity, a limitation examined further in Gap G3. To address these limitations in general‑purpose models, Xue et al. [13] introduced LLM4Fin, a domain‑specific LLM fine‑tuned on corpora from financial regulation. By incorporating both explicit and implicit domain knowledge,

9

two-shot, and prompt-chaining, the authors reported 79.6 percent coverage for the base model (Table II) and up to 90.3 percent coverage for the optimal prompt-chaining plus twoshot sequential configuration (Table III). These results show that high-coverage LLM-based test generation can be achieved under enterprise data-privacy constraints.

Figure 4. Average coverage of six general-purpose LLMs across decision, condition, MCDC, and execution criteria on twenty-five natural language requirements, based on the average values reported by Korraprolu et al. [18]. The six-model averages are 78.9 percent for decision coverage, 77.8 percent for condition coverage, 43.9 percent for MCDC, and 97.0 percent for execution. The marked execution-versusMCDC gap indicates that these models usually generate runnable tests but remain weaker at exercising compound logical conditions.

LLM4Fin achieved a peak business‑scenario coverage (BSC) of 98.18 percent on Dataset 3, with an average BSC of 91.89 percent across five datasets. In addition to improved coverage, LLM4Fin dramatically reduced test‑generation time from approximately 20 minutes to an average of 6.99 seconds per generation. In contrast, the general‑purpose ChatGPT model achieved only 49.65 percent average coverage and required approximately 19 minutes per generation. Figure 5 compares LLM4Fin with four baselines - human experts, human nonexperts, general-purpose ChatGPT, and ChatGLM - using average business-scenario coverage (BSC) and generation time. The figure highlights two results. First, LLM4Fin substantially improves coverage over generalpurpose ChatGPT, achieving an average BSC of 91.89 percent versus 49.65 percent. Second, it reduces average generation time from approximately 19 minutes to 6.99 seconds per case. Taken together, these results show that domain specialization can improve both effectiveness and efficiency in LLM-based test case generation.

Further optimization-oriented studies extend the applicability of LLMs to test generation. Alagarsamy et al. [20] introduced TestGPT, a GPT 3.5 model fine-tuned on a curated text-totestcase dataset, achieving 78.5 percent syntactic correctness, 67.09 percent requirement alignment, 61.7 percent code coverage, and an 18.9 percent mutation score across five large open-source projects, substantially outperforming thirty-one other variants across eight baseline LLMs (basic GPT 3.5turbo, Incoder, Starcoder, CodeT5, Bloom, CodeGemma, CodeLlama, and Gemini). Similarly, Takerngsaksiri et al. [21] proposed PyTester, which applies deep reinforcement learning to text-to-testcase generation and outperforms GPT3.5 on the APPS benchmark. Arora et al. [15] addressed trustworthiness through RAGTAG, an industrial retrieval-augmented generation framework evaluated at Austrian Post on two projects with bilingual German and English requirements. Generated test scenarios were assessed by four domain experts along five dimensions: relevance, coverage, correctness, coherence, and feasibility. Under the best-performing configuration, GPT-4.0 with fewshot prompting and single-chunk retrieval, the automatic scores were BLEU = 0.092, ROUGE = 0.419, and METEOR = 0.516. The low BLEU score is not surprising in this setting because test scenarios can admit multiple acceptable phrasings; accordingly, expert judgment provides the more informative evidence. The expert assessment found that RAGTAG generated scenarios aligned with the source requirements and feasible for practical testing. At the same time, the study indicates that current generative pipelines still depend on retrieval anchoring and human review, underscoring the absence of standardized hallucinationdetection methods and formal traceability artifacts, thereby reinforcing Gaps G1 and G2. Finally, Najmi and El-Dosuky [19] integrated ChatGPT with traditional NLP and deep learning components in a multistage framework for test case analysis, applying the approach to an optical character recognition system. Their study usefully bridges the purely pre-LLM and purely generative ends of the spectrum, thereby motivating the closer look at domain-shaped and hybrid architectures in the subsections that follow. C. Domain-Specific Applications

Figure 5. Average business-scenario coverage and generation time for LLM4Fin and four baselines, based on the average values reported by Xue et al. [13]. LLM4Fin achieves 91.89 percent average coverage and 6.99 seconds per case, compared with 49.65 percent and approximately 19 minutes for general-purpose ChatGPT.

If general-purpose LLM studies reveal the promise and instability of broad generative approaches, domain-specific applications show that performance can improve when requirements, domain knowledge, and evaluation settings are more tightly controlled. The studies in this subsection illustrate how architectural specialization can recover accuracy, speed, or robustness without reverting fully to the rigid constraints of earlier formal methods.

Building on this line of work, Yin et al. [14] demonstrated the on-premises deployment of a 4-bit quantized LLaMA-3 70B Instruct model for quality-assurance teams, emphasizing secure test case generation without data leakage. Evaluating multiple prompting strategies, including zero-shot, one-shot,

Medeshetty et al. [17] investigated domain-specific NLP for transforming natural-language automotive Electronic Control Unit (ECU) requirements into structured test-case specifications. Their study, conducted with Scania CV AB, combined rule-based information extraction with namedentity recognition (NER) and used 400 feature-element

10

documents from Polarion, including a manually annotated subset of 200 entities for NER training and 10-fold crossvalidation. Table I summarizes the comparative performance of the rulebased and NER-based extraction methods across increasing levels of requirement complexity. TABLE I. REPORTED ACCURACY OF DOMAIN-SPECIFIC NLP EXTRACTION TECHNIQUES [17]

Method

Variant

Key Metric

Notes

Rule-based IE

Category 1 (single signal)

Accuracy = 95%

Baseline

Rule-based IE

Category 2 (≤ 4 signals)

Accuracy = 75%

p = 0.037 vs manual (significant)

Rule-based IE

Category 3 (> 4 signals)

Accuracy = 60%

Degrades with complexity

NER (SVM)

All categories

NER Acc = 77.3%; Prec/Rec/F1 = 0.816/0.773/0.770

p = 0.806 vs manual (not significant)

NER (Random Forest)

All categories

NER Acc = 40.8%; Prec/Rec/F1 = 0.263/0.408/0.316

Substantially weaker than SVM

a)

Statistical findings: Rule-Based vs. Manual (Z-Test): p = 0.037 - statistically significant improvement over manual methods. NER vs. Manual: p = 0.806 - improvement not statistically significant. The rule-based method reduces test case generation time from ~24 hours to 5 minutes per case, but accuracy reduces with increased requirement complexity.

The rule-based extractor was strongly affected by requirement complexity, with accuracy declining from 95 percent for single-signal requirements to 75 percent for Category 2 and 60 percent for Category 3. Even so, it provided a substantial practical advantage over manual test-case generation, reducing effort from approximately 24 hours to 5 minutes per case; this improvement was statistically significant (MannWhitney U, p = 0.037). Among the NER variants, the SVM model performed best, reaching 77.3 percent accuracy with precision, recall, and F1 scores of 0.816, 0.773, and 0.770, respectively, whereas the Random Forest variant achieved only 40.8 percent accuracy. Despite this competitive extraction accuracy, the NER approach did not yield a statistically significant improvement over manual generation (p = 0.806). Taken together, these results indicate that deterministic rulebased extraction remains the stronger choice for wellstructured automotive ECU requirements. In contrast, NER performance depends heavily on model selection and does not yet translate into a measurable end-to-end advantage over manual practice. This contrast motivates continued interest in hybrid rule-based and learning-based designs for more complex requirements. A related pattern appears in another safety-critical domain. Shakthi et al. [16] combined NLP techniques, including NER, part-of-speech tagging, and dependency parsing, with fewshot and chain-of-thought prompting to generate test cases from ten satellite Functional Requirement Documents (FRD) at the U. R. Rao Satellite Center of the Indian Space Research Organization. Four LLMs were evaluated on the same task: GPT-4 generated 9 of 10 correct test cases (90 percent), Gemini-Pro generated 6 of 10 (60 percent), Llama-2-70B

generated 4 of 10 (40 percent), and Mistral-7B generated 2 of 10 (20 percent). This stratified result provides clear quantitative evidence of model-dependent variation in NL-totest generation accuracy under a common evaluation setting. D. Hybrid Symbolic-Neural Architectures Where domain-specific applications primarily narrow the problem space, a complementary line of work seeks to rebalance the architecture itself through hybrid symbolicneural designs that explicitly integrate formal reasoning with neural language understanding. By coupling the semantic generalization capabilities of neural models with the precision and auditability of symbolic constraint systems, these architectures aim to improve both the correctness and traceability of generated test cases. They address a key failure mode of purely generative LLM approaches: the hallucination of requirement-inconsistent test steps. Across the twenty-one primary studies surveyed, UMTG (Wang et al. [10]) is the only system to fully implement an integrated hybrid symbolic-neural architecture, with reported performance metrics in the NL-to-test domain. UMTG applies NLP-based semantic role labelling to extract control flow from use-case specifications, automatically generates Object Constraint Language (OCL) formal constraints from natural language sentences, and then applies constraint solving to produce executable test data. In two industrial automotive case studies, UMTG correctly translated 95 percent of use-case specification steps into formal OCL constraints and generated test cases that covered all expert-designed scenarios, as well as additional critical scenarios not previously considered [10]. This architectural pattern - neural NLP feeding a symbolic formal layer that drives test-data generation - represents the most tractable and empirically verified realization of hybrid symbolic-neural test-case generation reported in the literature. Bougzime et al. [34] provide a broader architectural survey of neuro-symbolic AI paradigms, including retrieval-augmented generation, graph neural networks, and reinforcement learning, and discuss how symbolic reasoning can reduce hallucinations and improve determinism in generative AI systems. However, this work remains theoretical and architectural in nature: it does not implement a test casegeneration system and reports no empirical performance metrics for that task. Other studies in this corpus that combine neural and rule-based elements do so primarily at the level of NLP pipelines; for example, Medeshetty et al. [17] compare rule-based information extraction with named-entity recognition rather than implementing full symbolic-neural integration. Despite growing recognition of the importance of hybrid architectures, as evidenced by their identification as a key research gap in multiple surveys [22], [23], this area remains one of the least explored in the field. At present, UMTG is the only system providing rigorous empirical evidence that hybrid symbolic-neural architectures can successfully balance automation, correctness, and formal traceability in natural language-driven test case generation. VI. COMPARATIVE ANALYSIS OF TOOLS AND FRAMEWORKS This section synthesizes the findings of Sections IV and V to answer RQ2 and RQ3 by comparing the tools and frameworks proposed for AI-based test case generation from natural language requirements. Building on the detailed discussion of techniques and architectures, the analysis

11

contrasts representative approaches across eras - traditional NLP pipelines, machine learning systems, and LLM-based and hybrid solutions using a uniform set of evaluation dimensions. Rather than re-describing individual methods, this section focuses on cross-cutting distinctions in automation level, ambiguity handling, domain applicability, traceability, evaluation rigor, and trustworthiness, as summarized in the comparative tables that follow. This structured comparison exposes recurring trade-offs and limitations that motivate the research gaps identified in Section VII. Over the past decades, the surveyed approaches have evolved through three distinct eras. The first, the Formalism Era, emphasized rule-based methods and formal specification techniques. This phase was followed by the Statistical/NLP Era, marked by a shift toward template-driven approaches and natural language processing pipelines. Most recently, the Hybrid/LLM Era has emerged, integrating symbolic reasoning, neural networks, and large language models. Figure 6 shows the chronological distribution of the twentyone confirmed primary studies across the three eras used in this survey. The empty Formalism Era band (2000–2012) reflects the effect of the strict I4 and I5 inclusion criteria rather than a lack of prior work: formal-specification approaches dominated that period but fall outside the present corpus because they do not jointly satisfy natural-language input and empirical test-generation validation. By contrast, the Statistical/NLP Era (2012–2022) and the Hybrid/LLM Era (2022–2025) contain the full set of confirmed primary studies, with the latter accounting for the most recent contributions.

Figure 6. Chronological distribution of the twenty-one confirmed primary studies across the three eras (formalism, statistical/NLP, and Hybrid/LLM) used in this survey. The primary studies span the period 2012–2025.

The absence of data points in the Formalism Era reflects the scope of this survey: the I4 and I5 criteria exclude the formalspecification studies that dominated that period. However, their historical role is discussed in the background references [31], [32], [39], [41], [45], [47], [50], as well as the mapping studies [23], [27]. To ground this era-level evolution in concrete empirical evidence, Table II enumerates all 21 verified primary studies in chronological order. For each study, the table records the associated system or publication venue, the core NLP or AI technique employed, the type of input requirements, the generated test artifact and application domain, and the reported quantitative performance metrics. The rightmost column lists the reference number that anchors each study to the corresponding entry in the reference list.

TABLE II. TWENTY-ONE VERIFIED PRIMARY STUDIES - QUANTITATIVE DATA System / Venue

Technique

Input Type

Output / Domain

Verified Performance

R ef

Litmus (NLDB 2012)

5-step NLP pipeline

NL function al requirem ents

Test cases / General

Testability → simplification → intent → synthesis → BVA. Qualitative.

1

Verma & Beg (IEEE ICETET 2013)

POS + parse tree

NL requirem ents

Test cases / General

POS + parse tree → knowledge graph → test inputs.

2

NAT2TES TSCR (SCP 2014)

CNL → SCR → TVEC

SysReqCNL requirem ents

Test vectors / Aerospa ce + Auto

Mutation 37.48% (TIS) → 95.52% (PC). 100% precision Embraer.

3

DSL from NL (ACM RACS 2015)

Domainspecific modelling

NL shall statemen ts

Test cases / Aerospa ce (GE)

GE Aviation NL → DSL → tests via M2T.

4

NLP Keywords (IEEE AEEICB 2017)

Keywordbased NLP

NL function al SRS

Test cases / General

Keyword structure extraction from SRS.

5

UML Memorizati on (ACM ICGDA 2019)

Memorizati on + UML activity diagrams

UML + NL annotatio ns

Test cases / Concurr ent systems

Memorization approach for UML activity diagram test generation.

6

MDE + NLP (IEEE COMPSAC 2019)

MDE + Stanford CoreNLP

NL user requirem ents

Test cases / General

NL reqs → metamodel → test cases via M2M transformatio n.

7

NAT2TES TCPN (SCP 2019)

CNL → Coloured Petri Net

SysReqCNL requirem ents

Test cases / Aerospa ce + Auto

~80% VM/NPP/PC, ~54% TIS, 85.07% peak PC.

8

SPECMAT E (IEEE ICST 2020)

NLP pattern + CEG

NL acceptan ce criteria

Test cases / Industri al (Allianz )

56% of 604 TCs. Missing negative TCs added. 961 user stories.

9

UMTG (IEEE TSE 2022)

NLP SRL → OCL + solver

NL usecase (RUCM)

Tests + traceabi lity / Automo tive

95% of usecase steps → OCL constraints. All expert + critical scenarios.

10

CiRA (JSS 2023)

RoBERTa conditional

Informal NL requirem ents

Accepta nce tests / Industri al

71.8% of 578 TCs. 80 missed TCs found. 3 partners.

11

Unified Boilerplate (JSS 2024)

Rupp + EARS boilerplate

NL requirem

Test templat

Pointis: 61.7%, Mdot:

12

12

System / Venue

Technique

Input Type

Output / Domain

Verified Performance

ents (PURE)

es / General

50.0%, Npac: 10.3%.

R ef

LLM4Fin (ICSE Companion 2024)

Domaintuned LLM + KB

NL financial rules

Accepta nce tests / FinTech

Avg BSC 91.89%; peak 98.18% (D3). ChatGPT avg 49.65%. 20 min → 7 sec.

13

On-Prem LLM (IEEE ICIIBMS 2024)

LLaMA-3 70B + prompt chain

NL user stories

Secure tests / Enterpri se

79.6% Llama3-70B (Tbl. II); 90.3% PromptChain+2-shot (Tbl. III).

14

RAGTAG (IEEE RE 2024)

RAG + LLMs (GPT3.5/4.0)

NL requirem ents (bilingua l)

Test scenario s/ Industri al (Austria n Post)

Expert: relevant/feasi ble. BLEU=0.092, ROUGE=0.41 9, METEOR=0. 516.

15

Satellite FRD (IEEE ICECCME 2024)

NLP (NER+POS +DP) + FewShot+CoT LLM

Satellite FRD (NL)

Test cases / Space (ISRO)

GPT-4=90%. GeminiPro=60%. Llama-270B=40%. Mistral7B=20%.

16

ECU NLP (IEEE ICHMS 2025)

Rule-based IE + NER SVM

NL ECU feature elements

Test specs / Automo tive

95%/75%/60 % by complexity. SVM 77.3% (p=0.037). 24 hr → 5 min.

17

LLM Benchmark (ISEC 2025)

6-LLM comparison

25 NL requirem ents

Coverag e metrics / Multidomain

Gemini decision 85.2%, cond. 82.9%. All LLMs MCDC ≤ 50.2%.

18

ChatGPT Framework (JCS 2025)

ChatGPT + NLP + DL

NL requirem ents

Test cases / OCR

4-stage ChatGPT+NL P+DL pipeline. Vol 21(5) 2025.

19

TestGPT (JSS 2025)

Fine-tuned GPT-3.5

NL requirem ents

Unit tests / General

78.5% syntactic, 67.09% alignment, 61.7% coverage, 18.9% mutation.

20

PyTester (JSS 2025)

Deep reinforceme nt learning

Text descripti ons

Test cases / General

RL text-totestcase. outperforms GPT-3.5 on APPS benchmark.

21

b)

Analysis of Tools & Frameworks - Era Evolution: Table II presents the 21 verified primary studies in chronological order, showing for each study the corresponding system or publication venue, the principal NLP or AI technique, the input requirements type, the generated test artifact and application domain, and the reported quantitative performance metrics.

A. Analysis of Table II - What the Twenty-One Primary Studies Collectively Show Three patterns emerge from the chronological quantitative data in Table II. First, the field’s center of gravity has decisively shifted into the Hybrid/LLM Era. Of the twenty-one primary studies, only six pre-date 2020; the remaining fifteen appear between 2020 and 2025, and nine of those fifteen are LLM-based or LLM-hybrid approaches (Xue et al.'s LLM4Fin [13], Yin et al.'s on-premises LLaMA3 70B [14], Arora et al.'s RAGTAG [15], Shakthi et al.'s satellite-FRD study [16], Korraprolu et al.'s six-LLM benchmark [18], Najmi and El-Dosuky's ChatGPT framework [19], Alagarsamy et al.'s TestGPT [20], Takerngsaksiri et al.'s PyTester [21], and Fischbach et al.'s CiRA [11]). The absolute peak numbers reported 98.18 percent peak Business-Scenario Coverage by LLM4Fin on Dataset 3, 95 percent OCL-translation accuracy by UMTG [10], 95.52 percent peak mutation score by NAT2TESTSCR [3] on the Embraer example, and 90.3 percent coverage by the on-premises LLaMA-3 70B system. They all cluster above 90 percent, suggesting that the ceiling for what NLdriven TCG can achieve lies not in the technique itself but in the measurement regime and the domain. Second, peak numbers and averages diverge sharply in LLM studies and converge in formal, controlled-NL studies. LLM4Fin's 98.18 percent peak versus 91.89 percent average (Dataset 3 vs. five-dataset average, [13]) and the GPT-4 90 percent versus Mistral-7B 20 percent spread on identical satellite FRDs [16] are representative: LLM outputs are more variable than formal CNL outputs, and averaging across datasets or models erodes the peak. The deterministic-CNL approaches, NAT2TESTSCR [3] and NAT2TESTCPN [8], show the opposite pattern: their mutation scores cluster tightly around 80 percent across four case studies, with a single outlier at 95.52 percent. This divergence is the empirical signature of the trade-off between traceability and expressiveness, as analyzed quantitatively in section VI-B. Third, industrial and safety-critical domains dominate the 2020-2025 cohort: Allianz Deutschland (SPECMATE [9], CiRA [11]), Austrian Post (RAGTAG [15]), Ericsson and Leopold Kostal (CiRA [11]), automotive ECUs (Medeshetty et al. [17], UMTG [10]), FinTech compliance (LLM4Fin [13]), and ISRO satellite FRDs (Shakthi et al. [16]). This concentration of industrial case studies is a strength of the corpus: validation is no longer limited to academic toy examples. But it also exposes the absence of published reproducibility packages. Most industrial studies report aggregate metrics without releasing the underlying requirements or test case data, which is a structural obstacle to the complexity-stratified benchmark that R3 proposes in section VIII-C. B. Related Work Summary While the preceding analysis focuses on the twenty-one confirmed primary studies, it is also necessary to situate these contributions within the broader research landscape. This subsection, therefore, summarizes representative related and background work that informs, contextualizes, or bounds the scope of AI-based test case generation from natural-language requirements.

13

Table III cross-references the twelve most representative related works, including selected primary studies already tabulated in Table II, as well as adjacent surveys and background references, using a seven-column schema that captures approach, input, output, evaluation design, core assumption, principal limitation, and inclusion status. The table is original to this survey (Contribution C2) and serves as the mechanism for comparing primary studies, related work, and background sources using uniform criteria. TABLE III. RELATED WORK SUMMARY - TWELVE REPRESENTATIVE STUDIES (CONTRIBUTION C2) Ref

Appro ach

Input

Outp ut

Evalu ation

Key Assumpti on

Key Limit ation

Status

[10]

UMTG: NLP SRL → OCL

NL usecase specs

Tests + tracea bility

2 indust rial studie s

RUCM template required

Doma in model requir ed

PRIM ARY

[3]

NAT2T ESTSC R: CNL → SCR

CNL require ments

Test vector s

4 case studie s; mutati on

Engineers write CNL

CNL trainin g neede d

PRIM ARY

[11]

CiRA: RoBER Ta conditi onal

Inform al NL require ments

Accep tance tests

Indust rial: 3 partne rs

Conditiona l structure present

Condi tionalcentri c only

PRIM ARY

[9]

SPEC MATE: NLP + CEG

NL accept ance criteria

Test cases

604 TCs; 56%

User stories with ACs

ACtype cover age

PRIM ARY

[15]

RAGT AG: RAG + LLMs

NL (biling ual)

Test scenar ios

4 expert s; 5 dimen sions

Domain context retrievable

Actio nseque nce accura cy

PRIM ARY

[13]

LLM4F in: domain LLM

NL financi al rules

Accep tance tests

BSC on 5 datase ts

Domain knowledge pre-coded

FinTe ch only

PRIM ARY

[14]

LLaM A-3 70B onprem

NL user stories

Secur e test cases

Cover age up to 90.3 %

GPU VRAM required

Huma n eval only

PRIM ARY

[22]

Yang RBTG Survey

N/A (surve y)

Taxo nomy

SLR; 267 paper s

Wide scope

LLM halluc inatio n unadd resse d

RELA TED WOR K

Ref

Appro ach

Input

Outp ut

Evalu ation

Key Assumpti on

Key Limit ation

Status

[49]

Cajica mutati on TCG

Forma l specs

Test cases

Cover age metri cs

Formal spec available

Fails I4 + I5

BAC KGR OUN D

[50]

Helke Z+ Isabell e

Z formal specifi cation

Test cases

Small exper iment

Z spec available

Fails I4 + I5

BAC KGR OUN D

c)

Related Work Summary: Table III compares twelve key related works using a seven-column framework covering methodology, inputs, outputs, evaluation, assumptions, limitations, and inclusion status. Created as Contribution C2 of this survey, it provides a uniform basis for comparing primary studies, related surveys, and background references.

The twelve rows in Table III span three inclusion categories: PRIMARY (seven entries), RELATED WORK (three entries), and BACKGROUND (two entries). Read vertically, the table exposes two structural observations that cut across these categories. First, the “Key Limitation” column converges on a small number of recurring weaknesses. Formal‑specification‑only background work consistently fails the natural language and empirical test generation criteria (Fails I4 + I5), as exemplified by Cajica [49] and Helke [50]. Older surveys, such as Boukhlif [24] and Garousi [27], are characterized by a pre‑LLM scope or by coverage broader than test case generation alone, limiting their ability to address LLM‑specific challenges. In contrast, LLM‑era primary studies exhibit domain‑restriction limitations, including FinTech‑only applicability in LLM4Fin [13] and enterprise‑constrained deployment assumptions in the on‑premises LLaMA‑3 system [14]. This clustering of limitations constitutes cross‑cutting evidence motivating R3’s call for a multi‑domain, complexity‑stratified benchmark. Second, the “Key Assumption” column reveals the implicit contract each approach requires users to accept to make requirements‑to‑test‑case generation tractable. These assumptions include Controlled Natural Language authoring in [3], RUCM templating in [10], explicit conditional structure in [11], retrievable domain context in [15], pre‑coded domain knowledge in [13], and GPU VRAM availability in [14]. Collectively, these assumptions demonstrate that the primary studies have internalized a fundamental lesson from the background formal‑methods

literature: unrestricted natural language is too expressive to map cleanly to executable tests without additional constraints. Each approach, therefore, imposes explicit structural, domain, or infrastructural assumptions to regain tractability.

[24]

Boukhl if NLPtesting SLR

N/A (surve y)

Classi ficati on

PRIS MA; 24 paper s

NLP testing feasible

PreLLM scope

RELA TED WOR K

[27]

Garous i NLPtesting mappin g

N/A (mapp ing)

57paper map

IST Q1, 2020

NLP applied to testing

Broad er than TCGonly

RELA TED WOR K

The positioning of this survey between related‑work surveys and confirmed primary studies is therefore not incidental. The primary studies inherit formal‑method rigor from the background layer while simultaneously extending the broad natural language applicability characterized in LLM‑era surveys. Table III makes this inheritance explicit and provides the interpretive bridge between foundational work and the empirical, LLM‑driven systems evaluated in this paper.

14

C. Six-Criteria Comparison and Gap Analysis The comparative patterns and recurring limitations identified across the primary studies and related work motivate the need for a uniform evaluative framework. To make these trade-offs explicit and systematically comparable across techniques and eras, we define a set of six evaluation criteria that capture the core dimensions along which existing approaches differ. The six criteria are defined to be consistently applicable across the majority of primary studies and directly tied to the research questions: C1 automation level (manual | semi auto | full); C2 ambiguity handling (Low | Medium | High); C3 domain applicability (Narrow | Medium | Broad); C4 traceability (Low | Medium | High); C5 evaluation thoroughness (Low | Medium | High); and C6 hallucination control (Low | Medium | High). Criteria that did not apply across ≥ 80 percent of studies consistently were removed or merged. Table IV applies the six criteria uniformly to eight representative primary studies, ranging from deterministic controlled natural language pipelines (NAT2TESTSCR), through conditional parse NLP approaches (CiRA), retrieval augmented LLM systems (RAGTAG, LLM4Fin), onpremises deployments (Yin et al.), multi-LLM benchmarks (Korraprolu et al.), to fine-tuned LLM solutions (TestGPT). The colored bottom row summarizes the overall pattern across criteria; the detailed gap analysis immediately following the table explains the structural trade-offs that the summary row only hints at. The eight studies in Table IV were not chosen arbitrarily; they were selected by purposive maximum-variation sampling from the twenty-one primary studies, using four explicit criteria. First, era coverage: the subset spans all three evolutionary eras, from the deterministic Statistical/NLP-era pipelines (NAT2TESTSCR [3], UMTG [10]) to the most recent Hybrid/LLM-era systems (RAGTAG [15], LLM4Fin [13], the on-premises LLaMA-3 system [14], the multi-LLM benchmark [18], and TestGPT [20]). Second, architectural diversity: the subset deliberately includes one instance of each major architectural class identified in Sections IV and V - controlled natural language translation, model-based constraint solving, conditional-parse NLP, retrieval augmented generation, domain-tuned LLM, on-premises quantized LLM, multi-model benchmark, and fine-tuned LLM - so that the comparison captures the full design space rather than one region of it. Third, evaluation sufficiency: only studies that reported sufficient methodological and empirical detail to be coded on all six criteria, without 'not reported' cells, were included, thereby keeping the matrix fully populated and therefore interpretable. Fourth, influence and industrial validation: each selected study is either a frequently cited reference point or reports a real industrial or safety-critical deployment. This is a deliberate analytical design choice rather than a convenience sample: the six criteria require interpretive High/Medium/Low coding of each study, and several of the remaining thirteen primary studies do not report sufficient detail to be coded on all six dimensions without introducing 'not reported' cells that would weaken rather than strengthen the comparison.

TABLE IV. SIX-CRITERIA COMPARISON - EIGHT REPRESENTATIVE PRIMARY STUDIES (CONTRIBUTION C3) Re f

Techniq ue

C1 Automa tion

C2 Ambig uity

C3 Domai n

C4 Traceabi lity

C5 Evaluat ion

C6 Hallucin ation

10

UMTG: NLP SRL → OCL

Semiauto

Mediu m

Mediu m-2 studies

HIGH OCL; RTTM artifact

HIGH industri al

HIGH formal layer

3

CNL + SCR + T-VEC

Full

HIGH CNL

Low CNL requir ed

HIGH SCR tables

HIGH mutatio n; 4 domain s

HIGH determini stic

11

CiRA RoBER Ta

Full

HIGH explicit conditi on parse

Mediu m-3 industr y partne rs

Medium requirem ents linked to TCs

HIGH 578 TCs; 3 partner s

Medium - NLP rules + RoBERT a

15

RAGT AG RAG+L LM

Full

Mediu mRAG retrieva l helps

Mediu mbiling ual industr ial

Low - no RTTM produced

HIGH 4 experts ;5 dims

Medium - RAG anchors context

13

LLM4F in domaintuned

Full

Mediu mdomain knowle dge helps

Low FinTe ch only

Low none reported

HIGH BSC; 5 dataset s

LOW LLM; uncontrol led

14

LLaMA -3 70B on-prem

Full

Low no explicit mechan ism

Low enterp rise only

Low none reported

Mediu m90.3%; human eval

LOW LLM; no control

18

LLM benchm ark - 6 models

Full

Low no mechan ism

Mediu mmultidomai n

Low none reported

HIGH 4 metrics ; 25 reqs

LOW correctne ss not evaluated

20

TestGP T finetuned

Full

Mediu mfeedbac k loop helps

Mediu mneeds source code

Low none reported

Mediu msyntax + covera ge

LOW LLM; uncontrol led

In summary, none of the eight studies attains a high rating on all six criteria. Traceability (C4) is low in five studies and medium in one, and hallucination control (C6) is low in four studies and medium in two, making them the weakest dimensions in the comparison. CiRA [11] and RAGTAG [15] achieve high evaluation thoroughness with medium hallucination control by constraining generation through explicit NLP structure and retrieval grounding, respectively, but neither provides formal traceability. Overall, the comparison indicates a consistent trade-off: as input expressiveness increases, formal traceability decreases. Although the six-criteria matrix is applied to eight representative studies for readability, its central patterns are consistent with the full twenty-one-study corpus characterized in Table II, and the matrix is therefore not a generalization from an unrepresentative sample. The traceability deficit is corpus-wide: across all twenty-one studies, only the controlled-input and model-based approaches (NAT2TESTSCR [3], NAT2TESTCPN [8], UMTG [10]) report a formal requirements-to-test traceability artifact, while every fully generative LLM-based study omits one. The absence of explicit hallucination handling is likewise corpus-wide: no primary study outside the external-anchoring group (UMTG [10], NAT2TESTSCR [3], CiRA [11], RAGTAG [15]) reports a hallucination detection or mitigation mechanism. Finally, the accuracy-degradation pattern in Gap finding 3 is confirmed beyond the subset by Medeshetty et al. [17] and Shakthi et al. [16], which are part of the twenty-one-study corpus but not of the eight-study matrix. The eight-study table is thus an expository distillation of the full corpus, and its conclusions hold at corpus scale.

15

d)

Six Criteria Comparison: In Table IV, we defined six evaluation criteria that capture the core dimensions along which existing approaches differ. These criteria were applied to eight purposely selected and representative studies within the twenty-one primary studies set.

D. Detailed Gap-Finding Analysis of Table IV Building on the six-criterion comparison introduced above, the matrix in Table IV exposes structural gaps that no single primary study fully addresses. Each gap identified below is diagnosed with reference to specific cell values in the table and maps directly to one of the four research gaps (G1-G4) formalized in section VII. •

Gap finding 1 - the automation vs traceability diagonal.

Reading the C1 (Automation) and C4 (Traceability) columns jointly yields the clearest structural pattern in the matrix. NAT2TESTSCR [3] achieves Full automation (C1) and High traceability (C4), but only by restricting inputs to Controlled Natural Language, resulting in Low domain applicability (C3). UMTG [10] similarly achieves High traceability with semi-automated processes by relying on RUCM-templated use cases. In contrast, every study that relaxes input restrictions to unconstrained natural language LLM4Fin [13], the on-premises LLaMA 3 system [14], Korraprolu et al.’s multi-LLM benchmark [18], and TestGPT [20] exhibits Low traceability (C4). The resulting pattern is effectively diagonal: as approaches move along the automation and expressiveness axis from [3] to [18], formal traceability is progressively traded for greater input flexibility. No row in the matrix escapes this trade-off. This diagonal provides the direct empirical evidence for Gap G2 (VII) and motivates the RTTM pipeline recommendation R2 (VIII B). •

Gap finding 2 - hallucination control collapses in pureLLM approaches.

Column C6 (Hallucination control) shows Low ratings for four of the eight studies: LLM4Fin [13], the on-premises LLaMA-3 system [14], Korraprolu et al.'s multi-LLM benchmark [18], and TestGPT [20], all pure-LLM systems. In contrast, the systems achieving High or Medium hallucination control - UMTG [10], NAT2TESTSCR [3], CiRA [11], and RAGTAG [15] do so by introducing an explicit external anchoring mechanism. UMTG relies on OCL constraints as a downstream verifier; NAT2TESTSCR uses SCR tables; CiRA applies explicit cause-effect graph parsing; and RAGTAG employs retrieval augmented generation to bind outputs to project-specific context. While the mechanisms differ, the pattern is consistent: effective hallucination control requires architectural commitments beyond the LLM itself. The combined HaluEval finding of a 19.5 percent general-response hallucination rate [53] and the 90-percent versus 20-percent performance spread between GPT-4 and Mistral-7B on identical satellite FRDs reported by Shakthi et al. [16] establishes the external quantitative baseline against which any future TCG-specific hallucination benchmark - R1 (in VIII-A) must be calibrated. •

Gap finding 3 - evaluation thoroughness reproducibility are inversely related.

and

At first glance, column C5 (Evaluation thoroughness) suggests a mature field: six of the eight studies score High. However, the studies achieving High C5 through industrial case study evaluations - UMTG’s automotive studies [10], CiRA’s three partner deployment [11], RAGTAG’s Austrian

Post study [15], and LLM4Fin’s five-dataset FinTech evaluation [13] are precisely those whose underlying requirements and test case data cannot be released for reproducibility. By contrast, the two studies with publicly available benchmarks, Korraprolu et al.’s twenty-five requirement multi-LLM benchmark [18] and TestGPT [20], achieve High and Medium C5 scores, respectively, but trade off industrial case-study depth. This constitutes the reproducibility vs industrial depth trade-off, providing concrete evidence for Gap G3 (accuracy degrades with requirement complexity) and motivating the complexitystratified benchmark recommendation R3 (VIII C). The benchmark proposed in R3 is explicitly designed to occupy the intersection of multi-domain scope, public availability, and complexity stratification that no existing study currently achieves. •

Gap finding 4 - compliance, on-premises deployment, and the enterprise ceiling.

Within Table IV, only Yin et al.'s on-premises LLaMA3 system [14] directly addresses the combined constraints of data residency, export-control sensitivity, and deployable model size imposed by real enterprise environments. This evidence is reinforced beyond the table by Shakthi et al.'s ISRO satellite-FRD deployment [16], discussed in Section V, which is part of the twenty-one-study corpus but is not one of the eight studies tabulated here. The on-premises LLaMA-3 system scores Low on C3 (domain applicability) because it does not generalize beyond its specific deployment context, and Low on C4 (traceability) because it reports no traceability artifacts. Although this system achieves 90.3 percent coverage, the highest empirically demonstrated ceiling in Table IV for a deployable enterprise-grade configuration, the absence of reported regulatory-compliance evidence (e.g., GDPR, HIPAA, DO-178C) across all studies constitutes the structural signal for Gap G4 and motivates the regulationcompliant deployment-blueprint recommendation R4 (VIIID). Taken together, the four gap findings- the automation-vstraceability trade-off, the collapse of hallucination control without external anchoring, the reproducibility-vs-industrialdepth tension, and the compliance void reveal a common architectural limitation. The field has not yet produced an approach that is simultaneously highly automated, formally traceable, domain-broad, and regulation-compliant. Section VII builds directly on this comparative evidence by formalizing the four corresponding research gaps, G1-G4, and grounding each in the primary-study corpus. Given that the foregoing discussion draws corpus-level inferences from the patterns observed in Table IV, a methodological clarification is warranted. The six-criterion comparison reported in Table IV is applied to a purposively selected subset of eight studies rather than to the full set of twenty-one primary studies. This constitutes a deliberate analytical choice because the six criteria require fine-grained interpretive coding, limiting the matrix to studies that can be coded completely on all criteria and avoiding the proliferation of “not reported” cells that a compulsory full-corpus matrix would introduce. The potential validity threat that subset selection might distort the gap analysis is mitigated in two respects. First, the subset was constructed through maximum-

16

variation sampling across all eras and architectural classes. Second, the patterns identified in the subset were explicitly cross-checked against the full corpus of twenty-one studies summarized in Table II, as noted above. Accordingly, the four research gaps G1-G4 are grounded in evidence from the corpus as a whole rather than solely in the eight studies tabulated in Table IV. VII. RESEARCH CHALLENGES, GAPS, AND EVIDENCE-BASED ANALYSIS The comparative analysis in Section VI shows that, despite substantial progress in NLP and LLM-based test generation, several structural limitations remain unresolved. Existing approaches continue to trade automation for traceability, broad applicability for architectural control, and industrial realism for reproducibility. Building on that evidence, this section addresses RQ4 by formalizing four research gaps: G1 (hallucination), G2 (traceability), G3 (complexity sensitivity), and G4 (compliance). Together, these gaps capture the principal obstacles to trustworthy, scalable, and deployable AI-based test case generation from natural language requirements. Each gap is grounded in specific quantitative and architectural evidence from the primary studies and, in turn, motivates the actionable recommendations developed in Section VIII. A. Gap G1 - Absence of Hallucination Detection in LLMGenerated Test Cases Gap G1 concerns the absence of explicit hallucinationdetection mechanisms in LLM-based test case generation. The six-criterion analysis shows that hallucination control remains weakly addressed: in Table IV, four of the eight representative studies - LLM4Fin [13], the on-premises LLaMA-3 system [14], Korraprolu et al.’s multi-LLM benchmark [18], and TestGPT [20] - receive Low ratings on criterion C6 (hallucination control). These systems primarily rely on unconstrained generative inference and provide no explicit architectural mechanism for detecting or mitigating requirement-inconsistent test steps. In contrast, the approaches that achieve High or Medium hallucination control - UMTG [10], NAT2TESTSCR [3], CiRA [11], and RAGTAG [15] all introduce an external anchoring mechanism. UMTG employs OCL constraints as a downstream verifier; NAT2TESTSCR relies on SCR tables; CiRA uses explicit cause-effect graph parsing; and RAGTAG applies retrieval augmented generation to bind outputs to project-specific requirement context. Although these mechanisms differ in implementation, the pattern is consistent: effective hallucination control requires architectural commitments beyond the LLM itself. This gap is further contextualized by external empirical evidence. Li et al.’s HaluEval benchmark reports that 19.5 percent of general LLM responses (977 of 5,000 evaluated responses, verbatim) contain hallucinations [53], underscoring the prevalence of the problem. In the domain of test case generation, Shakthi et al. [16] demonstrate a 90 percent versus 20 percent correctness gap between GPT 4 and Mistral 7B on identical satellite Functional Requirement Documents, confirming that hallucination frequency is strongly model-dependent and amplified by input complexity. Despite this evidence, no primary study provides a testcase-specific hallucination benchmark or a systematic post-

hoc detection mechanism tailored to NL-to-TCG pipelines. This absence constitutes Gap G1 and motivates Recommendation R1, which calls for the development of a dedicated hallucination evaluation benchmark for LLMgenerated test cases (VIII A). B. Gap G2 - No System Achieves Both Full Automation and Formal Traceability Gap G2 concerns the absence of any approach that simultaneously achieves full automation and formal traceability for unconstrained natural-language requirements. UMTG [10] and NAT2TESTSCR [3] provide formal requirements-to-test traceability, but only by imposing controlled or templated input conditions. CiRA [11] preserves a partial link between extracted conditional structures and generated acceptance tests and therefore represents the closest approximation to traceability among NLP-based systems operating on unconstrained natural language; however, it does not produce a formal requirements-to-test traceability matrix. In contrast, all fully generative LLM-based systems produce no traceability artifact [14], [22], reflecting the absence of an auditable linkage between input requirements and generated test cases. This loss of traceability constitutes a systematic limitation of current end-to-end generative approaches. External architectural precedents demonstrate that traceability is achievable but remains decoupled from test case generation. Sawada et al. [30] demonstrate operational NLP + ML-based requirements to test traceability at NASA JPL, providing a concrete example of scalable traceability support in an industrial setting. More recently, Ge et al. [55] (IEEE TSE 2025) demonstrate LLM-based cross-level requirements tracing, establishing an architectural precedent for integrating traceability mechanisms within the LLM paradigm. However, neither approach couples’ traceability directly with natural-language-driven test case generation. Taken together, these findings show that no existing approach simultaneously achieves full automation and formal traceability when operating on unconstrained natural language requirements. This absence constitutes Gap G2 and directly motivates Recommendation R2, which calls for an auditable requirements-to-test traceability matrix (RTTM) pipeline (VIII-B). C. Gap G3 - Accuracy Degrades with Requirement Complexity Gap G3 concerns the systematic degradation of test generation accuracy as requirement complexity increases. Medeshetty et al. [17] provide the clearest direct quantification of this effect, reporting a stepwise decline from 95 percent to 75 percent to 60 percent accuracy across singlesignal, up-to-four-signal, and more-than-four-signal requirement categories. This result establishes a direct empirical relationship between requirement complexity and test generation accuracy. Evidence from LLM-based approaches reinforces this pattern. Korraprolu et al. [18] confirm that all six general-purpose LLMs evaluated fail to exceed 50.2 percent Modified Condition/Decision Coverage (MCDC) for complex, multicondition requirements, with Gemini achieving the highest reported score. Similarly, Shakthi et al. [16] report a pronounced 90 percent to 20 percent drop in correctness between GPT 4 and Mistral 7B when evaluated on ten identical satellite Functional Requirement Documents,

17

demonstrating that model capability and input complexity interact nonlinearly in practice. Despite the consistency of these findings, empirical evaluation remains constrained by limitations in the dataset. Publicly available requirements datasets, including PURE [54] and PROMISE, are small-scale and limited in domain coverage, restricting their ability to support systematic, crosscomplexity evaluation [26], [23]. As a result, accuracy degradation with increasing requirement complexity is well documented but insufficiently benchmarked across domains and model classes. Taken together, these observations show that no existing study provides a standardized, complexity-stratified evaluation framework for NL-to-TCG systems. This absence constitutes Gap G3 and directly motivates Recommendation R3, which calls for the development of a shared, multi-domain benchmark with explicit stratification by requirement complexity (VIII C). D. Gap G4 - Ethical, Security, and Regulatory Compliance Gap G4 concerns the absence of regulation-compliant, enterprise-ready deployment patterns for AI-based test case generation. Within Table IV, only Yin et al.'s on-premises LLaMA-3 system [14] directly addresses the combined constraints of data residency, export-control sensitivity, and deployable model size that real enterprise environments impose. This evidence is reinforced beyond the table by Shakthi et al.'s ISRO satellite-FRD deployment [16], which is discussed in Section V as part of the twenty-one-study corpus but is not one of the eight tabulated studies. The on-premises LLaMA-3 system scores Low on C3 (domain applicability) because it does not generalize beyond its specific deployment context, and it is silent on C4 (traceability) because it produces no requirements-to-tests traceability matrix.

VIII. GUIDELINES, RECOMMENDATIONS, AND EMERGING TRENDS This section addresses RQ4, which concerns how the research gaps identified in the preceding analysis should be addressed. The four evidence-based research gaps formalized in Section VII motivate a corresponding set of actionable guidelines, each designed to directly address a specific structural limitation observed in current approaches to AIbased test-case generation from natural-language requirements. Figure 7 presents a one-to-one mapping between the four research gaps (G1-G4) and the recommended guidelines (R1R4), making the correspondence between identified limitations and proposed remedies explicit. Gap G1 highlights the absence of hallucination detection in LLM-generated test cases and motivates R1: the development of HalluTCGBench, a test-case-generation-specific hallucination benchmark. Gap G2 captures the structural tension between full automation and formal traceability, leading to R2, an auditable requirements-to-test traceability matrix (RTTM) pipeline. Gap G3 identifies systematic accuracy degradation as requirement complexity increases, motivating R3, a complexity-stratified benchmark spanning multiple domains. Finally, Gap G4 exposes unresolved ethical, security, and regulatory constraints, which R4 addresses through regulation-compliant, on-premises deployment blueprints. Each recommendation is defined as a concrete, implementable deliverable artifact whose construction would directly address its associated research gap and advance the state of practice toward more reliable, traceable, and deployable AI-based test case generation.

Despite these limitations, the on-premises LLaMA 3 system reported by Yin et al. [14] achieves 90.3 percent coverage, representing the highest empirically demonstrated ceiling in Table IV for a deployable enterprise-grade configuration. This result establishes an upper bound on performance under realistic enterprise constraints but also highlights the narrow applicability and limited auditability of current compliant deployments. More broadly, cloud-based LLM solutions introduce unresolved risks related to data leakage, licensing, bias, and regulatory exposure. No study in Table IV reports concrete evidence of compliance with regulatory standards such as GDPR, HIPAA, or DO 178C, underscoring the absence of validation frameworks for regulated deployment contexts. This lack of reported compliance evidence constitutes a structural barrier to adoption in safety-critical and regulated domains. Taken together, these observations show that no existing approach satisfies enterprise-grade requirements for security, compliance, and auditability while maintaining broad domain applicability and traceability. This absence constitutes Gap G4 and directly motivates Recommendation R4, which calls for the development of regulation-compliant, on-premises deployment blueprints for AI-based test-case generation (VIII D).

Figure 7. Mapping of the four identified research gaps (G1–G4) to the corresponding recommendations (R1–R4) and deliverable artifacts.

A. R1 - Develop HalluTCG-Bench (addresses G1) This recommendation proposes the development of a testcase-generation-specific benchmark for evaluating hallucination based on real-world natural-language requirements across multiple application domains. Empirical evidence from existing studies establishes a clear baseline for such a benchmark. The retrieval-augmented mitigation evaluated by Arora et al. [15] and the model capability variation reported by Shakthi et al. [16] together demonstrate that hallucination behavior is both architecture-dependent and domain-dependent. In particular, the 90 percent versus 20 percent correctness gap between GPT-4 and Mistral-7B on identical satellite Functional Requirement Documents

18

confirms that hallucination rates cannot be treated as a modelagnostic phenomenon. Li et al.’s HaluEval benchmark [53] provides a validated annotation methodology that can be adapted to the NL-toTCG context, offering a foundation for systematically labeling hallucination-related errors. Complementing this, PyTester [21] demonstrates a reinforcement-learning-based optimization pathway that can support controlled experimentation with hallucination mitigation strategies once such a benchmark is available. Taken together, these studies motivate the development of HalluTCG-Bench as a dedicated evaluation artifact for hallucination in LLM-generated test cases. By grounding evaluation in real requirements, standardizing annotations, and enabling cross-model comparison, HalluTCG-Bench would directly address Gap G1 and provide the empirical infrastructure needed to assess and improve hallucination behavior in NL-to-TCG pipelines. B. R2 - Architect a Requirement-to-Test Traceability Pipeline (addresses G2) This recommendation addresses Gap G2, which identified the lack of end-to-end, auditable traceability between natural language requirements and generated test cases in fully automated NL-to-TCG systems. As established in Section VII, existing approaches achieve traceability only under restrictive conditions or as partial artifacts, while fully generative LLM-based systems produce no requirements to test traceability artifacts at all. The partial traceability provided by CiRA [11], in which each generated test is anchored to an extracted conditional structure, represents the closest existing starting point within NLP-based systems operating on unconstrained natural language. This observation motivates the design of a provenance-tracking post-processing pipeline that can generate an auditable requirements-to-test traceability matrix (RTTM) alongside the test cases it generates, rather than treating traceability as an external or manual activity. Architectural precedents demonstrating that such traceability is feasible already exist beyond end-to-end NL-to-TCG systems. Sawada et al. [30] demonstrate operational NLP + ML-based requirements to test traceability at NASA JPL, while Ge et al. [55] (IEEE TSE 2025) show that large language models can be used for cross-level requirements tracing. In parallel, UMTG [10] provides a validated reference architecture for formal constraint extraction that leverages OCL-based representations. R2 therefore proposes the development of an auditable RTTM pipeline that integrates provenance tracking with test case generation, building on CiRA’s conditional anchoring [11], formal constraint modeling as exemplified by UMTG [10], and LLM-assisted traceability techniques demonstrated by Sawada et al. [30] and Ge et al. [55]. The objective is not to replace existing generation pipelines, but to complement them with an explicit, inspectable traceability artifact that preserves automation while restoring auditability. The adoption of such an RTTM pipeline would directly address Gap G2 by reconciling full automation with formal traceability, enabling NL-to-TCG systems to satisfy core requirements for verification, certification, and accountability in industrial and safety-critical contexts.

C. R3 - Build a Complexity-Stratified Benchmark (addresses G3) This recommendation addresses Gap G3, which identified systematic accuracy degradation as requirement complexity increases and the absence of standardized, complexity-aware evaluation frameworks for NL-to-TCG systems. As shown in Section VII, both traditional NLP-based approaches and LLM-based systems exhibit sharp performance drops when moving from simple to multi-condition requirements. Yet, existing evaluations remain fragmented, domain-limited, and difficult to compare across studies. R3 proposes the construction of a multi-domain, complexitystratified benchmark for natural-language-to-test-case generation. The benchmark would combine the PURE dataset [54], consisting of 79 software requirements specification documents and 34,268 sentences, with the real satellite Functional Requirement Documents (FRDs) used by Shakthi et al. [16], the user story corpora from SPECMATE [9], and the industrial requirements employed in CiRA [11], which are now available with ground-truth test artifacts. Together, these sources serve as the initial building blocks and provide coverage across multiple domains and levels of structural complexity. The benchmark can be continuously enhanced with additional data. By explicitly stratifying requirements by complexity (e.g., number of conditions, signals, or interacting constraints), the benchmark would enable consistent evaluation of how accuracy degrades as requirements become more complex, thereby addressing the reproducibility and comparability limitations identified in current studies. In addition, Garousi et al. [27] identify further annotated requirements corpora that could be incorporated to extend domain coverage and support broader empirical validation. The development of such a benchmark would directly address Gap G3 by providing a shared, publicly available evaluation framework that supports cross-domain comparison, complexity-aware analysis, and reproducible benchmarking of NL-to-TCG approaches. This, in turn, would enable clearer assessment of progress across techniques and architectures and support more reliable conclusions about their scalability and robustness (VIII C). D. R4 - Regulation-Compliant On-Premises Deployment Blueprints (addresses G4) This recommendation addresses Gap G4, which identified unresolved ethical, security, and regulatory constraints that limit the adoption of AI-based test case generation in safetycritical and enterprise environments. As shown in Section VII, only two primary studies in the corpus - Yin et al.'s onpremises LLaMA-3 deployment [14], which is tabulated in Table IV, and Shakthi et al.'s ISRO satellite-FRD deployment [16], discussed in Section V, explicitly address data residency, export-control sensitivity, and deployable model-size constraints. Even in these cases, broad domain applicability and formal traceability remain unaddressed. R4 proposes developing regulation-compliant, on-premises deployment blueprints for NL-to-TCG systems. These blueprints are intended to document architectural and operational patterns for deploying test case generation pipelines entirely within controlled enterprise environments, without reliance on external APIs or cloud-hosted inference endpoints.

19

Empirical evidence demonstrates both the feasibility and the limitations of such deployments. Yin et al. [14] report that a 4bit quantized LLaMA3 70B system can achieve up to 90.3 percent coverage under on-premises constraints, representing the highest reported empirical ceiling in Table IV for an enterprise deployable configuration. Shakthi et al. [16] further show that open-source LLMs enable secure in-house processing of export-controlled satellite requirements. However, neither study provides guidance on traceability artifacts, compliance documentation, or generalization beyond the deployment context. The proposed deployment blueprints would consolidate such operational evidence into a reusable reference framework that outlines how NL-to-TCG systems can be configured, audited, and validated under regulatory and confidentiality constraints. The adoption of these blueprints would directly address Gap G4 by enabling the secure, compliant, and inspectable deployment of AI-based test-case-generation systems in regulated domains, thereby supporting responsible industrial adoption (VIII-D). E. Emerging Trends Beyond the four guidelines above, five emerging trends are shaping the next decade of intelligent software testing. Hybrid symbolic-neural architectures combine rule‑based engines with the semantic reasoning capabilities of large language models (LLMs), allowing symbolic components to encode domain‑specific rules or compliance constraints, while LLMs generate candidate test cases and explanatory rationales [34]. Multimodal requirement understanding extends test case generation beyond text by integrating information from natural language descriptions, UML/GUI diagrams, and code artifacts, enabling more comprehensive end‑to‑end test synthesis. At the model‑adaptation level, self‑adapting LLMs incorporate continuous fine‑tuning through reinforcement learning driven by test‑execution feedback, often complemented by active‑learning strategies to validate ambiguous or low‑confidence cases [20]. From an assurance perspective, explainable AI for testing emphasizes traceable, transparent rationales that link requirements to generated test scripts, an essential property for auditability and regulatory compliance in safety‑critical domains [22]. Finally, integration with DevOps pipelines enables automated, real‑time regression testing and continuous integration/continuous delivery; however, modular AI components that interoperate cleanly with popular CI/CD tools and standardized APIs remain an active area of maturation [35]. F. Closing Synthesis Taken together, the analysis of twenty-one primary studies, the evidence-based identification of four structural research gaps (G1-G4), and the corresponding recommendations (R1-R4) provide a coherent roadmap for advancing AI-based test case generation from natural language requirements. The proposed benchmarks, traceability pipelines, and deployment blueprints address foundational limitations in hallucination control, traceability, complexity sensitivity, and regulatory compliance. At the same time, the emerging trends outlined in Subsection E hybrid symbolic-neural architectures, multimodal requirement understanding, self-adapting LLMs, explainable AI, and DevOps integration indicate how these solutions may

evolve into more robust, transparent, and industrially deployable testing ecosystems. Together, these findings answer RQ1-RQ4 by clarifying the state of the art and identifying concrete, evidence-driven directions for addressing its most consequential limitations. The following section concludes the survey by summarizing its principal findings, revisiting the research questions, and outlining the implications for future research and practice. IX. CONCLUSIONS Automated test case generation from natural language requirements has evolved from early rule-based parsing and deterministic grammars to contemporary large-languagemodel-based semantic reasoning. Foundational frameworks such as NAT2TESTSCR [3] and UMTG [10] established a critical bridge between linguistic analysis and formal testing by showing that natural language inputs can be translated into analyzable test artifacts. Among the surveyed systems, UMTG provides the most complete hybrid symbolic-neural realization, achieving 95 percent correct OCL constraint generation across two industrial automotive case studies. Building on this foundation, domain-specific LLM systems such as LLM4Fin [13] demonstrate that explicit domain knowledge can substantially improve performance in regulated settings, achieving a peak business-scenario coverage of 98.18 percent and a five-dataset average of 91.89 percent, compared with ChatGPT’s 49.65 percent average. In parallel, the on-premises deployment of quantized LLMs [14] demonstrates that high-coverage generation is feasible in security-sensitive environments, although rigorous benchmarking of such constrained deployments remains limited. This survey identified twenty-one primary studies on AIbased test case generation from natural language software requirements published between 2000 and 2025, following the strict application of inclusion criteria I4 and I5 and venuequality criterion E6. Five studies ([8], [9], [11], [15], [16]) were identified through backward snowballing on the reference list of Yang et al. [22] and verified through full-text review. Despite sustained progress, major challenges remain, including hallucinations in generated test cases, limited explainability and formal traceability, data scarcity for minority domains, and the absence of standardized evaluation benchmarks. Addressing these limitations will require balanced integration of hybrid symbolic-neural architectures, domain-knowledge grounding, and responsible AI testing practices. The four evidence-driven recommendations proposed in this survey - HalluTCG-Bench (R1), the requirements-to-test traceability matrix/pipeline - RTTM (R2), the complexity-stratified benchmark (R3), and the regulation-compliant deployment blueprint (R4) together provide a coherent roadmap for addressing structural gaps G1G4 and advancing trustworthy, deployable automation for requirements-based software testing. Overall, this paper provides an evidence-based survey that synthesizes twenty-one primary studies into a comparative framework, exposes the field’s central trade-offs, and derives four concrete research recommendations for advancing trustworthy, scalable, and industrially deployable AI-based test case generation from natural language requirements.

20

ACKNOWLEDGMENTS The author expresses sincere gratitude to the researchers whose work underpins this survey. Their foundational contributions to requirements‑based testing, natural language processing, and AI‑driven software testing have been essential in advancing the state of the art examined in this paper. The author also thanks colleagues and reviewers for their constructive feedback, which helped improve the clarity, rigor, and organization of this study. The author is particularly grateful to Dr. Yetunde Orimoloye for her valuable input during the review process. Her insightful feedback helped refine the research approach, and her expertise was instrumental in supporting data analysis and ensuring the accuracy and rigor of the results presented. REFERENCES [1]

A. Dwarakanath and S. Sengupta, "Litmus: Generation of test cases from functional requirements in natural language," in Proc. Int. Conf. Appl. Natural Lang. Inform. Syst. (NLDB), Berlin, Heidelberg: Springer, 2012, pp. 58–69. [2] R. P. Verma and M. R. Beg, "Generation of test cases from software requirements using natural language processing," in Proc. 6th Int. Conf. Emerg. Trends Eng. Technol., IEEE, 2013, pp. 140–147, doi: 10.1109/ICETET.2013.45. [3] G. Carvalho, D. Falcão, F. Barros et al., "NAT2TESTSCR: Test case generation from natural language requirements based on SCR specifications," Sci. Comput. Program., vol. 95, no. 3, pp. 275–297, 2014, doi: 10.1016/j.scico.2014.06.007. [4] O. Olajubu, S. Ajit, M. Johnson, S. Turner et al., "Automated test case generation from domain-specific models of high-level requirements," in Proc. ACM Res. Adaptive Convergent Syst. (RACS), Prague, Czech Republic, 2015, pp. 505–508, doi: 10.1145/2811411.2811555. [5] A. Ansari, M. B. Shagufta, A. S. Fatima, and S. Tehreem, "Constructing test cases using natural language processing," in Proc. 3rd Int. Conf. Adv. Electrical Electronics Information Commun. BioInformatics (AEEICB), IEEE, 2017, pp. 95–99, doi: 10.1109/AEEICB.2017.7972460. [6] S. Kamonsantiroj, L. Pipanmaekaporn, and S. Lorpunmanee, "A memorization approach for test case generation in concurrent UML activity diagram," in Proc. ACM Int. Conf. Geoinformatics and Data Analysis (ICGDA), 2019, pp. 20–25, doi: 10.1145/3318236.3318256. [7] S. C. Allala, J. P. Sotomayor, D. Santiago et al., "Towards transforming user requirements to test cases using MDE and NLP," in Proc. 43rd IEEE Annu. Comput. Softw. Appl. Conf. (COMPSAC), vol. 2, 2019, pp. 350–355, doi: 10.1109/COMPSAC.2019.10251. [8] B. C. F. Silva, G. Carvalho, and A. Sampaio, "CPN simulation-based test case generation from controlled natural-language requirements," Sci. Comput. Program., vol. 181, pp. 111–139, 2019, doi: 10.1016/j.scico.2019.04.001. [9] J. Fischbach, A. Vogelsang, D. Spies, A. Wehrle, M. Junker, and D. Freudenstein, "SPECMATE: Automated creation of test cases from acceptance criteria," in Proc. IEEE Int. Conf. Softw. Testing Verification Validation (ICST), 2020, pp. 321–331, doi: 10.1109/ICST46399.2020.00040. [10] C. Wang, F. Pastore, A. Goknil, and L. C. Briand, "Automatic generation of acceptance test cases from use case specifications: An NLP-based approach," IEEE Trans. Softw. Eng., vol. 48, no. 2, pp. 585–616, 2022, doi: 10.1109/TSE.2020.2998503. [11] J. Fischbach, J. Frattini, A. Vogelsang, D. Mendez et al., "Automatic creation of acceptance tests by extracting conditionals from requirements: NLP approach and case study," J. Syst. Softw., vol. 197, art. 111549, 2023, doi: 10.1016/j.jss.2022.111549. [12] J. W. Lim, T. K. Chiew, M. T. Su et al., "Test case information extraction from requirements specifications using NLP-based unified boilerplate approach," J. Syst. Softw., vol. 211, art. 112005, 2024, doi: 10.1016/j.jss.2024.112005. [13] Z. Xue, L. Li, S. Tian et al., "Domain knowledge is all you need: A field deployment of LLM-powered test case generation in FinTech domain," in Proc. IEEE/ACM 46th Int. Conf. Softw. Eng.: Companion (ICSE Companion), 2024, pp. 314–315, doi: 10.1145/3639478.3643532.

[14] H. Yin, H. Mohammed, and S. Boyapati, "Leveraging pre-trained large language models (LLMs) for on-premises comprehensive automated test case generation: An empirical study," in Proc. IEEE 9th Int. Conf. Intell. Informatics Biomed. Sci. (ICIIBMS), vol. 9, 2024, pp. 597–607, doi: 10.1109/ICIIBMS62405.2024.10790549. [15] C. Arora, T. Herda, and V. Homm, "Generating test scenarios from NL requirements using retrieval-augmented LLMs: An industrial study," in Proc. IEEE 32nd Int. Req. Eng. Conf. (RE), 2024, pp. 240–251, doi: 10.1109/RE59067.2024.00031. [16] S. Shakthi, P. Srivastava, R. Kumar, and S. G. Prasad, "Automated test case generation for satellite FRD using NLP and large language model," in Proc. IEEE Int. Conf. Electrical Comput. Commun. Mechatronics Eng. (ICECCME), 2024, pp. 1–8, doi: 10.1109/ICECCME62383.2024.10796866. [17] N. Medeshetty, A. N. Ghazi, S. Alawadi, and F. Alkhabbas, "From requirements to test cases: An NLP-based approach for highperformance ECU test case automation," in Proc. IEEE Int. Conf. Human-Machine Syst. (ICHMS), 2025, doi: 10.1109/ICHMS65439.2025.11154348. [18] B. R. Korraprolu, P. Pinninti, and Y. R. Reddy, "Test case generation for requirements in natural language — An LLM comparison study," in Proc. 18th Innovations Softw. Eng. Conf. (ISEC), 2025, pp. 1–5, doi: 10.1145/3717383.3717389. [19] A. Najmi and M. El-Dosuky, "Intelligent software testing for test case analysis framework using ChatGPT with natural language processing and deep learning integration," J. Comput. Sci., vol. 21, no. 5, pp. 1140–1155, 2025, doi: 10.3844/jcssp.2025.1140.1155. [20] S. Alagarsamy, C. Tantithamthavorn, W. Takerngsaksiri et al., "Enhancing large language models for text-to-testcase generation," J. Syst. Softw., vol. 230, pp. 1–26, 2025, doi: 10.1016/j.jss.2025.112353. [21] W. Takerngsaksiri, R. Charakorn, C. Tantithamthavorn, and Y.-F. Li, "PyTester: Deep reinforcement learning for text-to-testcase generation," J. Syst. Softw., vol. 224, art. 112381, 2025, doi: 10.1016/j.jss.2025.112381. [22] Z. Yang, R. Huang, C. Cui et al., "Requirements-based test generation: A comprehensive survey," ACM Trans. Softw. Eng. Methodol., 2025, doi: 10.1145/3771727. [23] A. Mustafa, W. M. N. Wan-Kadir, N. Ibrahim et al., "Automated test case generation from requirements: A systematic literature review," Comput. Mater. Continua, vol. 67, no. 2, pp. 1819–1833, 2021, doi: 10.32604/cmc.2021.014391. [24] M. Boukhlif, M. Hanine, N. Kharmoum et al., "Natural language processing-based software testing: A systematic literature review," IEEE Access, vol. 12, pp. 79383–79400, 2024, doi: 10.1109/ACCESS.2024.3407753. [25] J. Wang, Y. Huang, C. Chen et al., "Software testing with large language models: Survey, landscape, and vision," IEEE Trans. Softw. Eng., vol. 50, no. 4, pp. 911–936, 2024, doi: 10.1109/TSE.2024.3368208. [26] [26] A. Rodrigues, J. Vilela, and C. Silva, "A systematic mapping study on techniques for generating test cases from requirements," in Proc. 9th Int. Conf. Internet Things, Big Data and Security, SCITEPRESS, 2024, pp. 141–148, doi: 10.5220/0012399700003648. [27] [27] V. Garousi, S. Bauer, and M. Felderer, "NLP-assisted software testing: A systematic mapping of the literature," Inf. Softw. Technol., vol. 126, art. 106321, 2020, doi: 10.1016/j.infsof.2020.106321. [28] [28] L. Zhao, W. Alhoshan, A. Ferrari et al., "Natural language processing for requirements engineering: A systematic mapping study," ACM Comput. Surv., vol. 54, no. 3, art. 55, 2021, doi: 10.1145/3444689. [29] [29] D. Amalfitano, S. Faralli, J. C. R. Hauck, S. Matalonga, and D. Distante, "Artificial intelligence applied to software testing: A tertiary study," ACM Comput. Surv., vol. 56, 2023, doi: 10.1145/3616372. [30] [30] K. Sawada, M. Pomerantz, and G. Razo, "Intelligent requirementto-test-case traceability system via natural language processing and machine learning," in Proc. IEEE Space Mission Challenges Inform. Technol. (SMC-IT), 2023, doi: 10.1109/SMC-IT56444.2023.00017. [31] [31] R. D. Craig and S. P. Jaskiel, Systematic Software Testing. Norwood, MA: Artech House, pp. 5–9, 2002. [32] [32] I. K. El-Far and J. A. Whittaker, "Model-based software testing," in Encyclopedia of Software Engineering, Wiley, 2002, doi: 10.1002/0471028959.sof207. [33] [33] M. Fewster and D. Graham, Software Test Automation: Effective Use of Test Execution Tools. Addison-Wesley, pp. 1–574, 1999.

21

[34] [34] O. Bougzime, S. Jabbar, C. Cruz, and F. Demoly, "Unlocking the potential of generative AI through neuro-symbolic architectures: Benefits and limitations," arXiv:2502.11269, pp. 1–55, 2025. [35] [35] S. Polampally, K. Kudithipudi, V. K. Jyothi et al., "Leveraging AI for continuous quality assurance in Agile software development cycles," Cloud Computing and Data Science, vol. 7, no. 1, pp. 25–38, 2025. [36] [36] G. J. Myers, T. Badgett, T. M. Thomas, and C. Sandler, The Art of Software Testing, 2nd ed. Hoboken, NJ: Wiley, pp. 3–11, 2004. [37] [37] E. Dustin, T. Garrett, and B. Gauf, Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality. Pearson Education, pp. 1–10, 2009. [38] [38] A. Kao and S. R. Poteet, Natural Language Processing and Text Mining. Cham, Switzerland: Springer, pp. 1–264, 2007. [39] [39] J.-M. Bruel, S. Ebersold, F. Galinier et al., "The role of formalism in system requirements," ACM Comput. Surv., vol. 54, no. 5, pp. 1– 36, 2021, doi: 10.1145/3448975. [40] [40] M. Glinz, "On non-functional requirements," in Proc. 15th IEEE Int. Req. Eng. Conf., 2007, pp. 21–26, doi: 10.1109/RE.2007.45. [41] [41] B. Vaysburg, L. H. Tahat, and B. Korel, "Dependence analysis in reduction of requirement-based test suites," in Proc. ACM SIGSOFT Int. Symp. Softw. Testing and Analysis (ISSTA), 2002, pp. 107–111. [42] [42] Object Management Group, OMG Systems Modeling Language (OMG SysML), vol. 1, pp. 1–258, 2007. [43] [43] B. Berenbach, F. Schneider, and H. Naughton, "The use of a requirements modeling language for industrial applications," in Proc. 20th IEEE Int. Req. Eng. Conf., 2012, pp. 285–290, doi: 10.1109/RE.2012.6345813. [44] [44] A. Naumchev, "Seamless object-oriented requirements," in Proc. 2019 Int. Multi-Conf. Eng. Comput. Inform. Sci. (SIBIRCON), IEEE, 2019, pp. 0743–0748, doi: 10.1109/SIBIRCON48586.2019.8958127. [45] [45] E. G. Cartaxo, F. G. Neto, and P. D. Machado, "Test case generation by means of UML sequence diagrams and labeled transition systems," in Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC), 2007, pp. 1292–1297. [46] [46] S. Shaheen, H. Arshad, M. Mir, and M. T. Hashmi, "A case study: To discuss the construction of the finite state machine and Markovchain OP models for test case generation," in Proc. 5th Int. Conf. Adv. Comput. Sci. (ICACS), IEEE, 2024, pp. 1–9, doi: 10.1109/ICACS60934.2024.10473284.

[47] [47] L. Kadakolmath and U. D. Ramu, "Model-checking-based automated test case generation for Z formal specification of an urban railway interlocking system," in Proc. 4th Int. Conf. Emerg. Res. Electron. Comput. Sci. Technol. (ICERECT), IEEE, 2022, pp. 1–8, doi: 10.1109/ICERECT56837.2022.10060801. [48] [48] K. Saiki, S. Liu, H. Okamura, and T. Dohi, "A tool to support vibration testing method for automatic test case generation and test result analysis," in Proc. IEEE 21st Int. Conf. Softw. Qual. Reliab. Security (QRS), 2021, pp. 149–156, doi: 10.1109/QRS54544.2021.00025. [49] [49] R. J. Cajica, R. E. G. Torres, and P. M. Álvarez, "Automatic generation of test cases from formal specifications using mutation testing," in Proc. 18th Int. Conf. Electrical Eng. Comput. Sci. Autom. Control (CCE), IEEE, 2021, pp. 1–6, doi: 10.1109/CCE53527.2021.9633118. [50] [50] S. Helke, T. Neustupny, and T. Santen, "Automating test case generation from Z specifications with Isabelle," in Proc. 10th Int. Conf. Z Users, Springer LNCS 1212, 1997, pp. 52–71, doi: 10.1007/BFb0027300. [51] [51] M. Viggiato, D. Paas, C. Buzon, and C.-P. Bezemer, "Using natural language processing techniques to improve manual test case descriptions," in Proc. 44th Int. Conf. Softw. Eng.: Software Engineering in Practice (ICSE-SEIP), 2022, pp. 311–320, doi: 10.1145/3510457.3513045. [52] [52] B. Kitchenham and S. Charters, "Guidelines for performing systematic literature reviews in software engineering," Software Engineering Group, Keele Univ., UK, and Dept. of Computer Science, Univ. of Durham, UK, EBSE Tech. Rep. EBSE-2007-01, 2007. [53] [53] J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen, "HaluEval: A large-scale hallucination evaluation benchmark for large language models," in Proc. Conf. Empirical Methods Natural Lang. Processing (EMNLP), 2023, doi: 10.18653/v1/2023.emnlp-main.397. [54] [54] A. Ferrari, G. O. Spagnolo, and S. Gnesi, "PURE: A dataset of public requirements documents," in Proc. IEEE Int. Req. Eng. Conf. (RE), 2017, pp. 502–505, doi: 10.1109/RE.2017.50. [55] [55] C. Ge, T. Wang, X. Yang, and C. Treude, "Cross-level requirements tracing based on large language models," IEEE Trans. Softw. Eng., vol. 51, no. 7, pp. 2044–2066, Jul. 2025, doi: 10.1109/TSE.2025.3572094.

22

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500