arXiv:2604.22207v1 [cs.SE] 24 Apr 2026
Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations Anna Arnaudo
Riccardo Coppola
Maurizio Morisio
[email protected] Department of Control and Computer Engineering, Politecnico di Torino Torino, IT
Department of Control and Computer Engineering, Politecnico di Torino Torino, IT
Department of Control and Computer Engineering, Politecnico di Torino Torino, IT
Flavio Giobergia
Andrea Bioddo
Angelo Bongiorno
Department of Control and Computer Engineering, Politecnico di Torino Torino, IT
Politecnico di Torino Torino, IT
Politecnico di Torino Torino, IT
Luca Dadone Politecnico di Torino Torino, IT
Abstract
CCS Concepts
Due to the textual and repetitive nature of many Requirements Engineering (RE) artefacts, Large Language Models (LLMs) have proven useful to automate their generation and processing. In this paper, we discuss a possible approach for automating the GoalOriented Requirements Engineering (GORE) process by extracting functional goals from software documentation through three phases: actor identification, high and low-level goal extraction. To implement these functionalities, we propose a chain of LLMs fed with engineered prompts. We experimented with different variants of in-context learning and measured the similarities between input data and in-context examples to better investigate their impact. Another key element is the generation-critic mechanism, implemented as a feedback loop involving two LLMs. Although the pipeline achieved 61% accuracy in low-level goal identification — the final stage — these results indicate the approach is best suited as a tool to accelerate manual extraction rather than as a full replacement. The feedback-loop mechanism with Zero-shot outperformed stand-alone Few-shot, with an ablation study suggesting that performance slightly degrades without the feedback cycle. However, we reported that the combination of the feedback mechanism with Few-shot does not deliver any advantage, possibly suggesting that the primary performance ceiling is the prompting strategy applied to the ’critic’ LLM. Together with the refinement of both the quantity and quality of the Shot examples, future research will integrate Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting to improve accuracy.
• Software and its engineering → Requirements analysis; • General and reference → Empirical studies; • Computing methodologies → Natural language generation.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, Scotland, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Keywords Large Language Models, Software Engineering, Requirements Engineering, Goal Oriented Requirements Engineering, Prompt Engineering ACM Reference Format: Anna Arnaudo, Riccardo Coppola, Maurizio Morisio, Flavio Giobergia, Andrea Bioddo, Angelo Bongiorno, and Luca Dadone. 2026. Evaluating LLMBased Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
Rising interest in LLMs for software engineering [24] highlights their effectiveness in analysing and generating structured artifacts [2, 15]. In the field of Requirements Engineering (RE) some studies have explored the extraction of goal models from natural language requirement specifications [7], the extraction of domain models from textual requirements [5], the enhancement of Use Case definition with LLM-based agents [8], goal-model generation from user stories [20], class/behavioral model synthesis and benchmarking [6]. While most current applications of LLMs in software engineering rely on limited interactions with a single model instance, emerging research highlights a shift toward more collaborative paradigms. Integrating Large Language Models into multi-agent systems (MAS) represents a significant advancement, enabling agents to assume specialised roles, coordinate their actions, and collectively address complex software engineering challenges [12, 25]. Drawing inspiration from the literature, we envision a system architecture organised around multiple LLMs emulating the different steps typically involved in complex processes.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Moreover, increasing attention is directed towards prompting strategies and their effectiveness in supporting software engineering[]. For this reason, we experimented with different variants of in-context learning to assess how they affect the proposed architecture. The ultimate objective of our architecture is the refinement of high-level goals into increasingly concrete, operationalised forms, thus obtaining the list of the system’s functional requirements. Although mapping from low-level goals to API endpoints is not the main focus of this study, we include it in our conceptual architecture because it may provide useful insights about future extensions of this work. The contributions of this work can be listed as follows: • We proposed a novel architecture that decomposes the GORE process into multiple steps involving a feedback-loop mechanism; • We evaluated the proposed architecture by varying the prompting strategy of the generator model; • We demonstrated that the combination of Zero-shot prompting with a feedback-loop outperforms Few-shot prompting applied to GPT alone in the target tasks; • We measured the similarity between the Few-shot and the ground truth examples, analysing the possible implications on the performances of our system. The replication package of the study is available on Zenodo1 .
1.1
Research Questions (RQs)
We formalised the evaluation of the proposed architecture through the following RQs: • RQ1: What is the effectiveness of the multi-agent architecture in extracting the list of actors, and how it is influenced by Shot-prompting? • RQ2: What is the effectiveness of the multi-agent architecture in modelling high-level goals, and how it is influenced by Shot-prompting? • RQ3: What is the effectiveness of the multi-agent architecture in decomposing high-level goals into low-level goals, and how it is influenced by Shot-prompting? • RQ4: Does the critic mechanism actually improve the extraction of actors, high and low-level goals?
2 Background 2.1 The GORE Framework Introduced by Van Lamsweerde in 2001 [21], Goal-Oriented Requirements Engineering (GORE) is a technique with the primary objective of identifying all goals of a system, defined as Objectives that the system under consideration should achieve, which can be then mapped to functional or non-functional requirements. Goals can be formulated at different levels of abstraction, ranging from highlevel strategic concerns to low-level technical ones. The technique also includes the identification of the actors — i.e., stakeholders of the system, including its final users — goal prioritisation, and conflict detection. While low-level goals are typically derived from parent goals by asking ‘how’, the reverse path — goal abstraction 1 https://zenodo.org/records/18919525
Arnaudo et al.
— is achieved by answering ‘why’ [21]. For sake of simplicity, we consider only functional goals in our work, as the total number of non-functional goals may raise to an unmanageable amount, especially in case of manual datasets curation.
2.2
Large Language Models in Requirements Engineering
Zadenoori et al. [26] provide a comprehensive survey of LLM integration into RE, noting an exponential trajectory in publications beginning in 2023. Their analysis reveals that the majority of current studies utilise GPT-family models, with a heavy emphasis on Zeroshot (used in 44% of the surveyed studies) and Few-shot prompting (29%). A key finding of the survey is the current inclination towards using models without further task-specific optimisation; the authors argue that this trend highlights a requirement for more rigorous investigation into advanced model architectures and orchestration — as performed by this work. Moreover, significant hurdles remain concerning the deterministic reliability of these systems and the integration of human supervisory oversight. Sami et al. [19] developed a multi-agent system where a Product Owner agent generates user stories that are subsequently validated by a Quality Assurance agent against the INVEST [3] and ISO/IEC/IEEE 29148-2011 [1] standards. Their findings, which address a challenge similar to that addressed herein, emphasise the advantages of inter-agent communication and orchestration over a single, monolithic LLM instance. However, that study only covers the ’user story’ format, while in the present work we target system goals. Das et al. [7] propose NLP-driven techniques for extracting structured goal models from unstructured textual input, reducing manual effort and enhancing requirement elicitation accuracy. Similarly, recent work on API Alignment [11] integrates GORE with LLM-based techniques by leveraging multiple iterative prompts and the GPT model to extract goals and map them to existing APIs. The paper demonstrates the potential of LLMs for automating goal extraction and API selection, but it also suffers from limitations in validation, including inconsistent goal decomposition, unexplained omissions, and a lack of structured quality control. Our approach, building on existing work, aims to expand the investigation of a multi-LLM pipeline for goal extraction and to offer a further exploratory evaluation of its feasibility and limitations.
3 Approach 3.1 Architecture We based our architecture on a fixed-structure LLM chain, represented in Figure 1. Since GORE consists of clearly defined and sequential steps, agents’ inherent autonomy and flexibility were not well-suited. On the other hand, a structured chain ensures greater control over the execution flow. At the core of our architecture, two LLMs — GPT-4 and Llama 3.3 70B — collaborate through an iterative feedback loop. GPT-4 functions as the ’generator’, while Llama 3.3 70B operates as the ’evaluator’. Although the research prioritises the optimisation of prompting strategies and interaction patterns, this configuration has been chosen since it leverages GPT-4 for high-quality text generation and Llama for a lower-cost response evaluation. If Llama’s
Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
evaluation score is below 8.5/10 — that will be referred as Quality Threshold 2 — the critique is inserted into the prompt fed to GPT-4 in the next iteration. Until the Quality Threshold is not met, the system can perform a maximum of 3 iterations before continuing to the next phase, as we empirically found that this number is a good trade-off between accuracy and computational overhead. Moreover, we experimentally observed that if the ’generator’ and the ’critic’ agents fail to reach an agreement within this number of iterations, then it is unlikely to reach convergence. Future works may investigate in more depth the effect on performance when these parameters are varied.
3.2
Multi-step Pipeline
Table 1: Average cosine similarities between the Shot examples used in the prompts for the generator agent — i.e., GPT — and the different software projects constituting our ground truth dataset. Dataset
Task
Genome Nexus
Actors high-level goals low-level goals Actors high-level goals low-level goals Actors high-level goals low-level goals Actors high-level goals low-level goals Actors high-level goals low-level goals
Gestao Hospital
London Ambulance
As represented in Figure 1, the pipeline we propose is composed of multiple phases: Urban Maintenance
(1) Documentation Preprocessing (optional): It transforms a raw README file into a natural language project description, serving as a preprocessing step to improve the accuracy of downstream tasks. This is automated through the use of GPT-4, prompted as reported in Table 3; (2) Actor Identification: Actors are active entities that carry actions to achieve one or multiple goals [16]; (3) High-level Goals Extraction: High-level goals are the broad project objectives. For each identified actor, this step aims to extract the main goals that he may want to achieve when interacting with the software; (4) Low-level Goals Extraction: Low-level goals are specific, actionable objectives. Complex high-level goals are decomposed into a hierarchy of low-level goals; (5) API Mapping: Goals can be finally mapped to the API endpoints that implement the related functionalities. This step needs as input the set of low-level goals and the API documentation of the software under analysis. The first step is optional because natural language project descriptions were already available for the London Ambulance Service and Urbain Maintenance case studies (described in Section 4.2). Actors, high-level and low-level goals extraction is implemented through the iterative feedback loop described in Section 3.1, employing the prompts reported in Table 3.
3.3
Prompting Strategy
The chain integrates diverse in-context learning strategies, including Zero-shot, One-shot, and Few-shot. Three Shot examples have been manually curated for each task, by adapting documentation and requirements of existing open-source projects. Specifically, they are related to (i) an application that extracts statistics associated to GitHub accounts; (ii) a food delivery application; and (iii) a home maintenance service locator. Some examples of prompts are reported in Table 3, while Table 4 contains some instances of Few-shot examples integrated into prompts. The cosine similarities between the ground truth examples — described in Section 4.2 — and the Shot examples are reported in Table 1.
2 This threshold has been empirically derived, by qualitatively analysing the generated
goals.
Average per Task
Average Similarity 0.5137 0.5151 0.5323 0.5219 0.5113 0.5082 0.5319 0.5327 0.5327 0.5007 0.5067 0.5144 0.5171 0.5164 0.5219
Table 2: Average cosine similarities between the Shot examples fed to the critic agent — i.e., Llama — in each task and the different software projects forming the ground truth. Dataset
Task
Genome Nexus
Actors high-level goals low-level goals Actors high-level goals low-level goals Actors high-level goals low-level goals Actors high-level goals low-level goals Actors high-level goals low-level goals
Gestao Hospital
London Ambulance
Urban Maintenance
Average per Task
Average Similarity 0.5514 0.5407 0.5257 0.5289 0.4991 0.5066 0.5031 0.4687 0.5037 0.5382 0.5236 0.5631 0.5304 0.5080 0.5248
While our study evaluates diverse prompting strategies for the ’generator’ agent, the ’evaluator’ was restricted to a Few-shot configuration to mitigate scoring insensitivity. Empirical observations revealed that — absent explicit benchmarks for ’major’ versus ’minor’ errors — the Llama model lacked the necessary evaluative anchors to assign nuanced ratings. Without these calibrated exemplars, the model exhibited a systemic bias toward invariant scoring across disparate iterations. The cosine similarities between the ground truth examples and the Shot examples feed to the ’evaluator’ are reported in Table 2. At the present stage of development, the framework is restricted to functional requirements by embedding explicit constraints within
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Arnaudo et al.
Figure 1: Schema of the proposed architecture, consisting of a LLM chain the model prompts. However, the architecture maintains the flexibility to encompass also non-functional goals. This extension would necessitate removing the existing prompt instructions and updating both the ground-truth data and the Few-shot exemplars to reflect these broader goals. To enable the reproducibility of the results, the temperature has been set to zero to improve determinism in both GPT and Llama’s generations. Moreover, prompts are enhanced by a priming technique3 , as can be seen from the prompt samples reported in Table 3. Finally, it is fundamental to ensure that model outputs are structured and consistent. For this purpose, we use OpenAI’s Pydantic library4 , which allows us to enforce a structured schema for GPT-generated responses, while Llama outputs are formatted using standard JSON serialisation.
4 Evaluation Method 4.1 Evaluation Pipeline The evaluation procedure consists of the following steps: (1) Preprocessing: Stopwords remotion, stemming and lemmatisation are applied to reduce variation in text representation. This can enhance the accuracy of the subsequent similarity computation. Due to their simpler formulation, preprocessing is not applied to actor names, but only on goals descriptions; (2) Encoding: Both generated and reference data are transformed into vector embeddings using a BERT-based model; (3) Similarity Computation: Cosine similarity is calculated between generated and reference goal embeddings, forming a similarity matrix; (4) Maximum Weight Bipartite Matching: For each actor or goal, it is necessary to find the element of the ground
truth that it aims to resemble. To do so, we employ the algorithm proposed by Munkres et al. [18] to solve the Maximum Weight Bipartite Matching problem, which prescribes to model the generated and the reference items as nodes of a bipartite graph. In this undirected graph, the arcs are weighted by the cosine similarity between the two strings associated to the nodes. By finding the set of arcs generating the maximum sum of weights, each string is linked to its best match.5 (5) Computation of Precision, Recall and F1-score: For the computation of these performance metrics, we employed the formulas from Zhang et al. [27] reported below:6 𝑅𝑒𝑐𝑎𝑙𝑙 = |𝑋1 |
Í
⊤ (𝑖,𝑗 ) ∈ 𝐽 x𝑖 y 𝑗
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = |𝑌1 |
Í
⊤ (𝑖,𝑗 ) ∈ 𝐽 x𝑖 y 𝑗
F1-score = 2∗𝑅𝑒𝑐𝑎𝑙𝑙∗𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑅𝑒𝑐𝑎𝑙𝑙+𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 Where X is the set of embeddings from the generated strings; Y is the set of embeddings from the ground truth; J is the set of arcs identified in the previous step. The scalar product between x𝑖 and y𝑖 computes the cosine similarity between the relative strings. The approach of computing cosine similarity between BERT embeddings effectively distinguishes true semantic equivalence from surface-level resemblance, ensuring robust adherence to the ground truth. At the same time, solving the Maximum Weight Bipartite Matching problem allows us to effectively measure the similarity between the generated and the reference sets of actors or goals. 5 In case the lists of the generated and the reference strings have different lengths,
3 In the context of prompt engineering, priming is the practice of strategically providing
contextual input within a prompt to shape the model’s responses toward a desired style, reasoning process, or domain of information. 4 https://ai.pydantic.dev/models/openai/
some items remain unpaired. If this happens, those items contribute to the counting of false positives or false negatives respectively. 6 They applied these formulas on two sets of words, while we applied them to two sets of sentences.
Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 3: Prompts provided to the LLM model to accomplish the different tasks of the proposed pipeline. Preprocess the README file System prompt:
Prompt:
You a r e a t e c h n i c a l w r i t i n g a s s i s t a n t s p e c i a l i z e d i n s u m m a r i z i n g s o f t w a r e d o c u m e n t a t i o n . Your g o a l i s t o e x t r a c t a c l e a r , w e l l − w r i t t e n , and a c c u r a t e d e s c r i p t i o n o f a p r o j e c t from i t s README f i l e . The d e s c r i p t i o n s h o u l d be n a t u r a l and i n f o r m a t i v e , w i t h o u t u n n e c e s s a r y d e t a i l s o r i m p l e m e n t a t i o n s p e c i f i c s . Avoid m a r k e t i n g l a n g u a g e , vague c l a i m s , or f i l l e r content . T a l k a s you were a s t a k e h o l d e r d e s c r i b i n g t h e s y s t e m he wants t o be i m p l e m e n t e d ( e . g . , d u r i n g r e q u i r e m e n t s e l i c i t a t i o n ) .
Here i s t h e README f i l e o f a s o f t w a r e p r o j e c t :
[...]
E x p l a i n i t s p u r p o s e , t h e p r o b l e m i t a d d r e s s e s ( i f m e n t i o n e d ) , and i t s main f u n c t i o n a l i t i e s . Generate a new list of actors System prompt:
You a r e a h e l p f u l a s s i s t a n t e x p e r t i n s o f t w a r e e n g i n e e r i n g t a s k s , s p e c i a l i s e d i n e x t r a c t i n g end − u s e r s r o l e s from a h i g h l e v e l d e s c r i p t i o n o f a s o f t w a r e p r o j e c t . Your t a s k i s t o e x t r a c t t h e a c t o r s ( r o l e s o f end u s e r s o f t h e s y s t e m ) from t h e g i v e n d e s c r i p t i o n . I f a c t o r s a r e n o t e x p l i c i t l y mentioned , i n f e r them b a s e d on t y p i c a l u s e r s o f s i m i l a r s o f t w a r e s y s t e m s . Each e x t r a c t e d a c t o r name s h o u l d be a c c o m p a n i e d by a v e r y s h o r t d e s c r i p t i o n .
Prompt:
Now e x t r a c t t h e a c t o r s ( r o l e s o f end u s e r s ) from t h e f o l l o w i n g s o f t w a r e d e s c r i p t i o n . ∗∗ Description : ∗ ∗ { description } Generate a new list of high-level goals
System prompt:
You a r e a h e l p f u l a s s i s t a n t e x p e r t i n s o f t w a r e e n g i n e e r i n g t a s k s . You ' r e t a s k e d t o e x t r a c t high − l e v e l g o a l s from a s o f t w a r e d e s c r i p t i o n f o r e a c h p r o v i d e d a c t o r t h a t i s e x p e c t e d t o i n t e r a c t w i t h t h e s o f t w a r e . F o l l o w i n g t h e Goal − O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) frameworks , high − l e v e l g o a l s a r e s t r a t e g i c o b j e c t i v e s t h a t d e f i n e t h e ' why ' b e h i n d a s y s t e m . They a r e u s u a l l y a b s t r a c t , b u s i n e s s − o r i e n t e d , and i n d e p e n d e n t o f t e c h n i c a l i m p l e m e n t a t i o n . They r e p r e s e n t t h e n e e d s o f s t a k e h o l d e r s o r t h e o r g a n i z a t i o n . F o c u s : V i s i o n and j u s t i f i c a t i o n . G e n e r a t e ONLY t h e f u n c t i o n a l g o a l s .
Prompt:
B a s e d on your u n d e r s t a n d i n g o f t h e t y p i c a l n e e d s and i n t e r e s t s o f t h e f o l l o w i n g a c t o r s i n t h e f o l l o w i n g software project , help generate a l i s t of h i g l l e v e l goals . ∗∗ Description : ∗ ∗ { project_description } ∗ ∗ Actors : ∗ ∗ { actors } Generate a new list of low-level goals
System prompt:
You a r e a h e l p f u l a s s i s t a n t e x p e r t i n s o f t w a r e e n g i n e e r i n g t a s k s . E l i c i t low − l e v e l g o a l s f o r a s p e c i f i c s t a k e h o l d e r i n a s o f t w a r e p r o j e c t . Avoid g e n e r i c g o a l s . I n s t e a d , b r e a k them down i n t o a t o m i c a c t i o n s l i n k e d to system c a p a b i l i t i e s . Don ' t be t o o g e n e r i c , f o r example , a v o i d g o a l s l i k e ' make t h e s o f t w a r e f a s t ' , ' d e v e l o p a web i n t e r f a c e ' e t c . F o l l o w i n g t h e Goal − O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) framework , low − l e v e l g o a l s a r e t e c h n i c a l o b j e c t i v e s t h a t d e s c r i b e ' how ' t h e high − l e v e l g o a l s w i l l be a c h i e v e d . They a r e more c o n c r e t e and a r e e v e n t u a l l y r e f i n e d i n t o s p e c i f i c r e q u i r e m e n t s o r s o f t w a r e s p e c i f i c a t i o n s . F o c u s : I m p l e m e n t a t i o n and c o n s t r a i n t s . G e n e r a t e ONLY t h e f u n c t i o n a l g o a l s .
Prompt:
B a s e d on your u n d e r s t a n d i n g o f t h e t y p i c a l t a s k s t h a t compose t h e f o l l o w i n g s e q u e n c e o f high − l e v e l g o a l s , p r o v i d e i f p o s s i b l e a d e c o m p o s i t i o n o f g o a l s i n t o sub − g o a l s . Each low − l e v e l g o a l s h o u l d t h e o r e t i c a l l y c o r r e s p o n d to a s i n g l e a c t i o n of the a c t o r with the s o f t w a r e . ∗ ∗ High − l e v e l g o a l s : ∗ ∗ { h i g h L e v e l G o a l s } Critique the response previously generated
System prompt:
You ' r e an h e l p f u l a s s i s t a n t , e x p e r t i n t h e f i e l d o f s o f t w a r e e n g i n e e r i n g .
Prompt:
You ' r e an h e l p f u l a s s i s t a n t , e x p e r t i n t h e f i e l d o f s o f t w a r e e n g i n e e r i n g and s p e c i a l i s e d i n t h e Goal − O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) framework . F o l l o w i n g t h e Goal − O r i e n t e d R e q u i r e m e n t s E n g i n e e r i n g ( GORE ) framework : − an a c t o r i s a c t i v e e n t i t y t h a t h a s t h e c a p a b i l i t y t o p e r f o r m a c t i o n s t o a c h i e v e g o a l s . U n l i k e g o a l s , which a r e ' what ' o r ' why , ' a c t o r s a r e t h e ' who . ' − high − l e v e l g o a l s a r e s t r a t e g i c o b j e c t i v e s t h a t d e f i n e t h e ' why ' b e h i n d a s y s t e m . They a r e u s u a l l y a b s t r a c t , b u s i n e s s − o r i e n t e d , and i n d e p e n d e n t o f t e c h n i c a l i m p l e m e n t a t i o n . They r e p r e s e n t t h e n e e d s o f s t a k e h o l d e r s o r t h e o r g a n i z a t i o n . F o c u s : V i s i o n and j u s t i f i c a t i o n . − low − l e v e l g o a l s a r e t e c h n i c a l o b j e c t i v e s t h a t d e s c r i b e ' how ' t h e high − l e v e l g o a l s w i l l be a c h i e v e d . They a r e more c o n c r e t e and a r e e v e n t u a l l y r e f i n e d i n t o s p e c i f i c r e q u i r e m e n t s o r s o f t w a r e s p e c i f i c a t i o n s . F o c u s : I m p l e m e n t a t i o n and c o n s t r a i n t s . You can p r o p o s e new g o a l s t a k i n g i n t o a c c o u n t t h e a l r e a d y p r e s e n t o n e s . C o n s i d e r t h a t high − l e v e l g o a l s o f t e n answer t h e WHY q u e s t i o n , w h i l e low − l e v e l g o a l s o f t e n a d d r e s s t h e HOW. You must e n s u r e t h a t ONLY f u n c t i o n a l g o a l s a r e p r e s e n t . ∗∗ Description : ∗ ∗ { description } ∗ ∗ Actors : ∗ ∗ { actors }
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 4: Exemplars of Few-shot examples used either for prompting the ’generator’ or the ’critic’ LLM. Other instances can be found in the online appendix. Generation of high-level goals − D e s c r i p t i o n : CatWatch i s a web a p p l i c a t i o n t h a t t r a c k s and s t o r e s GitHub s t a t i s t i c s f o r a c c o u n t s . I t p r o v i d e s p r o j e c t p o p u l a r i t y and c o n t r i b u t o r d a t a t h r o u g h a REST API , o f f e r i n g a g g r e g a t e d s t a t s . − A c t o r s : [ { name : " GitHub a c c o u n t " , d e s c r : " I n d i v i d u a l s who own GitHub a c c o u n t s " } ] ∗ ∗ ∗ Output : ∗ ∗ ∗ [ " The s t a k e h o l d e r aims t o e f f o r t l e s s l y m o n i t o r t h e p o p u l a r i t y m e t r i c s o f t h e i r open s o u r c e p r o j e c t s a c r o s s v a r i o u s GitHub a c c o u n t s u s i n g CatWatch . " , " The s t a k e h o l d e r s e e k s a f e a t u r e t h a t h i g h l i g h t s t h e most a c t i v e c o n t r i b u t o r s and c o l l a b o r a t o r s i n t h e i r GitHub r e p o s i t o r i e s t h r o u g h CatWatch . " , " The s t a k e h o l d e r d e s i r e s a n o t i f i c a t i o n s y s t e m w i t h i n CatWatch t h a t a l e r t s them p r o m p t l y a b o u t s i g n i f i c a n t a c t i v i t i e s , s u c h a s new c o n t r i b u t i o n s o r r i s i n g p r o j e c t t r e n d s " , " The s t a k e h o l d e r i n s i s t s on CatWatch i m p l e m e n t i n g r o b u s t d a t a s e c u r i t y m e a s u r e s and c o m p l i a n c e w i t h p r i v a c y s t a n d a r d s t o s a f e g u a r d t h e i r GitHub a c c o u n t i n f o r m a t i o n . " , " The s t a k e h o l d e r r e q u i r e s CatWatch t o s e a m l e s s l y i n t e g r a t e w i t h t h e i r e x i s t i n g workflow t o o l s and d e v e l o p m e n t e n v i r o n m e n t s , e n h a n c i n g p r o d u c t i v i t y and u s e r e x p e r i e n c e . " , " The s t a k e h o l d e r aims t o a c c e s s d e t a i l e d a n a l y t i c s and r e p o r t s g e n e r a t e d by CatWatch , o f f e r i n g i n s i g h t s i n t o p r o j e c t p e r f o r m a n c e , community engagement , and o t h e r r e l e v a n t m e t r i c s . " ] Critic of high-level goals − D e s c r i p t i o n : CatWatch i s a web a p p l i c a t i o n t h a t t r a c k s and s t o r e s GitHub s t a t i s t i c s f o r a c c o u n t s . I t p r o v i d e s p r o j e c t p o p u l a r i t y and c o n t r i b u t o r d a t a t h r o u g h a REST API , o f f e r i n g a g g r e g a t e d s t a t s . − A c t o r s : [ { name : " GitHub a c c o u n t " , d e s c r : " I n d i v i d u a l s who own GitHub a c c o u n t s " } ] − High − l e v e l G o a l s : [ " The s t a k e h o l d e r aims t o e f f o r t l e s s l y m o n i t o r t h e p o p u l a r i t y m e t r i c s o f t h e i r open s o u r c e p r o j e c t s a c r o s s v a r i o u s GitHub a c c o u n t s u s i n g CatWatch . " , " The s t a k e h o l d e r s e e k s a f e a t u r e t h a t h i g h l i g h t s t h e most a c t i v e c o n t r i b u t o r s and c o l l a b o r a t o r s i n t h e i r GitHub r e p o s i t o r i e s t h r o u g h CatWatch . " , " The s t a k e h o l d e r wants t o u s e t h e s y s t e m t o t r a c k t h e h e a l t h , f e e d i n g s c h e d u l e s , and GPS l o c a t i o n s o f a c t u a l c a t s i n a r e s c u e s h e l t e r . " , " The s t a k e h o l d e r i n s i s t s on CatWatch i m p l e m e n t i n g r o b u s t d a t a s e c u r i t y m e a s u r e s and c o m p l i a n c e w i t h p r i v a c y s t a n d a r d s t o s a f e g u a r d t h e i r GitHub a c c o u n t i n f o r m a t i o n . " , " The s t a k e h o l d e r r e q u i r e s CatWatch t o s e a m l e s s l y i n t e g r a t e w i t h t h e i r e x i s t i n g workflow t o o l s and d e v e l o p m e n t e n v i r o n m e n t s , e n h a n c i n g p r o d u c t i v i t y and u s e r e x p e r i e n c e . " , " The s t a k e h o l d e r aims t o a c c e s s d e t a i l e d a n a l y t i c s and r e p o r t s g e n e r a t e d by CatWatch , o f f e r i n g i n s i g h t s i n t o p r o j e c t p e r f o r m a n c e , community engagement , and o t h e r r e l e v a n t m e t r i c s . " ] ∗ ∗ ∗ Score : ∗ ∗ ∗ 3/10 ∗ ∗ ∗ Comment : ∗ ∗ ∗ Out o f c o n t e x t . D e s p i t e t h e name " CatWatch , " t h e goal regarding tracking physical cats i s completely unrelated .
4.2
Experiment Setup
To evaluate the proposed architecture, we selected four samples of software projects, as reported in Table 5. Future work may experiment with more datasets. We selected two enterprise applications from the WFD (formerly EMB) dataset [4]. This dataset includes software projects with corresponding READMEs and Swagger-formatted API documentation. Our focus is on GestaoHospital7 , a public health management system, and GenomeNexus8 , which automates the annotation of cancer-related genetic variants. These projects constitute good examples of commissioned applications involving software engineering activities. 7 https://github.com/ValchanOficial/GestaoHospital 8 https://github.com/WebFuzzing/Dataset/tree/master/jdk_8_maven/cs/rest-
gui/genome-nexus
Arnaudo et al.
Table 5: Number of annotations present in the ground truth datasets adopted in this study. HL = High-level, LL = Lowlevel dataset
Actors
HL Goals
LL Goals
GestaoHospital GenomeNexus Urban Maintenance London Ambulance System
5 5 6 4
4 9 9 2
20 34 18 10
Total
20
24
82
Table 6: Example of output produced by the proposed system when processing the Urban Maintenance use case under the Few-shot setting. Red text represents false positives, while blue highlights false negatives. Examples of false positives are the high-level goals related to account registration and management. While low-level goals have not been reported for brevity, we observed that the relative low-level goals have not been consistently generated. Actors "Citizens", "Municipal Operators", "External Maintenance Personnel", "Administrators", "Non-registered Users", "Technical Office Staff", "Organizational Office Staff" High-level goals "Citizens should be able to submit reports by selecting locations on a map and providing necessary details, ensuring their voices are heard in urban management.", "Municipal operators need tools to review, approve, and manage reports effectively, ensuring timely responses to urban issues.", "External maintenance personnel should be able to receive assignments, update report statuses, and communicate with municipal operators to resolve issues efficiently.", "Administrators require access to both public and private statistics to monitor system performance and citizen engagement effectively.", "Non-registered users should be able to access public reports and statistics to foster trust and engagement with the municipality’s initiatives.", "The system should allow citizens to submit reports anonymously, ensuring their identity is not disclosed in public listings.", "Citizens should be able to create a new account through registration.", "Registered users should be able to manage their account information", "Registered users should be able to start following a report to receive timely notifications", "Administrators should be able to configure the auto-assignment for report categories., "External maintenance users should be able to communicate with the municipal staff, and eventually manage a report’s lifecycle through the platform."
Furthermore, we examined the London Ambulance Service case study, a seminal exemplar frequently cited within the GORE literature [14, 22]. The associated annotations were synthesised from the extant body of research. Finally, we included a software project sourced from an university course, describing an urban maintenance ticketing system. The relative annotations were manually curated by subject-matter experts within the teaching faculty. The ground truth was manually curated by three of the authors of this paper, producing the annotations described in Table 5 and available in the replication package.
Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 7: Results for the tasks described in Section 3.2, performed in Zero-Shot (ZS), One-Shot (OS), or Few-shot (FS) setting. Actors
Prec. Recall F1
5
High-Level Goals
However, analogous patterns cannot be found when considering neither system’s recall nor F1-score. Similarly, the cosine similarities between the case studies forming the ground truth and the Few-Shot examples used in GPT prompting seem not to be involved.
Low-Level Goals
ZS
OS
FS
ZS
OS
FS
ZS
OS
FS
0.75 0.78 0.76
0.68 0.80 0.74
0.78 0.67 0.72
0.63 0.61 0.62
0.57 0.60 0.59
0.63 0.59 0.61
0.78 0.51 0.61
0.72 0.49 0.59
0.77 0.45 0.57
Table 8: Performance metrics for the Actor identification task using Few-shot prompting, disaggregated by the individual software projects within the ground truth and ordered by precision. Dataset Genome Nexus Urban Maintenance London Ambulance Service Gestao Hospital
Results and Discussion
All the measurements obtained during the evaluation are arranged in Table 7, while a qualitative evaluation is reported in Table 6.
5.1
Extraction of Actors
As summarised in Table 7, the Zero-shot setting achieved the best F1-score — of about 0.76 — in actor extraction. At the same time, One-shot and Few-shot obtained comparable values for this metric. One-shot is associated with the highest recall (0.80), while Fewshot seem to favour precision — achieving the peak value of 0.78. This scenario highlights that the absence of Shot examples leads to the best trade-off between precision and recall, but the system’s behaviour can be shifted by applying different Shot-prompting strategies. To assess the impact of the refinement loop described in Section 3.1 — the core innovation of our architecture — Table 9 reports results from a pipeline using only the GPT model without any feedback mechanism. The comparison reveals that Actor Extraction does not benefit from the feedback loop. Although the observed differences are marginal — never exceeding three percentage points — the F1-scores achieved during the ablation study consistently outperform those of the full architecture across all prompting configurations, reaching a peak of 0.78 in the Zero-shot setting. To further evaluate the impact of the Few-shot prompts, Table 8 details the performance metrics across the ground truth case studies (already described in Section 4.2). Notably, in the Genome Nexus case the system achieved perfect precision, albeit with a limited recall of approximately 0.40. This precision peak aligns with Tables 1 and 2, which identify this dataset as having the highest cosine similarity to the Few-shot exemplars utilised in both GPT and Llama prompts. Regarding the Urban Maintenance dataset — which yielded the second-highest precision and a substantial recall of 0.70 — the average cosine similarity does not rank second for GPT prompts; however, it does hold the second-highest position for Llama-based prompts (Table 2). A consistent pattern emerges for the Gestao Hospital and London Ambulance Service datasets, which rank third and fourth in precision, respectively. Their precision scores correlate proportionally with the average cosine similarities observed in the Llama prompts. This suggests that the order of precision is preserved specifically in relation to the Llama Few-shot similarities, potentially indicating a direct correlation between the diversity of exemplars provided to the critic agent and the system’s overall precision.
Recall 0.40 0.70 0.76 0.85
Precision 1.00 0.84 0.51 0.71
F1-score 0.57 0.77 0.61 0.77
Answer to RQ1: Our evaluation reported an F1-Score of 0.76 in the optimal prompting configuration — specifically Zero-shot. This indicates that our architecture takes limited benefit from in-context learning. Furthermore, a distinct trade-off is evident between the maximum precision of 0.78 achieved via Few-shot and the maximum recall of 0.80 associated with One-shot prompting. The Llama ablation study yielded a slightly superior F1-score of 0.78 — associated with Zero-shot again. This suggests that the generator agent — i.e., GPT — is the primary responsible of the suboptimal exploitation of the Shot examples. Finally, we found a possible correlation between the precision in Actor extraction and the cosine similarity between the software project’s description and the Few-Shot examples provided to the critic agent — i.e., the Llama model. This correlation suggests that the Few-shot strategy applied to Llama should be enriched to improve the critic mechanism’s effectiveness.
5.2
Extraction of High-Level Goals
As reported in Table 7, high-level goal extraction demonstrates a distinct behavioural pattern across prompting settings compared to actor extraction. The Zero-shot strategy yielded the optimal overall F1-score, precision, and recall — recorded as 0.62, 0.63, and 0.61, respectively. These findings confirm the detrimental impact of Shot-prompting if integrated with the feedback mechanism. However, in the results of the Llama ablation study — reported in Table 9 — the opposite trend can be observed, with the supremacy of Few-shot prompting. This may suggest that — for what concerns the task of identifying high-level goals — the GPT model takes advantage from Shot examples, but this effect is neutralised by the critique mechanism. Table 9: Results for the Llama’s feedback ablation, performed in Zero-Shot (ZS), One-Shot (OS), or Few-shot (FS) setting. Actors
Prec. Recall F1
High-Level Goals
Low-Level Goals
ZS
OS
FS
ZS
OS
FS
ZS
OS
FS
0.80 0.77 0.78
0.80 0.74 0.77
0.86 0.67 0.75
0.65 0.46 0.54
0.66 0.46 0.54
0.68 0.53 0.60
0.79 0.38 0.51
0.76 0.36 0.50
0.79 0.40 0.53
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Arnaudo et al.
In high-level goals extraction, the contribution of Llama proves to be slightly beneficial: during the ablation study, the maximum F1score reached was 0.60, which is two points below the one relative to the complete architecture. These results may indicate that, while Few-shot prompting improves the ’generator’ model alone, high-level goals extraction takes greater advantage by the introduction of the feedback mechanism. Finally, in Table 10 we report the performance achieved by the complete architecture divided by case study, when Few-shot is applied. As was observed for the actors extraction task, a proportional relation — albeit not linear — can be found when comparing the precision metric achieved in each case study and the cosine similarities reported in Table 2 — which have been computed between the case studies documentation and the few Shot examples fed to the Llama model. This may suggest the primary bottleneck resided in the quality of the Few-Shot examples provided to the ’critic’ agent, and not to the ’generator’ model. Consistently with the results presented in Section 5.1, we did not observe any correlation between the similarity to the GPT’s Few-shot examples of each case study and the respective performance achieved by the system.
Table 10: Performance metrics for the high-level goals identification task using Few-shot prompting, disaggregated by the individual software projects within the ground truth and ordered by precision. Dataset Genome Nexus London Ambulance Service Urban Maintenance Gestao Hospital
Recall 0.51 0.75 0.53 0.81
Precision 0.76 0.21 0.69 0.46
F1-score 0.61 0.33 0.60 0.59
Answer to RQ2: The maximum F1-score reached by our architecture for the extraction of high-level goals is 0.62 (Zero-shot). The ablation of the critique mechanism reveals an opposite trend, with Few-shot achieving the highest F1-score — which was 0.60. This may suggest that the performance improvements achieved by the addition of the feedback loop overcome the ones related to the application of Few-shot to the GPT model alone. The performance with the feedback mechanism during Few-shot learning shows a specific trend: precision correlates proportionally — though not linearly — with the cosine similarity between the case studies and the examples provided to the Llama model. This pattern consistently mirrors the results previously observed during the actors extraction task, possibly indicating that the primary bottleneck resided in the quality of the in-context examples provided to the ’critic’ model.
5.3
Extraction of Low-Level Goals
As shown in Table 7, low-level Goal extraction yields a F1-score slightly lower than high-level goals extraction (0.61 versus 0.62). This can indicate that the errors may have propagated up to this point in the pipeline, effectively establishing a performance ceiling. Indeed, as detailed below, it is possible to confirm some patterns
already found when analysing low-level goals extraction in Section 5.2. In the Zero-Shot setting, the system achieves the best recall (0.78), precision (0.51), and F1-score (0.61). At the same time, Few-shot proves to be the best strategy in the absence of the feedback mechanism: in the results of the Llama’s ablation study — reported in Table 9 — Few-shot is associated with a great precision (0.78), a low recall (0.40), and the highest F1-score (0.53). These values confirm the phenomena already observed in the extraction of high-level goals, suggesting that the introduction of the feedback mechanism yields greater benefits than the application of Few-shot prompting to the GPT model alone. The iterative interaction between GPT and Llama failed to exploit Shot-prompting, which suggests that the primary bottleneck resides within the prompting of the Llama model. Following the established methodology, we compared the cosine similarities between the case studies and the Llama’s Few-shot exemplars — detailed in Table 2 — against the performance metrics achieved by the architecture in the Few-shot setting for each corresponding case study (Table 11). Differently from the previous two sections, we were unable to find any correlation pattern. This may suggest that the errors made during the extraction of high-level goals may have introduced perturbations in this final step. As can be seen by the prompts in Table 3, the models are tasked to extract low-level goals by starting only from the high-level ones. Future work may investigate this effect more deeply by isolating the generation of low-level goals.
Table 11: Performance metrics for the Low-level goals identification task using Few-shot prompting, disaggregated by the individual software projects within the ground truth and ordered by precision. Dataset Genome Nexus Gestao Hospital Urban Maintenance London Ambulance Service
Recall 0.25 0.56 0.61 0.68
Precision 0.85 0.76 0.69 0.68
F1-score 0.39 0.64 0.65 0.68
Answer to RQ3: The maximum F1-Score obtained for low-Level Goals extraction is 0.61 (Zero-shot). This value is slightly lower than the one achieved for the previous task in the pipeline, suggesting that the propagation of errors may have introduced performance ceiling. By removing the feedback loop from the architecture, the system achieves a maximum F1-score of 0.53 in the Few-shot setting. While this indicates that the introduction of the critic agent delivers a tangible advantage, it confirms that this benefit comes at the cost of neutralising the effects of in-context, as this phenomenon was already observed in the high-level goals extraction.
5.4
API Mapping
Given the limited F1-scores observed in the preceding phases, this component is considered exploratory rather than part of the
Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Table 12: Examples of generated API mappings High Level Goal Name Manage Healthcare Operations Manage Healthcare Operations
Low Level Goal Name Register a new hospital with essential details, including name, address, and contact information. Retrieve a list of all registered hospitals with their registration status and details.
API Name insertUsingPOST findAllUsingGET
formal evaluation. In Table 12, we therefore provide only a qualitative illustration of how such a mapping might look when applied to the extracted goals. These mappings should be interpreted as plausible candidates generated by the model rather than validated correspondences. Nevertheless, following a manual review of the project’s API documentation, the authors regard them as substantively sound. This provides preliminary insight into how such a component could support analysts during design or requirements traceability tasks.
the maximum number of iterations of the generation-critique loop (both defined in Section 3.1). Furthermore, our method stipulates that the final response produced by GPT-4 is retained once the maximum number of iterations has been reached. Although retaining the final iteration may be suboptimal compared to selecting the highest-scoring response, Llama’s feedback effectively mitigates quality degradation across successive outputs.
6 Limitations 6.1 Bias
6.4
Some bias may have been introduced through the examples employed in the in-context learning. Specifically, if the examples used were more closely aligned with the target problem, the results could have been biased towards higher performance without actually being the consequence of better design choices. This could be mitigated by expanding the evaluation to further benchmarks, or by introducing RAG-augmented Few-shot prompting.9 Moreover, a potential source of bias lies in the input software documents (Section 3.2). As highlighted in the prompt engineering literature [10, 17], the clarity and quality of information contained within input prompts are crucial determinants, and the adopted preprocessing strategy may not be enough for bare README files. Future work may assess the impact of feeding the system with more extensive documentation.
6.2
Threats to Construct Validity
In the present study, README files have been used as a proxy for preliminary documents from which the requirements might be elicited. To provide more realistic application scenarios, the approach should be evaluated with natural language requirements (e.g., transcripts of interviews with stakeholders) rather than README files. Furthermore, the multi-step pipeline enforces a ’waterfall’ methodology that precludes the modification of high-level goals during the elicitation of low-level objectives. This contrasts with the established literature [21], which advocates for the late-stage discovery of high-level goals through obstacle analysis and by addressing ’why’ queries relative to low-level goals. Future work may enhance architectural flexibility by integrating these additional mechanisms.
6.3
Threats to Internal Validity
We acknowledge that our study did not answer the proposed research questions exhaustively. Indeed, more experiments could be performed by varying the values of the Quality Threshold and 9 Retrieval Augmented Generation (RAG) with Few-shot prompting refers to the use
of a RAG system to retrieve the most pertinent Few-Shot examples at inference time, based on their similarity with the system’s current input.
Threats to External Validity
As previously stated, we concentrated solely on functional goals. While this limitation was imposed to keep the number of goals per use case manageable for manual annotation, we acknowledge that it may limit the generalisability of our results to real-world scenarios where non-functional goals are critical. Finally, it is worth noting that the reported results are highly dependent on the chosen architecture, the specific models employed, and the datasets used to validate the approach. Although they offer valuable insights into the capabilities of LLMs, these findings may not be generalisable to other combinations of models or alternative configurations (e.g., changing the number of max iterations) within a processing chain.
7
Conclusion and Future Work
We presented a semi-structured approach for automating parts of the RE process, assessing the impact of in-context learning, and exploiting models with diverging base knowledge for refining the outputs through an iterative feedback mechanism. However, the results were not entirely satisfactory, with an F1-score of 61% in low-level goals extraction — the last step of the pipeline. We acknowledge that the values observed are insufficient for fully automated use and would still require substantial manual supervision in practice. We therefore view the proposed approach not as a replacement for human annotation, but as a starting point that can assist and accelerate manual extraction. Improving recall is a key direction for future work, and systematic comparison with human recall on the same data would provide a more meaningful upper bound and evaluation target. Crucially, the ablation study — conducted by removing the Llamabased feedback mechanism — demonstrated that the proposed architecture yields consistent advantages over a conventional, linear pipeline utilising GPT in isolation. Moreover, we were able to demonstrate that our feedback-loop mechanism with Zero-shot prompting outperforms Few-shot prompting applied to the GPT model alone. Measurements of the cosine similarity between the case study descriptions and the shot examples provided to the GPT model revealed no correlation with performance per case study. This suggests a balanced variety among the in-context examples.
EASE 2026, 9–12 June, 2026, Glasgow, Scotland, United Kingdom
Conversely, analysis of the Few-shot examples used for Llama prompting indicated a potential correlation between their similarity to the case studies and the precision values achieved. This suggests that the quantity and variety of in-context examples provided to the Llama model should be expanded to improve robustness. While our approach may serve as an encouraging starting point for developing more accurate systems, we recognise that there remains substantial room for improvement. Future work may involve conducting repeated runs to assess stability, evaluating additional datasets (Section 4.2), and testing alternative embedding models — such as Sentence BERT — in place of BERT (Section 4.1). In its current form, our method - although iterative - did not involve a human in the loop, which may account for the suboptimal outcomes. Indeed, various studies in the RE literature [9, 13, 23] indicate that reliable results are difficult to achieve without involving humans. Furthermore, Llama’s evaluation process could be enhanced with an increased number of Few-Shot examples, which are fundamental to provide to the model the references to assign the scores. Moreover, the model could be provided with literaturegrounded instructions - drawing from RE and GORE studies - to guide its outputs towards a more informed evaluation, rather than depending exclusively on its internal knowledge and probabilistic reasoning. This enhancement could be realised through the integration of a Retrieval-Augmented Generation (RAG) system and the application of Chain of Thought (CoT) prompting. Additionally, our architecture could be modified to better align to the GORE procedures described in the literature. Specifically, a further loop mechanism should be introduced, encompassing both the high-level and low-level goals extraction phases to enable late discovery of high-level goals. Finally, our study was conducted without imposing constraints on computational resources or processing time. Although we relied on remote API calls to access both the GPT and Llama models, future research could examine in greater depth the computational and economic costs.
References [1] Systems and software engineering – Life cycle processes –Requirements engineering, 2011. ISBN: 9780738165912. [2] Akbar, M. A., Khan, A. A., and Liang, P. Ethical aspects of chatgpt in software engineering research. arXiv preprint arXiv:2306.07557 (2023). [3] Alliance, A. What does INVEST Stand For? | Agile Alliance, Dec. 2015. [4] Arcuri, A., Zhang, M., Golmohammadi, A., Belhadi, A., Duman, O., Seran, S., Galeotti, J. P., and Ghianni, H. WebFuzzing/EMB: v3.4.0, Jan. 2025. [5] Arulmohan, S., Meurs, M.-J., and Mosser, S. Extracting domain models from textual requirements in the era of large language models. In 2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) (2023), IEEE, pp. 580–587. [6] Bozyigit, F., Bardakci, T., Khalilipour, A., Challenger, M., Ramackers, G., Babur, Ö., and Chaudron, M. R. Generating domain models from natural language text using nlp: a benchmark dataset and experimental comparison of tools. Software and Systems Modeling 23, 6 (2024), 1493–1511. [7] Das, S., Deb, N., Cortesi, A., and Chaki, N. Extracting goal models from natural language requirement specifications. Journal of Systems and Software (2024), 111981. [8] De Vito, G., Palomba, F., Gravino, C., Di Martino, S., and Ferrucci, F. Echo: An approach to enhance use case quality exploiting large language models. In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (2023), IEEE, pp. 53–60. [9] Ebrahim, M., Guirguis, S., and Basta, C. Enhancing software requirements engineering with language models and prompting techniques: Insights from the current research and future directions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) (2025), J. Zhao, M. Wang, and Z. Liu, Eds., Association for Computational
Arnaudo et al.
Linguistics, pp. 486–496. [10] Errica, F., Sanvito, D., Siracusano, G., and Bifulco, R. What Did I Do Wrong? Quantifying LLMs’ Sensitivity and Consistency to Prompt Engineering. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (Albuquerque, New Mexico, 2025), Association for Computational Linguistics, pp. 1543–1558. [11] Feldt, R., and Coppola, R. Semantic api alignment: Linking high-level user goals to apis. In 2025 IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE) (2025), IEEE, pp. 17–20. [12] Feldt, R., Kang, S., Yoon, J., and Yoo, S. Towards autonomous testing agents via conversational large language models, 2023. [13] Ferrari, A., and Spoletini, P. Formal requirements engineering and large language models: A two-way roadmap. [14] Finkelstein, A., and Dowell, J. A comedy of errors: the london ambulance service case study. In Proceedings of the 8th International Workshop on Software Specification and Design, IEEE Comput. Soc. Press, pp. 2–4. [15] Kang, S., Yoon, J., and Yoo, S. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) (2023), IEEE, pp. 2312–2323. [16] Kavakli, E. Goal-oriented requirements engineering: A unifying framework. Requirements Engineering 6 (2002), 237–251. [17] Lin, Z. How to write effective prompts for large language models, Sept. 2023. [18] Munkres, J. Algorithms for the assignment and transportation problems. 32–38. [19] Sami, M. A., Waseem, M., Zhang, Z., Rasheed, Z., Systä, K., and Abrahamsson, P. AI based Multiagent Approach for Requirements Elicitation and Analysis. [20] Siddeshwar, V., Alwidian, S., and Makrehchi, M. A comparative study of large language models for goal model extraction. In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems (2024), pp. 253–263. [21] Van Lamsweerde, A. Goal-oriented requirements engineering: A guided tour. In Proceedings fifth ieee international symposium on requirements engineering (2001), IEEE, pp. 249–262. [22] Van Lamsweerde, A., and Letier, E. Handling obstacles in goal-oriented requirements engineering. 978–1005. [23] Vogelsang, A. Prompting the future: Integrating generative LLMs and requirements engineering. [24] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). [25] Yoon, J., Feldt, R., and Yoo, S. Autonomous large language model agents enabling intent-driven mobile gui testing, 2023. [26] Zadenoori, M. A., Dąbrowski, J., Alhoshan, W., Zhao, L., and Ferrari, A. Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review, Sept. 2025. arXiv:2509.11446 [cs]. [27] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. BERTScore: Evaluating text generation with BERT.