Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000. Digital Object Identifier 10.1109/ACCESS.2026.Doi Number
Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation SANTIAGO PEREZ-ACUNA1 , YOD-SAMUEL MARTÍN1 , (Member, IEEE), AND JUAN C. YELMO1
1
Information Processing and Telecommunications Center, ETSI Telecomunicación, Universidad Politécnica de Madrid, 28040 Madrid, Spain
Corresponding author: Yod-Samuel Martín (e-mail: [email protected]).
arXiv:2609.10316v1 [cs.SE] 9 Sep 2026
This work was supported in part by the Spanish Ministry of Science and Innovation through the project Re-InITS (Sistemas informáticos industriales confiables en la era de la computación en el continuo) under Grant PID2024-155230OB-C43.
ABSTRACT Translating high-level controls from security standards into concrete, system-specific requirements is central to cybersecurity requirements engineering. Large language models (LLMs) can accelerate this labor-intensive, recall-sensitive task, but any single run is unreliable: it misses valid safeguards while introducing plausible hallucinations, and outputs shift across runs and models. We reframe this variability as a resource: rather than selecting one output, we study post-generation ensembling, aggregating stochastic runs with information-retrieval data-fusion operators. We propose two strategies: Uniform fusion rewards mere cross-run agreement, whereas Naive-Bayes fusion weights each run by its estimated reliability. We evaluate both over 24 runs from 12 configurations across four model families, generated for ten ISO/IEC 27002:2022 controls and expert-judged against a gold standard of 72 valid requirements. Pooling every run’s output recovers all 72—whereas single configurations recover on average under half—but also 111 hallucinations. Fusion separates the wheat from the chaff, ranking valid requirements well ahead of hallucinations. In the areas under the precision–recall and ROC curves, Uniform fusion alone largely surpasses every original run and configuration—by 0.142 and 0.118 over the best configuration. Naive-Bayes weighting adds a further 0.039 and 0.052, reaching 0.864 and 0.869 while attaining useful operating points earlier. Internal validation confirms the stability of these gains: they stay positive in at least 92% of out-of-bag bootstrap resamples and every structured-perturbation sample. Post-generation fusion thus turns apparent noise into a practical asset: a lightweight layer giving analysts broader coverage and a better prioritized review queue—from affordable, below-frontier models alone. INDEX TERMS Computer security, ensemble learning, generative AI, requirements engineering, security management
I. INTRODUCTION
L
ARGE language models (LLMs) are beginning to be explored as assistants for drafting and refining softwarerequirements specifications—a task that remains both linguistically demanding and labor-intensive. Their promise is particularly compelling in cybersecurity where, abiding by security-by-design principles, engineers must translate high-level control definitions from standards such as ISO/IEC 27002 into concrete, system-specific requirements. In this setting, LLMs can relieve analysts working under time and budget constraints of much of the drafting effort— provided that the output they must then review remains manageable. A central challenge, however, is the diversity of LLM outputs, which stems from both stochasticity and inter-model variability. Sampling noise causes run-to-run differences even
VOLUME 14, 2026
under a fixed model and prompting scheme, and models, prompting styles, and decoding settings add further variability across configurations. In compliance-relevant settings, this diversity matters because requirements engineering (RE) is often recall sensitive: failing to surface a relevant requirement can leave protection gaps that may not be revisited later, whereas nonrelevant (but plausible) requirements are typically easier to flag during review. Consequently, practical adoption favors methods that improve coverage while keeping the growth in invalid requirements contained. Most empirical studies of LLM-assisted requirements engineering still evaluate single runs only. While this practice is reasonable for early benchmarking, it sits uneasily with the variability described above and, in particular, overlooks two salient properties of LLM behavior: stochastic diversity across runs and complementary differences across model 1
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
families and prompt designs. Recent surveys and position papers across software engineering and RE repeatedly flag stochasticity and prompt sensitivity as threats to reproducibility [1], [2], [3], yet evaluation protocols rarely account for run-to-run variability explicitly. Likewise, in our previous work [4], which evaluated several multi-step LLM pipelines instantiating ISO/IEC 27002 controls for an LLM-evaluation-focused testbed system, we observed substantial variability across both configurations and repeated runs, and many hallucinations proved not systematic but tied to specific models or runs. Taken together, these observations motivate a shift in perspective: instead of treating diversity purely as an evaluation nuisance, we treat it as a source of additional candidate requirements that, when combined appropriately, can improve recall while keeping the resulting increase in invalid requirements manageable. This shift also has an economic dimension: if aggregation compensates for the limitations of any individual run, strong coverage need not depend on a single frontier model, making an ensemble of more modest—cheaper or locally operable— models an attractive alternative; and because aggregation also ranks candidates by their support, it can reach the same coverage with less review effort. We therefore examine ensembling strictly as a postgeneration strategy. Rather than proposing new prompting pipelines or regenerating requirements, we reuse a frozen human-labeled corpus and treat each LLM run as a blackbox retrieval system that returns an unranked set of candidate requirements for each cybersecurity control. Following classical information retrieval (IR) theory, we model ensembling as a data-fusion problem over requirement sets. Our methods include voting-based aggregation, which rewards cross-run agreement, and quality-aware weighting derived from fixedparameter Naive-Bayes over run votes, estimating modelfamily reliability from repeated stochastic outputs. We study these ensembling strategies over 12 configurations spanning four model families (Llama 3.1 405B, Qwen-2 72B, Mixtral 8×22B, and GPT-4 Turbo), including multiple stochastic runs of the base generation pipelines. Ensembles are evaluated against a gold standard (see Section III) derived from expert annotation of the ISO/IEC 27002 instantiated requirements. This research makes the following key contributions: 1) Framing post-generation ensembling as a practical response, integrable into established security-RE processes, to the stochasticity and inter-model variability of LLM outputs: treating multi-run and multiconfiguration outputs as complementary signals rather than evaluation noise, and showing that fusing them acts on the order of review and not merely its volume— even plain support-based fusion prioritizes better supported candidates, raising recall without a commensurate rise in hallucinations. 2) Introducing a weighting scheme that treats runs as noisy observers of requirement validity, scaling each run’s votes by its model family’s reliability—a fixed2
parameter Naive-Bayes. By discounting support from less reliable sources, it refines this ordering and reaches useful operating points at shallower review depths. 3) Evaluating the resulting rankings with information retrieval data-fusion metrics that expose the trade-off between recall and hallucination burden across review depths, so that the fusion strategies can be compared at any compliance-relevant operating point rather than at a single fixed cut. The remainder of the article reviews related work (Section II); describes the materials and methods underpinning our ensembling strategies (Section III), the strategies themselves (Section IV), and the evaluation protocol (Section V); and presents the results (Section VI), their discussion (Section VII), and the conclusions (Section VIII). II. RELATED WORK
Our work sits at the intersection of three research threads, reviewed in turn below: 1) the emerging use of large language models for requirements engineering (RE); 2) LLMbased support for defensive cybersecurity, the landscape in which our target artifact class—cybersecurity requirements— is situated; and 3) ensemble and data-fusion techniques for combining model runs or configurations, from which our approach draws its machinery. Our previous research [4] highlighted two concerns that also recur in recent surveys on LLMs in software engineering and in RE: substantial output variance—across repeated runs and across models—and nontrivial hallucination rates [1], [5], [2]. The present article turns that variance into the input of an ensembling step: each LLM run is treated as a black-box retrieval system, where voting-based data fusion is applied across runs and configurations. A. LLMS FOR REQUIREMENTS ENGINEERING
Recent systematic reviews agree that LLM use in RE is promising but still immature: most studies remain researchprototype evaluations with scarce industrial deployment [5], [6]. Hemmat et al. survey natural language processing (NLP)and LLM-based approaches across the RE lifecycle and find practical, real-world integration of LLMs in RE still underexplored [5]. A complementary review by Zadenoori et al. catalogs 74 LLM-for-RE primary studies published in 2023– 2024, which rely mostly on GPT-style models, are usually evaluated in controlled environments, and are seldom integrated into complex workflows [6]. Beyond this immaturity, the literature voices concerns about the technology itself. Cheng et al. focus on generative artificial intelligence (AI) for RE, reviewing 238 articles published between 2019 and 2025 and highlighting reproducibility, controllability, and governance (including safetyand security-relevant concerns) as recurrent unresolved problems [2]. Sallou et al. examine the threats that LLM use poses to software-engineering research itself—including output variability and the limited reproducibility of closed, evolving models [3]. VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
These concerns materialize as practical obstacles. Norheim et al. review LLM applications to engineering-requirements tasks and report that, although LLMs can assist with structuring, reformulating, and checking requirements, practical use is constrained by recurring challenges: limited requirementsspecific datasets, inconsistent data annotation, inadequately defined RE use cases, and adoption hurdles in engineering practice [7]. Practitioner-oriented discussions add that fluent output can create a misleading veneer of correctness, potentially leading to downstream errors and costly rework that offsets automation benefits [8]. In security work, such residual flaws feed a familiar cost dynamic: security risk left in the software accumulates as security debt that must later be identified, managed, and repaid [9]. A conclusion that recurs across these studies is that human oversight remains necessary. Krishna et al. evaluate LLM assistance for drafting and revising software requirements specifications, finding that LLM drafts can be a productivity aid, though still short of expert-level output [10]. In the same vein, a role-prompted ChatGPT interview eliciting the requirements of a digital twin recovered about 83% of the domain expert’s reference elements while also producing items the expert marked as misinformation [11]. Hymel and Johnson compare LLM-generated requirements against those of human experts and find the LLM output rated at least as aligned and complete as the experts’; even so, they conclude that humans remain essential for domain knowledge, contextual understanding, and nuanced stakeholder needs, with their role shifting toward orchestration and oversight [12]. A further strand goes beyond oversight and casts LLMs and human engineers as collaborators, specifically around formal requirements: LLMs suggesting candidate formalizations whose soundness is then checked by formal verification [13], or translating informal natural-language requirements into verifiable formal specifications and proofs [14]. More broadly, across these RE-focused studies, nearly all investigations still treat LLM output from a single run or configuration as the evaluation unit. The surveys and position papers reviewed above explicitly flag stochasticity and sampling sensitivity as threats to reproducibility [2], [3], yet there is comparatively little work on turning this variance into an asset—for instance, by combining multiple runs or models to increase coverage. Our previous study documented precisely this untapped complementarity in a security-requirements task: repeated runs of the same configuration overlapped only partially in the requirements they retrieved, different models agreed less still, and even the best single configuration recovered only part of what all runs found together [4]. The research reported here takes up these observations as a whole: it engages several models through a generation pipeline of several steps, aligned with realistic practice; it answers the correctness and reliability concerns by ensembling multiple outputs—if LLMs are not reliable enough one at a time, fusion makes them so; and it streamlines, rather than replaces, human-in-the-loop intervention, ordering the candidate requirements so that oversight starts from the best supported VOLUME 14, 2026
ones and accounting explicitly for the cost of reviewing the hallucinations that survive. We develop our data-fusion response to this gap from Section III onward. B. LLMS FOR DEFENSIVE CYBERSECURITY
In parallel, there is fast-growing interest in applying LLMs to cybersecurity, with a maturity that varies markedly across the lifecycle of the artifacts addressed. Xu et al. provide a systematic literature review of LLMs for cybersecurity and find the surveyed work dominated by post-development, code-centric artifacts—source code, vulnerable code, bug-fix pairs—while requirements-stage activities are absent from their task taxonomy altogether [15]. Other surveys of the field report the same concentration—application catalogs running from vulnerability detection to secure code generation, again with no requirements-stage category [16], [17]. While on those post-development artifacts LLM assistance is well past the prototype stage—Divakaran and Peddinti discuss the opportunities and risks of deploying LLMs for cyber defense, for instance in interactive ‘‘security copilot’’ settings [18]—for pre-development, natural-language artifacts such as cybersecurity requirements, applications remain largely exploratory. The copilot vision itself extends to the early phases—LLMs prompted to identify and fill gaps in system specifications— though there as prospect rather than practice [19]. Recent surveys synthesize the landscape of LLMs across intrusion detection, malware analysis, phishing, and incident response, identifying opportunities but also serious concerns about hallucinations, attack-surface expansion, and how easily LLMs can be misused [20], [21]. Motlagh et al. similarly catalog uses of LLMs in cyber operations, mapping defensive applications to the National Institute of Standards and Technology (NIST) cybersecurity framework: studies concentrate on the Protect and Detect functions, leaving the Identify function among the flagged research gaps—and none addresses generating the cybersecurity requirements of a project [22]. Taken together, these maps place our target artifact class in a region they leave blank, at the exploratory end of the field’s maturity gradient; the present article contributes there the kind of prioritization layer that an early-phase security copilot would need to make its output reviewable. More specialized reviews focus on concrete technical tasks, and two of their traits matter here. First, their objects are post-development and largely not textual: Ferrag et al. survey generative AI and LLMs for cybersecurity and enumerate applications in threat intelligence, secure code generation, and red-teaming [23], and Sheng et al. review LLM-based approaches to vulnerability detection in software [24]—none of these targets requirements or comparable pre-development texts. Second, they surface challenges that carry over to our study: Sheng et al. note strong performance on benchmark datasets but an overreliance on synthetic, isolated code and limited evaluation on complex, real-world artifacts; the security-knowledge benchmarks cataloged in recent systematic literature reviews are closed-ended, multiple-choice questions [16], and benchmarks built from real-world threat3
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
intelligence reports find LLMs handle such closed items well but fall short on open-ended analyses demanding specialized security knowledge [25]. Within this broader cybersecurity landscape, only a subset of works explicitly targets security- or privacy-related requirements as early-lifecycle artifacts. Subahi leverages LLMs to evaluate the fulfillment of security requirements within requirements engineering [26]. Melo et al. review LLM applications in security requirements engineering, concluding that the research line is still emerging and lacks standardized evaluation metrics and benchmarks, which hinders reproducibility and the consolidation of results [27]. Adjacent work ensembles several fine-tuned transformer models to classify requirements into functional and non-functional categories, without a security-specific focus [28]. Our earlier work falls into this security-requirements niche. There we showed that a single, well-engineered configuration could retrieve a substantial fraction of a manually judged gold standard of ISO/IEC 27002-derived cybersecurity requirements for a given system under development, but also that 1) different runs discovered different subsets of those requirements, and 2) hallucinations were run- or model-scoped rather than systematic. Indeed, one of the prompting approaches back in that study explored a configuration that reissued the same request across four parallel runs and coalesced their outputs—aggregation without voting—raising recall but carrying every run’s hallucinations into the merged result, at a cost in precision [4, Sec. IV-E]. Building on that evidence, the present article supplies what the efforts reviewed above still lack—a systematic treatment of variability across runs— by investigating whether combining multiple runs and models through principled voting-based fusion can increase recall on cybersecurity requirements without proportionally inflating hallucinations. C. ENSEMBLING IN LLMS AND NLP
Ensemble methods have a long history in machine learning and information retrieval: combining multiple imperfect predictors often yields better accuracy than relying on any single one. Classic data-fusion work in IR distinguishes between rank-based and score-based fusion and identifies the chorus effect that makes fusion profitable [29], [30], as documents appearing in many systems’ results are more likely to be relevant. Fusion schemes exploiting this effect consistently outperform individual retrieval systems in many settings: scorebased rules such as CombSUM and CombMNZ [31], [30], combination of Boolean—hence binary—query results [32], and supervised probabilistic variants like ProbFuse, which learns segment-wise probabilities of relevance from training queries and sums them across systems [33]. Beyond IR, ensemble learning is well established in software engineering more broadly, most prominently in software defect prediction [34]; closer to our setting, data-fusion ideas themselves have recently been transferred to tasks such as code search, fusing the outputs of different search techniques as independent retrieval systems [35], and bug localization, combining 4
information-retrieval and spectrum-based signals [36]. For LLMs, surveys now discuss both LLM ensembles [37] and agentic compositions [38]. Following the taxonomy there adopted—ensembling before, during, or after inference—we distinguish three broad families, which differ in when fusion is applied relative to generation: 1) Routing approaches decide before generation which model should handle a particular query, e.g., on expertise criteria [39]; related cost-driven cascades instead try models sequentially until an answer is accepted [40]. Either way, only one model’s output is ultimately used per query, so these approaches do not directly exploit the diversity across runs that we seek. 2) Collaborative-decoding ensembles fuse models within generation, typically at the level of token probabilities [41], or by selecting which model of a pool generates each reasoning step, guided by a process reward model [42]. These methods require tight control over decoding—typically down to token-level logits or hidden states—which is often unavailable through application programming interfaces (APIs) and considerably more complex to instrument than fusing persisted outputs after the fact; they are therefore outside the scope of our study. 3) Output-aggregation ensembles combine complete responses after generation. Wang et al. propose selfconsistency, which samples multiple chain-of-thought explanations from a single model and returns the answer most consistent across samples, substantially improving accuracy on arithmetic- and commonsensereasoning benchmarks [43]. Jiang et al.’s LLM-Blender first collects outputs from several base LLMs, uses a learned pairwise ranker to score them, and then prompts a generator model to synthesize a final answer that fuses high-scoring candidates [44]. Li et al. and other recent work show that agreement-based voting over independently sampled answers can serve as a strong, trainingfree ensemble baseline [45]. Output-level stacking of several LLMs has also been applied to security-relevant classification tasks such as phishing detection [46]. Both IR data fusion and LLM output ensembling address a similar problem through similar methods—combining the outputs of several imperfect systems into a single, better result. From output-aggregation ensembling—the LLM-side term of art for the overall paradigm—we inherit the working conditions: complete responses combined after generation, with no access to model internals. From IR data fusion we take the machinery—the concrete combination rules operating inside that paradigm; indeed, each classical scheme reviewed above anticipates one element of our design (Section IV): CombSUM, additive score fusion itself; the combination of binary Boolean evidence, the vote counting underlying both of our strategies; and ProbFuse, the supervised, probability-derived computation of the reliability weights specific to our Naive-Bayes strategy. What requires adaptaVOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
tion is the structure of the inputs: whereas IR data fusion merges the ranked lists returned by several search systems, each of our LLM runs behaves as an unranked retrieval system returning a set of candidate requirements for a given control and system specification, so a run’s contribution reduces to a binary vote per candidate. Under this reading the chorus effect transfers directly—agreement across runs signals validity. The transfer has qualitative support in the source study’s consistency analysis: introducing different models increased diversity (reduced pairwise Jaccard similarity), and both inter-model and same-model consistency were higher on valid requirements than on hallucinations [4]. We study two voting-based fusion strategies: Uniform fusion, which counts votes and thereby operationalizes the chorus effect, and Naive-Bayes fusion, which further weights each vote by the estimated reliability of the model that produced it. Unlike output-aggregation methods built on auxiliary trained models, our strategies train no additional model—the NaiveBayes weights are fitted directly from adjudicated data—and operate purely post hoc on the runs’ persisted textual outputs. III. MATERIALS AND METHODS
This study builds on the results of a previous evaluation of LLM-augmented cybersecurity requirements generation [4], in which several LLMs repeatedly drafted system-specific cybersecurity requirements from standard-derived control templates, and experts subsequently adjudicated every candidate they produced. In this section, we restate that task setting and the frozen, expert-labeled corpus reused here, define the contingency-table metrics used over its universe of candidate requirements, and cast each LLM execution of the task as a black-box retrieval system whose outputs are fused into a single ranked review queue using the strategies of Section IV. A. TASK DESCRIPTION AND SOURCE CORPUS
As summarized in Fig. 1, the source study addressed an LLM entrusted with the task of drafting system-specific cybersecurity requirements, provided with 1) a natural-language description of the target system under development and 2) a parameterized cybersecurity control template derived from ISO/IEC 27002:2022, to be instantiated into concrete requirements for the target system. The problem is therefore not freeform ideation, but the contextualization of generic control intent into system-specific requirements. The execution of the task combines several steps in an LLM-based pipeline built with LangChain that concatenates applicability checks, the mapping of template placeholders to domain elements, the generation of free-text requirement instances, and JavaScript Object Notation (JSON) formatting. Ten cybersecurity control templates derived from ISO/IEC 27002:2022 and annotated with placeholders were used as input; their parameters denote elements such as user roles, sensitive data stores, or protected activities that must be bound to concrete elements in the specification of the system under development. The target system was AI4I4 (Automated Identification and Data Capture for Industry 4.0), VOLUME 14, 2026
FIGURE 1. Standards-driven cybersecurity requirements generation task (adapted from [4]).
a realistic project addressing AIDC technologies in the logistics processes of an automotive factory, created as a fit-forpurpose research testbed to benchmark LLM performance in this area. Its documentation is an approximately 5300-word natural-language specification covering the relevant domain concepts, roles, data stores, and use cases needed to ground cybersecurity requirements. The elicitation process goes through each control template in the input and generates the requirements that follow from its instantiation over the target system’s elements. Each execution of this process, which we refer to as a run of the task, produces its own set of candidate requirements. Runs are executed under a configuration that fixes model, prompt pipeline, and LLM settings; even repeated runs of the same configuration differ, through stochastic decoding. As for the models, their selection was inherited from the source study. Four were used, one from each of four families: three openweights and operable on local infrastructure (Llama 3.1 405B, Qwen-2 72B, Mixtral 8×22B), and one proprietary, accessed remotely as a commercial service (GPT-4 Turbo). This selection comprises models affordable as of today and provides a fitting testbed for the article’s central question: whether ensembles of such affordable (and hence not frontier-class) models can produce better results than any of their individual members. For each model, we reuse from the source study one base configuration, executed as four independent stochastic runs, and two additional single-run configurations with different prompting/pipeline and LLM-setting choices, as summarized in Table 1. The detailed prompt, pipeline, and hyperparameter settings were reported in the source study [4] and in its own reproducible research package [47]. For each run, the source corpus records the candidate requirements generated by the LLM. The control templates provide an ex-ante grouping of the candidates: each candidate is associated with the control template from which it was elicited, although our analysis below generally aggregates over controls. Raw generated requirements were then: 1) pooled across runs, collecting all candidate statements pro5
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
TABLE 1. Source-study configurations and their runs reused in the ensemble study
Model Llama 3.1 405B Llama 3.1 405B Llama 3.1 405B Qwen-2 72B Qwen-2 72B Qwen-2 72B Mixtral 8×22B Mixtral 8×22B Mixtral 8×22B GPT-4 Turbo GPT-4 Turbo GPT-4 Turbo
Run ID(s) a L0i –L0iv L1 L2 Q0i –Q0iv Q1 Q2 M0i –M0iv M1 M2 G0i –G0iv G1 G2
Variant description Base chain Cooler chain Warmer chain Base chain Cooler chain Warmer chain Base chain Cooler chain Warmer chain Base chain Cooler chain Warmer chain
a Subscripts i–iv index the four stochastic runs of each base configuration;
the variant configurations contributed one run each.
duced by the source runs; 2) decomposed, when a statement bundled several atomic requirements; and 3) canonicalized, so that semantically equivalent statements were represented once, mapping paraphrases or repeated rediscoveries to a single item. The pool also includes requirements elicited by human analysts following the same control-template instantiation process. During corpus construction, each canonical candidate requirement was expert-adjudicated as valid or hallucinated, with valid candidates mapped to existing humanelicited requirements when they expressed the same normative intent, or added as new nonredundant valid requirements when appropriate. Conversely, some valid requirements may have no support from any LLM run. This yields a frozen judged corpus over which all post hoc ensembling and evaluation are performed, without any further interaction with the underlying LLMs. The full requirement text behind these candidates is distributed with the source study’s reproducible research package [47], against which the candidate identifiers used throughout this article resolve; our own deposit records the reused run composition and the ensembling analysis (see the Appendix). B. JUDGED UNIVERSE AND CONTINGENCY-TABLE METRICS
Let U denote the universe of candidate requirements. Following the human adjudication process described above, it is partitioned into V, the gold-standard valid requirements, and H, the hallucinated candidates. In the version of the corpus used in this article, |U | = 183 and |V| = 72. The universe is delimited by this study’s participating sources, the same membership rule applying to valid and hallucinated candidates alike: U comprises the canonical candidates emitted by the 24 reused runs together with the requirements elicited by the human analysts. Consequently, V is the source study’s gold standard re-pooled over this universe’s participating runs: of its 76 published valid requirements, the 6 contributed only by exploratory source-study configurations are excluded here (alternative prompting pipelines, and a coalesced multibranch variant that would have skewed the run set toward one model), plus we include 2 further requirements surfaced 6
only by the repeated runs—which the source study analyzed for consistency but did not pool into its gold standard— for 72 in total; the hallucinated candidates of the excluded configurations likewise do not enter U. Let Runs be the set of LLM runs. For each run r ∈ Runs, let Ur ⊆ U denote the set of canonical candidate requirements generated by that run. Each run thus defines a binary vote ( 1, u ∈ Ur , yr (u) = u ∈ U. (1) 0, u ∈ / Ur , For any selected set A ⊆ U, such as the output Ur of a run, the counts of the contingency table (true/false positives/negatives) are TP(A) = |A ∩ V|,
FP(A) = |A ∩ H|,
FN(A) = |V \ A|,
TN(A) = |H \ A|.
(2)
True negatives are thus defined only within the finite set H of hallucinated candidates produced during corpus construction—not over every invalid statement one could conceivably write: our goal is to evaluate how well alternative fusion strategies prioritize the candidates that were actually produced, pooled, and judged. From these counts, we compute the recall or true-positive rate (TPR), the precision or positive predictive value (PPV), the specificity or true-negative rate (TNR), and the fall-out or false-positive rate (FPR): TPR(A) = TP(A)/(TP(A) + FN(A)), PPV(A) = TP(A)/(TP(A) + FP(A)), TNR(A) = TN(A)/(TN(A) + FP(A)), FPR(A) = 1 − TNR(A) = FP(A)/(TN(A) + FP(A)). (3) This represents a closed-world evaluation in the Cranfield/Text REtrieval Conference (TREC) tradition: systems are compared over a fixed test collection and a fixed set of relevance judgments [48], [49], [50]. The retrieval analogy is deliberate, with two peculiarities. First, there is a single, fixed query—in effect, ‘‘the requirements instantiating these controls on this system’’—rather than a stream of varying information needs. Second, the collection is the judged universe U itself, pooled from the outputs of all runs and from the humanelicited requirements, and then manually adjudicated. Within it, the valid requirements V play the role of the relevant items, and the hallucinated candidates in H that of the nonrelevant ones. None of these ratios is informative in isolation: recall can be driven arbitrarily high by selecting more candidates, at the expense of precision and specificity, and conversely. We therefore use two scalar summaries that aggregate both sides of this trade-off, weighted to encode how costly a missed valid requirement is relative to a spurious candidate in our review setting. The first is the Fβ -measure [51], [52], a weighted harmonic average of recall and precision, Fβ = (1 + β 2 ) PPV TPR/ β 2 PPV + TPR . (4) VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
For β = 1, this is the usual F1 -score. In this study, we use F2 , which places four times more weight on recall than on precision. The second weighted summary is a weighted Youden index. The usual Youden index summarizes the balance between sensitivity and specificity [53] as J = TPR+TNR−1. We use instead the weighted form [54] Jw = 2 w TPR + (1 − w) TNR − 1, (5) which reduces to J when w = 1/2. We adopt the costsensitive formulation of optimal operating points on the receiver operating characteristic (ROC) curve [55], [56] and compute w = ρ π/ ρ π + (1 − π) , (6) where π is the prevalence of relevant elements in our universe π = |V|/|U|, and ρ is a scenario-dependent false-negative to false-positive cost ratio. With ρ = 4 (a 4:1 preference to avoid false negatives over false positives), this gives w = 0.722. F2 and Jw aggregate different ratios, but both encode the same utility-oriented methodological preference: in this task, preserving valid requirements is more important than minimizing the number of candidates that experts must reject, as omitted controls can create security gaps, compliance risk, and expensive late rework, whereas a hallucinated candidate can be more easily discarded during review. Single-run outputs are used as original-output baselines. In addition, configuration-level summaries are provided as baselines that average the numerical evaluations of the runs sharing the same model, prompt/pipeline, and LLM settings. C. ENSEMBLING AS VOTING-BASED DATA FUSION
Post-generation ensembling may be treated as voting-based data fusion over the outputs of the black-box runs, as shown in Fig. 2. Each run casts a binary vote yr (u) on each candidate u ∈ UP , and these votes are combined into a fused score s(u) = r∈Runs wr yr (u) through per-run weights wr . Unlike the runs’ unranked output sets, the fused output is a single ranked list over U , obtained by sorting candidates by decreasing score; this ranking is the article’s main object of analysis. In a practical review workflow, candidates would then be inspected from the top of the ranking until the available review budget or an acceptance threshold is reached. Every such stopping point selects a prefix—a set to which the contingency-table metrics above apply directly—so a ranking defines a family of operating points, which Section V assesses both individually and across all review depths. The underlying intuition is the data-fusion ‘‘chorus effect’’: support for a candidate from several runs, especially runs of independent systems, provides evidence of relevance [57], [29]. At the same time, not all sources are equally reliable: giving more influence to runs from models that are empirically better at separating valid requirements from hallucinations may produce more faithful results. Section IV formalizes the concrete scoring rules and the weight choices that realize these intuitions as the two fusion strategies evaluated in this article: Uniform fusion and Naive-Bayes fusion. VOLUME 14, 2026
IV. IMPLEMENTATION
In this section, we introduce the fusion strategies, which convert the unranked outputs of the source LLM runs into a single scored ranking over the judged candidate universe U. A. WEIGHT-BASED ENSEMBLING
For each candidate requirement u ∈ U , the weight-based ensemble uses only the binary votes yr (u) cast by the source runs r ∈ Runs—the indicators of membership in each run’s output set Ur , as defined in Section III-B. A fusion strategy assigns a weight wr to each run and computes the aggregated score of the candidate requirement X s(u) = wr yr (u). (7) r∈Runs
Candidates are ranked by decreasing s(u)—the classical CombSUM rule from information-retrieval data fusion [58], [57], specialized to binary run-level evidence [32]. Because the inputs are unranked sets, (7) fuses agreement rather than ranks: the score is built post hoc from the generators’ votes, with no per-run ordering to merge. Although (7) allows arbitrary run weights, the strategies reported here do not estimate idiosyncratic weights for individual stochastic runs. The reason is that runs produced by the same configuration are repeated executions of the same model, prompt pipeline, and LLM settings; thus, assigning them different learned weights would mainly capture accidental variation in a particular corpus realization. Hence, all the strategies give equal weight to runs from the same configuration. In particular, the two strategies evaluated in the article are even coarser: Uniform fusion gives the same weight to all runs, whereas Naive-Bayes fusion gives the same weight to all runs produced by the same model. B. FUSION STRATEGIES
The first strategy we present is Uniform fusion, which assigns the same weight to every run, wunif = 1/|Runs|, r
∀r ∈ Runs.
(8)
The normalization constant has no effect on the ranking, but makes the score scale independent of the number of runs. Uniform fusion therefore ranks candidates by the amount of support they receive across runs: a candidate’s score is simply the fraction of runs that generated it. The second strategy is Naive-Bayes (NB) fusion, which is based on estimating how much a vote from each model should shift the evidence toward candidate validity. Let Yu ∈ {0, 1} denote the latent validity label of candidate u, with Yu = 1 for u ∈ V and Yu = 0 for u ∈ H. For each model m, let Runsm ⊆ Runs be the subset of runs produced by that model. The Naive-Bayes model treats the votes of runs in Runsm as conditionally independent repeated measurements given Yu , with shared true-positive and false-positive rates: TPRm = Pr(yr (u) = 1 | Yu = 1, r ∈ Runsm ) , FPRm = Pr(yr (u) = 1 | Yu = 0, r ∈ Runsm ) .
(9) 7
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
FIGURE 2. Post-generation ensembling: votes cast by parallel LLM runs are fused, through per-run weights, into a single ranked review queue over the judged universe U .
Both assumptions—conditional independence and shared rates across the runs of a model—are practical approximations: runs of the same model are not independent, as prompt and pipeline reuse induces dependence among their votes. Nonetheless, such assumptions do not hamper the validity of our results, as they are used only to derive this strategy’s weight vector; the empirical evaluation in Sections V and VI assesses the ranking induced by that vector, whether or not the assumptions hold. The result is a practical weighting rule, not necessarily an optimal one: better weightings exist for a given corpus, and our evaluation includes one, fitted numerically, as a benchmark for the attainable headroom (Section V). Because the corpus provides expert labels for all u ∈ U, the rates are estimated directly, as P P r∈Runsm u∈V yr (u) + 1/2 [ , TPRm = |Runsm | |V| + 1 P P (10) r∈Runsm u∈H yr (u) + 1/2 [ . FPRm = |Runsm | |H| + 1 The +1/2 terms in the numerators and +1 in the denominators are the result of Jeffreys regularization (equivalent to a Beta(1/2, 1/2) prior for each Bernoulli rate) to avoid zero or unit rate estimates, which would otherwise produce infinite log-odds weights. The same half-count correction is standard in probabilistic information retrieval weighting, notably in Robertson–Sparck Jones relevance weighting [59]. The corresponding model weight is the log-likelihood-ratio contribution of an observed vote toward requirement validity: [ m 1 − FPR [m TPR . wNB (11) m = log [ [ FPRm 1 − TPRm Each run r ∈ Runsm is then assigned wr = wNB in (7). m The weight is signed and deliberately not clipped at zero: a positive value indicates that a vote from model m increases 8
the posterior log-odds of validity, whereas a negative value would mark that model’s votes as evidence against validity. In this corpus, all estimated weights are positive. This produces a supervised weighted-voting rule: a candidate supported by models with high estimated recall over valid requirements and low estimated fall-out over hallucinations receives more score mass. Under the model of (9), the CombSUM score admits an exact posterior reading. By Bayes’ rule, the posterior log-odds of validity given the full vote profile is the prior log-odds plus one log-likelihood-ratio term per run: log(TPRm /FPRm ) for a run ofmodel m that voted for u, and log (1 − TPRm )/(1 − FPRm ) for one that did not—in the closed universe U, a non-generated candidate is an informative zero vote, not a missing judgment, so silence also counts as evidence. Rewriting each term as the silence contribution plus yr (u) wNB m —the weight of (11) is precisely the gap between the vote and silence contributions—and noting that every candidate thereby collects the silence contributions of all runs, the posterior log-odds equals s(u) plus Pa candidate-independent constant: the prior log-odds plus m |Runsm | log (1 − TPRm )/(1 − FPRm ) . Ranking by s(u) is therefore ranking by the NaiveBayes posterior Pr(Yu = 1 | {yr (u)}r∈Runs ): absent votes need no separate weight, and the prevalence prior, shifting all candidates equally, leaves the ranking unchanged. The reading is specific to the judged universe on which the rates are estimated: prospective use on a different system under development would apply these weights as fixed, best-effort estimates—the fixed-weight regime whose robustness the perturbation analysis of Section V-F probes—and the dataset dependence this entails is discussed in Section VII-C. This formulation is closely related to the well-known Dawid–Skene model for noisy observers [60], where each observer has a confusion matrix and the typically unknown true labels are inferred jointly with the observer reliabilities through expectation–maximization. Here, each LLM run acts VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
as a noisy observer that either retrieves or fails to retrieve a candidate, and the expert labels are known, so the observer reliabilities are estimated directly from the judged corpus as shown in (10). V. EVALUATION PROTOCOL
The evaluation compares the fusion rules with each other and with the original LLM outputs over the judged candidate universe U, attending to two complementary concerns: the coverage of valid requirements, captured by recall, and the hallucination burden, captured by precision and fall-out (equivalently, specificity), giving more relevance to the former through the combined yet recall-laden criteria F2 and Jw . Each of these metrics varies along the review depth k (the @k operating points), while the global metrics—average precision (AP) and the area under the ROC curve (ROC AUC)—summarize whole rankings by respectively integrating the combined precision–recall and ROC curves. Because vote-based scores produce frequent ties, all metrics are computed as tie-aware expected values. In addition, single runs and configuration averages are themselves evaluated with equivalent metrics, which place them on the same axes as the fused rankings. The results are framed with respect to reference curves and subject to internal-validation analyses to interpret whether the observed differences are robust to plausible changes in the evidence available for evaluation. A. RANKED OUTPUTS AND @k OPERATING POINTS
Both the ensembles and the single runs are interpreted as retrieval outputs over U, but with different score structures. A single run r produces an unranked set Ur : its generated candidates, those with vote yr (u) = 1, form one tied block above its non-generated remainder, with yr (u) = 0. An ensemble, by contrast, induces a meaningful ranked list by sorting candidate requirements by the fusion score s(u) from (7). For a ranked list, inspecting the first k candidates defines a rank cutoff, commonly denoted by the suffix @k in information retrieval. Changing k changes the operating point of the retrieval: larger k usually increases coverage, measured by recall, but also admits more hallucinated candidates, lowering precision and specificity. For a cutoff k, the Topk prefix set induces a contingency table (with TP@k, FP@k, FN@k, and TN@k). Therefore the operating-point metrics (such as P@k, R@k, Specificity@k, F2 @k, and Jw @k) — generically, M@k for an evaluation metric M — can be computed by applying the formulas from Section III-B to the set A = Topk . Degenerate cases are handled with common-sense defaults matching their natural limits: a prefix containing no true positive has F2 @k = 0 (resolving the 0/0 form), and a universe lacking valid items (possible in the resampled universes of the stability analysis, Section V-F) yields recall, AP, and ROC AUC equal to 0, while one lacking hallucinated candidates yields specificity and ROC AUC vacuously equal to 1. In our analysis, we will emphasize two fixed review depths. The first is kM = maxr∈Runs |Ur |, i.e., a review budget VOLUME 14, 2026
the size of the largest original LLM output (in this corpus, kM = 40). The second is kR = |V| (named after the usual Rprecision convention where R denotes the number of relevant items), which represents the earliest rank at which full recall could in principle be achieved (in this corpus, kR = 72). We also report the maximum values of F2 @k and Jw @k, together with the first rank at which each maximum is attained. These maxima summarize the best operating point under the chosen recall-sensitive utility view, and the associated rank indicates how early it is reached. B. CURVES AND GLOBAL RANKING METRICS
As explained above, each operating-point metric M@k yields a curve rather than a single value: plotted as an explicit function of the review depth, it shows how the quality of the review set evolves as the operating point deepens. Sweeping k also parametrizes — implicitly, with k running along each curve — two complementary views that integrate the tradeoffs between the individual metrics. The precision–recall (PR) curve plots P@k against R@k. This curve makes visible how many valid requirements are recovered (recall or TPR) for a given density of useful review items (precision or PPV). The ROC curve plots TPR@k (recall) against FPR@k (fall-out). The PR and ROC views are complementary: precision ties the PR view to the prevalence of valid candidates, whereas the ROC view, built from per-class rates, is insensitive to prevalence [61], [62]. Each combined criterion is a function of the two coordinates of one of these planes: F2 of precision and recall, Jw of TPR and FPR. Every point of the corresponding curve therefore fixes the criterion’s value, so F2 can be read directly off the PR curve and Jw off the ROC curve. The overall quality of each of these curves can, in turn, be captured by a single metric of its own: • AP can be understood as the area under the conventional stepwise precision–recall curve induced by ranking relevant items (slightly different from the geometric area under a visually plotted line segment curve [63]). It summarizes whether valid requirements are retrieved early and at high precision. • ROC AUC is the area under the empirical ROC curve. It summarizes pairwise separability between valid and hallucinated candidates, as it equals the probability that a randomly chosen valid requirement receives a higher score than a randomly chosen hallucinated candidate. C. TIE HANDLING
Discrete vote-based scores create frequent ties. Hence, a rank cutoff may fall inside an equal-score block; likewise, global metrics such as AP or ROC AUC may depend on how tied candidates are internally ordered. Four standard remedies exist: report only score-block-level values, break ties with an arbitrary secondary key, simulate random tie-breaks, or compute analytical expectations over random permutations within each tie block. Following tie-aware IR evaluation, we use the last [64], whose closed-form expectations cover the 9
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
operating-point metrics used here (precision, recall, F-scores) plus the global AP. After sorting by score, let the ranking be partitioned into maximal tie blocks. We take a uniformly random permutation over each tied block, which contains t items of which v are valid. When a prefix cuts through such a block, each additional position inside the block contributes an expected valid mass of v/t true positives and an expected hallucinated mass of (t − v)/t false positives. Thus, TP@k and FP@k are estimated by these fractional expected counts, and since every operating-point metric at a fixed k is an affine function of the counts, replacing them in its formula yields its exact expected value. This avoids arbitrary document-ID effects, is deterministic and reproducible, and reports the expected performance implied by the score information actually available to the fusion strategy. Note that the reported maximum-attaining rank for a given metric M (e.g., F2 or Jw ) is therefore the first k maximizing the expected curve, i.e., arg maxk E[M@k], rather than the expectation of a random maximizing rank, E[arg maxk M@k]. Ties affect the estimate only when the cut falls inside a tie block: blocks lying entirely within the topk prefix contribute their exact counts, so @k values at block boundaries involve no expectation at all. The global metrics AP and ROC AUC, in contrast, are not affine in the counts and require dedicated treatment under the same tie model. Expected AP is computed from closed-form block contributions under uniform random ordering within tied blocks: in a block occupying ranks b, . . . , b + t − 1 with v valid items among its t members and preceded by R0 valid items, the position b + j (for j = 0, . . . , t − 1) contributes v−1
v R0 + 1 + j t−1 · t b+j
(12)
in expectation to the sum of precisions at valid ranks, whose total over all blocks, divided by |V|, is the expected AP. Expected ROC AUC is computed from the pairwise preference probability, with each valid–hallucinated pair tied in the same block receiving half credit [61], [65]. D. COMPARABILITY METRICS FOR SINGLE RUNS AND CONFIGURATION AVERAGES
The curves and global metrics defined above describe ranked ensemble outputs. To place single runs on the same axes as the fusion rankings, each run is evaluated as a binary retrieval system over U: its induced two-level score yr (u) ranks the generated candidates as one tied block above the non-generated remainder, and the run’s equivalent metrics are those of the resulting two-block ranking. This does not imply that the LLM produced an inherent ordering: the two-level score just encodes membership in Ur . For this binary scorer, ROC AUC reduces to AUCr = (TPRr + TNRr )/2, and APr is the expected AP of the twoblock ranking under the tie model of Section V-C. In addition, configuration summaries are used as baselines for repeated runs of the same configuration; the equivalent metrics defined next place these averages on the same axes 10
as well. These should be read as descriptive baselines that smooth outlier runs, never as the output of some additional ensemble. Let Runsc be the runs belonging to configuration c. For each configuration c, the ROC coordinates are microaveraged over the common assessment pool: P P FP(Ur ) r∈Runsc TP(Ur ) , FPRc = r∈Runsc . TPRc = |Runsc | |V| |Runsc | |H| (13) To display the same configuration point in PR space, precision is reconstructed backward from population prevalence π = |V|/|U|: PPVc =
π TPRc . π TPRc + (1 − π) FPRc
(14)
The remaining operating-point metrics, including F2 and Jw , are computed from these micro-averaged rates; the equivalent operating point for these configuration summaries is placed at the mean output size X 1 |Ur |. (15) k̄c = |Runsc | r∈Runsc
Configuration ROC AUC is likewise obtained from configuration TPR and TNR through the same binary-scorer identity as for single runs, (TPRc + TNRc )/2, not by integrating a ROC curve. These averaging choices are deliberate: since each run has its own output size and hence its own precision denominator, the plain average of per-run precisions and recalls is not the precision–recall point of any actual retrieval, whereas micro-averaging the counts places the configuration at the operating point of its pooled contingency table. AP, in contrast, has no pooled counterpart — there is no single configuration-level ranking whose AP it would describe — so we report the macro-average of single-run AP values: X 1 APc = APr . (16) |Runsc | r∈Runsc
The configuration’s PR point and its APc are thus deliberately separate aggregates — micro-averaged rates versus a macroaverage over runs: absent a pooled configuration ranking, neither is derived from the other. E. BENCHMARKS: REFERENCE CURVES AND NORMALIZED UPLIFT
To contextualize the numerical differences observed between one strategy and another, and between these and single runs, we frame the computed curves within four reference curves, each bounding the results in a different sense: 1) The feasible region, imposed by a dataset with |V| valid requirements and |H| hallucinated candidates. This region represents the geometry of the finite judged universe, as no ranking can retrieve more than k valid items by rank k or more than |V| valid requirements in total; hence full precision is unattainable at review depths beyond |V|, bounding the values each operatingpoint metric M@k can take at any given k. VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
2) The expected random ranking, computed as a single tie block containing all candidates in U. Given population prevalence π = |V|/|U|, the expected counts at rank k are E[TP@k] = kπ and E[FP@k] = k(1 − π). Consequently, E[P@k] = π, E[R@k] = k/|U|, and E[FPR@k] = k/|U |. The random PR curve is therefore horizontal at the prevalence level π, whereas the random ROC curve follows the chance diagonal. Derived quantities such as F2 @k and Jw @k are computed from these expected point metrics. 3) The pattern oracle. Candidate requirements can be grouped into pattern blocks: two candidates share a block when every configuration gives them the same vote count, i.e., when their configuration-level vote patterns are identical. Candidates within the same pattern block will receive the same score under any fusion rule that only considers vote counts from each configuration. That is, if a pattern block contains both valid and hallucinated candidates, no weighting scheme that uses only those voting features can separate them. The pattern oracle produces the ranking that sorts these unbreakable pattern blocks by their observed internal precision (fraction of valid candidates), from highest to lowest, while leaving candidates inside each block tied. It therefore estimates the headroom available if vote patterns were exploited perfectly, without introducing any additional information. 4) The numerical linear-fusion benchmark, built by searching over nonnegative configuration-weight vectors in the CombSUM family, refining trial weight vectors by coordinate ascent on the simplex. At each step, one coordinate is swept over a grid, the remaining weights are rescaled, the induced ranking is rescored, and the move is accepted only if it improves the target tie-aware objective. This reference is heuristic rather than a certified optimum, because the objective is nonsmooth and depends on score-induced tie blocks. Nevertheless, it estimates the headroom potentially attainable by a linear combination of vote counts just by changing fusion weights. Comparing a reported fusion rule to this numerical benchmark isolates suboptimal weighting, whereas comparing that benchmark to the pattern oracle isolates irreducible vote-pattern ties. All four accompany the fusion strategy curves in the compound figure of Section VI (Fig. 3). For each global metric M (AP or ROC AUC), we report the normalized uplift of any given ranking approach, as M[approach] − M[random] , M[benchmark] − M[random] (17) where the approach may be a fusion strategy or a baseline such as the best single configuration. This expresses how much of the observed gap between random ordering and the numerical linear-fusion benchmark is recovered by a simple, interpretable fusion rule. upliftM [approach] = 100
VOLUME 14, 2026
F. INTERNAL VALIDATION AND STABILITY
The full-data results are apparent performance estimates on the same judged corpus used to estimate weights. Therefore, we need to probe internal validity and assess whether the relative conclusions about fusion rules are stable to plausible changes in what the evaluation treats as evidence. The primary internal-validation analysis is a requirement-level outof-bag (OOB) bootstrap with refitting: we draw 200 bootstrap replicates, each a sample of 183 candidates drawn with replacement from the 183 candidate requirements in U , and collapse duplicate in-bag candidates to preserve a set-valued retrieval universe. Data-dependent fusion strategy weights are then refitted on the unique in-bag candidates and evaluated on the out-of-bag complement. The same out-of-bag complement is used for all strategies in a replicate, thus differences are paired. The weight w of Jw , in contrast, keeps its fulldata value without refitting, both here and in the perturbation samples below, so the metric stays on a single utility scale and paired differences remain comparable. This analysis estimates whether the relative advantage of a strategy persists when the learned weights are applied to unseen candidates from the same judged universe; transfer to unseen control templates is not separately validated. The resulting contrasts are reported in the frame of estimation statistics: each is summarized by the magnitude of the paired difference in the metric’s own units — themselves probabilityscale quantities — and by how that magnitude varies across replicates, where percentile bands are interpreted as descriptive internal-validation ranges, not as classical confidence intervals [66], [67]. Because the replicates overlap and do not form an external validation sample, test statistics computed naively over them are anti-conservative [68]. The OOB procedure is related in spirit to m-out-of-n bootstrap subsampling ideas, but is not identical: we draw n times with replacement and then evaluate on the unsampled complement, so the effective in-bag size is random. In any case, the results obtained from this OOB procedure are intentionally conservative. Deduplication leaves an expected unique in-bag fraction of 1 − (1 − 1/183)183 ≈ 63% of the corpus; empirically, a mean of about 116 unique in-bag and 67 out-of-bag candidates over the 200 replicates. Performance estimated after training on such reduced bootstrap samples is known to be pessimistically biased — the learning-curve effect behind the ‘‘.632’’ correction of the leave-one-out bootstrap [66] — so OOB values are best read as lower bounds on the corresponding full-data performance. The complementary stability analysis is a fixed-weight perturbation analysis: we perturb the evidence while keeping the already-fitted fusion weights fixed. We use combined leave-one-out perturbations over runs and the ex-ante control slices—the per-template groups of candidate requirements noted in Section III-A—dropping one control slice and one run at a time, in a jackknife-like fashion. That way, we check whether results are driven by particular control templates or by idiosyncratic runs. Where available, we also report analytic intervals that 11
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
12
B. ENSEMBLING GAINS IN GLOBAL RANKING QUALITY
Table 3 confirms the operating-point picture with global metrics that summarize effectiveness over the whole ranking. If we normalize the results to the gap between random ordering and the numerical linear-fusion benchmark, the best
@
k
k 2@
Jw
Best config. (L0) k̄L0 = 35.25 0.887 0.434 0.036 Naive-Bayes kM = 40 0.950 0.528 0.018 Uniform kM = 40 0.941 0.523 0.021 Naive-Bayes kR = 72 0.743 0.743 0.167 Uniform kR = 72 0.712 0.712 0.187
F
R
k
Operating point k
FP
Approach
@
k
TABLE 2. Metrics of the best configuration and the fusion strategies at distinguished operating points
R@
A. ENSEMBLING EFFECTS ACROSS REVIEW DEPTHS
As the analyst descends through the ranked list of candidate requirements produced by a given fusion strategy, performance metrics vary, trading exhaustiveness against tolerance to hallucinations. Fig. 3 compares, over the judged universe, the retrieval behavior of the original LLM outputs and of the two fusion strategies. The upper panel plots F2 @k against the review depth k, providing an effort-aligned summary of recall and precision; the PR and ROC panels then show the same ranked outputs in terms of review yield and class separation. All results use the expected tie handling defined in Section V-C. The two fusion strategies appear as curves swept by the operating point k, with dedicated markers flagging the maximum F2 and Jw values each curve attains. The original outputs appear as points: individual runs, and configuration averages placed at their mean output size k̄c with micro-averaged coordinates (Section V-D). The distinguished review depths kM and kR appear as vertical references in the upper panel and are marked on the curves in the other two. The four reference curves of Section V-E frame the comparison: the shaded feasible region, the expected random ranking, the pattern oracle, and the numerical linear-fusion benchmark. For the latter, each panel draws the boundary given by the search’s best solution for the global objective underlying its
k
VI. RESULT ANALYSIS
This section presents the empirical evidence for the contributions of Section I: instantiating the protocol of Section V, we compare the rankings of the two fusion strategies (Uniform and Naive-Bayes) against each other and the original outputs—single runs and configuration averages as unranked baselines—first across review depths (Section VI-A) and then by global ranking quality (Section VI-B). Under both views, the findings unfold in a two-stage pattern: the mere agreement among runs already improves the order in which candidate requirements would be reviewed—Uniform fusion yields substantial gains over every original output—and the reliability-aware Naive-Bayes weighting refines that ordering further, reaching useful operating points earlier.
respective view: AP for the F2 @k and PR panels, and ROC AUC for the ROC panel. Finally, the background contours in the PR and ROC panels trace constant-F2 and constantJw levels, so that our two recall-sensitive criteria can be read directly off both views. One feature of the figure deserves clarification: the long final straight segment of the Uniform curve stems from its coarse support-count score. The large group of candidates supported by exactly one run forms one equal-score block, within which valid and hallucinated mass accumulates at a constant fractional rate under tie handling, producing straight segments in the F2 @k and ROC views. Naive-Bayes weighting, by contrast, separates part of this block according to the reliability of the supporting models, thus yielding a finergrained ordering. Note that the numerical benchmark is optimized for a global metric, not for pointwise dominance, so a strategy curve may locally rise above its boundary. As shown in all three panels, Uniform fusion already yields a substantial gain before any reliability information is introduced, and the improvement runs along every dimension of interest: its fused ranking offers better precision at a given recall and better recall at a given precision than the original outputs, together with lower fall-out and higher F2 and Jw at comparable review depths. Table 2 quantifies this at the two workload-aligned operating points. Under a conservative analyst workload, we may consider that the reviewer inspects at most the largest list that a single run might emit (kM = 40); there, Uniform fusion markedly improves all of P@kM , R@kM , and FPR@kM relative to the best configuration, and consequently also both recall-laden utility metrics F2 @kM and Jw @kM . A deeper representative operating point is the number of valid requirements in the gold standard (kR = 72). In turn, Naive-Bayes fusion adds a smaller but consistent improvement over Uniform: across the practically relevant range of review depths k, reliability weighting shifts the PR and ROC curves toward higher precision and lower fall-out for a given recall, higher recall for a given precision, and better combined recall-favoring metrics (higher F2 and Jw ).
P@
support the interpretation of selected scalar comparisons: DeLong-style inference for ROC AUC [69] and Wald approximations for fixed-threshold Youden-type operating-point quantities [54]. These intervals are full-data, per-comparison instruments, and they rest on an independence idealization: the judged candidates are treated as independent draws, whereas candidates cluster by the control template that elicited them — a clustering probed empirically by the control-slice perturbations rather than modeled analytically. The internal-validation argument itself is carried by the paired OOB and perturbation analyses, on a different ground: within each replicate or perturbation sample, all strategies are evaluated on the identical resampled universe, so the paired differences preserve the dependence between strategies — a pairing argument, not a correction for candidate clustering.
0.483 0.579 0.574 0.743 0.712
0.163 0.308 0.299 0.536 0.480
VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
TABLE 3. Global metrics for the best configuration (L0) and the fusion strategies: AP and ROC AUC with normalized uplifts (progress from the expected random ordering toward the numerical linear-fusion benchmark), and the maximum F2 and Jw operating points
Average Precision ROC AUC F2 operating point a Jw operating point a Approach Value Uplift Value Uplift k Value k Value Best configuration (L0) 0.684 56.8% 0.699 48.4% 35.25 0.483 35.25 0.163 Uniform fusion 0.825 86.2% 0.817 76.9% 183 0.764 79 0.489 Naive-Bayes fusion 0.864 94.1% 0.869 89.6% 91 0.831 91 0.679 a
For the best configuration, F2 and Jw are evaluated at its mean native output size k̄L0 = 35.25. For the fusion strategies, the table reports the maximum value over k and the first rank at which it is attained.
original configuration covers roughly half of it (56.8% for AP, 48.4% for ROC AUC), whereas Uniform fusion recovers 86.2% of the AP gap and 76.9% of the ROC AUC gap. The chorus effect thus emerges as a strong ranking signal: repeated support across runs prioritizes valid requirements more effectively than any original configuration alone. Naive-Bayes fusion further improves the results by weighting support according to model reliability: normalized uplifts reach 94.1% for AP and 89.6% for ROC AUC. These gains over Uniform are moderate in magnitude but consistent across the PR and ROC views, indicating that reliability weighting improves both early placement and pairwise separation. The maxima of F2 @k still call for a qualified reading on two axes at once, as the difference is as much one of review depth as of peak attainable quality: Naive-Bayes not only reaches a higher maximum (0.831 against 0.764) but also does so much earlier (k = 91), whereas Uniform peaks only after the candidate pool has been exhausted and the whole judged universe has effectively been admitted (k = 183). For Jw the contrast in value (0.679 against 0.489, both reached at comparable review depths) reflects a better recall–fallout trade-off under the chosen false-negative cost preference. The benefit of reliability-aware weighting therefore lies not merely in higher attainable maxima, but in a more useful ordering that reaches a favorable recall–hallucination balance within a shallower review. VII. DISCUSSION: IMPACT, STABILITY, AND SCOPE
This section reads the results of Section VI from three complementary angles, answering the questions a practitioner would raise before acting on them: what the improvements buy in practice (coverage, review effort, and fit with security requirements engineering processes; Section VII-A); whether the differences are stable under resampling and structured perturbations (Section VII-B); and within what scope the findings should be interpreted (Section VII-C). A. IMPACT OF RESULTS
The results presented in Section VI have two main implications. First, when each model run is viewed as a noisy observer that recovers a partial set of valid requirements, post-generation ensembling improves the recall achieved. Although the best single configuration (L0) covers on average less than half of the gold-standard requirements, the ensemble of the outputs produced by the different models, configurations, and runs recovers all 72 of them. VOLUME 14, 2026
Second, ensembling turns output variability into ranked evidence. Run-to-run stochasticity and inter-model differences manifest as dispersion in which valid requirements each run recovers and which hallucinations it introduces; the ordering induced by post-generation ensembling with appropriate weights allows reaching the same recall level with lower review budgets (i.e., going through shorter-length candidate lists), as candidate requirements supported by several runs— or by more reliable sources—surface earlier. Merely pooling the partial outputs already yields a more complete space of candidate requirements, and reviewing extra candidates to prune false positives may be an acceptable price for the added coverage—but this pruning entails a review cost, so the binding constraint in a human-in-the-loop workflow may eventually be a fixed number of items an analyst can review. Operationally, the question is not only ‘‘does ensembling generate more items?’’ but also ‘‘does it prioritize better items within the prefix that is actually reviewed?’’ As shown by ROC AUC, ensembling induces meaningful rankings where valid requirements are more consistently placed ahead of hallucinations; likewise, the density of hallucinations encountered early for the same ranking prefix length is reduced. Consequently, review effort is more productive: the budget concentrates on higher confidence candidates, and the choice of k lets the reviewer trade coverage against hallucination burden. Taken together, the two implications weigh most in a recall-sensitive activity such as cybersecurity requirements engineering, where a missed requirement costs more than the manual discarding of a spurious addition: pooled coverage supplies the missing valid requirements, and the fused ordering makes recovering them affordable within a bounded review budget. Within this picture, Uniform fusion is a finding in its own right: plain support aggregation, with no reliability information at all, already outperforms every original configuration and captures most of the attainable gain over random ordering (Section VI-B). Repeated support across runs is thus informative by itself—a rediscovered candidate is a better review target than one produced only once. Naive-Bayes fusion adds a further, reliability-weighted gain on top of this chorus effect by discounting support from less reliable models. Comparisons confined to peak utility values understate that refinement, as they miss the ordering advantage: Naive-Bayes not only reaches a higher maximum F2 than Uniform, but it also does so at roughly half the review depth (Section VI-B). Max F2 thus becomes the weakest of the internally validated 13
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
contrasts (Section VII-B): shared peak values conceal the ordering advantage of the earlier ascent. The granularity of each fusion rule also delimits what it can order. Uniform fusion cannot distinguish among the candidates supported by a single run, which form the large equalscore block noted in Section VI-A; the per-model NaiveBayes weights partially resolve this block by ordering singlevote candidates according to the reliability of their supporting model. The indistinguishability that remains is only partly a limitation of the weighting: candidates with identical vote patterns are inseparable to any vote-based rule, and the pattern oracle of Section V-E quantifies exactly that bound. The applicability conditions of the approach are modest: several LLM runs producing overlapping candidate sets that can be canonicalized into comparable units and judged. Postgeneration ensembling fuses and prioritizes what the runs produce; it does not recover valid requirements that appear in none of them. These properties translate into support for established security requirements engineering processes. A concrete example is Security Quality Requirements Engineering (SQUARE), a methodology from the Software Engineering Institute that organizes the process into nine steps, from agreeing on definitions to a final requirements inspection [70]. The source study already envisioned implementing SQUARE as a cooperative process between LLMs and human analysts [4]. Within that vision, the results reported here bring concrete improvements to three of the methodology’s steps: At elicitation (Step 6), fusing several runs enlarges the pool of standards-linked candidates beyond what any single run recovers. • At prioritization (Step 8), the reliability-weighted scores add an evidence-of-validity criterion—the strength of cross-run support—to the benefit and effort the step weighs. • At inspection (Step 9), the fused ranking hands reviewers a prioritized queue whose depth can be matched to the available inspection budget, spending it on the best supported candidates first—while final acceptance remains a human decision. •
FIGURE 3. Evaluation of the fusion strategies (Uniform and Naive-Bayes) and the original outputs (single runs and configuration averages). 14
Finally, the models ensembled here are not frontier models as of 2026—which points to an opportunity: ensembles of cheaper, lower tier, or open-weights, locally operable models may offer a practical route to results otherwise sought from a single frontier model, a direction already visible in systems that assemble open-source models into ensembles surpassing strong proprietary ones [71] and in multi-LLM systems shipped as a single product [72]. And even though the source study framed its model rankings as temporal snapshots rather than enduring hierarchies [4], we posit that the relative advantage of post-generation ensembling over its own constituent runs is the durable part of these results: agreement among imperfect generators remains informative whoever the generators are. The frontier may advance; the result holds.
VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
FIGURE 4. Paired Cumming estimation plot for AP under the requirement-level OOB bootstrap.
B. INTERNAL VALIDATION AND CONSTRUCT STABILITY
The full-corpus improvements of Section VI are apparent estimates, potentially sensitive to the particular corpus of requirements; we therefore examine whether the observed differences persist under out-of-bag resampling and structured perturbations of the available evidence. Following the estimation-statistics frame of Section V-F, the mean paired difference is the effect estimate of each contrast, the p05–p95 range describes the spread of the difference across replicates, and the win rate records how often it is positive; analytic confidence intervals complement these for the two metrics whose distribution theory is available. Fig. 4 provides internal-validation evidence through a paired Cumming estimation plot [73], [74] that presents paired contrasts of AP under the out-of-bag bootstrap procedure of Section V-F. The upper lane shows a paired trajectory plot (slopegraph): each faint line connects the AP values of a given out-of-bag test sample (making the paired structure explicit) across the three approaches; namely, the best single configuration (reselected within each replicate), Uniform fusion, and Naive-Bayes fusion. The superimposed trajectory connects the mean value for each approach. The two lower lanes show the paired difference distributions (deltas) relative to 1) the best configuration and 2) Uniform fusion, each summarized by a half-violin with its p05–p95 bar and mean. Table 4 extends the estimation-plot summary to the four principal metrics, presenting the improvement in two steps: VOLUME 14, 2026
Uniform fusion against the best single configuration, and Naive-Bayes fusion against Uniform fusion. For each comparison it reports the full-data difference; the OOB mean difference with its descriptive p05–p95 range and its win rate over the 200 overlapping replicates; and, where distribution theory permits, an analytic 95% confidence interval (CI) and two-sided p-value. The win rate is the proportion of replicates in which the difference is positive—the empirical probability of superiority Pr(d > 0) [75], with exact ties not counted as wins. The Monte √ Carlo error of each reported OOB mean difference—SD/ B, where SD is the standard deviation of the paired deltas and B = 200 is the number of replicates— stays below 0.007: the reported means are stable with respect to the finite number of replicates, a statement about simulation precision rather than population inference. The first comparison isolates the chorus effect: plain Uniform fusion already improves on the best single configuration on all four metrics in every replicate. Across them, the mean advantage amounts to roughly 3–4.5 times the replicate-toreplicate standard deviation of the paired deltas. Since the baseline is reselected on each replicate’s test sample, a choice that favors it, the comparison is conservative; the advantage is therefore not an artifact of fixing one configuration chosen once on the full dataset. The second comparison is more demanding, because Uniform already exploits the chorus effect; it thus isolates the added value of reliability-aware weighting. The NaiveBayes–Uniform differences in AP, ROC AUC, and max Jw remain positive throughout the central 90% OOB range, with win rates of 99.5%, 99.0%, and 100%, respectively. Max F2 is the weakest of the four contrasts, at a 92.5% win rate, its p05 of zero reflecting a residue of replicates in which the two strategies reach the same maximum; the difference in maximum value and in the review depth at which it is attained is discussed in Section VI-B. The total advantage of Naive-Bayes fusion over the best single configuration—the first difference lane of Fig. 4— amounts, for AP, to a mean OOB difference of 0.158, positive in every replicate. Analytic full-data intervals are available for ROC AUC and the Youden operating-point metrics, and provide a complementary view (Table 4). All four lie entirely above zero: for both metrics, Uniform fusion improves on the best single run—a stricter baseline, since a configuration’s strongest run can only exceed that configuration’s average—and NaiveBayes fusion improves on Uniform fusion. The intervals for the Youden metrics are pointwise traditional (Wald) confidence intervals [54] evaluated at the selected operating-point values for k. They treat sensitivity and specificity as independent binomial proportions estimated on the relevant and nonrelevant candidates, respectively, and they do not adjust for the preceding maximization over k, which also leaves the dependence induced by selecting the operating point on the same data unmodeled [78]. Finally, the joint control-slice/run perturbation (Section V-F) complements OOB refitting by applying the full15
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
TABLE 4. Paired internal-validation differences among the three approaches
OOB bootstrap Metric Full-data ∆ Mean ∆ AP 0.142 0.123 Uniform fusion vs. ROC AUC 0.118 0.111 best configuration Max F2 0.281 0.285 0.326 0.346 Max Jw c AP 0.039 0.035 Naive-Bayes vs. ROC AUC 0.052 0.050 Max F2 0.067 0.060 Uniform fusion d 0.190 0.160 Max Jw c Comparison
Analytic a
p05–p95 Win rate ∆ [0.051, 0.201] 100.0% — [0.048, 0.174] 100.0% 0.083 b [0.172, 0.385] 100.0% — [0.205, 0.496] 100.0% 0.241 b [0.012, 0.069] 99.5% — [0.012, 0.096] 99.0% 0.052 [0.000, 0.114] 92.5% — [0.054, 0.294] 100.0% 0.190
95% CI — [0.020, 0.145] — [0.071, 0.410] — [0.010, 0.094] — [0.055, 0.326]
pa — 0.010 — 0.005 — 0.015 — 0.006
a
A row’s OOB win rate and its analytic p quantify different comparisons and must not be read as one. Compared against the best single run for the metric (paired DeLong for ROC AUC, paired Wald for max Jw [76], [77]), rather than the best configuration used by the OOB columns. c Each side is evaluated at its own maximizing rank k, so a paired difference generally compares different operating points. d All columns share the Uniform baseline; the analytic ∆ thus equals the full-data ∆. b
corpus weight vector after removing one control slice and one run at a time. Under these fixed-weight perturbations, NaiveBayes fusion outperforms Uniform fusion on all four metrics in every one of the 240 samples. The conclusions about global ordering quality and weighted recall–specificity balance are therefore not driven by one particular control slice or one idiosyncratic run. C. SCOPE LIMITATIONS AND VALIDITY CONSTRAINTS
This exploratory study addresses only the post-generation aggregation stage: we reuse a frozen, expert-labeled corpus of generated outputs and evaluate IR-inspired voting and quality-weighted fusion rules over canonicalized requirement sets, holding fixed everything upstream—the elicitation, prompting, and generation that produced those outputs. Its conclusions therefore pertain to the behavior of outputaggregation ensembles, not to the intrinsic capabilities of any model, prompt library, or multi-step generation pipeline. It consequently inherits the external-validity constraints of the source study—a single standard and system, one language, and one expert team’s adjudication. However, the difference in construct attenuates them: our claims concern the relative effect of ensembling with respect to base performance, not the absolute performance of any model, so factors that shift the whole field of runs up or down weigh far less on the within-study contrasts we report than they would on an absolute claim. Moreover, the internal-validation analysis of Section VII-B substantiates this, showing those contrasts persisting under resampling and structured perturbation. Concretely, the empirical setting is a single Englishlanguage case study—ten ISO/IEC 27002 controls for AI4I4, from a text-only system description—with an expertconstructed gold standard and hallucination and canonicalization labels over the judged universe, and a fixed sample of stochastic LLM outputs under specific prompts, decoding settings, and model snapshots. The question our post hoc analysis answers—whether fusing already-produced runs improves the order of review—is posed in the same terms for any system whose requirements are to be specified, and the fusion operators themselves are largely domain agnostic; what varies 16
with different systems, domains, organizational contexts, and standards are the absolute precision–recall trade-offs and the magnitude (and relative ordering) of the ensembling gains, as judgments about relevance, atomicity, and merge/split decisions influence both the labels and the weighting signals derived from them. Evaluation frames ensembling as a ranked-retrieval problem and uses IR-inspired measures at shifting operating points (precision, recall, and fall-out, plus combined metrics such as F2 and Jw , and whole-ranking assessments such as AP and ROC AUC). This metric-focused design is appropriate for comparing fusion operators, but it treats all valid requirements as equally important and does not measure downstream security efficacy, audit outcomes, prioritization value, or human review cost. It is also, by construction, a closed-world evaluation: the evaluation is defined over the finite judged universe U (Section III-B), not the space of all conceivable requirements, and the true negatives are the judged hallucinated candidates that a ranking leaves unselected. This is a construct-validity scope note more than a defect, and it is intrinsically mitigated: the headline quantities are relative contrasts—paired differences and normalized uplifts—computed within the same closed universe for every approach; that universe is fully adjudicated, as every candidate carries an expert label, so, unlike depth-limited retrieval pooling, it contains no unjudged items; and no approach can rank a candidate that lies outside it. What remains genuinely universe relative is the absolute reading: recall and specificity are measured against U, valid requirements exist beyond it, and—since the space of invalid statements is effectively unbounded—an open-world notion of specificity would not be meaningful in any case. That valid requirements exist beyond U is not merely hypothetical: the exploratory source-study configurations excluded from this universe did surface further valid requirements, direct evidence that the requirement space extends past what any participating run produced. The sampling design is also uneven across configurations: each model’s base chain ran as four stochastic runs, whereas its cooler and warmer variants ran once each (Table 1), so the VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
base chain contributes more runs and could bias the comparison in its favor, since more draws yield both more votes in the fusion and more chances to surface rare valid requirements. Two features of the design limit this effect without removing it. First, configurations enter the comparisons through their per-run averages rather than individual runs, so a configuration’s standing reflects its typical run rather than its luckiest draw or sheer number of draws. Second, under Naive-Bayes fusion the vote of a model that behaves less consistently with the labeled corpus is discounted, so an over-sampled variant does not gain influence by volume alone unless its votes are also reliable; the weighting is per model, however, and does not equalize sampling among a model’s own temperature variants. Still, the more sampled variant retains a genuine recall advantage from its extra draws, so some temperaturedriven differences may reflect the number of runs rather than the decoding settings alone; replications that equalize runs per variant—or subsampling-based sensitivity analyses—would further strengthen causal claims. Relatedly, run-to-run variability is, in this design, less a threat to reproducibility than an exploited asset: because independent runs recover overlapping but not identical candidate sets, additional runs contribute complementary valid requirements and raise the recall available to fusion. This complementarity is visible in the source study’s consistency analysis [4]: within-configuration mean pairwise Jaccard similarity across runs ranges from 0.402 to 0.600 when hallucinations are included, rising to 0.647–0.788 once they are removed, while between-model similarity is substantially lower (≈0.219, or 0.387 without hallucinations). That is, runs converge more on true positives than on their idiosyncratic hallucinations (runs of the same model agree only moderately, and different models agree less still)—precisely the diversity, concentrated on the valid candidates, that voting-based fusion converts into ranked evidence. The two fusion strategies differ in how much they depend on the data at hand. Uniform fusion uses fixed, datasetindependent weights and requires no fitting; Naive-Bayes fusion derives its weights from the labeled corpus and is thus dataset dependent. The validity of the latter rests on two controls reported in Section VII-B: out-of-bag crossfitting, which trains the weights on a subset disjoint from the one used for evaluation, and a jackknife-like control-slice/run perturbation that applies the fixed full-corpus weights after removing one control slice and one run at a time. NaiveBayes is moreover proposed as a general weighting rule— one that improves on Uniform fusion, which in turn improves on any single configuration—rather than as the best attainable fusion; better fitting weights exist for this corpus (Section IV-B), but pursuing them would amount to overfitting this particular dataset rather than improving the general rule. Two of the reference curves against which the strategies are plotted—the pattern oracle and the numerical linear-fusion benchmark (Section V-E)—might be mistaken for improved fusion approaches, but, like the other reference curves, they presuppose the observed labels and so are diagnostic bounds, VOLUME 14, 2026
not deployable methods. The oracle marks an information bound that any vote-based scorer faces; the numerical benchmark estimates the headroom of linear weighting; and neither is a proposed method—in particular, the label-tuned benchmark contrasts with the article’s proposed Naive-Bayes weighting, which is validated by cross-fitting. A final caveat is temporal. The absolute results and the model ranking are a snapshot of a fast-moving field, as the source study likewise noted. We posit—and develop at greater length in Section VII-A—that the durable part is the relative advantage of ensembling over its own constituent runs, since that advantage is a property of run and model diversity rather than of any particular model, and should recur when today’s constituents are replaced by newer ones. This reading places our results within the broader evidence that aggregating the outputs of several imperfect LLMs improves on any single one [43], [45], [37], a regularity not tied to a specific model generation; the magnitude of the gain for stronger future constituents remains an open empirical question. Taken together, these constraints do not undermine the central contribution of this article—a controlled evaluation of voting and NaiveBayes post hoc aggregation over LLM runs—but they motivate the replications and the risk-aware, interactive-workflow extensions outlined in Section VIII. VIII. CONCLUSION
In this article, we have revisited a manually labeled corpus of cybersecurity requirements generated for ISO/IEC 27002 controls on the AI4I4 automotive logistics system, now focusing on a narrowly defined goal: how to improve the results by aggregating multiple LLM outputs once they have already been produced. By treating each LLM run as a black-box retrieval system over a canonicalized requirement universe, we framed post-generation ensembling as an information-retrieval data-fusion problem and evaluated voting- and quality-weighted ranking rules across 12 configurations from four model families. The results bear out the perspective advanced in the introduction: treating the runs’ disagreement as complementary evidence rather than evaluation noise, post hoc aggregation converts inter-run variability into a practical coverage advantage—and it does so chiefly by improving the order in which requirements are reviewed, besides the sheer volume recovered. Relative to the best single configuration, even the plainest form of agreement—Uniform fusion, which merely rewards candidate requirements supported by several runs—substantially increases recall at representative operating points without a matching rise in hallucinations. Layering model-reliability-aware weights on top improves the ordering further: by discounting support from less reliable sources, Naive-Bayes fusion pushes valid requirements ahead of hallucinations more decisively—visible as higher AP and ROC AUC—and thus offers better practical precision–recall trade-offs, reaching comparable coverage at shallower review depths. Internal validation indicates that these gains are stable, staying positive in at least 92% of out-of-bag resamples 17
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
and in every structured perturbation—though as an internal check it bears on robustness over this corpus, not external generality. Taken together, the study supports a lightweight, postgeneration workflow for AI-augmented security requirements generation—one that leaves the upstream models and prompts untouched and adds no model retraining or finetuning. Rather than committing to a single carefully tuned configuration, practitioners can sample multiple stochastic runs across heterogeneous models and prompts and then apply voting-based fusion with a reliability-aware weighting layer (Naive-Bayes fusion with per-source noise rates) to order items for human review. This preserves the diversity needed to surface rare or long-tail requirements while ranking likely valid content ahead of hallucinations. Such a workflow also integrates naturally into established security requirements engineering processes: the fused ranking simply feeds their existing elicitation, prioritization, and inspection activities with a better ordered candidate queue. Future work can extend this post-generation view in three directions. First, replication across additional standards, system artifacts, and languages would test the generality of the ensembling findings. Second, the reliability estimates could be complemented with control-specific checks that flag crossmodel, plausible-but-off-scope requirements able to survive pure consensus. Third, integrating these fusion operators into interactive analyst workflows would enable dynamic operating-point selection and clarify how ranked ensemble outputs translate into real compliance effort and risk reduction. Beyond these research directions, a production deployment would naturally go further than the generic Naive-Bayes rule evaluated here, calibrating the fusion weights to its own models, corpus, and accumulating adjudication feedback. In summary, this article shows that IR-inspired data fusion provides a principled and effective post hoc layer for aggregating LLM-generated cybersecurity requirements: the disagreement of stochastic generators is itself evidence, and it can be read after the fact, from the outputs alone. Postgeneration ensembling thus stands as an optimization layer in its own right—orthogonal to advances in prompting, finetuning, retrieval augmentation, or larger foundation models— ready to complement whichever of them a pipeline adopts. APPENDIX. REPRODUCIBLE RESEARCH
All artifacts required to replicate the evaluation reported here—the frozen candidate corpus, the ensembling code, and the computed metrics, curves, and result tables—are archived in a dedicated Zenodo deposit for this study (https://doi.org/ 10.5281/zenodo.21496481). This deposit is separate from the reproducible-research package of the source study that produced the underlying LLM runs and gold standard [47], which we reuse with its authors’ subsequent annotation corrections. Our artifacts refer to each candidate requirement—the 72 vetted valid requirements and the 111 hallucinated ones that together make up the judged universe—only by an identifier; the corresponding requirement text is resolved against that 18
source package. All ensembling reported here is offline post-processing over that frozen corpus: no new LLM calls are made and no requirements are regenerated. Every strategy operates only on the recorded per-run presence of each candidate, so the archived data fully determine the reported results. Because the underlying repository is a broader benchmarking toolkit, it computes more than this article reports—further ranking metrics, additional weighting families, and numerical optimizers among them. The article draws only on Uniform and Naive-Bayes fusion evaluated with the core metrics; the repository’s README file (supplied in the Zenodo record) documents the remaining strategies, metrics, and analyses, along with the mapping from their internal names to the labels used here, as well as a complete directory map, setup instructions, and usage examples. ACKNOWLEDGMENT
AI usage disclosure. During the preparation of this article, the authors worked in an iterative cycle with AI assistants— Anthropic Claude Code, OpenAI ChatGPT, GitHub Copilot, and OpenAI Codex—on writing, coding, and research tasks spanning all sections of the manuscript: the text and the analysis code grew through continuously interleaved authorand AI-written revisions, each contribution reviewed and built upon in the next round, with dedicated passes by adversarially instructed AI agents complementing the authors’ own reviews. All article contents and all significant code were revised by the authors, who are fully accountable for them. Source dataset. This study builds on the dataset of the authors’ previous work [4]: the recorded LLM runs and the expert-adjudicated gold standard are reused here, subject only to the annotation corrections recorded since. REFERENCES [1] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, ‘‘Large language models for software engineering: Survey and open problems,’’ in Proc. 2023 IEEE/ACM Int. Conf. Softw. Eng.: Future of Software Engineering (ICSE-FoSE), 2023, pp. 31–53. [2] H. Cheng, J. H. Husen, Y. Lu, T. Racharak, N. Yoshioka, N. Ubayashi, and H. Washizaki, ‘‘Generative AI for requirements engineering: A systematic literature review,’’ Software: Practice and Experience, vol. 56, no. 2, pp. 141–170, 2026. [3] J. Sallou, T. Durieux, and A. Panichella, ‘‘Breaking the silence: The threats of using LLMs in software engineering,’’ in Proc. 2024 IEEE/ACM Int. Conf. Softw. Eng.: New Ideas and Emerging Results (ICSE-NIER), 2024, pp. 102–106. [4] J. C. Yelmo, Y.-S. Martín, and S. Perez-Acuna, ‘‘Experimental evaluation of AI-augmented cybersecurity requirements generation leveraging LLMs’ capabilities,’’ IEEE Access, vol. 14, pp. 19 579–19 606, 2026. [5] A. Hemmat, M. Sharbaf, S. Kolahdouz-Rahimi, K. Lano, and S. Y. Tehrani, ‘‘Research directions for using LLM in software requirement engineering: A systematic review,’’ Front. Comput. Sci., vol. 7, 2025, Art. no. 1519437. [6] M. A. Zadenoori, J. Dabrowski, W. Alhoshan, L. Zhao, and A. Ferrari, ‘‘Large language models (LLMs) for requirements engineering (RE): A systematic literature review,’’ arXiv, Sep. 2025. [7] J. J. Norheim, E. Rebentisch, D. Xiao, L. Draeger, A. Kerbrat, and O. L. de Weck, ‘‘Challenges in applying large language models to requirements engineering tasks,’’ Design Science, vol. 10, 2024, Art. no. e16. [8] M. Borg, ‘‘Requirements engineering and large language models: Insights from a panel,’’ IEEE Softw., vol. 41, no. 2, pp. 6–10, 2024. VOLUME 14, 2026
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
[9] K. Rindell, K. Bernsmed, and M. G. Jaatun, ‘‘Managing security in software: Or: How I learned to stop worrying and manage the security technical debt,’’ in Proc. 14th Int. Conf. Availability, Reliability and Security (ARES), Canterbury, U.K., 2019, pp. 1–8. [10] M. Krishna, B. Gaur, A. Verma, and P. Jalote, ‘‘Using LLMs in software requirements specifications: An empirical evaluation,’’ in 2024 IEEE 32nd Int. Requirements Engineering Conf. (RE), Reykjavik, Iceland, 2024, pp. 475–483. [11] N. Blasek, K. Eichenmüller, B. Ernst, N. Götz, B. Nast, and K. Sandkuhl, ‘‘Large language models in requirements engineering for digital twins,’’ in Companion Proc. 16th IFIP WG 8.1 Working Conf. Practice of Enterprise Modeling (PoEM), ser. CEUR Workshop Proceedings, vol. 3645. Vienna, Austria: CEUR-WS.org, 2023. [Online]. Available: https://ceur-ws.org/Vol-3645/dte1.pdf [12] C. Hymel and H. Johnson, ‘‘Analysis of LLMs vs human experts in requirements engineering,’’ arXiv, 2025. [13] A. Ferrari and P. Spoletini, ‘‘Formal requirements engineering and large language models: A two-way roadmap,’’ Inf. Softw. Technol., vol. 181, 2025, Art. no. 107697. [14] J. Cao, Y. Lu, M. Li, H. Ma, H. Li, M. He, C. Wen, L. Sun, H. Zhang, S. Qin, S.-C. Cheung, and C. Tian, ‘‘From informal to formal—incorporating and evaluating LLMs on natural language requirements to verifiable formal proofs,’’ in Proc. 63rd Annu. Meeting Assoc. Comput. Linguistics (ACL), Vol. 1: Long Papers, Vienna, Austria, 2025, pp. 26 984–27 003. [15] H. Xu, S. Wang, N. Li, K. Wang, Y. Zhao, K. Chen, T. Yu, Y. Liu, and H. Wang, ‘‘Large language models for cyber security: A systematic literature review,’’ ACM Transactions on Software Engineering and Methodology, 2025. [16] J. Zhang, H. Bu, H. Wen, Y. Liu, H. Fei, R. Xi, L. Li, Y. Yang, H. Zhu, and D. Meng, ‘‘When LLMs meet cybersecurity: A systematic literature review,’’ Cybersecurity, vol. 8, no. 1, 2025, Art. no. 55. [17] Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, ‘‘A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly,’’ High-Confidence Computing, vol. 4, no. 2, 2024, Art. no. 100211. [18] D. M. Divakaran and S. T. Peddinti, ‘‘Large language models for cybersecurity: New opportunities,’’ IEEE Security & Privacy, vol. 23, no. 5, pp. 38–45, 2025. [19] L. Belzner, T. Gabor, and M. Wirsing, ‘‘Large language model assisted software engineering: Prospects, challenges, and a case study,’’ in Bridging the Gap Between AI and Reality (AISoLA 2023), ser. Lecture Notes in Computer Science, vol. 14380. Cham, Switzerland: Springer, 2024, pp. 355–374. [20] I. Hasanov, S. Virtanen, A. Hakkala, and J. Isoaho, ‘‘Application of large language models in cybersecurity: A systematic literature review,’’ IEEE Access, vol. 12, pp. 176 751–176 778, 2024. [21] N. O. Jaffal, M. Alkhanafseh, and D. Mohaisen, ‘‘Large language models in cybersecurity: A survey of applications, vulnerabilities, and defence techniques,’’ AI, vol. 6, no. 9, Sep. 2025, Art. no. 216. [22] F. N. Motlagh, M. Hajizadeh, M. Majd, P. Najafi, F. Cheng, and C. Meinel, ‘‘Large language models in cybersecurity: State-of-the-art,’’ in Proc. 11th Int. Conf. Inf. Syst. Security and Privacy (ICISSP), 2025, pp. 98–110. [23] M. A. Ferrag, F. Alwahedi, A. Battah, B. Cherif, A. Mechri, N. Tihanyi, T. Bisztray, and M. Debbah, ‘‘Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities,’’ Internet of Things and Cyber-Physical Systems, vol. 5, pp. 1–46, 2025. [24] Z. Sheng, Z. Chen, S. Gu, H. Huang, G. Gu, and J. Huang, ‘‘LLMs in software security: A survey of vulnerability detection techniques and insights,’’ ACM Computing Surveys, vol. 58, no. 5, 2026, Art. no. 134. [25] H. Ji, J. Yang, L. Chai et al., ‘‘SEvenLLM: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence,’’ arXiv, 2024. [26] A. F. Subahi, ‘‘Enhancing software sustainability: Leveraging large language models to evaluate security requirements fulfillment in requirements engineering,’’ Systems, vol. 13, no. 2, 2025, Art. no. 114. [27] A. Melo, L. Almeida, and L. Garcés, ‘‘Investigating the use of large language models in software security requirements: Results of a literature review,’’ in Anais do IV Workshop Brasileiro de Engenharia de Software Inteligente (ISE 2025), 2025, pp. 37–42. [28] T. Alsanoosy, ‘‘Large language model for requirements classification: An ensemble approach,’’ Procedia Computer Science, vol. 270, pp. 3648– 3657, 2025. [29] C. C. Vogt and G. W. Cottrell, ‘‘Fusion via a linear combination of scores,’’ Information Retrieval, vol. 1, no. 3, pp. 151–173, 1999. VOLUME 14, 2026
[30] S. Wu, Data Fusion in Information Retrieval, ser. Adaptation, Learning, and Optimization. Berlin, Germany: Springer, 2012, vol. 13. [31] J. A. Shaw and E. A. Fox, ‘‘Combination of multiple searches,’’ in Proc. 3rd Text REtrieval Conf. (TREC-3), Gaithersburg, MD, USA, 1995, pp. 105–108. [32] N. J. Belkin, P. Kantor, E. A. Fox, and J. A. Shaw, ‘‘Combining the evidence of multiple query representations for information retrieval,’’ Information Processing & Management, vol. 31, no. 3, pp. 431–448, 1995. [33] D. Lillis, F. Toolan, R. Collier, and J. Dunnion, ‘‘ProbFuse: A probabilistic approach to data fusion,’’ in Proc. 29th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retr. (SIGIR), Seattle, WA, USA, 2006, pp. 139–146. [34] F. Matloob, T. M. Ghazal, N. Taleb, S. Aftab, M. Ahmad, M. A. Khan, S. Abbas, and T. R. Soomro, ‘‘Software defect prediction using ensemble learning: A systematic literature review,’’ IEEE Access, vol. 9, pp. 98 754– 98 771, 2021. [35] S. Wang, M. Geng, B. Lin, Z. Sun, M. Wen, Y. Liu, L. Li, T. F. Bissyandé, and X. Mao, ‘‘Fusing code searchers,’’ IEEE Transactions on Software Engineering, vol. 50, no. 7, pp. 1852–1866, 2024. [36] T.-D. B. Le, R. J. Oentaryo, and D. Lo, ‘‘Information retrieval and spectrum based bug localization: Better together,’’ in Proc. 10th Joint Meeting Found. Softw. Eng. (ESEC/FSE), Bergamo, Italy, 2015, pp. 579–590. [37] Z. Chen, X. Lu, J. Li et al., ‘‘Harnessing multiple large language models: A survey on LLM ensemble,’’ arXiv, 2025. [38] H. Jin, L. Huang, H. Cai, J. Yan, B. Li, and H. Chen, ‘‘From LLMs to LLMbased agents for software engineering: A survey of current, challenges and future,’’ arXiv, 2024. [39] K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou, ‘‘Routing to the expert: Efficient reward-guided ensemble of large language models,’’ in Proc. 2024 Conf. North American Chapter Assoc. Comput. Linguistics (NAACL), Vol. 1: Long Papers, Mexico City, Mexico, 2024, pp. 1964–1974. [40] L. Chen, M. Zaharia, and J. Zou, ‘‘FrugalGPT: How to use large language models while reducing cost and improving performance,’’ Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id=cSimKw5p6R [41] Y. Huang, X. Feng, B. Li, Y. Xiang, H. Wang, T. Liu, and B. Qin, ‘‘Ensemble learning for heterogeneous large language models with deep parallel collaboration,’’ in Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024, pp. 119 838–119 860. [42] S. Park, X. Liu, Y. Gong, and E. Choi, ‘‘Ensembling large language models with process reward-guided tree search for better complex reasoning,’’ in Proc. 2025 Conf. Nations of the Americas Chapter Assoc. Comput. Linguistics (NAACL), Vol. 1: Long Papers, Albuquerque, NM, USA, 2025, pp. 10 256–10 277. [43] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, ‘‘Self-consistency improves chain of thought reasoning in language models,’’ in Proc. 11th Int. Conf. Learning Representations (ICLR), 2023. [Online]. Available: https: //openreview.net/forum?id=1PL1NIMMrw [44] D. Jiang, X. Ren, and B. Y. Lin, ‘‘LLM-blender: Ensembling large language models with pairwise ranking and generative fusion,’’ in Proc. 61st Annu. Meeting Assoc. Comput. Linguistics (ACL), Vol. 1: Long Papers, Toronto, ON, Canada, 2023, pp. 14 165–14 178. [45] J. Li, Q. Zhang, Y. Yu, Q. Fu, and D. Ye, ‘‘More agents is all you need,’’ Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id=bgzUSZ8aeg [46] H. Nasser, F. Trad, and A. Chehab, ‘‘Stacking large language models is all you need: A case study on phishing URL detection,’’ Journal of Artificial Intelligence and Soft Computing Research, vol. 15, no. 4, pp. 337–356, 2025. [47] J. C. Yelmo, Y.-S. Martín, and S. Perez-Acuna, ‘‘AI-augmented cybersecurity requirements generation using LLMs | reproducible research package,’’ 2025. [Online]. Available: https://doi.org/10.5281/zenodo.15641294 [48] C. W. Cleverdon, ‘‘The cranfield tests on index language devices,’’ Aslib Proceedings, vol. 19, no. 6, pp. 173–194, 1967. [49] E. M. Voorhees and D. K. Harman, Eds., TREC: Experiment and Evaluation in Information Retrieval. Cambridge, MA, USA: MIT Press, 2005. [50] M. Sanderson, ‘‘Test collection based evaluation of information retrieval systems,’’ Foundations and Trends in Information Retrieval, vol. 4, no. 4, pp. 247–375, 2010. [51] C. J. van Rijsbergen, Information Retrieval, 2nd ed. London, U.K.: Butterworths, 1979. [52] P. Christen, D. J. Hand, and N. Kirielle, ‘‘A review of the f-measure: Its history, properties, criticism, and alternatives,’’ ACM Computing Surveys, vol. 56, no. 3, pp. 73:1–73:24, 2023. 19
Perez-Acuna et al.: Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation
[53] W. J. Youden, ‘‘Index for rating diagnostic tests,’’ Cancer, vol. 3, no. 1, pp. 32–35, 1950. [54] D.-L. Li, F. Shen, Y. Yin, J.-X. Peng, and P.-Y. Chen, ‘‘Weighted youden index and its two-independent-sample comparison based on weighted sensitivity and specificity,’’ Chinese Medical Journal, vol. 126, no. 6, pp. 1150–1154, 2013. [55] M. Greiner, D. Pfeiffer, and R. D. Smith, ‘‘Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests,’’ Preventive Veterinary Medicine, vol. 45, no. 1–2, pp. 23–41, 2000. [56] N. J. Perkins and E. F. Schisterman, ‘‘The inconsistency of ‘‘optimal’’ cutpoints obtained using two criteria based on the receiver operating characteristic curve,’’ American Journal of Epidemiology, vol. 163, no. 7, pp. 670–675, 2006. [57] J. H. Lee, ‘‘Analyses of multiple evidence combination,’’ in Proc. 20th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retr. (SIGIR), Philadelphia, PA, USA, 1997, pp. 267–276. [58] E. A. Fox and J. A. Shaw, ‘‘Combination of multiple searches,’’ in Proc. 2nd Text REtrieval Conf. (TREC-2), ser. NIST Special Publication 500-215, Gaithersburg, MD, USA, 1994, pp. 243–252. [59] S. E. Robertson and K. Sparck Jones, ‘‘Relevance weighting of search terms,’’ Journal of the American Society for Information Science, vol. 27, no. 3, pp. 129–146, 1976. [60] A. P. Dawid and A. M. Skene, ‘‘Maximum likelihood estimation of observer error-rates using the EM algorithm,’’ Journal of the Royal Statistical Society. Series C (Applied Statistics), vol. 28, no. 1, pp. 20–28, 1979. [61] T. Fawcett, ‘‘An introduction to ROC analysis,’’ Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006. [62] J. Davis and M. Goadrich, ‘‘The relationship between precision-recall and ROC curves,’’ in Proc. 23rd Int. Conf. Mach. Learn. (ICML), Pittsburgh, PA, USA, 2006, pp. 233–240. [63] National Institute of Standards and Technology (NIST), ‘‘Common evaluation measures,’’ Appendix A (Evaluation Measures), TREC 2007 Results (TREC 16 Proceedings material), 2007. [Online]. Available: https://trec.nist.gov/pubs/trec16/appendices/measures.pdf [64] F. McSherry and M. Najork, ‘‘Computing information retrieval performance measures efficiently in the presence of tied scores,’’ in Advances in Information Retrieval: 30th European Conference on IR Research (ECIR 2008), ser. Lecture Notes in Computer Science, C. Macdonald, I. Ounis, V. Plachouras, I. Ruthven, and R. W. White, Eds. Berlin, Germany: Springer, 2008, vol. 4956, pp. 414–421. [65] D. Bamber, ‘‘The area above the ordinal dominance graph and the area below the receiver operating characteristic graph,’’ Journal of Mathematical Psychology, vol. 12, no. 4, pp. 387–415, 1975. [66] B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap, ser. Monographs on Statistics and Applied Probability. New York, NY, USA: Chapman & Hall/CRC, 1993, vol. 57. [67] D. N. Politis, J. P. Romano, and M. Wolf, Subsampling, ser. Springer Series in Statistics. New York, NY, USA: Springer, 1999. [68] C. Nadeau and Y. Bengio, ‘‘Inference for the generalization error,’’ Machine Learning, vol. 52, no. 3, pp. 239–281, 2003. [69] E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson, ‘‘Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach,’’ Biometrics, vol. 44, no. 3, pp. 837–845, 1988. [70] N. R. Mead, E. D. Hough, and T. R. Stehney II, ‘‘Security quality requirements engineering (SQUARE) methodology,’’ Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, USA, Technical Report CMU/SEI-2005-TR-009, Nov. 2005. [Online]. Available: https://www.sei.cmu.edu/library/ security-quality-requirements-engineering-technical-report/ [71] J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou, ‘‘Mixture-of-agents enhances large language model capabilities,’’ in Proc. 13th Int. Conf. Learning Representations (ICLR), 2025. [Online]. Available: https://proceedings.iclr.cc/paper_files/paper/2025/ hash/5434be94e82c54327bb9dcaf7fca52b6-Abstract-Conference.html [72] Y. Tang, E. Cetin, J. Xu, Q. Sun, S. Nielsen, V. Richard, H. Goda, I. Tymchenko, N. Nguyen, H. Lee, M. Ashiga, S. Kotyan, S. Kuroki, and T. Clanuwat, ‘‘Sakana Fugu technical report,’’ arXiv, 2026. [73] G. Cumming, Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis, ser. Multivariate Applications Series. New York, NY, USA: Routledge, 2012. [74] J. Ho, T. Tumkaya, S. Aryal, H. Choi, and A. Claridge-Chang, ‘‘Moving beyond P values: Data analysis with estimation graphics,’’ Nature Methods, vol. 16, no. 7, pp. 565–566, 2019. 20
[75] K. O. McGraw and S. P. Wong, ‘‘A common language effect size statistic,’’ Psychological Bulletin, vol. 111, no. 2, pp. 361–365, 1992. [76] D. Wenzel and A. Zapf, ‘‘Difference of two dependent sensitivities and specificities: Comparison of various approaches,’’ Biometrical Journal, vol. 55, no. 5, pp. 705–718, 2013. [77] H. Zhou and G. Qin, ‘‘Confidence intervals for the difference in paired Youden indices,’’ Pharmaceutical Statistics, vol. 12, no. 1, pp. 17–27, 2013. [78] L. E. Bantis, C. T. Nakas, and B. Reiser, ‘‘Construction of confidence regions in the ROC space after the estimation of the optimal Youden indexbased cut-off point,’’ Biometrics, vol. 70, no. 1, pp. 212–223, 2014.
SANTIAGO PEREZ-ACUNA is currently a Predoctoral Researcher with the Department of Telematics Engineering, Universidad Politécnica de Madrid (DIT-UPM), Spain. He received the B.Sc. degrees in computer engineering and in business administration from the University of Vigo, Spain, and the M.Sc. degree in network and telematic services engineering from the Universidad Politécnica de Madrid. His research focuses on the application of artificial intelligence to cybersecurity in industrial software systems. His main interests include AI-assisted requirements engineering, large language models, and the development of secure and trustworthy software in Industry 4.0 contexts.
YOD-SAMUEL MARTÍN (M’14) received the M.Sc. degree in telecommunications engineering from the Universidad Politécnica de Madrid (UPM), Spain, in 2004, the M.Phil. degree in 2006, and the Ph.D. degree in 2022. He has held different research and lecturing positions at UPM, in the Center for Open Middleware (COM), Information Processing and Telecommunications Center (IPTC), and Departamento de Ingeniería de Sistemas Telemáticos (DIT), where he is currently an Assistant Professor. His research interests include the management of different categories of extra-functional requirements (cybersecurity, privacy, accessibility, usability, internationalization) throughout the development lifecycle.
JUAN C. YELMO received the M.S. and Ph.D. degrees in telecommunications engineering from the Universidad Politécnica de Madrid, Madrid, Spain, in 1991 and 1996, respectively. He is currently an Associate Professor with the Universidad Politécnica de Madrid, where he has over 25 years of experience in software and internet services engineering. He has participated in numerous national and international research projects and has been involved in technology transfer and innovation activities, including research contracts with international companies, standardization efforts, open innovation initiatives, and consulting for university spinoffs. His current research interests include cybersecurity and AI-augmented software engineering.
VOLUME 14, 2026