Automating Quality Assessment of LLM-Generated Defeaters
arXiv:2607.06039v1 [cs.SE] 7 Jul 2026
T. Rohlinger, D. Ratiu, and S. Wagner
Abstract—High-integrity systems, such as autonomous vehicle fleets or large-scale energy infrastructures, rely on structured assurance cases, represented as rooted directed acyclic graphs (DAGs), to justify safety claims. To remain valid in the face of change, these cases must be robust against potential challenges, known as ’defeaters’. While large language models (LLMs) have recently enabled scalable generation of such defeaters, validating their quality remains a predominantly manual and subjective process. This paper presents an automated method for assessing LLM-generated defeaters using natural language processing (NLP) techniques. Our approach combines structural analysis of assurance case graphs with vector-based semantic embeddings and meta-classifiers trained on expert-assessed consensus defeaters. We evaluate our method through two case studies in the automotive and energy domains, quantifying human reviewer dissensus using Cohen’s kappa (κ < 0.442), which indicates low inter-rater agreement. Our automated approach achieves greater consistency with individual raters, improving (κ ≈ 40%). The method delivers an average F1-score of 0.84 across validation, reducing subjective variance through scalable, objective assessment. Our approach aims to advance the tool support for automation of assurance case synthesis. Index Terms—assurance case, defeater, NLP
inter-rater agreement calculated by cohens kappa [8]. Low inter-rater agreement indicates that expert judgments vary due to differing interpretations and domain knowledge. Furthermore, manual validation is resource-intensive and lacks scalability, particularly for dynamic systems with evolving safety boundaries, such as adaptive software. These limitations hinder the efficiency and reliability of assurance synthesis, making automated approaches necessary to enhance defeater validation. This work addresses the issue of subjectivity in manual defeater validation, as illustrated by the schematic figure 1. In Section III, we propose an NLP-based method that uses BERT embeddings [9] and meta-classifiers to objectively evaluate LLM-generated defeaters, enabling scalable assurance. Our experiment, detailed in Section IV, compares this approach with human evaluations and demonstrates improved consistency and reduced subjectivity in expert dissent. The results in Section V provide further support for the conclusions in Section VI with an outlook on future work.
I. I NTRODUCTION The research community aims to establish a validation method for safety case fragments, assuring over time-changing safety-critical systems that adapt to evolving safety boundaries [1], [2]. The pace of adaptation required is determined by the exposure of entities in, for example, autonomous driving fleets. These kinds of systems demand robust safety cases, enhanced by runtime monitoring feedback [3], thereby enabling advanced system-level assurance [4]. Defeaters, in the domain of safety assurance, refer to arguments or evidence that undermine the assurance claims made within the safety case [5]. Identifying and understanding these defeaters are essential for validating a safety case’s resilience against realworld scenarios. Manual creation and validation in this context is not scalable and relies on subjective expert judgment [6]. Generating defeaters using large language models (LLMs) increases confidence in the argumentation [7], thereby making safety assurance scalable and responsive. In Viger et al. (2024) [7], the generated defeaters were manually reviewed by two safety experts. The experts only agreed on the quality of the defeaters, with a low level of
Fig. 1. This is an assurance case graph in a goal-structured notation style [10], showing goals and sub-components as nodes (left). The diagram illustrates the manual reevaluation process of the generated defeaters and proposed automated AI assessment to mitigate subjective bias (right). We evaluated the approach on two assurance case studies (background), the Adaptive Cruise Control (ACC) (left) from [7] and Large Hadron Collider (CERN LHC) (right) from [11].
II. R ELATED W ORK © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Published version: doi:10.1109/ICSRS68021.2025.11422208.
LLM-generated Assurance Fragments Recent studies, such as Sivakumar et al. (2023) [12], demonstrated GPT4’s moderate success in generating Goal Structuring Notation (GSN)-based safety cases, a graph structure style, limited by
incomplete domain knowledge [13]. Kurzidem et al. (2023) [14] used meta-models to enable a holistic safety analysis to quantify the uncertainties. Gohar et al. (2024) [15] showed preliminary success in using LLMs to identify defeaters, while Shahandashti et al. (2024) [16] highlighted GPT-4 Turbo’s capability to generate domain-specific defeaters like software bugs or environmental risks, thereby further advancing automation. Graydon (2025), appendix, finds the redefinition of defeaters as hazards or failure conditions, distinct from logical flaws, suggesting that generic defeater searches may not replace safety analysis [17]. Defeaters are reasons that challenge a claim through defeasible reasoning [18]. Embeddings for Assurance Cases Fragments The Transformer architecture [19] has revolutionized natural language processing by enabling large language models (LLMs), such as GPT-4, to generate contextual embeddings that capture intricate semantic and syntactic relationships [9]. These embeddings serve as a foundation for advanced techniques like Named Entity Recognition (NER) [20]. By leveraging the contextual richness of LLM-generated embeddings, assurance case development can produce clear, domain-specific narratives within frameworks like Goal Structuring Notation (GSN), ensuring both structural integrity and semantic accuracy [13]. Existing research often overlooks automated quality assessments of generated claims and the use of advanced classification techniques to validate them. Recent studies that apply LLMs to the analysis of safety-critical systems include goal modeling [21], automated software proof generation [22], and LLM-driven defeater identification [7]. Odu et al. (2025) [23] compared LLM outputs and assurance cases, finding that outputs are ”relatively good” but not sufficient for certifying mission-critical systems without human intervention. Varadarajan et al. (2024) [24] propose a continuous assurance framework that emphasizes validity and soundness, employing LLMs to extract properties and environmental information from assurance case texts, processed by Prolog-based reasoning engines for semantic and logical analysis. While effective for structured validation, such hybrid approaches often rely on LLMs for natural language syntactic parsing rather than direct context semantic analysis. Verification and Validation Eliminative Argumentation (EA) in this regard provides a structured approach to constructing and validating assurance arguments by eliminating defeaters [18]. This iterative process builds confidence in claims by systematically addressing doubts, making it suitable for automation through NLP. Ensuring LLM reliability requires robust validation against real-world data, expert evaluations, and alignment with standards like ISO 26262, ISO 21448, and UL4600 [25]–[27]. These standards enhance the administration of domain-specific safety case generation and independent body compliance, confirming stakeholder acceptance. Rushby et al. [6] emphasize evaluating assurance arguments beyond compliance, advocating for semantic robustness. We assume that semantic robustness can be enhanced through precise claim localization in DAGs and objective defeater embedding analysis by NLP techniques. Validation strategies,
as discussed in [28], address risks such as output inaccuracy, lack of systematic coding frameworks, and ethical concerns in the validation and verification of LLM-generated code, emphasizing the need for rigorous processes to ensure reliable and safe outcomes. An iterative process builds confidence [6] in safety-critical systems, such as autonomous driving, by addressing doubts to support claims like “the system is safe”. Viger et al. (2024) [7] leveraged EA to generate defeaters for assurance cases, providing input for our work. Building on the defeaters generated by Viger et al., our approach uses the BERT architecture [9] to automate their evaluation and validates it with a trained model on expert judgment. This case study lays the foundation for experiments that can automatically evaluate LLM-generated defeater performance in safety-critical contexts. Summary Existing studies focus on generating assurance case fragments but do not address the systematic evaluation of their quality, leaving a gap in reliable and objective validation. In this work, we focus on automating the quality assessment of synthesized assurance arguments and the use of classifier strategies to validate defeaters. III. M ETHOD This section presents the methodology for assessing the quality of defeaters within assurance cases using Natural Language Processing (NLP) techniques. Our methodology extends the AI-Supported Eliminative Argumentation (AI-EA) framework by Viger et al. [7] comprising three phases: (1) the foundation experiment establishing a baseline through manual review, we will present a small insights from the provided dataset from Viger et al. to clarify the manual review task; (2) an inter-rater agreement analysis to quantify consensus and dissensus of the manual review; and (3) our novel extention, the automated review process. A. Manual Review Process The AI-Supported Eliminative Argumentation (AI-EA) study dataset includes 171 AI-generated defeaters from two industrial assurance cases (ACs): an Adaptive Cruise Control (ACC) system from the automotive domain including (317 nodes) and the CERN Large Hadron Collider Machine Protection System (MPS) from the nuclear domain with (506 nodes) [29]. Each generated defeater was structurally analyzed as a textual counterargument within the Eliminative Argumentation framework [18], comprising three components: “What” (the flaw), “Where” (the affected claim or system component), and “Why” (the rationale for the challenge). Reviewers scored these components as 0 (absent or non-compliant), 1 (present but generic), or 2 (specific and reasonable). Addressing defeaters strengthens safety arguments, thereby enhancing AC confidence. Below, we provide two example defeaters to illustrate their structure and the manual analysis process, as detailed in the study by Viger et al.. • ACC Defeater ID 1: “The effectiveness of the black box radar sensing and Machine Learning system in maintaining safe distance is uncertain; Where: reliance
on a system whose internal functioning is not analyzed; Why: without understanding the system’s internal functioning, potential failure modes may not be predictable or preventable.” Analysis: Rated 1 (generic) for “What” (vague uncertainty), 0 (non-compliant) for “Where” (lacks specific claim reference), and 2 (reasonable) for “Why” (clear failure mode rationale). This defeater prompts analysis of system behavior to address black box uncertainties. • CERN Defeater ID 1: “What: Total reliance on the Beam Loss Monitoring System (BLMS) for detecting intolerable beam loss; Where: CHILD CLAIM 9; Why: assumes BLMS always provides timely indications, risking system failure if BLMS faults.” Analysis: Rated 2 (reasonable) for “What” and “Where” (specific system and claim), but 1 (generic) for “Why” (broad fault assumption). This defeater flags a single point of failure, necessitating reliability evidence. These examples demonstrate how defeaters were manually analyzed individually for component quality and how addressing them (e.g., through evidence or refined claims) strengthens safety case confidence. B. Consensus and Dissensus Following the manual review process described in Section III-A, we assessed the inter-rater agreement between the two human reviewers (H1 and H2) to quantify the consistency of defeater ratings for the “What,” “Where,” and “Why” components across the ACC and CERN datasets. This analysis establishes the baseline variability in expert judgments, motivating the need for automated validation. We calculated Cohen’s kappa categorical using the cohen kappa score function from the scikit-learn library [30], accounting for agreement by chance. The methodology involved the following steps: 1) Observed Agreement (po ): Computed as the proportion of defeaters where H1 and H2 assigned the same rating (0, 1, or 2) for each component.
2) Marginal Probabilities: Probabilities that each rater assigns a specific rating to a component, regardless of the other rater’s choice. Obtained from the rating distributions. 3) Expected Agreement (pe ): Calculated as the sum of the products of corresponding marginal probabilities for each rating level. 4) Cohen’s Kappa (κ): Determined using the formula: po − p e κ= 1 − pe as described in [8]. For example, the ACC dataset with the “Why” component, reviewers agreed on 42 out of 75 defeaters, with an observed agreement (po = 0.56), and an expected agreement of (pe ≈ 0.57). The rating distributions were: • Reviewer 1: {0: 1, 1: 19, 2: 55} • Reviewer 2: {0: 1, 1: 24, 2: 50} The kappa values were: • What: κ = 0.383, indicating fair agreement. • Why: κ = −0.024, implying disagreement below chance. • Where: κ = 0.442, indicating moderate agreement. For the CERN dataset, the “Why” component, here the reviewers agreed on 61 out of 97 defeaters (po ≈ 0.629), with pe ≈ 0.565. The rating distributions were: • Reviewer 1: {1: 32, 2: 65} • Reviewer 2: {0: 2, 1: 26, 2: 69} The kappa values were: • What: κ = 0.167, indicating slight agreement. • Why: κ = 0.147, indicating slight agreement. • Where: κ = 0.329, indicating fair agreement. These results highlight significant inter-rater variability, particularly the negative kappa for the ACC “Why” component, underscoring the subjectivity in manual evaluations. This variability underscores the need for an automated, structured review process that follows qualitative, objective rating metrics. This will help reduce bias and improve consistency.
TABLE I D EFEATER E VALUATION M ETRICS FOR Q UALITY A SSESSMENT USING NLP Attributes Correctness (C)
Stability (St)
Relevance (R)
Completeness (Comp)
Novelty (N)
Interpretation Description Evaluates how accurately the expert reviews match the other judgments in rating components (”What”, ”Where”, and ”Why”). Associates consistent labels or probabilities for defeaters with high semantic similarity, ensuring stable recognition for defeaters with closely related meanings. Ensures that defeaters are applied to the assurance case and that relevant claims contribute meaningfully to the system’s reliability evaluation. Reviewer checks whether defeaters include necessary components (What, Where, Why) to address issues within the assurance case, ensuring comprehensive and contextually complete defeaters. Evaluates whether defeaters introduce new insights or challenges to avoid redundancy with existing defeaters and to promote originality in identifying potential issues.
Metric Evaluation based on NLP F1 scores assess the accuracy of each class (0, 1 and 2) in comparison to human reviewers H1 and H2. They measure the ability of the models to correctly classify the defeater labels. The proportion of identical predicted labels for defeater pairs with cosine similarity using BERT embeddings. Consistent labels across similar defeaters indicate robust performance. The average cosine similarity between the BERT embeddings of the defeaters and the elements of the assurance case graph (e.g. goal claims, strategies etc. DAG edges) indicates the degree of alignment with the case. The proportion of defeaters with non-empty What and Why components is extracted via text parsing. The Where component is covered by a regular expression filter, ensuring logical and contextual coverage within the assurance case hierarchy. Inverse of the minimum cosine similarity between a defeater’s BERT embedding and those of reference defeaters, where lower similarity suggests greater novelty and reduced redundancy.
C. Quality Assessment of Defeaters As discussed in Subsection III-B, manual validation of defeaters often exhibits significant inter-rater variability, reflecting inconsistencies in the subjective judgements of evaluators. This variability undermines the reliability and reproducibility of assurance case assessments, particularly in safety-critical domains where objective rigour is paramount. To address these issues, we have formalised a set of quality attributes for defeaters derived from established manual evaluation practices. These attributes are designed to be machine-processable, enabling standardised, automated assessment. Table I presents these attributes (left), their Interpretation Description (center), and how these attributes can be quantified by our NLP Metric (right) to enable machine-processable assessment. The attributes are: Correctness aligns automated ratings with human judgments using F1-scores. Stability ensures consistent labels for similar defeaters via BERT embeddings’ cosine similarity. Relevance reflects the defeater’s affiliation within the assurance case graph. Completeness verifies component presence through text parsing, informing field detection. Novelty assesses uniqueness with inverse similarity, diversifying features. This variability-driven approach shapes our automated review design, enhancing the reflective NLP features extraction for the machine learning models. D. Automated Review Process From the identification of the machine-processable quality attributes of the previous subsection III-C, we derived the Automated Review Process. Figure 2 illustrates this process chart with three primary nodes: Manual Review Process, Automated Review Process, and Evaluation, arranged to depict the transition from human assessment to automated validation and its performance assessment. The details of each node and its subcomponents are provided below, along with their positions and functionalities within a machine-learning cycle. This cycle is clustered on the upper rim of the process chart figure. 1) Data Collection: The Manual Review Process, positioned as the leftmost node in Figure 2, encompasses the initial
human-reviewed defeater data mentioned in subsection III-A, comprising the defeaters for each component. The Human node includes two sub-categories: Consensus Defeaters, where reviewers agree (H1=H2), and Dissensus Defeaters, where they disagree (H1xH2), reflecting inter-rater variability as discussed in section Consensus and Dissensus III-B. This bridges to the preprocessing node of the Automated Review Process, the central node in Figure 2. 2) Preprocessing: The embedding preprocessing employs the bert-base-uncased model [31] from Hugging Face, a transformer-based architecture with 12 layers and a 768dimensional output representation, as the initial choice. To generate embeddings for defeaters, the input text is tokenized with a maximum sequence length of 512 tokens. The tokenlevel embeddings produced by the model are aggregated into a single vector representation for each defeater by computing the mean across the token dimension. For creating the semantic relation of embeddings and the top-down hierachical structure, a directed graph of assurance case elements was constructed. This included nodes representing claims, evidence, strategies, contexts, or defeaters, and edges denoting hierarchical dependencies, facilitating the contextual analysis. This data preparation underpins the feature engineering. 3) Feature Engineering: Validity features were engineered to enhance classification, derived from the quality attributes outlined in section III-C Table I. These features support the automated validation process, interpreting the expert judgment in a machine-quantifiable format. Thereby, it provides a transparent, reproducible factor to enable objective evaluation. Is Linked: A binary indicator (1 if the defeater references a linked assurance case element, 0 otherwise), derived from the Relevance and Completeness attribute to assess applicability to the case claims. • Linked Element: Cosine similarity between the defeater’s BERT embeddings and the embedding of its linked assurance case element (if present, Is Linked=1), supporting Relevance by quantifying semantic alignment. • Similarity Path: Cosine similarity between the defeater’s BERT embedding and the concatenated text of the short•
Fig. 2. Automated review process and classifier integration
est path from the assurance case graph’s root to the linked element. Thereby reflecting the Stability attribute by ensuring consistent predictions for related elements. • Similarity Average: Mean cosine similarity between the defeater’s BERT embedding and the assurance case embeddings, enhancing Relevance by capturing overall contextual fit and its opposite Novelty to define whether defeaters are novel. Following, we will introduce the models that were trained on the features presented. 4) Model Selection and Training: Two classification approaches were implemented using scikit-learn: Simple Classifier: The Logistic Regression (LR) employs multinomial softmax regression with L2 regularization for multiclass classification and balanced class weights compensating for data imbalances. LR serves as a baseline or complementary model. Secondary Support Vector Machines (SVMs) that find the optimal hyperplane to separate classes in a feature space were trained on BERT embeddings for the What and Why components. The hyperparameters (kernel: {linear, rbf}, C: {0.1, 1, 10, 100}, gamma: {scale, 0.001, 0.01, 0.1}) were tuned using GridSearchCV (e.g., all combinations of kernel, C, gamma) with 5-fold stratified cross-validation (or fewer if class sizes were small ex. Why 0 class ratings), optimizing for weighted F1-score. These hyperparameters are tuned to find the best SVM configuration for classifying defeater components, balancing fit and generalization. Class imbalance was addressed using class weights, and probabilities were calibrated with CalibratedClassifierCV (sigmoid method), to better reflect true confidence levels. • Meta-Classifier: LR and SVMs combinations were trained on a feature set combining simple classifier probabilities and validity features (is linked, linked element, path similarity, similarity average) as introduced in subsection Feature Engineering III-D3. The same hyperparameter tuning and calibration process was applied as for the Simple Classifier. To address class imbalance, the Synthetic Minority Oversampling Technique SMOTE [32] was used. Additionally, a self-training approach using the SelfTrainingClassifier [33] with a confidence threshold of 0.9 was applied. This threshold determines the minimum confidence required to assign pseudo-labels to unlabeled data, reducing the risk of incorporating incorrect labels, enhancing model robustness.
IV. E XPERIMENT This experiment addresses the context gap between manual and automated validation of generated textual output, specifically defeaters, which are core elements in the argumentation of system safety. Defeaters require subjective review due to their role in challenging safety claims, and this study seeks to recreate such validation using Natural Language Processing (NLP) methods as elaborated in I, to achieve objective, biasindependent results. The forthcoming experiments extend the foundational research executed by Viger et al. [7], which supplies the ACC and CERN assurance cases as well as their corresponding defeater datasets (ACC and CERN) [29].
•
5) Evaluation: With the method section III establishing the automated review process, we will apply it to the ACC and CERN datasets [29], evaluating the performance of the models based on the scikit-learn, providing F1 scores and cohen kappa score Agreement Rates. Improving objectivity in the task performance by leveraging the graph-based embeddings for consistency, to prove the feasibility of AI-aided defeater validation for assurance synthesis.
A. Hypotheses and Research Questions We hypothesize that a vector-based embeddings approach outperforms subjective expert reviews as outlined in [7], in terms of consistency (measured by F1-score) and objectivity (measured by agreement on dissensus defeaters). This addresses the inter-rater variability observed in Section III-B. The experiment investigates two research questions: 1) RQ1: Is the validation of defeaters using an embedding vector-based method comparable to human expert evaluations in terms of accuracy and consistency? 2) RQ2: Does the embedding-based approach reduce subjectivity by providing an objective perspective on safety case defeater ratings? B. Experimental Setup The data [29] was obtained from the experimental results of Viger et al. [7], in which two reviewers collaborated with domain experts to independently analyse the 172 defeaters (75 for ACC and 97 for CERN) generated by GPT-4. These defeaters, covering nine argument fragments, were prompted by three strategies: background information, generic doubts, and context-specific challenges informed by Eliminative Argumentation (EA) [18]. Each defeater comprises “What” (the flaw), “Why” (the rationale), and “Where” (the affected claim or system component). The automated validation rates the “What” and “Why” components as 0 (non-compliant), 1 (generic), or 2 (reasonable), using expert annotations as the ground truth. The methodology, detailed in Section III-D, employs opensource BERT embeddings (“bert-base-uncased”) [9] to transform defeater texts and assurance case structures into 768dimensional vectors. Logistic Regression (LR), Support Vector Machine (SVM), and meta-SVM classifiers predict ratings. Therby incorporating defeater text and structural context in the DAGs (e.g., claim hierarchies via networkx). The experiment proceeds in two rounds: (1) using defeater text alone, and (2) integrating structural context via features to enhance predictions. Performance is evaluated against dissensus annotations (H1̸=H2) using precision, recall, and F1-score, addressing the quality attributes in Table I.
C. Data Preprocessing The ACC and CERN datasets, stored in ACCCERN defeater dataset.xlsx [29], contain 172 defeaters annotated by two human raters (H1, H2) for “What”, “Why”, and “Where” components on a {0, 1, 2} scale, yielding 688 samples (172 defeaters × 2 reviewers × 2 components). The dataset includes columns for Defeater ID, Defeater text, component ratings, and rater information. We split the data into: Consensus Sets: Defeaters where H1 and H2 agree on “What” and “Why” ratings (H1=H2). For ACC, 32 defeaters (31 initial, plus 1 dissensus for class balance); for CERN, 42 defeaters (40 initial, plus 2 dissensus). • Dissensus Sets: Defeaters with mismatched ratings (H1̸=H2) or insufficient samples, used for evaluation (42 for ACC, 50 for CERN after transfers to consensus sets). To address class imbalance in the training sets, we applied SMOTE [32] for the “What” and “Why” components. For each dataset, if a class had fewer than 5 samples, SMOTE was applied with a k-nearest neighbors parameter set to min(3, n − 1) a common default as it balances local information and generalization. Where n is the number of samples in the minority class. If n = 1, SMOTE cannot be applied (since k = n − 1 = 0), and the script falls back to generating synthetic defeaters using nlpaug with WordNet-based synonym augmentation. The final training sets, after SMOTE and synthetic augmentation, included: for ACC, 32 consensus defeaters supplemented with up to 5 synthetic samples per underrepresented class in “What” and “Why”; for CERN, 42 consensus defeaters similarly augmented. The evaluation sets remained unchanged in size, with 42 defeaters for ACC and 50 for CERN. For the context-informed evaluation step 2), a directed graph of assurance case elements (claims, evidence, strategies) was constructed using networkx to capture dependencies, with nodes and edges derived from the safety case CSV files (ACC.csv, CERN LHC.csv) [29]. Outputs, including raw datasets, consensus and dissensus sets, and training/evaluation splits, are saved as CSV files for reproducibility. •
D. Embeddings and Features We use the bert-base-uncased model [9] to generate 768-dimensional embeddings for defeater texts, computed as the mean of the last hidden state. Empty or invalid inputs return zero vectors. For the context-informed approach, a meta-classifier integrates the validity features introduced in section III-D3. • Is Linked: Binary indicator • Cosine Similarity: Between defeater & linked element. • Path Similarity: Between defeater & path route. • Average Cosine Similarity: Mean graph similarity. These features are normalised using min-max scaling and combined with the probabilities of the simple classifiers reassembled for the meta-classifier predictions, as illustrated in
the ’meta-classifier node’ of Figure 2 in the method section III. E. Model Description The experiment trains the two types of classifiers detailed in Subsection III-D4 for “What” and “Why” components: • Simple Classifiers: LR and SVM models predict labels {0, 1, 2} using BERT embeddings of defeater texts. • Meta-Classifiers: LR and SVM meta-classifiers refine predictions by integrating simple classifier probabilities and validity features as mentioned in the paragraph above. Labels are encoded with LabelEncoder, and min 2-fold cross-validation ensures balanced data training. Feature importance is computed via coefficients for linear kernels and permutation importance for non-linear kernels. 1) Extraction Logic: Defeaters were matched to safety case nodes using regex to identify ID-based references (e.g., ”Child Claims 5000”, ”Context 3001”). The logic targeted terms like ”Text Strategy”, ”Text Context”, ”Parent”, ”Child”, ”Evidence”, or numeric IDs in the ”Where” field. Defeaters with descriptive ’Where’ fields lacking structured references were rated 0 for positioning due to insufficient linkage to safety case nodes and were excluded from semantic analysis. Positional verification was performed by searching against the related safety case claims. F. Evaluation The evaluation is conducted in two rounds: 1) Text-Only: Simple classifiers predict ratings using only defeater text embeddings. 2) Context-Informed: Meta-classifiers incorporate structural DAG context (validity features) to improve predictions. Performance is assessed against dissensus annotations (H1̸=H2) using precision, recall, and weighted F1-score, as defined in Table I. The weighted F1 score is a metric that computes the F1 score for each class and then averages them, weighted by the number of samples in each class (support). Correctness is evaluated by comparing predictions to H1 and H2 ratings, while Stability, Relevance, Completeness, and Novelty are computed as described in Section III-C. An ablation study was conducted to evaluate the different configurations for class imbalance mitigation: baseline (SVM simple SVM Meta), SMOTE, non-consensus data inclusion, selftraining, and SMOTE with non-consensus data. We followed the default random state 42 for data splitting and initialization to produce the same result base across runs. G. Limitations The experiment faces several limitations: • Data Quality: Inconsistent or incomplete annotations, particularly for the “Where” component, impact embedding reliability. Class imbalance in training (e.g., few “0” ratings) is mitigated with dissensus integration; the small sample sizes limit robustness.
Human Bias: Variability in expertise for a domain influences annotations (Section III-B), introducing bias in the training data, which is partially addressed by focusing on consensus-defeater training and extracting objective context from the structure within the assurance case. • Model Complexity: BERT’s 768-dimensional embeddings capture rich semantics but could be enriched with more dimensionality. The LR and SVM classifiers may oversimplify nuanced defeater semantics based on the chosen parameters.
•
V. R ESULTS This section assesses classifier configurations for automated defeater evaluation, addressing RQ1 (accuracy and consistency with human raters) and RQ2 (subjectivity reduction) from Section IV-A. To our knowledge, it is the first approach automating the evaluation of generated defeaters; for that, we tested Simple LR, SVM, and meta-classifiers (LR and SVM variants) on “What” and “Why” components against human annotations (H1, H2). The “Where” component was evaluated with the extraction logic explained in Subsection IV-E1. We show below the weighted F1-score results in Table II for general match against both raters and in Table III for individual matches, additionaly listed the imbalance mitigation strategies introduced in Subsection III-D4; Table IV and V show the agreement results and class distribution the SVM simple SVM meta model achieved. A. RQ1 For RQ1, embedding-based classifier models demonstrate comparable consistency with human expert ratings, as evidenced by weighted F1 scores in Tables II and III. For the ACC dataset, the SVM meta, LR simple models achieve the highest General What F1 score of 0.85, with a strong What H2 Individual F1 of 0.65. Simple SVM excels with What H1 Individual F1 of 0.65 and Why H2 Individual F1 of 0.60. For the CERN dataset, Simple LR achieves a general What/Why F1 score of 0.86. In contrast, LR Meta and SVM Simple lead with a What H2 individual F1 score of 0.71 and a Why H2 individual F1 score of 0.61. SVM Meta and SVM Simple top the Why H1 individual F1 score at 0.60, which is balanced for both raters and shows individual accuracy. The SVM Meta and SVM Simple models achieve an F1 score of 0.84 across the ACC and CERN datasets, thereby also supporting RQ1, which demonstrates the accuracy of the artificial ratings. B. RQ2 For RQ2, classifier models reduce the impact of subjectivity in defeater validation, particularly in cases of significant disagreement, as demonstrated by the F1 scores in Table III. Balanced ratings across reviewers highlight this effect. The imbalance mitigation method involving self-training with iteratively refined predictions does not equalise performance among raters. Synthetic Minority Oversampling Technique (SMOTE) with artificial minority labels performs better, but the remaining imbalance is worse than with the SVM Meta and
SVM Simple approaches. Adding additional non-consensus defeaters does not improve the balance of performance against both reviewers. The subjectivity problem of independent raters is evident, shown by the class distributions of dissensus data in Table IV. A deeper analysis of the agreement rates for the best overall performance model (SVM Meta and SVM Simple) in Table V reveals further insights. The initial negative kappa values: ACC Why (-0.50), CERN Why (-0.40), and CERN What (-0.18) indicate worse-than-chance agreement in the human ratings. Using the automated review approach incorporating the model improved the values by approximately 40%. The ’Where’ component did not improve much but is included to provide a complete overview. It is not representative due to its static filter mechanism, which was explained in Subsection IV-E1, not considering general system descriptions. C. Result Discussion These findings suggest that embedding-based classifiers can mitigate the scalability and subjectivity issues of manual defeater validation, as outlined in Section I. By leveraging structured domain-specific embeddings, the models capture nuances relevant to automotive (ACC) and nuclear (CERN) assurance cases (ACs) [7], enhancing defeater evaluation shown by the results in Table II and III. However, the models lack human-like contextual understanding, occasionally missing critical domain-specific information (e.g., unmodeled risks in complex ACs) due to limitations in embedding patterns. This is noticeable in lower Individual F1 scores (e.g., 0.43 for ACC Why H1 from Table III), where nuanced defeater evaluation requires deeper contextual insight. The size of the validation datasets with Dissensus Defeaters is limited. The performance of the models could be statistically supported by larger validation sets. VI. C ONCLUSION Results from the model comparison Table II confirm that the models, trained on embeddings from the domains, can predict defeater quality and reflect expert tendencies, addressing Research Question 1. The meta-classifier integrates assurance case features, thereby enhancing contextual understanding; yet, it still exhibits training imbalances. Despite balanced class weights, LR models, including the meta-classifier, tend to favor majority classes due to imbalanced data distributions, aligning predictions more with Reviewer 2’s labels than Reviewer 1’s. Notably, SVM models showed slight bias, demonstrating consistent performance against both reviewers in the individual performance comparison Table III. For Research Question 2, this automated review approach reduces subjective uncertainties, as demonstrated by the improvement in inter-agreement rates shown in Table V. It provides an objective perspective on safety case defeaters by using semantic feature embeddings to achieve contextually informed evaluation. While this method has its advantages, like all supervised methods, it relies on expert-curated data and is not excluded from data change effects.
TABLE II M ODEL P ERFORMANCE C OMPARISON G ENERAL
Classifier
What Precision
Recall
Simple LR Simple SVM LR meta, LR simple LR meta, SVM simple SVM meta, LR simple SVM meta, SVM simple
0.64 0.8 0.82 0.71 0.84 0.8
0.80 0.82 0.80 0.76 0.86 0.82
Simple LR Simple SVM LR meta, LR simple LR meta, SVM simple SVM meta, LR simple SVM meta, SVM simple
0.82 0.8 0.83 0.83 0.85 0.86
0.91 0.82 0.89 0.89 0.85 0.86
Why F1
Precision
Recall
F1
0.85 0.84 0.85 0.85 0.84 0.85
0.92 0.84 0.92 0.92 0.86 0.92
0.88 0.84 0.88 0.88 0.85 0.88
0.82 0.82 0.87 0.85 0.83 0.82
0.91 0.77 0.87 0.85 0.79 0.79
0.86 0.77 0.86 0.85 0.79 0.79
Data ACC 0.71 0.81 0.73 0.71 0.85 0.81 Data CERN 0.86 0.80 0.86 0.86 0.85 0.86
TABLE III M ODEL P ERFORMANCE C OMPARISON I NDIVIDUAL
Classifier
What H1 Precision Recall
What H2 F1
Precision Recall
Why H1 F1
Precision Recall
Why H2 F1
Precision Recall
F1
Data ACC LR SVM LR meta, LR simple LR meta, SVM simple SVM meta, LR simple SVM meta, SVM simple
0.29 0.63 0.72 0.50 0.62 0.55
0.54 0.66 0.56 0.54 0.62 0.58
0.38 0.65 0.42 0.44 0.58 0.56
0.41 0.59 0.40 0.39 0.65 0.54
0.64 0.58 0.62 0.56 0.68 0.56
0.50 0.58 0.49 0.46 0.65 0.55
0.41 0.36 0.41 0.41 0.37 0.37
0.64 0.40 0.64 0.64 0.50 0.50
0.50 0.38 0.50 0.50 0.43 0.43
0.29 0.61 0.29 0.29 0.55 0.55
0.54 0.62 0.54 0.54 0.56 0.56
0.38 0.60 0.38 0.38 0.49 0.49
0.77 0.78 0.79 0.79
0.60 0.64 0.60 0.64
0.49 0.57 0.51 0.58
0.79 0.71 0.81 0.70
0.69 0.69 0.67 0.67
0.60 0.63 0.61 0.62
0.41 0.53 0.41 0.41
0.62 0.62 0.60 0.60
0.49 0.53 0.49 0.49
0.76 0.60 0.76 0.76
0.55 0.55 0.55 0.55
0.41 0.44 0.45 0.45
Imbalance Mitigation Self-Training SMOTE Non-Consensus SMOTE + Non-Consensus
Data CERN LR SVM LR meta, LR simple LR meta, SVM simple SVM meta, LR simple SVM meta, SVM simple
0.24 0.58 0.27 0.27 0.53 0.53
0.49 0.58 0.51 0.51 0.52 0.52
0.32 0.57 0.35 0.35 0.47 0.47
0.64 0.63 0.67 0.67 0.69 0.70
0.80 0.54 0.77 0.77 0.66 0.68
0.71 0.57 0.71 0.71 0.68 0.69
0.38 0.61 0.55 0.51 0.56 0.61
0.62 0.57 0.58 0.54 0.55 0.60
0.47 0.57 0.55 0.52 0.56 0.60
0.46 0.51 0.57 0.60 0.51 0.48
0.68 0.45 0.62 0.63 0.49 0.45
0.55 0.46 0.59 0.61 0.50 0.46
0.76 0.64 0.77 0.78
0.48 0.50 0.48 0.50
0.33 0.41 0.35 0.39
0.70 0.68 0.69 0.69
0.82 0.70 0.76 0.74
0.76 0.69 0.73 0.71
0.34 0.56 0.49 0.53
0.58 0.58 0.54 0.56
0.43 0.55 0.48 0.52
0.46 0.57 0.65 0.59
0.68 0.60 0.68 0.62
0.55 0.58 0.64 0.60
Imbalance Mitigation Self-Training SMOTE Non-Consensus SMOTE + Non-Consensus
TABLE IV C LASS D ISTRIBUTIONS FOR ACC/CERN DATASETS Task
ACC (42 defeaters)
CERN (50 defeaters)
H1
H1
H2
Model
(0,1,2) (0,1,2) (0,1,2)
What
0, 19, 23 Why 0, 15, 27 Where 10, 7, 25
0, 15, 27 0, 20, 22 1, 15, 26
0, 6, 36 0, 2, 40 19, 0, 23
H2
Model
(0,1,2) (0,1,2) (0,1,2)
0, 27, 23 0, 21, 29 2, 8, 40
0, 8, 42 0, 16, 34 3, 3, 44
2, 1, 47 0, 4, 46 13, 0, 37
TABLE V I NTER -R ATER AGREEMENT (C OHEN ’ S K APPA ) ACC/CERN Dataset
Task
H1/H2 H1/Model H2/Model
ACC
What Why Where
0.02 -0.50 0.33
0.34 -0.09 0.33
0.34 0.10 0.27
CERN
What Why Where
-0.18 -0.40 0.16
0.07 -0.06 0.35
-0.06 -0.03 0.40
Future Work Advanced LLMs or improved embedding techniques, such as fine-tuning on domain-specific datasets, could address contextual understanding limitations, opening directions for research to enhance classifier performance and calibrate appearing generalization problems for the generation task. Furthermore, future work should explore hybrid expert/embedding-based approaches in the safety engineering process to enhance validation reliability and scalability for dynamic systems at runtime. R EFERENCES [1] D. Ratiu, T. Rohlinger, T. Stolte, and S. Wagner, “Towards an argument pattern for the use of safety performance indicators,” in Computer Safety, Reliability, and Security. SAFECOMP 2024 Workshops, A. Ceccarelli, M. Trapp, A. Bondavalli, E. Schoitsch, B. Gallina, and F. Bitsch, Eds. Springer Nature Switzerland, pp. 160–172. [2] P. Koopman, How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety. Pittsburgh, PA, USA: Carnegie Mellon University, 2022. [3] E. Asaadi, E. Denney, J. Menzies, G. J. Pai, and D. Petroff, “Dynamic assurance cases: A pathway to trusted autonomy,” Computer, vol. 53, no. 12, pp. 35–46, 2020. [4] R. Tihomir, “Automated interpretation of fleet incidents to enable system level runtime assurance,” in 35th IEEE International Symposium on Software Reliability Engineering, ISSRE 2024 - Workshops, Tsukuba, Japan, October 28-31, 2024, 2024, pp. 91–94. [5] R. Bloomfield, K. Netkachova, and J. Rushby, “Defeaters and eliminative argumentation in Assurance 2.0,” Computer Science Laboratory, SRI International, Menlo Park, CA, Tech. Rep. SRI-CSL-2024-01, May 2024, additional arxiv 2405.15800. [6] J. Rushby, “The interpretation and evaluation of assurance cases,” in Proc. Comput. Sci. San Francisco, CA, USA: Elsevier, 2015. [Online]. Available: https://www.semanticscholar. org/paper/The-Interpretation-and-Evaluation-of-Assurance-Rushby/ 5196fbfbc98306ad2273fc61d420af8ce1452fe1
[7] T. Viger, L. Murphy, S. Diemert, C. Menghi, J. Joyce, A. Di Sandro, and M. Chechik, “AI-supported eliminative argumentation: Practical experience generating defeaters to increase confidence in assurance cases,” in 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE), 2024, pp. 284–294, ISSN: 2332-6549. [Online]. Available: https://ieeexplore.ieee.org/document/10771339/ [8] B. S. Everitt and D. C. Howell, Encyclopedia of statistics in behavioral science. Chichester, UK: John Wiley & Sons, 2005. [9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Association for Computational Linguistics, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423/ [10] T. Kelly and R. Weaver, “The goal structuring notation–a safety argument notation,” Proc Dependable Syst Networks Workshop Assurance Cases, 01 2004. [11] L. Millet, S. Diemert, C. Rees, T. Viger, M. Chechik, C. Menghi, and J. Joyce, “Assurance case arguments in the large: The CERN LHC machine protection system,” in Computer Safety, Reliability, and Security, J. Guiochet, S. Tonetta, and F. Bitsch, Eds. Springer Nature Switzerland, pp. 3–10. [12] M. Sivakumar, A. B. Belle, J. Shan, and K. K. Shahandashti, “Gpt-4 and safety case generation: An exploratory analysis,” 2023, arXiv:2312.05696. [Online]. Available: http://arxiv.org/abs/2312.05696 [13] ——, “Prompting gpt-4 to support automatic safety case generation,” Expert Syst. Appl., vol. 255, p. 124653, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417424015203 [14] I. Kurzidem, S. Burton, and P. Schleiss, “Safesens - uncertainty quantification of complex perception systems,” in 2023 IEEE 26th Int. Conf. Intell. Transp. Syst. (ITSC). IEEE, 2023, pp. 5805–5810. [15] U. Gohar, M. C. Hunter, R. R. Lutz, and M. B. Cohen, “CoDefeater: Using LLMs to find defeaters in assurance cases,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. Association for Computing Machinery, pp. 2262–2267. [Online]. Available: https://dl.acm.org/doi/10.1145/3691620.3695296 [16] K. Khakzad Shahandashti, M. Sivakumar, M. M. Mohajer, A. Boaye Belle, S. Wang, and T. Lethbridge, “Assessing the impact of GPT-4 turbo in generating defeaters for assurance cases,” in Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering. ACM, pp. 52–56. [Online]. Available: https://dl.acm.org/doi/10.1145/3650105.3652291 [17] M. S. Graydon and S. M. Lehman, “Examining proposed uses of llms to produce or assess assurance arguments,” NASA Peer Committee, Washington, DC, USA, Technical Report 20250001849, 2025. [Online]. Available: https://ntrs.nasa.gov/citations/20250001849 [18] J. B. Goodenough, C. B. Weinstock, and A. Z. Klein, “Eliminative argumentation: A basis for arguing confidence in system properties,” Carnegie Mellon University, Pittsburgh, PA, USA, Technical Report CMU/SEI-2015-TR-005, 2015. [Online]. Available: https://insights.sei. cmu.edu/documents/1248/2015 005 001 434813.pdf [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. [Online]. Available: https://papers.nips.cc/paper files/ paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html [20] I. Keraghel, S. Morbieu, and M. Nadif, “Recent advances in named entity recognition: A comprehensive survey and comparative study,” 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID: 267060999 [21] B. Chen, K. Chen, S. Hassani, Y. Yang, D. Amyot, L. Lessard, G. Mussbacher, M. Sabetzadeh, and D. Varró, “On the use of GPT-4 for creating goal models: An exploratory study,” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW). IEEE, pp. 262–271. [Online]. Available: https: //ieeexplore.ieee.org/document/10260905/ [22] E. First, M. N. Rabe, T. Ringer, and Y. Brun, “Baldur: Whole-proof generation and repair with large language models,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2023. Association for Computing Machinery, pp. 1229–1241. [Online]. Available: https://dl.acm.org/doi/10.1145/3611643.3616243
[23] O. Odu, A. B. Belle, S. Wang, S. Kpodjedo, T. C. Lethbridge, and H. Hemmati, “Automatic instantiation of assurance cases from patterns using large language models,” J. Syst. Softw., vol. 222, p. 112353, 2025. [Online]. Available: https://api.semanticscholar.org/CorpusID: 275911948 [24] S. Varadarajan, R. Bloomfield, J. Rushby, G. Gupta, A. Murugesan, R. Stroud, K. Netkachova, I. H. Wong, and J. Arias, “Enabling theorybased continuous assurance: A coherent approach with semantics and automated synthesis,” in Computer Safety, Reliability, and Security. SAFECOMP 2024 Workshops, A. Ceccarelli, M. Trapp, A. Bondavalli, E. Schoitsch, B. Gallina, and F. Bitsch, Eds. Springer Nature Switzerland, pp. 173–187. [25] ISO, “26262-1:2018 road vehicles – functional safety – part 1: Vocabulary,” Geneva, Switzerland, Tech. Rep., 2018, standard. [Online]. Available: https://www.iso.org/standard/68383.html [26] ——, “21448:2022 Road vehicles – Safety of the intended functionality (SOTIF),” Geneva, Switzerland, Tech. Rep., 2022, standard. [Online]. Available: https://www.iso.org/standard/77490.html [27] ANSI/UL, “4600:2023 standard for safety for the evaluation of autonomous products,” Northbrook, IL, USA, Tech. Rep., 2023, standard, 3rd ed. [Online]. Available: https://www.shopulstandards.com/ ProductDetail.aspx?productId=UL4600 3 S 20230317 [28] A. Turobov, D. Coyle, and V. Harding, “Using chatgpt for thematic analysis,” 2024, arXiv:2405.08828. [Online]. Available: http://arxiv.org/abs/2405.08828 [29] “Ai-ea implementation and data,” https://zenodo.org/records/13368055, accessed: 2025-07-10. [30] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. [31] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/ 1810.04805 [32] N. Chawla, K. Bowyer, L. Hall, and W. Kegelmeyer, “Smote: Synthetic minority over-sampling technique,” J. Artif. Intell. Res. (JAIR), vol. 16, pp. 321–357, 06 2002. [33] D. Yarowsky, “Unsupervised word sense disambiguation rivaling supervised methods,” in 33rd Annual Meeting of the Association for Computational Linguistics. Cambridge, Massachusetts, USA: Association for Computational Linguistics, Jun. 1995, pp. 189–196. [Online]. Available: https://aclanthology.org/P95-1026/