Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams

Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams Kevin Delcourt

Meriem Ben Chaaben

Abdelhamid Rouatbi

Université de Montréal DIRO Montreal, Canada kevin.delcourt@umontreal.ca

Université de Montréal DIRO Montreal, Canada meriem.ben.chaaben@umontreal.ca

Université de Montréal DIRO Montreal, Canada abdelhamid.rouatbi@umontreal.ca

Luciano Marchezan

Houari Sahraoui

Université de Montréal DIRO Montreal, Canada lucianomarchezan94@gmail.com

Université de Montréal DIRO Montreal, Canada sahraouh@iro.umontreal.ca

arXiv:2608.14315v1 [cs.SE] 14 Aug 2026

Abstract In software engineering, many semantic modeling tasks lack a unique ground truth, as human judgments are both costly and subjective. This paper explores mutation testing as a scalable alternative for evaluating semantic judges (e.g., LLM-based) of models. We propose a mutation testing approach in which controlled semantic defects are injected into domain class diagrams. Starting from pairs of PlantUML class diagrams and textual system descriptions, we apply mutation operators (e.g., removing a class) to generate faulty variants. A candidate judge is then evaluated based on its ability to detect the injected defects. We define 11 mutation operators for the task of comparing a domain class diagram against a textual description and evaluate the proposed approach against a conventional manual assessment of judgment validity. Across six judge configurations (three LLMs and two prompt variants), the automated mutation testing approach is largely consistent with the manual assessment in identifying the better-performing configurations. The results suggest that mutation testing may serve as a scalable proxy for analyzing semantic judges.

CCS Concepts • Software and its engineering → Model-driven software engineering; Software testing and debugging; • Computing methodologies → Natural language processing.

Keywords mutation testing, LLM-as-a-judge, large language models, modeldriven engineering ACM Reference Format: Kevin Delcourt, Meriem Ben Chaaben, Abdelhamid Rouatbi, Luciano Marchezan, and Houari Sahraoui. 2026. Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams. In

This work is licensed under a Creative Commons Attribution 4.0 International License. MODELS 2026, Málaga, Spain © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2809-9/2026/10 https://doi.org/10.1145/3822455.3838768

ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS 2026), October 04–09, 2026, Málaga, Spain. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3822455.3838768

1

Introduction

Models are widely used in software engineering, and in many modelbased approaches they serve as primary artifacts for describing, analyzing, and communicating aspects of software systems [8]. Domain-level class diagrams are important as they capture the main concepts of an application domain, their attributes, and relationships. Beyond their descriptive role, they also automate tasks such as model transformation and generation, among others [10]. A recurring challenge is assessing whether a model correctly reflects an informal or semi-formal textual description [9]. This is especially an issue when models must be assessed against naturallanguage requirements, which may be ambiguous or expressed at different levels of abstraction [6, 11]. Moreover, for many modeling tasks, there is no unique correct solution as different diagrams may satisfy the same description while making different yet logical choices [23]. These choices are often influenced by modeling conventions and by the context in which the model is used. As a result, evaluating a model by comparison to one reference model provides only a partial view of its correctness. Large Language Models (LLMs) are used for natural language understanding, abstraction, and cross-artifact reasoning [43], making them attractive for model engineering tasks involving textual descriptions [13]. In particular, LLMs are starting to be used as judges to assess whether a domain model is semantically adequate with respect to a textual specification [12]. However, using LLMs as model judges introduces a second evaluation problem: how can we determine whether the judgment produced by an LLM is itself reliable? LLMs are inherently non-deterministic, sensitive to prompting choices, and may produce judgments that appear plausible while overlooking important modeling defects [26]. Mutation testing offers a promising direction to address this problem. The technique evaluates a validation mechanism by introducing small, controlled defects into an artifact and checking whether they are detected [7]. It has been applied in MDE contexts, for instance, to assess the adequacy of test suites for model transformations [24], or for search-based model-driven engineering [33].

MODELS 2026, October 04–09, 2026, Málaga, Spain

However, existing approaches target the evaluation of rule-based validation oracles, rather than the evaluation of semantic judges. In this paper, we adapt this principle to evaluate LLMs used as semantic judges of domain class diagrams. We start from a model– description pair and introduce controlled defects through model transformation (e.g., by removing a class). The intuition is that, although it may be difficult to define a unique correct model for a textual description, it is easier to define changes that make a given model less adequate. These mutations create model variants whose defects are known by construction. An LLM judge can then be evaluated according to whether it detects or penalizes the defect introduced by the mutation. The behavior of the LLM judge on these mutants provides evidence of its sensitivity to different categories of modeling defects. In summary, we make the following contributions: (i) We propose a mutation-testing-based approach for validating semantic judges of domain-level class diagrams; (ii) We construct a mutated dataset of domain-level class diagrams by applying mutation operators that target classes, attributes, and associations to an existing curated dataset [37]; and (iii) We empirically show how the mutation-testing-based approach can be used to analyze and compare different LLMs and prompting strategies by comparing our approach to a standard manual evaluation.

2 Background and Related Work 2.1 Judging Semantic Tasks in Modeling We use the term semantic task to denote a task in which success is measured primarily by the semantic quality of the output. Examples include many natural-language tasks [1], such as text summarization [41], and emotion recognition in text [42]. In MDE, typical semantic tasks include comparing a domain model with a naturallanguage requirement [39], or assessing whether two alternative models make equivalent domain commitments [28]. For example, given a requirement about patients booking appointments with doctors, one diagram may explicitly model a Receptionist as an actor, whereas another may abstract this into a generic Booking concept. Which is preferable depends on the modeling purpose, stakeholder priorities, and downstream use. This creates an oracle problem. For many semantic modeling tasks, there is no single reference solution, and element overlap or syntactic well-formedness does not determine whether a model is the most appropriate interpretation of an informal requirement [20, 27].

2.2

Validating Semantic Judges

LLM-based semantic judges are increasingly used to evaluate openended modeling artifacts using rubrics, contextual information, and natural-language explanations [5, 18]. However, these judges inherit the same oracle problem. If the target quality is contextual or partly subjective, then the judge’s assessment cannot be validated by comparison with a unique ground truth. Semantic judges are commonly validated by comparing their assessments against human judgments. Benchmarks such as MTBench [4] measure agreement with human preferences and report that strong LLM judges can approximate human judgments in several settings [44]. However, human evaluation is costly, difficult to

Delcourt et al.

scale, and subjective, making human agreement alone an incomplete validation criterion, especially for model-related tasks [29]. Other approaches explore the use of LLM internal signals, in particular perplexity, as metrics for assessing output quality; that is, the more uncertain a model is about its answer, the more likely the answer is to be of low quality. For text-writing tasks, Murugadoss et al. [25] compared a perplexity-based evaluation method against human judgments and found a weak-to-moderate correlation. While promising, this kind of approach is limited because it is (i) not task-specific, and therefore needs to be validated for each new task, and (ii) specific to LLMs, making it difficult or impossible to apply to closed-source models. Finally, a natural extension is to use a jury of heterogeneous judges and aggregate their decisions, while treating inter-judge disagreement as an uncertainty signal. For example, The PoLL approach shows that panels of diverse models can reduce single-model bias [38], but multiple judges may still share correlated blind spots if trained on similar data [32, 40, 44].

2.3

Mutation testing

Mutation testing evaluates a validation oracle by introducing small, systematic changes into an artifact, producing mutants [7]. Each change is generated by a mutation operator. If the oracle detects the difference, the mutant is killed; otherwise, it survives. The aggregate metric is the kill ratio: the proportion of mutants killed. Surviving mutants reveal specific fault classes not detected by the oracle. Although mutation testing is most commonly associated with assessing test-suite adequacy, its underlying principle has been applied in other fields [21, 30]. In classic test-suite validation, approaches such as LLMorpheus prompt an LLM to generate contextaware mutants that better resemble realistic faults [34]. Similar approaches are being explored in MDE, for example to generate mutants of Simulink-Stateflow models [36], showing that LLMgenerated mutants are, on average, more diverse and take less time to generate than hand-made mutants. Another emerging direction uses mutation-style perturbations to evaluate LLMs and LLM-based judges. For instance, Crupi et al. use mutation-style perturbations to evaluate whether LLM judges can distinguish incorrect code from behavior-preserving variants [14]. In MDE, mutation testing has also been applied directly to modeling artifacts and model transformations. Mutation operators have been used to emulate realistic transformation faults and to guide the generation of mutant-killing rules [19, 35]. Mutation has also been applied at the model level, for example through mutation operators for UML class diagrams [17], or language-independent frameworks such as Wodel-Test [16]. However, these works evaluate rule-based validation oracles. The closest work to ours is Ahmed et al. [2], who manually introduced design smells into sequence diagram/textual requirement pairs and measured how various LLMs detected these issues, demonstrating the applicability of a mutation-like approach to LLM judge validation.

3 Mutation Testing for Semantic Tasks 3.1 Overview Figure 1 summarizes the proposed approach. The approach starts from a base dataset of model–description pairs, where each entry

Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams

Table 1: Mutation operators for domain class diagrams.

consists of a PlantUML domain class diagram and a textual description of the system being modeled. Mutation operators introduce known semantic defects into diagrams. For each generated mutant, the operator produces the mutated PlantUML diagram and a concise natural-language description of the injected defect. This defect description acts as a lightweight oracle for the evaluation of the judge. The semantic judge under test then reports design issues concerning the adequacy between the textual description and the mutated class diagram. A mutant is considered killed when the judgment identifies the injected defect. This decision is automated through a kill detection strategy that compares the defect description with the judge’s reported issues, which in our case is achieved through sentence-embedding similarity. The final outcome is the ratio of killed mutants, which provides a quantitative indication of the judge’s ability to detect semantic defects.

3.2

Application to Domain Class Diagram

Here, we describe the specific elements that need to be defined to apply our approach to domain class diagrams. To transfer this approach to new tasks, the following elements need to be adapted: Base Modeling Dataset. The base dataset provides the original inputs to which mutations are applied. It does not need to contain exhaustive annotations or gold-standard judgments. However, the approach assumes that the base dataset contains entries that are sufficiently coherent for the injected mutations to introduce identifiable defects. For example, in the task of checking a class diagram against a textual specification, if a low-quality class diagram in the base dataset already contains a superfluous class, then removing that class removes a defect instead of adding one. Therefore, the quality of the dataset must be relatively good. We discuss this limitation further in Section 5. Mutation Operators. We define 11 mutation operators for domain class diagrams, listed in Table 1. Each operator injects a semantic defect into the class diagram while keeping the textual description unchanged. All mutations are applied programmatically by parsing the PlantUML source, modifying the relevant elements according to model transformation principles, and producing a new PlantUML file. Each operator also generates a concise defect description, such as “the class Library is missing”, which serves as the lightweight oracle for the kill detection step. See the replication package [3] for additional operator implementation details. The operators are divided into two categories. Automated operators (M1–M5) are rule-based model transformations that select a target element at random and apply a fixed modification, such as removal or reversal. These can create, at low cost, semantic defects that a competent judge should detect. LLM-assisted operators (M6–M11) are used when the mutation requires generating a semantically meaningful model element. We use LLM assistance for these operators because the generated element must be realistic enough to make the mutant challenging to detect, while still constituting a defect under the modeling requirements adopted in this study. This set of 11 operators corresponds to defects that are meaningful in the context of domain class diagrams. However, the main challenge in evaluating semantic modeling tasks is that there is no single ground truth and, as such, no commonly accepted set of rules for defining what constitutes a good model [23]. Consequently, the

MODELS 2026, October 04–09, 2026, Málaga, Spain

ID

Impl. Transformation

M1 M2 M3

Auto Auto Auto

M4

Auto

M5

Auto

M6

LLM

M7

LLM

M8

LLM

M9

LLM

M10

LLM

M11

LLM

Expected violation

Remove a class. Remove an attribute. Remove a relationship.

Required concept is missing. Required property is missing. Required domain relation is absent. Add an irrelevant association Unjustified relation or multiplicity with multiplicities. appears. Reverse a non-symmetrical re- Direction or role semantics contralationship. dict the description. Add an implementation- Technical artefact appears as a dooriented class. main concept. Add an implementation- Technical property appears in a dooriented attribute. main class. Add an implicit domain class. Plausible but unjustified class is added. Add an implicit domain at- Plausible but unjustified property tribute. is added. Add a class from another do- Unrelated concept is introduced. main. Add an attribute from another Unrelated property is introduced. domain.

operators should be adapted each time the approach is applied in a new context. Semantic Judges. The semantic judges receive mutated entries and produce a judgment for the semantic task at hand, i.e., an assessment of the design issues in domain class diagrams when compared to their respective system descriptions. Although this work focuses on evaluating LLM-based judges, the approach can also be applied to other semantic validators, including rule-based or manual approaches. In all cases, the judge is evaluated according to whether it detects the defect intentionally introduced by the mutation operator. Mutant Kill Detection. Since we aim to automate the evaluation process, we define a step that automatically determines whether the judge identified the introduced mutations. To do so, we compare the injected defect description with each issue reported by the judge using sentence-embedding similarity. Let 𝑑𝑚 be the defect description associated with mutant 𝑚, and let I𝑚 be the set of issues reported by the judge for that mutant. We encode both 𝑑𝑚 and each issue 𝑖 ∈ I𝑚 using all-MiniLM-L6-v2 [31], a lightweight sentence transformer that offers a good trade-off between encoding quality and computational cost for short textual comparisons. We then compute the cosine similarity between the resulting embeddings. If the judge reports no issue, the mutant is considered alive. A mutant is considered killed when at least one reported issue reaches the similarity threshold 𝜏: killed(𝑚) = 1 ⇐⇒ max (∀𝑖 ∈ I𝑚 , cos(𝐸 (𝑑𝑚 ), 𝐸 (𝑖))) ≥ 𝜏 .

(1)

Other strategies may also be possible. Depending on the task, the comparison may rely on semantic similarity, natural-language entailment, keyword matching or a combination of these techniques.

4

Evaluation

To validate our mutation-testing-based approach, we compare it with a standard manual validation of LLM-based judges, as illustrated in Figure 2. This three-step evaluation process answers the following research questions: RQ1–Manual evaluation: How precise are the semantic issues reported by the evaluated judge configurations?

MODELS 2026, October 04–09, 2026, Málaga, Spain

Delcourt et al.

Base Modeling Dataset

Mutant Dataset

Domain Class Diagrams (PlantUML) + Textual Description

Mutated Diagrams + Textual Description + Defect Description

Example (Easy ModelSet - EasyLibrary)

1

Mutation Operators

1

Automatically add controlled defects to the base modeling dataset.

manages

0..*

name memberId

1 written by 1..*

0..*

0..*

Book

Member

written by

name

Does the judgement match the added defect description ?

name employeeId 0..1

is borrowed in makes 0..*

Automated Mutant Kill Detection

Librarian

processes

0..* 0..*

Author

Borrowing borrowDate

0..*

1

1..*

0..*

employs

name memberId

1

is borrowed in makes 0..*

1

1

0..*

ti tle isbn

1

Author name

Example: M8 - Add an implicit class i.e., a class that is plausible but unjustified along with the corresponding associations

Member

ti tle isbn

Example: gpt-5.3-codex (prompt P2) 1

serves

Book

Provides a list of model design issues

Library

manages serves

0..*

0..*

Semantic Judge Under Test

Example (mutated by gpt-5-mini)

Library

Mutant Judgements Semantic Judge Outputs

Borrowing

Example (generatd by gpt-5.3-codex)

borrowDate

Example: Semantic similarity comparaison between added defect and design issues using AllMiniLM-L6-V2

Design Issues

Textual description

Added defect

A small local library wants to organize its operations. The library manages a collection of books. Each book has a title and an ISBN [...]

The class `Librarian` is not mentioned in the domain description and should be removed.

1. Librarian is not supported by the domain description. No librarian concept or staffing process is mentioned, so this class is superfluous. [5 lines hidden]

Ratio of mutant killed

Figure 1: Mutation-testing-based approach for the validation of semantic judges. RQ2–Mutation-based approach: How effectively do the evaluated judge configurations detect controlled semantic defects introduced through mutation? RQ3–Comparative validity: To what extent can the mutation kill ratio reproduce the comparative conclusions obtained from a manual evaluation? Judges under test. We evaluate six LLM-based judge configurations, formed by three LLMs (open-source gemma4:e4b, and closed-source gpt-5-mini and gpt-5.3-codex) and two prompt variants (P1 and P2). The three LLMs were selected because they have different sizes and capabilities. Both prompts are zero-shot prompts that instruct the LLM to provide a list of class-diagram design issues, with P2 providing more detailed guidance than P1. The replication package [3] contains the full prompts. These conditions are intended to reflect a standard behavior in LLM-based judge development, where practitioners test different models and prompts to identify the most suitable configuration. Base Datasets. Table 2 presents the two datasets used in this study. For each dataset, size is measured through the number of model elements, i.e., classes, attributes, and associations. Lexical diversity is measured from diagram labels, i.e., class names, attribute names, and association labels. These metrics were computed using the CM benchmark for model sets introduced by Glaser et al. [15]. Table 2: Overview of the datasets used in this study. Metric

Easy

Golden [37]

Number of model–description pairs

5

45

Avg. elements per model Min. elements per model Max. elements per model

24.4 22 29

54.3 14 114

Total number of labels Avg. labels per model Total vocabulary size Avg. vocabulary per model

62 12.4 50 10.0

1190 26.4 776 17.2

The Golden ModelSet [37] is a collection of expert-curated domain class diagrams paired with textual descriptions. It contains 45 PlantUML class diagram and description pairs and provides realistic cases for semantic model validation, made by experts from various universities. Its diagrams are relatively complex, with an average of

54.3 elements per model. To complement it with simpler and more controlled examples, we define the Easy ModelSet, a small set of five smaller entries manually created by the authors. These easy cases are included in the replication package and were used to calibrate the approach on examples where the expected semantic defects are easier to inspect manually.

4.1

RQ1. Manual Evaluation

RQ1 estimates the precision of the issues reported by each judge configuration on non-mutated model–description pairs, providing a baseline measure of judgment validity. Setup. We evaluated the six judge configurations on ten nonmutated entries: all five entries from the Easy ModelSet and five selected at random from the Golden ModelSet. Each configuration produced one judgment per pair, for a total of 60 judgments. Four independent annotators with MDE expertise and at least master’slevel training participated in the annotation. Each annotator reviewed 30 judgments, such that every judgment was independently reviewed by two annotators. At the level of individual reported issues, annotators assigned one of three labels: valid, when the issue was supported by the diagram, the textual description, and the modeling criteria adopted in this study; invalid, when the issue was unsupported or resulted from an incorrect interpretation; and uncertain, for borderline cases. We assessed inter-annotator agreement using raw percentage agreement and nominal Krippendorff’s 𝛼, which accommodates the partially crossed annotation design [22]. All disagreements and uncertain cases were subsequently adjudicated by an additional annotator to obtain a final valid/invalid label for every reported issue. Precision is therefore calculated as the ratio of the number of valid issues to the total number of reported issues. Results. The 60 judgments contained 428 reported issues (86/342 for Easy/Golden ModelSet). The judges reported more issues for the Golden ModelSet pairs, which may be explained by the fact its entries are larger and more heterogeneous. The annotators assigned the same initial label to 65% of the issues. Krippendorff’s 𝛼 was 0.32, indicating limited agreement before adjudication. This result illustrates that deciding whether a natural-language criticism of a domain class diagram is semantically justified remains

Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams

MODELS 2026, October 04–09, 2026, Málaga, Spain

RQ1. Manual Evaluation. Base Modeling Datasets 50 model-description pairs Easy ModelSet (5) Golden ModelSet (45)

Judges Under Test 6 configurations 3 models (gpt-5-mini, gemma4:e4b, gpt-5.3-codex)

* 2 prompt variants (P1, P2)

Judgement of base model-description pairs subset of 10 pairs * 6 configurations ==> 60 judgements, 428 design issues

Application of 11 mutation operators ==> 547 mutated entries

Manual annotation design issues labels: valid, invalid or uncertain ==> each issue annotated 2 times by independent annotators

Judgement of mutated entries 547 entries * 6 configurations ==> 3282 judgements

RQ3. Comparative Validity.

Manual Judge Precisions #valid / #issues

Comparative analysis

Mutant Kill Detection semantic similarity (All-MiniLM-L6-V2) with 𝜏 = 0.55

Pearson correlation & pairwise agreement

Computed kill ratios #killed / #mutants

RQ2. Mutation-Based Approach.

RQ2. Mutation-Testing Evaluation

RQ2 evaluates whether the judge configurations detect known semantic defects injected through the mutation operators. Setup. We applied each mutation operator once to each applicable model–description pair. Operator applicability depends on the structure of the base diagram: in some cases, the base diagram from the Golden ModelSet does not include an asymmetric relationship, making M5 inapplicable, or does not include attributes, making M2 inapplicable. After filtering three cases of non-applicable operator– model combinations, the generated mutant dataset contains 547 mutants. We executed the six judge configurations once on each mutant, producing 3,282 mutant judgments. For each mutant, the semantic-similarity kill detector described in Section 3.2 determined whether at least one reported issue matched the injected defect. The similarity threshold was calibrated on a manually labeled set of 209 mutant judgments produced by gpt-5-mini and set to 𝜏 = 0.55, at which the kill detector reaches an optimal F1 score of 0.90, with a precision of 0.86 and a recall of 0.95. Further calibration details are provided in the replication package [3]. The mutation kill ratio we report therefore corresponds to the number of killed mutants divided by the total number of mutants. Results. The overall kill ratio was 82.4%. Figure 3b presents the results for each combination of model, prompt, and dataset. Mutants derived from the Easy ModelSet were killed more frequently than those from the Golden ModelSet, as injected defects may constitute a more evident departure in smaller and more regular diagrams. Pooled by model, gemma4:e4b obtains the lowest kill ratio at 73.4%, compared with 86.6% for gpt-5-mini and 87.1% for gpt-5.3-codex. Pooled by prompt, P2 again outperforms P1, with kill ratios of 87.0% and 77.7%, respectively.

4.3

RQ3. Comparative Evaluation

RQ1 and RQ2 measure different and complementary aspects of judge performance. RQ3 examines their convergent validity as instruments for comparing judge configurations. Setup. For the six judge configurations, we aggregated results across both datasets and constructed one entry for each configuration, containing both the corresponding manual precision and computed mutation kill ratio. To assess the correlation between the two measures, we computed Pearson’s correlation coefficient.

Dataset: Golden ModelSet 75% 56%

30%

25% 0%

3%

gemma4:e4b

gpt-5-mini

Precision

Dataset: Easy ModelSet Prompt P1 P2

gpt-5.3-codex

Prompt P1 P2 26%

42%

gemma4:e4b

100% 75% 50% 25% 0%

95%

Dataset: Easy ModelSet

76%

91%

87%

87%

36%

53%

gpt-5-mini

68%

60%

gpt-5.3-codex

Prompt P1 P2 gemma4:e4b

Dataset: Golden ModelSet

96%

Kill ratio

4.2

100% 75% 50% 25% 0%

(a) RQ1: Manual issue-level precision. Kill ratio

subjective, even for annotators with relevant modeling expertise. The adjudication process was therefore necessary to obtain a stable reference for the precision analysis. Figure 3a reports precision for each combination of model, prompt, and dataset. Pooled by model, gpt-5.3-codex achieves the highest precision at 63.7%, compared with 31.6% for gemma4:e4b and 35.5% for gpt-5-mini. Pooling by prompt, P2 reaches 48.6% precision, compared with 26.9% for P1.

Precision

Figure 2: Overview of the evaluation procedure.

gpt-5-mini

gpt-5.3-codex

65%

79%

83%

89%

83%

90%

Prompt P1 P2 gemma4:e4b

gpt-5-mini

gpt-5.3-codex

(b) RQ2: Mutation kill ratios.

Figure 3: Results for RQ1 and RQ2 across each LLM, prompt variant, and dataset. We also computed pairwise ordering agreement, defined as the proportion of configuration pairs for which the two instruments rank one configuration above the other in the same direction. Results. Across the six conditions, manual precision and mutation kill ratio have a Pearson correlation of 𝑟 = 0.624, indicating a
moderate-to-strong positive linear association. Among the 62 = 15 condition pairs, the two instruments produce the same ordering for 11 pairs (agreement rate of 73.3%). Some orderings were not conserved. For example, gpt-5-mini frequently detected injected defects (high kill ratio), while also reporting many unsupported issues on non-mutated diagrams (low manual precision).

5

Discussion

Mutation Testing as an Evaluation Instrument. The evaluation highlights that manual annotation and mutation testing are complementary instruments, as a judge may perform well on one dimension without performing well on the other. This is particularly important for verbose judges, as reporting many possible issues increases the probability of matching an injected defect and can therefore increase the kill ratio, even when many of the additional issues are invalid. This behavior was observed in RQ1, especially in the Easy ModelSet results. Nevertheless, RQ3 shows that mutation testing captures a useful comparative signal, leading to the same conclusions regarding prompt and LLM performance. Because mutant generation and kill detection are largely automated mutation testing is suited to preliminary comparisons and targeted diagnosis of known defect categories. Manual evaluation remains necessary to assess false positives, ambiguous judgments, and aspects of model quality not covered by the mutation operators. Diagnostic Value of the Mutation Operators. Aggregate kill ratios conceal substantial differences among defect categories. Table 3 complements the overall score with results for each mutation operator. M10 and M11, which add unrelated elements, are detected

MODELS 2026, October 04–09, 2026, Málaga, Spain

Delcourt et al.

particularly frequently. These operators remain useful as sanity checks, but if such easy mutants dominate the aggregate score, they can make judge performance appear stronger than it is on subtler modeling defects. They could therefore be retained as a separate basic-capability category, or omitted from repeated experiments when computational cost is the primary concern. Table 3: Overall kill ratio by mutation operator. Operator

M1

M2

M3

M4

M5

M6

M7

M8

M9

M10

M11

Kill ratio (%)

72.6

62.3

72.8

77.5

76.3

86.6

92.6

76.5

77.2

95.8

97.8

In contrast, operators with lower kill ratios identify concrete blind spots in the judge. A low score for relationship-orientation mutations, for example, suggests that the judge does not reason reliably about association semantics. This diagnostic information is difficult to obtain from a single aggregate comparison with human judgments, and can directly inform prompt refinement or the definition of additional task-specific examples. The mutation operators also make the evaluation requirements explicit. Their definition encodes the modeling experts’ assumptions about completeness, abstraction, relevance, and relationship semantics. This is an advantage when the intended criteria are explicit, but it also means that the resulting score is only meaningful with respect to the selected operators and their distribution. Threats to Validity. The RQ1 reference is based on 60 judgments over ten baseline entries. Only five of the 45 Golden ModelSet entries could be annotated due to limited capacity. These five were selected at random to limit selection bias, but a larger sample could change the observed precision values and the relative ordering of configurations. Moreover, manual annotation evaluates only reported issues and therefore doesn’t provide an exhaustive measure of semantic correctness. The initial Krippendorff’s 𝛼 = 0.32 also shows that the labels are sensitive to subjective interpretations. Adjudication resolves disagreements for analysis purposes, but does not eliminate the underlying ambiguity of the task. Mutation testing assumes that transformations degrade an acceptable model–description pair. This assumption can fail when a baseline model is of poor quality, e.g., already contains superfluous elements. We manually inspected 79 difficult cases identified during pilot runs and found that 69 were confirmed as valid degradations, while 10 were borderline mutants for which the transformation had little negative effect. These mutants were retained, which may underestimate the kill ratio of a competent judge. The RQ2 results depend on the semantic-similarity detector described in Section 4.2. Although the selected threshold achieves an F1 score of 0.90 on the calibration data, semantic similarity can still produce both false matches and missed paraphrases. The threshold was calibrated using judgments from gpt-5-mini; differences in wording across models or prompts may therefore affect transferability. A larger, independently held-out test set and comparisons with alternative embedding-based, entailment-based, structuredmatching, or LLM-based detectors would provide stronger evidence for the validity of the kill decisions. Limitations and Research Opportunities. Our study represents an initial investigation of mutation testing as an evaluation instrument for LLM-based semantic judges. The mutation operators target semantic defects in PlantUML domain class diagrams.

Although many of the underlying concepts (e.g., completeness, relevance, relationship semantics, and abstraction) are common across conceptual modeling languages, we do not have evidence on how the approach generalizes to other modeling notations, DSLs, or different semantic tasks. Thus, future research can investigate how mutation-testing performs as an evaluation approach for semantic tasks on other modeling languages and domains. Furthermore, although the choices of prompts and LLMs were sufficient to demonstrate the feasibility of the approach and to compare to manual analysis (RQ3), additional LLMs and prompts can be tested to explore the robustness of the observed trends from our results. Considering the mutations, our injection strategy applies a single semantic defect to each mutant. This design is limited regarding the complexity of real-world models, which often contain multiple interacting inconsistencies. This leads to opportunities to investigate richer mutation strategies that introduce combinations of defects with varying levels of subtlety and interaction. For example, one can adapt previously proposed UML mutation operators by ensuring they inject semantic defects in UML diagrams beyond class. Similarly, testing our approach with larger diagrams (thousands of elements) is challenging, as guaranteeing that the injected defects break the semantics is harder. More importantly, detecting the injected defects may be difficult for LLMs as larger models will lead to a larger context to be analyzed. Hence, future directions can investigate how to optimize the mutant injection in larger models, as well as how to pre-filter the context that the LLMs need to analyze to detect the defects, concerning the scalability of the mutation-based principle.

6

Conclusion

This work proposes a mutation-testing approach for evaluating LLM-based semantic judges of domain class diagrams against textual descriptions. We define a set of domain-class-diagram mutation operators, generate controlled semantic defects, and compare the usefulness of the approach to manually measured judge precision across multiple models and prompt variants. The results support mutation testing as a practical mechanism for the iterative development of semantic model judges. Once mutation operators and a kill detection strategy have been established, new models or prompts can be evaluated automatically against the same defect families. This makes it possible to track regressions, identify capability gaps, and refine prompts before investing in a larger expert annotation campaign. The approach is particularly attractive in MDE, where experts’ time is scarce and semantic tasks do not admit a unique ground truth. More broadly, our objective is to facilitate the engineering of reliable judges for semantic MDE tasks. Reliable semantic judges could then support downstream MDE activities such as model repair, interactive modeling assistance, quality control for generated-model datasets, and the validation of AI-based modeling tools.

Acknowledgments This work was partially funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), grant number RGPIN-2025-05677.

Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams

References [1] Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014). 81–91. [2] Khaled Ahmed, Jialing Song, Boqi Chen, Ou Wei, and Bingzhou Zheng. 2025. MCeT: Behavioral Model Correctness Evaluation using Large Language Models. In 2025 ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, 84–95. [3] Anonymous. 2026. Breaking Models to Test the Judge: A Mutation Testing Approach for Semantic Evaluators of Domain Class Diagrams (Replication Package). doi:10. 5281/zenodo.21084106 [4] Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, et al. 2024. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 7421–7454. [5] Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, et al. 2025. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 238–255. [6] Meriem Ben Chaaben, Lola Burgueño, Istvan David, and Houari Sahraoui. 2025. On the Utility of Domain Modeling Assistance with Large Language Models. ACM Trans. Softw. Eng. Methodol. (June 2025). doi:10.1145/3744920 [7] Christoph Bockisch, Deniz Eren, Sascha Lehmann, Daniel Neufeld, and Gabriele Taentzer. 2024. Mutation Testing of Java Bytecode: A Model-Driven Approach. In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems. 237–248. [8] Marco Brambilla, Jordi Cabot, and Manuel Wimmer. 2017. Model-driven software engineering in practice. Morgan & Claypool Publishers. [9] Antonio Bucchiarone, Jordi Cabot, Richard F Paige, and Alfonso Pierantonio. 2020. Grand challenges in model-driven engineering: an analysis of the state of the research: A. Bucchiarone et al. Software and Systems Modeling 19, 1 (2020), 5–13. [10] Lola Burgueño, Davide Di Ruscio, Houari Sahraoui, and Manuel Wimmer. 2025. Automation in Model-Driven Engineering: A Look Back, and Ahead. ACM Trans. Softw. Eng. Methodol. 34, 5 (2025). [11] Javier Cámara, Javier Troya, Lola Burgueño, and A Vallecillo. 2023. On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML: JJ. Cámara et al. Software and Systems Modeling 22, 3 (2023), 781–793. [12] Thibaut Capuano, Houari Sahraouiand Benoit Frenay, and Benoit Vanderose. 2022. Learning from Code Repositories to Recommend Model Classes. Journal of Object Technology 21, 3 (2022). [13] Kua Chen, Yujing Yang, Boqi Chen, José Antonio Hernández López, G Mussbacher, and D Varró. 2023. Automated domain modeling with large language models: A comparative study. In 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, 162–172. [14] Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. 2025. On the Effectiveness of LLM-as-ajudge for Code Generation and Summarization. IEEE Transactions on Software Engineering (2025). [15] Philipp-Lorenz Glaser, Lola Burgueño, and Dominik Bork. 2026. A Benchmarking Framework for Model Datasets. arXiv preprint arXiv:2603.05250 (2026). [16] Pablo Gómez-Abajo, Esther Guerra, Juan de Lara, and M G Merayo. 2021. WodelTest: a model-based framework for language-independent mutation testing: P. Gómez-Abajo et al. Software and Systems Modeling 20, 3 (2021), 767–793. [17] Maria Fernanda Granda, Nelly Condori-Fernández, Tanja EJ Vos, and Oscar Pastor. 2016. Mutation operators for UML class diagrams. In International Conference on Advanced Information Systems Engineering. Springer, 325–341. [18] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. The Innovation (2024). [19] Esther Guerra, Jesús Sánchez Cuadrado, and Juan de Lara. 2019. Towards effective mutation testing for ATL. In 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, 78–88. [20] Zakaria Hachm, Théo Le Calvar, Hugo Bruneliere, and Massimo Tisi. 2025. Towards LLM Agents for Model-Based Engineering: A Case in Transformation Selection. In 2025 ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). IEEE, 421–431. [21] Tobias John, Einar Broch Johnsen, and Eduard Kamburjan. 2026. Mutation-based testing of knowledge graphs. Empirical Software Engineering 31, 4 (2026), 99. [22] Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability. (2011). [23] Grischa Liebel, Jil Klünder, Regina Hebig, Christopher Lazik, et al. 2024. Human factors in model-driven engineering: future research goals and initiatives for MDE. Software and Systems Modeling 23, 4 (2024), 801–819. [24] Jean-Marie Mottu, Benoit Baudry, and Yves Le Traon. 2006. Mutation analysis testing for model transformations. In European Conference on Model Driven

MODELS 2026, October 04–09, 2026, Málaga, Spain

Architecture-Foundations and Applications. Springer, 376–390. [25] Bhuvanashree Murugadoss, Christian Poelitz, Ian Drosos, Vu Le, Nick McKenna, Carina Suzana Negreanu, Chris Parnin, and Advait Sarkar. 2025. Evaluating the evaluator: Measuring llms’ adherence to task evaluation instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 19589–19597. [26] Vittoriano Muttillo, Romina Eramo, R Rubei, and L Berardinelli. 2025. Coupling LLMs and Model-Driven Engineering to Support Synthetic Generation of BPMN Artifacts. In 2025 ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). IEEE, 556–565. [27] H James Nelson, Geert Poels, Marcela Genero, and Mario Piattini. 2012. A conceptual modeling quality framework. Software Quality Journal 20, 1 (2012), 201–228. [28] Lukas Netz, Judith Michael, and Bernhard Rumpe. 2024. From Natural Language to Web Applications: Using Large Language Models for Model-Driven Software Engineering. In Modellierung 2024. Gesellschaft für Informatik eV, 179–195. [29] Qian Pan, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. 2024. Human-Centered Design Recommendations for LLM-as-a-judge. In Proceedings of the 1st HumanCentered Large Language Modeling Workshop. 16–29. [30] Oszkár Semeráth, Rebeka Farkas, Gábor Bergmann, and Dániel Varró. 2020. Diversity of graph models and graph generators in mutation testing: O. Semeráth et al. International Journal on Software Tools for Technology Transfer 22, 1 (2020), 57–78. [31] Sentence Transformers. 2021. all-MiniLM-L6-v2. https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2. Hugging Face model card. Accessed: 2026-06-28. [32] Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2025. Judging the judges: A systematic study of position bias in llmas-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics. 292–314. [33] Daniel Strüber. 2017. Generating efficient mutation operators for search-based model-driven engineering. In International Conference on Theory and Practice of Model Transformations. Springer, 121–137. [34] Frank Tip, Jonathan Bell, and Max Schäfer. 2025. Llmorpheus: Mutation testing using large language models. IEEE Transactions on Software Engineering (2025). [35] Javier Troya, Alexander Bergmayr, Loli Burgueno, and Manuel Wimmer. 2015. Towards systematic mutations for and with ATL model transformations. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 1–10. [36] Pablo Valle, Shaukat Ali, and Aitor Arrieta. 2026. Exploring the Potential of Large Language Models in Simulink-Stateflow Mutant Generation. arXiv preprint arXiv:2602.04066 (2026). [37] Charlotte Verbruggen, Lukas Netz, Philipp-Lorenz Glaser, Marion Scholz, Christian Huemer, Marco Calamo, Bernhard Rumpe, Monique Snoeck, and Dominik Bork. 2025. Toward a Community-Curated Golden Dataset of UML Models. In 2025 ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). IEEE, 43–50. [38] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796 (2024). [39] Song Yang and Houari Sahraoui. 2022. Towards automatically extracting UML class diagrams from natural language specifications. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 396–403. [40] Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2025. Justice or prejudice? quantifying biases in llm-as-a-judge. In International Conference on Learning Representations, Vol. 2025. 102351–102390. [41] Haopeng Zhang, Philip S Yu, and Jiawei Zhang. 2025. A systematic survey of text summarization: From statistical methods to large language models. Comput. Surveys 57, 11 (2025), 1–41. [42] Shiqing Zhang, Yijiao Yang, Chen Chen, Xingnan Zhang, Qingming Leng, and Xiaoming Zhao. 2024. Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects. Expert Systems with Applications 237 (2024), 121692. [43] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Zican Dong, Yupeng Hou, Beichen Zhang, Yingqian Min, Junjie Zhang, Peiyu Liu, et al. 2026. A survey of large language models. Frontiers of Computer Science 20, 12 (2026), 2012627. [44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, et al. 2023. Judging LLM-as-ajudge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 2020, 29 pages.

Related documents

• eProcurement Ontology Fulfilment - core#128674
• Towards Term-based Verification of Diagrammatic Equivalence#677497
• Towards Term-based Verification of Diagrammatic Equivalence#674299
• Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development#1051889
• Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development#1043163