Comparing Domain-Model Similarity Metrics Against Human Expert Ratings

Comparing Domain-Model Similarity Metrics Against Human Expert Ratings Vasiliy Seibert Institute for Software and Systems Engineering, TU Clausthal Clausthal-Zellerfeld, Lower Saxony, Germany vasiliy.seibert@tu-clausthal.de

arXiv:2608.15250v1 [cs.SE] 15 Aug 2026

Abstract Domain models are a primary artefact in model-driven software engineering, where they capture the shared understanding between stakeholders and serve as the contractual basis for downstream software development. Automatic comparison of these semantic models has diverse application areas such as requirements engineering, education, automatic generation of domain models and model reuse and repository mining. The literature offers a variety of presented metrics, but for practitioners there is no defensible way to choose between them. The contribution of this paper is the implementation of five such metrics, their execution on a fixed set of 39 domain-model comparisons and the comparison of each metric’s output against the human expert ratings produced for the same comparisons. Two research questions are addressed. RQ1 asks how close, on average, each metric is to the human expert rating across the 39 comparisons. RQ2 asks how consistent each metric’s per-comparison distance from the human expert rating is. The findings reveal that no single metric achieves dominance across all criteria; rather, different metrics each yield competitive results on individual criteria — some closest on average, others best preserving the per-pair ordering — which suggests that an ensemble approach combining multiple metrics may serve as a viable substitute for human expert grading. The metric implementations are an artefact of this work and are published in accordance with the FAIR4RS recommendations (https://doi.org/10.5281/zenodo.20942596).

CCS Concepts • Software and its engineering → Software creation and management → Software verification and validation;

Keywords domain modeling, class diagrams, large language models, evaluation, metrics, no-free-lunch ACM Reference Format: Vasiliy Seibert. 2026. Comparing Domain-Model Similarity Metrics Against Human Expert Ratings. In . ACM, New York, NY, USA, 11 pages. https: //doi.org/10.1145/nnnnnnn.nnnnnnn

1

Introduction

Domain models are a primary artifact in model-driven software engineering (MDSE) [5, 6, 22]. A domain model captures the entities, attributes, and associations of an application domain in a notation that is read by stakeholders from different backgrounds — domain Conference’17, Washington, DC, USA 2026. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn

experts, requirements engineers, software architects, and developers — and serves as a basis for downstream software development. The diagram’s role is therefore twofold: it is the artefact on which all participants converge to gain a mutual understanding of the system to be built, and it is the input to subsequent design, code generation, and verification activities. A well-defined, deterministic metric for comparing domain models would benefit multiple areas. In automated generation of domain models [5, 6, 20, 21], the metric is the measurement of a generation approach’s performance against a reference. In education [3, 4, 10, 16], the metric could automate the grading of student-submitted class diagrams against an instructor’s reference and shorten the feedback loop. In requirements engineering [19, 22], the metric could let a modeler compare newly created models with models from other projects. In model reuse and repository mining [17–19, 22], the metric is the basis on which a repository query ranks candidates, a design-pattern detector flags matches, and a reuse decision is made. Without a well-defined, deterministic metric, all of these activities reduce to manual inspection. The metrics that have been proposed in the literature are diverse in approach. One approach is rule-based mistake detection [10, 16]: the metric enumerates a list of required-model properties (e.g., expected classes, expected attributes, expected associations) and reports the missing or extra elements as a mistake list, scored at the level of the mistake type. Another approach uses graph edit distance (GED) on attributed relational graphs [19]: the metric reduces the similarity between two models to the cost of the cheapest sequence of edit operations that transforms one graph into the other, and the similarity is reported as a normalised score derived from that cost. Yet another approach is structural matching on a domain-specific graph representation, exemplified by the UML Class Graph (UCG) [22]: the metric decomposes the similarity into an intra-cluster component (the quality of grouping related elements within one model) and an inter-cluster component (the alignment of clusters across the two models). A further approach is a semantic+structural pipeline [18]: the metric separates the structural similarity of the two models (computed on the diagram’s structure) from the semantic similarity (computed on the lexical information in the class names) and combines the two. One more approach is deep-learning-based similarity, exemplified by the SimGNN approach to UML use case models [17], in which the candidate and reference models are encoded as graphs and a graph neural network produces the similarity score.

Conference’17, July 2017, Washington, DC, USA

The diversity of the proposed metrics, combined with the fact that each is applied on its own dataset means that the field has many candidate metrics but no defensible way to choose between them. So that the field has arguments grounded in a common dataset for choosing one similarity metric over another, this paper implements five metrics from the literature [16, 18, 19, 22] and compares their outputs against human expert ratings on a fixed set of 39 domainmodel comparisons [6]. Two research questions are formulated before the comparison. RQ1 is formulated as follows: how close, on average, is each metric to the human expert rating across the 39 comparisons? The question matters because alignment with the human judgement on average is an indicator that the metric captures the notion of similarity that a human grader would apply, and is the operational form of a metric’s central tendency. RQ2 is formulated as follows: how consistent is each metric’s per-comparison distance from the human expert rating? The question matters because a metric that systematically undervalues or overvalues similarity can still be useful — what matters for many practical applications is that the metric preserves the ordering of similarity across pairs, so that two diagrams judged more similar by the human are also judged more similar by the metric; the per-comparison consistency is the operational form of a metric’s reliability as a ranker. Two risks attend this approach. The first is the risk of an incorrect implementation: the metrics in the literature are described at the level of a published paper, and a faithful re-implementation requires a careful reading of the source, which is not always unambiguous; an implementation error in any of the 5 metrics would invalidate the cross-metric comparison for that metric. The second is the risk of an unfair translation: the 5 metrics emit scores in different units, and the comparison requires each score to be translated into a per-element F1 (class, attribute, association) in the same scale as the human expert rating, a translation that is per-metric and that risks putting some metrics on a scale they were not designed to operate in. To address the first risk, a specification is derived from the publication of each metric, two different large language models implement the same specification, and the two implementations are tested for deterministic agreement: if both implementations produce the same result, the specification is assumed to be well-defined and the implementations are assumed to be correct, with the specification serving as the explicit record of the metric as understood in this work. To address the second risk, the specification is extended to translate the metric’s output into the per-element F1 scale of the human expert rating, and the translation is tested for partial-order preservation against the metric’s native output, so that the ordering of the metric results is consistent with the ordering of the original output across the 39 comparisons. To encourage reuse, the metric implementations are a supplementary research artifact to this work in accordance with the FAIR4RS recommendations [7] https://doi.org/10.5281/zenodo.20942596

Vasiliy Seibert

2 Background and Related Work 2.1 Related Work Education is one application area that would benefit from an established Domain Model Similarity Metric. Modi, Taher, and Mahmud [10] develops a Java-based tool that reads student and instructor solution diagrams as input and produces an automatic grade for the UML diagram types submitted in software-engineering and system-analysis courses. Boubekeur, Mussbacher, and McIntosh [4] combine a simple heuristic with machine-learning techniques to predict approximate letter grades. Bouali et al. [3] compares LLMgenerated scores against three teaching assistants on 92 student submissions of UML class diagrams in a software-design course. Singh, Boubekeur, and Mussbacher [16] propose a Mistake Detection System (MDS) that is a rule-based system which automatically indicates the exact location and type of the mistake in a student solution when compared with a correct solution, covering 83 out of 97 identified different types of mistakes that may exist in a student solution. On real student solutions, the authors report that, when synonyms are considered by MDS, the system achieves a recall of 0.93 and a precision of 0.79. The metric is evaluated on its own real-student dataset and is not cross-compared with other modelsimilarity metrics in the paper. Fauzan, Siahaan, Rochimah, and Triandini [8] decomposes the assessment into a structural similarity, calculated using the diagram’s structure while ignoring its lexical information, and a semantic similarity, calculated using the lexical information in the class diagram. The authors report that experts see semantic and structural similarities equally during assessment, and that the proposed approach shows agreement with experts in class-diagram similarity assessment. The two metrics are validated against expert agreement on the authors’ own student-diagram dataset. General model-driven engineering and requirements engineering are additional application areas. Chen, Yang, Chen, Hernández López, Mussbacher, and Varró [6] conduct a comparative study of using large language models for fully automated domain modeling on a dataset of examples with reference solutions created by modeling experts, framing domain modeling as a core softwareengineering activity. Chen, Chen, Varró, and Mussbacher [5] explore the use of GPT-4 for creating goal-oriented models in the Goal-oriented Requirement Language (GRL), positioning their work as a requirements-engineering application of LLM-based modeling. Yang, Chen, Chen, Mussbacher, and Varró [21] extend the same line of LLM-based domain modeling with a multi-step iterative framework and a self-reflection mechanism, again targeting general software-engineering modeling. Yang and Sahraoui [20] target model-driven engineering by proposing an automated approach for extracting UML class diagrams from natural-language software specifications. Model reuse and model-repository mining are a further application area. Song, Wang, Wang, Lin, and Hu [17] list model reuse, model validation, model retrieval, and pattern detection as concrete use cases for similarity calculation of UML models, and they focus their work on use case models, for which they observe that prior similarity-calculation research has been limited; they transform

Comparing Domain-Model Similarity Metrics Against Human Expert Ratings

UML use case models into use case graphs and then adopt the Similarity Graph Neural Network (SimGNN) to calculate the structural similarity between the resulting graphs, while semantic similarity is calculated separately with Term Frequency-Inverse Document Frequency (TF-IDF) and cosine similarity, and the two results are then combined; the authors report that their approach is evaluated by comparison to two baselines, namely cosine similarity combined with TF-IDF and cosine similarity combined with edit distance, and that it achieves higher accuracy and lower runtime than both. Čech [19] introduces a class-model distance computation framework that can be used for comparing class models in model repositories, with application to both pairwise model comparison and design-pattern detection. Yuan, Yan, and Ma [22] represent a UML class diagram as a graph called UCG and decompose the structural similarity of a class diagram into two aspects: intra-structure, which refers to the composition of each class, and inter-structure, which refers to the relationships between classes. For inter-structure similarity they propose an algorithm based on UCG Maximum Common Subgraph Sequence, and for intra-structure similarity they introduce UCG edit distance. The authors report experimental results both within a domain and across domains, but do not cross-compare their structural similarity measure with prior semantic or structural UML similarity metrics.

3 Comparison Methodology 3.1 Automated Domain Modeling with Large Language Models Chen, Yang, Chen, Hernández López, Mussbacher, and Varró [6] introduce a study in which they compare LLM-generated against expert-authored reference models. This paper uses this comparison as the basis for the literature-metric comparison in this work. The reference models span ten application domains: a bus transportation management system (BTMS), a second-hand delivery and pickup system (H2S), a medical laboratory test management system (LabTracker), a celebrations and order tracking system (CelO), a team sports management system (TeamSports), a smart home automation system (SHAS), a university OTS system, two board games (Block, Tile-O), and a hotel booking management system (HBMS). The Reference Models are mostly expressed as images. The authors apply Davinci (text-davinci-003, the GPT-3.5 completion model) [11], Turbo (gpt-3.5-turbo) [13], and GPT-4 [12] using five prompt settings: 0-shot (task instruction plus target description only), 1-shot-btms (one BTMS exemplar), 1-shot-h2s (one H2S exemplar), 2-shot (BTMS and H2S combined as exemplars), and chain-of-thought (CoT, sentence-by-sentence reasoning over H2S). Generated Domain Model The LLM Generated Domain Models describe Enumerations, Classes and Relationships according to the following structure: Enumerations: (...) TestResult(NEGATIVE, POSITIVE) Classes: (...)

Conference’17, July 2017, Washington, DC, USA

Requisition(validFromDate: Date) Test(group: String, duration: Integer) (...) Relationships: 1 Doctor associate 0..* Requisition 1 Patient associate 0..* Requisition 1 Requisition contain 1..* Test 1 Test associate 0..1 Appointment 1 Lab associate 0..* Appointment Inheritance: None Composition: (...) 1 Test contain 1 TestOutcome(result: TestResult, (...))

The eight models (ten examples minus BTMS and H2S, which appear in the prompts and are therefore excluded) contain in total 135 classes, 221 attributes, and 177 relationships, with the tenexample totals being 155 classes, 250 attributes, and 204 relationships. Crossed with the three LLMs and the five prompt settings, the paper therefore contains 3 × 8 × 5 = 120 generated-vs-reference comparisons. The grading itself is performed by human raters in a two-round consensus process. Each generated model is graded against the corresponding reference model per element type — class, attribute, relationship by applying four categories (c1, c2, c3 and c4) distinguishing exact matches, semantically equivalent matches, partial matches, and elements with no match in the reference model. Category 𝑐 1 captures generated elements that have a direct semantic counterpart of the same type in the reference model, e.g., the Person class of the reference model, which is matched against the User class of the generated model because both capture the same domain concept. Category 𝑐 2 captures generated elements that match an element of a different type in the reference model but are semantically equivalent to that element, e.g., the SecondHandArticle class, which the reference model represents with a boolean attribute discarded and the generated model represents with an enumeration Status taking the two literals AVAILABLE and DISCARDED; the two representations are semantically equivalent but differ in type. Category 𝑐 3 captures elements that only partially match a reference element, e.g., the 0..1 Route associate * Item relationship of the reference model, which is partially matched when the generated model marks Route as non-optional (1 Route associate * Item, a multiplicity mistake). Category 𝑐 4 captures generated elements that have no match in the reference model, e.g., the abstract UserRole class of the reference model, which has no equivalent modelling element in the generated model.

  1    𝑆 (𝑥) = 0.5   0 

if 𝑥 ∈ 𝑐1 or 𝑐2, if 𝑥 ∈ 𝑐3, if 𝑥 ∈ 𝑐4,

Conference’17, July 2017, Washington, DC, USA

Vasiliy Seibert

To turn the four categories into a numeric score per element, the authors define a scoring function 𝑆 (𝑥) that awards 1 point to generated elements in 𝑐 1 or 𝑐 2 (direct or semantically equivalent match), 0.5 points to elements in 𝑐 3 (partial match), and 0 points to elements in 𝑐 4 (no match). For each modelling element type, the precision and recall are then computed by summing 𝑆 (𝑥) over the generated and the reference elements separately, and dividing by the respective set sizes. For classes with 𝑚 generated elements and 𝑛 reference elements,

} Doctor "1" -- "0..*" Requisition Patient "1" -- "0..*" Requisition Requisition "1" *-- "1..*" Test Test "1" -- "0..1" Appointment Lab "1" -- "0..*" Appointment Test "1" *-- "1" TestOutcome @enduml

Í𝑖=𝑚 Precision𝐶 =

𝑖=1 𝑆 (𝐶𝑖 )

,

(1)

𝑚

Í𝑖=𝑛 Recall𝐶 =

𝑖=1 𝑆 (𝐶𝑖 )

,

(2)

𝑛

F1𝐶 =

2 × Precision𝐶 × Recall𝐶 . Precision𝐶 + Recall𝐶

(3)

and the same definitions apply with 𝐴 for attributes and 𝑅 for relationships (Equations 1–4 of [6]). The procedure is defined three times rather than once over an aggregate score, and the output of the grading is a 3-tuple of F1 values, ⟨F1𝐶 , F1𝐴 , F1𝑅 ⟩, one per element type. This 3-tuple is the de-facto output of the manual grading, and it is the unit of comparison that this work reuses as the ground truth.

3.2

Reusing Human Expert Ratings

In order to reuse the human expert ratings and test automated metrics against them, we must translate the data into a format that can be reliably processed by a parser. The Chen et al. paper distributes its models only as rendered images and a CSV of judgements. We therefore translated the 8 reference models and 39 generated models into PlantUML [15], producing a dataset of 39 PlantUML comparisons. Each comparison pairs a reference model with a generated model and carries the corresponding human F1 3-tuple ⟨F1𝐶 , 𝑒𝑥𝑡𝐹 1𝐴 , 𝑒𝑥𝑡𝐹 1𝑅 𝑎𝑛𝑔𝑙𝑒 from §3.1. Translated Generated Domain Model to PlantUML The LLM Generated Domain Models were manually translated to PlantUML: @startuml enum RepetitionInterval { WEEKLY MONTHLY HALF_YEARLY YEARLY } class Requisition { validFromDate : Date } class Test { group : String duration : Integer

As a consequence, every candidate metric in this study must accept as input two PlantUML domain-model strings — one reference, one generated — and output a 3-tuple of similarity scores ⟨𝑠𝐶 , 𝑠𝐴 , 𝑠𝑅 𝑎𝑛𝑔𝑙𝑒, one per modelling element type, with each 𝑠 ∈ [0, 1]. The three scores are aligned with the human categories. Metric M compute(ReferencePlantUML, GeneratedPlantUML) -> (class_score : [0, 1], attribute_score : [0, 1], association_score : [0, 1])

4

Implementation Methodology

An incorrect implementation of any of the five metrics would confound the comparison, because disagreement between a metric and the human expert rating could then be attributed to the implementation rather than to the metric itself, and the comparison’s validity rests on every implementation being a faithful restatement of its source paper. To address this risk, each metric is first restated as a specification before any code is written. The specification is expressed in a markdown file and Python Scripts containing docstrings, which docompose the metric into its functional-logical components and declares, for each component, the preconditions that its inputs must satisfy and the post-conditions that its output must satisfy, in the style of design by contract [9]. The specification is then handed, unchanged, to two independent large language models, each of which produces a separate Python implementation inside the predefined Python Scripts. The two implementations are executed on 39 domain-model comparisons that §3 describes, and they are required to produce identical output on every comparison; if they do, the specification is assumed to be unambiguous and the implementations are assumed to be a correct implementation. The specification, not the code, is the artifact that the comparison is based on: it is the explicit record of the metric as understood in this work, and every assumption or deviation that the implementation makes is written into it so that the comparison can be audited against the source paper. The second stage of the method extends the specification to translate the metric’s native output into the per-element (class, attribute, association) scale of the human expert rating, so that the five metrics become comparable both to each other and to the human judgement. The structure of a specification is illustrated by the following example, which restates the rule-based mistake detector of Singh, Boubekeur and Mussbacher [16]: metric(instructor_model, student_model):

Comparing Domain-Model Similarity Metrics Against Human Expert Ratings

Conference’17, July 2017, Washington, DC, USA

Figure 1: Spec → invariants → @icontract → code pattern. mapping |= mapClasses(instructor_model, student_model) mapping |= mapRelations((...)) mistakes = checkClasses(mapping) mistakes += checkRelations(mapping) mistakes += checkMissing(mapping) return mistakes requires: isValidModel(instructor_model), isValidModel(student_model) ensures: isValidMistakes(mistakes) (...) isValidModel(M): checks if model adheres to the data structure. class names are unique within M enumeration names are unique within M every relation references classes that exist in M every attribute belongs to exactly one class in M (...)

The example opens with a top-level metric function that expects two models as input, and both inputs must adhere to the isValidModel precondition; the function returns a list of mistakes, which must adhere to the isValidMistakes postcondition. The pre- and post-conditions are defined beforehand by the authors against the source paper, not by the LLM, so that the contract is fixed before any implementation is produced. Beneath the top-level function, the example decomposes the metric into five sub-functions — mapClasses, mapRelations, checkClasses, checkRelations, checkMissing — each with its own requires/ensures pair, so that the contract is layered: the post-condition of one function becomes the precondition of the next. The Invariants (e.g. isValidModel, isValidMapping, and isValidMistakes) are defined as @icontract.require or @icontract.ensure [14] decorators in the Python Scripts, where they are checked at runtime on every function call. The specification is therefore not only documentation but a contract that the code is

obliged to satisfy. The two independent implementations of each specification are produced by two open source large language models from different families, [1] and [2], chosen so that agreement between the two is not an artefact of a single model agreeing with itself. Both models are driven through the same agentic harness, opencode, which gives each model equivalent access to the specification file, the shared PlantUML parser, and the test directory, so that any difference in the produced implementation is attributable to the model’s interpretation of the specification rather than to differential tooling. Each implementation is produced in a fresh session from the same specification prompt, and the two sessions do not see each other’s code, which is the independence that the agreement test in the next paragraph relies on. The role of the large language models is deliberately bounded: the models implement the specification, they do not author it, and they do not judge whether the two implementations agree; the specification is written by the authors against the source paper, and the agreement check is a deterministic execution of both implementations on the same 39 inputs. The first test procedure, tS-1, follows a test-driven-development approach: the pre- and post-conditions are exposed to the large language model in the specification, and are embedded in the Python skeleton files as @icontract.require and @icontract.ensure decorators, so that they are executed on every function call; a contract violation raises an explicit runtime error rather than degrading silently. The second test procedure, tS-2, checks the two redundant implementations for determinism: both are executed on the 39 domain-model comparisons, and if they produce identical output on every pair, the specification is taken to be unambiguous; if they disagree, the specification is amended and the implementations are regenerated. The third test procedure, tS-3, is applied after the specification is extended to the second stage, where the metric’s native output is projected into the per-element (class, attribute, association) scale, and it verifies that the projection preserves the

Conference’17, July 2017, Washington, DC, USA

partial ordering of the native output, so that the relative ordering of model pairs is consistent between the first-stage output and the projected output.

4.1

Literature Metrics

Five candidate metrics are derived from four source papers. Metrik-1 (M-1) is based on the rule-based mistake detector of Singh, Boubekeur and Mussbacher [16]. The paper describes 97 distinct mistake types but reports that the presented rules cover only 83 of the 97 types; the remaining 14 types are flagged as out of scope. The native output of M-1 is a mistake list : a deduplicated, sorted list of detected mistakes. To extend the Metric, the mistake list is projected into the required triple of class-score, attribute-score, association-score. The projection works by bucketing the detected mistakes into three groups by their mistake identifiers and normalising each count by the corresponding element count of the instructor model, so that each score is 1. Metrik-2 (M-2) is based on the graph-edit-distance metric proposed by Čech [19] for attributed relational graphs. The method reduces the similarity between two models to the cost of the cheapest sequence of edit operations (insert node, delete node, insert edge, delete edge, substitute attribute) that transforms one graph into the other, where the cost weights are paper-specific. The native output of M-2 is a single normalised similarity scalar in 0,1. To extend the Metric, the scalar is projected into the required triple of class-score, attribute-score, association-score. The projection decomposes the aggregate edit cost into three components: vertex-level edit operations produce class-score, edge-level edit operations produce association-score, and attribute-level distance is computed as a Hungarian matching over the attribute sets of each mapped vertex pair to produce attribute-score, with each component normalised by the appropriate maximum cost so that each score is in 0,1. Metrik-3 (M-3) is based on the UCG structural-similarity metric of Yuan, Yan and Ma [22]. Each input model is first transformed into a UML Class Graph (UCG): classes become vertices, attributes become attribute vertices linked by attribute edges, and relationships become typed relationship edges whose tag encodes the relationship type (association, inheritance, aggregation, composition, dependency). A backtracking algorithm then finds all maximumcardinality sets of relationship edges in one graph that can be matched injectively to relationship edges in the other with the same tag and a consistent vertex mapping; this is the UML Maximum Common Subgraph (UMCS). From the UMCS, the inter-structure similarity is the size of the common subgraph normalised by the minimum number of relationship edges in either graph, and the intra-structure similarity is computed by comparing the attribute sets of each matched class-vertex pair via edit distance. The native output of M-3 is the weighted combination of the two, with the inter-structure component at 𝜃 = 0.9 and the intra-structure component at 0.1. To extend the Metric, the combined score is projected into the required triple of class-score, attribute-score, associationscore. The projection averages three structural sub-scores with

Vasiliy Seibert

three required F1 scores. Metrik-4 (M-4) and Metrik-5 (M-5) are both based on the semanticstructural similarity metric of Triandini [18]. The metric separates a semantic component (computed from class-name and relationship similarity via WordNet-based cosine similarity) from a structural component (computed from intra-cluster and inter-cluster graph-edit-distance similarity on a UCG representation), and combines the two with equal weighting. The native output is a 7-field result comprising the overall similarity, the semantic and structural components, and four sub-scores: propSim (property similarity), relSim (relationship similarity), intraSim (intra-structure similarity), and interSim (inter-structure similarity). To extend the Metric, the 7-field result is projected into the required triple of classscore, attribute-score, association-score. The two metrics share the same source paper and the same native output but differ in the projection step, because the paper’s description of how the model-level similarity is mapped into the triple admits two plausible readings: in M-4, the semantic sub-score maps to class-score, the property similarity to attribute-score, and the relationship similarity to association-score; in M-5, the intra-structure similarity maps to class-score and attribute-score, and the inter-structure similarity maps to association-score. Although M-3 and M-5 both operate on the same UCG representation, the two projections pair classes differently. M-3 pairs classes only as a by-product of its UMCS relationship-edge alignment: two classes are paired only because they appear as endpoints of matched relationship edges. M-5, by contrast, pairs classes directly via a greedy graph-edit-distance over per-class attribute subgraphs, so two classes are paired because their attribute vertices and attribute edges are structurally similar regardless of any relationship context. The two projections therefore surface different per-pair mistakes on the same 39 inputs, which is the practical reason for keeping both M-3 and M-5 in the comparison rather than collapsing them into a single metric.

5 Quantitative Results 5.1 The Consistency Table For a fixed metric 𝑚 and a fixed element type 𝑒 ∈ {class, attr, rel}, the consistency table reports four statistics computed from the 𝑁 = 39 per-pair signed deltas 𝑑𝑖(𝑒 ) = 𝑚𝑖(𝑒 ) − ℎ𝑖(𝑒 ) , where 𝑚𝑖(𝑒 ) is the metric’s per-element score on pair 𝑖 and ℎ𝑖(𝑒 ) is the human per-element F1 on pair 𝑖. The four quantities are: 𝑁

MAD𝑒 =

1 ∑︁ (𝑒 ) 𝑑 𝑁 𝑖=1 𝑖

The MAD is the mean absolute difference between the metric’s per-element score and the human F1, so a smaller MAD indicates that the metric is closer to the human F1 on average for that element type. 𝑁

|bias|𝑒 =

1 ∑︁ (𝑒 ) 𝑑 𝑁 𝑖=1 𝑖

Comparing Domain-Model Similarity Metrics Against Human Expert Ratings

The |bias| is the absolute value of the mean signed delta and captures a constant offset between the metric’s scores and the human F1: a non-zero |bias| with a small residual std means the metric is consistently off by a roughly constant amount (calibratable by a shift), and a near-zero |bias| means the metric is centred on the human F1 on average. v u t residual_std𝑒 =

𝑁
2 1 ∑︁ (𝑒 ) 𝑑 − bias𝑒 𝑁 − 1 𝑖=1 𝑖

The residual std is the sample standard deviation of the per-pair signed deltas around their mean (with the 1/(𝑁 − 1) denominator), expressed on the same scale as the MAD; a smaller residual std indicates that the per-pair distance is tightly distributed around its mean, while a larger residual std indicates that the per-pair distance varies across pairs.

(𝑒 ) − 𝑚¯ (𝑒 ) ℎ𝑖(𝑒 ) − ℎ¯ (𝑒 ) 𝑖=1 𝑚𝑖

Í𝑁 𝑟𝑒 = √︃ Í𝑁

(𝑒 ) − 𝑚¯ (𝑒 ) 𝑖=1 𝑚𝑖

2 √︃Í𝑁

2 (𝑒 ) − ℎ¯ (𝑒 ) 𝑖=1 ℎ𝑖

The Pearson 𝑟 is the linear correlation between the metric’s perelement score and the human per-element F1, so a high 𝑟 means the metric preserves the per-pair ordering of the human F1, while a near-zero 𝑟 means the metric’s per-pair score is statistically independent of the human’s. The MAD, residual std, bias, and Pearson r for each metric, on each of the 3 element types, are shown in Table 1. The class scores express how similar the two models are at the level of their classes — the entities that populate the domain. Across the 5 metrics, the class MAD spreads from 0.07 to 0.19. M-5 is the closest to the human F1 on average, with a MAD of 0.07, a bias of 0.03, and a residual std of 0.09. M-4 is a close second on MAD at 0.09, and has the tightest spread around its mean with a residual std of 0.07, as well as the highest Pearson r in the column at 0.42, meaning that M-4 preserves the per-pair ordering of the human F1 better than any other metric on the class element. M-1 sits in the middle: a MAD of 0.15, a residual std of 0.09, a bias of 0.14, and a Pearson r of 0.37, meaning it is moderately close on average but carries a constant underestimation of 0.14. M-3 is similar to M-1 on MAD at 0.17 and residual std at 0.09, but carries a larger bias of 0.17 and a slightly lower r of 0.36. M-2 is the weakest on class: the highest MAD at 0.19, the highest residual std at 0.12, the highest bias at 0.18, and the second-lowest r at 0.30, meaning that it is both far from the human on average and a poor ranker. The class scores admit no single dominant metric: the metric with the lowest MAD, M-5 at 0.07, is not the metric with the lowest residual std, M-4 at 0.07, or the highest r, M-4 at 0.42. A practitioner who wants the metric whose class score is closest to the human on average picks M-5; a practitioner who wants the metric whose class score preserves the human’s per-pair ranking picks M-4. The attribute scores express how similar the two models are at the level of their attributes — the properties that describe each class. Across the 5 metrics, the attribute MAD spreads from 0.14 to 0.23. M-4 is the closest to the human F1 on average, with a MAD of

Conference’17, July 2017, Washington, DC, USA

0.14, a bias of 0.09, and a residual std of 0.15. M-3 has the tightest spread around its mean with a residual std of 0.12, and at 0.65 the highest Pearson r not only on the attribute element but in the entire 15-row table, meaning that M-3 preserves the per-pair ordering of the human attribute F1 better than any other metric on any element. M-2 has the smallest bias at 0.06, meaning it is centred on the human on average, but its MAD is only middling at 0.15 and its r is low at 0.18, meaning that being centred on average does not translate into per-pair ranking accuracy. M-5 is comparable to M-2 on MAD at 0.15 and residual std at 0.15, but achieves a higher r at 0.38. M-1 is the weakest on attribute: the highest MAD at 0.23, the highest residual std at 0.19, the highest bias at 0.19, and a low r at 0.23, meaning that it is both far from the human on average and a poor ranker. The attribute scores admit no single dominant metric: the metric with the lowest MAD, M-4 at 0.14, is not the metric with the lowest residual std, M-3 at 0.12, or the highest r, M-3 at 0.65. A practitioner who wants the metric whose attribute score is closest to the human on average picks M-4; a practitioner who wants the metric whose attribute score preserves the human’s per-pair ranking picks M-3. The relationship scores express how similar the two models are at the level of their relationships. Across the 5 metrics, the relationship MAD spreads from 0.13 to 0.27, the widest spread in the table. M-1 and M-3 tie for the lowest MAD at 0.13, but the two metrics differ sharply on the other three statistics. M-3 is centred on the human with a bias of 0.00 and a residual std of 0.16. M-1 carries a bias of 0.08 with a residual std of 0.15, and at -0.13 the only negative Pearson r in the entire table, meaning that M-1 actually inverts the per-pair ordering of the human relationship F1 rather than merely failing to preserve it. M-3’s r of 0.05 is near-zero, meaning that M-3 is centred on the human on average but provides no per-pair ranking information. M-4 and M-5 tie for the tightest spread with a residual std of 0.11, and M-4 has the highest r in the column at 0.42, but both metrics carry the highest MADs at 0.27 and 0.26 respectively, and the highest bias values at 0.27 and 0.26, meaning that they are consistently off by a large constant amount despite preserving the per-pair ordering better than any other metric. M-2 sits in the middle: a MAD of 0.18, a residual std of 0.13, a bias of 0.17, and a Pearson r of 0.37, making it a moderate ranker with a moderate constant offset. The relationship scores admit no single dominant metric, and the trade-off is at its starkest here: the metrics that are closest to the human on average, M-1 and M-3 at MAD 0.13, are the metrics with the worst ranking performance, M-1 at r -0.13 and M-3 at r 0.05, while the metric with the best ranking, M-4 at r 0.42, is the metric with the worst MAD at 0.27. A practitioner who wants the metric whose relationship score is closest to the human on average picks M-1 or M-3; a practitioner who wants the metric whose relationship score preserves the human’s per-pair ranking picks M-4, accepting a constant overestimation of roughly 0.27.

6 Qualitative Analysis 6.1 Metrik-1 Metrik-1 is mid-column on class (MAD 0.15, r 0.37), weakest on attribute (MAD 0.23, r 0.23, worst in the column), and joint-best

Conference’17, July 2017, Washington, DC, USA

Vasiliy Seibert

Table 1: Per-element MAD, residual std, |bias|, and Pearson r for the 5 candidate metrics on the 39 pairs. Best value per column in bold; lower is better for MAD, r.std, and |bias|; higher is better for r. Metric

Class MAD

Class r.std

Class ∥ bias ∥

Class r

Attr MAD

Attr r.std

Attr ∥ bias ∥

Attr r

Rel MAD

Rel r.std

Rel ∥ bias ∥

Rel r

0.15 0.19 0.17 0.09 0.07

0.09 0.12 0.09 0.07 0.09

0.14 0.18 0.17 0.07 0.03

0.37 0.30 0.36 0.42 0.18

0.23 0.15 0.21 0.14 0.15

0.19 0.17 0.12 0.15 0.15

0.19 0.06 0.21 0.09 0.10

0.23 0.18 0.65 0.32 0.38

0.13 0.18 0.13 0.27 0.26

0.15 0.13 0.16 0.11 0.11

0.08 0.17 0.00 0.27 0.26

-0.13 0.37 0.05 0.42 0.20

M-1 M-2 M-3 M-4 M-5

on relationship (MAD 0.13) with the only negative r in the table (-0.13). It uses exact-name matching plus Levenshtein distance for classes, the same class mapping without a Levenshtein fallback for attributes, and an endpoint-class match for relationships. It agrees with experts on direct and near-direct class names (CelO 1-shot-BTMS: "Organizer", "Attendee", "Event", "Location" by exact name, "CheckListTask"—"Task" by Levenshtein) and on direct binary associations between name-matched classes (CelO 1-shotBTMS: "Organizer-Event", "Attendee-Event"), but disagrees on every renamed, absorbed, or merged element because the rename logic does not exist. Three divergences illustrate this: on LabTracker 2shot, "Person" maps to "Patient" with attributes merged downward, "TestType" is absorbed as "Test.testGroup", and "BusinessHour" is absorbed as "Lab.openingTime" and "Lab.closingTime"—experts credit all three as semantically equivalent, the metric reports all three as missing; "Person.lastName", "firstName", "emailAddress", and "password" are distributed across "Organizer" and "Attendee" after "Person" is flattened, all reported as missing where experts credit them; and the "Requisition-SpecificTest-Test" chain collapse, the "(Requisition, Lab)-to-Appointment" association-class decomposition, and the "Test-to-TestType" absorption of an association into an attribute are all reported as missing, producing the only negative r in the table. Levenshtein has no surface to operate on once a class, attribute, or relationship is absorbed or merged, so the pipeline misses every structural transformation that the human raters recognise.

6.2

Metrik-2

Metrik-2 is weakest on class (MAD 0.19, r 0.30), mid-column on attribute (MAD 0.15, |bias| 0.06—the smallest |bias| in the column), and moderate on relationship (MAD 0.18, r 0.37). It runs the Hungarian algorithm on a name/kind/attribute cost matrix for classes, then a within-class Hungarian on attribute sets, then graph edit distance on the matched graph for relationships. It agrees with experts when name and attribute-set both align (LabTracker direct-name matches on every element), but diverges whenever a low-cost match between unrelated classes is cheaper than leaving a reference class unmatched, and the wrong class pairing then propagates downstream. Three divergences illustrate this: on CelO 2-shot, "Person" is matched to "CelO" because the empty-attribute formula yields near-zero attribute-set distance—experts credit "Person" as equivalent to "Organizer" and "Attendee", not "CelO"; on LabTracker 2-shot, "Lab.address" is matched to "Lab.changeCancellationFee" (cost 0.752) because the formula weights name distance at only 0.5, so same-typed attributes with unrelated names still yield a cost the algorithm prefers—experts judge "Lab.address" dropped and

"Lab.changeCancelFee" a rename; and on CelO 2-shot, the upstream "Person"—"CelO" wrong pairing makes the "Person-PersonRole" association a deletion ("PersonRole" unmatched), whereas experts credit the equivalent class pair "Organizer"/"Attendee". The Hungarian cost function rewards any positive cost reduction without considering domain role, so wrong class pairings propagate through the attribute and relationship stages, and the small |bias| is misleading: wrong pairings cancel across the 39 pairs but the low r reflects unreliable per-pair ordering.

6.3

Metrik-3

Metrik-3 is near-bottom on class (MAD 0.17, r 0.36) and middle on the attribute column with the highest r in the table (MAD 0.21, r 0.65), and joint-best on relationship (MAD 0.13, |bias| 0.00—the only zero in the table) with r 0.05. It applies a UML Maximum Common Subgraph (UMCS) matcher on a UML Class Graph that pairs class vertices by attribute-set edit distance and relationship-edge common subgraph, and emits no per-attribute decisions—the attribute score is a structural proxy from set overlap. It agrees with experts on structurally distinctive classes that share both a structural signature and a domain role (TSS 1-shot-H2S: "ScoutingAssignment" and "Player" matched by name), and the proxy preserves the experts’ per-pair attribute ordering better than any other metric on any element, but disagrees whenever structural similarity is not aligned with domain semantics. Three divergences illustrate this: on TSS 1-shot-H2S, "Player" is paired to "ScoutingAssignment" (intra-similarity 0.680), collapsing two exact-name matches into one wrong pair; on TSS 1-shot-H2S, "Person.firstName" and "Person.lastName" merge into "Employee.name"—experts credit both as semantically equivalent but the metric’s structural proxy reflects only set overlap (2 vs 1), not the individual merge; and on LabTracker 2-shot, the metric pairs "Test", "TestResult", "Requisition", "TestType", and "SpecificTest" all wrong (five of six semantically wrong), yet the inter-structure score is moderate because commonsubgraph overlap is non-trivial at graph level. UMCS does not prioritise name identity over structural similarity, and the structural proxies cannot credit individual semantic-equivalence judgements, so the metric is centred on the experts on average (|bias| 0.00) but provides no per-pair ranking.

6.4

Metrik-4

Metrik-4 is best ranker on class (MAD 0.09, r 0.42), strongest on central tendency on attribute (MAD 0.14), and the starkest trade-off on relationship (worst MAD 0.27 with best r 0.42). It uses greedy

Comparing Domain-Model Similarity Metrics Against Human Expert Ratings

optimal-sum matching on a cosine of tokenised class names and an attribute-set similarity blend. It agrees with experts on direct class names and renames (CelO 1-shot-BTMS: "Location", "Organizer", "Event" at high cosine), but the three divergences all share the same root—the cosine blend and the class-derived association rule treat unrelated surface tokens and absorbed associations as equivalent to direct matches. Three divergences illustrate this: on CelO 1-shot-BTMS, "Attendee" is paired to "CelO" (cSim 0.429) because attribute-set similarity is 1.0—a generated class with no attributes gets maximum aSim—whereas experts credit "Attendee" a direct match; on LabTracker 2-shot, the "DayOfWeek"—"Patient" inherited class pairing scatters "DayOfWeek"’s 7 enum literals against "Patient"’s 6 attribute names, producing non-trivial scores for pairs like "Monday" vs. "dateOfBirth" that the metric reports as matches while experts judge all 7 to have no counterpart; and on CelO 2shot, the class-derived association rule reports "Location-Event" as matched because both endpoints are matched, even though the generated model lacks the association—experts judge it equivalent to "Event.locationName" and "Event.locationAddress" typed attributes, producing the systematic 0.27 overestimation. The same pipeline property that produces wrong class pairings on CelO 1-shot-BTMS also produces wrong attribute scatter on LabTracker 2-shot and constant relationship overestimation on CelO 2-shot: every divergence traces back to treating absorbed or absent structure as if it were directly matched.

6.5

Metrik-5

Metrik-5, Triandini’s intra-structure projection, is strongest on central tendency on class (MAD 0.07, |bias| 0.03) with the weakest class r (0.18), middle on attribute (MAD 0.15, r 0.38), and second-worst on relationship (MAD 0.26, r 0.20). For each matched class-vertex pair, similarity derives from overlap of attribute and neighbour sets; the relationship score uses per-edge cost minimisation on the full UCG graph. It agrees with experts on direct-name matches and structurally similar classes (LabTracker 1-shot-H2S: "Person", "Patient", "Doctor" direct matches), but diverges on absorption across element types, wrong inherited class pairings, and topology-dominated relationship matching. Three divergences illustrate this: on CelO 1shot-BTMS, reference "EventType" (a class) collapses to a generated "EventType" enum—experts judge them semantically equivalent but intraSim treats enums and classes as separate vertex types and cannot match across them; on CelO 1-shot-BTMS, "PersonRole" is paired to "TaskStatus" because both have 1 attribute and similar subgraphs, so the structural proxy computes similarity against the wrong generated class—experts judge "TaskStatus" equivalent to "CompletionStatus", not "PersonRole"; and on LabTracker 1-shotH2S, the "Person-to-PersonRole" association is matched to generated "Patient-to-Appointment" (topology similarity 0.986) because both are 2-node edges, but experts judge it to have no counterpart since "Person" and "PersonRole" are both dropped—this last pattern repeats across most of the 11 reference relationships. intraSim cannot match across element-type boundaries and rewards structural similarity even when domain semantics are wrong; interSim’s peredge cost is dominated by topology, not domain semantics.

Conference’17, July 2017, Washington, DC, USA

7

Discussion

RQ1: how close, on average, is each metric to the human expert rating across the 39 comparisons? The spread of MADs within each column — 0.07 to 0.19 on class, 0.14 to 0.23 on attribute, 0.13 to 0.27 on relationship — shows that the choice of metric changes the average distance by up to a factor of two on a given element. The answer to RQ1 is therefore per-element: the metric with the lowest MAD on the element of interest is the closest on average, but no single metric is closest on all three elements, and even the best MAD on the relationship element (0.13) corresponds to a metric whose per-pair ordering is uninformative, so closeness on average does not by itself justify substituting the metric for the human. RQ2: how consistent is each metric’s per-comparison distance from the human expert rating? The per-comparison distance is best preserved by different metrics on different elements: Metrik-4 on class with Pearson r 0.42, Metrik-3 on attribute with r 0.65, and Metrik-4 again on relationship with r 0.42. Which metric should practitioners use? The practitioner should pick the metric with the highest per-element Pearson r: Metrik-4 on class (r 0.42), Metrik-3 on attribute (r 0.65), and Metrik-4 on relationship (r 0.42), accepting that on relationship the best r carries the worst MAD (0.27), but noting that the bias is almost entirely a constant offset (|bias|/MAD = 1.00, bias +0.274, residual std 0.113), so a linear rescaling (slope 0.507, intercept 0.012) removes the offset and drops the MAD from 0.274 to 0.113, making Metrik-4 on relationship the best on both MAD and r simultaneously once calibrated — the trade-off is not a property of the metric but of the uncalibrated output Validity of the comparison Risk 1: Were the Metrics implemented correctly? Two independent large language models ([1] and [2]) implement the same specification through the same opencode harness, and the two implementations are required to produce identical output on all 39 comparisons (test procedure tS-2), so that an ambiguity that one model resolves arbitrarily is caught by the other model resolving it differently; and the specification, not the code, is the artifact the comparison is audited against. However, tS-2 tests agreement between the two implementations, not agreement between the specification and the paper, so a misreading that is consistent across both models is invisible to the agreement test. The specification is the explicit record of the metric as understood in this work, and every assumption or deviation is written into it, so a reviewer can audit it against the source paper, but the dual-implementation agreement does not by itself certify that audit. Risk 2: Did the Metrics suffer an unfair disadvantage through the translation of the output? The per-element projection is validated by test procedure tS-3, which checks that the projection preserves the partial ordering of the metric’s native output across the 39 comparisons, so that a pair ranked more similar by the native output is also ranked more similar by the projected per-element triple. The test passes for all five metrics, which is the condition under which the projected scores are used in the comparison. The residual risk is that tS-3 checks ordering, not absolute level: a projection can preserve the partial order while shifting the absolute scale. Is it valid to treat the human expert ratings from a consensus process as the ground

Conference’17, July 2017, Washington, DC, USA

truth against which the metrics are judged? The human expert F1 used as the ground truth in this work is the output of the two-round consensus grading process reported by Chen et al. [6], and three properties of that process bound the validity of the comparison. The first is the number of raters: the consensus process involves a small number of reviewers, and the absolute F1 values reflect one group’s reading rather than a representative sample of modelers, so the central-tendency statistics (MAD, |bias|) should be treated as relative comparisons between metrics rather than as calibrated estimates of the true human distance. The second is that the consensus process, while limited in panel size, does produce an ordering of the 39 pairs that aligns with human intuition about which diagrams are more similar, and that ordering is the part of the ground truth that generalises: it is less sensitive to rater panel size than the absolute F1 level, and it is the quantity that RQ2 (Pearson r) measures, so the per-pair ranking is a more robust finding than the central-tendency comparison. The third is the scope of the reviewers’ notion of equivalence: the ground truth is scoped to a semantic-equivalence notion; a practitioner whose notion of similarity is structural-only — two models are similar only if their structure matches — would find the human F1 over-credits transformations that several metrics structurally cannot detect, and the ground truth would be misleading for that use case without regrading under the stricter notion. The comparison is therefore valid as a test of each metric’s per-pair ranking against one documented human judgement under a semantic-equivalence scope, and the absolute F1 values should not be read as a population-calibrated scale. Why aren’t LLM-as-a-Judge approaches considered in this comparison? This study compares deterministic means of measuring domain model similarity. LLM-as-a-Judge approaches are probabilistic by nature, which means they are out of scope for this comparison. Are the implementations reusable? The metric implementations are released in accordance with the FAIR4RS recommendations [7] https://doi.org/10.5281/zenodo.20942596

8

Conclusion

There is a diversity of proposed metrics for comparing domain models, but no defensible way for practitioners to choose between them, since each is typically evaluated on its own dataset. So that the field has arguments grounded in a common dataset for choosing one similarity metric over another, this paper implements five metrics from the literature [16, 18, 19, 22] and compares their outputs against a human expert F1 on a fixed set of 39 domain-model comparisons [6]. To address the risk of a wrong implementation and an unfair interpretation of the native outputs, each metric is specified in writing, implemented independently by two different large language models, and required to produce identical output on 39 domain model comparisons, so the specification — not the code — is the audited artifact; the native score is then translated into a per-element F1 of similarity on class, attribute and relationship level, and the translation is required to preserve the native ordering. The five metrics were compared quantitatively (per-element MAD, residual std, |bias|, and Pearson r against the human F1) and qualitatively. No metric dominates on all statistics, however the individual metrics each perform well on a selected subset of statistics. We therefore suggest an ensemble of the best-performing metrics, in which each metric

Vasiliy Seibert

covers the statistics where it has the highest agreement with the human F1. We release all implementations in accordance with the FAIR4RS principles [7] (https://doi.org/10.5281/zenodo.20942596).

References [1] Moonshot AI. 2025. Kimi K2.7-Code (via Ollama). https://ollama.com/library/ kimi-k2.7-code [2] Zhipu AI. 2025. GLM-5.2 (via Ollama). https://ollama.com/library/glm-5.2 [3] Nacir Bouali, Marcus Gerhold, Tosif Ul Rehman, and Faizan Ahmed. 2025. Toward Automated UML Diagram Assessment: Comparing LLM-Generated Scores with Teaching Assistants. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: CSEDU. SciTePress - Science and Technology Publications, Porto, Portugal, 158–169. doi:10.5220/0013481900003932 [4] Younes Boubekeur, Gunter Mussbacher, and Shane McIntosh. 2020. Automatic assessment of students’ software models using a simple heuristic and machine learning. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (Virtual Event, Canada) (MODELS ’20). Association for Computing Machinery, New York, NY, USA, Article 20, 10 pages. doi:10.1145/3417990.3418741 [5] Boqi Chen, Kua Chen, Sami Hassani, Yilong Yang, Daniel Amyot, Lamine Lessard, Gunter Mussbacher, Mohammad Sabetzadeh, and Dániel Varró. 2023. On the use of GPT-4 for creating goal models: an exploratory study. In 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW). IEEE, Hannover, Germany, 262–271. doi:10.1109/REW57809.2023.00052 [6] Kua Chen, Yujing Yang, Boqi Chen, J. A. Hernández López, Gunter Mussbacher, and Dániel Varró. 2023. Automated Domain Modeling with Large Language Models: A Comparative Study. In 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS). IEEE, Västerås, Sweden, 162–172. doi:10.1109/MODELS58315.2023.00037 [7] Neil P. Chue Hong, Daniel S. Katz, and Michelle Barker. 2022. FAIR Principles for Research Software (FAIR4RS Principles). Zenodo. doi:10.15497/RDA00068 Version 1.0. [8] Reza Fauzan, Daniel Siahaan, Siti Rochimah, and Evi Triandini. 2021. Automated Class Diagram Assessment using Semantic and Structural Similarities. International Journal of Intelligent Engineering and Systems 14, 2 (2021), 52–66. doi:10.22266/IJIES2021.0430.06 [9] Bertrand Meyer. 1992. Applying “Design by Contract”. Computer 25, 10 (1992), 40–51. doi:10.1109/2.161279 [10] Salisu Modi, Hanan Abdulrahman Taher, and Hoger Mahmud. 2021. A Tool to Automate Student UML diagram Evaluation. Academic Journal of Nawroz University 10, 2 (2021), 189–198. doi:10.25007/ajnu.v10n2a1035 [11] OpenAI. 2023. GPT-3.5 models. https://web.archive.org/web/20240319094420/ https://platform.openai.com/docs/models/gpt-3-5. Original page GPT-3.5 models (online documentation); page since retired by OpenAI. Snapshot via the Wayback Machine captured 2024-03-19, accessed 2024-03-26. GPT-3.5 Turbo subsequently deprecated with shutdown scheduled for 2026-10-23 per https://platform.openai. com/docs/deprecations. [12] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] doi:10.48550/ arXiv.2303.08774 arXiv:2303.08774v6. [13] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). Curran Associates, Inc. doi:10.5555/3600270.3602281 [14] Marko Ristin. 2026. icontract: Design-by-contract for Python3 with informative violation messages and inheritance. https://github.com/Parquery/icontract. Also available on PyPI: https://pypi.org/project/icontract/. Original release July 31, 2018.. [15] Arnaud Roques and PlantUML Contributors. [n. d.]. PlantUML: Generate diagrams from textual description. GitHub repository. https://github.com/plantuml/ plantuml Accessed: 2026-06-15. [16] Prabhsimran Singh, Younes Boubekeur, and Gunter Mussbacher. 2022. Detecting Mistakes in a Domain Model. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (Montreal, Quebec, Canada) (MODELS ’22). Association for Computing Machinery, New York, NY, USA, 257–266. doi:10.1145/3550356.3561583 [17] Shizhe Song, Ye Wang, Xiaoyang Wang, Chengyi Lin, and Kun Hu. 2024. A Deep Learning-Based Approach to Similarity Calculation for UML Use Case Models. Expert Systems with Applications 251 (2024), 123927. doi:10.1016/j.eswa.2024. 123927 [18] Evi Triandini. 2021. Automated Class Diagram Assessment Using Semantic and Structural Similarities. International Journal of Intelligent Engineering and Systems

Comparing Domain-Model Similarity Metrics Against Human Expert Ratings

(2021). doi:10.22266/IJIES2021.0430.06 [19] Pavel Čech. 2019. Matching UML Class Models Using Graph Edit Distance. Expert Systems with Applications 130 (2019), 206–224. doi:10.1016/j.eswa.2019.04.008 [20] Song Yang and Houari Sahraoui. 2022. Towards Automatically Extracting UML Class Diagrams from Natural Language Specifications. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (Montreal, Quebec, Canada) (MODELS ’22). Association for Computing Machinery, New York, NY, USA, 396–403. doi:10.1145/3550356. 3561592

Conference’17, July 2017, Washington, DC, USA

[21] Yujing Yang, Boqi Chen, Kua Chen, Gunter Mussbacher, and Dániel Varró. 2024. Multi-step Iterative Automated Domain Modeling with Large Language Models. In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages Systems (Linz, Austria) (MODELS Companion ’24). Association for Computing Machinery, New York, NY, USA, 587–595. doi:10.1145/3652620.3687807 [22] Zhongchen Yuan, Li Yan, and Zongmin Ma. 2020. Structural Similarity Measure Between UML Class Diagrams Based on UCG. Requirements Engineering 25, 2 (2020), 213–229. doi:10.1007/s00766-019-00317-w

Received 2026-06-13

Related documents

• Comparing Domain-Model Similarity Metrics Against Human Expert Ratings#615228
• Comparing Domain-Model Similarity Metrics Against Human Expert Ratings#614261
• REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations#798854
• REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations#797440
• Guided Uncertainty-Aware Robust Domain Transfer#1130308