arXiv:2606.24721v1 [cs.SE] 23 Jun 2026
Two-Level vs. Multi-Level Modelling: An Empirical Study of Cascading Maintenance Burden 3rd Bowen Jiang
1st Yuhong Fu
2nd Weixing Zhang
Adelaide University Mawson Lakes, Australia 0000-0003-2093-2326
Karlsruhe Institute of Technology Karlsruhe, Germany 0000-0003-2890-6034
4th Haowei Cheng
Waseda University Karlsruhe Institute of Technology Tokyo, Japan Karlsruhe, Germany 0009-0008-2265-6437 0009-0001-8168-0866
5th Georg Grossmann
6th Karamjit Kaur
7th Matt Selway
8th Markus Stumptner
Adelaide University Mawson Lakes, Australia 0000-0003-4415-2228
Adelaide University Mawson Lakes, Australia 0000-0003-0255-1060
Adelaide University Mawson Lakes, Australia 0000-0001-6220-6352
Adelaide University Mawson Lakes, Australia 0000-0002-7125-3289
Abstract—When a core definition in a software system changes, every dependent artefact must be updated to remain consistent. This cascading update problem pervades software maintenance and recurs in Model-Driven Engineering (MDE), where the evolution of a metamodel forces every conforming model to be migrated in lockstep. The dominant MDE paradigm, twolevel modelling (2LM), fragments domain knowledge across two artefacts that must be kept mutually consistent, making this coevolution a source of inconsistencies and maintenance effort. An alternative paradigm, multi-level modelling (MLM), unifies these artefacts into one and is claimed in the literature to reduce coevolution burden. Prior empirical work has examined MLM’s applicability and modelling effort, but its effect on co-evolution and maintenance burden has not yet been tested in a controlled, paired comparison against 2LM. We hypothesise that, for semantically equivalent evolution scenarios, MLM’s structural unification yields fewer post-change inconsistencies and a smaller modification footprint than the 2LM split-artefact arrangement. To test this hypothesis, we present a pre-registered, mutation-based empirical comparison of coevolution behaviour in 2LM and MLM; empirical results will follow in Stage 2. From a curated corpus of published 2LM coevolution scenarios, we construct semantically equivalent MLM counterparts, apply identical evolution mutations to both, and measure outcomes through automated consistency checking and pre-registered hypothesis tests. Positive controls and a blinded mapping protocol guard against bias favouring MLM. This design provides the first empirical framework for assessing whether paradigm-level structural choices affect the cascading maintenance burden in model-based systems, operationalising co-evolution burden as two automatically measurable outcome variables, and delivering a reusable benchmarking protocol to support replication and extension. Index Terms—Software Maintenance, Cascading Update, Model Co-Evolution, Model-Driven Engineering, Controlled Experiment
I. Introduction When a core definition in a software system changes, every artefact that depends on it must be updated to remain consistent. This cascading update problem is a defining concern of software maintenance, recognised since Lehman’s laws of continuing change [1]: database schema evolution forces
application-layer migration [2], and API breaking changes propagate to client code, tests, and downstream libraries [3], [4]. In each case, much of the cost lies not in the original change but in the cascade of dependent updates it triggers. The same cascading update recurs in Model-Driven Engineering (MDE) [5], [6], a software engineering subfield in which models serve as primary development artefacts from which code, configurations, and analyses are derived. In MDE, the core definition of a system resides in its metamodel, while one or more conforming models populate that vocabulary with concrete instances. When the metamodel evolves, every conforming model must be migrated in lockstep. This metamodel– model co-evolution is the MDE-specific instance of the broader cascading update problem, and it is a source of inconsistencies and maintenance effort [7]–[9]. MDE underpins long-lived industrial systems in domains such as automotive, avionics, and energy infrastructure, where models drive code generation, simulation, and certification, and co-evolution costs accumulate over decade-long maintenance cycles. The dominant MDE paradigm, two-level modelling (2LM), embodies this metamodel-and-model split precisely. An alternative paradigm, multi-level modelling (MLM), unifies the metamodel and the model into a single artefact that internally spans multiple ontological levels, and is claimed in the literature to thereby reduce the burden of co-evolution [10]–[14]. The technical machinery of both paradigms, including the SLICER realisation of MLM used in this study, is detailed in Section II. Prior work has begun to evaluate MLM empirically: its applicability and modelling complexity benefits have been assessed against 2LM in specific settings [10]. However, to our knowledge no controlled study has isolated the coevolution/maintenance dimension by applying identical evolution changes to paired 2LM and MLM artefacts and measuring the resulting inconsistency and modification burden [15]. Existing maintenance-oriented evidence remains largely conceptual or based on illustrative examples, leaving practitioners without guidance on whether MLM’s purported maintenance benefits materialise in practice.
The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports
We address this gap with a pre-registered, mutation-based empirical comparison of co-evolution behaviour in 2LM and MLM. Drawing on a curated corpus of published 2LM co-evolution scenarios, we construct semantically equivalent MLM counterparts under a pre-registered mapping protocol, subject both versions to identical evolution mutations, and measure post-change inconsistencies and modification effort using automated consistency checking. The use of preregistration, paired statistical analysis, positive controls, and a blinded mapping protocol jointly safeguards the comparison against bias favouring MLM. This Stage 1 report contributes: (i) a pre-registered design for comparing the maintenance burden of two structurally distinct ways of organising an evolving system’s artefacts; (ii) an operationalisation of coevolution burden as two automatically measurable outcomes, post-change inconsistency count and restoration modification count, each at its paradigm’s native edit granularity; and (iii) a reusable benchmarking protocol (corpus, mappings, mutation taxonomy, analysis pipeline) supporting replication and extension. II. Background This section provides the technical background needed to understand the design choices in the rest of the paper. a) Two-Level Modelling: The dominant MDE paradigm is two-level modelling (2LM), embodied by frameworks such as the Eclipse Modelling Framework (EMF) and the OMG’s Meta-Object Facility (MOF). In 2LM, a metamodel (e.g., an Ecore file) defines the domain vocabulary, while a conforming model populates this vocabulary with concrete instances. This two-level separation fragments domain knowledge across two artefacts that must remain mutually consistent: any change to the metamodel (e.g., renaming a class, adding a mandatory attribute) typically obliges a corresponding migration of every conforming model [8], [9]. b) Multi-Level Modelling and SLICER: Multi-level modelling (MLM) is an alternative MDE paradigm. Rather than splitting domain knowledge across a metamodel and a model, MLM fixes a single generic linguistic metamodel (a small, stable kernel defining object, attribute, and relationship) and represents all domain knowledge in one artefact spanning as many ontological levels as the domain requires. Within it, an intermediate-level object can simultaneously instantiate the level above and type the level below. For example, PumpModel, CentrifugalPump-X200 (which instantiates PumpModel yet itself acts as a type), and the physical PumpSerial-7341 can coexist in one artefact under one linguistic metamodel. In SLICER [12], a relationship-based MLM framework, this dual role is realised through five explicit inter-object relations: standard instantiation (InstN), instantiation with extension (InstX), and specialisation by extension (SpecX) introduce new ontological levels, while specialisation by refinement (SpecR) and subset-by-specification (SbS) operate within a level. The level structure of a model thereby emerges dynamically from the combined network of these relations.
The reference implementation is in DoME, a Smalltalk-based environment providing built-in validation and structural diff over SLICER artefacts [16]. We select SLICER on three methodological grounds: (i) it provides an explicit, published mapping procedure from 2LM artefacts [12, Section 6], a hard prerequisite for the blinded mapping protocol and round-trip check (C3, Section III); (ii) DoME offers a scriptable interface to both validation and structural diff, enabling automated measurement of InconsCount and ModCount; (iii) its relation-based level mechanism minimises the gap between mapping protocol primitives and measured artefact primitives. We treat SLICER as a representative MLM realisation because it instantiates the three structural properties that define the paradigm: arbitrary ontological levels emerging from the level-introducing relations, crosslevel propagation of intensional content along instantiation and specialisation chains, and the dual instantiation/classification role of intermediate-level objects. The confirmatory claim is accordingly scoped to SLICER-realised MLM. Whether the measured effects transfer to potency-based or formal-logic realisations such as LML [17], FMMLx [11], MultEcore [18], or DeepTelos [19] is an external-validity question we defer to the multi-framework replication outlined in Section VII. Bias risks introduced by this selection are addressed by the Mapping Protocol Bias and Tool Maturity controls detailed in Section VI. c) Co-Evolution as Intra-Model Maintenance: MLM reframes co-evolution without eliminating it: an upper-level edit still leaves dependent lower-level objects inconsistent, but the inconsistency is now governed by the fixed linguistic metamodel and checkable within one tool rather than across two separately maintained files. Whether this reframing yields measurable reductions in inconsistency risk and modification effort is the question this study tests. III. Hypotheses We operationalise the conjectured advantage of MLM along two dimensions of co-evolution burden: the risk of inconsistency an evolution step introduces, and the effort needed to absorb it. We state one primary hypothesis per dimension. Both are one-sided, predicting that MLM’s artefact unification reduces the measured quantity. The null hypotheses are tested at 𝛼 = 0.05 with multiple-comparison correction across the two tests (Section VI). Following the RR Author’s Guide, each hypothesis carries a pre-specified positive control (C1, C2), and a shared mapping control (C3) covers both. a) H1 — Inconsistency risk: Hypothesis. For semantically equivalent scenarios, identical evolution mutations yield, on average, fewer post-change inconsistencies under MLM than under 2LM, each count measured under its paradigm’s native consistency checking. Rationale. In 2LM, inter-artefact synchronisation can lag or remain incomplete, generating inconsistencies after an evolution step (Section II). In MLM the same step occurs within one artefact under a fixed linguistic metamodel, so no separate artefact can fall out of sync and the conditions for inconsistency are
The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports
narrower. We therefore expect fewer post-change inconsistencies under MLM. Control (C1). Applying trivial leaf-level mutations (operations that touch only leaf elements, modifying neither the metamodel structure in 2LM nor any MLM object that types an instantiation relation or parents a level-introducing specialisation) must leave the paired inconsistency-count difference statistically indistinguishable from zero, verified by a two-sided Wilcoxon signed-rank test [20] that fails to reject at 𝛼 = 0.05. b) H2 — Modification effort: Hypothesis. For semantically equivalent scenarios, restoring consistency after identical evolution mutations incurs, on average, a smaller paradigmnative modification footprint under MLM than under 2LM. We attribute any such reduction to structural unification, not to differences in per-edit cognitive or operational cost, which lie outside our scope. Rationale. In 2LM, modification effort is dominated by the cascading migration of conforming models across two artefacts [8], [9] (Section II); in MLM as realised by SLICER the change is expressed once on the defining object, its effect mediated by the recorded instantiation and specialisation relations rather than by manual migration. We therefore expect fewer paradigm-native modifications under MLM. Control (C2). The same trivial leaf-level mutations (C1) must leave the paired element-modification-count difference statistically indistinguishable from zero, verified by an automated diff over the canonical serialisation of each artefact pair before and after restoration. c) Shared control (C3): The round-trip mapping check translates a 2LM scenario into its MLM counterpart under the pre-registered mapping protocol (Section VI) and back, and must recover the original 2LM artefact up to a pre-registered notion of structural equivalence. The round-trip establishes that the mapping is information-preserving and invertible, but it does not by itself establish semantic equivalence between the two representations. We therefore pair it with a pre-registered conformance-preservation check: a fixed set of instance-level conformance tests must return identical accept/reject verdicts on the 2LM model and on its MLM counterpart, so that an instance valid under one representation is valid under the other. On a stratified random sample, semantic faithfulness is additionally audited by an independent reviewer blinded to the hypotheses, and any scenario failing either check is excluded. Together these ensure the MLM artefact is a faithful counterpart of the original rather than a paradigm-favouring re-encoding. Failure on any control invalidates the corresponding test for the affected condition and is reported in the Stage 2 paper as required by the RR protocol. IV. Variables This section operationalises every variable underlying H1, H2, and the controls. Independent and dependent variables are deterministically computable from the artefacts, so that two researchers applying the protocol to the same scenario obtain identical measurements; confounders are listed with the
TABLE I Notation used throughout the paper. Variables Paradigm InconsCount ModCount Complexity
Modelling paradigm, ∈ {2LM, MLM} Post-change inconsistency count (primary outcome, H1) Element modification count (primary outcome, H2) Scenario complexity stratum (small/medium/large)
SLICER relation kinds InstN Standard instantiation InstX Instantiation with extension SpecX Specialisation by extension SpecR Specialisation by refinement SbS Subset-by-specification
strategy that neutralises or absorbs them. Table I summarises the notation. A. Independent Variable 1) Modelling Paradigm (Paradigm): Description. The MDE paradigm under which a co-evolution scenario is represented and evolved. Scale type. Nominal, two levels. Operationalisation. 2LM and MLM scenarios are represented and evolved under EMF and SLICER [12] respectively, with DoME as the MLM reference implementation (see Section II for details). B. Dependent Variables 1) Post-Change Inconsistency Count (InconsCount; primary outcome for H1): Description. The number of distinct consistency violations reported by automated checking on the artefact(s) after applying an evolution mutation, before any restoration is attempted. Scale type. Ratio (non-negative integer count). Operationalisation. On each side, InconsCount is the count of violations reported by the paradigm’s native validator under a pre-registered, fixed configuration: for 2LM, Ecore well-formedness, model-to-metamodel conformance, and OCL invariant violations via the EMF Validation Framework; for MLM, linguistic-metamodel and object-constraint violations via DoME’s validator. The exact configurations are pre-registered in the replication package. 2) Element Modification Count (ModCount; primary outcome for H2): Description. The number of paradigm-native edits needed to return the artefact(s) from the post-mutation state to a consistent one, judged by the same checker configuration used for InconsCount. “Paradigm-native” means edits counted at the native granularity of each paradigm’s reference tooling (EMF for 2LM, DoME for MLM). Cross-paradigm comparisons of ModCount therefore compare structural edit footprint, not per-edit human labour. Scale type. Ratio (non-negative integer count). Operationalisation. ModCount is computed by EMF Compare (2LM) and DoME’s structural diff (MLM) over canonical serialisations before and after restoration, counting four atomic operations (add, delete, update, move) with equal weight over
The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports
each paradigm’s native elements (classes, attributes, references, instances, slots in 2LM; objects, attributes, slots, and the five SLICER relation kinds in MLM). Restoration is deterministic and pre-registered (Section VI), with no human-in-the-loop migration. C. Confounding Variables 1) Scenario Complexity (Complexity): Description. The intrinsic size and structural richness of a scenario, which may raise both InconsCount and ModCount independently of Paradigm. Scale type: ordinal, three strata (small, medium, large). Control strategy: stratified sampling. Using preregistered thresholds on element count and on the depth of the longest level-introducing relation chain (InstN/InstX/SpecX chains in MLM; metamodel-to-model classification depth in 2LM), the corpus (Section V) is partitioned into strata, each contributing a balanced number of pairs; the paired Wilcoxon test (Section VI) further absorbs scenario-level variation, since each scenario contributes both a 2LM and an MLM observation. 2) Mutation Type: Description. The category of evolution operation applied to a scenario; because different operations exert different synchronisation pressure in 2LM, it could drive the H1/H2 effects asymmetrically if left uncontrolled. Scale type: nominal, enumerated by the pre-registered mutation taxonomy (Section VI), derived from Herrmannsdoerfer et al.’s catalogue [21] and adapted to SLICER objects and the five relation kinds (InstN, InstX, SpecX, SpecR, SbS). Control strategy: each scenario receives the full set of preregistered mutations (full-factorial within scenario) wherever mechanically applicable, every paired difference formed within a matched (scenario, mutation) cell so that mutation type is identical across paradigms within each pair. We do not balance across mutation-type categories, and balancing is not required: the primary tests operate on within-pair differences in which mutation type is constant by construction, so any main effect cancels and cannot confound the paradigm comparison. To keep the residual frequency imbalance transparent, we pre-register reporting of the realised count of applicable mutations per category and per Complexity stratum, and report per-mutation effects as a planned secondary breakdown. The primary tests aggregate across mutation types within scenario pairs. 3) Mapping Protocol Bias: Description. Asymmetries inadvertently introduced when translating a 2LM scenario into its SLICER counterpart under the meta-property decomposition of [12, Section 6], which could artificially favour MLM. Control strategy: the mapping protocol is fully pre-registered before any scenario is mapped and is applied by an operator blinded to the mutation set. Although the operator is aware of the study’s directional hypotheses, as is unavoidable given their role in the research team, operator discretion is structurally bounded by the deterministic meta-property decomposition (Section VI), in which the annotation pattern uniquely determines the SLICER relation kind. To bound the residual latitude in the annotation step itself, a second operator, likewise blinded to the hypotheses, independently annotates a pre-registered random sample of scenarios, and inter-annotator agreement (Cohen’s 𝜅) is
reported, with any disagreement resolved by the pre-registered tie-breaking rules. The mapping is additionally validated by the round-trip check (Section III). 4) Tool Maturity: Description. The greater maturity and wider adoption of EMF (the 2LM reference tooling) over DoME (the SLICER reference tooling) may affect the accuracy of consistency-violation and structural-diff detection. Control strategy: validators and diff tools on both sides are configured to a consistent, pre-registered semantic scope (well-formedness, conformance, and invariants; element-level structural diff); to compensate for DoME’s relative immaturity, every MLMside InconsCount and ModCount measurement is subject to a pre-registered manual audit by an independent reviewer blinded to the corresponding 2LM measurement, which may flag a measurement as instrument failure (excluding it from the test) but cannot revise its value, with all decisions logged in the replication package. Residual threat and bias direction: the audit cannot rule out that DoME under-reports violations relative to EMF, so we bound this by direction: under-reporting biases InconsCount downward for MLM, the direction H1 predicts, so a null or 2LM-favouring result on H1 is robust, and a significant MLM-favouring result is reported in Stage 2 with the caveat that part of the reduction may reflect validator coverage; the replication package publishes every violation flagged by either validator for independent scrutiny. 5) Cross-Paradigm Edit Granularity: Description. InconsCount and ModCount are counted in each paradigm’s native units, and since 2LM and MLM expose different element kinds (coupled metamodel-and-model edits versus single-artefact edits over SLICER’s relations), one conceptual change may decompose into a different number of native units in each. Control strategy: ModCount captures structural edit footprint at each paradigm’s native granularity, not human labour (Section IV-B), so any paradigm-level asymmetry is read as evidence about structural cost, not person-time; to keep the mapping from amplifying this asymmetry, the mutation taxonomy is anchored at the conceptual-change level (e.g., “introduce a mandatory attribute on concept 𝐶”) and instantiated independently against each paradigm’s native operations, and the round-trip check (C3) confirms this anchoring introduces no MLM-favouring decomposition. The atomic operations counted by DoME’s diff are of comparable fineness to EMF Compare’s, so ModCount differences are not expected to be dominated by tool granularity. Residual threat: as with Tool Maturity, coarser MLM operations would bias ModCount downward for MLM, the direction H2 predicts, so a null or 2LM-favouring result is robust and a significant MLM-favouring result is reported in Stage 2 with the corresponding caveat. V. Datasets This study involves no human participants. Its empirical units are co-evolution scenarios drawn from publicly available, peerreviewed sources in the MDE literature. We deliberately rely on third-party scenario sources rather than constructing scenarios ourselves, so that selection is not influenced by knowledge of subsequent mutations or anticipated paradigm-level outcomes.
The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports
a) Source Corpora: The corpus is assembled from four public sources of complementary character: (i) large-scale, industrial-character scenarios from the long-running GMF evolution history [22]; (ii) canonical small-scale scenarios from Herrmannsdoerfer et al.’s coupled-operator catalogue [21], which also defines our mutation taxonomy (Mutation Type, Section IV); (iii) medium-complexity scenarios with known migration solutions from Hebig et al.’s co-evolution survey [23]; (iv) small- and medium-scale scenarios from Iovino et al.’s multi-language metamodel-evolution impact study [24]. b) Selection Criteria: Evaluated by the corpus operator before any mutation, a scenario is eligible if: (i) its original 2LM artefacts (a metamodel and at least one conforming model) are publicly available or reconstructible; (ii) it specifies an unambiguous before-state; (iii) it is expressible under the SLICER mapping protocol (Section VI) without out-of-scope constructs; and (iv) at least one element can receive at least one mutation from the pre-registered taxonomy. A scenario is excluded if its sources are incomplete or inconsistent, if it relies on EMF tooling extensions with no documented semantics, or if it is a syntactic variant of an accepted scenario (only the more general variant is kept). c) Target Sample Size and Stratification: The target number of paired scenarios is determined by the pre-registered statistical power analysis described in Section VI. Based on the published volumes of the four sources, we expect the number of scenarios meeting our eligibility criteria to exceed the sample size of 𝑁 = 55 required for adequate power. As an early feasibility gate, we screen the corpus against the eligibility criteria before any mutation is applied and record the realised eligible count per source and per Complexity stratum. If this screen projects fewer than 𝑁 = 47 eligible scenarios, we enact the pre-registered source-expansion contingency (secondary sources, in the priority order fixed in the replication package) before proceeding; if expansion still falls short, we report the shortfall and restrict primary inference to the strata that remain adequately powered, documenting the deviation in the Stage 2 paper. The final eligible count is reported in the Stage 2 paper. Each eligible scenario is assigned to a Complexity stratum using the pre-registered thresholds of Section IV, and strata are populated to the per-stratum minimum implied by the power analysis, drawing on the secondary sources where a primarysource stratum falls short. d) Data Availability: All artefacts and analysis scripts will be released in a public replication package upon Stage 1 acceptance, per the ICSME open science policy. e) Threats to Corpus Validity: The four primary sources are concentrated in the Eclipse/EMF ecosystem, which may bias the corpus toward 2LM idioms favoured by that toolchain and away from corpora developed under GME, MetaEdit+, or other 2LM frameworks [25]. To mitigate this, the Hebig et al. survey corpus [23] contributes scenarios drawn from heterogeneous co-evolution approaches across multiple research groups, and the Iovino et al. study [24] spans multiple modelling languages beyond GMF. A residual threat is that scenarios published in the literature may over-represent “hard” co-evolution cases that
motivated the original publications, inflating absolute InconsCount and ModCount values. Because the primary tests are paired within scenarios, this absolute inflation does not bias the paradigm-level comparison; we report descriptive statistics on absolute magnitudes alongside the paired tests so that readers can assess external validity directly. A distinct concern is that scenarios sourced from EMF-oriented communities may embody 2LM design conventions and thus sample the region of the design space where MLM’s advantages are least pronounced. This biases the comparison against MLM: a demonstrated MLM advantage on such a corpus is a conservative lower bound, while the corpus cannot manufacture an advantage that does not exist. The complementary question, whether MLM’s benefits are larger on models designed natively for multi-level representation, falls outside the scope of this confirmatory test and is left to the future work of Section VII. VI. Execution Plan Pipeline. The execution proceeds in five pre-registered stages, applied uniformly to every eligible scenario: (1) Mapping: an operator blinded to the mutation set and to the directional hypotheses translates the 2LM scenario into its SLICER counterpart under the pre-registered mapping protocol, which applies the meta-property decomposition of Selway et al. [12, Section 6] so that each construct’s annotation pattern determines its SLICER relation kind. The round-trip check (C3) is run on the result, and any failure terminates inclusion. (2) Mutation application: each mechanically applicable mutation from the pre-registered taxonomy is applied independently to the 2LM and SLICER artefacts, producing one mutated artefact pair per (scenario, mutation). (3) Inconsistency measurement: InconsCount is recorded immediately after mutation, before any restoration, using the pre-registered EMF Validation Framework configuration (2LM) and DoME validator configuration (MLM). (4) Restoration: a deterministic, pre-registered restoration procedure resolves violations by applying the minimum-edit closure under a fixed priority of repair operators; restoration is fully scripted and admits no human-in-the-loop decisions. (5) Modification measurement: ModCount is computed by EMF Compare (2LM) and DoME’s structural diff (MLM) over canonical serialisations of the artefacts before and after restoration. Restoration procedure. The restoration script systematically resolves violations in the post-mutation artefact to produce a fully consistent state; the modification distance is reported as ModCount (Section IV-B). The script proceeds violation-byviolation in a pre-registered traversal order. For each violation, it selects a repair from a fixed catalogue ranked by a pre-registered priority list. The priority ordering encodes two standard maintenance heuristics: prefer non-destructive over destructive repairs, and local over widely propagating ones. The priority list is structurally identical across paradigms but instantiated against the artefact kinds available on each side. The SLICER-side instantiation reuses the semantic conflict resolution catalogue developed in prior work [26]. The 2LM-side instantiation draws its repairs from the established EMF coupled-evolution opera-
The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports
tors [21], the community-standard migration catalogue for that paradigm, so that neither side is restored with an ad hoc operator set. Both catalogues are constrained to the same minimum-edit objective, so ModCount reflects the edit footprint required to regain consistency rather than the relative sophistication of either catalogue. We nonetheless acknowledge a residual asymmetry, that the SLICER catalogue [26] was developed by some of the present authors whereas the 2LM operators are third-party, and we bound it by direction: a more effective MLM catalogue can only lower ModCount for MLM, the direction H2 already predicts, so it cannot manufacture a spurious 2LM result nor mask a real one. Should H2 favour MLM, robustness check (iii) below re-tests it with the authored catalogue removed. The highest-priority repair that yields a syntactically valid result is applied. Ties are broken lexicographically over repair names. Restoration terminates when no violations remain, or when a pre-registered iteration cap is reached, in which case the scenario is flagged as unrestorable and excluded from H2 (but retained for H1, since InconsCount is measured before restoration). The unrestorable rate is reported per paradigm and per stratum as a quality indicator. Statistical analysis. H1 and H2 are tested by paired one-sided Wilcoxon signed-rank tests with Holm correction [27] across the two primary tests at 𝛼 = 0.05. Effect sizes are reported as the matched-pairs rank-biserial correlation 𝑟 𝑟 𝑏 with 95% bootstrap confidence intervals. We use 𝑟 𝑟 𝑏 because it is native to the Wilcoxon signed-rank test: it is computed from the test’s signed-rank statistic and therefore quantifies exactly the paired comparison the significance test evaluates, it inherits the test’s distribution-free assumptions (unlike a parametric 𝑑 𝑧 , which our skewed, non-negative counts would violate), and it is bounded in [−1, +1] on Cohen’s small/medium/large scale, commensurate with the smallest effect of interest 𝑟 𝑟 𝑏 = 0.40 used below. Its confidence interval is obtained by nonparametric bootstrap over scenario pairs. Outcome-neutral controls C1 and C2 (Section III) are evaluated as two-sided Wilcoxon tests under the same 𝛼, and primary inference is conditional on failure to reject in C1, C2, and the structural pass of C3. Scenario–mutation pairs flagged as instrument failure by the manual audit are excluded from the corresponding hypothesis test; the instrument-failure rate is reported per paradigm and per stratum. Power analysis. Sample size is determined a priori for the more conservative of the two primary tests. Targeting power 1 − 𝛽 = 0.80 at the Holm-corrected level 𝛼′ = 0.025 with a one-sided paired Wilcoxon test, and a pre-registered smallest effect of interest of 𝑟 𝑟 𝑏 = 0.40 (a medium effect by Cohen’s conventions [28], in line with SE experiment benchmarks [29]), the required minimum sample is 𝑁 = 47 paired scenarios. To absorb potential attrition from manual-audit instrumentfailure flags, we pre-register a target sample size of 𝑁 = 55 (≈ 17% buffer); should the post-audit effective sample fall below 𝑁 = 47, the deviation will be reported in the Stage 2 paper. Robustness checks. Three pre-registered sensitivity analyses accompany the primary inference. (i) An alternative restoration priority list, swapping the top two repair operators, is run on the full corpus to verify that H2 conclusions are not driven by
the specific priority pre-registration. (ii) A leave-one-sourceout re-analysis re-computes the primary tests with each of the four corpus sources omitted in turn, exposing any singlesource dominance. (iii) H2 is re-computed with the SLICERside restoration restricted to a paradigm-generic, minimumedit repair set that excludes the semantic conflict-resolution catalogue of [26] and mirrors the generic operators used on the 2LM side, confirming that any MLM advantage does not depend on the authored catalogue. Disagreement between the primary analysis and any robustness check is reported in the Stage 2 paper without retro-fitting hypotheses. VII. Work in Progress and Future Work This manuscript is a Stage 1 Registered Report: it fixes the design, hypotheses, and analysis plan, while the confirmatory results are reported in the Stage 2 paper. The protocol is therefore presented without outcome data by design. Work to date has fixed the variable operationalisation (Section IV) and the corpus eligibility criteria (Section V); the mapping protocol, mutation taxonomy, restoration procedure, and validator configurations are being finalised and frozen. Following Stage 1 In-Principle Acceptance, Stage 2 execution is scheduled over approximately twelve months, leaving a buffer before the EMSE submission deadline of September 11, 2027: • Months 1–2. Freeze the replication package: mapping protocol, mutation taxonomy over the SLICER relation kinds, restoration procedure, and validator configurations. • Months 3–5. Screen the corpus against the eligibility criteria and run an early feasibility gate: if the projected eligible count falls below 𝑁 = 47, enact the pre-registered source-expansion contingency before proceeding. Build SLICER counterparts under the blinded mapping protocol, including round-trip verification (C3). • Months 6–8. Apply the pre-registered mutations and measure InconsCount and ModCount automatically. • Months 9–10. Evaluate the outcome-neutral controls (C1, C2), then run the Holm-corrected tests of H1 and H2 and the planned per-stratum breakdown. • Months 11–12. Manuscript preparation, internal review, and submission. Any deviation from this plan that materially affects the protocol will be documented in the Stage 2 paper. Beyond Stage 2, the design generalises in two directions. First, the mapping protocol can be re-targeted to other multilevel frameworks, testing whether the measured maintenance effects are specific to SLICER or hold across MLM realisations. Second, because ModCount measures structural edit footprint rather than engineering effort, a reduction in edits is a necessary but not sufficient indicator of reduced maintenance cost. Establishing how structural footprint maps to developer time requires controlled human-subject studies, which, together with richer edit-cost models and a broader set of evolution operators, we leave to future work. Complementary approaches that leverage LLMs to support co-evolution [30] suggest further directions for automating the restoration step.
The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports
References [1] M. Lehman, “Programs, life cycles, and laws of software evolution,” Proceedings of the IEEE, vol. 68, no. 9, pp. 1060–1076, 1980. [2] D. I. K. Sjøberg, “Quantifying schema evolution,” Inf. Softw. Technol., vol. 35, no. 1, pp. 35–44, 1993. [3] R. Robbes, M. Lungu, and D. Röthlisberger, “How do developers react to API deprecation?: the case of a smalltalk ecosystem,” in SIGSOFT 2012. ACM, 2012, p. 56. [4] D. Dig and R. E. Johnson, “How do apis evolve? A story of refactoring,” J. Softw. Maintenance Res. Pract., vol. 18, no. 2, pp. 83–107, 2006. [5] D. C. Schmidt, “Guest editor’s introduction: Model-driven engineering,” Computer, vol. 39, no. 2, pp. 25–31, 2006. [6] M. Brambilla, J. Cabot, and M. Wimmer, Model-Driven Software Engineering in Practice, Second Edition, ser. Synthesis Lectures on Software Engineering. Morgan & Claypool Publishers, 2017. [7] A. Cicchetti, D. D. Ruscio, R. Eramo, and A. Pierantonio, “Automating coevolution in model-driven engineering,” in ECOC 2008. IEEE Computer Society, 2008, pp. 222–231. [8] M. Herrmannsdoerfer, S. Benz, and E. Jürgens, “COPE - automating coupled evolution of metamodels and models,” in Proc. of ECOOP 2009, ser. LNCS. Springer, 2009, pp. 52–76. [9] D. E. Khelladi, R. Bendraou, R. Hebig, and M. Gervais, “A semi-automatic maintenance and co-evolution of OCL constraints with (meta)model evolution,” J. Syst. Softw., vol. 134, pp. 242–260, 2017. [10] J. de Lara, E. Guerra, and J. S. Cuadrado, “When and how to use multilevel modelling,” ACM Trans. Softw. Eng. Methodol., vol. 24, no. 2, pp. 12:1– 12:46, 2014. [11] U. Frank, “Multilevel modeling - toward a new paradigm of conceptual modeling and information systems design,” Bus. Inf. Syst. Eng., vol. 6, no. 6, pp. 319–337, 2014. [12] M. Selway, M. Stumptner, W. Mayer, A. Jordan, G. Grossmann, and M. Schrefl, “A conceptual framework for large-scale ecosystem interoperability and industrial product lifecycles,” Data Knowl. Eng., vol. 109, pp. 85–111, 2017. [13] Y. Fu, G. Grossmann, K. Kaur, M. Selway, and M. Stumptner, “Multi-level risk modelling for interoperability of risk information,” in Proc. of IN4PL 2022. SCITEPRESS, 2022, pp. 242–249. [14] ——, “Towards the integration of multi-level and multi-view modelling for interoperability,” in MODELS 2023 Companion (Proc. MULTI Workshop). IEEE, 2023, pp. 679–688. [15] W. Zhang, J. Holtmann, D. Strüber, R. Hebig, and J.-P. Steghöfer, “Supporting meta-model-based language evolution and rapid prototyping with automated grammar transformation,” Journal of Systems and Software, vol. 214, p. 112069, 2024. [16] Y. Fu, M. Selway, G. Grossmann, K. Kaur, and M. Stumptner, “Modelling a warehouse with SLICER: A contribution to the MULTI warehouse challenge,” in MODELS 2024 Companion (Proc. MULTI Workshop). ACM, 2024, pp. 828–837. [17] C. Atkinson and R. Gerbig, “Flexible deep modeling with melanee,” in Modellierung 2016, 2.-4. März 2016, Karlsruhe - Workshopband, ser. LNI. GI, 2016, pp. 117–122. [18] F. Macı́as, A. Rutle, and V. Stolz, “Multecore: Combining the best of fixed-level and multilevel metamodelling,” in MODELS 2016 Companion (Proc. of MULTI Workshop), ser. CEUR Workshop Proceedings. CEURWS.org, 2016, pp. 66–75. [19] M. A. Jeusfeld and B. Neumayr, “Deeptelos: Multi-level modeling with most general instances,” in Proc. of ER 2016, ser. LNCS, 2016, pp. 198– 211. [20] F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics, vol. 1, pp. 196–202, 1945. [Online]. Available: https://api.semanticscholar. org/CorpusID:53662922 [21] M. Herrmannsdoerfer, S. Vermolen, and G. Wachsmuth, “An extensive catalog of operators for the coupled evolution of metamodels and models,” in SLE 2010, ser. LNCS. Springer, 2010, pp. 163–182. [22] M. Herrmannsdoerfer, D. Ratiu, and G. Wachsmuth, “Language evolution in practice: The history of GMF,” in SLE 2009, ser. LNCS. Springer, 2009, pp. 3–22. [23] R. Hebig, D. E. Khelladi, and R. Bendraou, “Approaches to co-evolution of metamodels and models: A survey,” IEEE Trans. Software Eng., vol. 43, no. 5, pp. 396–414, 2017. [24] L. Iovino, A. Pierantonio, and I. Malavolta, “On the impact significance of metamodel evolution in MDE,” J. Object Technol., vol. 11, no. 3, pp. 3: 1–33, 2012.
[25] W. Zhang, D. Strüber, and R. Hebig, “Development and evolution of xtextbased dsls on github: an empirical investigation,” Empirical Software Engineering, vol. 31, no. 3, p. 48, 2026. [26] Y. Fu, G. Grossmann, K. Kaur, M. Selway, and M. Stumptner, “Conflict management for multi-level models in collaborative modelling environments,” in MODELS 2025 Companion. IEEE, 2025, pp. 502–511. [27] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979. [Online]. Available: http://www.jstor.org/stable/4615733 [28] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988. [29] V. B. Kampenes, T. Dybå, J. E. Hannay, and D. I. K. Sjøberg, “A systematic review of effect size in software engineering experiments,” Inf. Softw. Technol., vol. 49, no. 11-12, pp. 1073–1086, 2007. [30] W. Zhang, B. Jiang, Y. Fu, A. Koziolek, R. Hebig, and D. Strüber, “Leveraging llms to support co-evolution between definitions and instances of textual dsls: A systematic evaluation,” arXiv preprint arXiv:2602.11904, 2026.
The 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME 2026) – Registered Reports