Multi-Agent Specification-based Metamorphic Testing of FMU-Based Simulations Ashir Kulshreshtha∗ , Abdullah Mughees∗ , Gaadha Sudheerbabu∗ , Tanwir Ahmad∗ , Kristian Klemets† Dragos Truscan∗ , and Mikael Manngård‡ , ∗ Åbo Akademi University, Finland ([email protected]) † University of Turku, Finland ([email protected])
arXiv:2605.25101v1 [cs.SE] 24 May 2026
‡ Novia University of Applied Sciences, Finland ([email protected])
Abstract—In many industrial domains, the Functional Mockup Interface (FMI) is used to exchange simulation models as Functional Mock-up Units (FMUs) across different partners using various modelling tools. This opens up the possibilities for simulation-based verification and validation using FMUs for ensuring reliable system behaviour. However, deriving effective test oracles for these simulation models remains challenging due to the absence of explicit expected outputs. This limits the applicability of conventional testing approaches, which require access to the internal workings of the systems. Metamorphic testing (MT) addresses this limitation by leveraging metamorphic relations (MRs), but extracting such relations from specifications remains largely a manual and error-prone process. To address this challenge, we propose an LLM-powered multi-agent workflow for specificationbased metamorphic testing of FMU-based simulation models. The approach takes functional and interface specifications as input and orchestrates multiple agents to extract requirements and derive MRs. These MRs are expressed using Given–When–Then patterns to structure input conditions (Given), transformations (When), and expected output behaviours (Then). These relations are then used to generate metamorphic test cases, execute simulations, and evaluate output consistency across multiple sessions. We evaluate the approach on a Lube Oil Cooling system FMU, demonstrating its ability to automatically generate meaningful MRs and corresponding test cases. Preliminary results indicate that the proposed workflow can effectively support the systematic verification and validation of dynamic simulation models by reducing manual effort and improving test generation. Index Terms—Metamorphic testing, multi-agent systems
I. I NTRODUCTION In modern industrial systems, model-based design is an increasingly integral approach to system development, in which components are designed and validated through simulation prior to physical implementation [1]. Organizations typically use various specialized modelling and simulation tools for model-based development and often collaborate with internal and external partners in this process. Systems development strategy using component-based design in such collaborative settings necessitates a standardized way of model exchange and co-simulation. The Functional Mock-up Interface (FMI) [2] is a standard for tool-independent exchange of dynamic simulation models. It defines a container and an interface to exchange simulation models as Functional Mock-up Units (FMUs). An FMU is a ZIP archive that typically contains: (i) binaries and/or source to execute the model, (ii) model description, and (iii) optional
resources (such as documentation files, maps and tables needed by the model, and/or all object libraries or DLLs that are utilized). This encapsulation brings along several challenges for validation and verification of the FMU components: a) often the source code of the model and its internal functionalities are not included in the package with it, allowing only for specification-based testing techniques to be applied, and b) the test oracles are not explicitly defined since typically the testing of such simulation models is done manually by domain experts. However, the simulation model of a component/system is designed based on the specifications of the real system it represents and is available for validation early in the development. This opens up the possibility of early simulationbased testing using specification-based testing techniques. Metamorphic testing (MT) is a specification-based testing technique that can be used to test systems lacking explicit test oracles [3]. MT checks whether multiple executions of the system under test (SUT) satisfy specific necessary properties, called metamorphic relations (MRs). It starts with a seed input and derives one or more follow-up inputs by applying a metamorphic transformation. Instead of checking one output against a fixed expected value, the test verdict is assigned based on whether the MR that links the seed and follow-up outcomes holds [4]. MT has been applied in many application domains, including simulation and modelling [5]–[9]. Despite prior studies on generating MRs and MT for simulation models, there is still no FMU-focused approach that starts from functional and interface specifications and continues through MR generation, test generation, simulation execution, and test-quality assessment. Moreover, identifying suitable MRs remains a practical challenge when exploring the applicability of metamorphic testing to new application domains [10], [11]. FMU-based simulations introduce practical constraints, for example, the internal model implementation may be unavailable, the executable interface is limited to exposed FMU variables, and not every requirement in the functional specification can be directly converted into a valid input-output MR. These constraints motivate the need for an automated workflow that identifies MRs from specifications, then designs, executes metamorphic tests, and reports requirement coverage for test adequacy and mutation score for test quality.
Large language models (LLMs) are increasingly used for relations demonstrated its usability [12]–[14]. In [12], a zeroautomated test generation, and recent studies [12]–[14] show shot prompting strategy is used to generate MRs and these MRs that LLMs can identify MRs in different application domains, are verified by domain experts. Their study, conducted across such as web applications, autonomous driving systems, and nine software systems, suggested that ChatGPT-generated MRs embedded systems. Although these studies demonstrate the can be used to generate innovative MRs and highlighted the effectiveness of LLMs in generating innovative and diverse need for human-in-the-loop validation. ChatGPT is used to MRs, they also emphasize the need for a domain expert to generate MRs for autonomous driving systems [13], and it validate the MRs to ensure a high accuracy. Open challenges is reported that the approach can generate diverse MRs that remain in systematically identifying effective MRs and reducing effectively improve coverage using a feedback-based prompting reliance on domain experts [11]. strategy. The third study [14] uses a few-shot prompting strategy In this paper, we investigate the following research questions: to derive MRs from natural-language requirements and convert them into executable forms using a domain-specific language • RQ1: Can a multi-agent workflow systematically perform automated MT from the functional and interface specification called SMRL. Their work shows that LLMs can help identify transformation-based test relations and reduce manual effort. of an FMU? • RQ2: To what extent does the approach extract specificationB. LLM-assisted Metamorphic Testing linked MRs and executable metamorphic test cases based LLM-assisted metamorphic testing has also been applied to on the FMU specification? embedded systems and the simulation and modelling domain. • RQ3: What is the quality of the generated test cases? More recent work [16] presents a multi-agent tool that applies • RQ4: What are the main runtime costs of the workflow metamorphic testing to autonomous driving systems. It derives across extraction, MR generation/refinement, test generation, MRs from driving rules, creates follow-up tests from existing and test execution? tests, executes them, and reports violations. In contrast, our To answer these research questions, we introduce approach targets FMU-based dynamic simulation models. Our AgenticMeta, an LLM-powered multi-agent workflow to approach differs mainly in how it infers MRs, structures tests, support metamorphic testing of FMU-based simulation models and refines them. We also address practical instability in using the functional and interface specifications as the primary large language model outputs by separating generation and input. The key contributions of this paper are as follows: refinement. A stable model produces initial MRs, and a stronger • A multi-agent approach for FMU-based metamorphic testing model refines the results to better match the specification and that extracts requirements from functional and interface reduce unsafe patterns. This can improve repeatability, but full specifications and then identifies and selects MRs using determinism is still difficult in LLM-based pipelines [17]. LLM-powered agents; Compared with prior MT work for simulation models, • The extracted MRs are specified as GWT requirement AgenticMeta focuses specifically on FMU-based simulations patterns, allowing for better processing by LLMs to generate where the testable interface is constrained by the variables metamorphic tests with customized input transformations exposed in the FMU model description. Earlier GWT-based and output relations for simulation models; MT work for dynamic simulation models [9] provides a useful • A feasibility evaluation on a Lubricating Oil Cooling system structure for expressing MRs; this paper extends that work FMU across multiple independent sessions, reporting require- by automating requirement extraction, MR generation, MR ment coverage, test execution outcomes, runtime statistics, refinement, test generation, and validation through LLM-based and output-level mutation analysis. agents. Recent LLM-assisted MR-generation studies mainly We will evaluate the approach on a dynamic simulation focus on producing candidate MRs, whereas AgenticMeta model, a simplified version of a Lubricating Oil Cooling (LOC) automates the entire MT process, integrating MR refinement, system [15], packed as an FMU. The LOC system models a test execution, and output-level mutation analysis into a single lube oil cooler that transfers the heat from the lubrication oil process. Compared with multi-agent MT for autonomous to the cooling water circuit of a marine engine unit. driving systems, our target domain is FMU-based dynamic The rest of this paper is organized as follows. Section II simulation, where inputs and outputs are continuous timediscusses related work. Section III presents the AgenticMeta series variables and where valid MRs must conform to the workflow and its main components. Section IV describes FMU interface constraints. Therefore, the primary contribution the experimental design and reports the evaluation results. of this work resides not in the application of MT or LLMs Section V discusses the limitations. Section VI concludes the in isolation, but in the integration of these components into a paper and outlines future work. traceable, FMU-oriented MT workflow. II. R ELATED W ORK
III. OVERVIEW OF THE A PPROACH
A. LLM-assisted MR Generation
A. Conceptual approach
The application of LLMs for automating different phases of MT has been investigated. Several studies exploring the applicability of ChatGPT for automatically generating metamorphic
Our approach aims to augment the main steps of the metamorphic testing process, namely, systematic identification and selection of MRs from functional and interface specifications,
and metamorphic test generation in patterns similar to Gherkin’s language Given-When-Then (GWT) requirement patterns [18] with an LLM-powered multi-agent workflow for validation of dynamic simulation models. The workflow comprises four phases: extraction, MR generation, test generation, and test execution, as illustrated in Figure 1. The first three phases are performed by specialized agents, whereas the test execution phase is performed by deterministic custom tools. The overall process is coordinated by a Coordinator node that orchestrates interactions between agent-based and deterministic components.
states the metamorphic input relation for transforming the seed inputs to morphed inputs, and Then step states the acceptance criteria as a metamorphic output relation between the outputs generated using the seed inputs and follow-up inputs. In the test generation step, the test design specified in each MR is converted into seed and follow-up input signals as a time series, each over a specified time interval for each SUT input. In the test execution step, which is the last step in the workflow, we run and evaluate the generated time series against the FMU system. B. Multi-agent workflow The agents, central to our workflow, perform well-defined tasks in a coordinated manner to facilitate LLM-assisted metamorphic testing. The approach follows a ’hub and spoke’ architecture where the pipeline is controlled by a Coordinator node which acts as the hub monitoring, orchestrating the workflow and functioning of the agents with clearly defined roles. The approach is iterative and incremental. The inputs to the workflow are: the FMU file of the system under test, including the model description, functional specification as pdf and binaries of the model, and a set of configuration parameters for the workflow, as follows: The following parameters can be used to customize the workflow for a given case study: system name: Name of SUT. system abv: Short abbreviation for the SUT. • fmu path: Path to the FMU file used for simulation. • output dir: Directory name where run outputs are saved. • max iterations: Max number of iterations per run. • mr count: Max number of MRs to generate per iteration. • test cases per mr: Max number of test cases to generate per MR. • llm provider: LLM service provider identifier. • •
Fig. 1: Overview of the Multi-Agent workflow The approach takes as input the simulation model, packed as an FMI-standard compliant FMU, which comprises the functional specification, model description and execution binaries to facilitate the metamorphic testing. In the first step of our approach, we extract system properties to be selected for MR selection from the functional and interface specifications of the simulation model. The system properties specified in the functional specification are extracted as test conditions. In addition to this, the cause-and-effect relationship between the input and output variables in a specified property is extracted from the specification document. From the model description XML file in the FMU package, the model variables and the interface specification, including inputs, outputs, parameters, and their details (name, description, variability, unit, datatype, minimum, and maximum values), are extracted. The extraction of relevant information and storing it in such a structured manner as specified above streamlines the identification and selection of the MRs in the next phase. Subsequently, in the MR generation step, MRs are generated with an MR identifier and are mapped to the test condition identifier it corresponds to, along with the test scenario description. The generated MR has the metamorphic source and follow-up test specified in the GWT pattern with Given, When, and Then keywords. The Given states the initial conditions to generate seed inputs for the source test execution, When
1) Coordinator Node: The Coordinator node maintains the global execution state, determines the phase of the workflow, and routes flow to the corresponding node. This design separates control-flow management from task-specific execution, enabling each LLM agent or deterministic module to operate on a well-defined responsibility while the Coordinator governs the overall progression of the pipeline. A shared graph state is a central context that manages execution information, tracks intermediate states, and propagates coordination signals across nodes throughout the workflow. At each workflow step, the Coordinator inspects the phase field in the shared graph state and applies a phase-to-node mapping to select the next executable component. After an iteration is completed, it records execution-related statistics. If the configured maximum number of iterations has not yet been reached, the Coordinator prepares for the next iteration by initialising iteration-specific directories and resetting iteration local artifacts to their initial state. When the iteration count is exhausted, the Coordinator marks the workflow as completed and exits the execution according to the configured stopping condition.
modelDescription.xml
Coordinator Node
Documentations Binaries
Fmu package Model Description Parser
modelDescription.xml Functional specification
Extractor Agent First Iteration Yes
No
Functional Description Markdown Generator
The output in nested JSON format comprises of the following information to facilitate the MR generation: system summary test conditions (categorized by property-type if relevant details are present in the functional specification), and model variables. 3) MR Generator Agent: In each iteration, the MR Generator Agent proposes a number of MRs as specified by the mr count workflow configuration parameter. The task prompt designed for the MR Generator Agent guides the generation of candidate MRs. An excerpt of the task prompt, which constraints the generated MRs to follow the schema and to assign priorities as per the rules, is shown in Listing II.
Extracted Previous Knowledge Extractions
Centralised Storage
Fig. 2: Workflow of Extractor Agent 2) Extractor Agent: The Extractor Agent operates by transforming the input functional specification document through a structured conversion pipeline, then formulating an output in an LLM-friendly content format (see Figure 2). It transforms the functional specification document from PDF to Markdown, a structured format that enhances the processing and accuracy of agents that accept it as input for proceeding with further steps of MT. The system prompt to define the role of the agent for performing the information extraction is shown in Listing I. System Prompt Role: You are a domain expert in Functional Mockup Interface (FMI) and Functional Mockup Unit (FMU) systems, with strong experience in requirement extraction and structured evidence preservation. Core Principles: 1. Preserve document context 2. Accept equivalent table layouts (grid/pipe/html/text) as the same semantic structure. 3. Document is authoritative — prioritize document facts as the single source of truth. 4. Verbatim evidence — every extracted fact (relationship, formula, constraint, test condition) must be grounded in a direct quote or precise reference from the document. 5. Tables are primary sources — treat input/output tables, parameter tables, and model description tables as the authoritative metadata source for variable specifications. 6. Precision over inference 6. Preserve relationships — for causal links and behavioral rules, write actionable statements that capture the trigger, condition, direction of change, and operational consequence. 7. Valid JSON output only — return well-formed JSON without markdown fences, commentary, or embedded natural language.
Listing I: System prompt of Extractor Agent The model interface data and the constraints for input and output variables, including names, units, causality, and allowed ranges, are extracted from the modelDescription.xml file of the FMU. A custom XML parser script extracts and converts the information to JSON format. To summarize, the Extractor Agent generates extraction output in a structured schema as defined in the task prompt.
Task Prompt Task: - Generate up to mr count NEW, UNIQUE, CONCRETE and test conditions focused Metamorphic Relations (MRs). - Use concise test conditions focused **Given-When-Then** form. - Each MR MUST include: id, req ids, scenario, given, when, then. - Prevent duplicates with current & previous MRs. - Keep the wording compact and clear in given, when and then. Prioritise MR generation categories in this order: priority Constraint by Category: - Behavioral - Performance
Listing II: Task Prompt of MR Generator Agent The prompt explicitly requires the candidates to be different from those generated in previous iterations by providing the agent with the accumulated MR history of up to three previous generations. Each candidate MR is assigned a unique MR identifier and linked to the requirements it addresses by listing the test condition identifier and variable relationship identifier. The agent generates MRs as GWT patterns under a strict JSON schema using the rules defined in the task prompt, so that subsequent stages can reliably generate metamorphic tests. Figure 3 shows an example MR generated by the agent in the structured schema as per the rules defined in the task prompt in Listing II. In the example, MR001, the initial condition values for the seed input signals are stated in the Given part. The metamorphic transformation in the When part of the inferred MR states that the input signal engine_load transforms from its seed value to a morphed value using the relational operator ’increase’. The agent generates transformations aligned by the rules specified in its task prompt for the When using patterns such as ’STEP’, ’RAMP’, considering the scenario and system properties to be validated. The metamorphic output relation to determine the test outcome is stated in the Then part using comparative and temporal operators such as the ones listed below: Increases than: Implies that morphed outputs of the selected output variable should eventually increase than the corresponding seed outputs. • Eventually Decreases than: Implies that morphed outputs of the selected output variable should eventually decrease than the corresponding seed outputs.
• Eventually
Fig. 3: MR generated by the agent in Given-When-Then format to: Implies that morphed outputs of the selected output variable should always be proportional to the corresponding seed outputs. • Equal to: Implies that morphed outputs of the selected output variable should always be equal to the corresponding seed outputs. • Settles within: Implies that seed and morphed outputs of the selected output variable should settle within X seconds to a defined set point value. In total, the agent generates mr count ∗ iteration count number of candidate MRs, where mr count is defined by MR Generator Agent and iteration count by the Coordinator node for metamorphic test generation. These generated MRs will be subsequently passed to the next agent for further validation before proceeding to the test generation. 4) MR Refiner Agent: validates if the system property specified as a test condition from the functional specification is correctly captured and translated into MRs by the MR Generator Agent. The validation criteria for each MR take into consideration the factual correctness, category consistency, constraint compliance, causal validity, and testability. Any missing detail or incorrect information in each MR is verified and refined using refinement rules by the MR Generator Agent. In the output schema of the refined MRs, a field named feedback provides the information on the refinement performed for each MR, in order to enhance the explainability of the decision, and the field named dropped with values as either ’true’/’false’ indicates whether the particular MR should be accepted with refinement or discarded. The repair loop is limited to a configurable number of attempts as defined in the pipeline configuration of the Coordinator node. The refined version of the example MR shown in Figure 3 is presented in Figure 4, where the refined part is highlighted in green, and the feedback on the refinement performed is highlighted in blue. 5) Test Generator Agent: The Test Generator Agent generates a configurable count of metamorphic tests for each MR received for the previous phase. In this step, the simulation parameters of the model, such as the start and end times, and the system I/O constraints, are considered. The seed input values are assigned based on the initial conditions in the extraction output. The concrete values for performing the seed to follow• Proportional
Fig. 4: MR refined by the agent and the feedback provided
up transformation of input signals are defined in this step using a sampling algorithm. The algorithm uses the initial condition values extracted from the functional specification as the baseline for generating follow-up inputs using the input relation in the When part of MR. The test design also accounts for the rule that any step/ramp change to any input signal should be placed near the start of the simulation time window (∼ 10-25%). The rules also reinforce that any set-point values among the input signals should remain unchanged. Each MR/refined MR from the previous phase guides the generation of one or more tests. 6) Test Validator Agent: The Test Validator Agent primarily validates the test inputs and MRs in the test case. It checks the test case against the functional specification and the metamorphic relation patterns used to generate the test inputs. The validation is performed based on repair rules such as data-type and boundary checks for test input values, type of metamorphic relation pattern, and simulation-specific time constraints. In the output schema of validated tests, two fields named fixed and dropped capture the validation decision, and a summary is provided by the agent in the field validation summary. The decision categorizes a test as any one of the following: • If the test case is fully valid, keep it unchanged and set fixed=false, dropped=false. • If the test case can be safely repaired, repair it, apply the smallest possible fix, and set fixed=true, dropped=false. • If the test case is fundamentally broken or unsafe to repair, set fixed=false, dropped=true. To summarize, the valid/repaired tests are passed to the test instantiator, which generates the input time series for all the input signals. The test input signals for which the metamorphic transformation is applied capture the evolution of values over time, and the other inputs are represented as constant vectors. These generated input signals are wrapped into an input object and proceed to the test execution phase. C. Implementation AgenticMeta is implemented in Python as a layered workflow built around LangGraph [19]. The implementation separates the
pipeline into five layers: the orchestration layer, the specification C. Evaluation metrics and variable extraction layer, the MR generation and refinement We evaluate AgenticMeta using the following metrics: layer, the test generation and validation layer, and the test Requirements coverage denotes the number of requirements execution layer. Each layer is integrated with a persistence extracted from the specification and mapped into metamorphic mechanism that stores its inputs, outputs, intermediate results, test scenarios produced per test session. Test Case Summary and state, enabling traceability, debugging, and post-run eval- enumerating generated test cases, their MR assignments, uation across the complete workflow. This separation helps and pass/fail outcomes, providing an indicator of workflow with debugging and evaluation, since intermediate artifacts are effectiveness. Runtime statistics to measure the efficiency of stored after each phase, the errors can be traced to a specific the workflow across different phases and units. layer, and individual components can be modified or replaced Mutation score to measure the quality of metamorphic tests without affecting the rest of the workflow. The layered design generated by the approach successfully executed (e.g., without also allows different pipeline settings to be used at different any syntax errors) against the SUT. Since we do not have access stages, enabling isolated evaluation of each stage, more flexible to the internal specification of the FMU, we imitate possible reasoning early on, and more stable behaviour during execution. design and implementation mistakes by applying systematic Overall, this structure makes AgenticMeta easier to extend and changes (mutations) on the output signals. These mutations adapt to new tasks. are defined based on a set of mutation operators that are systematically applied, one at a time, to each output; this will IV. E VALUATION This section evaluates AgenticMeta in order to provide an result in a slightly incorrect version (mutant) of the FMU In this work, we use the following mutation operators: answer to the research questions discussed in Section I. • Mirror Mutation: Replaces an output signal with its mirror A. Case Study version from the start time step of the simulation horizon. For evaluation, we use a simplified version of a • Crossover Mutation: Select two output time series of different Lubricating Oil Cooling (LOC) system. The LOC system metamorphic relations for a given testcase and pick a specific has a proportional–integral (PI) controlled valve that crossover site and interchange the values of the time series regulates the lubrication oil temperature at a constant signals after the crossover site. set-point at the engine inlet. The controller aims to keep • Polynomial Mutation: Given a variable x in a time series in the lubrication oil temperature at the outlet within the the range (xmin , xmax ), we slightly change it by a random specified boundary values under all operating conditions. amount determined by a polynomial distribution. The Lubricating Oil Cooling system has the following The process starts with the execution of the generated test input variables, temperature_cooling_liquid_in, cases on the original version of the FMU. To save on the mass_flow_cooling_liquid_in, engine_load, simulation time, we apply the mutation operators directly on and setpoint_temperature_oil (held constant the recorded output time series for each output of the passed throughout the simulation) and following output test cases. Each application of a mutation operand to an output variables temperature_oil, position_valve, time series will result in a new mutant. We compare the seed temperature_cooling_liquid_out, and output and the mutated output using the output MRs and assign mass_flow_cooling_liquid_out. The simulation a pass/fail verdict. Finally, we calculate the test adequacy as model of the LoC system has been implemented and tested a mutation score, with values between 0 and 1, as the ratio manually by the designers, and in the context of this evaluation, between the number of mutants on which tests failed (killed it is used as the ground truth. The case study and tool were mutants) and the total number of mutants created. deployed locally on a Windows-based PC (13th Gen Intel Core i9-10900X, 3.70 GHz, 64 GB RAM, 4 GB dedicated D. Results graphics card, 2 TB storage). We use FMPy [20], a Python We evaluated the workflow across 10 independent sessions library, to simulate the FMU and execute the tests generated of the LOC case study. Each run used the same configuration. by the workflow. B. Workflow configuration 1) RQ1: AgenticMeta successfully executed the complete We used the following workflow configuration for our workflow, including extraction, MR generation, refinement, test experiments. Some of these values were chosen based on generation, validation, instantiation, simulation, and mutation empirical evaluations. All agents use OpenAI as the model analysis in all ten sessions. provider and the GPT-5.4-mini model. Reasoning effort is The results show that the proposed architecture is feasible set to none for the Extraction agent, low for test generation, for metamorphic testing of FMU-based simulation models. medium for MR generation and test validator, and high for the Across all ten sessions, the workflow successfully progressed MR refinement agent. The Extraction agent uses temperature from requirement extraction to executable metamorphic test 0 (temperature is only applicable with reasoning effort: none). generation and execution under the specified configuration. It Each run consists of 5 iterations, with up to 5 MRs per iteration was also able to perform mutation-based quality assessment and 5 test cases per MR. on top of the core execution.
TABLE I: Runtime statistics across 10 sessions Phase level statistics (s)
Sessions
Unit level statistics (s)
Total Exec.
Extraction
MR Gen
Test Gen
Test Exec
Gen. Time/TC
Gen. Time/MR
Exec. Time/TC
1 2 3 4 5 6 7 8 9 10
771.39 913.54 881.11 1074.30 900.78 951.92 1021.93 943.33 1000.65 908.27
81.74 78.70 81.04 88.94 79.27 83.76 80.68 81.05 97.31 93.94
540.28 617.23 619.95 773.78 612.57 655.39 740.52 662.22 705.96 567.56
110.24 160.21 132.72 159.77 150.62 157.65 149.59 147.51 151.85 184.36
39.14 57.41 47.40 51.81 58.31 55.12 51.14 52.55 45.53 62.41
2.69 2.67 2.77 2.90 2.55 2.43 3.05 2.68 2.81 2.60
45.02 41.14 47.68 48.36 40.84 40.96 46.28 47.30 50.42 31.53
18.82 15.23 18.36 19.53 15.27 14.64 20.86 17.15 18.53 12.79
Average (s)
936.72
84.64
649.55
150.45
52.08
2.72
43.95
17.12
TABLE II: Coverage statistics across 10 sessions Sessions
MR Summary & Requirement Coverage (%)
Test Summary
Mutation Coverage
Gen. MR’s
Dropped
Refined
Coverage (%)
Generated
Passed (%)
Failed (%)
Generated
Killed
Score
1 2 3 4 5 6 7 8 9 10
14 16 14 17 15 16 16 15 17 18
2 1 1 1 0 0 0 1 3 0
12 15 13 16 15 16 16 14 14 18
64.71 35.29 58.82 70.59 52.94 82.35 70.59 64.71 45.83 58.82
41 60 48 55 59 65 49 55 54 71
85.37 76.67 91.67 83.64 72.88 63.08 53.06 70.91 64.81 78.87
14.63 23.33 8.33 16.36 27.12 36.92 46.94 29.09 35.19 21.13
104 176 162 174 162 94 138 158 78 126
65 83 85 83 96 63 82 91 46 57
0.63 0.47 0.52 0.48 0.59 0.67 0.59 0.58 0.59 0.45
Average
15.80
0.90
14.90
60.47
55.70
74.10
25.90
137.20
75.10
0.56
2) RQ2: Across the 10 sessions, the workflow generated an average of 15.80 MRs and 55.70 test cases per run. The average requirement coverage was 60.47% (see Table II, MR Summary & Requirement Coverage). These results indicate that the workflow can repeatedly generate specification-linked MRs and executable metamorphic test cases based on the FMU interface. The results also show that the workflow can repeatedly produce specification-linked MRs and executable tests. The produced MRs were manually checked by a domain expert and considered to be meaningful and consistent with the system specification. However, full requirement coverage was not achieved because not all extracted requirements could be converted into valid MRs over the exposed FMU interface. Manual inspection suggested two main causes: • a few extracted requirements are generic statements about system behaviour and do not define a directly manipulable input-output relation. • a few requirements described output behaviours that cannot be directly influenced by the available FMU inputs. Such requirements were often removed during the MR refinement stage because the agent reasoned that a valid follow-up output could not be guaranteed with the given inputs. It was observed that the MRs that were not covered or dropped were mainly those based on indirect, generic, or weakly controllable behaviours. In the LOC case study, MRs based on direct relations between controllable inputs and
observable outputs were more stable, while MRs requiring indirect influence over variables were more likely to be refined or removed. This indicates that the approach is most effective when the specification states explicit causal links between input perturbations and expected output responses. The test case pass rate across simulation sessions averaged 74.10% (see Table II, Test Summary). This indicates that most generated tests were semantically aligned with the generated MRs. As the implementation of the LOC is considered to be correct, the failing tests in this case are considered false positives and a limitation of our workflow, which we will address in future work. Further investigation showed that several failures occurred when the output variables initially moved in the expected direction but settled at a different value before satisfying the specified relation. Other failures occurred when the output settled outside the specified tolerance range or around a value different from the setpoint values. Both these situations can be due to the ramp-up values defined by the workflow. These observations indicate that future versions of the workflow should include stronger validation of temporal operators and tolerance values. 3) RQ3: The average mutation score across the ten sessions was 0.56, corresponding to 56% of generated mutants killed (see Table II, Mutation Coverage). These results indicate that the majority of mutations are detected by the evaluator, but they do not exhaustively cover the model behaviour. The mutation results should be interpreted cautiously. Be-
cause mutations are applied to recorded output time series simulation models. The approach used functional and interface rather than to the FMU implementation, the score reflects specifications to extract test-relevant requirements, generate and the sensitivity of the MRs to selected output deviations, not refine MRs, produce executable metamorphic test cases, and the full fault-detection capability of the workflow against evaluate test outcomes using metamorphic oracles and mutation implementation defects. In addition, the pass rate and mutation analysis. The feasibility evaluation was performed on the LOC score do not always increase together. The reason was that a system, which shows that the workflow can stably execute higher pass rate means more test cases satisfy their original the complete testing across repeated sessions. AgenticMeta MRs, and mutation analysis is then applied to a larger set of generated specification-linked MRs and executable test cases, passing tests. This increases the number of generated mutants. achieved an average requirement coverage of 60.47%, an However, some mutation operators are less effective for certain average test-case pass rate of 74.10%, and an average mutation relation types. In these cases, the mutated outputs may still score of 56%. These results indicate that the approach is satisfy the relation, so the mutants are not killed. Consequently, feasible for supporting automated metamorphic testing of FMUsessions with more passing tests may generate more surviving based simulations. At the same time, the results also highlight mutants, which reduces the overall mutation score even when important limitations. Requirement coverage is constrained by the test pass rate is high. the quality and testability of the specification. 4) RQ4: The average test-generation time is 2.72 seconds Future work will extend the workflow by transforming the per test case, the average MR-generation and refinement time current deterministic Coordinator node into an LLM-based is 43.95 seconds per accepted MR, & the average execution agent. This would allow the Coordinator to make adaptive time per test case is 17.12 seconds (see Table I, Unit level decisions during execution, such as prioritizing uncovered restatistics). These results show that the overall time taken for quirements, deciding when additional MR refinement is needed, the complete loop from extraction to execution is very low selecting alternative strategies, and determining whether the per test case. The compact test-case schema generated by the workflow should continue or stop based on coverage and testtest generator agent is effective in keeping the test generation quality feedback. In addition, future work will aim to improve phase relatively fast. mutation analysis by designing mutation operators that are The main efficiency bottleneck is the LLM-based MR more closely aligned with the semantics of the evaluated MRs. generation and refinement. The refinement phase remains This would make mutation-based assessment more robust for comparatively expensive due to the ’high’ reasoning effort FMU systems. of MR Refiner Agent, and also it checks generated MRs against ACKNOWLEDGMENTS both structured extraction output and the original functional specification, making it a resource-intensive process. The result This work was funded by the Finnish Ministry of Educasuggests that future optimization should focus on reducing tion and Culture’s Doctoral Education Pilot under Decision unnecessary MR refinement calls, prioritizing uncovered reNo. VN/3137/2024-OKM-6 (The Finnish Doctoral Program quirements more selectively, caching stable extraction artifacts, Network in Artificial Intelligence, AI-DOC) and Business and using less expensive models or reasoning settings for lowFinland via the Virtual Sea Trial project (VST), under grant risk validation steps. 7187/31/2023. V. T HREATS TO VALIDITY R EFERENCES In this section, we discuss the possible threats to the validity of our study. First, the evaluation is conducted on a single [1] J. Cederbladh et al. Early validation and verification of system behaviour in model-based systems engineering: A systematic literature review. ACM FMU-based case study, the LOC system. Although LOC is Transactions on Software Engineering and Methodology, 33(3), 2024. representative of dynamic simulation models with continuous [2] T. Blochwitz et al. Functional mockup interface 2.0: The standard for inputs and outputs, the findings may not generalize to models tool independent exchange of simulation models. In 9th international modelica conference, pp. 173–184. The Modelica Association, 2012. with substantially different control logic, interface structures, [3] T. Y. Chen et al. Metamorphic testing: a new approach for generating or operating characteristics. next test cases. arXiv preprint arXiv:2002.12543, 2020. Second, the workflow depends on the quality of the func- [4] H. Liu et al. A new method for constructing metamorphic relations. In 12th International Conference on Quality Software. IEEE, 2012. tional specification. As the implementation uses document conversion tools and evidence-preserving extraction prompts, [5] S. Segura and Z. Q. Zhou. Metamorphic testing 20 years later: A handson introduction. In Proceedings of the 40th International Conference on the generated MRs are directly constrained on the quality of Software Engineering: Companion Proceeedings, pp. 538–539, 2018. the specification document. In addition, the workflow cannot [6] A. Núñez and R. M. Hierons. A methodology for validating cloud models using metamorphic testing. annals of telecommunications-annales des reliably generate MRs for variables that are not described in télécommunications, 70(3):127–135, 2015. model description.xml, since the model interface description [7] M. Lindvall et al. Metamorphic model-based testing of autonomous sysis treated as the authoritative source for available FMU inputs tems. In 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET), pp. 35–41. IEEE, 2017. and outputs. VI. C ONCLUSIONS & F UTURE W ORK This paper proposed AgenticMeta, a multi-agent workflow for specification-based metamorphic testing of FMU-based
[8] M. Olsen and M. Raunak. Increasing validity of simulation models through metamorphic testing. IEEE Trans. on Reliability, 68(1), 2018. [9] G. Sudheerbabu et al. Validation of dynamic simulation models using metamorphic testing and given-when-then patterns. In Modelica Conferences, pp. 139–146, 2025.
[10] S. Segura et al. A survey on metamorphic testing. IEEE Transactions on Software Engineering, 42(9):805–824, 2016. [11] T. Y. Chen et al. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR), 51(1):1–27, 2018. [12] Q. H. Luu et al. Can chatgpt advance software testing intelligence? an experience report on metamorphic testing. arXiv:2310.19204, 2023. [13] Y. Zhang et al. Automated metamorphic-relation generation with chatgpt. In Proceedings of the 47th IEEE Annual Computers, Software, and Applications Conference (COMPSAC), pp. 1–6. IEEE, 2023. [14] S. Y. Shin et al. Towards generating executable metamorphic relations using large language models. In Intl. Conf. on the Quality of Information and Communications Technology, pp. 126–141. Springer, 2024. [15] NoviaRDISeafaring. Virtual Sea Trial Project. https://github.com/ Novia-RDI-Seafaring/fmu-opc-hackathon/tree/main/fmus/loc, 2024. [16] L. Liang et al. AutoMT: A Multi-Agent LLM Framework for Automated Metamorphic Testing of Autonomous Driving Systems. arXiv preprint arXiv:2510.19438v1, 2025. [17] B. Atil et al. Non-determinism of ”deterministic” llm settings. arXiv preprint arXiv:2408.04667, 2024. [18] M. Wynne and A. Hellesoy. ”The cucumber book: behaviour-driven development for testers and developers”. Pragmatic Bookshelf, 2012. LangGraph. https://reference.langchain.com/python/ [19] LangChain. langgraph/overview, 2024. Accessed: 2026-04-30. [20] DassaultSystémes. FMPy. https://fmpy.readthedocs.io/en/latest/, 2017. Accessed: 2026-04-21.