Metamorphic Testing of Transpilers via Mutation Consistency of Programs Enea Raffaele Ilario Papaleo [email protected] University of Milano-Bicocca Milano, Italy
Luca Guglielmo
[email protected] University of Milano-Bicocca Milano, Italy
arXiv:2607.29247v1 [cs.SE] 31 Jul 2026
Abstract Transpilers are increasingly used for software development, especially in industrial domains that rely on domain-specific languages (DSLs), to allow engineers to work with familiar concepts and appropriate abstractions. Ensuring the correctness of these instruments is therefore critical in many industrial settings. This paper observes that existing approaches for compiler testing hardly generalize to transpilers. Differential testing approaches are hindered as multiple equivalent implementations of the transpiler under test are seldom available in practice. The approaches based on metamorphic testing assume the ability to execute the compiled binaries, an assumption that cannot be always made for transpilers, which oftentimes produce results expressed as source code, requiring complex toolchains, hardware-in-the-loop setups, and depending on non trivial inputs. This paper introduces a novel metamorphic testing technique tailored to transpilers. Instead of reasoning about the runtime behavior of compiled programs, our approach defines metamorphic relations directly over the source code produced by the transpiler. These relations capture a property that we call mutation consistency of the (transpiled) programs: mutation-style changes in the input DSL program must induce predictable and structurally consistent changes in the generated output. We implemented this idea in a tool, MCP-Tester, and evaluated it through a case study conducted in the context of a technology-transfer project. Our current empirical results indicate that the proposed approach can effectively reveal faults that would remain undetected with pure fuzzing.
CCS Concepts • Software and its engineering → Software testing and debugging.
Keywords Transpiler testing, Metamorphic testing, Program Mutations.
1
Introduction
Transpilers, also known as source-to-source compilers, play a central role in modern industrial software development. A transpiler translates programs written in a source language into code expressed in another (target) programming language, enabling engineers to work with higher-level or domain-specific concepts, while obtaining code that is more suitable for compilation and execution [2, 3, 7]. Their relevance has grown significantly with the widespread adoption of domain-specific languages (DSLs) [8, 10, 28]. DSLs allow domain experts to express system behavior using familiar concepts and terminology, but the resulting specifications typically require translation into a general-purpose programming
Giovanni Denaro
[email protected] University of Milano-Bicocca Milano, Italy
language before they can be compiled or deployed. As a consequence, the correctness of transpilers becomes mission-critical: faults in the translation process may silently propagate into the generated code and ultimately into deployed systems. Despite their importance, automated testing techniques for transpilers remain largely unexplored. A possible workaround is to borrow solutions devised for compiler testing, drawing on the assumption that a transpiler is a "sort of" compiler. Existing research on compiler testing offers two main families of approaches: differential testing and metamorphic testing. Differential testing compares the outputs of multiple, supposedly equivalent compiler implementations [6, 13, 20, 24], but is inapplicable when only a single implementation exists, as it is common for many industrial transpilers. Metamorphic testing for compilers [4, 5, 11, 12, 19, 27] defines relations over source programs whose compiled binaries should behave equivalently at runtime, exploiting the expected equivalences as automated test oracles. These techniques draw on a fundamental assumption: the compiler produces executable binaries that can be run with available or easily derivable inputs. However, this assumption often breaks down in the case of transpilers, as their output is itself source code, often requiring complex toolchains, hardware-in-the-loop setups, or domain-specific simulators for compilation and execution. Moreover, the mapping of the input space across the original and the transpiled sources can be far than trivial. In such settings, existing compiler-oriented metamorphic testing approaches may become ineffective or even inapplicable. This paper introduces a novel metamorphic testing technique specifically designed for transpilers. Instead of reasoning about the runtime behavior of compiled binaries, our approach embraces metamorphic relations defined directly over the source code produced by the transpiler. These metamorphic relations specify how mutation-style changes in the input source code are expected to consistently map onto corresponding equivalencies and differences in the output source code produced by the transpiler under test. We call the underlying property Mutation Consistency of transpiled Programs (MCP). This shift from runtime equivalence of compiled binaries to structural consistency of generated source code enables metamorphic testing in contexts where traditional compiler-testing techniques cannot operate. This idea emerged from a technology-transfer project with an industrial partner, whose transpiler translates control-logic DSL programs into executable code for industrial automation systems. This collaboration provided both the motivation and the opportunity to evaluate our technique in a realistic setting. So far, we conducted a first case study in which we applied the MCP metamorphic testing for validating the partner’s transpiler. The case study
Papaleo et al.
also provided the ground for the experiment that we report in this 1 paper, in which we injected faults into the bytecode of the partner’s 23 transpiler and assessed the ability of our approach to reveal those 4 faults. The empirical results indicate that the MCP approach can 5 6 effectively detect faults that would remain unexposed with pure 7 fuzzing, testifying the practical significance and the potential of 8 9 the approach proposed in this paper. 10 The remainder of the paper is organized as follows. Section 2 11 reviews existing approaches to compiler testing and highlights 12 why they do not necessarily generalize to transpilers. Section 3 13 14 introduces the MCP approach and presents MCP-Tester, an auto- 15 mated test generator based on the MCP approach. Section 4 reports 16 17 our empirical evaluation conducted during the case study with 18 the industrial partner. Section 5 discusses related work. Section 6 19 20 summarizes our contribution and future research directions. 21
2
State of the Art and Overview of the Proposed Approach
In this section, we survey the main approaches to applying metamorphic testing for compiler testing, acknowledging that these approaches can also be used to test transpilers. We center the discussion around a sample bug of a flawed transpiler which reveals the limitations of the existing approaches. We then refer to the same example to highlight the distinctive characteristics of the approach that we propose in this paper.
2.1
Working Example
As a working example, we introduce a sample transpiler, which aims to compile source programs written in a custom domainspecific language (DSL) into corresponding C code. We consider a DSL that mimics (modulo adaptations and simplifications for the purpose of making the presentation accessible) a DSL defined by an industrial partner. This language is designed to support their domain analysts in specifying the control logic of industrial plants by means of a state-machine-based formalism, that the analysts (who are not professional IT programmers) can use more effectively than conventional programming languages. They can then use the transpiler to translate the DSL control logics into actual C programs, which can be compiled and deployed. Figure 1 shows a sample logic written in the considered DSL. It defines a finite-state machine that represents the behavior of a simple traffic light. The program begins with the declarations of the set of states that the traffic light can assume (line 2), the relevant actuator commands (line 3 and line 4), and a configuration parameter that can be set to activate or deactivate the camera (line 5). Next, lines 7–17 contain the core of the state machine logic. Like in most embedded software programming, logic is split into two phases, the initialization phase and the cycle phase, with the former being executed only once on power-on, and the latter looping at fixed time intervals until the machine loses power. In particular, line 8 specifies the initialization logic that must be executed on power-on to properly set up the state machine. In this case, the logic ensures that the traffic light activates its red light at power-on. The transition to instruction is used to instruct the internal logic to queue up a transition to the specified state at the end of the cycle. The cycle logic is located at lines 9–16. Initially, the presence of a
declarations { v a r i a b l e s t a t e : E n u m e r a t i o n { Red , Green , Y e l l o w } a c t u a t o r command camera ( argument e n a b l e : B o o l e a n ) a c t u a t o r command s p e a k e r ( argument e n a b l e : B o o l e a n ) c o n f i g u r a t i o n parameter e n a b l e C a m e r a : B o o l e a n } fsm { i n i t { t r a n s i t i o n t o Red } cycle { i f ( enableCamera ) { i f ( s t a t e i s Red ) s e n d command camera ( e n a b l e = t r u e ) e l s e s e n d command camera ( e n a b l e = f a l s e ) } s e n d command s p e a k e r ( e n a b l e = s t a t e i s Red ) t r a n s i t i o n to s t a t e . next } } configure { s e t parameter e n a b l e C a m e r a t o f a l s e s e t p u l s e t o 10 s e c o n d s }
Figure 1: A sample traffic-light controller in a custom DSL camera for this specific traffic light installation is checked (line 10), so that the proper enabling or disabling logic can be carried out: if the traffic light is currently showing a red light, then the camera must be enabled, which is accomplished by the use of the send command construct as shown on line 11; on the other hand, the camera must be disabled in all other situations, thus line 12 sends the opposite command as needed. Following this, the traffic light must illuminate the next light in the sequence–which we assume to be the usual red-green-yellow cycle–and enable or disable the pedestrian speaker accordingly. The code at lines 14–15 accomplishes this behavior by leveraging the ordering of the states previously declared at line 2. Lines 18–21 indicate the actual configuration. Line 19 indicates that this particular traffic light is not equipped with a camera. Line 20 sets the time interval between successive executions of the loop to ten seconds. The transpiler translates the DSL definitions, like the ones above, into an equivalent program in C. Figure 2 shows the translation produced for the logic defined in Figure 1. In our example, we assume that the transpiler has a fault that causes the entries in an enumeration to be translated in the opposite order with respect to how they were declared. In this case, the bug manifests itself between lines 0–0 of trafficlight.h in Figure 2. The resulting code is still compilable and executable, but its behavior is now no more compatible with the logic encoded by the DSL. In fact, the error makes the traffic light cycle according to a red-yellow-green cycle, rather than the one specified in the DSL. The code is still compilable and executable, but it is no more compliant with the DSL logic. Infact, the behavior of the traffic light itself now erroneusly but consistently cycles through the lights according to a red-yellow-green cycle.
2.2
Current Techniques for Metamorphic Testing of Compilers
At the state of the art, one can address the problem of generating test cases for a transpiler by leveraging existing research on compiler testing. A complete approach shall encompass the two main tasks of (i) generating test programs for exercising the compiler under test,
Metamorphic Testing of Transpilers via Mutation Consistency of Programs
trafficlight.h # pragma once # i n c l u d e < s t d b o o l . h> # i n c l u d e " fsm . h " typedef struct { _Bool p_enableCamera ; } trafficlight_t ; t y p e d e f enum { / / Bugged Yellow = 0 , Green , Red , } trafficlight_state_t ; void t r a f f i c l i g h t _ f s m _ i n i t ( t r a f f i c l i g h t _ t ∗ ) ; void t r a f f i c l i g h t _ f s m _ c y c l e ( t r a f f i c l i g h t _ t ∗ , t r a f f i c l i g h t _ s t a t e _ t ) ; _Bool get_enableCamera ( t r a f f i c l i g h t _ t ∗ ) ;
trafficlight.c # i n c l u d e " actuator_command . h " # include " actuator_command_ids . h " # i n c l u d e " fsm . h " # include " t r a f f i c l i g h t . h" void t r a f f i c l i g h t _ f s m _ i n i t ( t r a f f i c l i g h t _ t ∗ instance ) { f s m _ t r a n s i t i o n _ t o ( Red ) ; } void t r a f f i c l i g h t _ f s m _ c y c l e ( t r a f f i c l i g h t _ t ∗ instance , trafficlight_state_t current_state ) { i f ( get_enableCamera ( i n s t a n c e ) ) { i f ( ( c u r r e n t _ s t a t e == Red ) ? B_TRUE : B_FALSE ) { s e n d _ a c t u a t o r _ c o m m a n d ( CAMERA_ACTCMD_ID , B_TRUE ) ; } else { s e n d _ a c t u a t o r _ c o m m a n d ( CAMERA_ACTCMD_ID , B_FALSE ) ; } } s e n d _ a c t u a t o r _ c o m m a n d ( SPEAKER_ACTCMD_ID , ( c u r r e n t _ s t a t e == Red ) ? B_TRUE : B_FALSE ) ; fsm_transition_to ( ( current_state + 1) % 3 ) ; } _Bool get_enableCamera ( t r a f f i c l i g h t _ t ∗ i n s t a n c e ) { r e t u r n i n s t a n c e −> p _ e n a b l e C a m e r a ; }
Figure 2: The output of the execution of a (bugged) transpiler on the code in Figure 1
and (ii) defining oracles for checking the results of the generated test cases [4]. The solution for the former task can be engineered by grounding on the grammar of the input language (e.g., the DSL of our example). In contrast, identifying test oracles is most often a major challenge, due to both missing a specification of the behavior of the target compiler and the hardness of coping with the semantic richness of the input and output languages. Researchers have explored two main approaches for defining oracles while generating test cases for compilers: differential testing and metamorphic testing [4]. Differential testing requires the existence of multiple, supposedly equivalent implementations of the compiler, in order to cross-check the results obtained while testing a compiler against the results of an equivalent counterpart. In this paper, we aim at addressing scenarios in which equivalent implementations are not available, and thus we consider solutions based on differential testing out of our scope. On the other hand, metamorphic testing relies on metamorphic relations, which specify how particular changes to the test input are expected to change the output. So far, researchers have proposed metamorphic testing of compilers grounded on metamorphic relations that exploit either nonexecuted code regions (dead code) of the test programs, or transformation rules to obtain test programs yielding equivalent outputs.
Metamorphic Relations via Dead Code Manipulation. A possible approach is to manipulate a test program is by modifying a region of the code that is never executed. In this way, we obtain a second test program that, by construction, should yield the same output as the original test program when they are both executed with the same input. Thus, we can define a (metamorphic) test oracle by checking that the two executable programs, obtained through compilation of the test programs we previously obtained, actually exhibit the same behavior when executed with a given set of inputs. Should there be a bug in the compiler under test, we expect that this type of test oracle may fail for some possible dead code manipulations. Existing techniques are focused on either dead code elimination or code insertion in dead regions [11, 12]. For example, with reference to the sample test program outlined in Figure 1, the test generator can notice that the logic comprised by the if statement at line 10 is dead code, as variable enableCamera is configured with the value false. Thus, it can straightforwardly obtain equivalent test programs, by removing or adding statements within the body of the if statement. Then, it can run the transpiler of the example against a pair of equivalent test programs, turn the resulting C code into corresponding binaries, and execute those binaries to check the equivalence of their outputs. The test fails upon observing any difference in the output, evidence that the transpiler under test produced non-equivalent C programs out of equivalent DSL sources. Otherwise, the test passes and the test generator can iterate the process for other pairs of equivalent test programs, while also potentially considering further (automatically generated) test programs other than the one in Figure 1. However, we observe that this approach is unable to detect the fault of the transpiler we have exemplified above. Even though the C code generated by the transpiler for both our sample test program and the equivalent test programs created by the test generator are all affected by the bug, all the corresponding binaries behave the same (and fail equivalently) at runtime. Therefore, metamorphic oracles based on dead code manipulation will report a "test passed" verdict in all cases. Metamorphic Relations via Equivalent Expressions, Assignments, and Submodules. Other authors construct equivalent test programs by manipulating an initial test program according to semanticpreserving program transformations. Tao et al. obtain equivalent programs based on equations that provide basic building blocks to construct equivalent expressions, which in turn allow for building equivalent assignments and submodules [27]. Similarly, it is possible to inject into the test programs suitably synthesized semanticpreserving code [19], or reorder some statements in the code, or even apply other types of semantic-preserving program transformations [5]. For instance, with reference to our sample test program, the test generator can detect that the two last statements in the cycle block for the fsm in the test program are order independent: changing the light of the traffic light and enabling or disabling the speaker are two actions that can be carried out in any order. It can thus obtain an equivalent test program by swapping the two statements with each other, execute the transpiler against both equivalent test programs, and verify the oracle that the binaries compiled from the corresponding C code produce the same output at runtime.
Papaleo et al.
Nevertheless, this approach is also unable to detect the fault of the transpiler, just as the one we analyzed previously. In fact, the binaries obtained from the equivalent programs still behave the same (and fail equivalently) at runtime.
With reference to our example, let 𝑃 ′ be the test program of Figure 1. Then, the test generator can satisfy the precondition of the metamorphic relation by generating a test program 𝑃 ′′ that is identical to Figure 1 but in which it replaces line 2 with the following: s t a t e : E n u m e r a t i o n { NEWVAL , Red , Green , Y e l l o w }
2.3
Overview of Our Technique
In this paper, we propose a novel technique for testing transpilers, which exploits metamorphic testing in a radically different way from the existing approaches on compiler testing. We ground on the observation that a transpiler is distinctively different from a traditional compiler, as it is not designed to produce an executable binary; rather, it renders the given program to the corresponding source code. For instance, the transpiler of our working example renders programs written in the DSL of Figure 1 to the corresponding source code in C, i.e. the C code of Figure 2. This fundamental difference introduces technical issues that do not necessarily show up in compiler testing, but also brings new opportunities. On one hand, executing the programs produced by a transpiler can be less straightforward than in the case of the binaries yielded by a compiler. For example, executing the C code produced by the transpiler of our example requires the test generator to first compile those C programs. This in turn may require the resolution of any deployment-environment-related runtime dependencies, including dependencies that may be outside the scope of the specific problem domain of the transpiler. Once the test generator succeeds in compiling the transpiled programs into binaries, it must then identify suitable inputs to execute those binaries. The compiler testing approaches surveyed above often rely on generating random inputs for this purpose. A technical difference is that, in transpiler testing, the test generator shall elicit inputs that fit the declarations in the output sources (the C code of our example) and the deploying environment, rather than address directly the input space of the original sources (the DSL programs of our example). On the other hand, as the output of a transpiler is source code written in a programming language, we see the opportunity to define oracles that predicate directly on the textual representation of the output code. In the solution proposed in this paper, we abandon the approach of evaluating the test cases by executing the binaries of the semantically-equivalent programs, and embrace the idea of defining metamorphic relations that specify how changes in the input sources are expected to map into observable differences in the output sources. We exemplify our approach with reference to the working example. A possible metamorphic relation of the type defined in our approach could be the following: Let 𝑃 ′ and 𝑃 ′′ be two test programs that differ only in the declaration of a single enumerative type 𝐸𝑁𝑈 𝑀, such that 𝐸𝑁𝑈 𝑀 is declared in both programs, but it includes an additional value identifier 𝑁 𝐸𝑊 𝑉 𝐴𝐿 in 𝑃 ′′ . Then, after transpiling 𝑃 ′ and 𝑃 ′′ , we expect that the output sources 𝑂 ′ and 𝑂 ′′ shall have no difference but in the declaration of the type that corresponds to 𝐸𝑁𝑈 𝑀, where we expect that 𝑂 ′′ reflects the value 𝑁 𝐸𝑊 𝑉 𝐴𝐿 of 𝑃 ′′ with the proper ordinal value, while 𝑂 ′ lacks it.
where 𝑁 𝐸𝑊 𝑉 𝐴𝐿 is the identifier of an additional value of the enumeration. Next, the test generator would run the transpiler on both test programs and compute the textual diff between the C code of the output sources, to crosscheck that the postcondition stated in the metamorphic relation correctly holds. With the buggy transpiler of our example, the textual diff would appear as follows: @@ t r a f f i c l i g h t . h ; −7 ,5 + 7 , 6 @@ t y p e d e f enum { Yellow = 0 , Green , Red , + NEWVAL, } trafficlight_state_t ;
At this point, the test generator would signal a failure report, noticing that the textual diff reflects the identifier 𝑁 𝐸𝑊 𝑉 𝐴𝐿 with a different ordinal value (3, as it is the fourth value of the enumeration) than the expected one (0, as it was added as the first element of the enumeration). In the next section, we define our technique for metamorphic testing of transpilers in detail.
3
The MCP Approach to Metamorphic Testing of Transpilers
Our approach to metamorphic testing of transpilers relies on metamorphic relations that specify how mutation-style changes in the input source code are expected to consistently map onto corresponding equivalencies and differences in the output source code produced by the transpiler under test. We call the property that underlies these metamorphic relations mutation consistency of the (transpiled) programs (MCP). In this section, we introduce both the concept of MCP metamorphic relations and the general workflow of a test generator exploiting such metamorphic relations, to which we refer as MCP-Tester.
3.1
MCP Metamorphic Relations
An MCP metamorphic relation can be defined as a pair ⟨mutation operator, consistency checkers⟩, capturing the metamorphic relation operationally. Namely, a mutation operator represents a punctual transformation that can be applied to an input source code, say 𝑖, to produce a follow-up source code, say 𝑖 ′ , such that 𝑖 ′ is equal to 𝑖 except for the change introduced by the transformation. Correspondingly, the consistency checkers specify a set of checks for validating code-level equivalences and code-level differences that we expect to hold between the output source code that the transpiler yields when executed against the source code 𝑖 and 𝑖 ′ , respectively, consistently with the change done according to the given mutation operator. MCP metamorphic testing leverages MCP metamorphic relations for steering the generation of test cases, while automatically assessing their outcomes through oracles instantiated from the metamorphic relations.
Metamorphic Testing of Transpilers via Mutation Consistency of Programs
Formally, an MCP metamorphic relation ⟨mutation operator, consistency checkers⟩ enables test oracles specifying that for each pair of input sources, ⟨𝑖, 𝑖 ′ ⟩, which relate with each other as they mutually differ only for the changes determined by the given mutation operator, i.e., 𝑖 ′ = mutation operator(𝑖), then the corresponding pair of output sources produced by the transpiler 𝑇 under test, ⟨𝑜 = 𝑇 (𝑖), 𝑜 ′ = 𝑇 (𝑖 ′ )⟩, must also be related with each other as they shall reflect the specific set of expected code-level equivalencies and differences, i.e., consistency checkers(𝑜, 𝑜 ′ ) must hold. For instance, the following MCP metamorphic relation captures the metamorphic relation that we informally introduced in Section 2.3, relatively to adding a new value in the enumeration of the program in Figure 1: MCP metamorphic relation: Augment Enumeration Mutation Operator: given an input source 𝑖 accepted by the transpiler under test, generate another input source 𝑖 ′ by adding a new value identifier 𝑁 𝐸𝑊 𝑉 𝐴𝐿 to an Enumeration entity that is part of an input source 𝑖. Consistency Checkers: (i) Equivalence Check: the output sources 𝑜 and 𝑜 ′ consist of exactly equal C code, except for the enum definition translating the mutated Enumeration. (ii) Difference Check: the output sources 𝑜 and 𝑜 ′ differ in that the enum definition translating the mutated Enumeration in 𝑜 ′ includes the value identifier 𝑁 𝐸𝑊 𝑉 𝐴𝐿, preceded (respectively, followed) by the value identifier that immediately precedes (respectively, follows) 𝑁 𝐸𝑊 𝑉 𝐴𝐿 in the mutated Enumeration in 𝑖 ′ , if any.
3.2
MCP Metamorphic Testing
Algorithm 1 outlines the general workflow of our technique for MCP metamorphic testing of transpilers, to which we refer to as MCP-Tester. MCP-Tester takes as input a transpiler under test, a set of MCP metamorphic relations valid for the given transpiler, a set of seed input sources accepted by the transpiler, and three hyper-parameters that control the testing budget allowance and its segmentation across the given metamorphic relations and seed inputs. MCPTester proceeds through three nested loops, which iterate through the seed input sources, the MCP metamorphic relations and the possible ways of applying mutations, respectively, as follows. In the first loop, lines 2-3, if further testing time is still available (parameter max_time), it selects one of the available seed input sources to play the role of base input source (the input source 𝑖 following the terminology of Section 3.1) throughout the metamorphic testing steps thereon. In the second loop, lines 4-6, if there is further testing time and up to a maximum number of attempts (parameter max_mr), MCPTester selects an MCP metamorphic relation (line 4) to steer the test generation process. It then inspects the base input source 𝑖 (line 6), searching for all possible locations where the mutation operator of the metamorphic relation can be actually applied. The found locations are called spots in the algorithm. For instance, if MCP-Tester is testing the sample (bugged) transpiler of our working example, assuming that in the first loop it
Algorithm 1: MCP-Tester’s workflow Input :𝑡𝑟𝑎𝑛𝑠𝑝𝑖𝑙𝑒𝑟 , the transpiler under test 𝑀𝑅, the set of MCP metamorphic relations 𝑆, a set of seed input sources 𝑚𝑎𝑥_𝑡𝑖𝑚𝑒, maximum testing budget 𝑚𝑎𝑥_𝑚𝑟 , num. of 𝑚𝑟 ∈ 𝑀𝑅 to consider, per input source 𝑚𝑎𝑥_𝑠𝑝𝑜𝑡𝑠, num. of applications of each 𝑚𝑟 ∈ 𝑀𝑅 Output : 𝐹 , the set of identified failures 𝐹 ← ∅; while CanContinue(max_time) do 3 𝑖 ←PickOne(S); 4 while CanContinue(max_time, max_mr) do 5 𝑚𝑟 ←PickOne(MR); 6 𝑆𝑃𝑂𝑇 𝑆 ←FindSpots(𝑚𝑟 , 𝑖); 7 while CanContinue(max_time, max_spots) do 8 𝑠𝑝𝑜𝑡 ←PickOne(SPOTS); 9 𝑖 ′ ←MutateProgram(𝑖, 𝑚𝑟 , spot); 10 𝑜 ←InvokeTranspiler(𝑡𝑟𝑎𝑛𝑠𝑝𝑖𝑙𝑒𝑟 , 𝑖); 11 𝑜 ′ ←InvokeTranspiler(𝑡𝑟𝑎𝑛𝑠𝑝𝑖𝑙𝑒𝑟 , 𝑖 ′ ); 12 if ¬VerifyOracles(𝑜, 𝑜 ′ , 𝑚𝑟 ) then 13 𝐹 ← 𝐹 ∪ ⟨𝑖 , 𝑖 ′ , 𝑚𝑟 ⟩; 14 end 15 end 16 end 17 end 18 return 𝐹 ; 1 2
selected the source program of Figure 1 as seed input source, and assuming that in the second loop it selected the MCP metamorphic relation exemplified in Section 3.1, then it will determine that, according to the mutation operator of the metamorphic relation at hand, the new value identifier 𝑁 𝐸𝑊 𝑉 𝐴𝐿 can be added at four possible spots (§ the set 𝑆𝑃𝑂𝑇 𝑆 at line 6 of Algorithm 1): (i) before the Red identifier, (ii) between the Red and Green identifiers, (iii) between the Green and Blue identifiers, or (iv) after the Blue identifier. Eventually, in the third loop, lines 7-14, if there is further testing time and up to a maximum number of attempts (parameter max_spots), MCP-Tester executes the actual metamorphic testing steps. It selects a possible spot (line 8), generates a follow up input source 𝑖 ′ by applying the mutation operator of the metamorphic relation at hand at the given spot (line 9), executes the transpiler against both the input sources 𝑖 and 𝑖 ′ (lines 10 and 11, respectively) and verifies the test oracles against the outcomes from the transpiler, 𝑜 and 𝑜 ′ , according to the consistency checkers of the metamorphic relation at hand (line 12). Finally, it collects all failing test cases (line 13) and reports them as output at the end (line 18). Continuing with the example, MCP-Tester iteratively applies the metamorphic testing steps by inserting the new value identifier NEWVAL at the various possible spots of the seed input source of Figure 1. Eventually, it might add the identifier 𝑁 𝐸𝑊 𝑉 𝐴𝐿 as the first value of the Enumeration in the program, and observe the failure of the
Papaleo et al.
test oracles, as the value Red that immediately follows 𝑁 𝐸𝑊 𝑉 𝐴𝐿 in the follow-up input source 𝑖 ′ does not appear in the expected location in the transpiled output 𝑜 ′ . Consequently, MCP-Tester would report a test failure. Algorithm 1 describes the general workflow of MCP-Tester, but several extensions and optimizations could be easily incorporated. The set of seed input sources could comprise existing input sources, but also input sources automatically generated, or a mix of existing and automatically generated input sources. It may even be incrementally augmented with the follow-up input sources generated at line 9, or with a selection of these follow-up input sources, chosen according to strategies aimed at identifying those more interesting for further mutations in subsequent testing steps. The strategies for iterating through the seed input sources, the metamorphic relations and the mutation spots (the PickOne functions in the algorithm) can also be customized, for example to avoid picking the same items multiple times. Finally, the translation of the source program (line 10) could also be made more efficient. In fact, it is entirely independent of the metamorphic relation and picked spot, and can therefore be executed immediately after the choice of the input source (line 3), while caching its result. Our current implementation1 works with automatically generated seed input sources, and iterates through seed input sources, metamorphic relations and mutation spots in random order, caching the corresponding selections to avoid re-picking, and resetting each cache upon exhausting all available options at a given selection point. Additionally our implementation already implements the optimization involving the early translation of the source program and its caching. We leave for future work to investigate possible strategies for feeding the follow-up input sources as additional seeded inputs.
4
Case Study and Empirical Evaluation
We devised and investigated the MCP approach in the context of a technology-transfer project joint with an industrial partner. As we already mentioned in Section 2, the industrial partner worked with a custom DSL for defining control logics of industrial plants, by using a state-machine-based formalism and according to a syntax that is pervasively inspired by the natural-language terminology of their engineers, in the fashion of the sample program of Figure 1. The industrial partner used a transpiler developed in Java in their laboratories to translate the DSL control logics into C programs. In this section we describe a case study, in which we specialized the MCP metamorphic testing approach to the transpiler developed by our industrial partner. For confidentiality reasons, we cannot disclose the identity of the partner, nor the technical details of their transpiler or their DSL. Thus, we present the MCP metamorphic relations discussed in the remainder of this section by referring to an anonymized version of the partner’s DSL, mimicking the DSL that we already introduced in the working example in Figure 1, while nonetheless keeping strict correspondence with the characteristics of the actual constructs of the partner’s DSL and the way we addressed those constructs in the actual setting of the case study.
4.1
Instantiating MCP Metamorphic Relations
In the case study we instantiated a set of MCP metamorphic relations for the DSL constructs that can appear in the declarations section of a control-logic definition. A subset of the constructs are exemplified in the declarations section of the sample program in Figure 1, while in Figure 3 we report the nine MCP metamorphic relations that we derived for the possible constructs, referring to the DSL grammar and the documented requirements of the transpiler. As explained in Section 3.1, each metamorphic relation consists of a mutation operator and a set of consistency checkers. In the figure, we documented the mutation operator for all nine metamorphic relations, while, due to the space limitations, we documented the consistency checkers only for the first metamorphic relation. The omitted consistency checkers are available for reference in the repository of our tool.2 In the figure, the titles of the metamorphic relations refer to the behavior of the corresponding mutation operators, each handling the injection of a construct that can be part of the declarations section of a control-logic definition. The first three metamorphic relations (Add configuration parameter, Add actuator command and Add enumeration-typed variable) correspond to the constructs actually exemplified in the sample program in Figure 1, whereas the other matamorphic relations refer to other types of variables or other types of entities (namely, input events and attributes) that can be also added. The mutation operators are documented in the first paragraph of the description of the metamorphic relations, including a template code snippets representing the grammatical structure of the construct to be added. For instance, for the first metamorphic relation, Add configuration parameter, the mutation operator augments the declarations section by adding (at a valid position of the section) a line that starts with the keyword configuration parameter, followed by a valid name (<NAME>) of the parameter, which must be of type Boolean. This produce a declaration that respects the same syntax as the one at Figure 1, line 5. The second paragraph of the metamorphic relation Add configuration parameter exemplifies the documention of the consistency checkers. Each checker can represent either an equivalence check or a difference check. The formers verify that given code areas are unchanged across the C sources yielded by the transpiler for the base and follow-up test cases, respectively, while the latter ones predicate on the differences found therein. For the metamorphic relation Add configuration parameter, we specified three checkers, documenting the expected equivalences and differences, with reference to the C code that the considered transpiler produces as output. The first consistency checker specify an equivalence check ensuring that the addition of a configuration parameter impacts only on the relevant files. This enables the detection of failures arising from faults in the transpiler for which the specification of a configuration parameter yields unwanted cascading effects on the other files of the translated source code. The second consistency checker verifies that the boolean parameter has been properly added to the struct in the proper header file. Similarly, the third consistency checker makes sure that the 1 https://doi.org/10.5281/zenodo.19340835.
2 https://doi.org/10.5281/zenodo.19340835.
Metamorphic Testing of Transpilers via Mutation Consistency of Programs
Add Configuration Parameter Mutation Operator: Add a new configuration parameter to the declarations section like: configuration parameter <NAME>: Boolean Consistency Checkers: • Equivalence Check: the output source consists of exactly equal code, except for the file hosting the configuration parameters for the affected module; • Difference Check: the struct containing the logic parameters in the module configuration file contains a new field named p_<NAME> of type _Bool; • Difference Check: the module configuration file contains a new getter get_<NAME> to access the parameter’s value.
Add Integer Variable with Counter Behavior Mutation Operator: Add a new variable to the declarations section like: variable <NAME>: Integer like counter Consistency Checkers: [Omitted for space reasons] Add Integer Variable with Timer Behavior Mutation Operator: Add a new variable to the declarations section like: variable <NAME>: Integer like timer(duration = <DURATION> seconds) Consistency Checkers: [Omitted for space reasons]
Add Actuator Command Add Boolean Variable Mutation Operator: Add a new actuator command to the declarations section like: actuator command <NAME>(argument arg: <TYPE>) Consistency Checkers: [Omitted for space reasons]
Mutation Operator: Add a new variable to the declarations section like: variable <NAME>: Boolean Consistency Checkers: [Omitted for space reasons] Add Input Event
Add Enumeration-Typed Variable Mutation Operator: Add a new variable to the declarations section like: variable <NAME>: <ENUMERATION-TYPE> Consistency Checkers: [Omitted for space reasons]
Mutation Operator: Add a new input event to the declarations section like: input event <NAME>(argument arg: <TYPE>) Consistency Checkers: [Omitted for space reasons] Add Attribute
Add Integer Variable with Default Value Mutation Operator: Add a new variable to the declarations section like: variable <NAME>: Integer like default(value = <VALUE>) Consistency Checkers: [Omitted for space reasons]
Mutation Operator: Add a new attribute to the declarations section like: attribute <NAME> = <VALUE> Consistency Checkers: [Omitted for space reasons]
Figure 3: The MCP metamorphic relations instantiated in our case study transpiler has added a proper getter in the produced C code. For instance, if the mutation operator of the metamorphic relation Add configuration parameter is applied to add a line as: c o n f i g u r a t i o n parameter MCPTEST_a26c5 : B o o l e a n
at the end of the declarations section in the sample code of Figure 1, then the difference checks verify that the diff-snippet between the C sources yielded by the transpiler for the base and follow-up test case, respectively, matches the following differences: @@ t r a f f i c l i g h t . h ; −4 ,3 + 4 , 4 @@ typedef struct { _Bool p_enableCamera ; + _ B o o l p_MCPTEST_a26c5 ; } trafficlight_t ; @@ t r a f f i c l i g h t . h ; − 1 4 , 0 + 1 4 , 1 @@ + _ B o o l get_MCPTEST_a26c5 ( t r a f f i c l i g h t _ t ∗ ) ; @@ t r a f f i c l i g h t . c ; − 2 3 , 0 + 2 3 , 3 @@ + _ B o o l get_MCPTEST_a26c5 ( t r a f f i c l i g h t _ t ∗ i n s t a n c e ) { + r e t u r n i n s t a n c e −>p_MCPTEST_a26c5 ; + }
Question RQ1 studies the effectiveness of our metamorphic testing approach for transpilers, which relies on the test cases and the test oracles induced by the MCP metamorphic relations. We addressed this research question by seeding faults into the transpiler and measuring how many of those faults MCP-Tester can reveal, as well as the portion of detected faults that were specifically exposed thanks to the metamorphic oracles. Question RQ2 corresponds to an ablation study in which we adapt MCP-Tester to work as a fuzzer that applies the same mutations used in MCP-Tester, but without exploiting knowledge of the underlying metamorphic relations and oracles. This eliminates the overhead that MCP-Tester pays for checking the metamorphic oracles, allowing the generator to spend the saved time on additional fuzzing. In this way, we assess whether the overhead introduced by MCP metamorphic testing is justified by the benefits it provides over pure fuzzing.
4.3 4.2
Research Questions
In the case study, we executed MCP-Tester against the partner’s transpiler, relying on the metamorphic relations defined above. Our investigation was driven by two main research questions: RQ1: Can MCP-Tester effectively detect faults? RQ2: Can MCP-Tester outperform a baseline that only fuzzes the transpiler?
Experimental Settings
We developed MCP-Tester in Java. It provides an API to customize the MCP metamorphic relations accessible both from Java and through YAML configuration files. The transpiler under test is specified in a YAML configuration file. The tool is open source and accessible anonymously for reviewers.3 3 https://doi.org/10.5281/zenodo.19340835.
Papaleo et al.
For the case study, we connected MCP-Tester to the transpiler under test, and configured it to handle the nine metamorphic relations of Figure 3. To increase the throughput of test cases per metamorphic relation, we exploited parallelism by distributing the nine relations across four parallel instances of the tool. Each instance was configured to handle a mutually disjoint subset of metamorphic relations (3, 2, 2 and 2 metamorphic relations, respectively). Accordingly, we set the max_mr parameter of each instance to 3, 2, 2, and 2, respectively, to ensure that every metamorphic relation is picked exactly once after selecting a seed input (§Algorithm 1, line 4). Furthermore, we set the max_spots parameter to 4, such that each metamorphic relation is applied at four different locations within each seed input (§Algorithm 1, line 3). We provided MCP-Tester with automatically generated seed inputs, relying on the tool rmutt.js [26] to generate control logic sources based on the grammar of the partner’s DSL. We executed rmutt.js for 6 hours, retaining only the sources that the transpiler accepted as valid control logics. We ended up with 297 valid sources. A testing session consisted in executing the four parallel instances of MCP-Tester configured as above against the partner’s transpiler with a time budget (max_time) of 12 hours. To account for the random choices that underlie the MCP-Tester algorithm, we executed 10 repetitions of the 12 hour testing session, configured as above. During each 12-hour testing session, the seed inputs were first randomly shuffled and then processed in the same order by all four parallel instances of MCP-Tester. Each testing session has been executed on a cloud resource running Windows Server 2025 Standard 24H2 build 26100.6584, equipped with 12 processors, 32 GB of RAM and a 100 GB SSD disk. To answer RQ1, we evaluated the effectiveness of MCP-Tester in detecting faults by systematically injecting seeded faults into the transpiler’s bytecode. To this end, we developed a custom bytecode instrumentor that targets the opcodes implementing conditional branches (ifeq, ifne, ifnonnull, ifnull) in the core code of the transpiler (i.e., excluding the external libraries like ANTLR). Each injected fault forces the branch condition to evaluate deterministically to either true or false. While this strategy enables the generation of a large number of seeded faults, it also suffers of inefficiencies that may negatively impact the evaluation of MCP-Tester. On one hand, many injected faults may lead to trivial crashes of the transpiler. Such trivial crashes are both scarcely representative of realistic faults that a developer would introduce and can be straightforwardly revealed even without any specific test oracle. On the other hand, executing every generated test case against every faulty version of the transpiler would be prohibitively expensive, also because of the large amount of test cases that MCP-Tester generates in the 12 hour testing sessions. To mitigate such issues, we heuristically refined the fault injection strategy as follows. First, in a preparatory experiment, we executed the transpiler against a sample DSL control logic (provided by our partner), considering every injected fault in a separate execution. We discarded all injected faults that caused the transpiler to crash in the preparatory experiment. Moreover, we exploited a heuristic fault-selection strategy that does not impact the number of test executions done by MCP-Tester, while maximizing the number of injected faults evaluated at least
once in a testing session: It aims at maintaining that, in a testing session, we evaluate each test case against a distinct injected fault. In detail, the intrumentor incorporates all injected faults at once (but the ones discarded in the preparatory experiments). At runtime, the instrumentation activates only the first executable injected fault that has not been executed in any previous test cases (if any). An injected fault is executable when the execution traverses the corresponding branch. This approach ensures broad consideration of potentially interesting faults while controlling the number of tested faulty variants of the transpiler. MCP-Tester evaluates the injected faults only while executing follow-up test cases, recording which injected faults crashed the transpiler or were detected by the metamorphic oracles. Whereas the crashes indicate faults that MCP-Tester would necessarily detect, failed oracles may be spurious alarms, as the base test case considered the non-faulty transpiler. Thus, after completing each testing session, we performed a confirmation step: for each injected fault that MCP-Tester detected via a metamorphic oracle, we re-executed the pair of test cases that revealed the fault, to confirm if the fault gets indeed revealed according to the metamorphic relations comprised by those test cases. We removed the unconfirmed outcomes. We remark tat this confirmation step is specific to our experimental protocol and is not part of the MCP-Tester technique itself; therefore, we execute the confirmation step offline with respect to the 12-hour time budget used in the main experiments. For RQ2, we turned MCP-Tester into a corresponding fuzzer version, which exploits the same mutation operators and iteratively executes mutated test cases, although ignoring the underlying metamorphic relations and oracles. As with common fuzzers, there are no oracles, and it reports failures upon observing the transpiler crashes. We used the same configuration of the 12-hour testing sessions as above, seeding the same injected faults with the same heuristic strategy. Moreover, for each testing session executed with MCP-Tester, we recorded the seed of the random number generator used in the tool, and executed a corresponding fuzzing session which uses exactly the same seed. In this way, the fuzzing session shuffles the input sources in the same order as the corresponding MCP-Tester session, as well as it applies the mutation operators in the same order and at the same code locations. This improves the comparability of the results.
4.4
Results
Figure 4 reports the results of our experiments, for the 10 testing sessions of 12 hours executed with both MCP-Tester and the fuzzer version of the tool. Each pair of rows in the table reports on a testing session, siding (first and second row or each pair or rows) the results from the execution (with same seed of the random number generator) with either MCP-Tester or the fuzzer version, respectively. For each testing session, we report the number of runs of the transpiler (column Transpiler runs), that is, the number of executed test cases, the number of unique injected faults evaluated at least once (column Injected faults), as well as the number of exposed crashes (column Crashes) and confirmed oracle failures (column Oracle failures). We observe that the number of unique injected faults is generally lower than the number of transpiler runs. This is due to two primary
Metamorphic Testing of Transpilers via Mutation Consistency of Programs
Testing session 1 Testing session 2 Testing session 3 Testing session 4 Testing session 5 Testing session 6 Testing session 7 Testing session 8 Testing session 9 Testing session 10
Transpiler runs
Injected faults
Crashes
Oracle failures
MCP-Tester
1952
689
13
33
Fuzzer
1952
692
13
-
MCP-Tester
2116
714
15
40
Fuzzer
2128
714
13
-
MCP-Tester
352
134
5
6
Fuzzer
360
133
5
-
MCP-Tester
864
324
5
20
Fuzzer
864
324
5
-
MCP-Tester
2206
707
12
28
Fuzzer
2194
707
12
-
MCP-Tester
2074
714
16
40
Fuzzer
2086
714
14
-
MCP-Tester
2128
714
12
39
Fuzzer
2120
714
12
-
MCP-Tester
2110
714
14
40
Fuzzer
2110
714
14
-
MCP-Tester
1816
688
12
27
Fuzzer
1816
693
13
-
MCP-Tester
2128
714
14
43
Fuzzer
2128
714
14
-
Figure 4: Results of MCP-Tester and the Fuzzer version per testing session causes. First, some transpiler runs may not traverse any yet-nonexecuted fault. Second, since transpiler runs are done in parallel, synchronization delays may lead them to activate the same injected fault. Furthermore, no data on oracle failures is available for the fuzzer version of the tool, as the fuzzer does not check oracles. The results in the table provide empirical evidence of the merit of MCP metamorphic testing, as the number of injected faults detected thanks to the metamorphic-testing oracles is significant, specially when compared to the number of faults that cause crashes. The failures reported based on the oracles correspond to cases in which the transpiler fails silently, as it indeed produces the output sources, even though the code in those sources is an incorrect translation of the code in the input sources. As a matter of facts, those failures cannot be detected with pure fuzzing: as the transpiler does not crash in those cases, a fuzzer can only observe that the output was indeed produced, being it agnostic about the actual code in those sources. Answering RQ1, we observe that in the testing sessions with MCP-Tester the number of failures detected with metamorphic oracles ranges between 6 and 43, with a median value of 39, whereas the number of crashes ranges between 5 and 16, with a median value of 13.
The testing sessions with the fuzzer version of the tool, where we ablated the use of the metamorphic oracles, show that the overhead of MCP-Tester over a pure fuzzer (based on the same mutation operators) is negligible. In several cases the fuzzer version ends up with executing the same amount of transpiler runs as MCP-Tester, or just a few more. This can be explained because the application of the mutations and the execution of the transpiler dominate the execution time of the tool, while the evaluation of the oracles takes little time. If we consider the first testing session, the fuzzer version of our tool executed 1,952 transpiler runs, running four tool instances for 12 hours, which on average corresponds to generating and executing a test case every 89 seconds4 , whereas the evaluation of the oracles takes hundredths of seconds. The comparison between the number of injected faults evaluated in the testing sessions with MCP-Tester and the corresponding testing sessions with the fuzzer version, respectively, and the comparison of the number of crashes observed in those testing sessions, respectively, confirm that there is no significant gain in dismissing the metamorphic oracles, conversely it pays the penalty of not exposing many failures. In summary, answering RQ1, our case study provides empirical evidence that the MCP approach to metamorphic testing of 4With some particular DSL source inputs the transpiler may take even more than 10
minutes, as it happened for example in testing session 3.
Papaleo et al.
transpilers is effective, as it is able to expose significant amounts of injected faults. Answering RQ2, we observed negligible overhead between using or not using the metamorphic oracles, as we can execute almost equal amounts of transpiler runs and reveal comparable amounts of crashes in both cases.
4.5
Threats to Validity
We could not compare MCP-Tester with baselines derived from existing fuzzers and existing approaches to metamorphic testing of compilers (such has the ones mentioned in section 2.2). Common fuzzers, like AFL, would generally fail in generating valid sources, while all existing approaches for metamorphic testing of compilers rely on executing the compiled programs, which is inapplicable in our case (as we discussed). We mitigated this issue by creating a baseline that adapts MCP-Tester to work as a fuzzer, applying the same mutations used in MCP-Tester. We do not claim that our results generalize beyond the specific transpiler and metamorphic relations considered in our experiment. As future work we plan to increase the number and diversity of metamorphic relations considerd for the partner’s DSL, evaluate the effectiveness of our technique for other transpilers. A threat to the validity of the results obtained in our experiment may come from the injected seeded faults that are only a subset of all possibly representative faults. While this is true, we affirm that our subset of injected faults is significant enough to show the benefits of our technique. Internal validity may be threatened by the high number of random choices in our technique (spot selection, DSL selection) and empirical setup (fault seeding) in combination with a limited time budget. To mitigate this issue, we repeated our experiments 10 times demonstrating that our technique is consistently able to identify potential faults.
5
Related Work
In Section 2, we have already commented on why existing approaches to metamorphic testing of compilers [5, 11, 12, 19, 27] can hardly be directly applied for testing transpilers, such as the one used in our case study. Those approaches assume that the compiler under test produces executable programs for which inputs are readily available or easily derived, and their metamorphic relations assert that specific changes in the source programs should preserve the runtime behavior of the compiled programs. In contrast, transpilers yield results that are themselves source code, which in many practical cases may depend on non trivial compilation and execution requirements. In our case study, the transpiler generates control-logic programs that can run only when integrated with dedicated hardware or specialized simulators, and whose inputs depend on the structure and dynamics of the controlled plant. Under these conditions, the above-mentioned techniques for metamorphic testing of compilers are inapplicable. Our approach overcomes this limitation by defining metamorphic relations directly over the source code produced by the transpiler. Other compiler-testing techniques rely on pure fuzzing [16, 23, 29–31], which is limited to revealing only observable crashes (as our case study confirms), or exploit differential testing [6, 13, 20, 24], which is applicable only when multiple, supposedly equivalent
implementations of the transpiler exist, a condition that is seldom met in practice and not satisfied in our setting. Our approach defines metamorphic relations that ground on mutating the input source code in the style of classic mutation analysis of programs [9, 15]. Program mutations have been used by other authors in the literature on metamorphic testing. For instance, some approaches automatically learn which mutants enable predefined metamorphic relations [18] or instantiate metamorphic relations by inferring how results change when applying given mutants [25]. Some authors relied on mutation testing to evaluate the effectiveness of metamorphic relations [1, 14, 17, 21, 22]. All these research efforts address different problems than transpiler testing.
6
Conclusions
Motivated by the industrial relevance of transpilers, this paper introduced a metamorphic testing technique, MCP-Tester, specifically designed to address automated testing of transpilers. MCP-Tester distinctively predicates metamorphic oracles over the source code that the transpiler under test produces as output, whereas existing approaches to metamorphic testing of compilers draw on executing the compiled binary. Our approach leverages on program mutations applied to the input sources provided to the transpiler. It grounds on metamorphic relations which specify how the mutations in the input sources shall consistently map onto corresponding equivalencies and differences in the output sources produced by the transpiler under test. We studied the effectiveness of MCP-Tester for testing the transpiler of an industrial partner, and used injected faults to assess the ability of MCP-Tester to reveal faults. The empirical results reported in the paper confirm that the proposed approach has the potential to reveal faults that would remain unexposed with pure fuzzing. We are currently working to extend our case study with metamorphic relations for all the constructs of the DSL of our partner, and to experiment the MCP testing approach to other transpilers. Data Availability Statement: Our tool MCP-Tester is open source and accessible anonymously on Zenodo at https://doi.org/ 10.5281/zenodo.19340835. The same repository also includes the integral version of the metamorphic relations of Figure 3. For confidentiality reasons, we cannot disclose other data of the case study discussed in the paper, but the anonymized information in Section 4.
References [1] Jon Ayerdi, Valerio Terragni, Gunel Jahangirova, Aitor Arrieta, and Paolo Tonella. 2024. Genmorph: Automatically generating metamorphic relations via genetic programming. IEEE Transactions on Software Engineering 50, 7 (2024), 1888–1900. [2] Andrés Bastidas Fuertes, María Pérez, and Jaime Meza Hormaza. 2023. Transpilers: A Systematic Mapping Review of Their Usage in Research and Industry. Applied Sciences 13, 6 (2023). doi:10.3390/app13063667 [3] Patryk Chaber and Maciej Ławryńczuk. 2016. Effectiveness of PID and DMC control algorithms automatic code generation for microcontrollers: Application to a thermal process. In 2016 3rd Conference on Control and Fault-Tolerant Systems (SysTol). IEEE, 618–623. [4] Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A survey of compiler testing. Acm Computing Surveys (Csur) 53, 1 (2020), 1–36. [5] Alastair F Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. 2017. Automated testing of graphics shader compilers. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 1–29.
Metamorphic Testing of Transpilers via Mutation Consistency of Programs
[6] Chris Hawblitzel, Shuvendu K Lahiri, Kshama Pawar, Hammad Hashmi, Sedar Gokbulut, Lakshan Fernando, Dave Detlefs, and Scott Wadsworth. 2013. Will you still compile me tomorrow? static cross-version compiler validation. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. 191–201. [7] Matthias Hirzel and Herbert Klaeren. 2016. Code coverage for any kind of test in any kind of transcompiled cross-platform applications. In Proceedings of the 2nd International Workshop on User Interface Test Automation. 1–10. [8] Paul Hudak. 1997. Domain-specific languages. Handbook of programming languages 3, 39-60 (1997), 21. [9] René Just. 2014. The Major mutation framework: Efficient and scalable mutation analysis for Java. In Proceedings of the 2014 international symposium on software testing and analysis. 433–436. [10] Tomaž Kosar, Sudev Bohra, and Marjan Mernik. 2016. Domain-specific languages: A systematic mapping study. Information and Software Technology 71 (2016), 77–91. [11] Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (Edinburgh, United Kingdom) (PLDI ’14). Association for Computing Machinery, New York, NY, USA, 216–226. doi:10.1145/2594291.2594334 [12] Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding deep compiler bugs via guided stochastic program mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (Pittsburgh, PA, USA) (OOPSLA 2015). Association for Computing Machinery, New York, NY, USA, 386–399. doi:10.1145/2814270.2814319 [13] Shaohua Li and Zhendong Su. 2023. Finding unstable code via compiler-driven differential testing. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 238–251. [14] Huai Liu, Xuan Liu, and Tsong Yueh Chen. 2012. A new method for constructing metamorphic relations. In 2012 12th international conference on quality software. IEEE, 59–68. [15] Yu-Seung Ma, Jeff Offutt, and Yong-Rae Kwon. 2006. MuJava: a mutation system for Java. In Proceedings of the 28th international conference on Software engineering. 827–830. [16] Michaël Marcozzi, Qiyi Tang, Alastair F Donaldson, and Cristian Cadar. 2019. Compiler fuzzing: How much does it matter? Proceedings of the ACM on Programming Languages 3, OOPSLA (2019), 1–29. [17] Johannes Mayer and Ralph Guderlei. 2006. An empirical study on the selection of good metamorphic relations. In 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Vol. 1. IEEE, 475–484. [18] Aravind Nair, Karl Meinke, and Sigrid Eldh. 2019. Leveraging mutants for automatic prediction of metamorphic relations using machine learning. In Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation (Tallinn, Estonia) (MaLTeSQuE 2019). Association for Computing Machinery, New York, NY, USA, 1–6. doi:10.1145/3340482.3342741 [19] Kazuhiro Nakamura and Nagisa Ishiura. 2016. Random testing of C compilers based on test program generation by equivalence transformation. In 2016 IEEE Asia pacific conference on circuits and systems (APCCAS). IEEE, 676–679. [20] Georg Ofenbeck, Tiark Rompf, and Markus Püschel. 2016. RandIR: differential testing for embedded compilers. In Proceedings of the 2016 7th ACM SIGPLAN Symposium on Scala. 21–30. [21] Md Shaik Sadi, Fei-Ching Kuo, Joshua WK Ho, Michael A Charleston, and Tsong Yueh Chen. 2011. Verification of phylogenetic inference programs using metamorphic testing. Journal of Bioinformatics and Computational Biology 9, 06 (2011), 729–747. [22] Prashanta Saha and Upulee Kanewala. 2019. Fault detection effectiveness of metamorphic relations developed for testing supervised classifiers. In 2019 IEEE International conference on artificial intelligence testing (AITest). IEEE, 157–164. [23] Emin Gün Sirer and Brian N Bershad. 1999. Using production grammars in software testing. ACM SIGPLAN Notices 35, 1 (1999), 1–13. [24] Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding and analyzing compiler warning defects. In Proceedings of the 38th International Conference on Software Engineering. 203–213. [25] Chang-ai Sun, Yiqiang Liu, Zuoyi Wang, and Wing Kwong Chan. 2016. 𝜇 MT: a data mutation directed metamorphic relation acquisition methodology. In Proceedings of the 1st International Workshop on Metamorphic Testing. 12–18. [26] tachuris. 2024. tachuris/rmutt.js. https://github.com/tachuris/rmutt.js originaldate: 2015-05-14T06:54:15Z. [27] Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique. In 2010 Asia Pacific Software Engineering Conference. 270–279. doi:10.1109/APSEC.2010.39 [28] Andrzej Wąsowski and Thorsten Berger. 2023. Domain-Specific Languages. Springer. [29] Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation. 283–294.
[30] Takahide Yoshikawa, Kouya Shimura, and Toshihiro Ozawa. 2003. Random program generator for Java JIT compiler test system. In Third International Conference on Quality Software, 2003. Proceedings. IEEE, 20–23. [31] Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal program enumeration for rigorous compiler testing. In Proceedings of the 38th ACM SIGPLAN conference on programming language design and implementation. 347–361.