J OURNAL OF I NFORMATION S CIENCE AND E NGINEERING 41, 43-60 (2025) DOI: 10.6688/JISE.202501 41(1).0003
Input Relation Prompting for Metamorphic Testing on Query-Based Systems E NG -S HEN T U AND S HIN -J IE L EE
arXiv:2607.25603v1 [cs.SE] 28 Jul 2026
Department of Computer Science and Information Engineering National Cheng Kung University Tainan, Taiwan [email protected]; [email protected]
Testing query-based systems (QBSs) presents significant challenges due to the absence of ground truth for validation and the extensive time and effort required for manual testing. This paper addresses these challenges by proposing an approach that assists testers in identifying metamorphic relations (MRs) for metamorphic testing (MT) instead of solely and exhaustively relying on prerequisite domain knowledge. MT is an approach rising in popularity employed to alleviate the oracle problem by applying input transformation rules (MRs) to a program. The proposed approach helps the tester by prompting MRs that describe the relationships between inputs and outputs, enabling fault detection when the expected relationship is not met. Unlike traditional testing approaches, this approach does not rely on pre-defined test cases or concrete ground truth, making it suitable for the testing of real-world QBSs. Furthermore, the proposed approach can be combined with other testing methods such as combinatorial testing and fuzz testing, expanding the possibilities for QBS testing. A conducted case study of a real-world web application demonstrates the applicability and potential of the proposed approach. Overall, this research contributes to advancing the field of metamorphic testing and provides a valuable tool for QBS testers to enhance their testing efficiency. Keywords: metamorphic testing, metamorphic relation, software, query-based system, testing
1.
Introduction
Query-based systems (QBSs) are software applications that rely on query processing sub-systems to allow users to efficiently retrieve information from a database. These systems can range from simple search engines to complex business intelligence systems. The testing of these QBSs involves verifying that the system correctly processes user queries and returns the expected results. When testing QBSs, due to the immense volume of data and the complexity of QBSs, there are a number of challenges that arise when testing a given QBS. The two main challenges we would like to draw attention to are: • In QBSs, the correct output of the queries is either unknown or hard to compare with the observed output; therefore, it can be assumed that there is a lack of ground Received August 10, 2023; revised November 15, 2023; accepted December 20, 2023.
1
2
E NG -S HEN T U , S HIN -J IE L EE
truth for most QBSs that require testing.[1][2] The ground truth mentioned here refers to a correct version of the software under test to refer to for comparison when testing for errors. The absence of ground truth would lead to the challenge of indistinguishable correct and incorrect behavior regarding the output of the QBS under test, which is known as the Oracle Problem[3] in software testing. • When given a QBS to test, manual testing and checking for errors or anomalies require an extensive amount of time and effort due to the vast volume of data.
Fig. 1. Metamorphic Testing Flowchart.
Dealing with the first challenge, metamorphic testing (MT)[4], a software testing technique rising in popularity, is known to effectively alleviate the oracle problem[5, 6] in multiple cases, including database systems[7]. MT focuses on identifying faults in software by applying input transformation rules or metamorphic relations (MRs) to a program. It is based on the ideology that if two inputs are related by a known transformation, then the output of the program should also be related predictably, see Fig.1. In a query-based system like a search engine, an example of metamorphic testing would be to alter a search query slightly and then compare the results. For instance, if the original query is ”all universities in Taiwan”, a metamorphic test might involve changing the query to ”all universities in Asia” and then checking whether the search results of the first query are included in the second query since all universities in Taiwan are also universities in Asia. This tests the search algorithm’s ability to recognize and correctly handle hierarchical or inclusive relationships between different sets of data. The goal of MT is to test the functionality of a software program without relying on a set of pre-defined test cases, which is well-suited for testing QBSs since not all QBSs have a wel. Instead, the tester defines a set of metamorphic relations that describe the relationships between different inputs and outputs. If the output does not match the expected relationship with the input, a fault is detected. However, to effectively apply the use of MT[4] in the testing of QBSs, one of the main challenges lies in MR identification[8]. The identification of MRs usually exhaustively relies on domain knowledge, which in most cases, might not be available without additional resources, especially when the tester is not well-versed with domain knowledge on the subject of the data.
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
3
Fig. 2. The Proposed Approach.
Not all QBSs have well-defined or explicit specifications readily available when tested; therefore, this paper proposes an approach that supports real-world scenarios where an not all QBSs under test have ground truth, detailed specifications, or MRs available. The proposed approach will assist the process of identifying and defining MRs through prompting based on metamorphic relation patterns (MRPs) in the QBS and performing testing. The proposed approach relies less on prerequisite domain knowledge and does not require ground truth, making the approach more accessible and user-friendly to testers. Furthermore, the proposed approach can even be applied to and or combined with other testing methods, such as combinatorial testing (CT) and fuzz testing (FT). This increases the value of the proposed approach in terms of performing a more comprehensive test for errors on the given QBS. The organization of this paper is as follows. The MRPs will be outlined in Section 2 Metamorphic Relation Patterns; details of the proposed approach will be described in Section 3 Methodology; and related work will be addressed in Section 6 Related Work. The experiments of the proposed approach will be demonstrated via two case studies in Section 4 Experiments. The limitations and future work of the proposed approach will be discussed in Section 5 Discussion and the conclusion will be drawn in Section 7 Conclusion.
2.
Metamorphic Relation Patterns
There are seven MRPs that will be frequently used and mentioned in this paper. These MRPs were introduced in detail by Segura et al.[1] in 2019, and detailed examples for these MRPs can be found in both the original work[1] and subsequent research[9]. Here we provide a brief introduction for these seven MRPs, please note that the relational algebra notations used below are defined and specified in [1]: • Input Equivalence: This pattern represents relations where source and followup test cases have equivalent inputs, resulting in the same output items in the
4
E NG -S HEN T U , S HIN -J IE L EE
same order. As mentioned in previous work [1], let q1 = T (σ (R, c1 )) and q2 = T (σ (R, c2 ), ⟨o2 ⟩) be two queries defined under the same relation R. Input equivalence states that the outputs of q1 and q2 should be the same when conditions are equivalent, c1 ≡ c2 , and ordering sequence is equivalent, o1 ≡ o2 , i.e., (c1 ≡ c2 ∧ o1 ≡ o2 ) ⇔ (T (σ (R, c1 ), ⟨o1 ⟩) = T (σ (R, c2 ), ⟨o2 ⟩)) • Shuffling: This pattern represents relations where the source and follow-up outputs contain the same items regardless of the ordering criteria specified as input. As mentioned in previous work [1], let q = σ (R, c) be a query such that its result contains several attributes {ai }t=1,n that can be used as ordering criteria with the T operator. Shuffling states that the result of the query q ordered by a given attribute ai should contain the same elements as the same query ordered by any other attribute a j , i.e., ∀i, j :1 . . . n|i ̸= j• itemsT (q, {ai ⟩) = itemsT (q, ⟨a j ⟩) • Conjunctive Conditions: This pattern involves refining a query with additional conjunctive conditions, where the results of each test case should be included in the previous ones. As mentioned in previous work [1], let c be a complex condition formed by the conjunction of simpler conditions, i.e., c = ∧t=1...n ci . Conjunctive conditions states that the result of a query after one condition is added conjunctively to its selection condition should be a subset of the original query, i.e., ^
∀i = 2 · ·n • σ (R,
. . . ck ) ⊆ σ (R,
k=1··i
∧
k=1...i−1
ck )
• Disjunctive Conditions: This pattern involves expanding a query with disjunctive conditions, where the results of each test case should be a subset of the following ones. As mentioned in previous work [1], let c be a complex condition formed by the disjunction of simpler conditions, i.e., c = Vi=1...n ci . Disjunctive conditions states that the result of a query when one condition is added disjunctively to its selection condition should contain the result of the original query, i.e., ∀i = 2 . . . n • σ (R,
_ k=1...i−1
ck ) ⊆ σ (R,
_
ck )
k=1...i
• Disjoint Partitions: This pattern represents relations where the outputs of followup test cases should have no items in common because the queried relation can be partitioned based on input attribute values. As mentioned in previous work [1], let q = σ (R, c) be a query such that its result contains at least one attribute a p whose domain is a discrete set of values, e.g., {v1 , v2 , . . . vn }. Disjoint partitions states that, when a conjunctive condition of the form a p = vi appears, the result of two queries should be disjoint if the values with which a p is compared are different, i.e., ∀i, j : 1 . . . n|i ̸= j• σ (R, c ∧ a p = vi ) ∩ σ (R, c ∧ a p = v j ) = ⊘
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
5
• Complete Partitions: This pattern represents relations where the union of followup outputs should contain the same items as the source output because the queried relation can be partitioned based on input attribute values. As mentioned in previous work [1], let q = σ (R, c) be a query such that its result contains at least one attribute a p whose domain is a discrete set of values, e.g., {v1 , v2 , . . . vn }. Complete partitions states that the result of q should be equal to the union of the results of the n queries formed by a conjunctive condition of the form a p = vi to the condition in q for each possible value of a p , i.e., σ (R, c) =
[
.σ (R, c ∧ a p = vi )
i=1.n
• Partition Difference: This pattern represents relations where the outputs of followup test cases are pairwise disjoint, and their union contains the same items as the source output because the queried relation can be partitioned based on input attribute values. As mentioned in previous work [1], let q = σ (R, c) be a query such that its result contains at least one attribute a p whose domain is a discrete set of values, e.g., {v1 , v2 , . . . vn }. Partition difference states that the difference between q and the union of the results of the k queries formed by adding a conjunctive condition of the form a p = vi to the condition in q for k different possible values of a p , should be equal to the result of the union of the results of the n − k queries formed by adding a conjunctive condition of the form a p = vi to the condition in q for the n − k different possible values of a p , i.e., ∀k : 1 . . . (n−1)• σ (R, c)−
[
.σ (R, c ∧ a p = vi ) =
i=1.k
[
. . . σ (Rn, c ∧ a p = v j )
j=k+1
3. 3.1
Methodology
Overview
Our proposed approach is focused on helping the tester discover and identify input relations by prompting them with metamorphic relation patterns (MRPs)[1] detected within the QBS to perform metamorphic testing and check for errors or anomalies in a given QBS. Based on real-world scenarios for QBS testing, with a given QBS, there is usually no ground truth, hence the need for testing. In this view, when faced with the task of testing the given QBS through querying, testers encounter the Oracle problem[3] and do not have correct results for validation. This implies that with a given set of initial test cases, the queried results are a mix of correct or incorrect results. The proposed approach theorizes that if at least some portion of the queries yields accurate results, it should be possible for the tester to discover input relations when prompted with MRPs[1] that can be detected between the results of the given test cases.
6
E NG -S HEN T U , S HIN -J IE L EE
This works even if only some of the results are correct since MRs should hold true consistently within the QBS, so if there lies a logical explanation, it is indeed possible to identify these MRs when prompted with partially correct results. For the proposed approach, the MRPs for QBSs[1] previously mentioned in Sec. 2 prompt the tester to identify MRs within the QBS under test.
Fig. 3. Operational Concept of the Proposed Approach.
3.2
The Proposed Approach
Given a QBS to test and an n number of test cases, later described in Section 2.2.1, the QBS is queried to yield n initial results, keeping in mind that the initial N results could contain both correct and incorrect results. Next, MRP detection is performed for the generated results which then creates a comparison table, which specifies MRP relations detected between the n results, the process of comparison table generation is shown in Section 3.2.2. With the comparison table, the tester can then observe and identify input relations for MT and potentially find errors and or anomalies in the QBS. The overview of the proposed approach can also be seen in Fig. 2. In the following subsections, a more detailed description as well as a running example of the proposed approach will be given in accordance with Fig. 3, which contains a detailed visualization of the operational concept of the proposed approach when given a QBS to test and an initial set of test cases. 3.2.1
Obtaining Initial Test Case Results
In Fig. 3, f denotes the function of querying the QBS under test, I denotes the initial test cases input, O denotes the queried output of I, n denotes the total number of initial test cases, and the following equation, Eq. (1), shows the input and output relation function
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
7
of querying the QBS under test: f (Ii ) = Oi
(1)
where Ii denotes an input test case and Oi denotes the corresponding output test case. Referencing Stage 1 in Fig. 3, given the initial test cases I as input, the QBS under test is queried with the initial test cases, and the results O are saved for comparison in the next step. For the initial test cases, the proposed approach shows high flexibility and compatibility, which implies that the proposed approach can work well with numerous types of test cases generated through other testing methods. However, it is still worth noting that the better designed the initial test cases are, the easier it will be for the tester to identify input relations. In this context, test cases targeted toward singular or pairwise parameters typically yield more obvious prompts to the tester compared to test cases that cover a complex combination of multiple parameters. For example, given the following hypothetical QBS and initial test cases, obtaining the initial test case results would yield: • QBS Under Test: An Animal Image Search Engine • Initial Test Case Queries and their Outputs: – Cats: All cat images in the system. – Funny Birds: All funny bird images in the system. – Dogs: All dog images in the system. – Pandas: All panda images in the system. – Funny Dogs: All funny dog images in the system. – Funny Cats: All funny cat images in the system. – Birds: All bird images in the system. – Cute Dogs: All cute dog images in the system. We will be following this running example for each step. 3.2.2
MRP Detection
With O from Sec. 3.2.1, the comparison table is then generated by performing MRP Detection. Referencing Stage 2 in Fig. 3, MRP Detection is conducted by making comparisons between all pairwise combinations of O, and the results of these comparisons will be held in the generated comparison table, see Fig. 4 for visualization of the process. As for the details of comparison table generation, the pseudocode for the process is provided, see Algorithm 1, in which the comparison table is returned for the next stage in the proposed approach. In the implementation of the proposed approach, the comparison table is created in the form of a data frame and saved to an Excel file for observation. Following the running example, while testing for the MRP conjunctive condition in Section.2, we should be able to build the comparison table through comparing the results. From the generated comparison table Table.1, we can see that the output of the test cases Funny Birds, Funny Dogs, Funny Cats, and Cute Dogs are subsets of the queries Birds, Dogs, Cats, and Dogs respectively. Please note that this process will be done for all MRPs in Section.2.
8
E NG -S HEN T U , S HIN -J IE L EE
Fig. 4. Visualization of Comparison Table Generation.
Algorithm 1 Comparison Table Generation results ← table of results of initial test cases comparisonTable ← 0/ for all queryResulta ∈ results do for all queryResultb ∈ results do if MRP detected in queryResulta & queryResultb then a ← index of queryResulta in results b ← index of queryResultb in results add comparisonTable[a] to results[b] end if end for end for return comparisonTable
3.2.3
Human Observation
With the generated comparison table, the tester can observe the MRP relations between the outputs in O, map them back to their respective inputs in I, and identify potential input relations, which serve as the key to performing MT. Referencing Stage 3 in Fig. 3, prompted by the comparison table generated from Sec. 3.2.2, if there exists an intuitive logical explanation behind the input relation identified, then it is likely that an MR can be defined for the QBS. In this stage, even though the tester does not need to rely on prerequisite domain knowledge extensively to identify these input relations of the QBS, the tester could be required to make assumptions about what the input relations could be. However, with the proposed approach, we claim that being prompted by MRP detection and the generated comparison table makes this task significantly easier. Also, since the comparison table is stored in the form of a data frame so it is also possible for the tester to manipulate the
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
Queried Results (O) Cats Funny Birds Dogs Pandas Funny Dogs Funny Cats Birds Cute Dogs
9
Conjunctive Conditions Funny Cats None Funny Dogs, Cute Dogs None None None Funny Birds None
Table 1. Comparison table of running example.
generated comparison table to aid the process of input relation identification. In the running example, from human observation, we can intuitively infer that this could be because all funny birds are birds, all funny dogs are dogs... and so on. Therefore, through this process, we are able to assume and identify an input relation, that is, the results of the query with keyword f unny followed by any given animal A should be a subset of the results of the query A. Next, we will then be able to perform MT with this input relation on all animals in the QBS. 3.2.4
Testing for Errors
Once MRs have been identified and defined, these MRs can then be used to perform MT[4] on the QBS. Referencing Stage 4 in Fig. 3, for each MR, for all inputs Ii and I j that fit the input relation, their corresponding outputs Oi and O j are checked to see if they fulfill the output relations defined by the MR. If they do not fulfill the defined relation, then the test case fails and an error will have been detected within the QBS under test. Furthermore, it is worth noting that inputs Ii and I j do not necessarily have to be within the initial test cases, they could be newly generated test cases that fit the relation specified by the identified MR to perform MT[4] on the QBS under test. For the running example, if test cases arise where the input relation does not hold, e.g. the query f unny pandas includes an image that is not a panda, which leads to the result of the query not being a subset of the query pandas, then that means we have found a potential error in the system.
4. 4.1
Experiments
Experiments Overview
For the proposed approach, the experiments were conducted in the form of a case study, applying the proposed approach combined with widely known test case generation approaches to a given real-world QBS. The QBS Under Test: The experiments were performed on the National Cheng Kung University Advanced Course Search Website[10], a real-world QBS with no ground truth of query results available. This website belongs to National Cheng Kung University (NCKU) and is frequently used by students of NCKU to perform search queries for the
10
E NG -S HEN T U , S HIN -J IE L EE
Fig. 5. The National Cheng Kung University Course Search Webpage[10].
courses of the current semester. In Fig. 5, we can see the structure of the website as well as the query parameters and an example query result. For the following case study, the values of the column: Dept/Inst Code Serial Number Course System Number represent the result of each query. Fuzz Testing Case Study: In this case study, the proposed approach is applied to FT-generated test cases. FT is a widely known software testing technique that involves providing unexpected and random inputs to a program to identify vulnerabilities or bugs. It helps uncover potential issues by stressing the system with invalid or unexpected data, improving overall software security and reliability, as seen in practice in [11][12]. For this case study, the FT test cases are targeted toward the ”Instructor” query parameter. Combinatorial Testing Case Study: In this case study, the proposed approach is applied to CT-generated test cases. CT is a commonly used method in software testing with the overarching aim to minimize the number of test cases. By selecting a subset of possible combinations rather than testing every single combination, CT reduces the number of test cases required while still achieving high test coverage. CT can help identify interaction-related faults and improve testing efficiency. Applications of CT can be seen in [13]. For this case study, the combinatorial test cases are targeted toward the query parameters ”Instructor” and ”Day”. 4.2 4.2.1
Fuzz Testing Case Study Test Case Generation and Initial Querying
In this experiment, FT was applied to test case generation for the ”Instructor” search bar on the QBS webpage. During test case generation, a web crawler was scripted to get all the unique values listed in the instructor column of the QBS, which should include all
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
11
Fig. 6. Fuzz Testing Case Study: Comparison Table for Input Equivalence.
names of teachers in NCKU. Once the crawled data is saved in the form of test cases, the QBS is then queried with these test cases, and the results are then saved for the next stage of the approach, MRP detection and comparison table generation. 4.2.2
MRP Detection and Comparison Table Generation
In this case study, despite being able to generate comparison tables for all MRPs, only the comparison table for the MRP Input Equivalence is demonstrated, as it showed the most prominent results, see Fig. 6, the rest are excluded for concision. Following Eq. (1), the MRP input equivalence condition equation is defined as: f (Ii ) ≡ f (I j )
(2)
where Ii and I j in Eq.( 2) are said to be input equivalent. In Fig. 6, the Fuzz column contains all the names of teachers in National Cheng Kung University, the Query Results column holds the queried results, and the Equivalence Found column holds a list of inputs within the initial test case that yield an equivalent result to the Fuzz input.
Fig. 7. Fuzz Testing Case Study: Observation of Teacher Name Input Relation.
4.2.3
Input Relation Identification
Upon observation of the input equivalence comparison table, it is not hard to find an input relation between some of the teacher names. In Fig. 7, it can be observed that given the instances where there is only one case of equivalence, for each instructor, there
12
E NG -S HEN T U , S HIN -J IE L EE
is an equivalence between his or her English and Chinese name, which is logical since querying an Instructor’s English name should return the same results as querying their Chinese name. This prompts the identification of an input equivalence MR equation for the Instructor parameter, which, following Eq. (1) and Eq. (2), can be defined as: f (TeacherEnglishName) ≡ f (TeacherChineseName).
(3)
This MR can then be used to test the QBS[10] under test in the next stage.
Fig. 8. Fuzz Testing Case Study: Errors found in the QBS under test[10].
4.2.4
Check for Errors in QBS
Now that the Eq. (3) has been induced, as an MR of the QBS under test[10], MT[4] can now be performed to check for errors. In this case, since all the teachers’ names were scraped and added during initial test case generation, no new test cases need to be generated to perform testing, and the results of the initial test cases can be conveniently used to validate the QBS under test. To check for errors, the results are validated by checking if there is an English Name counterpart to the Chinese Name and vice versa in the Equivalence Found column. 4.2.5
Results and Remarks
After testing, errors were indeed found in the QBS under test, e.g. in Fig. 8, it is obvious there is no English Name counterpart to the Chinese Name in the Fuzz Column. Similar errors can also be found in Fig. 6, and after complete validation, approximately half of the test cases exhibit this error. This finding can subsequently be reported to the owner of the website. In addition, it is also worth noting that even though the error in Fig. 8 can be visually seen in Fig. 7, it does not substantively affect the tester’s judgment when prompted by the comparison table to identify the input relation Eq. (3) mentioned in Sec. 4.2.3. Through this experiment, it is shown that the proposed approach can indeed be used to discover metamorphic relations and detect errors in QBSs even when the initial test cases do not yield completely correct results. Furthermore, this experiment also shows the compatibility and successful combination of the proposed approach and FT. 4.3 4.3.1
Combinatorial Testing Case Study Test Case Generation and Initial Querying
In the second experiment, the proposed approach is applied to CT. For the combinatorial test cases, the parameters targeted are the ”Instructor” and ”Day” parameters, see
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
13
Fig. 9. Combinatorial Testing Case Study: Comparison Table for Conjunctions.
Fig. 10. Combinatorial Testing Case Study: Filtered and Sorted Comparison Table.
Fig. 5, in the QBS under test [10]. For combinatorial test case generation, PICT[14] was used to generate pairwise test cases for the query parameters, specified as follows: • Instructor: A total of 10 instructors, each with more than 25 courses in the QBS, were chosen. • Day: A value from 0 − 7, with 1 − 7 denoting the 7 days of the week and 0 defined as no selection of the day, which yields results of 1 − 7 combined. With 10 instructors and 8 days, 80 test cases were generated with PICT[14], and the QBS under test[10] was queried with all 80 pairwise combinations. 4.3.2
Comparison Table Generation
For the demonstration of the results of this experiment, only the comparison table of the MRP Conjunctive Conditions, see Fig. 9, is shown. However, while the other MRP
14
E NG -S HEN T U , S HIN -J IE L EE
comparison tables are excluded for concision, it is worth noting that the MRPs Input Equivalence, Disjoint Partitions, Complete Partitions, and Partition Difference all yield well-performing MRs that can be identified with this set of test cases. In this case study, following Eq. (1), the conjunctive condition equation is defined as: j ∈ f (Ii ), ∀ j ∈ f (I j )
(4)
where I j is a subset of Ii . In Fig. 10, the Instructor and Day columns contain the query parameters mentioned in Sec. 4.3.1, the Query Results column contains the course results of the QBS under test[10], and the Conjunctions Found column specify the subsets of the query found within the test cases, which are listed in the form [Instructor Name, Day]. 4.3.3
Input Relation Identification
The original Conjunctions comparison table, see Fig. 9, is visually messy and complex, making it difficult to prompt the identification of any input relations. However, once the Conjunctions column is filtered by name and the data frame is sorted by Day, the resulting filtered and sorted comparison table can be seen in Fig. 10, which is significantly more readable. Upon observation of the filtered and sorted comparison table in Fig. 10, it can be observed that for each instructor, [Instructor Name, Day=[1-7]] are subsets of the test case [Instructor Name, Day=0], which can be used to prompt the identification of an MR for Conjunctive Conditions. The MR equation, following Eq. (1) can be defined as: f (Name, Day = [1 − 7]) ∈ f (Name, Day = 0).
(5)
Upon further observation, since Day=0 was defined as the union of Days 1 − 7, this observation can also be used to define an MR for Complete Partitions. This second MR equation can be defined as: f (Name, Day = 0) ≡ ∪( f (Name, Day = [1 − 7])).
(6)
These MRs can then be used to test the QBS[10] under test in the next stage. 4.3.4
Check for Errors in QBS
With the induced MR equations, Eq. (5) and Eq. (6), as MRs of the given QBS[10], MT[4] is performed to check for errors. In this case study, all test cases for both MRs passed, and no errors were detected. 4.3.5
Results and Remarks
Even though no errors were found with these identified MRs in the QBS under test[10], it can be inferred from this result that Eq. (5) and Eq. (6) are MRs that hold consistently within the QBS under test[10]. Furthermore, This experiment shows the compatibility and successful combination of the proposed approach and CT and even goes on to show that the proposed approach can help alleviate the oracle problem that CT might be challenged with when testing QBSs.
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
5. 5.1
15
Discussion
Limitations
Though versatile and arguably applicable to real-world scenarios, the proposed approach does have limitations that affect its performance. • The proposed approach relies on the assumption that at least some portion of the queries of the initial test cases are correct. That is, if the portion of correct queries is too low, the tester may not be able to infer the input relations when prompted by the generated comparison table. • Following the previous limitation, if there are 0 correct queries in the initial test case, then the proposed approach will fail to prompt correct MRs to the tester, nor will it be able to detect whether such errors that exist within the initial test cases. 5.2
Future Work
This research unlocks intriguing research opportunities and challenges waiting to be addressed, including: • Automation: Future work in this regard could be focused on how to automate the process that requires human observation and intervention to provide a fully automated solution. • Combining with other testing methods: In this paper, the case studies demonstrate the proposed approach combined with FT and CT; however, there could be other combinations that may yield valuable insight. • Application in other domains: This paper specifically focuses on QBSs; however, the proposed approach could have the potential to benefit other software systems beyond QBSs where MRPs can be identified.
6.
Related Work
In this section, we will address some related work as well as how this paper contributes to new findings. The related work presented in this section mainly focuses on metamorphic relation generation and metamorphic testing. 6.1
Metamorphic Relation Patterns for QBSs
In 2019, Segura et al.[1] presented a catalog of metamorphic relation patterns for Query-Based Systems[1] from their observation that the MRs used to test different types of QBSs are very similar, regardless of their domain, as all of them exploit standard query features. This related work serves as the basis of the prompts that the proposed approach can provide to a tester when identifying MRs.
16
6.2
E NG -S HEN T U , S HIN -J IE L EE
Automated Metamorphic Relation Generation for QBSs
In 2022, Sergio et al.[9] proposed an automated way of generating MRs through constraint programming[9]; however, their proposed method requires the tester to provide a lightweight specification of the query parameters of the QBS, this process of specification inference might require extensive domain knowledge and may not be available or easy to induce. In addition, their proposed method also requires a source test case that contains no errors for their program to generate MRs. In comparison, the proposed method of this paper does not require such specification, nor does it need test cases with complete ground truth to work, arguably making it more applicable to real-world QBSs that require testing. 6.3
Enhancing Existing Testing Methods with MT
In Sec. 4.3, we see the compatibility of the proposed approach and CT. The combination of MT and CT has been a rising topic of research, discussed in cases such as [15], as CT is often faced with the Oracle problem[3], which MT can alleviate[5]. In this view, Enhance CT With Metamorphic Relations[16] presents a CT methodology to enhance traditional CT by accounting for metamorphic relations; however, this method mainly focuses on the mix of CT and MT and assumes that the tester already has the MRs whereas the proposed approach prompts the tester to identify the MRs.
7.
Conclusion
In this paper, an approach for prompting the identification of MRs for MT in realworld QBSs is proposed. The proposed approach is efficient in terms of manpower since the only process that requires extensive time and effort is the observation of the comparison table and identification of input relations. Furthermore, the proposed approach does not require ground truth for the QBS under test, alleviating the Oracle problem[3] and making the application of this approach in the testing of real-world QBSs arguably more practical than methods that require source test cases or ground truth in order to test and validate a given QBS. Furthermore, through the conducted experiments, it can also be established that the proposed approach can be applied and successfully combined with other testing methods such as FT and CT, opening the door to more possibilities in terms of research on MT as well as improving the ease, efficiency, and effectiveness of QBS testing.
ACKNOWLEDGMENT This research is sponsored by the Ministry of Science and Technology under the grant NSTC 112-2221-E-006 -084 -MY2 in Taiwan.
I NPUT R ELATION P ROMPTING FOR M ETAMORPHIC T ESTING
17
REFERENCES 1. S. Segura, A. Durán, J. Troya, and A. Ruiz-Cortés, “Metamorphic relation patterns for query-based systems,” in 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET), 2019, pp. 24–31. 2. E. J. Weyuker, “On testing non-testable programs,” The Computer Journal, Vol. 25, no. 4, 1982, pp. 465–470. 3. E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, Vol. 41, no. 5, 2015, pp. 507–525. 4. T. Y. Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic testing: A new approach for generating next test cases,” 2020. 5. H. Liu, F.-C. Kuo, D. Towey, and T. Y. Chen, “How effectively does metamorphic testing alleviate the oracle problem?” IEEE Transactions on Software Engineering, Vol. 40, no. 1, 2014, pp. 4–22. 6. S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortés, “A survey on metamorphic testing,” IEEE Transactions on Software Engineering, Vol. 42, no. 9, 2016, pp. 805– 824. 7. M. Lindvall, D. Ganesan, R. Árdal, and R. E. Wiegand, “Metamorphic model-based testing applied on nasa dat – an experience report,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2, 2015, pp. 129–138. 8. T. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys, Vol. 51, 01 2018, pp. 4:1–4:27. 9. S. Segura, J. C. Alonso, A. Martin-Lopez, A. Durán, J. Troya, and A. Ruiz-Cortés, “Automated generation of metamorphic relations for query-based systems,” in 2022 IEEE/ACM 7th International Workshop on Metamorphic Testing (MET), 2022, pp. 48–55. 10. N. Computer and N. Center, “Ncku advanced course search webpage,” 2023, accessed on May 16, 2023. [Online]. Available: https://course.ncku.edu.tw/index. php?c=qry11215&m=en query 11. G. Klees, A. Ruef, B. Cooper, S. Wei, and M. Hicks, “Evaluating fuzz testing,” in Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, ser. CCS ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 2123–2138. [Online]. Available: https: //doi.org/10.1145/3243734.3243804 12. J. Liang, M. Wang, Y. Chen, Y. Jiang, and R. Zhang, “Fuzz testing in practice: Obstacles and solutions,” in 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2018, pp. 562–566. 13. C. Nie and H. Leung, “A survey of combinatorial testing,” ACM Comput. Surv., Vol. 43, no. 2, feb 2011. [Online]. Available: https://doi.org/10.1145/1883612. 1883618 14. microsoft, “Pairwise independent combinatorial testing,” 2022, accessed on May 18, 2023. [Online]. Available: https://github.com/microsoft/pict 15. F. Wotawa, “Combining combinatorial testing and metamorphic testing for testing a logic-based non-monotonic reasoning system,” in 2018 IEEE International Confer-
18
E NG -S HEN T U , S HIN -J IE L EE
ence on Software Testing, Verification and Validation Workshops (ICSTW), 2018, pp. 348–351. 16. X. Niu, Y. Sun, H. Wu, G. Li, C. Nie, L. Yu, and X. Wang, “Enhance combinatorial testing with metamorphic relations,” IEEE Transactions on Software Engineering, Vol. 48, no. 12, 2022, pp. 5007–5029.
Eng-Shen Tu (杜霙笙) is a research assistant in the Software Engineering and Intelligent Test Automation Lab at National Cheng Kung University in Tainan, Taiwan. Currently, he is working towards his Bachelor’s degree in Computer Science and Information Engineering at the same university. Though still early in his academic journey, James continues to explore and deepen his understanding of his chosen fields, always keen to contribute to new knowledge and innovative solutions in the domain of Information Science and Software Engineering.
Shin-Jie Lee (李 信 杰) is an Associate Professor in Computer and Network Center at National Cheng Kung University (NCKU) in Taiwan and holds joint appointments from Department of Computer Science and Information Engineering at NCKU. His current research interests include software engineering and web test automation. He is the creator and team lead of SideeX project, serving as a basis for the most popular open source record-playback test automation tool in the world - Selenium IDE. He received his Ph.D. degree in Computer Science and Information Engineering from National Central University in Taiwan in 2007.