Department of Computing Technologies Software Testing and Reliability (SWE30009)

Semester 2, 2021

Assignment 3

1. Basics of Metamorphic Testing

Metamorphic testing is a technique that was proposed in the 90s for overcoming the
test Oracle problem by using known properties of an input domain. For example l ets
suppose you were trying to implement the mathematical sine function. Suppose sin
claims to compute sine correctly o 100 significant figures. What should sin (2.45)
return? If an Oracle for sine is not present, you don't know the answer to that question
In this case there is no oracle so we don’t know. So you don't have an Oracle a priori but
we know a few things we know the claim about the function: sin (x)=sin (x+2π). It's
going to compute things correctly to a hundred significant figures and also know quite
a lot about trigonometry for example sin (2.45) and sin(2.45+2π) are same for real
numbers for any X. Then this is a bug. So, what we could do is we could try sin(2.45)
and we could try sine of 2.45 plus two times pi and we could ask are these results the
same up to one hundred significant figures and if they're not then there's a bug in the
implementation of sine. Because we know from mathematics they should be the same
as real numbers and we know that sine. Our implementation is meant to have this
guarantee. If we just say that sine always returns zero that will satisfy this property. So
this isn't something that is going to be verified. It's not something that could be used
to replace any kind of sanity testing but it's a way of doing intensive follow-up testing.

The lack of test oracle has made performance testing a challenging task. Metamorphic
testing enables the test cases generation by exploiting the metamorphic relations in the
absence of oracle. A program is said to be untestable or nontestable if the outputs of some
inputs cannot be verified the inputs and outputs of multiple executions of the program
under test. . Metamorphic testing is used to detect faults especially functional faults or bugs
in a multiple domains. For example web servers, simulation tools and simulators, machine
learning etc.

Test Oracle actually determines whether a test has passed or failed. These oracles can be
manual or automated. In The process of oracles the output of the system under test for a
given test-case input is compared to the output that the oracle determines that product
should have.  For example we have a test case and a source to generate these test cases
and we get out of it a particular test case and run it against a program under test and then it
is decided whether the test is passed or failed. Most of the time we don’t have a true oracle
and it is very difficult to write one so we have to put up with an approximate oracle. In this
 case we create a consistency oracle and what we do is to check the program under test
against the previous version of the program under test and then we check to see whether
or not they had same behavior in all. The properties of oracles are mentioned below:

Soundness and Completeness

If an oracle determines correctly that determines when a test fails then it is

Working of metamorphic testing:

In metamorphic testing we start out with an input that you provide you also supply a set of
transformations that are going to monkey with that input from there. The tester computes
all combinations that are possible of those transformations that apply that to the system
input.

For every possible combination of those transformations tester gets a new input for the
tester to compute an output for the original input and every transformed input and you
enforce that some relation holds between those outputs

Features:

For N transformations there are 2 N-1 test cases if the order does not matter. If the order
of the transformations doesn't matter, you know exactly what the input is so you can say
specific stuff about it and you don't need to know the outputs before time which eases
the developer’s burden.

2. Metamorphic Relations

Each input consists of one list:

SI = SI-L1
FI = FI-L1, F1-L2

1.

Given SI = SI-L1
Define FI = FI-L1,FI-L2 such that
FI-L2=SI-L1, and FI-L2 is constructed from SI-L1 by deleting the smallest element

FO ??? SO???
Example:

removeDuplicates( [-3, 9, 16, 9,7, -5, 128, 23, 5, -20]) = removeDuplicates(  -3, 9, 16, 9,7, -5,
128, 23, 5])

2.

Given SI = SI-L1
Define FI = FI-L1,FI-L2 such that
FI-L2=SI-L1, and FI-L2 is constructed from SI-L1 by deleting the Largest element

FO ??? SO???

Example:

RemoveDuplicates ( [-3, 9, 16, 9,7, -5, 128, 23, 5, -20]) = removeDuplicates(  -3, 9, 16, 9,7, -5, 23,
5])

3.

Given SI = SI-L1
Define FI = FI-L1, FI-L2 such that
FI-L2=SI-L1, and FI-L2 is constructed from SI-L1 by deleting most repeated number

FO ??? SO???

Example:

RemoveDuplicates ( [-3, 9, 16, 9,7, -5, 128, 23, 5, -20]) = removeDuplicates([-3, 9, 16, 9,7, -5,
23, 5, -20])

4.

Given SI = SI-L1
Define FI = FI-L1, FI-L2 such that
FI-L2=reverse of SI-L1 is constructed from SI-L1 by deleting the least repeated element

FO??? SO???

Example:

RemoveDuplicates ( [-3, 9, 16, 9,7, -5, 128, 23, 5, -20]) = removeDuplicates( [ 9, 9, -20])
5.

Given SI = SI-L1
Define FI = FI-L1,FI-L2 such that
FI-L2=SI-L1, and FI-L2 is constructed from FI-L1-SI-L1

FO ??? SO???

Example:

RemoveDuplicates ( [-3, 9, 16, 9,7, -5, 128, 23, 5, -20]) = removeDuplicates( [])

3. Application Domains for Metamorphic Testing

The most popular Application domains for metamorphic testing are:

1. Web Services and Applications 16%

Use of metamorphic service for Service–Oriented Applications (SOA) which is done by

Used to automate the process, and evaluated it with three subject web services and
mutation analysis.

Analyze the XML description of the service composition to select adequate

Used for the detection of inconsistencies in online web search applications like Google, Yahoo!
and Live Search

Used to detect faults in the software under test (verification) and heck whether the program
behaves as the user expects (validation).

Assessment of quality related properties such as reliability, usability or performance in the

Used for image generation techniques for testing image processing programs.

Used for the generation of follow–up test cases and the assessment of test results.

Used for mesh simplification programs using pattern classification and metamorphic testing.
Used to detect test cases erroneously labeled as passed by a trained pattern classifier.
Used to evaluate the effectiveness of test data generation techniques and metamorphic
relations for a jpeg2000 image encoder.

Used to detect faults in morphological image operations such as dilation and erosion.

Used for casting simulation, exploiting the properties of the Medial Axis geometry function.

Used to check the conformance between network protocols and network simulators.

Used for the detection of faults in open queuing network modeling, a technique for planning
the ]capacity of computer and communication systems.

Used for detection of faults in a Monte Carlo modeling program for the simulation of
photon propagation.

Used to systematically test health care simulation programs, and presented a case study with
two real–world simulators and mutation testing.

Used to detect unexpected behavior when simulating cloud provisioning and usage.

Used for the detection of bugs related to energy consumption in distributed systems
as cloud environments.

4. Impact of Source Test Cases on the Effectiveness of Metamorphic Testing

Case I:

In this case we are given with 100 source test cases only. We don’t know about
domain, domain size and number of failure. In order to calculate we need number of
failures. Without intuition we can say that Random testing is better than
proportional sampling strategy. The main reason of this verdict is that the number of
distributions is unknown as this information is necessary for PSS. Generally we know
that The P-measure of PSS is not less than that of random testing The E-measures of
PSS and random testing are equal.

Case II:

We say that there are 4 Partitions with input domain equal to 500, number of
failures causing inputs equal to 25. As we know number of test cases is 100. Size of
partitions is 50,100,150 and 200 respectively. Where number of test cases is also
divided in 4 parts such that n1=10,n2=20,n3=3,n4=40. This distribution will result in
sampling rate of 0.2 for all partitions. In this case PSS is better than the random
testing.

Θ=m/d
 M=25 d=500

Θ=25/500=0.05

Pr=1-(1-0.05)100=0.994

Er=nθ

Er=100*0.05=5

For Random Testing

E=n*θ

Ep=n1 θ1+n2 θ2+n3 θ3+n4 θ4

n1=10,n2=20,n3=3,n4=40

θ1=0

θ2=0

θ3=0.1

θ4=0.2

Ep=30*0.1+40*0.2=11

In the above case Expected number of failures (E-measure) of PSS is more than Random
testing which gives strong reason to use PSS instead of random testing.