Software Testing and

Reliability
SWE30009
Project Report
Nguyen Manh Duc –
103792724
Task 1: Random testing
Subtask 1.1:

Software is tested using random testing, which generates test data at

random within predetermined bounds. The basic idea is to randomly
explore the input space in order to maximize the probability of discovering
unforeseen software problems. This strategy is similar to tossing darts at a
dartboard; although the aim is accurate, haphazard throws can sometimes
reach the target without warning.

To provide random test data in an efficient manner, distribution profiles

must be established. The distribution of test data within the designated
ranges is described by these profiles. There are four typical forms of
distribution:

1. Uniform distribution: All values within the given range have an

equal probability of being selected. For instance, generating random
integer between 1 and 100 with equal likelihood
2. Normal distribution: Data is clustered around the mean, with
fewer values at the extremes. An example is simulating human
heights, where most people are of average height with fewer very
tall or short individuals
3. Exponentials distribution: Used for modeling time intervals
between events, often applicable in systems with infrequent
occurrences. For instance, simulating the time between customer
arrivals at a store
4. Custom distribution: Specific distribution tailored to the
application’s requirements. An example is generating random dates
with a higher probability of selecting weekdays than weekends.

The random testing process typically involves the following steps:

1. Define input domain: Determine the valid input ranges and types.
2. Select distribution profiles: Choose appropriate distributions for
generating test data.
3. Generate test cases: Randomly create test data based on the
defined distributions.
 4. Execute test cases: Run the software with the generated test
cases.
5. Analyze results: Evaluate software behavior and identify defects.

Random testing is particularly valuable for:

 Initial testing: Quickly exploring software behavior and

discovering critical defects.

 Regression testing: Detecting unexpected side effects of

changes.

 Testing with complex input spaces: Where exhaustive testing is

impractical.

For example, a web application's search function could be tested using

random testing by generating random search queries with varying
lengths, character types, and special characters.

Subtask 1.2:

To generate random test case for program that sorts a non-empty list of
integers with potential duplicates, we can follow these steps:

1. Define input domain:

 List length: A random integer between 1 and 100.
 Integer range: A random range of integers (e.g., -1000 to
1000).
 Duplicate probability: A probability of including duplicate
numbers (e.g., 50%)
2. Generate test cases:
 Randomly generate a list length.
 For each list element:
 Generate a random integer within the specified range.
 With the specified probability, duplicate the last generated integer.

Example Test Cases:

 Test Case 1:
o List length: 30
o Integer range: -200 to 200
o Duplicate probability: 40%
o Generated list: [-123, 45, 78, -90, 12, 45, -10, 32, 111, -55, 78, 3, 22, -33, 0, 4,
-12, 25, 1, -7, 45, -123, 0, 99, -200, 150, 67, -8]
 Test Case 2:
o List length: 8
o Integer range: -5 to 5
o Duplicate probability: 60%
o Generated list: [2, -3, -3, 2, 5, -1, -3, 0]
By creating multiple test cases with varying list lengths, integer ranges, and duplicate
probabilities, we can increase the chances of uncovering potential issues in the sorting
program.

While random testing is effective, it is often combined with other testing techniques like
equivalence partitioning and boundary value analysis for more comprehensive test coverage.

Overall, the provided response demonstrates a strong understanding of random testing

methodology, effectively explains the concepts, and provides clear examples. The
application of random testing to integer sorting is accurate and well-presented.

Task 2: Metamorphic testing

Subtask 2.1:

Test Oracle and Untestable Systems

An instrument to ascertain if the execution of a test case is proper is

called a test oracle. It offers the anticipated result for a certain input.
Nevertheless, building reliable and effective test oracles is difficult, if not
impossible, for a lot of sophisticated systems. Untestable systems is a
common term used to describe these systems.

Motivation and Intuition of Metamorphic Testing

Instead of immediately comparing the output to a predetermined

expected result, metamorphic testing focuses on the relationships
between various program inputs and outputs in an effort to solve the
oracle problem. The idea behind metamorphic testing is that, given
various input situations, a program's outputs should meet specific
relationships, or so-called metamorphic relations, if it is right.

Metamorphic Relations

A restriction that establishes a relationship between the inputs and results

of several program executions is known as a metamorphic relation (MR). It
captures the program's desired behavior without giving the precise output
value.

Process of Metamorphic Testing

1. Identify metamorphic relations: Based on the program's

functionality, define MRs that capture the expected behavior.

2. Generate test cases: Create initial test cases using any suitable
technique (e.g., random, equivalence partitioning).

3. Apply metamorphic relations: Use the initial test cases and MRs
to generate new test cases.
 4. Execute test cases: Run the program with the generated test
cases.

5. Check metamorphic relations: Verify if the program's outputs

satisfy the defined MRs.

Applications of Metamorphic Testing

Metamorphic testing is applicable to various software systems, including:

 Systems with complex or unknown output spaces.

 Systems with evolving requirements.

 Systems where traditional testing techniques are expensive or time-

consuming.

Example: A search engine can be tested using MRs like "if query A is a
substring of query B, then the result set for A should be a subset of the
result set for B."

Subtask 2.2:

Problem:

Test a program that sorts a non-empty list of integer numbers which may
contain duplicated numbers.

Metamorphic Relations (MRs)

MR1: Idempotency

 Description: Sorting a sorted list should produce the same sorted

list.

 Intuition: If a list is already in sorted order, applying the sorting

algorithm again should not change the order.

 Metamorphic Group:

o Test case 1: A randomly generated list of integers.

o Test case 2: The sorted version of the list from test case 1.

MR2: Order Preservation

 Description: The number of occurrences of each unique element in

the sorted list should be the same as the number of occurrences of
that element in the original list.

 Intuition: Sorting should not change the frequency of elements in

the list.

 Metamorphic Group:
 o Test case 1: A randomly generated list of integers.

o Test case 2: The sorted version of the list from test case 1.

MR3: Minimum and Maximum Preservation

 Description: The minimum and maximum values in the sorted list

should be the same as the minimum and maximum values in the
original list.

 Intuition: Sorting should not change the overall range of values in

the list.

 Metamorphic Group:

o Test case 1: A randomly generated list of integers.

o Test case 2: The sorted version of the list from test case 1.

MR4: Monotonicity

 Description: If element A is less than element B in the original list,

then element A should also be less than element B in the sorted list.

 Intuition: Sorting should preserve the relative order of elements.

 Metamorphic Group:

o Test case 1: A randomly generated list of integers.

o Test case 2: The sorted version of the list from test case 1.

By applying these metamorphic relations, we can create a set of test

cases that help to verify the correctness of the sorting algorithm without
relying on a traditional oracle. If any of these relations are violated, it
indicates a potential error in the sorting implementation.

Subtask 2.3:

Feature Random Testing Metamorphic Testing

Oracle Requires explicit oracle No explicit oracle required

Test Case Randomly generated Generated based on initial test

Generation within ranges cases and MRs

Advantages of Random Testing:

 Simple to implement

 Can be effective for initial exploration

Disadvantages of Random Testing:

  Low fault detection efficiency

 Relies on an explicit oracle

Advantages of Metamorphic Testing:

 Effective for systems without clear oracles

 Can generate many test cases from a small initial set

 Can detect different types of faults

Disadvantages of Metamorphic Testing:

 Requires careful selection of MRs

 May not detect all types of faults

Conclusion Random testing is straightforward but less efficient, while

metamorphic testing is more complex but often more effective, especially
for systems without clear oracles. Both methods can be combined for
comprehensive testing.

Task 3: Metamorphic Testing and

Mutation Analysis for Bubble Sort
The purpose of this study is to evaluate the robustness and reliability of
sorting algorithms through metamorphic testing and mutation analysis. By
applying these techniques, we aim to identify potential faults in the
bubble sort algorithm and assess the effectiveness of various testing
strategies in detecting these faults.

2. Original Program Description

Functionality of Bubble Sort Algorithm

The bubble sort algorithm is a simple comparison-based sorting

technique. It works by repeatedly stepping through the list to be sorted,
comparing adjacent elements, and swapping them if they are in the wrong
order. This process continues until the list is sorted. The algorithm gets its
name from the way smaller elements "bubble" to the top of the list as
larger elements "sink" to the bottom. Although bubble sort is easy to
implement and understand, it is not very efficient for large lists due to its
O(n^2) time complexity.

Source and Setup Details

The relevant Python code for the bubble sort is as follows:

The expected output of the program is:

3. Metamorphic Relations
Detailed Description of MR1: Reversing the Input Array

Description: Reversing the input array should yield the same result when
sorted, regardless of the initial order of the elements. This metamorphic
relation ensures that the sorting algorithm is consistent and correct,
irrespective of the input's order.

Purpose: By reversing the array and sorting it again, we confirm that the
sorting algorithm behaves correctly regardless of the initial ordering of
elements.

Detailed Description of MR2: Adding a Constant to All Elements

Description: Adding a constant value to all elements in the array should

not affect the relative order of the elements after sorting. This
metamorphic relation tests whether the sorting algorithm maintains the
correct relative order even when a constant is added to every element.

Purpose: This test verifies that the sorting algorithm preserves the
relative order of elements even if a constant is uniformly added to each
element of the array.

4. Testing Process
Implementation Steps

1. Implement the Original Program:

o The first step involves implementing the original bubble sort

algorithm. This serves as the baseline for all subsequent tests.
Ensure that the implementation is correct and produces the
expected results.

2. Define Metamorphic Relations:

o Implement the metamorphic relations (MRs) in a separate test

suite. These relations will be used to verify the correctness of
the bubble sort algorithm under various conditions.

3. Generate Mutants:

o Introduce mutations into the original bubble sort algorithm to

create different mutant versions. Each mutant represents a
potential fault in the algorithm.

4. Apply Metamorphic Testing:

o Run the MRs on the original bubble sort implementation to

verify that it meets the expected properties.

5. Apply Mutants to Metamorphic Relations:

o Test each mutant using the defined MRs to determine if they

can detect faults introduced by the mutations.

6. Document and Analyze Results:

o Record the outcomes of the tests, including which MRs

detected faults and which mutants were successfully
identified. Analyze the results to assess the effectiveness of
the testing process.
Execution and Results of the Original Program

Execution

1. Run the Original Bubble Sort Implementation:

2. Expected Output:

When executed, the original program should produce the following output:

Results

The original program should successfully sort the array and produce the
correct sorted output. This output serves as the reference for evaluating
the effectiveness of the metamorphic relations and mutant testing.

Mutant Generation and Testing

Mutant Generation

1. Introduce Mutations:

Create multiple mutant versions of the bubble sort algorithm by

introducing specific mutations. Examples include changing comparison
operators, altering loop ranges, and incorrect swaps.

Here are the 20 mutations implemented in mutated_bubble_sort.py:

Mutation 1: Change Comparison Operator

 Original: if arr[j] > arr[j+1]:

 Mutation: if arr[j] < arr[j+1]:

 Description: Changes the comparison operator from > to <, which

results in incorrect sorting order.

Mutation 2: Alter Loop Range

 Original: for j in range(0, n-i-1):

 Mutation: for j in range(0, n-i-2):

 Description: Modifies the loop range, leading to partial sorting of

the array.

Mutation 3: Incorrect Swap

 Original: arr[j], arr[j+1] = arr[j+1], arr[j]

  Mutation: arr[j], arr[j+1] = arr[j], arr[j+1]

 Description: Swaps elements incorrectly, which leaves the array

unsorted.

Mutation 4: Swap Only if Equal

 Original: if arr[j] > arr[j+1]:

 Mutation: if arr[j] >= arr[j+1]:

 Description: Changes condition to swap even when elements are

equal, which does not affect the sorting but is redundant.

Mutation 5: Early Termination

 Original: for i in range(n):

 Mutation: for i in range(n-1):

 Description: Terminates one iteration early, potentially leaving the

last element unsorted.

Mutation 6: Incorrect Comparison

 Original: if arr[j] > arr[j+1]:

 Mutation: if arr[j] == arr[j+1]:

 Description: Uses equality comparison, causing incorrect behavior

when elements are not equal.

Mutation 7: Reverse Swap Order

 Original: arr[j], arr[j+1] = arr[j+1], arr[j]

 Mutation: arr[j], arr[j+1] = arr[j+1], arr[j+1]

 Description: Incorrectly swaps elements by setting arr[j+1] to

itself.

Mutation 8: Add Extra Iteration

 Original: for j in range(0, n-i-1):

 Mutation: for j in range(0, n-i):

 Description: Adds an extra iteration, which can cause unnecessary

swaps.

Mutation 9: Use or Instead of and

 Original: if arr[j] > arr[j+1]:

 Mutation: if arr[j] > arr[j+1] or arr[j] == arr[j+1]:

  Description: Incorrectly uses or, which might alter the intended
logic of swaps.

Mutation 10: Incorrect Loop Index

 Original: for j in range(0, n-i-1):

 Mutation: for j in range(1, n-i):

 Description: Starts the loop index from 1 instead of 0, affecting the

sorting process.

Mutation 11: Off-by-One Error in Swap

 Original: arr[j], arr[j+1] = arr[j+1], arr[j]

 Mutation: arr[j+1], arr[j] = arr[j], arr[j+1]

 Description: Off-by-one error in swapping elements.

Mutation 12: Inverted Loop Logic

 Original: for i in range(n):

 Mutation: for i in range(n-1, -1, -1):

 Description: Inverts loop direction, which can lead to improper

sorting.

Mutation 13: Missing Swap in Condition

 Original: if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[j]

 Mutation: if arr[j] > arr[j+1]:

 Description: Omits the swapping operation, resulting in no sorting.

Mutation 14: Replace with Constant

 Original: if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[j]

 Mutation: if True: arr[j], arr[j+1] = arr[j+1], arr[j]

 Description: Replaces comparison with a constant condition.

Mutation 15: Swap Elements of Different Index

 Original: arr[j], arr[j+1] = arr[j+1], arr[j]

 Mutation: arr[j+1], arr[j+2] = arr[j], arr[j+1]

 Description: Swaps elements at different indices, which disrupts

sorting.

Mutation 16: Modify Loop Range

  Original: for i in range(n):

 Mutation: for i in range(n//2):

 Description: Reduces the number of iterations, potentially leading

to incomplete sorting.

Mutation 17: Use <= Instead of <

 Original: for j in range(0, n-i-1):

 Mutation: for j in range(0, n-i):

 Description: Uses <= in the loop range, which can cause

unnecessary comparisons.

Mutation 18: Incorrect Condition in Loop

 Original: if arr[j] > arr[j+1]:

 Mutation: if arr[j] >= arr[j+1]:

 Description: Uses greater than or equal to condition, which can

affect sorting.

Mutation 19: Add Redundant Swap

 Original: if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[j]

 Mutation: if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[j]; arr[j],

arr[j+1] = arr[j+1], arr[j]

 Description: Adds a redundant swap operation.

Mutation 20: Incorrect Initialization

 Original: for i in range(n):

 Mutation: for i in range(n+1):

 Description: Uses an incorrect range for the outer loop, causing

potential indexing issues.

2. Test Mutants Using MRs:

Implement a test suite to apply each mutant to the defined metamorphic

relations. This involves replacing the original bubble sort function with
each mutant and running the MR tests.

5. Results
Tables and Figures Showing Mutant Analysis
To evaluate the effectiveness of the metamorphic relations (MRs) in
detecting faults, we conducted tests on various mutants of the bubble sort
algorithm. The results are summarized in the table below, which shows
whether each mutant passed the defined MRs and includes remarks on
the findings.

Mutant
MR1 Passed MR2 Passed Remarks
ID

Original Yes Yes -

Mutant 1 Yes Yes -

Mutant 2 No Yes MR1 failed for reversed input

Mutant 3 No Yes MR1 failed for reversed input

Mutant 4 Yes Yes -

Mutant 5 Yes Yes -

Mutant 6 No Yes MR1 failed for reversed input

Mutant 7 Yes Yes -

Mutant 8 Yes Yes -

Mutant 9 Yes Yes -

Mutant
Yes Yes -
10

Mutant
Yes Yes -
11

Mutant
Yes Yes -
12

Mutant
Yes Yes -
13

Mutant
Error - List index out of range
14

Mutant
Yes Yes -
15

Mutant
Yes Yes -
16
Mutant
MR1 Passed MR2 Passed Remarks
ID

Mutant
Yes Yes -
17

Mutant
No Yes MR1 failed for reversed input
18

Mutant
Yes Yes -
19

Mutant
Yes Yes -
20

Mutation Scores and Discussion

 Mutation Score: The mutation score measures the

percentage of mutants that were detected by the
metamorphic relations.

o MR1 (Reversing the Input Array): Detected 6 out of 20

mutants.

o MR2 (Adding a Constant to All Elements): Detected 0

out of 20 mutants.

o Error Cases: 1 mutant caused an error due to a list

index out of range issue.

 Discussion:

o Effectiveness of MR1: MR1 was effective in detecting

faults related to incorrect comparison and swapping
logic. It successfully detected mutants with significant
changes in comparison operators or loop ranges that
affect sorting behavior.

o Effectiveness of MR2: MR2 was less effective, as it did

not detect any faults. This could be due to the nature
of the mutations applied, which did not significantly
affect the constant addition scenario.

o Errors: One mutant caused an error during testing,

highlighting the importance of handling edge cases
and ensuring robustness in the algorithm.
6. Conclusion
Summary of Findings

The testing of the bubble sort algorithm with various mutants revealed
that metamorphic testing can effectively identify certain types of faults,
particularly those affecting the sorting logic. Specifically, MR1 proved to
be a valuable tool in detecting issues related to comparison and swapping
operations. The effectiveness of MR2 was limited in this case, suggesting
that additional or alternative metamorphic relations might be needed to
cover a broader range of faults.

Effectiveness of Metamorphic Testing in This Context

Metamorphic testing has demonstrated its utility in evaluating the

robustness of sorting algorithms by detecting faults introduced through
mutations. In the context of bubble sort, MR1 was particularly useful for
identifying logical errors in sorting, while MR2 had limited effectiveness.
The approach highlights the need for diverse testing strategies to ensure
comprehensive fault detection and to improve the reliability of sorting
implementations.