EFFECTIVENESS OF MUTATION TESTING TOOL IN OBJECT ORIENTED SYSTEMS

A DISSERTATION
Submitted in partial fulfillment of the requirement for the award of Degree of Master of Technology in Information Technology Discipline
Submitted To

RAJIV GANDHI PRODYOGIKI VISHWAVIDYALAYA, BHOPAL (M.P.) Submitted By:

Shikha Singh Enrollment No

DEPARTMENT OF INFORMATION TECHNOLOGY TECHNOCRATS INSTITUTE OF TECHNOLOGY, BHOPAL

ABSTRACT
Software testing allows programmers to determine the quality of the software. Mutation testing is a branch of software testing which does more than this. It helps determine whether the test cases that have been created, effectively detect all the possible faults in the software. This allows the development of better test sets, thus, ensuring that maximum software quality is achieved. This may seem very inviting to software testers, but there is a major issue being faced which has hindered the widespread use of mutation testing. Mutation testing works by seeding faults in the software program. Various mutation operators are used to create these faulty programs. These programs are called mutants. The mutants depict software faults that may be caused by programmers while writing the software. Test cases are then executed on these mutants to determine if they have been killed or not. Test sets that kill all the mutants are considered to be good as they successfully detect all the possible program faults. Sometimes, it is difficult to kill all the mutants. The reason is that some of the mutants, although syntactically different than the original program, are still semantically the same. Any test case will not be able to differentiate between the original program and the mutant. Such mutants are called equivalent mutants.

This thesis targets this issue by determining which mutation operators create mutants, which are more probable to create equivalent mutants. A tool called MuJava, Jody is used for this purpose. An empirical analysis has been carried out for this purpose, which helps determine which mutation operators develop more equivalent mutants.

INTRODUCTION

Computer software can have two kinds of problems: faults or failures. A failure does not permit a program to perform its intended function. A fault is an error in the correctness of a programs semantics. Testing is a process of determining faults. Software testing is performed to ensure that the algorithms used in a program are correct, the program can execute correctly in all possible scenarios and it conforms to all the requirements specified. Correctness, completeness and quality are some of the characteristics that all software programs must meet. Software testing can be performed in a number of ways. There is no fixed process; it is a simple method of trial and error and investigating various problems encountered whilst execution. Software testing is performed using test cases. These make use of some variables and determine the actual output of the program against the expected output. Good test cases help determine faults that occur rarely.

1.1 Mutation Testing Mutation testing is one of the many types of methodologies used for testing a program. While other software testing techniques focus on the correct functionality of the program, mutation testing focuses on the test cases used to test the programs. The main idea is to create good sets of test cases rather than trying to find all the faults in a particular program. Good test cases are those which are able to discover all the easy and hard-to-find faults in the program. An ideal test case will detect all the possible faults in a software program. Mutation testing is a white-box testing technique i.e. it examines the internal structure of a program to detect faults. It is used to determine the effectiveness of test cases i.e. how good they are in detecting faults. Mutation testing has been developed using two basic ideas: 1. Competent Programmer Hypothesis: Software programs usually differ from the correct version of the program in minute ways. Most of the programs written are nearly correct. 2. Coupling Effect Hypothesis: Larger programming faults are coupled with smaller faults of the same nature.

These hypotheses form the platform for mutation testing. Since incorrect programs differ from the correct versions in minute ways, mutation testing works by making the correct programs incorrect by using the concept of fault seeding. This is done by the use of mutation operators. These operators are simple rules which create different versions of the same program with minor changes. These versions are called mutants. Typically, mutation operators replace operands with other similar operands or delete entire statements, etc. The process used in mutation testing can be explained simply in the following steps: 1. Suppose we have a program P 2. Apply mutation operators to P to produce mutant P 3. Apply test case t to P and P 4. Analyze outputs of P and P 5. If the outputs are different, then test case is effective i.e. it can detect the fault and has killed the mutant. 6. If the outputs are same, there could be two reasons: i. The mutant is difficult to kill. Write another test case to detect this fault. ii. The mutant has the same semantic meaning as the original program i.e. it is equivalent.

We need to achieve a high mutation score for any given set of test cases. This mutation score indicates how well a particular test set has detected faults in a program. Mutation score can be defined as: Mutation score = # of mutants killed / total # of non-equivalent mutants Where 0 <= M.S. <= 1 or 0% <= M.S. <= 100% A low score means that the majority of faults cannot be detected accurately by the test set. A higher score indicates that most of the faults have been identified with this particular test set. A good test set will have a mutation score close to 100% or ideally 100%.

A simple example of mutation testing has been illustrated below:

Program P

Apply mutation operation

C=a*b; D=c/a;

Mutant P

c = a + b; d = c / a;

C=a**b; D=c/a;

Apply test case t to P and P Result R Result R

If R != R, then t has killed mutant P

If R == R, then mutant P is still alive

Figure 1: Process of Mutation testing

Mutation testing faces a huge challenge. Since every program may have a fault in many possible ways, one problem is that a large number of mutants are created for very small programs. Another problem is that sometimes most of these programs are equivalent to the original program. The latter problem is the topic of discussion for this thesis. This project focuses on the use of mutation operators to eliminate the problem of equivalent mutants. The technique adopted suggests that some mutation operators may contribute more towards the creation of equivalent mutants. Hence, a detailed analysis is carried out for all the mutation operators used in MuJava (mutation testing tool). Scenarios are sketched which indicate situations always creating equivalent mutants for certain operators. For the extension of this thesis, it is suggested that the scenarios sketched out be transformed into algorithms and incorporated into the respective mutation operators.

LITERATURE REVIEW

The aim of mutation testing is to generate a good set of test cases that successfully help find all possible faults in a software program. We need to achieve a high mutation score i.e. a score close to 100%. To get the maximum mutation score, we need a test set which successfully kills all the mutants. But some of the mutants created are equivalent. Therefore, to get the highest possible score, all the nonequivalent mutants need to be detected. An equivalent mutant is one, which is syntactically different from the original program, but semantically the same. Such mutants only contribute in increasing the computational cost. They do not help in establishing whether a particular test case is effective in discovering faults in a program or not. The problem of determining whether a mutant is equivalent to the original mutant is theoretically decidable. Sample Program P if(x == 2 && y == 2) z = x + y;

 Mutant P if(x == 2 && y == 2) z = x * y; The value of Z will be equal to 4; any test set will be unable to determine any faults with this program because the value will always be equal to 4. Example of Equivalent Mutants Mutation testing is not popular in the industry. Reason being, that a large number of mutants are generated for very small programs. This incurs a high computational cost, as each test case needs to be executed on each mutant. The number of mutants generated for a software unit is proportional to the product of the number of data references and the number of data objects. To reduce the computational cost, some approaches have been proposed which are explained below:

2.1 Do Smarter approaches These approaches work by dividing computational cost amongst several machines or by preserving some information, such as compiled code, so that the same code need not be generated repetitively. They may also avoid complete execution of the entire program. E.g. suppose there is a program that has 10,000 lines of code. Assuming the 5th line is changed to create 3 different mutants. It will take a long time to compile the mutated program again and again. To avoid this, the compiled form of the original program is saved. This compiled form of the original program is changed for the single line modified in the mutants. Thus, computation cost is saved. 2.1.1 Weak Mutation Weak mutation is an approximation technique. It compares the internal states of the mutated program and the original program immediately after executing the mutated portion of the program. If the state of the mutated program is different from that of the original program, the test case has killed the mutant. Otherwise the mutant is still alive. The first tool developed for mutation testing called Mothra used this strategy.

Original program result = obj.add(a,b) Display(result);

Mutant result = obj.multiply(a,b) /* inspect value of result */ Display(result);

2.1.2 Distributed Architectures This approach works by dividing computational cost over multiple machines. Some work has been carried out on vector processors, SIMD machines, Hypercube (MIMD) machines, and Network (MIMD) computers to adapt them for mutation analysis. This is easy to perform as each mutant is independent of the other mutants and can be executed individually. 2.2 Do Faster approaches These approaches work by generating and running each mutant program as quickly as possible.

2.2.1 Schema-based Mutation Analysis Many mutation systems compile their programs using interpretive methods. This makes the task of compilation low and more tedious to build. Untch has built a Mutant Schema Generation system for this purpose. This system encodes all the mutants into one source-level program to create a metamutant . This metamutant is compiled once and executed in the same programming environment. Since only a fraction of code is different for each schema, repetitive runs of large programs can be avoided on the compiler, thus saving computational cost. Original program result = obj.add(a,b) Switch(n) Case 1: result = obj.add(a,b) Case 2: result = obj.subtract(a,b) Case 3: result = obj.multiply(a,b) Case 4: result = obj.divide(a,b) Case 5: result = obj.modulus(a,b) Case 6: Figure 4: Example of Schema-based mutation analysis Metamutant

2.2.2 Separate Compilation Approach This method avoids the interpretative style of execution. It creates, compiles, executes and runs each mutant individually. When mutant run times greatly exceed individual compilation/link times, a system based on such a strategy will execute 15 to 20 times faster than an interpretative system. One example of such a system is Proteum. 2.3 Do Fewer approaches These approaches run lesser number of mutants. A subset of mutants is selected from all the mutants created in such a way that it is sufficient to determine a good set of test cases. 2.3.1 Selective Mutation The aim of selective mutation is to achieve maximum coverage by using the least number of mutation operators. Operators are selected in such a way that most of the mutants are generated from them. E.g. multiple mutation operators, which produce the same mutant, are discarded when generating mutants. This idea was developed by Offutt and was called selective mutation. Since lesser operators are used, the number of mutants produced also decreases significantly.

2.3.2 Mutation Sampling This technique randomly selects a subset of mutants produced. This subset is then tested with the specified test set to determine its sufficiency. If not satisfied, another subset of mutants is selected randomly. Such subsets are continuously selected till we find one which helps determine the effectiveness of the particular test set. Another method of sampling does not use a priori fixed size of mutants; it uses a Bayesian sequential probability ratio test to determine whether a statistically appropriate sample size of mutants has been reached. 2.4 Techniques Some techniques have been proposed which deal with the problem of equivalent mutants. Some of these have been implemented using some algorithms. Tools have also been created for mutation testing. It was imperative to study these tools as this thesis involves the use of one of these tools.

2.4.1 Overcoming the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution This technique makes use of genetic algorithms. It is suggested that the use of genetic algorithms may help reduce the problem of generating a large number of mutants as well as the problem of equivalent mutants. It is argued that this strategy helps attain selective mutation without the need of decreasing the mutation operators applied to the programs. This technique performs three main operations: 1. Evolution of subsets of mutants against a fixed set of test cases: In this step, mutants are created and validated against a fixed set of test cases. The score generated indicates the performance level for the mutant against a particular set of test cases. A low score indicates that the mutant is difficult to kill. Subsets of these mutants are then created. Each subset corresponds to an individual of the genetic algorithm. A subset of non-equivalent mutants is then selected with a higher adequacy score. Genetic algorithms are then used to evolve this set. The fitness function used ensures that the mutants selected are such that they are definitely killed by some test case. Uniform crossover is used to enable the selection of highly fit individuals in the population.

2. Evolution of subsets of test cases against a fixed set of test mutants: The same strategy is used to create a subset of test cases. Test sets are generated randomly. Mutation scores are then assigned to each set of test cases by executing them against a fixed set of mutants. This mutation score corresponds to the performance of the test set against a given set of mutants. Again, each test set is evolved using a fitness function; it results in another set of test cases which when executed against a set of mutants give a higher adequacy score. 3. Combine the above two sets using co-evolution: The two populations generated above are then combined using a fitness function. The score of each mutant is re-evaluated with respect to the present population of test cases. Similarly, the score of each test case is re-evaluated with respect to the present population of the mutants. This is due to the design of the fitness functions. The fitness function created assigns an adequacy score of 0 to all the mutants which are difficult to kill since they are more likely to create equivalent mutants. And as explained, only mutants and test cases with higher adequacy scores are selected.

Summarizing the study presented in this paper, selective mutation has been achieved by the use of genetic algorithms without reducing the mutation operators used thus reducing the computational cost. Also, equivalent mutants are not considered to determine the effectiveness of a particular test set due to the ingenious design of the fitness function. 2.4.2 Using Program Slicing to Assist in the Detection of Equivalent Mutants It is often argued that it is difficult to apply mutation testing to large programs. This is because the detection of equivalent mutants becomes more tedious and almost impossible since it is done manually. It is debated in this paper that the creation of program slices makes detection of equivalence easier. A formal notation is used. Let the program be p and mutant p. An input is applied to both p and p. p and p can be compared by either considering their internal states or final output. To determine whether a mutant has been killed or not, three different choices are considered: 1. Strong mutation: Under strong mutation, either the final output of the two programs is considered (strong output mutation) or the final state of p and p is considered (strong state mutation). For strong output mutation, an equivalent mutant will produce the same value of variable x as the original program. Similarly, for strong state mutation, an

equivalent mutant will produce the same state of variable x as the original program at the end of execution. 2. Weak mutation: In weak mutation, if the value of some variable x is different immediately after the execution of some point in the program, say n, for both programs p and p when some input is applied, then the mutant is killed. Hence, a mutant will be equivalent only when the value of x is identical in p and p at any point n. 3. Firm mutation: Firm mutation creates p by mutating some part of p, for example a loop, a control structure, etc. It then examines p and p after applying some input to determine if the mutant has been killed or not. If the value of variable x is the same after the final node in the structure, the mutant is equivalent. The strategy discussed in this paper uses weak mutation. A new Boolean variable, say z, is introduced in the program p. A slice is then created on this variable and analyzed to determine if the mutant has been killed or not. The value of z is initially set to true. Every time the node n is met, the value of z is changed. If the mutant is killed it becomes false, otherwise it remains true. Equivalence is indicated if the value of z remains true at the end of execution.

Program p 1 substring (int from, int to, char target[], char source[]){ 2 from++; 3 start = from; 4 if(from < to){ 5 while(from != to){ 6 target[from-start] = source[from]; 7 from++; 8} 9 target[to-start] = /0; 10 } 11 else 12 target[0] = /0; 13 } Sample program to show strategy using program slicing to detect equivalent mutants

Mutant p 1 substring (int from, int to, char target[], char source[]){ 2 from++; 3 start = from; 4 if(from < to){ 5 while(from != to){ 6 target[from-start] = source[from]; 7 from++; 8} 9 target[from-start] = /0; 10 } 11 else 12 target[0] = /0; 13 } Example of mutant created for use with program slicing

The Boolean variable z is introduced at this point on line 2. The statement on line 10 shows that the variable to has been modified to the variable from in p. A slice is then created with respect to z and the variable from. This slice is then tested to determine if the mutant has been killed or is still alive at the end of test execution. Mutant p to be sliced 1 substring (int from, int to, char target[], char source[]){ 2 z = TRUE; 3 from++; 4 start = from; 5 if(from < to){ 6 while(from != to){ 7 target[from-start] = source[from]; 8 from++; 9} 10 z = z && from == to; 11 target[from-start] = /0;

12 } 13 else 14 target[0] = /0; 15 } Sliced mutant 1 substring (int from, int to, char target[], char source[]){ 2 z = TRUE; 3 from++; 5 if(from < to) 6 while(from != to) 8 from++; 10 z = z && from == to; 15 } Introduction of Boolean variable z in p & the creation of a program slice using z

The authors also discuss that creating amorphous slices may reduce the problem of detecting equivalent mutants even more by applying further transformations to the traditional slices. Amorphous Slice of mutant p substring (int from, int to, char target[], char source[]){ z = TRUE; from++; if(from < to) z = TRUE; } Creation of an amorphous slice from a traditional program slice for the detection of Equivalent mutants An interesting observation stated in the paper states that an ideal amorphous slicing algorithm would yield this slice for all equivalent mutants, and the equivalent mutant problem would disappear.

In summary, the use of program slices can ease the process of detection of equivalent mutants. it reduces the work carried out manually in identifying equivalent mutants. Also, it has been suggested that equivalent mutants will always create slices which are identical for all mutants. 2.4.3 Using Compiler Optimization Techniques to Detect Equivalent Mutants The strategy presented in this paper makes use of compiler optimization techniques. The authors of this paper suggest that many equivalent mutants are either optimizations or de-optimizations of the original program. Equivalent mutants can always be detected by their optimized code by the use of algorithms. Following are the 6 techniques explained in this paper: 1. Dead code detection Mutants that change dead code, i.e. code that will never be executed or is irrelevant, will not affect the output of the program and will thus be equivalent. 2. Constant propagation It creates a constant table, which keeps entries of variables with constant definitions.

Mutants that are still alive at the end of the execution of a test case can be checked for equivalence using this table. A mutant that has an entry in this table will be equivalent. 3. Invariant propagation This technique works by saving the relationship between 2 variables, also called an invariant, in an invariant table. An equivalent mutant will have the same definitions in the invariant table, thus not changing the value in the mutated program form that in the original program. 4. Common sub-expressions This technique does not identify equivalent mutants directly. It is used in conjunction with other compiler optimization techniques. It keeps track of all the temporary variables created during compilation and determines the equality of relationships between variables if any. Normal Code T1 = B + C T2 = T1 - D A = T2 T3 = B + C T4 = T3 D X = T4 Optimized Code T1 = B + C T2 = T1 D A = T2 X = T2

Figure 5: Common Sub-expression Detection Example 5. Loop invariant detection Here, an equivalent mutant is one, which changes the boundary of a loop by moving it inside or outside when the code is compiled. 6. Hoisting and sinking This also uses the same technique as loop invariant detection. Equivalent mutants will generate the same result regardless of hoisting or sinking the code. 2.4.4 Automatically Detecting Equivalent Mutants and Infeasible Paths The authors of the paper presume that the problem of detecting equivalent mutants is in-fact a type of a feasible path problem. The feasible path problem states that some test requirements are infeasible because of the nature of the program semantics. These requirements cannot be satisfied by the program. The strategy presented makes use of the constraint-based testing technique. This technique specifies a number of mathematical properties, which need to be met by any test case. Assuming that P is a program, M is a mutant of P on statement S, and t is a test case for P. three mathematical conditions need to be met to kill a mutant:

1. Reachability: The test case must execute the mutated statement. I.e. if t cannot reach S, then t will never kill M. 2. Necessity: To kill a mutant, the test case must cause the mutant to have an incorrect state if it reaches the mutated statement. I.e. for t to kill M, it is necessary that if S is reached, the state of M immediately following some execution of S must be different from the state of P at the same point. 3. Sufficiency: The test case must cause the final state of the mutant to be different from the original program. I.e. the final state of M differs from that of P. These mathematical conditions are then used to apply constraints on the programs. The constraints are used to determine which mutants are equivalent and which are not.

2.5 Tools As part of the literature survey, all the tools that have been created for mutation testing were also studied. None of these tools detect equivalent mutants automatically. The tester has to detect them by hand. Tools have been developed for a number of languages: 2.5.1 Mothra It is the oldest tool developed for mutation testing. It was developed as a project at Georgia Institute of Technologys Software Engineering Research Centre. This project was initiated in 1986 and developed using C language. It was developed for programs written in FORTRAN and made for the Linux platform. Mothra is a set of tools that can be called separately or called using one of Mothras interfaces. Since it is based on a set of data structures, many researchers have tried to expand Mothra by adding new tools. Mothra works by creating intermediate code of the program. This requires more compilations and an increase in performance costs.

2.5.2 Jester/Nester/Pester The tool that was developed first was Jester. It applied mutation testing to Java programs. Jester works for Java code with JUnit tests. Jester does simple modifications to the programs such as changing If statements to true or false, etc. After making these modifications, it runs tests on the modified programs. It then generated web pages displaying the results of the tests using a built-in script. Pester is the same as Jester only that it was written for programs created in Python. Test cases are written in PyUnit. Nester is also a variation of Jester written for C# programs. Unfortunately, no literature was available to study it in more detail. 2.5.3 MuJava MuJava is a tool written for java programs. It uses two sets of mutation operators; method-level and class-level. MuJava creates mutants using the various methodlevel and class-level operators. It then runs test cases on them and evaluates the mutation coverage for them. Test cases are written as separate classes that call methods in the classes that need to be tested.

Figure 6: Structure of MuJava

Mutant generator

Mutant executor

Behavior mutant generator

Behavior mutant

Behavioiural mutant executr

Compile time reflection+ms g

Standard java API[47]

Structured mutantgenrato r

Structural mutant

Structural mutant executar

Bytecode transformatio n

Different engines create the structural and behavioral mutants. For the behavioral mutants, compile time reflection is used to analyze the original program. The MSG engine then uses compile time reflection as well to create a meta-mutant program. For the structural mutants, the original source code is compiled using the Java compiler. BCEL API is then used to add or delete class members in the resulting byte code translation. 3.2 Motivation Mutation testing has a high computational cost. A large number of mutants are created for very small programs. This is because each program can have various faulty versions. Each version could be a simple replacement of an addition operator, for example, with all other arithmetic operators in a particular language. All these mutants need to be compiled and executed against a specific set of test cases. In addition to this high cost, sometimes most of the mutants created are equivalent. Equivalent mutants produce the same output for the original program as well as the modified program or the mutant. These issues have prevented mutation testing to gain wide-spread acceptance in the software industry. The aim of this thesis is to help determine a technique by the use of which the number of equivalent mutants produced is reduced.

Quite a few techniques have been proposed to date to deal with this issue as discussed in chapter 2. Some algorithms have been proposed for them, but none of them have been implemented so far. Most authors have resorted to combining other software engineering strategies such as genetic algorithms and compiler optimization techniques with mutation testing. In view of this thesis, this has made the process of detecting equivalence more arduous. The implementation of any of these strategies would require the creation of an entirely new tool for mutation testing rather than reusing any one of those already developed. The strategy discussed here makes use of MuJava, a mutation testing tool developed in 2003. None of the other mutation testing tools have been used due to lack of support for the Windows platform. Also, some of the tools did not contain enough literature to support their operation. The main idea is that some mutation operators may develop a greater number of equivalent mutants as compared to other operators. To support this argument, the mutation operators of MuJava are discussed in great detail. An analysis is carried out to see which operators create a large number of equivalent mutants.

3.3 The Strategy For this project, the main aim is to help identify those mutation operators that contribute more towards creating equivalent mutants. The procedure that was followed is explained below: 1. Test programs were created for method-level and class-level operators. These test programs serve as original programs for which mutants will be generated later. 2. Two different approaches were adopted for method-level and class-level operators a. For method-level operators, different programs were developed each for arithmetic, shift, logical operators etc. Then all the method-level mutation operators were applied to each type of program to create the mutants. b. For the class-level operators, programs were written for the common data structures of Linked list. A Stack class was then derived from it. Programs were then written which used these classes. These programs differed for each of the class-level mutation operators. All the class-level mutation operators were applied to all the programs to create mutants. 3. Empirical data was gathered indicating which mutation operators created a greater number of mutants as compared to other mutation operators.

4. Also, a manual inspection was conducted on all the mutants to discover the equivalent versions. 5. Java programs were then created containing the test cases for each of the method-level and class-level operators original programs created in step 2. These test cases perform branch coverage on the original program. 6. The test cases were executed against the original programs using MuJava. 7. The mutation scores were obtained. 8. Also, it was determined that how many mutants were left alive by each of the operators at the end of test execution. 9. Of the mutants left alive, the equivalent mutants were determined. 10. An empirical analysis was conducted to determine which mutation operators contribute more towards creating equivalent mutants.

RESEARCH METHODOLOGY

The tool used in this project is MuJava. Before explaining the strategy adopted for this project, it is imperative to explain the working of MuJava in more detail. 3.1 MuJava Mutation operators make syntactic changes to the program under test. These syntactic changes depict usual syntactical mistakes made by programmers while writing code. MuJava implements a do faster approach to mutation testing to save compilation time. This approach has been adopted primarily for object-oriented programs. The architecture of MuJava makes use of the Mutant Schemata Generation (MSG) approach. This approach works by creating one meta-mutant of all the mutant programs and requires only two compilations: compilation of the original program and compilation of the meta-mutant program. MuJava uses two types of mutant operators: Operators that change the structure of the program Operators that change the behavior of the program

The method-level operators implemented in MuJava. The operators considered for MuJava modify the expressions by inserting, replacing or deleting the primitive operators. MuJava contains six types of primitive operators. 1. Arithmetic operators 2. Relational operators 3. Conditional operators 4. Shift operators 5. Logical operators 6. Assignment operators Some of these operators have short-cut versions as well. Because of this, some operators are subdivided into binary, unary and short-cut versions. In all, there are 12 method-level operators in MuJava. Since MuJava has been created for Java programs, only operators used in Java are Considered for the creation of the tool

Table 1: Method-level operators in MuJava

3.2 Arithmetic Operators Arithmetic operators perform mathematical computations on all integers and floating-point numbers. The arithmetic operators supported in Java are: For binary: op + op (addition), op op (subtraction), op * op (multiplication), op / op (division) and op % op (modulus). For unary: + (plus indicating positive value), - (minus indicating negative value) For short-cut: op++ (post-increment), ++op (pre-increment), op-- (postdecrement), --op (pre-decrement) The arithmetic operators used in MuJava are explained below [9]: AORB / AORB U / AORS: Arithmetic Operator Replacement These operators replace binary, unary or short-cut arithmetic operators with other binary, unary or short-cut operators, respectively. AOIU / AOIS: Arithmetic Operator Insertion These operators insert unary or short-cut arithmetic operators, respectively. AODU / AODS: Arithmetic Operator Deletion These operators delete unary or short-cut arithmetic operators, respectively.

3.3 Relational Operators Relational operators compare the values of two operands. The relational operators supported in Java are: op > op (greater than), op >= op (greater than or equal to), op < op (less than), op <= op (less than or equal to), op == op (equal to) and op != op (not equal to). Since these operators require two operands, only replacement is allowed for these operators. The relational operator used in MuJava is explained below: ROR: Relational Operator Replacement This operator replaces relational operators with other relational operators.

3.4 Conditional Operators Conditional operators or bitwise operators perform computations on the binary values of its operands. These operators exhibit short-circuit behavior. The conditional operators supported in Java are: For binary: op && op (conditional AND), op || op (conditional OR), op & op (bitwise AND), op | op (bitwise OR) and ^ (bitwise XOR). For unary: !op (bitwise logical complement) The arithmetic operators in MuJava are explained below : COR: Conditional Operator Replacement This operator replaces binary conditional operators with other binary conditional operators. COI: Conditional Operator Insertion This operator inserts unary conditional operators. COD: Conditional Operator Deletion This operator deletes unary conditional operators.

3.5 Shift Operators Shift operators require two operands. They manipulate the bits of the first operand in the expression by either shifting them to the right or the left. The shift operators supported in Java are: op >> op (signed right shift), op << op (signed left shift) and op >>> op (unsigned right shift). The arithmetic operators in MuJava are explained below: SOR: Shift Operator Replacement This operator replaces binary shift operators with other binary shift operator.

3.6 Logical Operators Logical operators perform logical comparisons to produce a Boolean result for comparison statements. The logical operators supported in Java are: For binary: op & op (AND), op | op (OR) and op p (XOR). For unary: ~op (bitwise complement) The arithmetic operators in MuJava are explained below: LOR: Logical Operator Replacement This operator replaces binary logical operators with other binary logical operators. LOI: Logical Operator Insertion This operator inserts unary logical operators. LOD: Logical Operator Deletion This operator deletes unary logical operators.

3.7 Assignment Operators Assignment operators set the values of an operand. The short-cut assignment operators perform a computation on the right hand operand and then assign its value to the left hand operand. The assignment operators supported in Java are: For short-cut: op += op (addition assignment), op -= op (subtraction assignment), op *= op (multiplication assignment), op /= op (division assignment), op %= op (modulus assignment), op &= op (bitwise AND assignment), op |= op (bitwise OR assignment), op ^= op (bitwise XOR assignment), op <<= op (right shift assignment), op >>= op (left shift assignment), op >>>= op (unsigned right shift assignment). The arithmetic operators in MuJava are explained below: ASRS: Assignment Operator Replacement Short-cut This operator replaces short-cut assignment operators with other short-cut assignment operators.

3.8 Class Level Mutation Operator The operators considered for MuJava have been divided in four categories according to their usage in Object Oriented programming. The first 3 groups target features common to all OO languages. The last group includes features that are specific to Java. These groups are: 1. Encapsulation 2. Inheritance 3. Polymorphism 4. Java-specific Features Like method-level operators, the class-level operators make changes to the program syntax by inserting, deleting or modifying the expressions under test. Operators have been defined for each category. In all, there are 29 class-level operators in MuJava. These operators are explained in detail below. All code examples have been taken from.

3.8.1 Encapsulation Encapsulation in OOP deals with data hiding. It is the ability of an object to create a boundary around its data and methods. Encapsulation allows a programmer to define the access levels for various objects. For this purpose many access modifiers are used. Specifying the wrong access modifier can lead to incorrect results. There is only one mutation operator in MuJava, which deals with encapsulation. AMC: Access Modifier Change This operator replaces the access modifiers for various instance variables and methods in a program. This allows a tester to ensure that the correct level of accessibility is used in a program. Original Code public Stack s; Mutants private Stack s; protected Stack s;

3.8.2 Inheritance Inheritance allows the data and methods of one class (parent class) to be used by another class (child class). Inheritance promotes code reusability. Eight mutation operators have been developed for MuJava that deal with inheritance. Variable shadowing allows a child class to hide or shadow the variables in the parent class. This may cause incorrect variable to be accessed. Mutation operators IHD and IHI test this issue in OOP. IHD: Hiding variable deletion This operator deletes a hiding variable in a child class. In this way, the variable in the parent class can be accessed. Original Code class List { int size; } Mutants class List { int size; .. }

class Stack extends List { // int size; } IHI: Hiding variable insertion

class Stack extends List { int size; .. }

This operator inserts a hiding variable in a child class. It is reverse of IHD. Newly Defined and overriding methods in a subclass reference the hiding variable whereas inherited methods reference the hidden variable as before. Original Code class List { int size; } class Stack extends List { } Mutants class List { int size; . } class Stack extends List { int size; }

A child class can modify the behavior of its parent class by creating a method with the same name and arguments as the parent class. This is called method overriding. Mutation operators IOD, IOP and IOR are used to test issues related to method overriding in MuJava. IOD: Overriding method deletion This operator deletes the entire declaration of the overriding method in the child class. All references to this function then access the method in the parent class. Original Code class Stack extends List { void push(int a){ } } Mutants class Stack extends List { . // void push(int a){ . } }

 IOP: Overridden method calling position change Sometimes the overriding method in the child class may need to call the method it overrides in the parent class. It is necessary to test that the method in the parent class is called at the right point in the program otherwise it would cause the program to be in an incorrect state. IOR: Overridden method rename This operator is used to check if an overriding method adversely affects other methods. It renames the method being overridden in the parent class so that the overriding method in the child class doesnt affect the method in the parent class. The super keyword allows a programmer to access the member variables and functions of the parent class when using variable shadowing or method overriding. Mutation operators ISI and ISD are used to test issues regarding use of the super keyword in MuJava. ISI: super keyword insertion This operator inserts the super keyword so that any references to the variable or Method go to the overridden method or variable.

 ISD: super keyword deletion This operator deletes the super keyword so that any references to the variable or Method go to the overriding method or variable. The constructors of the parent class are not inherited like other methods. Whenever an object of a child class is created, it automatically invokes the default constructor of the parents class before invoking its own constructor. The child class can also use the super keyword to invoke a specific parent class constructor. Mutation operator IPC is used to test this in MuJava. IPC: explicit call of a parents constructor deletion This operator deletes calls to the parent classs constructor. This causes the default Constructor of the parent class to be called.

3.8.3 Polymorphism Polymorphism allows objects to react differently to the same method. It is implemented by having many methods with the same name. Ten mutation operators have been developed for MuJava that deal with polymorphism. PNC: new method call with child class type This operator changes the constructor used to instantiate an object i.e. it changes the type with which the object is instantiated. It makes the object reference refer to an object of a different type than that with which it is declared Original Code Parent a; a = new Parent(); Mutants Parent a; //a = new Child();

PMD: member variable declaration with parent class type This operator changes the declared type of an object reference to the parent of the Original declared type.

Original Code Child b; b = new Child();

Mutants // Parents b; b = new Child();

3.9 Method-level Mutation Operators There are 12 method-level operators in MuJava. To develop mutants for these operators the following strategy was used: 1. One type of mutation operator was chosen, e.g. Arithmetic mutation operators i.e. AOR, AOI, AOD. 2. A simple Java program (say ArithOper.java) was developed, which used this type of mutation operator, in this case, arithmetic operators. 3. MuJava was started and all the method-level operators were selected. 4. Using MuJava, mutants were created for ArithOper.java. 5. Another java program was created (say ArithOperTest.java) which contained the test cases for ArithOper.java. These test cases must exercise branch coverage for ArithOper.java. 6. ArithOperTest.java was run against ArithOper.java to determine how many mutants were killed and the mutation score was calculated. 7. Of the mutants still alive at the end of the execution, the equivalent mutants were determined manually.

8. The entire process from steps 1 to 7 was repeated to develop java programs for each type of method-level mutation operator i.e. relational, conditional, shift, logical and assignment. Using the above process, the data collected in the first instance indicates the number of mutants created for each method-level operator.

OBJECTIVE OF THE STUDY This thesis targets this issue by determining which mutation operators create mutants, which are more probable to create equivalent mutants. A tool called MuJava, Jody is used for this purpose. An empirical analysis has been carried out for this purpose, which helps determine which mutation operators develop more equivalent mutants.

EXPERIMANTAL SETUP & RESULTS The muJava system requires that the Java CLASSPATH be modified to include the muJava jar and the Java tools.jar files. Also, the general PATH variable must include the java bin directory; this is usually set automatically when java was installed, but not always. Then one GUI (Java applet) is used to generate mutants, the tester must create tests, and another GUI is used to run mutants. 1. Environment Settings for the muJava System There are three steps to setting up the environment for muJava, (1) CLASSPATH, (2) setting the config file, and (3) creating subdirectories. i. The Java CLASSPATH must include two Java jar files and one standard Java jar file. tools.jar is standard with Java compilers and is probably located in the "lib/" directory. The two Java files are mujava.jar and openjava2005.jar, which are downloaded from this site. One slightly awkward requirement is that the CLASSPATH must include the location of the classes under test when generating mutants, but NOT when running mutants. (If it does, no mutants can be killed.) This location is the "classes/" directory under where the mujava.config is stored.

In a DOS window, use the following command (assuming that classes is under C:\mujava):
set CLASSPATH=%CLASSPATH%;C:\mujava\mujava.jar;C:\mujava\openjava2005.jar;C:\j2sdk1.4.0_01\lib\to ols.jar;C:\mujava\classes

In a Cygwin window, use the following command:

To change your CLASSPATH permanently in Win2000 and WinXP, go to Start-settings-Control Panel. Double-click System, go to the Advanced tab, and choose Environment Variables. Edit the CLASSPATH variable or create a new variable if there is none. Add the full path to mujava.jar and openjava2005.jar to the CLASSPATH. In Unix, set the CLASSPATH environment variable. Assuming the jar files are stored in the home directory of user gpd:

CLASSPATH=$CLASSPATH:/home/gpd/mujava.jar:/home/gpd/openjava2 005.jar:/java1.4/j2sdk1.4.0_01/lib/tools.jar ; export CLASSPATH

Note that the syntax will vary under different shells, and it will be more convenient to put the command in your setup file such as .login, or .bashrc. ii. Next, modify the mujava.config file to point to a directory where you wish to store source Java files and muJava temporary files. The directory must be

the complete path (either Windows or Unix). For example, the config file may contain the line: MuJava_HOME=C:\home\gpd\exp. IMPORTANT: It is necessary to copy the config file to the directory you run the muJava system in. iii. Finally, create a directory structure for the muJava system in the $MuJava_HOME directory. Assuming your MuJava_HOME directory is called MuJava, the subdirectories should look like:

There should be four subdirectories, used in the following ways.

<TDMuJava_HOME\src directory for Java files to be tested directory for compiled classes of Java files from MuJava_HOME\classes MuJava_HOME\src MuJava_HOME\testset directory for test sets MuJava_HOME\result directory for generated mutants

You can create these subdirectories by hand or by using the muJava class "mujava.makeMuJavaStructure".
java mujava.makeMuJavaStructure

We have identified several potential problems with installing Java.

Java does not work with Java 1.5. It is important that the MuJava_HOME variable NOT have a trailing slash. This will confuse Java.

If you have a different version of the java compiler and the JVM, Java may get confused. This happens sometimes when a new application on your computer updates the JVM. If you have problems compiling or killing mutants, we suggest deleting all Java components and reinstalling the latest version.

If your tools.jar file is out of date (pre Java 1.4, we think), parts of Java may not work.

2. Generating Mutants with muJava Important note: You should run all commands in a directory that contains "mujava.config" i. Put the source files to test to MuJava_HOME\src directory. muJava does not check for compilation errors, so all Java files should compile correctly. If the Java file to test needs other Java files or class files, they should also be placed in MuJava_HOME\src. For example, suppose you want to test B, which is a child class of A. Then, you should put both A.java and B.java into MuJava_HOME\src. If the file has a package structure, you should store the entire package underneath MuJava_HOME\src. ii. Compile all the Java files in MuJava_HOME\src and copy the .class files into the MuJava_HOME\classes\ directory. iii. Start the GUI from the command line. Use it to generate mutants: java mujava.gui.GenMutantsMain iv. This command should bring a up a screen similar to the following:

i.

Select the files you want to mutate by clicking in the boxes on the left. Select the mutation operators you want to use by slecting their boxes. Then push RUN.

ii.

Note: The class mutation operators produce far fewer mutants. Also note that a number of status messages go to the command window, but not the GUI.

iii.

After mutants are generated, you can view the mutants in the "Class Mutants Viewer" and "Traditional Mutants Viewer" tabs, as shown in the following two figures.

iv.

v.

vi.

vii.

You may be interested in knowing where the mutant versions of the classes are stored. They are underneath the MuJava_HOME\result\ directory. The following example shows the directory Stack underneath result, with objectoriented mutants in class_mutants and traditional mutants in a separate directory.

3. Making a test set A testset in muJava is a Java file that contains executable test scripts. Each test is a method that contains a sequence of calls to methods in the class under test. Each test method returns a string result that is used to compare outputs of mutants with outputs of the original class. Each test method should start with the string "test". The test methods and the test class should have public access. Below is an example of a testset class for the class Stack. Its name is StackTest. StackTest contains two test case: test1() and test2(). The testset .class file should be in the directory MuJava_HOME\testset\. (By the way, although these test scripts are reminiscient of JUnit tests, Java does not accept JUnit tests. The first version of Java predated JUnit so we implemented a general test driver facility.)
public class StackTest { public String test1() { String result = ""; Stack obj = new Stack(); obj.push(2); obj.push(4); result = result + obj.isFull(); result = result + obj.pop(); return result; } public String test2() {

String result = Stack obj = new obj.push(5); obj.push(3); result = result result = result return result; } }

""; Stack();

+ obj.pop(); + obj.pop();

4. Running mutants. Run the mutants from another GUI. Start it with the following command:

java mujava.gui.RunTestMain

Note: Your CLASSPATH must not include MuJava_HOME\classes\. If it does, no mutants can be killed. You should see the following GUI. You can select which collection of mutants to run, and which testset to use. The "Class Mutants Viewer" and "Traditional Mutants Viewer" tabs will show the source view of the mutants. You can design tests to kill mutants by finding a live mutant, then analyzing the program to decide what input will kill it. Remember that between 5% to 20% of the mutants are typically equivalent.

CONCLUSION AND FUTURE WORK Detailed work has been carried out to determine which mutation operators contribute more towards creating equivalent mutants. The following results were obtained: Method-level operators: The majority of mutants were created for AOIS, ROR and LOI. Almost half of the mutants created for AOIS were found to be equivalent. However, majority of the mutants created for ROR and LOI were killed successfully. All the mutants created for COI failed the software. Class-level operators: Majority of mutants were created for PRV, JSI, IHI and IOD. Almost 50% of the mutants created for PRV, failed the software. Of the remaining mutants, almost 80% were killed and the remaining was found to be equivalent. Nearly 95% of the mutants created for PMD caused MuJava to fail. All the mutants created for JDC were found to be equivalent.

Many class-mutation operators such as IOR, ISI, etc, which created a very low number of mutants, were found to be equivalent. Various scenarios have been sketched out for method-level operator AOIS and class-level operators IHD, IHI, IOR, PRV, JID and JDC. These scenarios indicate situations which always create equivalent mutants. Looking at these results, different conclusions can be made for method-level mutation operators and class-level mutation operators. For method-level operators, AOIS is more probable to create equivalent mutants. For class-level operators, PRV usually creates the greatest number of mutants. But most of these mutants are simply program failures. Also, operators that create a very low number of mutants as compared to other class-level operators are also most likely to create equivalent mutants. For the future work of this project, the operators that create a larger number of equivalent mutants can be tweaked such that they do not create equivalent mutants. This can be done by using the scenarios sketched out in chapter 7. These scenarios can be incorporated in the algorithms implemented for the concerned operators. Thus, the creation of equivalent mutants can be avoided.

APPENDIX A USER GUIDE for MuJava MuJava requires 2 jar files and one config file. These files can be found at: http://ise.gmu.edu/~ofut/mujava/ These 2 files, (mujava.jar & adaptedOJ.jar) should be placed in the same directory. Lets assume the directory is C:\MuJava\. Before executing the software, the following steps should be taken: 1. Your computer must have a JRE installed and the Java path set. To learn how to do this refer to the java.sun.com website for details. 2. After this, firsts et the classpath using the following command: set CLASSPATH=%CLASSPATH%;C:\mujava\mujava.jar;C:\mujava\adaptedOJ.jar; C:\j2sdk1.4.0_01\lib\tools.jar;C:\mujava\classes This command allows you to run MuJava to create mutants. To run test cases a different path must be set which is explained later. 3. After this open the mujava.config file. You should see

MuJava_HOME=C:\ofut\mujava.

Change this to point to MuJava_HOME=C:\MuJava_Home. This command tells MuJava where to find the source files and the test programs when executing MuJava. 4. Now we need to make a directory structure for MuJava. All the source programs and test programs will be contained in this directory structure. This structure can be made either manually or by using the command: java mujava.makeMuJavaStructure The directory structure will now look like: C:\MuJava_Home\classes C:\MuJava_Home\testset C:\MuJava_Home\src C:\MuJava_Home\result Now, you will have 2 different directories; one will be C: \MuJava and the other will have C:\MuJava_Home\. The config file should be in the directory of MuJava.

Creating Mutants 1. Create your source file (say AccessorModifier.java) that you want to test in C:\MuJava_Home\src. 2. Compile this file using the DOS prompt and the c ommand javac AccessorModifier.java. This command will create a class file in the folder C:\MuJava_Home\src\. 3. Remove this file from the C:\MuJava_Home\src\ and copy it to the folder C:\MuJava_Home\classes\. 4. Now change the path in the dos prompt to point to C:\MuJava\ instead of C:\MuJava_Home\src\. 5. Now set the classpath using: set classpath=%classpath%;c:\MuJava\mujava.jar;c:\MuJava\adaptedOj.jar;c:\J2ee\jdk \lib\tools.jar;c:\MuJava_Home\classes; 6. Then use the following command to start MuJava:

java mujava.gui.GenMutantsMain 7. Now select the programs from the list shown for which you want to create the mutants. Also select the mutation operators for which you want to create mutants. Then press Generate. Whilst mutants are being generated, many messages will be sent to the command prompt in the background. The Generate button will be disabled. When all the mutants have been generated, this button will turn yellow again and the different mutants that have been generated can then be viewed.

APPENDIX B PROGRAME WRITTEN FOR Mujava (source code) Method-level Mutation Operators The programs written in this section were developed to create mutants for method-level operators in MuJava. They were written arbitrarly. Test cases written for all the programs are also included. ArithOper.java /* @author: Maryam Umar */ public class ArithOper { public String oper(int i){ int val = 1 + 2 + 3*i + (4 + 8)/3; String result = ""; result = Integer.toString(val); return result; } } ArithOperTest.java /* @author: Maryam Umar */

public class ArithOperTest{ public String test1(){ String result = ""; ArithOper obj = new ArithOper(); result = result + obj.oper(0); return result; } public String test2(){ String result = ""; ArithOper obj = new ArithOper(); result = result + obj.oper(1); return result; } public String test3(){ String result = "";

ArithOper obj = new ArithOper(); result = result + obj.oper(2); return result; } public String test4(){ String result = ""; ArithOper obj = new ArithOper(); result = result + obj.oper(3); return result; } } AssignOper.java /* @author: Maryam Umar */ public class AssignOper { public String operation(int i){

String result = ""; i += 10; result = Integer.toString(i); return result; } } AssignOperTest.java /* @author: Maryam Umar */ public class AssignOperTest{ public String test1(){ String result = ""; AssignOper obj = new AssignOper(); result = result + obj.operation(0); return result; }

public String test2(){ String result = ""; AssignOper obj = new AssignOper(); result = result + obj.operation(1);

//All Mutants Generator Code

package mujava;

import openjava.mop.*; import openjava.ptree.*;

import java.io.*; import mujava.op.*; import mujava.op.basic.*; import mujava.op.util.*; import mujava.util.Debug;

/** * <p>Generate all mutants</p> * <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma * @version 1.0 */

public class AllMutantsGenerator extends MutantsGenerator {

boolean existIHD = false;

String[] classOp; String[] traditionalOp;

public AllMutantsGenerator(File f) { super(f); classOp = MutationSystem.cm_operators; traditionalOp = MutationSystem.tm_operators; }

public AllMutantsGenerator(File f, boolean debug) { super(f, debug);

classOp = MutationSystem.cm_operators; traditionalOp = MutationSystem.tm_operators; }

public AllMutantsGenerator(File f, String[] cOP, String[] tOP) { super(f); classOp = cOP; traditionalOp = tOP; }

void genMutants() { if (comp_unit == null) {

System.err.println(original_file + " is skipped."); } ClassDeclarationList cdecls = comp_unit.getClassDeclarations(); if (cdecls == null || cdecls.size() == 0) return;

if (traditionalOp != null && traditionalOp.length > 0) { Debug.println("* Generating traditional mutants"); MutationSystem.clearPreviousTraditionalMutants(); MutationSystem.MUTANT_PATH MutationSystem.TRADITIONAL_MUTANT_PATH; CodeChangeLog.openLogFile(); genTraditionalMutants(cdecls); CodeChangeLog.closeLogFile(); } =

if (classOp != null && classOp.length > 0) { Debug.println("* Generating class mutants"); MutationSystem.clearPreviousClassMutants(); MutationSystem.MUTANT_PATH MutationSystem.CLASS_MUTANT_PATH; CodeChangeLog.openLogFile(); genClassMutants(cdecls); CodeChangeLog.closeLogFile(); } } =

void genClassMutants(ClassDeclarationList cdecls) {

genClassMutants1(cdecls); genClassMutants2(cdecls); }

void genClassMutants2(ClassDeclarationList cdecls) { for (int j=0; j<cdecls.size(); ++j) { ClassDeclaration cdecl = cdecls.get(j); if (cdecl.getName().equals(MutationSystem.CLASS_NAME)) { DeclAnalyzer mutant_op;

if (hasOperator(classOp, "IHD")) {

Debug.println(" Applying IHD ... ... "); mutant_op = new IHD(file_env, null, cdecl); generateMutant(mutant_op);

if (((IHD)mutant_op).getTotal() > 0) existIHD = true; }

if (hasOperator(classOp, "IHI")) { Debug.println(" Applying IHI ... ... "); mutant_op = new IHI(file_env, null, cdecl); generateMutant(mutant_op); }

if (hasOperator(classOp, "IOD")) { Debug.println(" Applying IOD ... ... "); mutant_op = new IOD(file_env, null, cdecl); generateMutant(mutant_op); }

if (hasOperator(classOp, "OMR")) { Debug.println(" Applying OMR ... ... "); mutant_op = new OMR(file_env, null, cdecl); generateMutant(mutant_op); }

if (hasOperator(classOp, "OMD")) { Debug.println(" Applying OMD ... ... "); mutant_op = new OMD(file_env, null, cdecl); generateMutant(mutant_op); }

if (hasOperator(classOp, "JDC")) { Debug.println(" Applying JDC ... ... "); mutant_op = new JDC(file_env, null, cdecl); generateMutant(mutant_op); } } }

void genClassMutants1(ClassDeclarationList cdecls) { for (int j=0; j<cdecls.size(); ++j) { ClassDeclaration cdecl = cdecls.get(j); if (cdecl.getName().equals(MutationSystem.CLASS_NAME)) { String qname = file_env.toQualifiedName(cdecl.getName()); try { mujava.op.util.Mutator mutant_op;

if (hasOperator(classOp,"AMC")) { Debug.println(" Applying AMC ... ... "); mutant_op = new AMC(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "IOR")) { Debug.println(" Applying IOR ... ... "); try { Class parent_class = Class.forName(qname).getSuperclass(); if (!(parent_class.getName().equals("java.lang.Object"))) {

String temp_str = parent_class.getName(); String result_str = ""; for (int k=0; k<temp_str.length(); k++) { char c = temp_str.charAt(k); if (c == '.') { result_str = result_str + "/"; } else { result_str = result_str + c; } }

File f = new File(MutationSystem.SRC_PATH, result_str + ".java"); if (f.exists()) { CompilationUnit[] parent_comp_unit = new CompilationUnit[1]; FileEnvironment[] parent_file_env = new FileEnvironment[1]; this.generateParseTree(f, parent_comp_unit, parent_file_env); this.initParseTree(parent_comp_unit, parent_file_env); mutant_op = new IOR(file_env, cdecl, comp_unit); ((IOR)mutant_op).setParentEnv(parent_file_env[0], parent_comp_unit[0]); comp_unit.accept(mutant_op); } } } catch (ClassNotFoundException e) {

System.out.println(" Exception at generating IOR mutant. File : AllMutantsGenerator.java "); } catch (NullPointerException e1) { System.out.print(" IOP ^^; "); } }

if (hasOperator(classOp, "ISD")) { Debug.println(" Applying ISD ... ... "); mutant_op = new ISD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "IOP"))

{ Debug.println(" Applying IOP ... ... "); mutant_op = new IOP(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "IPC")) { Debug.println(" Applying IPC ... ... "); mutant_op = new IPC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PNC")) {

Debug.println(" Applying PNC ... ... "); mutant_op = new PNC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PMD")) { Debug.println(" Applying PMD ... ... "); // if(existIHD){ mutant_op = new PMD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); //} }

if (hasOperator(classOp, "PPD"))

{ Debug.println(" Applying PPD ... ... "); // if(existIHD){ mutant_op = new PPD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); // } }

if (hasOperator (classOp, "PRV")) { Debug.println(" Applying PRV ... ... "); mutant_op = new PRV( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PCI")) { Debug.println(" Applying PCI ... ... "); mutant_op = new PCI( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PCC")) { Debug.println(" Applying PCC ... ... "); mutant_op = new PCC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PCD"))

{ Debug.println(" Applying PCD ... ... "); mutant_op = new PCD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JSD")) { Debug.println(" Applying JSC ... ... "); mutant_op = new JSD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JSI")) {

Debug.println(" Applying JSI ... ... "); mutant_op = new JSI( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JTD")) { Debug.println(" Applying JTD ... ... "); mutant_op = new JTD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JTI")) { Debug.println(" Applying JTI ... ... ");

mutant_op = new JTI( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JID")) { Debug.println(" Applying JID ... ... "); mutant_op = new JID( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "OAN")) { Debug.println(" Applying OAN ... ... "); mutant_op = new OAN( file_env, cdecl, comp_unit );

comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "EOA")) { Debug.println(" Applying EOA ... ... "); mutant_op = new EOA( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "EOC")) { Debug.println(" Applying EOC ... ... "); mutant_op = new EOC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op);

if (hasOperator(classOp, "EAM")) { Debug.println(" Applying EAM ... ... "); mutant_op = new EAM( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "EMM")) { Debug.println(" Applying EMM ... ... "); mutant_op = new EMM( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

} catch (ParseTreeException e ) { System.err.println( "Encountered errors during generating mutants." ); e.printStackTrace(); } } } }

/** * Compile mutants into bytecode */ public void compileMutants() {

if (traditionalOp != null && traditionalOp.length > 0) { Debug.println("* Compiling traditional mutants into bytecode"); MutationSystem.MUTANT_PATH MutationSystem.TRADITIONAL_MUTANT_PATH; super.compileMutants(); } =

if (classOp != null && classOp.length > 0) { Debug.println("* Compiling class mutants into bytecode"); MutationSystem.MUTANT_PATH MutationSystem.CLASS_MUTANT_PATH; super.compileMutants(); } } =

void genTraditionalMutants(ClassDeclarationList cdecls) { for(int j=0; j<cdecls.size(); ++j) { ClassDeclaration cdecl = cdecls.get(j);

if (cdecl.getName().equals(MutationSystem.CLASS_NAME)) { try { mujava.op.util.Mutator mutant_op; boolean AOR_FLAG = false;

if (hasOperator(traditionalOp, "AORB"))

{ Debug.println(" Applying AOR-Binary ... ... "); AOR_FLAG = true; mutant_op = new AORB(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); } if (hasOperator(traditionalOp, "AORS")) { Debug.println(" Applying AOR-Short-Cut ... ... "); AOR_FLAG = true; mutant_op = new AORS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "AODU"))

{ Debug.println(" Applying AOD-Normal-Unary ... ... "); mutant_op = new AODU(file_env, cdecl, comp_unit); ((AODU)mutant_op).setAORflag(AOR_FLAG); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "AODS")) { Debug.println(" Applying AOD-Short-Cut ... ... "); mutant_op = new AODS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "AOIU"))

{ Debug.println(" Applying AOI-Normal-Unary ... ... "); mutant_op = new AOIU(file_env, cdecl, comp_unit); ((AOIU)mutant_op).setAORflag(AOR_FLAG); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "AOIS")) { Debug.println(" Applying AOI-Short-Cut ... ... "); mutant_op = new AOIS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "ROR"))

{ Debug.println(" Applying ROR ... ... "); mutant_op = new ROR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "COR")) { Debug.println(" Applying COR ... ... "); mutant_op = new COR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "COD")) {

Debug.println(" Applying COD ... ... "); mutant_op = new COD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "COI")) { Debug.println(" Applying COI ... ... "); mutant_op = new COI(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "SOR")) { Debug.println(" Applying SOR ... ... ");

mutant_op = new SOR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "LOR")) { Debug.println(" Applying LOR ... ... "); mutant_op = new LOR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "LOI")) { Debug.println(" Applying LOI ... ... "); mutant_op = new LOI(file_env, cdecl, comp_unit);

comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "LOD")) { Debug.println(" Applying LOD ... ... "); mutant_op = new LOD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(traditionalOp, "ASRS")) { Debug.println(" Applying ASR-Short-Cut ... ... "); mutant_op = new ASRS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op);

} } catch (ParseTreeException e) { System.err.println( "Exception, during generating traditional mutants for the class " + MutationSystem.CLASS_NAME); e.printStackTrace(); } } } } }

Class Mutant Generator

package mujava;

import openjava.mop.*; import openjava.ptree.*; import java.io.*; import mujava.op.*; import mujava.op.util.*; import mujava.util.Debug;

/** * <p>Generate class mutants according to selected * class mutation operator(s) from gui.GenMutantsMain.

* * * *

The original version is loaded, mutated, and compiled. Outputs (mutated source and class files) are in the class-mutants folder. </p>

* <p> Currently available class mutation operators: * * * * * * * * * * (1) AMC: Access modifier change, (2) IHD: Hiding variable deletion, (3) IHI: Hiding variable insertion, (4) IOD: Overriding method deletion, (5) IOP: Overriding method calling position change, (6) IOR: Overriding method rename, (7) ISI: Super keyword insertion, (8) ISD: Super keyword deletion, (9) IPC: Explicit call to parent's constructor deletion, (10) PNC: New method call with child class type,

* * * * * * * * * * * * * * *

(11) PMD: Member variable declaration with parent class type, (12) PPD: Parameter variable declaration with child class type, (13) PCI: Type cast operator insertion, (14) PCC: Cast type change, (15) PCD: Type cast operator deletion, (16) PRV: Reference assignment with other compatible variable, (17) OMR: Overloading method contents replace, (18) OMD: Overloading method deletion, (19) OAN: Arguments of overloading method call change, (20) JTI: Java-specific this keyword insertion, (21) JTD: Java-specific this keyword deletion, (22) JSI: Java-specific static modifier insertion, (23) JSD: Java-specific static modifier deletion, (24) JID: Java-specific member variable initialization deletion, (25) JDC: Java-supported default constructor creation,

(26) EOA: Java-specific reference assignment and content assignment

replacement, * (27) EOC: Java-specific reference comparison and content assignment

replacement, * * * </p> * <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma * @version 1.0 */ (28) EAM: Java-specific accessor method change, (29) EMM: Java-specific modifier method change

public class ClassMutantsGenerator extends MutantsGenerator { boolean existIHD = false;

String[] classOp;

public ClassMutantsGenerator (File f) { super(f); classOp = MutationSystem.cm_operators; }

public ClassMutantsGenerator (File f, boolean debug) { super(f, debug); classOp = MutationSystem.cm_operators; }

public ClassMutantsGenerator (File f, String[] cOP) { super(f); classOp = cOP; }

/** * Verify if the target Java source and class files exist, * generate class mutants */ void genMutants() { if (comp_unit == null) { System.err.println(original_file + " is skipped.");

} ClassDeclarationList cdecls = comp_unit.getClassDeclarations();

if (cdecls == null || cdecls.size() == 0) return;

if (classOp != null && classOp.length > 0) { Debug.println("* Generating class mutants"); MutationSystem.clearPreviousClassMutants(); MutationSystem.MUTANT_PATH MutationSystem.CLASS_MUTANT_PATH; CodeChangeLog.openLogFile(); genClassMutants(cdecls); CodeChangeLog.closeLogFile(); } =

/** * Apply selected class mutation operators * @param cdecls */ void genClassMutants (ClassDeclarationList cdecls) { genClassMutants1(cdecls); genClassMutants2(cdecls); }

/** * Apply selected class mutation operators: IHD, IHI, IOD, OMR, OMD, JDC * @param cdecls

*/ void genClassMutants2 (ClassDeclarationList cdecls) { for (int j=0; j<cdecls.size(); ++j) { ClassDeclaration cdecl = cdecls.get(j); if (cdecl.getName().equals(MutationSystem.CLASS_NAME)) { DeclAnalyzer mutant_op;

if (hasOperator(classOp, "IHD")) { Debug.println(" Applying IHD ... ... "); mutant_op = new IHD(file_env, null, cdecl); generateMutant(mutant_op);

if ( ( (IHD)mutant_op).getTotal() > 0 ) existIHD = true; }

if (hasOperator(classOp, "IHI")) { Debug.println(" Applying IHI ... ... "); mutant_op = new IHI(file_env, null, cdecl); generateMutant(mutant_op); }

if (hasOperator(classOp, "IOD")) { Debug.println(" Applying IOD ... ... ");

mutant_op = new IOD(file_env, null, cdecl); generateMutant(mutant_op); }

if (hasOperator(classOp, "OMR")) { Debug.println(" Applying OMR ... ... "); mutant_op = new OMR(file_env, null, cdecl); generateMutant(mutant_op); }

if (hasOperator(classOp, "OMD")) { Debug.println(" Applying OMD ... ... "); mutant_op = new OMD(file_env, null, cdecl);

generateMutant(mutant_op); }

if (hasOperator(classOp, "JDC")) { Debug.println(" Applying JDC ... ... "); mutant_op = new JDC(file_env, null, cdecl); generateMutant(mutant_op); } } } }

/** * Apply selected class mutation operators:

* * *

AMC, IOR, ISD, IOP, IPC, PNC, PMD, PPD, PRV, PCI, PCC, PCD, JSD, JSI, JTD, JTI, JID, OAN, EOA, EOC, EAM, EMM

* @param cdecls */ void genClassMutants1(ClassDeclarationList cdecls) { for (int j=0; j<cdecls.size(); ++j) { ClassDeclaration cdecl = cdecls.get(j);

if (cdecl.getName().equals(MutationSystem.CLASS_NAME)) { String qname = file_env.toQualifiedName(cdecl.getName()); try

{ mujava.op.util.Mutator mutant_op;

if (hasOperator(classOp, "AMC")) { Debug.println(" Applying AMC ... ... "); mutant_op = new AMC(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "IOR")) { Debug.println(" Applying IOR ... ... "); try {

Class parent_class = Class.forName(qname).getSuperclass(); if ( !(parent_class.getName().equals("java.lang.Object")) ) { String temp_str = parent_class.getName(); String result_str = "";

for (int k=0; k<temp_str.length(); k++) { char c = temp_str.charAt(k); if (c == '.') { result_str = result_str + "/"; } else {

result_str = result_str + c; } }

File f = new File(MutationSystem.SRC_PATH, result_str + ".java"); if (f.exists()) { CompilationUnit[] parent_comp_unit = new CompilationUnit[1]; FileEnvironment[] parent_file_env = new FileEnvironment[1]; this.generateParseTree(f, parent_comp_unit, parent_file_env); this.initParseTree(parent_comp_unit, parent_file_env); mutant_op = new IOR(file_env, cdecl, comp_unit); ((IOR)mutant_op).setParentEnv(parent_file_env[0], parent_comp_unit[0]); comp_unit.accept(mutant_op);

} } } catch (ClassNotFoundException e) { System.out.println(" Exception at generating IOR mutant. AllMutantsGenerator.java "); } catch (NullPointerException e1) { System.out.print(" IOP ^^; "); } } file :

if (hasOperator(classOp, "ISD")) {

Debug.println(" Applying ISD ... ... "); mutant_op = new ISD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "IOP")) { Debug.println(" Applying IOP ... ... "); mutant_op = new IOP(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "IPC")) { Debug.println(" Applying IPC ... ... ");

mutant_op = new IPC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PNC")) { Debug.println(" Applying PNC ... ... "); mutant_op = new PNC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PMD")) { Debug.println(" Applying PMD ... ... "); //if(existIHD){

mutant_op = new PMD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); //} }

if (hasOperator(classOp, "PPD")) { Debug.println(" Applying PPD ... ... "); // if(existIHD){ mutant_op = new PPD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); //} }

if (hasOperator(classOp, "PRV"))

{ Debug.println(" Applying PRV ... ... "); mutant_op = new PRV( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PCI")) { Debug.println(" Applying PCI ... ... "); mutant_op = new PCI( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PCC")) {

Debug.println(" Applying PCC ... ... "); mutant_op = new PCC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "PCD")) { Debug.println(" Applying PCD ... ... "); mutant_op = new PCD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JSD")) { Debug.println(" Applying JSC ... ... ");

mutant_op = new JSD( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JSI")) { Debug.println(" Applying JSI ... ... "); mutant_op = new JSI( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JTD")) { Debug.println(" Applying JTD ... ... "); mutant_op = new JTD( file_env, cdecl, comp_unit );

comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JTI")) { Debug.println(" Applying JTI ... ... "); mutant_op = new JTI( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "JID")) { Debug.println(" Applying JID ... ... "); mutant_op = new JID( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op);

if (hasOperator(classOp, "OAN")) { Debug.println(" Applying OAN ... ... "); mutant_op = new OAN( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "EOA")) { Debug.println(" Applying EOA ... ... "); mutant_op = new EOA( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "EOC")) { Debug.println(" Applying EOC ... ... "); mutant_op = new EOC( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "EAM")) { Debug.println(" Applying EAM ... ... "); mutant_op = new EAM( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

if (hasOperator(classOp, "EMM")) { Debug.println(" Applying EMM ... ... "); mutant_op = new EMM( file_env, cdecl, comp_unit ); comp_unit.accept(mutant_op); }

} catch (ParseTreeException e ) { System.err.println( "Encountered errors during generating mutants." ); e.printStackTrace(); } } }

/** * Compile class mutants into bytecode */ public void compileMutants() { if (classOp != null && classOp.length > 0) { Debug.println("* Compiling class mutants into bytecode"); MutationSystem.MUTANT_PATH MutationSystem.CLASS_MUTANT_PATH; super.compileMutants(); } } } =

Compile Test Case

package mujava;

import mujava.MutationSystem; import mujava.util.Debug; import mujava.util.ExtensionFilter; import com.sun.tools.javac.Main; import java.io.*;

/** * <p>Description: </p> * <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma

* @version 1.0 */

public class compileTestcase { public static void main(String[] args) { Debug.setDebugLevel(3); File f = new File(MutationSystem.TESTSET_PATH); String[] s = f.list(new ExtensionFilter("java")); String[] pars = new String[2+s.length]; pars[0] = "-classpath"; pars[1] = MutationSystem.CLASS_PATH;

for (int i=0; i<s.length; i++)

{ pars[i+2] = MutationSystem.TESTSET_PATH + "/" + s[i]; } try { // result = 0 : SUCCESS, result = 1 : FALSE int result = Main.compile(pars,new PrintWriter(System.out)); if (result == 0) { Debug.println("Compile Finished"); } } catch (Exception e) { e.printStackTrace(); }

} } Exception Mutant Generator

package mujava;

import openjava.ptree.*; import java.io.*; import mujava.op.exception.*; import mujava.op.util.*; import mujava.util.Debug;

/** * <p>Generate exception-related mutants

* </p> * <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma * @version 1.0 */

public class ExceptionMutantsGenerator extends MutantsGenerator { String[] exceptionOp;

public ExceptionMutantsGenerator(File f) { super(f); exceptionOp = MutationSystem.em_operators; }

public ExceptionMutantsGenerator(File f, boolean debug) { super(f, debug); exceptionOp = MutationSystem.em_operators; }

public ExceptionMutantsGenerator(File f, String[] eOP) { super(f); exceptionOp = eOP; }

void genMutants() {

if (comp_unit == null) { System.err.println(original_file + " is skipped."); }

ClassDeclarationList cdecls = comp_unit.getClassDeclarations(); if (cdecls == null || cdecls.size() == 0) return;

if (exceptionOp != null && exceptionOp.length > 0) { MutationSystem.clearPreviousMutants(); MutationSystem.MUTANT_PATH MutationSystem.EXCEPTION_MUTANT_PATH; CodeChangeLog.openLogFile(); genExceptionMutants(cdecls); =

CodeChangeLog.closeLogFile(); } }

/** * Compile exception-related mutants into bytecode */ public void compileMutants() { if (exceptionOp != null && exceptionOp.length > 0) { Debug.println("* Compiling exception-related mutants into bytecode"); MutationSystem.MUTANT_PATH MutationSystem.EXCEPTION_MUTANT_PATH; super.compileMutants(); } =

void genExceptionMutants(ClassDeclarationList cdecls) { for (int j=0; j<cdecls.size(); ++j) { ClassDeclaration cdecl = cdecls.get(j);

if (cdecl.getName().equals(MutationSystem.CLASS_NAME)) { try { mujava.op.util.Mutator mutant_op; if (hasOperator(exceptionOp, "EFD")) {

Debug.println(" Applying EFD ... ... "); mutant_op = new EFD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(exceptionOp, "EHC")) { Debug.println(" Applying EHC ... ... "); mutant_op = new EHC(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(exceptionOp, "EHD")) { Debug.println(" Applying EHD ... ... ");

mutant_op = new EHD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(exceptionOp, "EHI")) { Debug.println(" Applying EHI ... ... "); mutant_op = new EHI(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator(exceptionOp, "ETC")) { Debug.println(" Applying ETC ... ... "); mutant_op = new ETC(file_env, cdecl, comp_unit);

comp_unit.accept(mutant_op); }

if (hasOperator(exceptionOp, "ETD")) { Debug.println(" Applying ETD ... ... "); mutant_op = new ETD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); } } catch (ParseTreeException e) { System.err.println( "Exception, during generating traditional mutants for the class " + MutationSystem.CLASS_NAME); e.printStackTrace(); }

} } } } Make MUJAVA Structure

package mujava;

import java.io.*;

/** * <p>Description: </p> * <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma

* @version 1.0 */

public class makeMuJavaStructure {

public static void main(String[] args) { MutationSystem.setJMutationStructure(); makeDir(new File(MutationSystem.SYSTEM_HOME)); makeDir(new File(MutationSystem.SRC_PATH)); makeDir(new File(MutationSystem.CLASS_PATH)); makeDir(new File(MutationSystem.MUTANT_HOME)); makeDir(new File(MutationSystem.TESTSET_PATH)); }

static void makeDir(File dir){

System.out.println("\nMake " + dir.getAbsolutePath() + " directory..."); boolean newly_made = dir.mkdir(); if(!newly_made){ System.out.println(dir.getAbsolutePath() + " directory exists already."); }else{ System.out.println("Making " + dir.getAbsolutePath() + " directory " + " ...done."); } } }

Mutants Generator

package mujava;

import openjava.mop.*; import openjava.ptree.*; import openjava.tools.parser.*; import openjava.ptree.util.*; import java.io.*; import java.util.*; import java.lang.reflect.Constructor; import mujava.op.util.*; import mujava.util.*; import com.sun.tools.javac.Main;

/** * <p>Generate mutants according to selected mutation * * * * operator(s) from gui.GenMutantsMain. The original version is loaded, mutated, and compiled. Outputs (mutated source and class files) are in the -mutants folder. </p>

* <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma * @version 1.0 */

public abstract class MutantsGenerator { //private boolean debug = false;

/** Java source file where mutation operators are applied to */ File original_file; // mutation file

/** mutation operators to apply */ String[] operators = null;

FileEnvironment file_env = null; CompilationUnit comp_unit = null;

public MutantsGenerator(File f) { this.original_file = f; initPrimitiveTypes(); }

public MutantsGenerator(File f, boolean debug_flag) { this(f); //debug = debug_flag; }

public MutantsGenerator(File f, String[] operator_list) { this(f); operators = operator_list; }

public MutantsGenerator(File f, String[] operator_list, boolean debug_flag) {

this(f, operator_list); //debug = debug_flag; }

/** * Generate and initialize parse tree from the original Java source file. * Generate mutants. Arrange and compile the original Java source file. * @return * @throws OpenJavaException */ public boolean makeMutants() throws OpenJavaException { Debug.print("-------------------------------------------------------\n"); Debug.print("* Generating parse tree. \n" );

generateParseTree(); Debug.print("..done. \n" ); //System.out.println("0"); Debug.print("* Initializing parse tree. \n" ); initParseTree(); Debug.print("..done. \n" ); //System.out.println("1"); Debug.print("* Generating Mutants \n" ); genMutants(); Debug.print("..done.\n" ); //System.out.println("2"); Debug.print("* Arranging original soure code. \n" ); arrangeOriginal(); //System.out.println("3"); compileOriginal();

Debug.print("..done. \n" ); Debug.flush(); return true; }

abstract void genMutants();

/*void generateMutant(OJClass mutant_op){ try { mutant_op.translateDefinition(comp_unit); }catch (Exception ex){ System.err.println("fail to translate " +mutant_op.getName()+" : " + ex); ex.printStackTrace(); } }*/

/** * Generate mutants from Java bytecode */ void generateMutant(DeclAnalyzer mutant_op) { try { mutant_op.translateDefinition(comp_unit); } catch (Exception ex) { System.err.println("fail to translate " + mutant_op.getName() + " : " + ex); ex.printStackTrace(); }

/** * Arrange the original source file into an appropriate directory */ private void arrangeOriginal() { if (comp_unit == null) { System.err.println(original_file + " is skipped."); }

ClassDeclarationList cdecls = comp_unit.getClassDeclarations(); for (int j=0; j<cdecls.size(); ++j) {

ClassDeclaration cdecl = cdecls.get(j); File outfile = null; try { outfile = new File(MutationSystem.ORIGINAL_PATH,

MutationSystem.CLASS_NAME + ".java"); FileWriter fout = new FileWriter( outfile ); PrintWriter out = new PrintWriter( fout ); MutantCodeWriter writer = new MutantCodeWriter( out ); writer.setClassName(cdecl.getName()); comp_unit.accept( writer ); out.flush(); out.close(); } catch ( IOException e ) {

System.err.println( "fails to create " + outfile ); } catch ( ParseTreeException e ) { System.err.println( "errors during printing " + outfile ); e.printStackTrace(); } } }

/** * Initialize parse tree * @throws OpenJavaException */ private void initParseTree() throws OpenJavaException

{ try { //System.out.println("OJSystem.env0 :" + OJSystem.env ); comp_unit.accept(new TypeNameQualifier (file_env)); //System.out.println("OJSystem.env1 :" + OJSystem.env ); MemberAccessCorrector corrector = new MemberAccessCorrector(file_env); // System.out.println("OJSystem.env2 :" + OJSystem.env ); comp_unit.accept(corrector); //System.out.println("OJSystem.env3 :" + OJSystem.env ); } catch (ParseTreeException e) { throw new OpenJavaException("can't initialize parse tree"); }

/** * Initialize parse tree * @param parent_comp_unit * @param parent_file_env */ void initParseTree(CompilationUnit[] parent_comp_unit,FileEnvironment[]

parent_file_env) { try { parent_comp_unit[0].accept(new TypeNameQualifier(parent_file_env[0])); MemberAccessCorrector corrector = new

MemberAccessCorrector(parent_file_env[0]); parent_comp_unit[0].accept(corrector);

} catch (ParseTreeException e) { System.err.println("Encountered errors during analysis."); e.printStackTrace(); } }

/** * Generate parse tree * @throws OpenJavaException */ private void generateParseTree() throws OpenJavaException { try

{ comp_unit = parse(original_file);

String pubcls_name = getMainClassName(file_env, comp_unit);

if (pubcls_name == null) { int len = original_file.getName().length(); pubcls_name = original_file.getName().substring(0, len-6); }

file_env = new FileEnvironment(OJSystem.env, comp_unit, pubcls_name); ClassDeclarationList typedecls = comp_unit.getClassDeclarations();

for (int j = 0; j < typedecls.size(); ++j)

{ ClassDeclaration class_decl = typedecls.get(j); OJClass c = makeOJClass(file_env, class_decl); OJSystem.env.record(c.getName(), c); recordInnerClasses(c); }

} catch (OpenJavaException e1) { throw e1; } catch (Exception e) { System.err.println("errors during parsing. " + e);

System.out.println(e); e.printStackTrace(); } }

/** * Evaluate whether a parse tree is successfully generated * @param f * @param comp_unit * @param file_env * @return */ boolean generateParseTree(File f, CompilationUnit[] comp_unit,

FileEnvironment[] file_env) { try

{ comp_unit[0] = parse(f); String pubcls_name = getMainClassName(file_env[0], comp_unit[0]); if (pubcls_name == null) { int len = f.getName().length(); pubcls_name = f.getName().substring(0, len-6); }

file_env[0] pubcls_name);

new

FileEnvironment(OJSystem.env,

comp_unit[0],

ClassDeclarationList typedecls = comp_unit[0].getClassDeclarations(); for (int j=0; j<typedecls.size(); ++j) { ClassDeclaration class_decl = typedecls.get(j);

if ( class_decl.getName().equals(MutationSystem.CLASS_NAME) ) { if ( class_decl.isInterface() || class_decl.getModifiers().contains(ModifierList.ABSTRACT) ) { return false; } }

OJClass c = makeOJClass(file_env[0], class_decl); OJSystem.env.record(c.getName(), c); recordInnerClasses(c); } }

catch (Exception e) { System.err.println("errors during parsing. " + e); e.printStackTrace(); return false; } return true; }

/** * Record inner-classes * @param c */ private static void recordInnerClasses( OJClass c ) {

OJClass[] inners = c.getDeclaredClasses(); for (int i = 0; i < inners.length; ++i) { OJSystem.env.record( inners[i].getName(), inners[i] ); recordInnerClasses( inners[i] ); } }

/** -> to move to OJClass.forParseTree() **/ private OJClass makeOJClass( Environment env, ClassDeclaration cdecl ) { OJClass result; String qname = env.toQualifiedName( cdecl.getName() ); Class meta = OJSystem.getMetabind( qname ); try

{ Constructor constr = meta.getConstructor( new Class[]{ Environment . class,OJClass . class, ClassDeclaration . class } ); Object[] args = new Object[]{env, null, cdecl}; result = (OJClass) constr.newInstance(args); } catch (Exception ex) { System.err.println("errors during gererating a metaobject for " + qname); ex.printStackTrace(); result = new OJClass( env, null, cdecl ); } return result; }

/** * Prepare a compilation unit * @param file * @return * @throws OpenJavaException */ private static CompilationUnit parse( File file ) throws OpenJavaException { Parser parser; try { parser = new Parser(new java.io.FileInputStream( file ) ); } catch ( java.io.FileNotFoundException e ) {

System.err.println( "File " + file + " not found." ); return null; }

CompilationUnit result; try { System.out.println( "File " + file ); result = parser.CompilationUnit( OJSystem.env ); } catch (ParseException e) { throw new OpenJavaException(" can't generate parse tree"); } catch (Exception e)

{ e.printStackTrace(); result = null; } return result; }

/** * * @param env * @param comp_unit * @return * @throws ParseTreeException */ private static String getMainClassName(FileEnvironment env,

CompilationUnit comp_unit) throws ParseTreeException {

ClassDeclaration cd = comp_unit.getPublicClass(); if (cd != null) { return cd.getName(); } else return null; }

/** * Compile mutants */

public void compileMutants() { File f = new File(MutationSystem.MUTANT_PATH);

String[] s = f.list(new MutantDirFilter());

for (int i=0; i<s.length; i++) { File target_dir = new File(MutationSystem.MUTANT_PATH + "/" + s[i]); String[] target_file = target_dir.list(new ExtensionFilter("java"));

Vector v = new Vector(); for (int j=0; j<target_file.length; j++) { v.add(MutationSystem.MUTANT_PATH + "/" + s[i] + "/" + target_file[j]);

String[] pars = new String[v.size()+2];

pars[0] = "-classpath"; pars[1] = MutationSystem.CLASS_PATH; for (int j=0; j<v.size(); j++) { pars[2+j] = v.get(j).toString(); } try { // result = 0 : SUCCESS, result = 1 : FALSE //int result = Main.compile(pars,new PrintWriter(new

FileOutputStream("temp"))); int result = Main.compile(pars);

if (result == 0) { Debug.print("+" + s[i] + " "); } else { Debug.print("-" + s[i] + " "); // delete directory File dir_name = new File(MutationSystem.MUTANT_PATH + "/" + s[i]); File[] mutants = dir_name.listFiles(); boolean tr = false;

for (int j=0; j<mutants.length; j++) { // [tricky solution] It can produce loop -_-;;

while (!tr) { tr = mutants[j].delete(); } tr = false; }

while (!tr) { tr = dir_name.delete(); } } } catch (Exception e) {

System.err.println(e); } } Debug.println(); }

/** * Compile original java source file */ private void compileOriginal() { String[] pars= { "-classpath", MutationSystem.CLASS_PATH, MutationSystem.ORIGINAL_PATH MutationSystem.CLASS_NAME + ".java"}; try + "/" +

{ // result = 0 : SUCCESS, result = 1 : FALSE //int result = Main.compile(pars,new PrintWriter(new

FileOutputStream("temp"))); Main.compile(pars); } catch (Exception e) { System.err.println(e); } }

private static void initPrimitiveTypes() { OJSystem.initConstants();

/** * Determine whether a string contain a certain operator * @param list * @param item * @return true if a string contain the operator, false otherwise */ protected boolean hasOperator (String[] list, String item) { for (int i=0; i<list.length; i++) { if (list[i].equals(item)) return true; }

return false; } } Test Executer

package mujava;

import java.io.*; import java.lang.annotation.Annotation; import java.lang.reflect.*; import java.net.URL; import java.net.URLClassLoader; import java.util.ArrayList; import java.util.HashMap;

import java.util.List; import java.util.Map; import java.util.Properties; import java.util.Vector; import mujava.test.*; import mujava.util.*;

import org.junit.*; import org.junit.internal.RealSystem; import org.junit.runner.JUnitCore; import org.junit.runner.Result; import org.junit.runner.notification.Failure; import org.junit.runners.*;

/**

* <p>Description: </p> * <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma * @update by Nan Li May 2012 integration with JUnit * @version 1.0 */

public class TestExecuter { Object lockObject = new Object();

//int TIMEOUT = 3000; int TIMEOUT; final int MAX_TRY = 100;

Class original_executer;

Object original_obj;

// instancitation of the test set class

volatile Object mutant_result;

Class mutant_executer;

// test set class for a mutant // test set object for a mutant

volatile Object mutant_obj;

Method[] testCases; volatile Method testcase;

String whole_class_name; String testSet; boolean mutantRunning = true;

//original test results

Map<String, String> originalResults = new HashMap<String, String>(); //results for mutants Map<String, String> mutantResults = null; //JUnit test cases List<String> junitTests = new ArrayList<String>(); //result of a test case Result result = null; //results as to how many mutants are killed by each test Map<String, String> finalTestResults = new HashMap<String, String>(); //results as to how many tests can kill each single mutant Map<String, String> finalMutantResults = new HashMap<String, String>();

public TestExecuter(String targetClassName) {

int index = targetClassName.lastIndexOf("."); if(index<0){ MutationSystem.CLASS_NAME = targetClassName; }else{ MutationSystem.CLASS_NAME targetClassName.substring(index+1,targetClassName.length()); } =

MutationSystem.DIR_NAME = targetClassName; MutationSystem.CLASS_MUTANT_PATH MutationSystem.MUTANT_HOME+"/"+targetClassName +"/"+MutationSystem.CM_DIR_NAME; MutationSystem.TRADITIONAL_MUTANT_PATH MutationSystem.MUTANT_HOME+"/"+targetClassName +"/"+MutationSystem.TM_DIR_NAME; = =

MutationSystem.EXCEPTION_MUTANT_PATH MutationSystem.MUTANT_HOME+"/"+targetClassName +"/"+MutationSystem.EM_DIR_NAME;

whole_class_name = targetClassName;

public void setTimeOut(int msecs){ TIMEOUT = msecs; }

public boolean readTestSet(String testSetName){ try{ testSet = testSetName; // Class loader for the original class

OriginalLoader myLoader = new OriginalLoader(); System.out.println(testSet); original_executer = myLoader.loadTestClass(testSet); original_obj = original_executer.newInstance(); set class if(original_obj == null){ System.out.println("Can't instantiace original object"); return false; } // initialization of the test

// read testcases from the test set class testCases = original_executer.getDeclaredMethods(); if(testCases==null){ System.out.println(" No test case exist "); return false; }

}catch(Exception e){ System.err.println(e); return false; } return true;

boolean sameResult(Object result1,Object result2){ if( !(result1.toString().equals(result2.toString())) ) return false; return true; }

public

TestResult

runClassMutants()

throws

NoMutantException,NoMutantDirException{

MutationSystem.MUTANT_PATH MutationSystem.CLASS_MUTANT_PATH; TestResult test_result = new TestResult(); runMutants(test_result, ""); return test_result; }

public

TestResult

runExceptionMutants()

throws

NoMutantException,NoMutantDirException{ MutationSystem.MUTANT_PATH MutationSystem.EXCEPTION_MUTANT_PATH; TestResult test_result = new TestResult(); runMutants(test_result, ""); return test_result; } =

public TestResult runTraditionalMutants(String methodSignature) NoMutantException,NoMutantDirException{

throws

MutationSystem.MUTANT_PATH MutationSystem.TRADITIONAL_MUTANT_PATH; String original_mutant_path = MutationSystem.MUTANT_PATH;

TestResult test_result = new TestResult();

if(methodSignature.equals("All method")){ try{ //setMutantPath(); //computeOriginalTestResults(); File f = new File(MutationSystem.TRADITIONAL_MUTANT_PATH, "method_list"); FileReader r = new FileReader(f);

BufferedReader reader = new BufferedReader(r); String readSignature = reader.readLine(); while(readSignature != null){

MutationSystem.MUTANT_PATH = original_mutant_path + "/" + readSignature; try{ runMutants(test_result, readSignature); }catch(NoMutantException e){ } readSignature = reader.readLine(); } reader.close(); }catch(Exception e){ System.err.println("Error TraditioanlMutantsViewerPanel.java"); in update() in

} }else{ MutationSystem.MUTANT_PATH methodSignature; runMutants(test_result, methodSignature); } return test_result; } /** * get the result of the test under the mutanted program * @deprecated * @param mutant * @param testcase * @throws InterruptedException */ = original_mutant_path + "/" +

void runMutants(Object mutant,Method testcase) throws InterruptedException{ mutantRunning = true; try{ // testcase execution mutant_result = testcase.invoke(mutant_obj,null); }catch(Exception e){ // execption occurred -> abnormal execution mutant_result = e.getCause().getClass().getName()+" : "

+e.getCause().getMessage(); } mutantRunning = false; synchronized(lockObject){ lockObject.notify(); } //throw new InterruptedException(); }

synchronized void waitUntilAtLeast(long timeOut) throws InterruptedException{ wait(timeOut); }

/** * get the mutants for one method based on the method signature * @param methodSignature * @return * @throws NoMutantDirException * @throws NoMutantException */ private String[] getMutants (String methodSignature) throws

NoMutantDirException, NoMutantException{

// Read mutants

//System.out.println("mutant_path: MutationSystem.MUTANT_PATH);

"

File f = new File(MutationSystem.MUTANT_PATH);

if(!f.exists()){ System.err.println(" There is no directory for the mutants of " + MutationSystem.CLASS_NAME); System.err.println(" MutationSystem.CLASS_NAME); throw new NoMutantDirException(); } Please generate mutants for " +

// mutantDirectories match the names of mutants String[] mutantDirectories = f.list(new MutantDirFilter());

if(mutantDirectories == null || mutantDirectories.length == 0){

if(!methodSignature.equals("")) System.err.println(" No mutants have been generated for the method " + methodSignature + " of the class" + MutationSystem.CLASS_NAME); else System.err.println(" No mutants have been generated for the class " + MutationSystem.CLASS_NAME); // System.err.println(" Please check if zero mutant is correct."); // throw new NoMutantException(); }

return mutantDirectories; }

/** * compute the result of a test under the original program

*/ public void computeOriginalTestResults(){

Debug.println("\n\n======================================== Generating Original Test Results

========================================"); try{ //initialize the original results to "pass" //later the results of the failed test cases will be updated for(int k = 0;k < testCases.length;k++){ Annotation[] annotations = testCases[k].getDeclaredAnnotations(); for(Annotation annotation : annotations) { //System.out.println("name: " + testCases[k].getName() + annotation.toString() + annotation.toString().indexOf("@org.junit.Test")); if(annotation.toString().indexOf("@org.junit.Test") != -1){

//killed_mutants[k]= ""; each test case

// At first, no mutants are killed by

originalResults.put(testCases[k].getName(), "pass"); junitTests.add(testCases[k].getName()); finalTestResults.put(testCases[k].getName(), ""); continue; } } }

JUnitCore jCore = new JUnitCore(); //result = jCore.runMain(new RealSystem(), "VMTEST1"); result = jCore.run(original_executer);

//get the failure report and update the original result of the test with the failures

List<Failure> listOfFailure = result.getFailures(); for(Failure failure: listOfFailure){ String nameOfTest = failure.getTestHeader().substring(0,

failure.getTestHeader().indexOf("(")); String testSourceName = testSet + "." + nameOfTest;

//System.out.println("failure message: " + failure.getMessage() + failure.getMessage().equals("")); String[] sb = failure.getTrace().split("\\n"); String lineNumber = ""; for(int i = 0; i < sb.length;i++){ if(sb[i].indexOf(testSourceName) != -1){ lineNumber = sb[i].substring(sb[i].indexOf(":") + 1, sb[i].indexOf(")")); } }

//put the failure messages into the test results if(failure.getMessage() == null) originalResults.put(nameOfTest, nameOfTest + ": " + lineNumber + "; " + "fail"); else{ if(failure.getMessage().equals("")) originalResults.put(nameOfTest, nameOfTest + ": " + lineNumber + "; " + "fail"); else originalResults.put(nameOfTest, nameOfTest + ": " + lineNumber + "; " + failure.getMessage()); }

} System.out.println(originalResults.toString());

// System.out.println(System.getProperty("user.dir")); // System.out.println(System.getProperty("java.class.path")); // System.out.println(System.getProperty("java.library.path"));

} catch(Exception e){ System.out.println(e.getMessage()); e.printStackTrace();

//

original_results[k]

e.getCause().getClass().getName()+"

"

+e.getCause().getMessage(); // // ); Debug.println("Result for " + testName + " : " +original_results[k] ); Debug.println(" [warining] " + testName + " generate exception as a result "

// ----------------------------------

}finally{ //originalResultFileRead(); } }

private TestResult runMutants(TestResult tr, String methodSignature) throws NoMutantException,NoMutantDirException{ try{

String[] mutantDirectories = getMutants(methodSignature);

int mutant_num = mutantDirectories.length; tr.setMutants();

for(int i = 0;i < mutant_num;i++){ // set live mutnats tr.mutants.add(mutantDirectories[i]); }

// result againg original class for each test case // Object[] original_results = new Object[testCases.length]; // list of the names of killed mutants with each test case //String[] killed_mutants = new String[testCases.length];

Debug.println("\n\n======================================== Executing Mutants ========================================"); for(int i = 0; i < tr.mutants.size(); i++){ // read the information for the "i"th live mutant String mutant_name = tr.mutants.get(i).toString(); finalMutantResults.put(mutant_name, "");

JMutationLoader mutantLoader = new JMutationLoader(mutant_name); //mutantLoader.loadMutant(); mutant_executer = mutantLoader.loadTestClass(testSet); mutant_obj = mutant_executer.newInstance(); Debug.print(" " + mutant_name);

try{ // Mutants are runned using Thread to detect infinite loop caused by mutation Runnable r = new Runnable(){ public void run(){ try{ mutantRunning = true;

//original test results

mutantResults = new HashMap<String, String>(); for(int k = 0;k < testCases.length;k++){ Annotation[] annotations = testCases[k].getDeclaredAnnotations(); for(Annotation annotation : annotations) { //System.out.println("name: " + testCases[k].getName() + annotation.toString() + annotation.toString().indexOf("@org.junit.Test")); if(annotation.toString().indexOf("@org.junit.Test") != 1){ //killed_mutants[k]= ""; // At first, no mutants are killed by each test case mutantResults.put(testCases[k].getName(), "pass"); continue; } }

JUnitCore jCore = new JUnitCore(); result = jCore.run(mutant_executer);

List<Failure> listOfFailure = result.getFailures(); for(Failure failure: listOfFailure){ String nameOfTest = failure.getTestHeader().substring(0, failure.getTestHeader().indexOf("(")); String testSourceName = testSet + "." + nameOfTest;

//System.out.println(testSourceName); String[] sb = failure.getTrace().split("\\n"); String lineNumber = ""; for(int i = 0; i < sb.length;i++){ //System.out.println("sb-trace: " + sb[i]);

if(sb[i].indexOf(testSourceName) != -1){ lineNumber sb[i].substring(sb[i].indexOf(":") + 1, sb[i].indexOf(")")); =

} } //get the line where the error happens /* String tempLineNumber = ""; if(failure.getTrace().indexOf(testSourceName) != -1){ tempLineNumber failure.getTrace().substring(failure.getTrace().indexOf(testSourceName) = +

testSourceName.length() + 1, failure.getTrace().indexOf(testSourceName) + testSourceName.length() + 5); System.out.println("tempLineNumber: tempLineNumber); " +

lineNumber tempLineNumber.indexOf(")"));

tempLineNumber.substring(0,

//System.out.print("LineNumber: " + lineNumber); } */

//get the test name that has the error and save the failure info to the results for mutants if(failure.getMessage() == null) mutantResults.put(nameOfTest, nameOfTest + ": " + lineNumber + "; " + "fail"); else if(failure.getMessage().equals("")) mutantResults.put(nameOfTest, nameOfTest + ": " + lineNumber + "; " + "fail"); else mutantResults.put(nameOfTest, nameOfTest + ": " + lineNumber + "; " + failure.getMessage()); }

System.out.println(mutantResults.toString()); mutantRunning = false; synchronized(lockObject){ lockObject.notify(); }

}catch(Exception e){ e.printStackTrace(); //System.out.println("e.getMessage()"); //System.out.println(e.getMessage()); } } };

Thread t = new Thread(r);

t.start();

synchronized(lockObject){ lockObject.wait(TIMEOUT); // Check out if a mutant is in infinite loop } if(mutantRunning){ //System.out.println("check point4"); t.interrupt(); mutant_result = "time_out: more than " + TIMEOUT + " seconds"; //mutantResults.put(nameOfTest, nameOfTest + ": " + lineNumber + "; " + failure.getMessage()); } }catch(Exception e){ mutant_result = e.getCause().getClass().getName()+" : "

+e.getCause().getMessage(); }

//determine whether a mutant is killed or not //update the test report boolean sign = false; for(int k = 0;k < junitTests.size();k++){ String name = junitTests.get(k); if(!mutantResults.get(name).equals(originalResults.get(name))){ sign = true; //update the final results by tests if(finalTestResults.get(name).equals("")) finalTestResults.put(name, mutant_name); else finalTestResults.put(name, finalTestResults.get(name) + ", " + mutant_name); //update the final results by mutants if(finalMutantResults.get(mutant_name).equals(""))

finalMutantResults.put(mutant_name, name); else finalMutantResults.put(mutant_name, finalMutantResults.get(mutant_name) + ", " + name); } } if(sign == true) tr.killed_mutants.add(mutant_name); else tr.live_mutants.add(mutant_name);

mutantLoader = null; mutant_executer = null; System.gc(); }

for(int i = 0;i < tr.killed_mutants.size();i++){ tr.live_mutants.remove(tr.killed_mutants.get(i)); } /* System.out.println(" Analysis of testcases "); for(int i = 0;i < killed_mutants.length;i++){ System.out.println(" test " + (i+1) + " kill ==> " + killed_mutants[i]); } */ }catch(NoMutantException e1){ throw e1; }catch(NoMutantDirException e2){ throw e2; }

/*catch(ClassNotFoundException e3){ System.err.println("[Execution 1] " + e3); return null; } */catch(Exception e){ System.err.println("[Exception 2]" + e); return null; } System.out.println("test report: " + finalTestResults); System.out.println("mutant report: " + finalMutantResults); return tr; }

void erase_killed_mutants(Vector v){ System.out.println("Deleting directories of killed mutants");

for(int i=0;i<v.size();i++){ System.out.print(v.get(i).toString()+" "); erase_directory(v.get(i).toString()); } }

void erase_directory(String mutant_name){ File mutant_dir = new

File(MutationSystem.MUTANT_PATH+"/"+mutant_name); File[] f = mutant_dir.listFiles(); boolean flag = false; for(int i=0;i<f.length;i++){ while(!flag){ flag = f[i].delete(); } flag = false;

while(!flag){ flag = mutant_dir.delete(); } } } Traditional Mutants Generator

package mujava;

import openjava.ptree.*; import java.io.*; import mujava.op.basic.*;

import mujava.op.util.*; import mujava.util.Debug;

/** * <p>Generate traditional mutants according to selected * * * * * * <p>Currently available traditional mutation operators: * * * * (1) AORB: Arithmetic Operator Replacement (Binary), (2) AORU: Arithmetic Operator Replacement (Unary), (3) AORS: Arithmetic Operator Replacement (Short-cut), (4) AODU: Arithmetic Operator Deletion (Unary), operator(s) from gui.GenMutantsMain. The original version is loaded, mutated, and compiled. Outputs (mutated source and class files) are in the traditional-mutants folder. </p>

* * * * * * * * * * * *

(5) AODS: Arithmetic Operator Deletion (Short-cut), (6) AOIU: Arithmetic Operator Insertion (Unary), (7) AOIS: Arithmetic Operator Insertion (Short-cut), (8) ROR: Rational Operator Replacement, (9) COR: Conditional Operator Replacement, (10) COD: Conditional Operator Deletion, (11) COI: Conditional Operator Insertion, (12) SOR: Shift Operator Replacement, (13) LOR: Logical Operator Replacement, (14) LOI: Logical Operator Insertion, (15) LOD: Logical Operator Deletion, (16) ASRS: Assignment Operator Replacement (short-cut)

* </p> * <p>Copyright: Copyright (c) 2005 by Yu-Seung Ma, ALL RIGHTS RESERVED </p> * @author Yu-Seung Ma

* @version 1.0 */

/* upsorn: 11/08/2009 - need to work on nesting * * * * */ (17) SID: If-Statement Deletion, (18) SFD: For-Statement Deletion, (19) SSD: Switch-Statement Deletion, (20) SWD: While-Statement Deletion

public class TraditionalMutantsGenerator extends MutantsGenerator { String[] traditionalOp;

public TraditionalMutantsGenerator(File f)

{ super(f); traditionalOp = MutationSystem.tm_operators; }

public TraditionalMutantsGenerator(File f, boolean debug) { super (f, debug); traditionalOp = MutationSystem.tm_operators; }

public TraditionalMutantsGenerator(File f, String[] tOP) { super(f); traditionalOp = tOP;

/** * Verify if the target Java source and class files exist, * generate traditional mutants */ void genMutants() { if (comp_unit == null) { System.err.println (original_file + " is skipped."); }

ClassDeclarationList cdecls = comp_unit.getClassDeclarations();

if (cdecls == null || cdecls.size() == 0) return;

if (traditionalOp != null && traditionalOp.length > 0) { Debug.println("* Generating traditional mutants"); MutationSystem.clearPreviousTraditionalMutants();

MutationSystem.MUTANT_PATH MutationSystem.TRADITIONAL_MUTANT_PATH;

CodeChangeLog.openLogFile();

genTraditionalMutants(cdecls);

CodeChangeLog.closeLogFile();

} }

/** * Compile traditional mutants into bytecode */ public void compileMutants() { if (traditionalOp != null && traditionalOp.length > 0) { try { Debug.println("* Compiling traditional mutants into bytecode"); String original_tm_path =

MutationSystem.TRADITIONAL_MUTANT_PATH; File f = new File(original_tm_path, "method_list");

FileReader r = new FileReader(f); BufferedReader reader = new BufferedReader(r); String str = reader.readLine();

while (str != null) { MutationSystem.MUTANT_PATH = original_tm_path + "/" + str; super.compileMutants(); str = reader.readLine(); } reader.close(); MutationSystem.MUTANT_PATH = original_tm_path; } catch (Exception e) { e.printStackTrace();

System.err.println("Error TraditionalMutantsGenerator.java"); } } }

at

compileMutants()

in

/** * Apply selected traditional mutation operators: * * AORB, AORS, AODU, AODS, AOIU, AOIS, ROR, COR, COD, COI, SOR, LOR, LOI, LOD, ASRS, SID, SWD, SFD, SSD

* @param cdecls */ void genTraditionalMutants(ClassDeclarationList cdecls) {

for (int j=0; j<cdecls.size(); ++j)

{ ClassDeclaration cdecl = cdecls.get(j); //take care of the case for generics String tempName = cdecl.getName(); if(tempName.indexOf("<") != -1 && tempName.indexOf(">")!= -1) tempName = tempName.substring(0, tempName.indexOf("<")) +

tempName.substring(tempName.lastIndexOf(">") + 1, tempName.length());

if (tempName.equals(MutationSystem.CLASS_NAME)) { try { mujava.op.util.Mutator mutant_op; boolean AOR_FLAG = false;

try

{ //generate a list of methods from the original java class //System.out.println("MutationSystem.MUTANT_PATH: MutationSystem.MUTANT_PATH); File f = new File(MutationSystem.MUTANT_PATH, "method_list"); FileOutputStream fout = new FileOutputStream(f); PrintWriter out = new PrintWriter(fout); " +

mutant_op comp_unit, out);

new

CreateDirForEachMethod(file_env,

cdecl,

comp_unit.accept(mutant_op); out.flush(); out.close(); } catch (Exception e) {

System.err.println("Error in writing method list"); return; }

if (hasOperator (traditionalOp, "AORB") ) { Debug.println(" Applying AOR-Binary ... ... "); AOR_FLAG = true; mutant_op = new AORB(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "AORS") ) { Debug.println(" Applying AOR-Short-Cut ... ... ");

AOR_FLAG = true; mutant_op = new AORS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "AODU") ) { Debug.println(" Applying AOD-Normal-Unary ... ... "); mutant_op = new AODU(file_env, cdecl, comp_unit); ((AODU)mutant_op).setAORflag(AOR_FLAG); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "AODS") ) {

Debug.println(" Applying AOD-Short-Cut ... ... "); mutant_op = new AODS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "AOIU") ) { Debug.println(" Applying AOI-Normal-Unary ... ... "); mutant_op = new AOIU(file_env,cdecl,comp_unit); ((AOIU)mutant_op).setAORflag(AOR_FLAG); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "AOIS") ) {

Debug.println(" Applying AOI-Short-Cut ... ... "); mutant_op = new AOIS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "ROR") ) { Debug.println(" Applying ROR ... ... "); mutant_op = new ROR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "COR") ) { Debug.println(" Applying COR ... ... ");

mutant_op = new COR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "COD") ) { Debug.println(" Applying COD ... ... "); mutant_op = new COD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "COI") ) { Debug.println(" Applying COI ... ... "); mutant_op = new COI(file_env, cdecl, comp_unit);

comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "SOR") ) { Debug.println(" Applying SOR ... ... "); mutant_op = new SOR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "LOR") ) { Debug.println(" Applying LOR ... ... "); mutant_op = new LOR(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op);

if (hasOperator (traditionalOp, "LOI") ) { Debug.println(" Applying LOI ... ... "); mutant_op = new LOI(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "LOD") ) { Debug.println(" Applying LOD ... ... "); mutant_op = new LOD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "ASRS") ) { Debug.println(" Applying ASR-Short-Cut ... ... "); mutant_op = new ASRS(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

// upsorn: First attempt: statement deletion operator // // // // // // } if (hasOperator (traditionalOp, "SDL") ) { Debug.println(" Applying SDL ... ... "); mutant_op = new SDL(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op);

/* if (hasOperator (traditionalOp, "SID") ) { Debug.println(" Applying SID ... ... "); mutant_op = new SID(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); }

if (hasOperator (traditionalOp, "SWD") ) { Debug.println(" Applying SWD ... ... "); mutant_op = new SWD(file_env, cdecl, comp_unit); comp_unit.accept(mutant_op); } */

} catch (ParseTreeException e) { System.err.println( "Exception, during generating traditional mutants for the class " + MutationSystem.CLASS_NAME); e.printStackTrace(); } } } } }

REFERENCES
Offutt, J. A. & Untch, R. H. (October 2000). Mutation 2000: Uniting the Orthogonal. Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdf Bakewell, G. (2010). Mutation-Based Testing Technologies Close the Quality Gap in Functional Verification for Complex Chip Designs. Retrieved January 18, 2011 from http://electronicdesign.com/article/eda/Mutation-Based-Testing-Technologies-Close-theQuality-Gap-in-Functional-Verification-for-Complex-Chip-Designs/4.aspx Offut, J. A. (June 1995). A Practical System for Mutation Testing: Help for the Common Programmer. Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/practical.pdf Usaola, M. & Mateo, P. (2010). Mutation Testing Cost Reduction Techniques: A Survey. IEEE Software, 27(3), 80-86. Retrieved from ABI/INFORM Global. (Document ID: 2012103051). Alexander, R. T.; Bieman, J. M.; Ghosh, S.; & Ji, B. (2002). Mutation of Java Objects. Retrieved from http://www.cs.colostate.edu/~bieman/Pubs/AlexanderBiemanGhoshJiISSRE02.pdf Nester - Free Software that Helps to do Effective Unit Testing in C#. Retrieved January 23, 2011 from http://nester.sourceforge.net/

 Kolawa, Adam (1999). Mutation Testing: A New Approach to Automatic ErrorDetection. Retrieved January 18, 2011 from http://www.parasoft.com/jsp/products/article.jsp?articleId=291

Eclipse Profiler plugin. http://sourceforge.net/projects/eclipsecolorer. M. Hafiz. User Manual of Muatation Engine. https://netfiles.uiuc.edu/ mhafiz/www/ResearchandPublications/MutationTestingTool.htm jclasslib byte-code viewer.EJ-Technologies, http://www.ej-technologies.com/ products/jclasslib/overview.html J. Offutt. Investigation of the software testing coupling effect. ACM Transactions on Software Engineering Methodology, vol. 1, pp. 3-18. J. Offutt, Ammei Lee, G. Rothermel, R. Untch, and C. Zapf. An experimental determination of sufficient mutation operators. ACM Transaction on Software Engineering Methodology, 5(2):99-118, April 1996. http://javabdd.sourceforge.net/ Sourceforge. Metrics, http://metrics.sourceforge.net/. accessed May 2010. Sourceforge. Jumble. http://jumble.sourceforge.net/, 2007. accessed May 2010. P. Trinidad, D. Benavides, A. Ruiz-Corts, S. Segura, and A.Jimenez. Fama framework. In 12th Software Product Lines Conference (SPLC), 2008. J. Whaley. Javabdd. http://javabdd.sourceforge.net/. accessed May 2010.