Scribd
Upload
95 views25 pages

Mutation Testing: Stuart Anderson

Mutation testing involves making small changes or mutations to a program and using test cases to determine if they can detect the changes, called killing the mutants. It is used to evaluate test adequacy by measuring the percentage of mutants killed and can provide an estimate of residual defects in a program. Common types of mutations include value, decision, and statement mutations that model errors like changing variables or conditions. Mutation testing aims to model real defects and can be automated using mutation testing tools.

Uploaded by

Sri Ram
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
95 views25 pages

Mutation Testing: Stuart Anderson

Mutation testing involves making small changes or mutations to a program and using test cases to determine if they can detect the changes, called killing the mutants. It is used to evaluate test adequacy by measuring the percentage of mutants killed and can provide an estimate of residual defects in a program. Common types of mutations include value, decision, and statement mutations that model errors like changing variables or conditions. Mutation testing aims to model real defects and can be automated using mutation testing tools.

Uploaded by

Sri Ram
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 25

Mutation Testing

Stuart Anderson

Stuart Anderson

Mutation Testing

c 2011

Overview
Mutation testing is a structural testing method, i.e. we use the structure of the code to guide the test process. We cover the following aspects of Mutation Testing: What is a mutation? What is mutation testing? When should we use mutation testing? Mutations Examples Mutation testing tools
Stuart Anderson Mutation Testing c 2011

What is a mutation?
A mutation is a small change in a program. Such small changes are intended to model low level defects that arise in the process of coding systems. Ideally mutations should model low-level defect creation.

Stuart Anderson

Mutation Testing

c 2011

What is Mutation Testing?


Mutation testing is a structural testing method aimed at assessing/improving the adequacy of test suites, and estimating the number of faults present in systems under test. The process, given program P and test suite T , is as follows: We systematically apply mutations to the program P to obtain a sequence P1, P2,... Pn of mutants of P . Each mutant is derived by applying a single mutation operation to P . We run the test suite T on each of the mutants, T is said to kill mutant Pj if it detects an error. If we kill k out of n mutants the adequacy of T is measured by the quotient k/n. T is mutation adequate if k = n. One of the benets of the approach is that it can be almost completely automated.
Stuart Anderson Mutation Testing c 2011

When should we use mutation testing?


Structural test suites are directed at identifying defects in the code. One goal of mutation testing is to assess or improve the ecacy of test suites in discovering defects. When we are carrying out structural testing we are worried about defects remaining in the code. Often we are keen to measure the Residual Defect Density (RDD) in the program P under test. The Residual Defect Density is usually measured in defects per thousand lines of code. Advocates of mutation testing argue that it can provide us with an estimate of the RDD of a program P that has satised all the tests in a test suite T.
Stuart Anderson Mutation Testing c 2011

Using Mutation Testing to Estimate the RDD


We want to estimate the RDD of Program P given that it has satised all the tests in test suite T . We follow the procedure: Suppose we have an estimate r of the RDD of programs produced by our development process before they are subject to test (this could be gathered using production data and eld experience, or it could be based on the number of faults our tests have already detected). Generate n mutants of the program P . Test each mutant with the test suite T . Find the number, k , of mutants that are killed by T . To yield a non-zero RDD we need to test enough mutants to ensure that 0 < k < n. Use r. (n k )/k as the estimate for the RDD of the tested program. k/n is a measure of the adequacy of T in nding defects in P .
Stuart Anderson Mutation Testing c 2011

Slide 5: Using Mutation Testing to Estimate the RDD


Alternative non-RDD-based approach in P&Y, p.322.

Assumptions
The validity of this rests on many assumptions: That mutations are a good model for defects. That defects are usually independent That the construction of T is not inuenced by knowledge of the mutation process (i.e. we do not use knowledge of the mutation process to build tests that are better at nding defects generated by mutations than normal defects). If we are interested in making condent estimates of very low RDDs we will need very large numbers of mutants. For example, if our development process left us with 10 defects per kLoc before test and we want to be condent our RDD after test is lower that 0.1 per kLoC then we need to test many mutants to be condent of such an RDD estimate.
Stuart Anderson Mutation Testing c 2011

An Approach to Mutation
Ideally we need systematically to apply mutations to the program under test. This would involve some criterion of applicability. Usually we consider mutation operators in the form of rules that match a context and create some systematic mutation of the context to create a mutant. The simple approach to coverage is to consider all possible mutants but this may create a very large number of mutants (in the case of estimating RDDs the value and condence required of the estimated RDD would control the number of mutants to be generated). Mutation testing is best supported by tools because of the potentially very large numbers of mutations to be generated during testing.
Stuart Anderson Mutation Testing c 2011

Kinds of Mutation
Value Mutations: these mutations involve changing the values of constants or parameters (by adding or subtracting values etc), e.g. loop bounds being one out on the start or nish is a very common error. Decision Mutations: this involves modifying conditions to reect potential slips and errors in the coding of conditions in programs, e.g. a typical mutation might be replacing a > by a < in a comparison. Statement Mutations: these might involve deleting certain lines to reect omissions in coding or swapping the order of lines of code. There are other operations, e.g. changing operations in arithmetic expressions. A typical omission might be to omit the increment on some variable in a while loop. A wide range of mutation operators is possible...
Stuart Anderson Mutation Testing c 2011

Outts Mutations for Inter-Class Testing

Stuart Anderson

Mutation Testing

c 2011

10

Value Mutation
Here we attempt to change values to reect errors in reasoning about programs. Typical examples are: Changing values to one larger or smaller (or similar for real numbers). Swapping values in initialisations. The commonest approach is to change constants by one in an attempt to generate a one-o error (particularly common in accessing arrays). Coverage criterion: Here we might want to perturb all constants in the program or unit at least once or twice.

Stuart Anderson

Mutation Testing

c 2011

Example

11

Value Mutation

Stuart Anderson

Mutation Testing

c 2011

12

Decision Mutation
Here again we design the mutations to model failures in reasoning about conditions in programs. As before this is a very limited model of programming error really modelling slips in coding rather than a design error. Typical examples are: Modelling one-o errors by changing < to <= or vice versa (this is common in checking loop bounds). Modelling confusion about larger and smaller, so changing > to < or vice versa. Getting parenthesisation wrong in logical expressions e.g. mistaking precedence between && and || Coverage Criterion: We might consider one mutation for each condition in the program. Alternatively we might consider mutating all relational operators (and logical operators e.g. replacing || by && and vice versa)
Stuart Anderson Mutation Testing c 2011

Example

13

Decision Mutation

Stuart Anderson

Mutation Testing

c 2011

14

Statement Mutation
Here the goal is primarily to model editing slips at the line level these typically arise when the developer is cutting and pasting code. The result is usually omitted or duplicated code. In general we might consider arbitrary deletions and permutations of the code. Typical examples include: Deleting a line of code Duplicating a line of code Permuting the order of statements. Coverage Criterion: We might consider applying this procedure to each statement in the program (or all blocks of code up to and including a given small number of lines).
Stuart Anderson Mutation Testing c 2011

Example

15

Statement Mutation

Stuart Anderson

Mutation Testing

c 2011

16

Observations
Mutations model low level errors in the mechanical production process. Modelling design errors is much harder because they involve large numbers of coordinated changes throughout the program. Ensuring test sets satisfy coverage criteria are often enough to ensure they kill mutants (because mutants often do not make sense and so provoke a failure if they are ever executed). Black-box test sets are poorer at killing mutants wed expect this because black-box tests are driven more by the operational prole than by the need to cover statements. We could see mutation testing as a way of forcing more diversity on the development of test sets if we use a black-box approach as our primary test development approach.
Stuart Anderson Mutation Testing c 2011

17

Concepts from the literature


Syntactic vs semantic size of a mutant the size the source change a mutant involves, versus the size of its eect on program behaviour. It has been hypothesised that mutation operators which produce semantically small faults are better (because semantically large faults will be caught by most tests). Justication for elimination of certain types of mutation. Competent programmer hypothesis the program under test is close to the correct program. So exploring the space of small mutations will lead us to that program. Coupling eect hypothesis tests for detecting simpler faults will be sucient also for detecting more complex faults. So even though many faults are a product of logical errors with wide consequences in the code, small mutants will lead to recognition of these faults.
Stuart Anderson Mutation Testing c 2011

18

Mutation Testing Tools


There is a range of possible mutation tools. Recently Outt and others have created MuJava, a tool for creating Java mutants. MuJava: An Automated Class Mutation System, Yu-Seung Ma, Je Outt and Yong-Rae Kwon. Journal of Software Testing, Verication and Reliability, 15(2):97-133, June 2005. http://dx.doi.org/10.1002/stvr.v15:2 Their system is designed specically to include a range of mutation operators that target OO languages in particular. They incorporate an ecient version of generating a metamutant that is capable of behaving like all mutants of the program (using Java reection to instantiate operators at execution time).
Stuart Anderson Mutation Testing c 2011

MuJava

19

Mutant Generation Interface

Stuart Anderson

Mutation Testing

c 2011

MuJava

20

Mutant Analysis Interface

Stuart Anderson

Mutation Testing

c 2011

MuJava

21

Test Execution Interface

Stuart Anderson

Mutation Testing

c 2011

22

Summary
Mutation testing can be a useful addition to the test process. It is laborious and really requires tool assistance if it is to be cost-eective. Improving Residual Defect Density estimates requires very large numbers of mutants if we are to have condence in the results. Object Orientation has a wide range of structural and operational mutants that are specic to objects. Tools like mu-Java use features of Java to enable the ecient generation and test of mutants. Even with ecient techniques execution times can be very slow for large numbers of mutants.
Stuart Anderson Mutation Testing c 2011

23

Required Readings
Textbook (Pezz` e and Young): Chapter 16, Fault-Based Testing MuJava: An Automated Class Mutation System, Yu-Seung Ma, Je Outt and Yong-Rae Kwon. Journal of Software Testing, Verication and Reliability, 15(2):97-133, June 2005. http://dx.doi.org/10.1002/stvr.v15:2 A. Jeerson Outt, Ammei Lee, Gregg Rothermel, Roland H. Untch, and Christian Zapf. 1996. An experimental determination of sucient mutant operators. ACM Trans. Softw. Eng. Methodol. 5, 2 (April 1996), 99-118. http://dx.doi.org/10.1145/227607.227610

Stuart Anderson

Mutation Testing

c 2011

You might also like