Software Testing

Lecture 2 – Clear Testing Goals (usually) Mean

Good Testing
SECTION 1 – THE GOALS OF
TESTING

2
Question – what goals might you have when
testing?

• Find the maximum number of bugs

• Know whether you have undiscovered bugs
• Comply with regulator-set standards
• Have a compelling defence in a court case
• Do any of the above in the minimum time and
cost
• …
Evaluation is the centrepiece of decision making!

• Should I buy new shoes?

– Are my current shoes comfortable, watertight and
fashionable?
• Should I change the way I test?
– Is my current testing finding bugs before the
customers do?

4
Question – How do you know if you’re achieving
those goals?

5
SECTION 2 – SOFTWARE TESTING
METRICS
Definition

Software Metrics
A system or standard of
measurement of software, which
gives a numerical value to some
property of interest.
Exhaustive testing is best
• Exhaustive coverage of possible inputs
– (and input sequences)
• Exhaustive coverage of possible context states
– E.g. operating system state
– E.g. hardware state
– E.g. (for reactive systems) external world state

8
Exhaustive testing is impossible
• But that doesn’t mean it’s not a useful goal

9
Definition
Coverage
A measure of the proportion of a
structure or a domain that a
program, a test case, a test suite
exercises.
What’s a desirable coverage criterion?

Path Coverage
A test suite achieve path coverage if it
results in each path through the program
being taken at least once.
 Example
Coverage of the red red path: a:=1
50% b:=a+2
(1/2 paths) If (b>3)

true false

Result:=b Result:=a
b:=a

b:=a
Exercise: minimum of test cases to cover all paths

public static int max(boolean considerA, int a, int b, int c) {

if (considerA && a>b) {
if (a>c)
return a;
else
return c;
} else
if (b>c)
return b;
else
return c;
}
Question - Is path coverage the same as exhaustive
testing?

• No
• Path coverage ensures that each control flow
path has been seen at least once
• But it doesn’t ensure that each path is tried
with all possible variable values
Path coverage is weaker than
exhaustive testing

if (client == TK){
System.out.println(“You may access the secret ST
files”)
} else {
System.out.println(“Sorry “ + client.toString() + “
you may cannot come in here”)
}

Path coverage met by

• TK object
• (any other object of suitable type)

Exhaustive test would eventually crash this with

• null

15
Path coverage is incredibly expensive

• E.g. 10 non-nested if statements is 1024 paths

– 11 is 2048…
• For a little more see
http://blog.parasoft.com/bid/66549/The-Prob
lem-with-Path-Code-Coverage

16
Lets scale our ambitions right down

Statement Coverage
A test suite achieves statement coverage if
it results in each statement in the program
being run at least once

17
Statement coverage is very weak

int abs(int x) {
x<0 ans = x;
if (x < 0)
ans = -x;
x>=0
return ans;
}
Statement coverage is very weak

int abs(int x) {
x<0 ans = x;
if (x < 1)
ans = -x;
x>=0
return ans;
}

Test case: abs(-4)

Achieved statement coverage
Statement coverage is very weak

int abs(int x) {
x<0 ans = -x;
if (x < 1)
ans = -x;
x>=0
return ans;
}

Test case: abs(-4)

Achieved statement coverage

Didn’t test half the inputs at all

A compromise position

Branch Coverage
A test suite achieves branch coverage if it
results in each branch in the program being
taken at least once

21
Branch Coverage Example

int abs(int x) {
x<0 int ans = x;
if (x < 1)
ans = -x;
x>=0
return ans;
}
Branch coverage often beats statement
coverage

int abs(int x) {
x<0 int ans = x;
if (x < 1)
ans = -x;
x>=0
return ans;
}

Test case 1: abs(-4)

Branch coverage often beats statement
coverage

int abs(int x) {
x<0
int ans = x;
if (x < 1)
ans = -x;
x>=0 return ans;
}

Test case 1: abs(-4)

Test case 2: abs(4)
Achieved branch coverage
Branch coverage often beats statement
coverage

int abs(int x) {
x<0
int ans = -x;
if (x < 1)
ans = -x;
x>=0 return ans;
}

Test case 1: abs(-4)

Test case 2: abs(4)
Achieved branch coverage
Question - Is branch coverage the same as path
coverage?

• No

• Path coverage ensures that each control flow

path has been seen at least once
• Branch coverage ensures that each individual
branch has been taken
• …but there may be combinations of branches
that haven’t been.
• Any questions so far?

27
Quick Quiz
1. Draw the CFG for this code int function(int x){

2. Find a test suite that achieves int y = 0;

statement coverage for this code if (x > 10)
y = 1;
3. Find a test suite that achieve
branch coverage for this code else
y = 2;
4. Can you find a test suite that
if (x > 5)
achieves statement coverage but
not branch coverage? y = y*2;
if (x > 0)
5. Is there a test suit that achieves
y = y*3;
branch coverage but not
statement coverage? return y;
}
Quick Quiz – An Answer
int function(int x){
X<=10 X>10
int y = 0;
if (x > 10)
y = 1;
else
X>5
y = 2;
X<=5 if (x > 5)
y = y*2;
if (x > 0)
X>0
y = y*3;
return y;
X<=0
}
Quick Quiz – An Answer
X<=10 X>10 int function(int x){
int y = 0;
if (x > 10)
y = 1;
else
X>5 y = 2;
X<=5 if (x > 5)
y = y*2;
if (x > 0)
X>0 y = y*3;
return y;
X<=0 }
function(20)
function(7)
Statement coverage but not branch
coverage
Quick Quiz – An Answer
X<=10 X>10 int function(int x){
int y = 0;
if (x > 10)
y = 1;
else
X>5 y = 2;
X<=5 if (x > 5)
y = y*2;
if (x > 0)
X>0 y = y*3;
return y;
X<=0 }
function(20)
function(-7)
Statement coverage and branch
coverage
Quick Quiz – An Answer
X<=10 X>10 int function(int x){
int y = 0;
if (x > 10)
y = 1;
else
X>5
y = 2;
X<=5 if (x > 5)
y = y*2;
if (x > 0)
X>0 y = y*3;
return y;
X<=0
}

Impossible to achieve branch coverage

without achieving statement coverage
A problem with branch coverage
if (averageMark >= 50 AND hardFailCount == 0){
grade = "pass";
} else {
grade = "fail";
}

Branch coverage met by:

• (50,0)  “pass”
• (49,1)  “fail”

33
A problem with branch coverage
if (averageMark >= 50 OR hardFailCount == 0){
grade = "pass";
} else {
grade = "fail";
}

Branch coverage met by:

• (50,0)  “pass”
• (49,1)  “fail”

34
Condition Coverage
• Each time one has a branching instruction (if,
for, while…) that contains one or several
conditions, each condition’s outcome (True or
False) is a possibility
• Condition coverage is the percentage of these
possibilities that were executed by a test suite.
if (a || b)

100% obtained with (a,b) = (true, false) and (false,true)

What does Condition Coverage require here?

if (averageMark >= 50 OR hardFailCount == 0){

grade = "pass";
} else {
grade = "fail";
}

Condition coverage met by:

• (50,0)  “pass” (true, true – don’t need)
• (49,1)  “fail” (false, false – don’t need)
• (50,1)  “pass” (true, false) wrong behaviour – we found a bug!
• (49,0)  “pass” (false, true) wrong behaviour – we found a bug!

36
 Modified Condition/Decision Coverage

• Consists of:
– 100% branch coverage
– 100% condition coverage
– Each entry/exit point is exercised
– Each condition affects the behaviour
independently
if (a || b)

DO-178B, Software Considerations in Airborne

Systems and Equipment Certification
What does MC/DC need?
if (averageMark >= 50 OR hardFailCount == 0){
grade = "pass";
} else {
grade = "fail";
}

Condition coverage met by:

• (50,0)  “pass” (true, true)
• (49,1)  “fail” (false, false)
• (50,1)  “pass” (true, false) wrong behaviour – we found a bug!
• (49,0)  “pass” (false, true) wrong behaviour – we found a bug!

38
Notes on MC/DC
• Significant because DO-178B mandates it for
avionics software
• Amman & Offutt book calls this ACC (with
variants GACC, CACC and RACC)
– Helpfully, neither “MC/DC” nor “ACC” appears in
their index
– Section 3.2.1 covers this (bottom of p108 in my
version)

39
• Any questions?

40
SECTION 3 – METRIC
SUBSUMPTION
Quick Quiz – An Answer
X<=10 X>10 int function(int x){
int y = 0;
if (x > 10)
y = 1;
else
X>5
y = 2;
X<=5 if (x > 5)
y = y*2;
if (x > 0)
X>0 y = y*3;
return y;
X<=0
}

Impossible to achieve branch coverage

without achieving statement coverage
Criteria Subsumption
• Criterion A subsumes criterion B if every test
suite that satisfies criterion A also satisfies
criterion B

• Branch coverage subsumes statement

coverage
– If a test suite achieves branch coverage it must
also achieve statement coverage
Criteria Subsumption – A Warning
• If criterion A subsumes criterion B it means all
test suites satisfying criterion A satisfy criterion
B
• However, it does not mean that all failures
found by a test suite satisfying criterion A will
be found be a test suite satisfying criterion B
• In general, test suites meeting subsuming
criteria do find more failures
– But might not
Criteria Subsumption Problem
int abs(int x) { Test suite 1:
int ans = x; abs(10)
if (x < 2) abs(-10)
ans = -x Achieves branch and
return ans; statement coverage
}
Test suite 2:
abs(1)
Achieves statement
coverage only
Criteria Subsumption Problem
int abs(int x) { Test suite 1:
int ans = x; abs(10) = 10 (passed)
if (x < 2) abs(-10) = 10 (passed)
ans = -x Achieves branch and
return ans; statement coverage
}
Test suite 2:
abs(1)
Achieves statement
coverage only
Criteria Subsumption Problem
int abs(int x) { Test suite 1:
int ans = x; abs(10) = 10 (passed)
if (x < 2) abs(-10) = 10 (passed)
ans = -x Achieves branch and
return ans; statement coverage
}
Test suite 2:
abs(1) = -1 (failed)
Achieves statement
coverage only
• Any questions?

48
SECTION 4 – METRICS AND
METHOD EVALUATION
Question – You’ve evaluated your testing
on a project, and you’ve got good coverage.
Does that mean that the approach you
used is a good method for all projects?

• No!
• Different projects will have different testing
requirements
• Rigorous evaluation requires confidence over
multiple projects
SECTION 5 – SOME METRICS AREN’T DIRECTLY
ABOUT TESTING, BUT HAVE IMPLICATIONS FOR
TESTING
So, you want to know how much testing you
have to do, and how much it will cost you…

• You can use software metrics to asses the size

of your system
Size-related metrics
• Lines of Code (Locs)
• Number of classes or header files
• Number of methods per class
• Number of attributes per class
• Size of compiled code
• Memory footprint
Examples: locs
• STRING_32 has 2766 locs
• EiffelBase has 65416 locs (in 251 classes)
• Apache has around 89000 locs
• EiffelStudio has 3 million locs
• Windows XP has 40 million locs
Examples: Classes and Header Files
• EiffelBase (Estudio 6.1) has 251 classes
• XCode for MacOSX Leopard has 14040 header
files
• Java 6 SDK has 7556 classes
Examples
• EiffelBase (Eiffel Studio 6.1) has 58 routines
per class on average (14541 routines in total,
251 classes)
• Gobo (in EiffelStudio 6.1) has 9.6 routine per
class on average (19214 routines in total, 1993
classes)
Not all parts of a program deserve equal testing
effort

• You can take parts of a program and calculate

complexity metrics
• Cyclomatic Complexity
• Number of states
• Coupling metrics
• Inheritance metrics

57
Cyclomatic Complexity
• The metric measures the number of possible paths in a
program (in a subroutine)
• How to count it:
– M=E-N+2P
(M= Cyclomatic Complexity, E number of edges in control-flow
graph, N number of nodes of the graph, P number of entries)
– M = 1+ number of if, from…until, select, for, while…
• Pitfalls:
– Difficult to understand for non-specialists
– What does it really measure?
Example

a:=1
E=5, N=5, P=1
b:=a+2
M=2
b>3
yes no
Result:=b Result:=a
b:=a

b:=a
Number of distinct states
• The metric measures the number of states in a
model of the program

• How to count it:

– Have a model (sic!) and count the number of nodes

• Pitfalls:
– Model generally non-trivial to make
Example

open closed

Distinct states: 2
Interdependence
• The metric measures the number of
interdependences between two modules
• How to count it:
– Count all direct references in one class to another
• Pitfalls:
– How to count dependencies that use inheritance?
– Are direct calls to be used?
Inheritance Metrics
• These metrics measures how much classes
inherit one from another.
• How to count them:
– Measure the number of parents or descendants
• Pitfalls:
– How does that relate to the quality of the code?
Inheritance Example: EiffelBase
Once you’ve got a test process in place,
you might want to measure how well
your development is going

65
Bugs per line of code
• The metric measures the number of bugs
found per line of code
• How to count it:
– Count the number of bugs divide it by the number
of lines of code
• Pitfalls:
– What if the testing is weak?
– Can you compare results for two programs?
Examples
• Firefox 1.5: .000077 (locs: 2’172’520, open
bugs: 169)
• Apache Http 2: .00065 (locs: 89000, open
bugs: 58)
• Any questions?

68
SECTION 6 – A USEFUL SUMMARY
OF SMAT SO FAR
Definitions (Dynamic) Testing Techniques
Testing Exploratory testing
Fault, Error and Failure Specification-based testing
Model-based testing
Fuzz testing
Classes of Testing Partition testing
Dynamic / static Boundary testing
Black / white / grey –box Coverage-based testing
Functional / non-functional Fault injection

Testing Scopes Coverage Criteria

Unit Statement
Integration Branch
System
Acceptance Condition
MC/DC
Regression
Exhaustive
• Any questions?

Now

By email
(I’ll answer in next lecture)

71