Software testing

““ Testing
Testing isis the
the process
process of
of
executing
executing aa program
program withwith the
the
intent
intent of
of finding
finding errors.”
errors.”
Glen
GlenMyers
Myers

1
Limits of software testing

• Testing can only prove the presence of bugs - never their absence

WHY???

2
Exhaustive testing

Assume that there are

520 possible different
execution flows. If
we execute one test
per millisecond, it
would take 3.170
years to test this
program!!
loop < 20x

3
ThePeculiarity of
Testing
• Testing is the process of executing a program with
the intent of finding an error
• A good test case has a high probability of finding an
as-yet undiscovered error
• A successful test is one that uncovers an as-yet
undiscovered error

4
 Who tests the software
better??

developer independent tester

Understands the Must learn about the
system system, but, will
but, will test attempt to break it
“gently”
and, is driven by
and, is driven by
“delivery” quality 5
Testing Principles

• All testing should be traceable to customer requirements

• Tests should be planned long before testing begins
• The Pareto principle applies to software testing as well
• Testing should begin in the small and progress toward
testing in the large
• To be most effective testing should be performed by a third
party

6
Testability = developing a
program be tested easily
• Operability - “The better it works, the more efficiently it can be
tested”
• Observability - the results are easy to see, distinct output is
generated for each input, incorrect output is easily identified
• Controllability - processing can be controlled, tests can be
automated & reproduced
• Decomposability - software modules can be tested independently
• Simplicity - no complex architecture and logic
• Stability - few changes are requested during testing
• Understandability - program is easy to understand

7
 Test case design

“Bugs lurk in corners

and congregate at
boundaries…”
Boris Beizer

OBJECTIVE to uncover errors

CRITERIA in a complete manner
CONSTRAINT with a minimum of effort and time
8
Test Cases and Test
Data
• Test data: Inputs which have been devised to test the system
• Test cases: Inputs to test the system and the predicted outputs from
these inputs if the system operates according to its specification

9
• Depending on the type of data input, we can identify three levels of
program correctness possible, probable, and absolute correctness

10
levels of program correctness

11
Possible correctness
 Obtaining correct output for some arbitrary input (single set).
 If the outcome is wrong, the program cannot possibly be correct.
 For example a multiply program: for input 2 and 3, if result is 6; is
possibly correct.
Probable correctness
 Obtaining correct output for a number of carefully selected inputs.
 If all potential problematic areas are checked in this way, the program
is probably correct.
 Try several values, including the obvious “problem” values such as
zero, largest negative, 1, -1 and so on.
Absolute correctness
 can be demonstrated only by a test that involves every possible
combination of inputs.
 Requires huge amount of time; it is therefore not practical {However
for some programs we can prove correctness mathematically}.
 Imagine the same above test for a 32 bit machine (how many
combinations are possible? (Hundreds of Millions).
12
 Approaches to tests

requirements

output

input

events

Black Box Testing White Box Testing

13
White box testing

Also called glass

box testing

CANNOT BE
EXHAUSTIVE!!!

14
Approaches to White Box Testing

• Exercise all independent paths in a program at least once

• Exercise all logical decisions on their true and false sides
• Execute all loops at their boundaries and within their operational
bounds
• Exercise internal data structures to assure their validity

15
 Why bother with white box testing?

• Black box testing: (1)Requirements fulfilled and (2)

Interfaces available and working
• Question: Why white box testing?

• Logical errors and incorrect assumptions are inversely

proportional to the probability that a program path will
be executed
• We often believe that a logical path is not likely to be
executed when, in fact, it may be executed on a
regular basis
• Typographical errors are random
16
Logical Errors

17
 Exhaustive testing

If-then-else There are 520=1014 (approx.)

possible paths!

loop < 20x

Selective Testing
18
 Selective testing

a selected path Selective:

Basis Path testing

Condition testing
Dataflow testing
Loop testing

19
Basis set

• Basis set = a set of paths that will execute all

statements and all conditions in a program at least
once (can have redundancies)
• The basis set is not unique
• Cyclomatic complexity defines the number of
independent paths in the basis set
• Goal: Define test cases for basis set

20
Basis path testing

Rationale I want to test all

Rationale:
statements of a
program at least
once.
Compute the cyclomatic complexity of the
program P - V(P)
Use it to determine how many test you have to
do:
V(P) provides an upper bound of tests that
must be executed to guarantee coverage of all
program statements
21
Basis path testing

1. We start with computing V(P):

1 Since V(P) = 4, there are four
paths
2
Path 1: 1,2,3,6,7,8
4 3
Path 2: 1,2,3,5,7,8
5 6
Path 3: 1,2,4,7,8
Path 4: 1,2,4,7,2,4…7,8
7
2. We derive test cases to exercise
8 these paths.
22
Documenting test cases

• Name
• Number
• Values for the inputs
• Expected outputs
• Short description (if needed)

23
Basis path testing notes

• you don’t need a flow chart, but

the picture will help when you
trace program paths
• count each simple logical test,
compound tests count as 2 or
more
• basis path testing should be
applied to critical modules

24
Example

Program
void
void f()
f() {{
if(count
if(count >> 10){
10){
fixed_count
fixed_count == fixed_count
fixed_count ++ count;
count;
done
done == 1;
1;
}} else
else if(count
if(count >5){
>5){
fixed_count
fixed_count --;
--;
}} else
else {{
fixed_count
fixed_count == count
count ** 4;
4;
}}
}}

25
 Flow Graph

void f()
11

if(count > 10)

22
if(count >5)
44 33
fixed_count = fixed_count + count;
fixed_count = count * 4; 55 66 done = 1;
77

88 fixed_count --;

return;

26
Derivation of the test cases

• V(P) = 3 (2 conditions + 1)
• Independent paths: 11

(a) 1, 2, 3, 8
(b) 1, 2, 4, 5, 7, 8 22

(c) 1, 2, 4, 6, 7, 8 44 33

• Test cases: 55 66
(a) count  13 77
(b) count  8 88
(c) count  2

27
 Example
PROCEDURE average
INTERFACE RETURNS average, total.input, total.valid;
INTERFACE ACCEPTS value, minimum, maximum;
TYPE value[1:100] IS SCALAR ARRAY;
TYPE average, total.input, total.valid;
minimum, maximum, sum IS SCALAR;
TYPE i IS INTEGER;
i = 1;
total.input = total.valid = 0;
1 sum = 0; 2
DO WHILE value[i] <> -999 AND total.input < 100
increment total.input by 1; 3
4
IF value[i] >= minimum AND value[1] <= maximum
6
5 THEN increment total.valid by 1;
sum = sum + value[i]
7 ELSE skip
8 ENDIF
increment i by 1;
9 ENDDO
IF total.valid >10
0
11
12 THEN average = sum/total.valid;
ELSE average = -999;
13 ENDIF
END average

28
 …Example
1

Flowgraph for average 2

3
10

4
12 11

13 5

Determine the:
6
1. Cyclomatic complexity
7
8
2. Independent paths
9 29
 Condition Testing

• Condition testing is a test case design method that

is based on testing all possible conditions in a
program
• A condition in a program may contain:
• Boolean operators and variables
• Parenthesis
• Relational operators
• Arithmetic expressions All of them can fail!!
• ….
Condition testing focus on testing
each condition in the program
30
Kinds of condition testing

• Branch testing: for each condition C, the true and the false
branches resulting from it and every simple condition in C need to be
executed at least once
• Domain testing: it uses information on the domain;
• given a relational condition in the program such as a<b I have to test it with
with values of a and b a<b, a=b, a>b and a and be VERY close
• given a boolean condition with n variables, I have to test it for each
combination of the values (that is 2n cases)
• There are other approaches, such as Branch and Relational operator
testing

31
Data Flow Testing

• Data flow testing identifies paths in the program that go from the
assignment of a value to a variable to the use of such variable, to
make sure that the var is properly used.
• The DU (Definition - Usage) testing strategy requires that each DU
chain is covered at least once

32
 Data Flow Testing

Considering x, there are two DU

[1]
[1] void
void k()
k() {{ paths:
[2]
[2] xx == 11;
11; (a) [2]-[4]
[3]
[3] if
if (p(cond1))
(p(cond1)) {{
[4] yy == xx ++ 1;
(b) [2]-[6]
[4] 1;
[5]
[5] }} else
else if if (q(cond2))
(q(cond2)) {{ We need therefore to derive test
[6]
[6] ww == xx ++ 3;
3; cases to match the following
[7]
[7] }} else
else {{
[8] ww == yy ++ 1;
conditions:
[8] 1;
[9]
[9] }} (a) k() is executed and
[10]}
[10]} p(cond1) is true
(b) k() is executed and
p(cond1) is false and
q(cond2) is true
33
Loops

• Cornerstone of every program

• Loops can lead to non-terminating programs
• Error in indexes of loops are very easy to occur

Loop Testing is a white-box testing

technique that focuses only on the validity
of loop constructs 34
 Loop testing

simple loop

nested concatenated unstructured

loops loops loops
35
Loop testing: simple loops

• Minimum conditions - simple loops

• 1. skip the loop entirely
• 2. only one pass through the loop
• 3. two passes through the loop
• 4. m passes through the loop m < n
• 5. (n-1), n, and (n+1) passes through the loop

where n is the maximum number of allowable passes

36
Nested loops

• Just extending simple loop testing: number of

tests increases geometrically
• Reduce the number of tests:
• start at the innermost loop; set all other loops to minimum values
• conduct simple loop test; add out of range or excluded values
• work outwards while keeping inner nested loops to typical
values
• continue until all loops have been tested

37
Concatenated loops

• Same strategies as simple loops if the loops counters are

independent
• Same strategies as nested loops if the loops counters
depend on each other:

int k;
for (k=0;k<10;k++)
k=0;k<10;k++
{
w();
if p(m) break;
}
for (;k<10;k++)
;k<10;k++ {
r();
}
38
 How do we select the strategy?
• The requirements of reliability of the system
• The resources available
• The kind of programs
• functional languages do not have loops…
• The tools available to support the testing process

39
Theoretical Foundation of Testing

• Mathematical model: Let P be a program, and let D and R denote its input (Data) and output
(Result) ranges. That is, D is a set of all data that can correctly be supplied to P, and the results of P’s execution,
if any, are elements of R.

TEST EXAMPLE : Statement Coverage Criteria: We can use above model to define test as: Select a test set T such
that, by executing P for each d (subset of D) in T, each elementary statement of P is executed at least once.

Test Case Design Examples

 Number of test cases in a test does not necessarily contribute: To test this wrong program

If x > y Then
max = x;
else
max = x;
endif;

 The test case set [x =3, y= 2; x= 2, y= 3] is able to detect the error. Whereas
 Test case set [x=3, y = 2; x =4, y = 3, x = 5, y = 1] is not, although it contains
more test cases.

40
Black box testing

requirements

output

input

events

41
 Categories of errors in black box
testing
• incorrect or missing functions
• interface errors
• errors in data structures or external database access
• performance errors
• initialization and termination errors

42
Techniques for black box
testing
• Graph-Based Testing Methods
• Equivalence Partitioning
• Boundary Value Analysis
• Comparison Testing

43
Graph-Based Testing

• Complex Systems are hard to understand

• They can be modeled as interacting objects
• Test can be performed to ensure that all the
required relations are in place

Pressman, p. 472
44
Equivalence Partitioning.. (to
design test cases)
• Input data and output results often fall into
different classes where all members of a class are
related

• Each of these classes is an equivalence partition

where the program behaves in an equivalent way
for each class member
• Test cases should be chosen from each partition

45
Equivalence Partitioning.. (to
design test cases)

46
 Defining Equivalence Classes

• Input condition is a range: one valid and two invalid classes are
defined
• Input condition requires specific value: one valid and two
invalid classes are defined
• Input condition is boolean: one valid and one invalid class are
defined

 Then define one test case for each equivalence class

47
Defining Equivalence Classes

Input or Valid Equivalence Invalid Equivalence

Output Event Classes Classes

Enter a 1–9 < 1, > 9

non-zero digit Letters and other non-
numeric characters

Enter the first First character is a First character is not a

letter of a capital letter letter
name First character is a
lower case letter

Draw a line From 1 to 4 inches long No line

Max. 4 inches Longer than 4 inches
Not a line (a curve)

48
 Defining Equivalence Classes

Characteristic Valid equivalence Invalid equivalence

class class
First char must be Letter Non-letter
alphabetic
Next three numerics All numeric One char non-
numeric
Range 100-500 In range (i) Above range
(ii) Below range

Equivalence class table for customer-acc-number

49
Boundary value testing

• Range a..b
• test cases: a, b, just above a, just below b
• Number of values:
• test cases: max, min, just below min, just above max
• Output bounds should be checked
• Boundaries of externally visible data structures shall be checked (e.g.
arrays)

50
 Error-Guessing

• Some people design the test cases by

intuition and experience.

• The basic idea is to enumerate a list of

possible errors and then write test cases
based on the list.

51
Testing Phase

What content should be included in a software test

plan?

- Testing tasks and assignments and

schedule
- Selected test strategy and test models
- Test methods and criteria
- Required test tools and environment
- Problem tracking and reporting
- Test cost estimation

52
Define Testing Strategy

• Select the testing technique that will be used for testing like Stress
Testing, Security Tests, performance Tests, Load Tests etc)
• Also consider the resources that will be employed for testing the
project. Resources are Hardware, Software, and Human.

53
 Example: Windows
Calculator
R-001:
The users should be able to add two numbers and view their result on the display.

Use Case: UC01 Add Two Numbers

Actors: User
Purpose: Add two numbers and view their result
Overview: The user inputs two numbers and then adds them and checks
the result, displayed on the screen.
Type: Primary, Real
Cross References: R-001
Typical Course of Events

Actor Action System Response

1.The actor opens the calculator. The keypad and
display screen appears.

2. The actor input the first number by clicking on the 3. The digit is displayed on the screen.
keypad or using keyboard.
4. The actor clicks or presses the “+” key.
5.The actor then adds the second number as (2). 6. The pressed digit is displayed on the screen.

7. The actor clicks the “=” key. 8. The sum of the two digits is displayed on the screen.

54
 Test Case

Test Case ID: T-101 Test Item: Add Numbers

Wrote By: (tester name) Junaid Documented Date: 26th April 2005
Test Type: Manual Test Suite#: NA
Product Name: Windows Calculator Release and Version No.: V 1.0
Test case description:
Add any two Numbers
Operation procedure:
Open Calculator
Press “1”
Press “+”
Press “2”
Press “=”
Pre-conditions: Post-conditions:
Calculator Opened Result Displayed
Inputs data and/or events: Expected output data and/or events:
1+2= 3
Required test scripts (for auto): NA
Cross References: (Requirements or Use Cases)R-001, UC-01

55
 Bug Report

Problem ID: B-101 current software name: Windows Calculator

Release and Version No.: V 1.0
Test type: Manual Reported by: Junaid
Reported date: 26th April 2005 Test case ID: T-101
Subsystem (or module name): Calculation Feature Name (or Subject): Add
Numbers
Problem type (REQ, Design, Coding,): Coding
Problem severity (Fatal, Major, Minor,): Major
Problem summary and detailed description:
On adding two numbers the result is not correct.
Cause analysis: NA
How to reproduce? Why Required:
Attachments: Nil

56
Debugging

 Debugging is not the same thing as testing.

 Testing is the attempt to identify errors in a program or module;
debugging is the task of identifying the causes of errors and
removing it
 In most circumstances it is better to totally separate the two
activities. The reasons being:
 Further test runs may shed some light on the nature of the
problem.
 Secondly, there is an inherent danger in fixing bugs on the
fly- you might create more errors.

57
 Bugs and Debugging

 The bug is the name of an error - anything goes wrong with our
program, when it runs.

 Debugging of course is to removing/fixing up the error.

 Error identification is generally simple, as we know the program
outcomes. Still not trivial (minor).
 If a test run produces unexpected outcome, this indicates there is an
error somewhere; unfortunately, a correct outcome does not
necessarily indicate absence of errors.
 However, careful choice of test data prevents certain kinds of errors
from persisting / continuing but it may still be possible for some errors
to escape and still the final outcome appears correct :
 There is no “magic formula” to ensure this should not happen.
 Once the error is spotted, next task is to locate its cause.
 Of course, if a systematic approach has been adopted throughout, this
should not be too difficult.
58
 Debugging Techniques
(1) Cause Analysis
 wrong variable used, or logic is wrong
 variable not initialized.

(2) Binary Chop

 If the cause analysis is not fruit full to look for the clues of the cause.
 Print or display statements can be inserted at the points in the program
to provide current values of key variables, which may help in finding
error.
 Problem is where to insert such statements. This is done by binary
choping; it works as:
(1) Identify a point in the module, where everything was known to be
correct.
(2) Identify the point where something wrong detected (may be test
result or error message)
(3) Identify a point roughly mid-way between 1-2 one, insert
statement here.
(4) See where error is before or after it.
(5) Repeat above to pin ort the affected area.
59
 One thing to be noted is that do not be misled by the apparent nature of
an error indication.
 If the driver of a car in your front gives a hand signal, (apart from
many other things/possibilities) only one thing can be absolutely sure;
that window is open.
 Examine the values of memory variables by introducing additional
PRINT and DISPLAY statements at selected parts.
 Approach the code with a fresh and open mind, even when looking at it
for nth time.
 Only the output of a program is relevant while analyzing a failure. The
error in the output is direct indicator of what has failed.
 Keep in mind, some time an error can escalate and show up at a point
far from where it was caused.
 Making a change in order to fix an error might cause other errors.

 Take care to document any alteration to the logic.

 There can be two classes of errors: logical and coding errors.
60
 Logical errors should be trapped and corrected before the program is
encoded and tested. At testing level it should be removed by
reexamining the logic at the highest possible level.
 Coding errors are results of misuse of the language facilities and
inadequate understanding of, or familiarity with, languages in use.
Tools Available

 Symbolic debuggers (nowadays) and program tracers: Allow single

stepping break point insertion, variable display, selected part execution,
etc.

 Traces allow high-level program display at running statements and

modules. Some also allow the facility of ‘log’ files.

61
Testing Strategies
 There are basically 3 levels of testing unit, integration, and system
testing. But the complete testing include the fourth one (i.e.)
acceptance testing.
 These different levels of testing attempt to detect different types of
faults.
 The relation of faults introduced in different phases and different levels
of testing, are shown as:
refers
Client needs Acceptance testing.

refers
Requirements System Testing

refers
Design Integration testing

refers
Code Unit Testing

 Unit testing: involves test of internal logic of modules.

62
 Unit Testing

Objective: Find differences between specified units and their imps.

Unit: component ( module, function, class, objects, …)
Unit test environment:

Driver Test cases

Test result

Unit under
Effectiveness?
test • Partitioning
• Code coverage

Stub Stub Dummy modules

63
Targets for Unit Test Cases

• Module interface
• Ensure that information flows properly into and out of the module
• Local data structures
• Ensure that data stored temporarily maintains its integrity during all steps in
an algorithm execution
• Boundary conditions
• Ensure that the module operates properly at boundary values established to
limit or restrict processing
• Independent paths (basis paths)
• Paths are exercised to ensure that all statements in a module have been
executed at least once
• Error handling paths
• Ensure that the algorithms respond correctly to specific error conditions

64
Common Computational Errors in
Execution Paths

• Misunderstood or incorrect arithmetic precedence

• Mixed mode operations (e.g., int, float, char)
• Incorrect initialization of values
• Precision inaccuracy and round-off errors
• Incorrect symbolic representation of an expression (int vs. float)

65
Drivers and Stubs for
Unit Testing

• Driver
• A simple main program that accepts test case data, passes such data to the
component being tested, and prints the returned results
• Stubs
• Serve to replace modules that are subordinate to (called by) the component
to be tested
• It uses the module’s exact interface, may do minimal data manipulation,
provides verification of entry, and returns control to the module
undergoing testing
• Drivers and stubs both represent overhead
• Both must be written but don’t constitute part of the installed software
product

66
Integration Testing

• Objectives:
• To expose problems arising from the combination
• To quickly obtain a working solution from components.
• Problem areas
• Internal: between components
• Invocation: call/message passing/…
• Parameters: type, number, order, value
• Invocation return: identity (who?), type, sequence
• External:
• Interrupts (wrong handler?)
• I/O timing
• Interaction

67
Regression Testing

• Regression Testing is defined as a type of software testing to

confirm that a recent program or code change has not
adversely affected existing features.
• Regression Testing is nothing but a full or partial selection of
already executed test cases that are re-executed to ensure
existing functionalities work fine.
• This testing is done to ensure that new code changes do not
have side effects on the existing functionalities. It ensures
that the old code still works once the latest code changes
are done.

68
 Why Regression Testing?

• There is a need for regression testing whenever the code is

changed, and you need to determine whether the modified code
will affect other parts of the software application.

• Moreover, regression testing is needed when a new feature is

added to the software application.

• Regression tests may also be performed when a functional or

performance defect/issue is fixed.

69
 When can we perform Regression
Testing?

• New functionality is added to the application: This happens when new features or
modules are created in an app or a website. The regression is performed to see if the
feature is working properly.

• In case of change requirement: When any significant change occurs in the system,
regression testing is used. This test is done to check if these shifts have affected other
features.

• After a defect is fixed: The developers perform regression after fixing a bug issue in any
functionality. This is done to determine if the changes made while fixing the issue have
affected other related features.

• Once the performance issue is fixed: After fixing any performance issues, regression
testing is done to see if it has affected other functionalities.

• While integrating with a new external system: Regression testing is required whenever
the product integrates with a new external system.
70
How to do Regression Testing in Software Testing

Retest All
This is one of the methods for Regression Testing, specifically employing a regression
testing suite, in which all the tests in the existing test bucket or suite should be re-
executed. This is very expensive as it requires huge time and resources.
Regression Test Selection
Regression Test Selection is a technique in which some selected test cases from test suite
are executed to test whether the modified code affects the software application or not.
Test cases are categorized into two parts, reusable test cases which can be used in further
regression cycles and obsolete test cases which can not be used in succeeding cycles.
Prioritization of Test Cases
Prioritize the test cases depending on business impact, critical & frequently used
functionalities. Selection of test cases based on priority will greatly reduce the regression
test suite.

71
System Testing

 A classic system-testing problem is “finger pointing” this occurs when

an error is uncovered, and each system element developer blames the
other for the problem.
 Rather than indulging in such nonsense, the software engineer should
anticipate potential interfacing problem and
 Design error-handling paths that test all information coming from other
elements of the system.
 Conduct a series of tests that simulate bad data or other potential errors
at the software interface.
 Record the results of tests to use as” evidence” if finger pointing does
occur; and
 Participate in planning and design of system tests to ensure that
software is adequately tested.
72
Different Types
• Recovery testing
• Tests for recovery from system faults
• Forces the software to fail in a variety of ways and verifies that recovery is
properly performed
• Tests reinitialization, checkpointing mechanisms, data recovery, and restart for
correctness
• Security testing
• Verifies that protection mechanisms built into a system will, in fact, protect it
from improper access
• Stress testing
• Executes a system in a manner that demands resources in abnormal quantity,
frequency, or volume
• Performance testing
• Tests the run-time performance of software within the context of an integrated
system
• Often coupled with stress testing and usually requires both hardware and
software instrumentation
• Can uncover situations that lead to degradation and possible system failure

74
 Testing in the V-Model

Requirements Acceptance
Customer test
Developer
Architectural System test

s
t e st
Design
Wri

Run
Integration test
te te

Detailed
Design Functional
st s

Module Structural
Unit test
implementation
75
Some Terminology

• Failure
• A failure is said to occur whenever the external behavior does not conform to system
spec.
• Error
• An error is a state of the system which, in the absence of any corrective action, could
lead to a failure.
• Fault
• An adjudged cause of an error.

76
Some Terminology

fault, bug, error,

It is there in the program… defect
Fault

Program state
Error

Observed
Failure

77
 Fault, Error , Failure

Fault: A fault is the underlying defect in a system. It may be a bug in the code or a hardware defect that could lead to
incorrect behavior.
Example: A developer writes an incorrect line of code due to a logic mistake, such as using > instead of >= in a condition.

if temperature > 30: # Should be >= 30 to trigger the correct behavior

activate_cooling_system()

Error: An error is the manifestation of the fault during execution. When the system is running, and this faulty code is executed, it leads to incorrect
system state.
Example: When the temperature is exactly 30°C, the cooling system is not activated because the code uses > instead of >=.
This is an error in the system's internal state.

Failure: A failure occurs when the system does not deliver the expected service to the user due to the error. It is the visible incorrect behavior
of the system from the user's perspective.
Example: The user notices that when the temperature hits 30°C, the cooling system does not activate, resulting in a failure of the system to
maintain the desired temperature.
In this case:
•Fault: The wrong condition > instead of >=.
•Error: At 30°C, the system enters an incorrect state where the cooling system is not triggered.
•Failure: The cooling system does not work as expected, leading to overheating.

78
 Testing Activities

Subsystem Requirements
Unit System
Code Test Analysis
Design Document
Tested Document User
Subsystem
Subsystem Manual
Unit
Code Test
Tested Integration Functional
Subsystem
Test Test
Integrated Functioning
Subsystems System

Tested Subsystem

Subsystem Unit
Code Test
All
Alltests
testsby
bydeveloper
developer
79
 Testing Activities continued

Client’s
Global Understanding User
Requirements of Requirements Environment

Functioning Validated Accepted

System Performance System Acceptance System Installation
Test Test Test

Usable
Tests
Testsby
byclient
client System
Tests
Testsby
bydeveloper
developer
User’s understanding
System in
Use

Tests
Tests(?)
(?) by
byuser
user
80
Comparison of White & Black-box
Testing
• White-box Testing: • Both types of testing are needed
• Potentially infinite number of paths • White-box testing and black box
have to be tested testing are the extreme ends of a
• White-box testing often tests what testing continuum.
is done, instead of what should be
done • Any choice of test case lies in
• Cannot detect missing use cases between and depends on the
following:
• Black-box Testing:
• Number of possible logical paths
• Potential combinatorical explosion
of test cases (valid & invalid data) • Nature of input data
• Often not clear whether the • Amount of computation
selected test cases uncover a • Complexity of algorithms and data
particular error structures
• Does not discover extraneous use
cases ("features")

81
Software Quality
assurance (SQA)
Lecture 4

82
 Outline
 Introduction
 Software Life Cycle
 Quality Control
 Infrastructure
 Management
 Standards
 Conclusion & Summary
83
 Management
Project Progress Control
Quality Metrics
Cost of Quality
Discussion & Summary

84
 1. Project Progress Control
Overview:
 The components of project progress control

 Progress control of internal projects and external

participants
 Implementation of project progress control regimes

 Computerized tools for software progress control

85
 Components of Project Progress
Control
 Control of risk management activities
 Project schedule control
 Project resource control
 Project budget control

86
 Control of Risk Management
Activities

Initial list of risk items come from the contract reviews

and project plan
Systematic risk management activities required:
 Periodic assessment about the state of the software risk
items
 Based on this reports the project managers are expected to
intervene and help arrive at a solution in the more extreme
cases

87
 Project Schedule Control
 Compliance of the project with its approved and
contracted timetables
 Control is based mainly on milestone reports
which are set (in part) to facilitate identification of
delays and other periodic reports
 Milestones set in contracts, especially dates for
delivery, receive special emphasis
 Focus on critical delays (which may effect final
completion of the project)
 Management interventions:
Allocation of additional resources
Renegotiating the schedule with the customer

88
 Project Resource Control
Main control items:
 Human resources
 Special development and testing equipment (real-time

systems; firmware)
Resource control primarily focuses on professional human
resources but applies to other assets too.
Real-time software and firmware projects require strict resource
monitoring through periodic reports.
Small deviations in resource use can mask larger cumulative
impacts if project progress is delayed.
Internal allocation reviews help detect budget strains, such as
imbalanced senior vs. junior analyst hours.
Early detection of deviations allows for resource reallocation, team
reorganization, or project plan revisions.

89
 Project Budget Control
Main budget items:
 Human resources
 Development and testing facilities
 Purchase of COTS software
 Purchase of hardware
 Payments to subcontractors
 Control is based on the milestone reports and other
periodic reports
 Usually budget control has the highest priority, but
only the combination of all control aspects ensure the
required

90
 Progress control of internal
projects and external participants
Problem: In practice project control provides only a
limited view of the progress of internal software
development and an even more limited view on the
progress made by external project participants
Internal projects have by definition no external
customer and therefore tend to occupy a lower place
among management's priorities. Therefore, the full
range of project progress control should be employed
More significant efforts are required in order to achieve
acceptable levels of control for an external project
participant due to the more complex communication
and coordination
91
 Implementation of project progress
control regimes
Procedures:
Allocation of responsibilities for
 Person or management for progress control
 Frequency of reporting from each of the unit levels and
administrative level
 Situation requiring the project leader to report immediately
to management
 Situation requiring lower management to report immediately
to upper management
Management audits of project progress which deals mainly with
(1) How well progress reports are transmitted by project
leaders and by lower- to upper-level management
(2) Specific management control activities to be initiated
92
 Implementation of project progress
control regimes
Remarks:
Project progress control may be conducted on
several managerial levels in large software
development organizations coordination becomes
essential
Project leaders base there progress reports on
information gathered from team leaders

93
 Computerized Project Progress
Control
Required for non trivial projects
Automation can reduce costs considerably

94
 Computerized Control of risk
management
Listsof software risk items by category and their
planned solution dates
Listsof exceptions of software risk items –
overrun solution dates

95
 Control

Classified lists of delayed activities.

Classified lists of delays of critical activities – delays
that can affect the project’s completion date.
Updated activity schedules, according to progress reports
and correction measures.
Classified lists of delayed milestones.
Updated milestones schedules, according to progress
reports and applied correction measures.
96
 Computerized Project Resource
Control
 Project resources allocation plan
for activities and software modules
for teams and development units
for designated time periods, etc.
 Project resources utilization – as specified above
 Project resources utilization exceptions – as specified
above
 Updated resource allocation plans generated
according to progress reports and reaction measures
applied

97
 Computerized Project Budget
Control
 Project budget plans
for activity and software module
for teams and development units
for designated time periods, etc.
 Project budget utilization reports — as specified
above
 Project budget utilization deviations – by period or
accumulated – as specified above
 Updated budget plans generated according to
progress reports ports and correction measures
98
 2. Quality Metrics
“You can’t control what you can’t measure”
[DeMarco1982]

Overview:
Objectives of quality measurement
Classification of software quality metrics
Process metrics
Product metrics
Implementation of software quality metrics
Limitations of software metrics
The function point method

99
 Definition
IEEE definition of software quality metrics:
A quantitative measure of the degree to
which an item possesses a given quality
attribute.
A function whose inputs are software data
and whose output is a single numerical value
that can be interpreted as the degree to
which the software possesses a given quality
attribute.
100
 Objectives
Facilitate management control, planning and
managerial intervention. Based on:
Deviations of actual from planned performance.
Deviations of actual timetable and budget
performance from planned.
Identify situations for development or
maintenance process improvement (preventive
or corrective actions). Based on:
Accumulation of metrics information regarding the
performance of teams, units, etc.

101
 Requirements
General requirements
Relevant
Valid
Reliable
Comprehensive
Mutually exclusive

Operative requirements
Easy and simple
Does not require independent data collection
Immune to biased interventions by interested parties

102
 Classifications
Classification by phases of software system:
Process metrics: metrics related to the software
development process
Maintenance metrics: metrics related to software
maintenance (product metrics in [Galin2004])
Product metrics: metrics related to software artifacts

Classification by subjects of measurements

Quality
Timetable
Effectiveness (of error removal and maintenance services)
Productivity

103
 Software Size/Volume Measures

KLOC: classic metric that measures the size of

software by thousands of code lines.
Number of function points (NFP): a measure of the
development resources (human resources) required to
develop a program, based on the functionality
specified for the software system.

104
 Error Counted Measures
Calculation of NCE Calculation of WCE

Error severity class Number of Errors Relative Weight Weighted

Errors
a b c D=bxc

low severity 42 1 42

medium severity 17 3 51

high severity 11 9 99

Total 70 --- 192

NCE 70 --- ---

WCE --- 192

Number of code errors (NCE) vs. weighted number of code errors (WCE)

105
 Process Metrics Categories
Software process quality metrics
Error density metrics
Error severity metrics
Software process timetable metrics
Software process error removal
effectiveness metrics
Software process productivity metrics

106
 Error Density Metrics
Code Name Calculation formula

CED Code Error Density NCE

CED = -----------
KLOC
DED Development Error Density NDE
DED = -----------
KLOC
WCED Weighted Code Error Density WCE
WCDE = ---------
KLOC
WDED Weighted Development Error WDE
WDED = ---------
Density KLOC
WCEF Weighted Code Errors per WCE
WCEF = ----------
Function Point NFP
WDEF Weighted Development Errors per WDE
WDEF = ----------
Function Point NFP

NCE = The number of code errors detected by code inspections and testing.
NDE = total number of development (design and code) errors) detected in the development process.
WCE = weighted total code errors detected by code inspections and testing.
WDE = total weighted development (design and code) errors detected in development process.

107
 Error Severity Metrics
Code Name Calculation formula
ASCE Average Severity of Code WCE
ASCE = -----------
Errors NCE
DED Average Severity of WDE
ASDE = -----------
Development Errors NDE

NCE = The number of code errors detected by code inspections and testing.
NDE = total number of development (design and code) errors) detected in the
development process.
WCE = weighted total code errors detected by code inspections and testing.
WDE = total weighted development (design and code) errors detected in
development process.

108
 Software Process Timetable Metrics
Code Name Calculation formula

TTO Time Table Observance MSOT

TTO = -----------
MS
ADMC Average Delay of Milestone TCDAM
ADMC = -----------
Completion MS

MSOT = Milestones completed on time.

MS = Total number of milestones.
TCDAM = Total Completion Delays (days, weeks, etc.) for all milestones.

109
 Error Removal Effectiveness Metrics
Code Name Calculation formula

DERE Development Errors NDE

DERE = ----------------
Removal Effectiveness NDE + NYF
DWERE Development Weighted WDE
DWERE = ------------------
Errors Removal WDE+WYF
Effectiveness

NDE = total number of development (design and code) errors) detected in the
development process.
WCE = weighted total code errors detected by code inspections and testing.
WDE = total weighted development (design and code) errors detected in
development process.
NYF = number software failures detected during a year of maintenance service.
WYF = weighted number of software failures detected during a year of maintenance
service.

110
 Process Productivity Metrics
Code Name Calculation formula

Development Productivity DevH

DevP DevP = ----------
KLOC
Function point Development DevH
FDevP FDevP = ----------
Productivity NFP
Code Reuse ReKLOC
CRe Cre = --------------
KLOC
Documentation Reuse ReDoc
DocRe DocRe = -----------
NDoc

DevH = Total working hours invested in the development of the software system.
ReKLOC = Number of thousands of reused lines of code.
ReDoc = Number of reused pages of documentation.
NDoc = Number of pages of documentation.

111
 Maintenance Metrics Categories (1/2)
Help desk service (HD):
software support by instructing customers regarding the
method of application of the software and solution for
customer implementation problems (depends to a great
extent on “user friendliness”)

Related metrics:
HD quality metrics:
HD calls density metrics - measured by the number of calls.
HD calls severity metrics - the severity of the HD issues raised.
HD success metrics – the level of success in responding to HD calls.
HD productivity metrics.
HD effectiveness metrics.

112
 Maintenance Metrics Categories (2/2)
Corrective maintenance:
Correction of software failures identified by
customers/users or detected by the customer service team prior to
their discovery be the customer (directly related to the software
development quality)

Related metrics:
Corrective maintenance quality metrics.
Software system failures density metrics
Software system failures severity metrics
Failures of maintenance services metrics
Software system availability metrics
Corrective maintenance productivity metrics
Corrective maintenance effectiveness metrics

113
 HD Calls Density Metrics
Code Name Calculation formula

HDD HD calls density NHYC

HDD = --------------
KLMC
WHDD Weighted HD calls density WHYC
WHYC = ------------
KLMC
WHDF Weighted HD calls per WHYC
WHDF = ------------
function point NMFP

NHYC = the number of HD calls during a year of service.

KLMC = Thousands of lines of maintained software code.
WHYC = weighted HD calls received during one year of service.
NMFP = number of function points to be maintained.

114
 Severity of HD Calls Metrics and HD Success
Metrics
Code Name Calculation formula

ASHC Average severity of HD WHYC

ASHC = --------------
calls NHYC

NHYC = the number of HD calls during a year of service.

WHYC = weighted HD calls received during one year of service.

Code Name Calculation formula

HDS HD service success NHYOT

HDS = --------------
NHYC

NHYNOT = Number of yearly HD calls completed on time during one year of service.
NHYC = the number of HD calls during a year of service.

115
 HD Productivity and Effectiveness Metrics
Code Name Calculation formula

HD Productivity HDYH
HDP HDP= --------------
KLNC
Function Point HD HDYH
FHDP FHDP = ----------
Productivity NMFP
HD effectiveness HDYH
HDE HDE = --------------
NHYC

HDYH = Total yearly working hours invested in HD servicing of the software system.
KLMC = Thousands of lines of maintained software code.
NMFP = number of function points to be maintained.
NHYC = the number of HD calls during a year of service.

116
 Failures Density Metrics
Code Name Calculation formula
NYF
SSFD Software System Failure SSFD = --------------
Density KLMC
WYF
WSSFD Weighted Software WFFFD = ---------
System Failure Density KLMC
WYF
WSSFF Weighted Software WSSFF = ----------
System Failures per NMFP
Function point

NYF = number of software failures detected during a year of maintenance service.

WYF = weighted number of yearly software failures detected during one year of
maintenance service.
NMFP = number of function points designated for the maintained software.
KLMC = Thousands of lines of maintained software code.

117
 Failures Density Metrics
Code Name Calculation formula

ASSSF Average Severity of WYF

ASSSF = --------------
Software System NYF
Failures

NYF = number of software failures detected during a year of maintenance service.

WYF = weighted number of yearly software failures detected during one year.

Code Name Calculation formula

MRepF Maintenance Repeated RepYF

MRepF = --------------
repair Failure metric - NYF

NYF = number of software failures detected during a year of maintenance service.

RepYF = Number of repeated software failure calls (service failures).

118
 Availability Metrics
Code Name Calculation formula
NYSerH - NYFH
FA Full Availability FA = -----------------------
NYSerH
NYSerH - NYVitFH
VitA Vital Availability VitA = -----------------------------
NYSerH
NYTFH
TUA Total Unavailability TUA = ------------
NYSerH
NYSerH = Number of hours software system is in service during one year.
NYFH = Number of hours where at least one function is unavailable (failed) during one year,
including total failure of the software system.
NYVitFH = Number of hours when at least one vital function is unavailable (failed) during
one year, including total failure of the software system.
NYTFH = Number of hours of total failure (all system functions failed) during one year.

Note: NYFH ≥ NYVitFH ≥ NYTFH and 1 – TUA ≥ VitA ≥FA

119
 Software Corrective Maintenance
Productivity and Effectiveness Metrics
Code Name Calculation formula

Corrective Maintenance CMaiYH

CMaiP CMaiP = ---------------
Productivity KLMC
Function point Corrective CMaiYH
FCMP FCMP = --------------
Maintenance Productivity NMFP
Corrective Maintenance CMaiYH
CMaiE CMaiE = ------------
Effectiveness NYF

CMaiYH = Total yearly working hours invested in the corrective maintenance of the software
system.
NYF = number of software failures detected during a year of maintenance service.
NMFP = number of function points designated for the maintained software.
KLMC = Thousands of lines of maintained software code.

120
 Product Metrics Categories
Product metrics can also be used for general
predictions or to identify anomalous components.
Classes of product metric
Dynamic metrics which are collected by
measurements made of a program in execution;
Static metrics which are collected by measurements
made of the system representations;
Dynamic metrics help assess efficiency and
reliability; static metrics help assess complexity,
understandability and maintainability.

121
 Static Metrics
Static metrics have an indirect relationship with
quality attributes (see also static analysis)

122
 Software Product Metrics
Software metric Description
Fan in/Fan-out Fan-in is a measure of the number of functions or methods that call some other function
or method (say X). Fan-out is the number of functions that are called by function X. A
high value for fan-in means that X i s tightly coupled to the rest of the design and
changes to X will have extensive knock-on effects. A high value for fan-out suggests
that the overall complexity of X m ay be high because of the complexity of the control
logic needed to coordinate the called components.
Length of code This is a measure of the size of a p rogram. Generally, the larger the size of the code of a
component, the more complex and error-prone that component is likely to be. Length of
code has been shown to be one of the most reliable metrics for predicting error-
proneness in components.
Cyclomatic complexity This is a m easure of the control complexity of a p rogram. This control complexity may
be related to program understandability. I discuss how to compute cyclomatic
complexity in Chapter 22.
Length of identifiers This is a measure of the average length of distinct identifiers in a p rogram. The longer
the identifiers, the more likely they are to be m eaningful and hence the more
understandable the program.
Depth of conditional This is a measure of the depth of nesting of if-statements in a program. Deeply nested if
nesting statements are hard to understand and are potentially error-prone.
Fog index This is a measure of the average length of words and sentences in documents. The higher
the value for the Fog index, the more difficult the document is to understand.

123
 Object-oriented Metrics
Object-oriented
metric Description

Depth of inheritance This represents the number of discrete levels in the inheritance tree where sub-
tree classes inherit attributes and operations (methods) from super-classes. The
deeper the inheritance tree, the more complex the design. Many different object
classes may have to be understood to understand the object classes at the leaves
of the tree.

Method fan-in/fan- This is directly related to fan-in and fan-out as described above and means
out essentially the same thing. However, it may be appropriate to make a
distinction between calls from other methods within the object and calls from
external methods.

This is the number of methods that are included in a class weighted by the
Weighted methods
complexity of each method. Therefore, a simple method may have a complexity
per class of 1 and a large and complex method a much higher value. The larger the value
for this metric, the more complex the object class. Complex objects are more
likely to be more difficult to understand. They may not be logically cohesive so
cannot be reused effectively as super-classes in an inheritance tree.

Number of This is the number of operations in a super-class that are over-ridden in a sub-
overriding class. A high value for this metric indicates that the super-class used may not be
operations an appropriate parent for the sub-class.

124
 The Measurement Process
A software measurement process may be part of a
quality control process.
(1) Define software quality metrics
(2) Select components to be assessed
(3) Data collection
(4) Identify anomalous measurement
(5) Analyse anomalous measurement
Data collected during this process should be
maintained as an organisational resource.
Once a measurement database has been
established, comparisons across projects become
possible.

125
(1) Defining
Software
Quality
Metrics

126
 (3) Data Collection
A metrics programme should be based on a
set of product and process data.
Data should be collected immediately (not
in retrospect) and, if possible,
automatically.
Three types of automatic data collection
Static product analysis;
Dynamic product analysis;
Process data collation.

127
 Data Accuracy
Don’t collect unnecessary data
The questions to be answered should be
decided in advance and the required data
identified.
Tell people why the data is being collected.
It should not be part of personnel evaluation.
Don’t rely on memory
Collect data when it is generated not after a
project has finished.

128
 (5) Measurement Analysis
 It is not always obvious what data means
Analysing collected data is very difficult.
 Professional statisticians should be consulted
if available.
 Data analysis must take local circumstances
into account.

129
 Measurement Surprises
Reducing the number of faults in a program
leads to an increased number of help desk
calls
The program is now thought of as more reliable and
so has a wider more diverse market. The percentage
of users who call the help desk may have decreased
but the total may increase;
A more reliable system is used in a different way from
a system where users work around the faults. This
leads to more help desk calls.

130
 Limitations of Quality Metrics
Budget constraints in allocating the necessary resources.
Human factors, especially opposition of employees to evaluation of their
activities.
Validity Uncertainty regarding the data's, partial and biased reporting.
[Galin2004]

Metrics assumptions:
A software property can be measured.
The relationship exists between what we can measure and what we want to
know. We can only measure internal attributes but are often more interested in
external software attributes.
This relationship has been formalised and validated.
It may be difficult to relate what can be measured to desirable external quality
attributes.
[Sommerville2004]

"Not everything that counts is countable; and not

everything that is countable counts."

131
Examples of Software Metrics that
exhibit Severe Weaknesses

Parameters used in development process

metrics:
KLOC, NDE, NCE.
Parameters used in product (maintenance)
metrics:
KLMC, NHYC, NYF.

132
Factors Affecting Parameters used
for Development Process Metrics
 Programming style (KLOC).
 Volume of documentation comments (KLOC).
 Software complexity (KLOC, NCE).
 Percentage of reused code (NDE, NCE).
 Professionalism and thoroughness of design review
and software testing teams: affects the number of
defects detected (NCE).
 Reporting style of the review and testing results:
concise reports vs. comprehensive reports
(NDE,NCE).
133
Factors Affecting Parameters used
for Maintenance Metrics
 Quality of installed software and its documentation
(NYF, NHYC).
 Programming style and volume of documentation
comments included in the code be maintained
(KLMC).
 Software complexity (NYF).
 Percentage of reused code (NYF).
 Number of installations, size of the user population
and level of applications in use: (NHYC, NYF).

134
Factors Affecting Parameters used
for Product Metrics
 Programming style
 Architectural styles/Architectures
 Thorough understanding of the domain
(more intensive use of inheritance)
 Level of reuse and use of frameworks

135
 The Function point approach for software
sizing was invented by Allan Albrecht in
1979
 The measure of Albrecht - Function Point
Analysis (FPA) - is well known because of its
great advantages:
 Independent of programming language and technology.
 Comprehensible for client and user.
 Applicable at early phase of software life cycle.

136
 The Function Point Method
 The function point estimation process:
Stage 1: Compute crude function points
(CFP).
Stage 2: Compute the relative complexity
adjustment factor (RCAF) for the project.
RCAF varies between 0 and 70.
Stage 3: Compute the number of function
points (FP):
FP = CFP x (0.65 + 0.01 x RCAF)

137
 Complexity Levels
In function point analysis (FPA), measuring the complexity of different elements such as
user inputs, outputs, online queries, logical files, and external interfaces is crucial for
calculating the function point count. Here are the criteria used to determine their
complexity levels:
1. User Inputs (External Inputs - EI)
User inputs refer to data entered into the system from external sources. The complexity
levels depend on the number of data elements and the types of processing involved.
Low Complexity (1 point):
Simple data input with minimal processing (e.g., data entry forms).
Few data elements (1 to 5).
Average Complexity (2 points):
Moderate data input requiring validation or minor calculations.
More data elements (6 to 19).
High Complexity (3 points):
Complex data input with significant processing (e.g., calculations, validation, or
multiple screens).
Many data elements (20 or more).

138
 Complexity Levels
2. User Outputs (External Outputs - EO)
User outputs are the results presented to the user, including reports and
messages. Their complexity is based on the number of data elements and
the type of output processing.
Low Complexity (1 point):
Simple reports or messages with minimal formatting.
Few data elements (1 to 5).
Average Complexity (2 points):
Moderate reports or messages with some processing or formatting.
More data elements (6 to 19).
High Complexity (3 points):
Complex reports or messages requiring significant formatting,
sorting, or calculations.
Many data elements (20 or more).

139
 Complexity Levels
3. User Online Queries (External Inquiries - EQ)
Online queries refer to user interactions where they request information or
perform a search. Complexity is based on the number of data elements and
the query’s processing complexity.
Low Complexity (1 point):
Simple queries with minimal processing (e.g., display a single record).
Few data elements (1 to 5).
Average Complexity (2 points):
Queries with moderate processing, such as retrieval and display of
multiple records.
More data elements (6 to 19).
High Complexity (3 points):
Complex queries with significant processing, such as aggregating or
sorting large datasets.
Many data elements (20 or more).

140
 Complexity Levels
4. Logical Files (Internal Logical Files - ILF)
Logical files are internal data structures used for storing information. The
complexity depends on the number of data elements and the file’s structure.
Low Complexity (1 point):
Small files with simple structures (e.g., a single table or list).
Few data elements (1 to 5).
Average Complexity (2 points):
Files with moderate complexity, such as multiple related tables or files.
More data elements (6 to 19).
High Complexity (3 points):
Large or complex files with many relationships or intricate data
structures.
Many data elements (20 or more).

141
 Complexity Levels
5. External Interfaces (External Interface Files - EIF)
External interfaces refer to data that is used by the system but maintained outside the system boundary. Complexity is based on
the number of data elements and the integration complexity.
Low Complexity (1 point):
 Simple data exchanges with external systems (e.g., file imports or exports).
 Few data elements (1 to 5).
Average Complexity (2 points):
 Moderate external interfaces with some integration or data transformation.
 More data elements (6 to 19).
High Complexity (3 points):
 Complex external interfaces requiring significant data transformation, validation, or integration.
 Many data elements (20 or more).
General Considerations for Complexity Level:
Data Elements: These are the individual units of data handled by the system, like fields or variables in files or records.
Processing Logic: Complexity increases when data requires more elaborate processing such as calculations, sorting, or
aggregation.
External Interactions: Integration with external systems or user interfaces introduces more complexity due to the need for
more robust data handling or error-checking mechanisms.
File Structure: More complex structures, such as those with relationships or hierarchical data, increase the complexity level.
The function point calculation process involves assigning these complexity values to each component based on its
characteristics and then summing them up to determine the overall function point count for the software application.

142
 Function Point Method
An example: The Attend
Master

143
 Attend Master Software System
Attend-Master is a basic employee attendance
system that is planned to serve small to medium-
sized businesses employing 10–100 employees.
The system is planned to have interfaces to the
company’s other software packages: Human-
Master, which serves human resources units, and
Wage-Master, which serves the wages units.
Attend-Master is planned to produce several
reports and online queries.

144
145
 Calculation of CFP
Crude Function Points
Analysis of the software system as presented in
the DFD summarizes the number of the various
components:
Number of user inputs – 2
 Number of user outputs – 3
Number of user online queries – 3
Number of logical files – 2
Number of external interfaces – 2.
The degree of complexity (simple, average or
complex) was evaluated for each component.
146
 Crude Function Points: Calculation
Complexity level Total
Software Simple average complex CFP
system Count Weight Points Count Weight Points Count Weight Points
Factor Factor Factor
components
A B C= D E F= G H I=
AxB DxE GxH

User inputs 1 3 3 --- 4 --- 1 6 6 9

User outputs --- 4 --- 2 5 10 1 7 7 17
User online 3 4 6
queries 1 3 1 4 1 6 13
Logical files 1 7 7 --- 10 --- 1 15 15 22
External 5 7 10
interfaces --- --- --- --- 2 20 20
Total CFP 81

147
 Calculation of RCAF
Relative Complexity Adjustment Factor

148
No Subject Grade

1 Requirement for reliable backup and recovery 0 1 2 3 4 5

2 Requirement for data communication 0 1 2 3 4 5
3 Extent of distributed processing 0 1 2 3 4 5
4 Performance requirements 0 1 2 3 4 5
5 Expected operational environment 0 1 2 3 4 5
6 Extent of online data entries 0 1 2 3 4 5
7 Extent of multi-screen or multi-operation online data input 0 1 2 3 4 5
8 Extent of online updating of master files 0 1 2 3 4 5
9 Extent of complex inputs, outputs, online queries and files 0 1 2 3 4 5
10 Extent of complex data processing 0 1 2 3 4 5
11 Extent that currently developed code can be designed for reuse 0 1 2 3 4 5
12 Extent of conversion and installation included in the design 0 1 2 3 4 5
13 Extent of multiple installations in an organization and variety of customer 0 1 2 3 4 5
organizations
14 Extent of change and focus on ease of use 0 1 2 3 4 5
Total = RCAF 41

149
FP = CFP x (0.65 + 0.01 x RCAF)

FP = 81 x (0.65 + 0.01 x 41) = 85.86

150
 Converting NFP 2 KLOC
• The estimates for the average number of lines of
code (LOC) required for programming a function
point are the following:

For C++:
KLOC = (85.86 * 64)/1000 = 5.495 KLOC

151
 The function point metric was originally designed to be applied to business
information systems applications.
 The data dimension was emphasized to the exclusion of the functional and
behavioral (control) dimensions.
 The function point measure was inadequate for many engineering and
embedded systems
 Feature points : A superset of the function point, designed for
applications in which algorithmic complexity is high (real-time, process
control, embedded software applications).

 UCPs: Use case points (UCPs) allow the estimation of an application’s

size and effort from its use cases. UCPs are based on the number of actors,
scenarios, and various technical and environmental factors in the use case
diagram.

152
 UCPs
UCPs are based on the number of actors, scenarios, and
various technical and environmental factors in the use case
diagram.
The UCP equation is based on four variables:
Technical complexity factor (TCF)
Environment complexity factor (ECF)
Unadjusted use case points (UUCP)
Productivity factor (PF)
which yield the equation:
UCP = TCP * ECF * UUCP * PF

153
 Advantages & Disadvantages
Main advantages
Estimates can be prepared at the pre-project stage.
Based on requirement specification documents (not specific
dependent on development tools or programming languages), the
method’s reliability is relatively high.
Main disadvantages
While tools exist to support FPA, many steps in the process (e.g.,
identifying and categorizing functions) require manual analysis and are not
fully automatable.
Estimates based on detailed requirements specifications, which are not
always available.
The entire process requires an experienced function point team and
substantial resources.
The evaluations required result in subjective results.
Successful applications are related to data processing. The method
cannot yet be universally applied.

154
 3. Cost of Quality
Overview:
Objectives of cost of software quality metrics
The classic model of cost of software quality
Galin’s extended model for cost of software
quality
Application of a cost of software quality
system
Problems in the application of cost of
software quality metrics

155
 Objectives
In general – it enables management to achieve
economic control over SQA activities and
outcomes. The specific objectives are:
Control organization-initiated costs to prevent
and detect software errors.
Evaluation of the economic damages of software
failures as a basis for revising the SQA budget.
Evaluation of plans to increase or decrease of
SQA activities or to invest in SQA infrastructure on
the basis of past economic performance.

156
 Performance Comparisons
 Control Budgeted expenditures (for SQA
prevention and appraisal activities).
 Previous year’s failure costs
 Previous project’s quality costs (control costs
and failure costs).
 Other department’s quality costs (control costs
and failure costs).

157
 Evaluating SQA Systems
Cost metrics examples:
Percentage of cost of software quality out of
total software development costs.
Percentage of software failure costs out of
total software development costs.
Percentage of cost of software quality out of
total software maintenance costs.
Percentage of cost of software quality out of
total sales of software products and software
maintenance.

158
Model of Software Quality Costs
Prevention costs

Costs of
control costs
Appraisal costs
Cost of
software
quality
Internal failure
Costs of costs
Failure of
control costs
External failure
costs

159
 Prevention Costs
Investments in development of SQA infrastructure
components
Procedures and work instructions
Support devices: templates, checklists etc
Software configuration management system
Software quality metrics
Regular implementation of SQA preventive
activities:
Instruction of new employees in SQA subjects
Certification of employees
Consultations on SQA issues to team leaders and others
Control of the SQA system through performance of:
Internal quality reviews
External quality audits
Management quality reviews

160
 Appraisal Costs
Costs of reviews:
Formal design reviews (DRs)
Peer reviews (inspections and walkthroughs)
Expert reviews
Costs of software testing:
Unit, integration and software system tests
Acceptance tests (carried out by customers)
Costs of assuring quality of external
participants

161
 Internal Failure Costs
 Costs of redesign or design corrections
subsequent to design review and test findings
 Costs of re-programming or correcting programs
in response to test findings
 Costs of repeated design review and re-testing
(regression tests)

162
 External Failure Costs
Typical external failure costs cover:
Resolution of customer complaints during the warranty period.
Correction of software bugs detected during regular operation.
Correction of software failures after the warranty period is over even if
the correction is not covered by the warranty.
Damages paid to customers in case of a severe software failure.
Reimbursement of customer's purchase costs.
Insurance against customer's claims.

Typical examples of hidden external failure costs:

Reduction of sales to customers that suffered from software failures.
Severe reduction of sales motivated by the firm's damaged reputation.
Increased investment in sales promotion to counter the effects of past
software failures.
Reduced prospects to win a tender or, alternatively, the need to under-
price to prevent competitors from winning tenders.

163
 Galin’s Extended Model of
Software Quality Costs
Prevention costs

Costs of Appraisal costs

Control costs
Managerial
Cost of preparations and
software control costs
quality
Internal failure
costs
Costs of External failure
Failure of costs
control costs
Managerial failure
costs
164
 Managerial Preparation and
Control Costs
 Costs of carrying out contract reviews
 Costs of preparing project plans, including quality
plans
 Costs of periodic updating of project and quality
plans
 Costs of performing regular progress control
 Costs of performing regular progress control of
external participants’ contributions to projects

165
 Managerial Failure Costs
 Unplanned costs for professional and other resources,
resulting from underestimation of the resources in the
proposals stage.
 Damages paid to customers as compensation for late
project completion, a result of the unrealistic schedule in
the Company’s proposal.
 Damages paid to customers as compensation for late
completion of the project, a result of management’s failure
to recruit team members.
 Domino effect: Damages to other projects planned to be
performed by the same teams involved in the delayed
projects. The domino effect may induce considerable
hidden external failure costs.

166
 Application
Definition of a cost of software quality model
and specification of cost items.
Definition of the method of data collection for
each cost item.
Application of a cost of software quality
system, including thorough follow up.
Actions taken in response to the findings.

167
Cost of Software Quality
Balance by Quality Level
Quality
costs Total
cost of
Minimal software
total cost quality
of
software
quality

Total
Total control
failure of costs
control
costs
low high Software quality level
Optimal
software
quality
level

168
 Problems of Application
General problems:
Inaccurate and/or incomplete identification and classification of quality costs.
Negligent reporting by team members
Biased reporting of software costs, especially of “censored” internal and external
costs.
Biased recording of external failure costs - “camouflaged” compensation of
customers for failures.

Problems arising when collecting data on managerial costs:

Contract review and progress control activities are performed in a “part-time
mode”. The reporting of time invested is usually inaccurate and often neglected.
Many participants in these activities are senior staff members who are not
required to report use of their time resources.
Difficulties in determination of responsibility for schedule failures.
Payment of overt and formal compensation usually occurs quite some time after
the project is completed, and much too late for efficient application of the lessons
learned.

169
 4. Discussion & Summary
Components of Project Progress Control
requires (1) control of risk management activities,
(2) project schedule control, (3) project resource
control, and (4) project budget control. Even though
budget control has usually the highest priority,
only the combination of all control tasks ensures
the required coverage of risks
Metrics facilitate management control, planning
and managerial intervention based on deviations of
actual from planned performance to identify
situations for development or maintenance process
improvement (preventive or corrective actions)
170
 Discussion & Summary
Metrics must be relevant, valid, reliable, comprehensive, and
mutually exclusive. In addition, it must be easy and simple,
should not require independent data collection, and should
be immune to biased interventions by interested parties.
Available metrics for control are process metrics, which are
related to the software development process, maintenance
metrics, which are related to software maintenance (called
product metrics in [Galin2004]), and product metrics, which
are related to software artifacts.
Metrics are used to measure quality, accordance with
timetables, effectiveness (of error removal and
maintenance services), and productivity.

171
 Discussion & Summary
A software measurement process may should be
part of a quality control process which (1) defines
the employed software quality metrics, (2) selects
components to be assessed, (3) collects the data,
(4) ensures that anomalous measurements are
identified, and (5) are analyzed. But due to the
complex interdependencies unexpected effects are
possible (measurement surprises)
The optimal software quality level is reached
when the sum of the total failure of control costs
and total control costs is minimal.

172
Lecture 5

173
Outline

174
III.1 Foundations

Quality Control is the series of

inspections, reviews and tests used
throughout the development cycle to
ensure that each work product meets the
requirements placed upon it.
[Pressman2004]

175
 Verification & Validation
Verification: refers to the set of activities that
ensure that softwarecorrectly implements a specific function.
Validation: refers to a different set of activities that ensure that
the software that has been built is traceable to customer
requirements.
Boehm [Boehm81]:
Verification: “Are we building the product right?”
Validation: “Are we building the right product?”
The definition of V&V encompasses many of SQA activities,
including
formal technical reviews, quality and configuration audits
performance monitoring, different types of software testing
feasibility study and simulation

176
 Terminology: Dynamic/Static
Analysis
Dynamic analysis:
The process of evaluating a system or component based on
its behavior during execution.
Contrast with: static analysis.
See also: demonstration; testing.
[IEEE-610.12-1990]

Static analysis:
The process of evaluating a system or component based on
its form, structure, content, or documentation.
Contrast with: dynamic analysis.
See also: inspection; walk-through.
[IEEE-610.12-1990]
177
 Dynamic vs. Static Analysis
Dynamic analysis (testing):
execution of system components;
running the software
Static analysis:
investigation without operation;
pencil and paper reviews etc.
Modelling (mathematical representation)

178
 Terminology: Testing

Several definitions:
“Testing is the process of executing a program or
system with the intent of finding errors.”
[Myers1979]

Testing is the process used to verify or validate a

system or its components.
[Storey1996]

179
 Testing (according to IEEE)
(1) The process of operating a system or component under specified
conditions, observing or recording the results, and making an
evaluation of some aspect of the system or component.
(2) (IEEE Std 829-1983 [5]) The process of analyzing a software
item to detect the
differences between existing and required conditions (that is, bugs)
and to evaluate the features of the software items. See also:
acceptance testing; benchmark; checkout; component testing;
development testing; dynamic analysis; formal testing; functional
testing; informal testing; integration testing; interface testing;
loopback testing; mutation testing; operational testing;
performance testing; qualification testing; regression testing;stress
testing; structural testing; system testing; unit testing.
[IEEE-610.12-1990]
180
 Quality Control
Dynamic Activities
Life cycle phase Static &
Modelling
analysis analysis
Requirements
Process X X
Top-level design X X
Detailed design X X
Implementation X X
Integration testing X X X
System validation X X

181
 III.2 Static Analysis Techniques
Overview
 Reviews and Inspections
Walkthroughs, inspections, personal
reviews
Formal technical reviews

Summary
Other Techniques
Control-flow analysis, data-flow analysis, metrics, …

182
 II.2.1 Reviews and Inspections
A family of techniques
 Personal reviews
 Inspections
 Walkthroughs
 Formal technical reviews
Review / inspect
To examine closely, With an eye toward
correction or appraisal
 People (peers) are the examiners 183
 Purpose/Objectives
 Verify that software meets its requirements software
follows predefined standards software is developed in
uniform manner
 Catching errors
Sooner
More and different
 Make projects more manageable
To identify new risks likely to affect the project
 Improving communication
Crossing organization boundaries
 Providing Education
 Making software visible
184
 (1) Personal Review
Features
 Informal
 Done by the producer
Implications
 Not objective
 Available to any developer
 Different mindset (A personal review often
reflects the developer's own thought process,
which may differ from that of external
reviewers, potentially limiting critical insights.)
185
Need for review
Conducting personal reviews helps identify issues
early, ensuring that the product is thoroughly
checked before completion.

Product completion
Personal reviews contribute to the overall
progress toward product completion by
addressing preliminary flaws efficiently.

186
 (2) Inspections
Inspection is More formal type of walkthrough
•Used to identify specific errors or problems
•assume details are correct, but look for missing items at a global level
•look for occurrences of a particular kind of bug such as buffer overflow.
Features
Team reviews materials separately

Team and producers meet to discuss

May review selected product aspects only

Implications
Focus on important issues
If you know what they are
More material per meeting
Less preparation time

187
Process and Participants

188
 (3) Walkthroughs
Features
 Less formal
 Producer presents or provides information
Implications
 Larger groups can attend (education)
 More material per meeting
 Less preparation time

Disadvantage: Harder to separate

Product and presenter
 The walkthrough process may focus more on the presenter’s
performance than on critically evaluating the actual product.
Explanation and justification
 During the walkthrough, explanations of the product may
unintentionally turn into justifications for its shortcomings, making it
difficult to identify real issues.
189
Process and Participants

190
What to omit and what to include?
Inclusion:
Sections of complicated logic Critical sections,
where defects severely damage essential system
capability Sections dealing with new environments
Sections designed by new or inexperienced team
members
Omission:
“Straightforward” sections (no complications)
Sections of a type already reviewed by the team in
similar past projects Sections that, if faulty, are not
expected to effect functionality Reused design and
code Repeated parts of the design and code

191
 (4) Formal Technical Review
Features
 Formal
 Scheduled event
 Defined procedure
 Reported result
 Independent review team
 Producers not present
Implications
 More preparation time
 Less material per meeting
Product must stand or fall on its own

192
 Formal Technical Review (FTR)
 Process
Phases (Orientation) and procedures (Checklists)
 Roles
Author, Moderator, Reader, Reviewer, Recorder
 Objectives
Defect removal, requirements elicitation, etc.
 Measurements
Forms, consistent data collection, etc.

193
 FTR Process
 How much to review
 Review pacing
 When to review
 Pre-meeting preparation
 Meeting pace

194
 How Much to Review?
 Tied into meeting time (hours)
 Should be manageable
 Break into chunks if needed

195
 Review Pacing
 How long should the meeting last?
Based on:
 Lines per hour?
 Pages?
 Specific time frame?

196
 When to Review?
 How much work should be completed before the
review
 Set out review schedule with project planning
 Again, break into manageable chunks
 Prioritize based on impact of code module to overall
project

197
 Pre-Meeting Preparation
 Materials to be given to reviewers
 Time expectations prior to the meeting
 Understand the roles of participants
 Training for team members on their various
roles
 Expected end product

198
 Pre-Meeting Preparation (2/2)
How is document examination conducted?
 Ad-hoc
 Checklist
 Specific reading techniques (scenarios or
perspective-based reading)

Preparation is crucial to effective reviews

199
 FTR Team Roles

 Select the correct participants for each role

 Understand team review psychology
 Choose the correct team size

200
 FTR Team Roles (2/2)
 Author
 Moderator
 Reader
 Reviewer
 Recorder (optional?)

Who should not be involved and why?

201
202
 Team Participants
 Must be actively engaged
 Must understand the “bigger picture”

203
 Team Psychology
Stress
Conflict resolution
Perceived relationship to performance reviews

204
 Team Size
 What is the ideal size for a team?
 Less than 3?
 3-6?
 Greater than 6?
 What is the impact of large, complex projects?
 How to work with globally distributed teams?

205
 FTR Objectives
 Review meetings can take place at various stages of
the project lifecycle
 Understand the purpose of the review
 Requirements elicitation
 Defect removal
 Other
 Goal of the review is not to provide solutions
 Raise issues, don’t resolve them

206
 FTR Measurements
 Documentation
 Sample forms
 Inspection metrics

207
 Documentation
 Forms used to facilitate the process
 Documenting the meeting
 Use of standards
 How is documentation used by:
 Managers
 Developers
 Team members

208
 Sample Forms

 NASA Software Formal Inspections Guidebook

 Sample checklists
Architecture design
Detailed design
Code inspection
Functional design
Software requirements

209
 Beyond the FTR Process
 Impact of reviews on the programmer
 Post-meeting activities
 Review challenges
 Survey of reviews and comparisons
 Future of FTR

210
 Impact on the Programmer
 Should reviews be used as a measure of
performance during appraisal time?
 Can it help to improve commitment to their
work?
 Will it make them a better reviewer when
roles are reversed?
 Improve teamwork?

211
 Post-Meeting Activities
Defect correction
How to ensure that identified defects are
corrected?
What metrics or communication tools are
needed?
Follow-up
Feedback to team members
Additional phases of reviews
Data collection for historical purposes
Gauging review effectiveness

212
 Review Challenges
 Distributed, global teams
 Large teams
 Complex projects
 Virtual vs. face-to-face meetings

213
 Survey of Reviews
 Reviews were integrated into software development
with a range of goals
 Early defect detection
 Better team communication
 Review approaches vary widely
 Tending towards nonsystematic methods and
techniques

Source: Ciolkowski, M., Laitenberger, O., & Biffl, S. (2003). Software reviews,
the state of the practice. Software, IEEE, 20(6), 46-51.

214
 Survey of Reviews (2/2)
 What were common review goals?
 Quality improvement
 Project status evaluation
 Means to enforce standards
 Common obstacles
 Time pressures
 Cost
 Lack of training (most train by participation)

215
 Do We Really Need a Meeting?
 “Phantom Inspector” (Fagan)
The “synergism” among the review team
that can lead to the discovery of defects not
found by any of the participants working
individually
 Meetings are perceived as higher quality
 What about false positives and duplicates?

216
 A Study of Review Meetings
 The need for face-to-face meetings has never
been questioned
 Meetings are expensive!
 Simultaneous attendance of all participants
 Preparation
 Readiness of work product under review
 High quality moderation
 Team personalities
 Adds to project time and cost
(15-20% overhead)

217
A Study of Review Meetings (2/3)
 Studied the impact of:
 Real (face-to-face) vs. nominal (individual)
 groups
 Detection effectiveness (number of defects
detected)
 Detection cost
 Significant differences were expected

218
 A Study of Review Meetings (3/3)
 Results
 Defect detection effectiveness was not
significantly different for either group
 Cost was less for nominal than for real groups
(average time to find defects was higher)
 Nominal groups generated more issues, but
had higher false positives and more duplication

219
 Does Openness and Anonymity
Impact Meetings?
 “Working in a group helped me find errors
faster and better.”
 “Working in a group helped me understand
the code better.”
 “You spent less time arguing the issue
validity when working alone.”
 “I could get a lot more done in a given
amount of time when working by myself.”

220
 Study on Successful Industry Uses
 Lack of systematic execution during preparation
and detection
 60% don’t prepare at all, only 50% use
checklist, less than 10% use advance reading
techniques
 Reviews are not part of an overall improvement
program
 Only 23% try to optimize the review process

221
 Study on Successful Industry Uses
(2/3)
 Factors for sustained success
 Top-management support is required
 Need evidence (external, internal) to warrant
using reviews
 Process must be repeatable and measurable for
continuous improvement
 Techniques need to be easily adaptable to
changing needs

222
 Study on Successful Industry Uses
(3/3)
 Repeatable success tends to use well defined
techniques
 Reported success (NASA, Motorola, IBM)
 95% defect detection rates before testing
 50% overall cost reduction
 50% reduction in delivery time

223
 Future of FTR
1. Provide tighter integration between FTR and the
development method
2. Minimize meetings and maximize asynchronicity in
FTR
3. Shift the focus from defect removal to improved
developer quality
4. Build organizational knowledge bases on review
5. Outsource review and in-source review knowledge
6. Investigate computer-mediated review technology
7. Break the boundaries on review group size
224
Category/Attribute Management Technical Inspection Walkthrough
Objective Ensure Progress. Evaluate conformance to Detect and identify Detect defects.
Recommend corrective specifications and plans. defects. Examine alternatives.
action. Ensure proper Ensure change integrity. Verify resolution. Forum for learning.
allocation of resources.
Delegated Controls Management team charts Review team petitions Team chooses All decisions made by
Decision making course of action. management of technical predefined product producer. Change is the
Decisions leadership to act on dispositions. Defects prerogative of the
are made at the meeting recommendations. must be removed. producer.
or as a result of
recommendations.

Change verification Change verification left to Leader verifies as part of Moderator verifies rewo rk.Change verification left to
other project controls. review report. other project controls.

Group Dynamics 2 or more people 3 or more people 3-6 people 2-7 people
Recommended size
Attendance Management, technical Technical leadership and College of peers meet Technical leadership and
leadership, and peer mix. peer mix. with documented peer mix.
attendance.
Leadership Usually the responsible Usually the lead engineer. Trained moderator. Usually producer.
manager.
Procedures Moderate to high, Moderate to high, Relatively low. Relatively low.
Material volume depending on the specific depending on the specific
“statement of objectives” “statement of objective”
at the meeting. for the meeting.
Presenter Project representative Software element Presentation by “reader” Usually the producer
representative. other than producer.
Data collection As required by applicable Not a formal project Formally required. Not a formal project
policies, standards, or requirement. May be requirement. May be
plans. done locally. done locally.
Outputs Management review Technical review report. Defect list. Defect Walkthrough report.
Reports report. summary. Inspection
report.

Database entries Any schedule changes No formal database Defect counts, No formal database
must be entered into the required. characteristics, severity, required.
project tracking database. and meeting attributes 225
are kept.
 Cost-Benefit Analysis
 Fagan reported that IBM inspections found
90% of all defects for a 9% reduction in
average project cost
 Johnson estimates that rework accounts for
44% of development cost
 Finding defects, finding defects early and
reducing rework can impact the overall cost of
a project

226
 Cost of Defects
 What is the impact of the annual cost of software
defects in the US?
$59 billion
 Estimated that $22 billion could be avoided by
introducing a best-practice defect detection
infrastructure
 Source: NIST, The Economic Impact of Inadequate
Infrastructure for Software Testing, May 2002

227
 Software Inspections
Why are software inspections not widely used?
 Lack of time
 Not seen as a priority
 Not seen as value added (measured by loc)
 Lack of understanding of formalized techniques
 Improper tools used to collect data
 Lack of training of participants
 Pits programmer against reviewers

228
 Other Techniques
Covered:  Static analysis:
Static Code Analysis
The process of
Formal Methods
evaluating a system
Model Checking
or component based
on its form,
Static analysis:
structure, content,
investigation without
operation; or documentation.
pencil and paper reviews [IEEE-610.12-1990]
etc.
Modelling (mathematical
representation)

229
 Static Analysis Techniques (1/2)
Static Code Analysis
 Control flow analysis
 data flow analysis
 symbolic execution
 Timing analysis
Performance/scheduling theories
Formal Methods
Discrete mathematics, logics, temporal logics, process
algebra, …
Model Checking
Fully automatic analysis of formal models w.r.t. a formal
specification
230
 III.3 Dynamic Testing Techniques
Common definition [Myers1979] :
Testing is a process of executing a program with the
intent of finding an error.
A good test case is one that has high probability of finding
an undiscovered error.
A successful test is one that uncovers an as-yet
undiscovered error.
Also: testing to assess reliability
(not considered here)

231
Test:
(1) An activity in which a system or component is executed under
specified conditions, the results are observed or recorded, and an
evaluation is made of some aspect of the system or component.
(2) To conduct an activity as in (1).
(3) (IEEE Std 829-1983 [5]) A set of one or more test cases.
(4) (IEEE Std 829-1983 [5]) A set of one or more test procedures.
(5) (IEEE Std 829-1983 [5]) A set of one
or more test cases and procedures.
[IEEE-610.12-1990]

232
 Objectives
Design tests that systematically uncover defects
 Maximize bug count: uncover as many defects (or bugs) as possible
 Using the minimum cost and efforts, within the limits budgetary/scheduling
 Guide correction such that an acceptable level of quality is reached
Help managers make ship / no-ship decisions
 Assess quality (depends on the nature of the product)
 Block premature product releases; Minimize technical support costs
 It is impossible to verify correctness of the product by testing, but you can
 prove that the product is not correct or you can demonstrate that you didn’t
 find any errors in a give time period.
Minimize risks (especially safety-related lawsuit risk):
 Assess conformance to specification
 Conform to regulations
Find safe scenarios for use (find ways to get it to work, in spite of the bugs)
Indirect objectives
To compile a record of software defects for use in error prevention
(by corrective and preventive actions)

233
 Testing Fundamentals
Software Testing is a critical element of Software QualityAssurance
 It represents the ultimate review of the requirements specification,
the design, and the code.
 It is the most widely used method to insure software quality.
 Many organizations spend 40-50% of development time in testing.
Testing is the one step in software engineering process that could be
viewed as destructive rather than constructive.
“A successful test is one that breaks the software.” [McConnell 1993]
A successful test is one that uncovers an as yet undiscovered
defect.
Testing can not show the absence of defects, it can only show that
software defects are present.
For most software exhaustive testing is not possible.

234
 Test Cases & Set of Test Cases
“Adequate” Set of Test Cases?
Describe how to test a
system/module/function
Objective:
Description must identify to uncover as many defects as
system state before executing possible
the test
Test set criteria:
system/module/function does the set cover the system in
to be tested a complete/sufficient manner

parameter values for the test Constraints:

with a minimum/appropriate of
expected outcome of the test effort and time
Should enable automation!
So, how many? Depends on method!

235
Model of Testing (1/2)
Inputs causing
anomalous
behaviour
Input test data I

System

Outputs which reveal

the presence of
defects
Output test results
Oe

236
 Model of Testing (1/2)
Unexpected
outcome
The Environment
environment model

The Program
program model Tests

Nature and Bug

psychology model Expected
outcome

237
 Testing Approaches
Black box testing
1. Testing that ignores the internal mechanism of the
system or component and focuses solely on the outputs in
response to selected inputs and execution conditions
2. Testing conducted to evaluate the compliance of a system
or component with specified functional requirements
(sometimes named functional testing)
White box testing
Testing that takes into account the internal mechanism of a
system or component(sometimes named structural testing)
IEEE

238
 III.3.2 Black Box Testing
Techniques
An approach to testing Black box testing:
where the program is (1) Function testing
considered as a “black- (2) Domain testing
box” (3) Specification-based testing
(4) Risk-based testing
requirements
(5) Stress testing
(6) Regression testing
(7) User testing
input output (8) Scenario testing
(9) State-model based testing
events (10) High volume automated testing
(11) Exploratory testing

Not orthogonal!
239
 (1) Function Testing
variable in isolation. Test each function / feature /
Usually start with fairly simple function testing
focuses on a single function and tests it with
middle-of-the-road values don’t expect the
program to fail but if it fails the algorithm is
fundamentally wrong, the build is broken, or …
Later switch to a different style which often involves
the interaction of several functions
These tests are highly credible and easy to evaluate
but not particularly powerful

240
 (2) Domain Testing
Essence: sampling
Equivalence class testing:
Look at each variable domains (including invalid values)
reduce a massive set of possible tests (domain) into a small group by
partitioning
picking one or two representatives from each subset.
look for "best representative“: a value that is at least as likely to
expose an error as any other
Boundary value analysis: A good test set for a numeric variable hits
every boundary value because boundary / extreme-value errors are common
use the minimum, the maximum, a value barely below the minimum, and a
value barely above the maximum bugs are sometimes dismissed, especially
when you test extreme values of several variables at the same time (corner
cases)

241
 Equivalence Partitioning
 Input data and output
results often fall into
Valid inputs
different classes Invalid inputs

 Each of these classes is an

equivalence partition
where the program behaves System

in an “equivalent” way
 Test cases should be
chosen from each partition
Outputs

242
 Equivalence Partitioning
an approach that divides the input domain of a program
into classes of data from which test cases can be derived.
Example: Suppose a program computes the value of the
function. ( X −1) ∗( X 2) This function defines
the following
valid and invalid equivalence classes:
X < = -2 valid
-2 < X < 1 invalid
X >= 1 valid
Test cases would be selected from each equivalence
class.
243
 Boundary Value Analysis (1/2)
 a testing technique where test cases are
designed to test the boundary of an input
domain. “Bugs lurk in corners
 Boundary value analysis complements and congregate at
boundaries…”
and can be used in conjunction with Boris Beizer

equivalence partitioning.
Example (cont.): boundary values of the
three ranges have to be included in the
test data. That is, we might choose the
following test cases:
X <= -2 MIN, -100, -2.1, -2
-1.9, -1, 0, 0.9
-2 < X < 1 1, 1.1,100, MAX
X >= 1
244
 Boundary Value Analysis (2/2)
If an input condition specifies a range [a, b]: Example: [-3, 10],
test values: 0 (interior point); 0 is always good candidate!
test values: -3, 10 (extreme points)
test values: -2, 9 (boundary points)
test values: -4, 11 (off points)
If an input condition specifies a number values:
minimum and maximum
values just above and below minimum and
maximum
Also for more complex data (data structures):
array
input condition: empty, single element, full element, out-of-
boundary
search for element: element is inside array or not
You can think about other data structures: list, set, stack, queue,
and tree 245
 (3) Specification-Based Testing
Check the program against a reference document design specification,
a requirements list, a user interface description, a model, or a user
manual.
Rational:
the specification might be part of a contract products must conform to
their advertisements safety-critical products must conform to any
safety-related specification.
Observation
Specification-driven tests are often weak, not particularly powerful
representatives of the class of possible tests
Different approaches:
focus on the specification: the set of test cases should include an
unambiguous and relevant test for each claim made in the spec. look
further, for problems in the specification: explore ambiguities in the
spec. or examine aspects of the product that were not well-specified
246
 (4) Risk-Based Testing
Process
(1) Imagine a way the program could fail (cf. error guessing)
(2) design one or more tests to check for it
a good risk-based test = a powerful representative that address
a given risk.
“complete”: a list of every way the program could fail.
If tests tie back to
significant failures in the field or
well known failures in a competitor’s product failure will be
highly credible and highly motivating
However, many risk-based tests are dismissed as academic
(unlikely to occur in real use).
Test for possible problem carry high information value
(We learn a lot whether the program passes the test or not)
247
 (5) Stress Testing
Different definitions of stress testing:
Hit the program with a peak burst of activity and see it fail.
Testing conducted to evaluate a system or component at or beyond the
limits of its specified requirements with the goal of causing the system to
fail. [IEEE Standard 610.12-1990]
Driving the program to failure in order to watch how the program fails.
Keep increasing the size or rate of input until either the program finally fails
or
you become convinced that further increases won’t yield a failure.
Look at the failure and ask what vulnerabilities have been exposed and which
of them might be triggered under less extreme circumstances.
Discussion:
stress test results as not representative of customer use
One problem with stress testing is that a failure may not be easily analyzed
(must includes enough diagnostic support)
stress-like tests can expose failures that are hard to see if the system is not
running with a high load (e.g., many concurrent tasks)

248
 (6) Regression Testing
Process
(1) design tests with the intent of regularly reusing them
(2) repeat the tests after making changes to the program.
A good regression test is designed for reuse.
adequately documented and maintainable
designed to be likely to fail if changes induce errors in the addressed part
of the program
Discussion:
First run of the tests may have been powerful, but after passed
many times detecting defects is not very likely (unless there have
been major changes or changes in part of the code directly involved
with this test.) Usually carries little information value
Automation can make regression testing very “cheap”

249
 Requirements changes
Regression Testing Process
Affect design, coding, and

Software Change
testing document update

Analysis

 Design changes

Software Change
Affect coding, tests, associated

Impact Analysis
components, system architecture,

related component interactions

Define regression
testing strategy
 Implementation changes

Build Regression
Test Suite
Affect test cases, test data, test scripts,

test specification (see also code change impact)

Run RegressionTests at
 Test changes

different levels
Affect other tests and test documentation

Report Retest
 Document changes

Results
Affect other document

250
 Regression Testing Strategies
Random: The test cases are randomly selected from the

existing test suite.

Retest-all: Run all the tests in the existing suite.

Safe: The test selection algorithm excludes no test from the

original test suite that if executed would reveal faults in the

modified program.

Based on modifications: Place an emphasis on selecting existing

test cases to cover modified program components and those that

may be affected by the modifications.

Dataflow/coverage based: Select tests that exercise data

interactions that have been affected by modifications.

Remark ([Juriso+2004]):

For small programs and set of test cases is small, then retest-all.

Use safe techniques for large programs and programs with a large

number of test cases.

251
 (7) User Testing
The testing done by real users (Not testers pretending to be users)
User tests might be designed by
the users,
the testers, or
other people (sometimes even by lawyers; acceptance tests in a contract)
Continuum:
tests can be designed in such detail that the user merely executes them reports
whether the program passed or failed
a carefully scripted demonstration, without much opportunity to detect defects
Leave the user some freedom to
cognitive activity while providing enough structure to ensure effective reports (in a way that
helps readers understand and troubleshoot the problem)
detect problems a user will encounter in real use is much more difficult
Failures found in user testing are typically credible

252
Beta tests:
often described as cheap, effective user tests
but in practice they can be quite expensive to
administer
you may not yield much information.
automation required!

253
 (8) Scenario Testing
Check how the program copes with a scenario
Good scenario test should be
credible,
motivating,
easy to evaluate, and
complex.
How often run a given scenario test?
Some create a pool of scenario tests as regression tests
Others run a scenario once or a small number of times and then design
another scenario rather than sticking with the ones they’ve used before
Scenarios help to develop insight into the product
Early in testing (understand the product at all) or
Late in testing (understand advanced uses of a stabilized product)
harsher test: rarely be used by typical users (cf. stress testing)

254
 (9) State-Model-Based Testing
Model the visible behavior of the program as a state machine
Checking for conformance with the state machine model
Approach is credible, motivating and easy to troubleshoot
Tradeoffs:
state-based testing often involves simplification and if the model is
oversimplified, failures exposed by the model can be false negatives and
difficult to troubleshoot
more detailed models find more bugs, but can be much harder to read and
maintain
Experiences shows that much of the bug-finding happens while
modeling the state models rather than testing
coverage criteria / stopping rule (cf. [Al-Ghafees&Whittaker2002])

255
 State-Model-Based Testing
Uncovers the following issues:
Wrong number of states.
Wrong transition for a given state-input combination.
Wrong output for a given transition.
States or sets of states that have become dead.
States or sets of states that have become unreachable.

Test sequence:
Define a set of covering input sequences that
starting from the initial state and get back to the initial state
For each step in the input sequence, defines the expected next state, the expected
transition, and the expected output code

Remarks:
Completeness: transitions and outputs for every combination of input and states
Instrument the transitions to also capture the sequence of states (not just the outputs)
Count the states

256
 Variant: Syntax Testing
Test model: BNF-based syntax instead of state machine
E.g., for testing command-driven software
Dirty syntax testing:
a) to exercise a good sample of single syntactic errors in all
commands in an attempt to break the software.
b) to force every diagnostic message appropriate to that
command to be executed.
Example test cases: Syntax Graph for <real_number>:
.
correct ones <digital> <digital>
1.54683e5,
1.6548
incorrect without loops:
.05 E
1. e <digital>
1.1e
D +
incorrect with loops: d -
123456.78901.1
1.12345e6789D-123
257
 (10) High-Volume Automated
Testing
Massive numbers of tests, comparing the results against a partial
oracles.
The simplest partial oracle: crashing
State-model-based testing (if the stopping rule is not based on a coverage
criterion) or also more general notion of stochastic state-based testing
[Whittaker1997].
Randomly corrupt input data
Repeatedly feeds random data to the application under test and a reference
oracle in parallel [Kaner2000] (back-to-back testing)
Run arbitrarily long random sequence of regression tests to let memory
leaks, stack corruption, wild pointers or other garbage that cumulates over
time finally causes failures. Probes (tests built into the program) can be
used to expose problems. [Kaner2000]
An almost complete automation for the testing is required
The low power of individual tests is make up for with massive numbers

258
 (11) Exploratory Testing
any testing which uses information gained while testing to better test
continuum between
purely scripted (the tester does precisely what the script specifies and nothing
else) to purely exploratory (no pre-specified activities, not documentation
beyond bug reports is required)
 prototypic case/ “freestyle exploratory testing”
exploratory testers continually learn about the software, the context, the ways it
could fail, its
weaknesses, and the best ways to test the software.
At the same time they are also test the software and report the problems
Test cases are good to the extent that they advance the tester’s knowledge
(goal-driven)
as the tester gains new knowledge the goals may also change quickly
which type of testing: what is most likely to reveal the information the tester is
looking for

259
 Not purely spontaneous:
studying competitive products,
review failure histories of this and analogous
products
interviewing programmers and users,
reading specifications, and
working with the product.
 state models can serve as an exploratory testing
tool

260
 III.3.3 White Box Testing
Techniques
 Black box testing:

Requirements fulfilled

Interfaces available

and working

 But can we also exploit

the internal structure

of a component,

interactions between

objects?

white box testing

White box testing:

(1) Control flow testing

(2) Data flow testing

(3) Mutation testing

261
 (1) Control Flow Testing: Example
procedure XYZ is 1
A,B,C: INTEGER;
begin 2
1. GET(A); GET(B); GET(C);
2. if (A > 15 and C > 20) then
3 9
3. if B < 10 then
4. B := A + 5;
5. else 4 6
6. B := A - 5;
7. end if
8. else 7
9. A := B + 5;
10. end if;
10
end XYZ;

262
 Control Flow Testing
a) Statement coverage: The test cases are

generated so that all the program statements are

executed at least once.

b) Decision coverage (branch coverage): The test

cases are generated so that the program

decisions take the value true or false.

c) Condition coverage: The test cases are

generated so that all the program conditions

(predicates) that form the logical expression of the

decision take the value true or false.

d) Path coverage: Test cases are generated to

execute all/some program paths.

263
 a) Statement Coverage
Statement 1

coverage: 2
The test cases are generated so
that all the program statements
are executed at least once. 3 9

4 6

10

264
 b) Decision / Branch Coverage
Decision 1

coverage 2
(branch
coverage): The 3 9
test cases are
generated so 4 6
that the program
decisions take
the value true or 7
false.
10

265
 c) Condition Coverage (1/2)
1
Conditioncoverage: The
test cases are generated 2
so that all the program
conditions (predicates) 3 9
that form the logical
expression of the 4 6
decision take the value
true or false.
7

10

266
Simple Condition Coverage
Composed condition:
((A and B) or C) A B C (A and B) or C
Test cases must cover all TRUE TRUE TRUE TRUE
the values true or false for TRUE TRUE FALSE TRUE
each conditions (predicates)
of the logical expression TRUE FALSE TRUE TRUE

combinatorial explosion TRUE FALSE FALSE FALSE

only coverage for simple FALSE TRUE TRUE TRUE
condition coverage FALSE TRUE FALSE FALSE
FALSE FALSE TRUE TRUE
Simple condition: (A) FALSE FALSE FALSE FALSE
degenerates to decision
coverage but simple Remarks:
condition coverage does not complete condition evaluation
ensure decision coverage assumed
Otherwise compiler dependent
execution semantics
267
 Other Condition Coverage Types
Simple condition coverage: require that the tests must
cover both true and false for each conditions (predicates) of the logical
expression.
Condition/decision coverage: require in addition that the coverage w.r.t. the
decision is also complete.
Minimal multiple condition coverage: also cover all sub expressions of the
Boolean formula
Modified condition/decision coverage: requires that
every atomic condition can influence the overall output
(required for level A equipment by RTCA DO-178 B).
Multiple condition coverage: test all combinations of the atomic conditions.

268
 d) Path Coverage
1
Path coverage: Test cases are
generated to 2
execute all/some program paths.
Why path coverage?
logic errors and incorrect 3 9
assumptions are inversely
proportional to a path’s execution
4 6
probability we often believe that a
path is not likely to be executed; in
fact, reality is often counter 7
intuitive it’s likely that untested
paths will contain some undetected
defects 10

269
Exhaustive Testing Not Possible
There are 1014 possible
paths!
If we execute one test
per millisecond, it
would take 3.170 years
to test this program!!

Out of question

270
 Basic Path Testing
 Terminology:
execution path: a path that connects a start node and a terminal
node.
Two paths are independent if they have disjoint node or edge sets
 Steps:
convert the unit into a “flow graph”
compute a measure of the unit's logical complexity
use the measure to derive a “basic” set of independent
execution paths for which test cases are determined

Remark: Path coverage set is not unique!

271
 Cyclomatic Complexity
Cyclomatic complexity (McCabe complexity) is a metric,
V(G), that
describes the logical complexity of a flow graph, G.
V(G) = E - N + 2 where E = #edges in G and N = #nodes in G
Studies have shown:
V(G) is directly related to the number of errors in source code
V(G) = 10 is a
practical upper
limit for testing modules in this range
are more error prone

modules

V(G)
272
 Path Coverage Testing
Next, we derive the independent paths:
1
Since V(G) = 4, we have 4 paths
Path 1: 1,2,3,6,7,8 2

Path 2: 1,2,3,5,7,8 4 3
Path 3: 1,2,4,7,8
5 6
Path 4: 1,2,4,7,2,4…7,8
Finally, we derive test cases to
exercise these paths, i.e. choose
7
inputs that lead to traversing the
paths 8

273
simple loop Testing Loops

nested
loops
concatenated unstructured
loops loops 274
 Loop Testing: Simple Loops

Minimum conditions - simple loops

1. skip the loop entirely

2. only one pass through the loop

(boundary)

3. two passes through the loop

(interior)

4. m passes through the loop m < n

5. (n-1), n, and (n+1) passes through the loop

where n is the maximum number of allowable passes

275
 Testing Nested Loops

Just extending simple loop testing:

number of tests explodes
Reduce the number of tests:
start at the innermost loop; set all other
loops to minimum values
conduct simple loop test; add out of range
or excluded values
work outwards while keeping inner nested
loops to typical values
continue until all loops have been tested

276
 Comparison: Control Flow Testing
a) Statement coverage: The test cases are generated so
that all the program statements are executed at
least once.
b) Decision coverage (branch coverage): The test cases are
generated so that the program decisions take the
value true or false.
c) Condition coverage: The test cases are generated so
that all the program conditions (predicates) that
form the logical expression of the decision take the value
true or false.
d) Path coverage: Test cases
are generated to execute
all/some program paths.

277
a) Statement Simple condition
coverage coverage

b) Decision
Condition/decision
coverage
coverage

Minimal multiple
condition coverage
Basic
path
coverage
Modified
condition/decision
coverage

Path
coverage
Multiple condition
coverage

278
 (2) Data Flow Testing
Basic Definitions:
1
V = the set of variables.
N = the set of nodes
E = the set of edges 2 read(x,y) def (2) = {i}
i:=1; c-use(2) = {x,y}
Definition/write on variables (nodes):
p-use(3,4) = {i}
def(i) = { x in V | x has a global i<=2 3 i>2 p-use(3,5) = {i}
def in block I }
4 5
p-use(6,8) = {y}
Computational-use of variables (nodes): writeln(“hello”); p-use(6,7) = {y}
y>=0
y<0 6
c-use(i) = { x in V | x has a global i:= i+1;
8
c-use in block i } def (4) = {i} 7
c-use(4) = {i}
Predicate-use of variables (edges): 9 writeln(x);
c-use(8) = {x}
p-use(i,j) = { x in V | x has a p-use
in edge (i,j) } 10

279
 Idea of Data Flow Testing
Idea: coverage of the a def-clear 1

path between definition and

usage 2 read(x,y) def (2) = {i}
i:=1; c-use(2) = {x,y}
Examples:
p-use(3,4) = {i}
One path per def (all-defs) i<=2 3 i>2 p-use(3,5) = {i}
One path per def to all c-uses
(all-c-uses) 4 5
p-use(6,8) = {y}
One path per def to all p-uses writeln(“hello”); p-use(6,7) = {y} 6
y<0
y>=0
(all-p-uses) i:= i+1;
def (4) = {i} 7 8
One path per def to all p-uses or c-use(4) = {i}
c-uses (all-uses) 9 writeln(x
c-use(8) = {x}
All path (without loops) from defs
to all p-uses or c-uses (all du- 10

paths)

280
 Data Flow Testing
1
All node with an effective path from a def to a c-use
for variable x:
dcu(x, j) = { j in N | x in c-use(j) and read(x,y) def (2) = {i,x,y}
there is a def-clear path wrt 2
x from i to j } i:=1; c-use(2) = {}

All edges with an effective path from a def to a p-p-use(3,4) = {i} i>2 p-use(3,5) = {i}
use for variable x: i<=2 3

dpu(x, j) = {(j,k) in E | x in p-use(j,k) and

there is a def-clear path wrt x 4 5
from i to (j,k) } p-use(6,8) = {y}
writeln(“hello”); p-use(6,7) = {y} y>=0
All effective path from a def to use without loops: i:= i+1; y<0 6

du-c(x,i,j) = { i,…,k,j in N* | x in def(i), x in

c-use(j), the path is def-clear wrt x, def (4) = {i} 7 8
and no node is visit twice} c-use(4) = {i}
9 writeln(x);
du-p(x,i,j) = { i,…,k,j in N* c-use(8) = {x}
p-use(k,j), and the path i,…,k is
def-clear wrt x, and no node i,…,k 10
is visit twice}
du(x,i,j) = du-c(x,i,j) ∪du-c(x,i,j)

281
 Types of Data Flow Testing
All-defs: Test cases are generated to cover each definition of each
variable for at least one use of the variable.
All c-uses: Test cases are generated so that there is at least one path
of each variable definition to each c-use of the variable.
All p-uses: Test cases are generated so that there is at least one path
of each variable definition to each p-use of the variable.
All-c-uses/some-p-uses: Test cases are generated so that there is at
least one path of each variable definition to each c-use of the variable. If
there are variable definitions not covered, use p-uses.
All-p-uses/some-c-uses: Test cases are generated so that there is at
least one path of each variable definition to each p-use of the variable. If
there are variable definitions not covered, use c-uses.

282
 Types of Data Flow Testing
All uses: Test cases are generated so that there is at least one path of
each variable definition to each use (read: p-use and c-use) of the
definition.
All-du paths: Test cases are generated for all the possible paths of
each definition of each variable to each use of the definition not
taking
loops into account.

283
 Comparison: Data Flow Testing
If time is an issue: All-Paths
All p-uses should be used instead
of all-uses All-DU-Paths
All-uses should be used
All-Uses
instead of all-du-paths
to generate fewer tests: All-C-Uses/Some-P-Uses All-P-Uses/Some-C-Use

use the test cases All-C-Uses All-DEFS All-P-Uses

covered by the (stronger) decision

coverage
criteria.
All-du-paths is usable in statement
coverage
practice.
284
 (3) Mutation Testing
Strong (standard) mutation: Test cases are generated to cover all
the mutants generated by applying all the mutation operators defined
for the programming language in question.

Selective (or constrained) mutation: Test cases are generated to

cover all of the mutants generated by applying some of the mutation
operators defined for the programming language in questions.

Weak mutation: Test cases are generated to cover a given

percentage of mutants generated by applying all the mutation
operators defined for the programming language in question.

285
 III.3.4 Comparison
Black box tests: White box tests:
tester has no access to tester can access
information about the information about the
system implementation system implementation

Characteristics: Characteristics:
Good for independence of Simplifies diagnosis of
tester results
But not good for formative Can compromise
tests independence
Hard to test individual How much do they need to
modules... know?

286
 III.4 Testing Process & Activities
Beforehand:
Requirement analysis
design analysis

Testing Process & Activities

(1) Unit test
(2) Integration test
(3) Function test
(4) Performance test
System test
(5) Acceptance test
(6) Installation test

287
 Testing Activities
Unit Design System Other Customer User
test specifications functional software requirements environment
requirements requirements specification
Unitcode

Unit
test
Unitcode
Integration Function Performance Acceptance Installation
test test test test test

Integrated Functioning Verified, Accepted

modules system validated system
software
Unit
test SYSTEM
IN USE!
Unitcode

288
 III.4.1 Unit Testing
Individual components are tested independently to ensure their
quality.
The focus is to uncover errors in design and implementation,
including
data structure in a component
program logic and program structure in a component
component interface
functions and operations of a component
There is some debate about what constitutes a “unit”. Some
common definitions:
the smallest chunk that can be compiled by itself
a stand alone procedure of function
something so small that it would be developed by a single person
Unit testers: developers of the components.

289
 Example: Test GCD Algorithm
Testing a unit designed to compute the “greatest common divisor”
(GCS) of a pair of integers (not both zero).
GCD(a,b) = c where
c is a positive integer
c is a common divisor of a and b (e.g., c divides a and c divides
b)
c is greater than all other common divisors of a and b.
For example
GCD(45, 27) = 9
GCD (7,13) = 1
GCD(-12, 15) = 3
GCD(13, 0) = 13
GCD(0, 0) undefined

290
 Test Planning
How do we proceed to determine the tests cases?
1.Determine appropriate equivalence classes for the
2.input data.
3.Determine the boundaries of the equivalence classes.
4.Design an algorithm for the GCD function.
5.Analyze the algorithm using basic path analysis.
6.Then, choose tests cases that include the basic path
7.set, data form each equivalence class, and data at and
8.near the boundaries.

291
1.
GCD
function gcd (int a, int b) {
Algorithm
2. int temp, value;
3. a := abs(a); 1
4. b := abs(b);
5. if (a = 0) then
6. value := b; // b is the GCD 5
else if (b = 0) then
8. raise exception; 7
9. else
10. loop 6
11. temp := b; 9

12. b := a mod b;
13. a := temp;
14. until (b = 0) 10

15. value := a;
16. end if; 17
17. return value;
18. end gcd
18
292
 Example: GCD Test Planning
Equivalence Classes
Although the the GCD algorithm should accept any
integers as input, one could consider 0, positive
integers and negative integers as “special” values.
This yields the following classes:
a < 0 and b < 0, a < 0 and b > 0, a > 0 and b < 0
a > 0 and b > 0, a = 0 and b < 0, a = 0 and b > 0
a > 0 and b = 0, a > 0 and b = 0, a = 0 and b = 0
Boundary Values
a = -231, -1, 0, 1, 231-1 and b = -231, -1, 0, 1, 231-1
Basic Path Set
V(G) = 4
(1,5,6,17,18), (1,5,7,18), (1,5,7,9,10,17,18),
(1,5,7,9,10,9,10,17,18)

293
Example GCD Test Plan

294
Test Description / Data Expected Results Test Experience / Actual Results
Basic Path Set

path (1,5,6,17,18) (0, 15) 15

path (1,5,7,18) (15, 0) 15

path (1,5,7,9,10,17,18) (30, 15) 15

path (1,5,7,9,10,9,10,17,18) (15, 30) 15

Equivalence Classes

a < 0 and b < 0 (-27, -45) 9

a < 0 and b > 0 (-72, 100) 4

a > 0 and b < 0 (121, -45) 1

a > 0 and b > 0 (420, 252) 28

a = 0 and b < 0 (0, -45) 45

a = 0 and b > 0 (0 , 45) 45

a > 0 and b = 0 (-27, 0) 27

a > 0 and b = 0 (27, 0) 27

a = 0 and b = 0 (0 , 0) exception raised

Boundary Points

(1 , 0) 1

(-1 , 0) 1

(0 , 1) 1

295
(0 , -1) 1
 Test Implementation
Once one has determined the testing strategy, and the units to tested, and
completed the unit test plans, the next concern is how to carry on the tests.
If you are testing a single, simple unit that does not
interact with other units (like the GCD unit), then one can
write a program that runs the test cases in the test plan.

However, if you are testing a unit that must interact with

other units, then it can be difficult to test it in isolation.

The next slide defines some terms that are used in implementing and running test
plan.

296
 Test Implementation Terms
Test Driver
a class or utility program that applies test cases to a component being tested.
Test Stub
a temporary, minimal implementation of a component to increase controllability
and observability in testing.
When testing a unit that references another unit, the unit must either be complete
(and tested) or stubs must be created that can be used when executing a test case
referencing the other unit.
Test Harness
A system of test drivers, stubs and other tools to support test execution

297
 Test Implementation Steps
Here is a suggested sequence of steps for testing a unit:

1.Once the design for the unit is complete, carry out

Complete a test plan for a unit.
create stubs for not yet completed referenced other units
Create a driver (or set of drivers) for the unit
construction of test case data (from the test plan)
2.Once the implementation is complete, carry out
Use the driver (or set of drivers) to test the unit
execution of the unit, using the test case data
provision for the results of the test case execution to be printed or logged as
appropriate

298
 Testing Tools

There are a number of tools that have been

developed to support the testing of a unit or system.
googling “Software Testing Tools” will yield
thousands of results.
JUnit testing (http://www.junit.org/index.htm) is a
popular tool/technique that can be integrated into the
development process for a unit coded in Java.

299
 III.4.2 Integration Testing
A group of dependent components are tested together to ensure their
the quality of their integration unit.

The focus is to uncover errors in:

Design and construction of software architecture
Integrated functions or operations at sub-system level
Interfaces and interactions between them
Resource integration and/or environment integration

Integration testers:
either developers and/or test engineers.

300
 III.4.2 Integration Testing
Tests complete systems or subsystems composed of
integrated components
Integration testing should be black-box testing with tests
derived from the
specification
Main difficulty is localising errors
Incremental integration testing reduces this problem

301
 Integration Testing Strategies
Incremental testing strategies:
Bottom-up testing: Integrate individual components in levels until
the complete system is created
Top-down testing: Start with high-level system and integrate from
the top-down replacing individual components by stubs where
appropriate
Outside-in integration: Do bottom-up and top-down testing at the
same time such that the final integration step occurs in a middle layer
Non incremental testing strategies:
Big bang testing: put all together and test it
Remark: In practice, most integration involves a combination of
these strategies

302
 Big Bang Integration Testing

combine (or integrate) all parts at once.

Advantages:

simple
Disadvantages:

hard to debugging, not easy to isolate errors

not easy to validate test results

impossible to form an integrated system

303
 Bottom Up Integration Testing
Idea:Modules at the lowest levels are integrated at first, then by moving
upward through the control structure.
Integration process (five steps):
1. Low-level modules are combined into clusters that perform a specific
software sub-function
2. A driver is written to coordinate test case input and output
3. Test cluster is tested
4. Drivers are removed and clusters are combined moving upward in the
program structure

304
 Bottom Up Integration Testing
Pros and cons of bottom-up integration:
no stubs cost
need test drivers
no controllable system until the last step

305
 Bottom Up Integration Testing
Stage 4 M11

Integration B Integration c

Stage 3 M9
M10

Integration A

Stage 2 M8

Stage 1 M1 M2 M3 M4 M5 M6 M7

306
 Top Down Integration Testing
Idea:
Modules are integrated by moving downward through the control
structure.
Modules subordinate to the main control module are incorporated into
the
system in either a depth-first or breadth-first manner.

307
Integration process (five steps):
1. the main control module is used as a test driver, and the stubs are
substituted
for all modules directly subordinate to the main control module.
2. subordinate stubs are replaced one at a time with actual modules
3. tests are conducted as each module is integrated
4. On completion of each set of tests, another stub is replaced with
the real
module
5. regression testing may conducted
Pros and cons top-down integration:
stub construction cost
major control function can be tested early.

308
 Top Down Integration Testing
Integration D
Integration C

Integration B
Integration A
Stage 1 M11

Stage 2 M9 M10

Stage 3 M8

Stage 4 M6 M7

Stage 5 M1 M2

Stage 6 M3 M4 M5
309
 Test Stubs and Test Drivers
Top-down testing of module M8 Bottom-up testing of module M8

Module
tested in an Drive
M9 earlier of M9
stage

M8 Module M8 Module
on test on test

Modules
Stub Stub tested in an
of M1 of M2 M1 M2 earlier
stage

310
 Regression Testing & Integration
A module firewall:
closure of all possible
affected modules and
in a program based on a
control-flow graph. Main

Reduction of the
4

software regression
testing to a smaller M1 M2 M3
scope (the firewall)
3
This implies that: Changed Module

re-test of the changed

M4 1 M5 M6 M7
module and its affected
modules 2

re-integration for all of

A module firewall:
M8 - M5, M1, and Main
related integration links Re-testing at the unit level: M5
Re-integration steps: 2, 3, 4

311
 Comparison of the Approaches
Problems of non-incremental integration:
hard to debugging, hard to isolate errors not easy to validate test results
impossible to form an integrated system
Pros and cons top-down integration:
stub construction cost
major control function is tested early
Pros and cons of bottom-up integration:
no stubs cost
need test drivers
no controllable system until the last step
Pros and cons outside-in integration:
stub construction cost
major control function is tested early
need test drivers

312
Architectural validation
Top-down/Outside-in integration testing is better at discovering errors in
the system architecture
System demonstration
Top-down/Outside-in integration testing allows a limited demonstration
at an early stage in the development
Test implementation
Often easier with bottom-up integration testing
Test observation
Problems with all approaches. Extra code may be required to observe
tests

313
 III.4.3 Function Testing
The integrated software is tested based on requirements to ensure that
we have a right product (validate functional requirements).
The focus is to uncover errors in:
System input/output
System functions and information data
System interfaces with external parts
User interfaces
System behavior
Function testers: External interfaces
test engineers or SQA people.
User System
interface
User (Operations &
Functions
& Behavior)

314
 III.4.4 Performance
Performance testing is designed to test
Testing
Relevant attributes:
run-time performance of software within System process speed
the context of an integrated system (Max./Min./Average)
(validate non-functional requirements).
System throughput
The focus areas are: (Max./Min./Average)
Confirm and validate the specified system System latency
performance requirements. (Max./Min./Average)
Check the current product capacity to System utilization
answer the questions from customers and (Max./Min./Average)
marketing people. System availability
Identify performance issues and (component-level/system-level)
performance degradation in a given System reliability (component-
system level/system-level)
System behavior in the special load System scalability
conditions (component-level/system-level)
System successes/failures rates
Performance testers:
test engineers or SQA people.

315
 III.4.5 Acceptance Testing
The system software is tested as a whole. It verifies all elements mesh
properly to make sure that all system functions and performance are
achieved in the target environment.

The focus areas are:

System functions and performance
System reliability and recoverability (recovery test)
System behavior in the special conditions (stress)
System user operations (acceptance test/alpha test)
Hardware and software integration and collaboration
Integration of external software and the system
System testers:
test engineers or SQA people.

316
 III.4.6 Installation Testing
The installation testing is designed to test the installation procedure
and its supported software.

The focus areas are:

Hardware and software integration and collaboration
Integration of external software and the system

Installation testers: test engineers or SQA people.

317
 III.5 Management & Automation
Management

The testing process

Scope: Unit, integration & system testing

Test methodology

Test planning

Test execution

Alpha and beta site testing programs

Scope: customer or user

Automation:
Cost effective testing

318
 Test Management
Test management encompass:
management of a test team
management of a test process
management of test projects
Management role:
Motivation: Motivate quality work and stimulate creative solutions
Methodology: Test methodology, process, standards, and test criteria
Mechanization/Selection/Development: test tools, configuration
management of test suites, integrated test environment
Measurement: Test cost, complexity and efforts, engineer
performance, test effectiveness, product quality

319
 Roles in Testing
A test engineer’s role:
A test manager’s role:
Responsibilities:
Play a leadership role in:
Ensure that testing is
planning projects
performed and documented as
motivating people
defined by the employed
build a team and manage
testing approach
Basic skills:
engineers

Technical awareness of testing

Play as a controller in

methods, tools, and criteria

product evaluation

Organized; systematic
performance evaluation

planning
Play as a supporter in:

Inner determination to
assist and train engineers

discover
enforce and control test

problems Pay good attention

methods, standards, and criteria

select and develop test tools

to the details
Ability to related to others and
resolve conflicts 320
 III.5.1 The Testing Process
Phases: Determining the test
(1) Determining the test methodology
methodology phase
Planning the tests
(2) Planning the tests
(3) Test design
Designing the tests
(4) Test implementation

Performing the tests

(implementation)

321
 (1) Determining the Test
Methodology
Determining the appropriate SQA standard

Depends on the application domain

The risk associated with the product

Determine the software testing strategy

Big Bang, Bottom-Up, Top-Down, …

Where to use white-box testing

Where can automated testing be applied?

322
 (2) Planning the Test

Scope:

Unit test, Integration test, system test

Must address:

a) What to test?

b) Which sources to be used for the test?

c) Who is performing the tests?

d) Where to perform the tests?

e) When to terminate the tests?

323
 a) What to test?
Must address:
Which modules should be unit tested?
Which integrations should be tested?
Determine priorities for allocating test resources
to the individual tasks or applications

Rating:
Damage severity level: estimate the severity of
results in case the module/application fails
Software risk level: probability of failure

324
 Damages to Customers and Users
1. Endangers the safety of human beings
2. Affects an essential organizational function with no system
replacement capability available
3. Affects functioning of firmware, causing malfunction of an
entire system
4. Affects an essential organizational function but a
replacement is available
5. Affects proper functioning of software packages for
business applications
6. Affects proper functioning of software packages for a
private customer
7. Affects functioning of a firmware application but without
affecting the entire system.
8. Inconveniences the user but does not prevent accomplishment of the system’s capabilities

325
 Damages to the Software Developer
1. Financial losses

Damages paid for physical injuries

Damages paid to organizations for

malfunctioning of software

Purchase cost reimbursed to customers

High maintenance expanses for repair of failed

systems

2. Non-quantitative damages

Expected to affect future sales

Substantially reduced current sales

326
 Issues Affecting Software Risk
Level
Module/application issues
1. Magnitude
2. Complexity and difficulty
3. Percentage of original software (vs. percentage of reused software)
Programmer issues
4. Professional qualifications
5. Experience with the module's specific subject matter.
6. Availability of professional support (backup of
knowledgeable and experience).
7. Acquaintance with the programmer and the ability to
evaluate his/her capabilities.

327
 Priorities/Ratings for
Modules/Applications
A = Damage severity level
B = Software risk level
Combined Ratings
C=A+B
C=k*A+m*B
C=A*B (≈Risk = Severity x Probability)
Employ Ratings to:
Assign testing priorities
Allocate resources for testing
Determine required techniques

328
 Example: Super Teacher
Combined rating method
Application Damage Software A +B 7xA+2xB AxB
Severity risk
Factor A Factor B
1. Input of test results 3 2 5 (4) 25 (5) 6 (4)
2. Interface for input and output of pupils’ data to 4 8 (1) 36 (1) 16 (1)
4
and from other teachers
3. Preparation of lists of low achievers 2 2 4 (5-6) 18 (7) 4 (5-6)
4. Printing letters to parents of low achievers 1 2 3 (7-8) 11 (8) 2 (8)
5. Preparation of reports for the school principal 3 3 6 (3) 27 (3) 9 (3)
6. Display of a pupil’s achievements profile 4 3 7 (2) 34 (2) 12 (2)

7. Printing of pupil’s term report card 3 1 3 (7-8) 23 (6) 3 (7)

8. Printing of pupil’s year-end report card 4 1 4 (5-6) 26 (4) 4 (5-6)

Super Teacher is a software package designed to support

teacher in managing the grades of elementary school pupils.
The package contains eight applications.
329
 b) Which Sources to be used for
the test?
Alternatives:
Random samples of real life cases
(Preferable – Stratified sampling of real life cases )
Taken from real application context
Refers to existing customers, shipment, products, …
Synthetic test cases (simulated test cases)
Prepared by the test designer
Refers to combinations of system operating conditions
and parameters (set of input data)
Designed to cover all known/expected operating
situations

330
 Comparison of Test Data Sources
Implication Random Synthetic
Effort Low, especially where expected results High; parameters must be
are available determined and expected
results calculated
Size Relatively high; repetitions are frequent; Relatively small; repetitions
in order to obtain a sufficient number of can be avoided
non-standard situations a relatively large
number is required
Efforts to High efforts (low efficiency) Low effort (high efficiency);
perform tests
Effectiveness Relatively low; Relatively high; due to good
(probability of No coverage of erroneous cases coverage
error detection) Some ability to identify unexpected Good coverage of
errors erroneous situations
Little possibility to identify
unexpected errors

331
 c) Who is performing the tests?
Unit test:
developer or tester
Integration test:
developer or tester
System test:
independent tester
Large Systems:
Multiple test teams developer independent tester
Small organizations:
Understands the Must learn about the
Testing by another system but, will system, but, will
development team test “gently ” and, attempt to break it
Outsourcing of testing is driven by and, is driven by
“ delivery ” quality

332
 d) Where to perform the tests?
Unit test & Integration test:
Developer site
System test:
Developer Site or target site?
If external tester: consultant site
Simulate target site as good as possible
Still unpredicted failures of the installed system possible!

333
 e) When to terminate the tests?
Alternatives:

i.The completed implementation route

ii.The mathematical model application route

iii.The error seeding route

iv.The dual independent testing team route

v.Termination after resources have petered

vi.out
334
 i. Completed Implementation Route

Approach:
Testing is terminated once:
The entire test plan has been carried out
Error free (“clean”) results have been
achieved for all regression tests

Discussion:
Fits to perfection approach
Disregards budget and timetable constraints

335
 ii. Mathematical Model
Application Route
Approach:
A mathematical model is used to estimate the percentage of
undetected errors based on the rate of error detection
Define acceptable level of undetected errors
Determine an abort rate that corresponds to a predefined
level of undetected errors which is considered acceptable
Testing terminates when
The error detection rate declines below the abort rate

336
 ii. Mathematical Model
Application Route
Discussion:
The mathematical model may not fully represent the project
characteristics
Simple models provide no estimates about the severity of the undetected
errors

337
 iii. Error Seeding Route
Approach:
Errors of various types are seeded (hidden) in the tested
software prior to testing
Assumption: percentage of discovered seeded errors
corresponds to
the percentage of real errors
Define acceptable percentage of undetected errors
Testing terminates when
The residual percentage of undetected seeded errors
declines
below the acceptable percentage

338
 iii. Error Seeding Route
Discussion:
Additional workload for tester
Seeding distribution is totally based on past experience while testers
might experience “new and original” errors in every new project
only applicable for similar/familiar systems

339
 iv. Dual Independent Testing
Approach:
Team Route
Two teams implement the testing

process independently

Define acceptable percentage or

number of undetected errors

Assumption: statistically independent

testing by both teams and random

detection of errors

Testing terminates when

The residual percentage/number of

undetected errors is below the

acceptable percentage/number

Discussion:

Only applicable when there are two

independent test teams

Random detection of errors is a

strong assumption

Methodological basis may be

questioned when both teams share

similar testing experience or methodology

340
 v. Termination after Resources
Approach: have petered out …
Testing terminates when
budget or time allocated for testing run out
Discussion:
Undesirable but not uncommon
Systematic follow-up (for all cases):
Data collection about errors detected during development and regular
use (first
6 or 12 month)
Analysis of the error data: compare estimates supplied by the models
with the real figures
Comparative analysis of the severity of errors: Are the errors detected
during
regular use (first 6 or 12 month) more sever than that detected during
development?

341
 Software Test Plan (STP) - Template
1 Scope of the tests 3.3 Cross-reference to the relevant design
1.1 The software package to be tested document and the requirement documen
(name, version and revision) 3.4 Test class
3.5 Test level (unit, integration or system
1.2 The documents that provide the basis tests)
for the planned tests 3.6 Test case requirements
2 Testing environment 3.7 Special requirements (e.g.,
2.1 Sites measurements of response times,
security requirements)
2.2 Required hardware and firmware 3.8 Data to be recorded
configuration
2.3 Participating organizations 4 Test schedule (for each test or
2.4 Manpower requirements test group) including time
2.5 Preparation and training required of estimates for:
the test team 4.1 Preparation
4.2 Testing
3 Tests details (for each test) 4.3 Error correction
3.1 Test identification 4.4 Regression tests
3.2 Test objective

342
 (3) Test Design
Product of the test design stage:

Detailed design and procedures for each test

Test case database/file

Is carried out on the basis of the software test plan

(as documented by a STP)
The test case database/file may be documented by
a “software test procedure” and “test case file” document
or in a single document called “software test description”

343
 Software Test Description (STD)
1 Scope of the tests
- Template
3. Testing process
1.1 The software package to be tested 3.1 Instructions for input, detailing every
(name, version and revision) step of the input process
1.2 The documents providing the basis 3.2 Data to be recorded during the tests
for the designed tests (name and
version for each document) 4. Test cases (for each case)
2 Test environment (for each test) 4.1 Test case identification details
2.1 Test identification (the test details 4.2 Input data and system settings
are documented in the STP) 4.3 Expected intermediate results (if
2.2 Detailed description of the operating applicable)
system and hardware configuration 4.4 Expected results (numerical,
and the required switch settings for message, activation of equipment,
the tests etc.)
2.3 Instructions for software loading 5. Actions to be taken in case of
program failure/cessation
6. Procedures to be applied
according to the test results
summary
344
(4) Test Implementation
Phase activities:
Testing
Corrections of
detected errors
Re-testing
(regression
testing)
Is the quality of
the software
acceptable?
(Determine
termination)

345
 Test Execution
Test execution can be performed:
using manual approach
using a systematic approach
using a semi-automatic approach
Basis activities in test execution are:
Select a test case
Set up the pre-conditions for a test case
Set up test data
Run a test case following its procedure
Track the test operations and results
Monitor the post-conditions of a test case & expect the test
results
Verify the test results and report the problems if there is any
Record each test execution
346
 Software Test Report (STR) -
Template
1. Test identification, site, schedule and participation
1.1 The tested software identification (name, version and
revision)
1.2 The documents providing the basis for the tests (name and
version for each document)
1.3 Test site
1.4 Initiation and concluding times for each testing session
1.5 Test team members
1.6 Other participants
1.7 Hours invested in performing the tests
2. Test environment
2.1 Hardware and firmware configurations
2.2 Preparations and training prior to testing
347
 III.5.2 Alpha and Beta Site Tests
Programs
Alpha & Beta tests:

Test from the user perspective

Replace acceptance test

Should NOT replace testing by the developer/tester

Possible usually only for COTS

Alpha Site Tests:

Customer examine the program by applying to it the specific

requirements of his organization

Is done at the developer site

Beta Site Tests:

Free offer to potential users

Is done at the “early” user sites (called “beta sites”)

Often hundreds or even thousands participants

348
 Pro & Cons of Alpha and Beta
Site Tests
Advantages

Identification of unexpected errors.

A wider population in search of errors.

Low costs.

Disadvantages

A lack of systematic testing.

Low quality error reports.

Difficult to reproduce the test environment.

Much effort is required to examine reports.

Remark:

Support for online crash reports

everybody becomes a “beta” tester

349
 III.5.3 Test Automation
What is Software Test Automation?

Software test automation refers to the activities and efforts that intend to
automate engineering tasks and operations in a software test process
using well-defined strategies and systematic solutions.

Major objectives:
Reduce software testing time, cost, and efforts
Increase the quality of software testing
Apply well-defined test methods through tools
Relieve the complicated and redundant work from test
engineers
350
 Scope of Test Automation
Could be performed in three different scopes:
Enterprise-oriented test automation, where the major
focus of test automation efforts is to automate an
enterprise-oriented test process so that it could be used and
reused to support different product lines and projects in an
organization.
Product-oriented test automation, where test automation
activities are performed to focus on a specific software
product line to support its related testing activities.
Project-oriented test automation, where test automation
effort is aimed at a specific project and its test process.

351
 Main Types of Automated Testing
Code auditing
Module size, levels of nesting, levelsof subroutine nesting,
prohibited
constructs (GOTO), check naming conventions, check
programming
styles, check documentation styles, and other forms of
automated
static analysis: unreachable code, …
Test Management
Test plans, test results, and correction follow-up
E.g., Coverage monitoring: collect coverage information
such as
line coverage
Test execution
Functional testing: Automation regression testing; Output comparator
Load testing: Simulate load characteristics
Maintenance follow-up
352
 Automated and Manual Testing
by Phase
Testing process phase Automated Manual
testing testing
Test planning M M
Test design M M
Preparing test cases M M
Performance of the tests A M
Preparing the test log and test reports A M
Regression tests A M
Preparing the tests log and test reports M M
including comparative reports
M = phase performed manually,
A= phase performed automatically

353
 Maturity Levels of Test Automation
Level 4: Optimal
A systematic solutions to manage test information, execute tests, and generate
tests, and measure test coverage.
Level 3: Automatic
Besides the test management and test execution tools, a
Level 4: Optimal
software test process at this level is supported with additional
solutions to generate software tests using systematic methods.
Systematic Test Level 2: Repeatable
Measurement &
Optimization Level 3:Automatic Also systematic solutions to execute software
Systematic
tests in a systematic manner.
Test Generation Level 2: Repeatable

Systamatic
Level 1: Initial Test Execution
Systematic solutions to Control
Level 1: Initial
create, update, and Systematic Test
manage all types of Information
software test information Management

354
Maturity Levels and
Level 4: Automatic test measurement
Tools test coverage and metrics
measurement
test cost and complexity measurement
Level 3: Automatic test generation
black-box/white-box test generation
Level 2: Automatic test execution
Level 4: Optimal test control and execution
Level 1: Automatic test management
Systematic Test test case management, and
Measurement &
Optimization Level 3:Automatic documentation
test script management
Systematic
Test Generation Level 2: Repeatable

Systematic
Test Execution
Control Level 1: Initial

Systematic Test
Information
Management
355
 Essential Needs & Basic Issues
Essential Needs: Basic Issues:
A dedicated work force for
Poor manually performed
test automation software test process
The commitment from Late engagement of
senior managers and software test automation in
engineers a software product life
The dedicated budget and
cycle
project schedule Unrealistic goals and
A well-defined plan and
unreasonable expectations
strategy
Organization issues
Talent engineers and cost-
Lack of good
effective testing tools
understanding and
experience of software test
automation
Maintenance of automated

software tests and tools

356
A Software Test Automation Process
Steps:
Test automation planning: identify test
automation focuses, objectives, strategies,
Plan Software requirements, schedule and budget.
Test Automation
Test automation design: draw out the
detailed test automation solutions to
Design Test Automation achieve the major objectives and meet the
Strategies & Solutions
given requirements in a test automation
Select and plan.
Develop & Implement Test
Automation Solutions
Evaluate
Available
Test tool development: realize the test
Software
Testing
automation solutions; the developed tools
Tools must be reliable, usable and reusable
Introduce and Deploy Test
Automation Solutions Test tool deployment: introduce and
deploy the tools; including basic user
Review and Evaluate
training and user support
Software Test Automation Review and evaluation : a review should
be conducted to identify limitations and
evaluate the provided features; valuable
feedback to the test automation group for
further improvements and enhancements. 357
 Test Tool Types
Test Tool Types Basic Descriptions of Different Types of Test Tools
Test Information Systematic solutions and tools support test engineers and quality
Management assurance people to
create, update, and maintain diverse test information, including test
cases, test scripts,
test data, test results, and discovered problems.
Test Execution and Systematic solutions and tools help engineer set up and run tests, and
Control collect and
validate test results.
Test Generation Systematic solutions and tools generate program tests in an automatic
way.
Test Coverage Analysis Systematic solutions and tools analyze the test coverage during a test
process based
on selected test criteria.
Performance Testing Systematic solutions and tools support program performance testing and
and performance
Measurement measurement.
Software Simulators Programs are developed to simulate the functions and behaviors of
external systems,
or dependent subsystems/components for a under test program.
Regression Testing Test tools support the automation performance of regression testing and
activities,
including test recording and re-playing.

358
 Classification of Software Test Tools
There are three types of test information
management systems and tools.
Test information management tool: It supports engineers to create, update,
and
manage all types of test information, such as testing requirements, test cases,
data, procedures, and results.
Test suite management tool: It enables engineers to create, update, and
manage various software test scripts for test execution.
Problem management tool: It helps engineer
bookkeeping and manage the discovered problems during a test process.
There are three types of software simulators:
Model-driven program simulators: based on a specific model, such as a finite
state machine
Data-driven program simulators: functional program based on the given
inputs and return the pre-defined outputs.
Message-driven program simulators: communication process to generate
outgoing messages, based on the pre-defined incoming messages.
359
Regression test tools:
Software change analysis: This refers to a systematic facility that
identifies various types of software changes and discovers their ripple effects
and impacts.
Software test selection: This refers to a systematic facility that assists
engineers to select reusable tests for regression testing based on software
change information.
Test change analysis: This refers to a systematic solution that identifies the
necessary test changes based on the given software changes. The major task of
test change analysis is to find reusable tests and identify obsolete tests so that
new tests could be added, and existing tests will be updated.
Test recorder and re-player: This refers to a software tool that records the
executed tests, and re-plays them for re-testing. Mercury
Interactive’s Winrunner is a typical re-testing
tool that is able to record and re-play user-
system interactive sequences for window-
based programs using pre-defined test scripts.

360
 Classification of Software Test Tools
Types of Test Tools Test Tool Vendors Test Tools
Problem Management Tools Rational Inc. ClearQust, ClearDDTS
Microsoft Corp. PVCS Tracker
Imbus AG Imbus Fehlerdatenbank
Test Information Management Tools Rautional Inc. TestManager
Mercury Interactive TestDirectory
Test Suite Management Tools Evalid TestSuiter
Rational Inc. TestFactory
SUN JavaTest, JavaHarness
White-Box Test Tools McCabe & Associates McCabe IQ2
Junit
IBM IBM COBOL Unit Tester
IBM ATC
- Coverage Assistant
- Source Audit Assistant
- Distillation Assistant
- Unit Test Assistant

361
Classification of Software Test Tools
Test Execution Tools OC Systems Aprob
Softbridge ATF/TestWright
AutoTester AutoTester
Rational Inc. Visual Test
SQA Robot
Mercury Interactive WinRunner
Sterling Software Vision TestPro
Compuware QARun
Seque Software SilkTest
RSW Software Inc. e-Test
Cyrano Gmbh Cyrano Robot
Code Coverage Analysis Tools Case Consult Corp. Analyzer, Analyzer Java
OC Systems Aprob
IPL Software Product Group Cantata/Cantata++

ATTOL Testware SA Coverage

Compuware NuMega TruCoverage
Software Research TestWorks Coverage
Rational Inc PureCoverage
SUN JavaScope
ParaSoft TCA
Software Automation Inc Panorama
362
Classification of Software Test Tools
Load Test and Performance Tools Rational Inc. Rational Suite PerformanceStudio
InterNetwork AG sma@rtTest

Compuware QA-Load
Mercury Interactive LoadRunner
RSW Software Inc. e- Load
SUN JavaLoad
Seque Software SilkPerformer
Client/Server Solutions, Inc. Benchmark Factory
Regression Testing Tools IBM Regression Testing Tool(ARTT)
Distillation Assistant
GUI Record/Replay Software Research eValid
Mercury Interactive Xrunner
Astra Astra QuickTest
AutoTester AutoTester, AutoTester One

363
 Automated Testing
Advantages
Reduce manual software testing operations and eliminate redundant
testing efforts.
Few manpower resources required for performing of tests.
Shorter duration of testing.
Execute much more software tests and achieve a better testing
coverage in a very limited schedule.
Accuracy and completeness of performance.
Accuracy of results log and summary reports.
Comprehensiveness of information.
Produce more systematic repeatable software tests, and generate more
consistent testing results.
Performance of complete regression tests.
Performance of test classes beyond the scope of manual testing.

364
 Automated Testing
Disadvantages
High investments required in package purchasing and training.
High package development investment costs.
High manpower requirements for test preparation.
Considerable testing areas left uncovered.

365
 III.6 Paradigm & Domain
Specific Aspects
Paradigms: object-oriented, component-
based, agent-oriented, …
Testing of systems which apply object-
oriented technologies (OOA/OOD/OOP/…)

Applications domains: business, finance,

web, embedded systems, …
Testing embedded systems

366
 III.6.1 Testing Object-Oriented
Systems
Relatively little attention has been paid to considerations for testing OO implementations

Lack of attention is not lack of need; New opportunities for error come with the increased
power of OO languages:
each lower level in an inheritance hierarchy is a new context for inherited features
correct behavior at an upper level in no way guarantees behavior at a lower level
polymorphism with dynamic binding dramatically increases the number of execution
paths static analysis of source code is of relatively little help with these
situations
encapsulation is an obstacle to controllability and operability of implementation state

Components offered for reuse should be highly reliable:

each reuse is a new context of usage and re-testing is warranted
likely that more testing is needed to obtain high reliability in OO systems

367
 OO & Unit Testing
Unit of testing in OO context:

encapsulation drives definition of classes and objects:

each class and class instance (object), package attributes (data),

and operations (methods or services) that manipulate data

smallest testable unit is encapsulated class or object, rather

than the conventional module, because

a class can have a number of operations/methods and particular

methods may exist in a number of different classes

class testing for OO software ≡unit testing for conventional

software

class testing driven by encapsulated operations and state

behavior of class

368
 OO Testing: Easier vs. Harder
Easier
Modularity Inheritance Harder
Quicker
Polymorphism
development
Dynamic
Encapsulation binding
Small methods
Complex
Reuse interfaces
Interfaces
identified More integration
early

369
 Class Testing & States
Functional
model of
processing

OO model of
processing

Problem: The state

often due information
hiding not directly
accessible, but can only
be accessed using public
class operations
370
 Testing & Inheritance
Problems:

Inheritance defines new context for methods

in the child class, the method may be redefined and must

then be tested when the child is implemented

behavior of inherited methods can be changed because

methods, that are called within methods, have to be

tested per class

Polymorphism

can result in an number of alternative paths as object of

the parent class can be substituted by any parent class

object

371
 Object-Oriented Testing Issues
The object class is the basic testing unit for

system developed with OO techniques.

This is especially appropriate if strong cohesion and

loose coupling is applied effectively in the class

design.

An incremental testing approach is dictated by

The package/class design decomposition.

The incremental nature of the development process.

Information hiding restricts/constrains testing to

using a white-box approach.

Encapsulation motivates testing based on the class interface.

372
 Levels of Testing
1. Testing individual methods associated with objects:
black-box and white-box testing
2. Testing individual object classes:
black-box testing with “equivalence class” testing extended to
cover related operation sequences
structural (white-box) testing (requires a different type of analysis)
3. Testing clusters of objects :
strict top-down/bottom-up iteration inappropriate to create groups
or related groups explicit clustering
scenario-based testing appropriate
4. Testing the object-oriented system:
verification/validation against system requirements is carried out
as for any other type of system
373
 Class Testing (1/2)

Class test cases are created by examining the

specification of the class.

If you have a state model for the class, test

each transition - devise a driver that sets an

object of the class in the source state of the

transition and generates the transition event.

Class Constraints/Invariants should be

incorporated into the class test.

All class methods should be tested, but testing

a method in isolation from the rest of the class

is usually meaningless

374
 Class Testing (2/2)
To test a class, you may need instances
of other classes.

Interaction diagrams provide useful

guidance in constructing class test cases:
They provide more specific knowledge about
how objects of one class interact with
instances of other classes.
Messages sent to a class object provide a
guidance on which test cases are most critical for that class.

375
 Method Testing (1/2)

A public method in a class is typically tested

using a black-box approach.
Start from the specification, no need to look at the
method body.
Consider each parameter in the method signature,
and identify its equivalence classes.
Incorporate pre-conditions and post-conditions in
your test of a method.
Test exceptions.
For complex logic, also use white-box testing or
static testing.

376
 Method Testing (2/2)

For private methods, either

modify the class (temporarily) so that it can
be tested externally:
change the access to public
or incorporate a test driver within the class
or use static test methods, such as program
tracing or symbolic execution

377
 Incremental Testing
Incremental testing indicates that we test each

unit in isolation, and then integrate each unit,

one at a time, into the system, testing the

overall system as we go.

Classes that are dependent on each other

called class clusters, are good candidates for

an increment integration.

Candidate class clusters:

Classes in a package

Classes in a class hierarchy.

Classes associated with the interaction diagram for a

use case.

378
A Class Test Order for OO Programs
3
4
4.1 ButtonList Sensor
AG 3.4
AS AS
AG 3.3
AS
AS
AS
4.2 Event Scene
Button
AS I 3.1
AS
I 2
ButtonState AG
3 TextButton 3.2 World I
AS
AS
AS
I
1 I 1 2
1.1 5.2 I
Subject Control CanvasRep Canvas
AS
I
AS
AS AS AS
MonoScene

1.2 InstructorItr 5 ControlState 5.1

AG

379
 Integration Test Plan
An Integration Test checks and verifies that the
various components in a class cluster
communicate properly with each other.
With this plan, the emphasis is not on whether
system functionality is correctly implemented,
but rather do all the cluster classes “fit” together
properly.
A single test plan, for each cluster, which tests
the communication between all cluster
components is typically sufficient.

380
 Increment Test Planning
Determine the classes to be tested in an

increment.

Determine the appropriate “class clusters”.

Develop a unit test plan and test driver for each

class.

Develop an integration test plan and test driver

for each class cluster.

Develop a test script that details the

components to be tested and the order in which

the plans will be executed.

381
 III.6.1 Testing Embedded Systems

What is an embedded system?

What is the correct behavior?

Development Stages
Simulation
Prototyping
Pre-Production
Production

382
 What is an Embedded System?
Software
embedded into a
technical
system that
interacts with the
real physical
world, controlling
some specific
hardware.
Interaction with 383
 What is the Correct Behavior?
Continuous and discrete
signals in the
Time domain
Value domain
Exact match is not realistic
tolerance in
value and time

384
 Different Development Stages
(1) Simulation
(2) Prototyping
(3) Pre-Production
(4) Production

385
 (1) Simulation
One-way simulation (model test)
Test plant or embedded
software model

Feedback-simulation (model-in-the-loop)
Let the models of the plant and the
embedded software interact

Rapid Prototyping
Let the “real” plant (or a
simplified version) interact
with the model of the
embedded software (on
high performance hardware;
floating-point arithmetic)

386
 (2) Prototyping (1/3)
(1) Embedded software
Compiled for host
Compiled for target
platform
(2) Processor
High performance host
Target platform
emulator
(3) Rest of the embedded
system
Simulated on the host
Experimental hardware
platform
Prototype hardware
platform
(4) Plant
Statically (signal
generator)
Dynamically
(simulation model)

387
 Prototyping (2/3)
(1) Software unit and software
integration test
Same environment as MIL
Final software instead of model
a) Host environment
b) Target (emulator)
(2) Hardware/software
integration
test
Test hardware integration
Signal processors

Output monitoring with oscilloscopes

or logic analyzers
In-circuit test equipment (not
restricted to outputs)
Real-time simulation of the plant

388
 Prototyping (3/3)
(3) System integration test
Integrate all hardware parts (often prototypes)
Similar to hardware integration (but complete!)
(4) Environmental test
Measuring influence of the system on the environment (EMC)
Measuring roboustness of the system w.r.t. environment
conditions (temperature, humidity, EMI, shock or vibration, …)

389
 (3) Pre-Production
Applicable tests:
Test a pre-production unit
Use pre-production unit Acceptance test
Execute with the real plant Qualification test
Safety execution test
Goals: Test of the production and
Prove that all requirements are maintenance
met test facilities
Demonstrate conformity with Inspection and/or test by government
officials
relevant
standards/norms Typical test techniques:
Demonstrate that production Real-life testing
effort and schedule can be met Random testing
Demonstrate maintainability Fault injection
and mean- time-to-repair Remark:
requirements Special tools for gathering and
Demonstrate product to recording
(potential) the data as well as fault injection are
customers required
Often environment test facilities
(climate
chambers, vibration tables, …) are also
used 390
 (4) Production

Production (or Post-development stage):

Development and test of the production

facilities (often equally important for quality)

Often a first article inspection of similar

style as the pre-production test is required

Production and maintenance tests on

production units may be required for quality
control

391
Comparison of the Test Levels

392
 III.7 Discussion & Summary
Besides dynamic analysis (testing), which includes the execution of
the software, also effective manual static analysis techniques such as
reviews or inspections are available.
Automatic or semi-automatic static analysis techniques are static
code analysis, formal methods, or model checking.
Two main approaches to testing (dynamic analysis) are black box
testing, which ignores the internal mechanism of the system or
component (sometimes named functional testing), or white box
testing, which takes into account the internal mechanism of a system
or component (sometimes named structural testing).
The testing activities during development should include, unit tests,
integration tests, and system tests. System tests may be divided into
function tests, performance tests, acceptance tests, and installation
tests.

393
 III.7 Discussion & Summary
Test management should result in a testing process which includes (1)
determining the test methodology, (2) planning the tests, (3) design the
individual tests, and (4) implement the designed tests.
The criteria when to terminate the tests is of critical importance.
Mathematical models, error seeding, or dual independent testing teams
are available approaches to guide termination.
Test automation can help to reduce software testing time, cost, and
efforts while increase the quality of software testing by applying well-
defined test methods through tools. Systematically employed can they
relieve some complicated and redundant work from test engineers.
The paradigm employed for the system development is relevant for
testability: e.g., object-oriented systems have advantages as well as
drawbacks w.r.t. testing
The application domain is relevant for the required testing: e.g.,
embedded systems and their software must be subject to a whole series
of test configurations

394
Software Configuration
Management

395
Outline

• Software Configuration Management (SCM)

• What is SCM
• What is software configuration
• Sources of Change
• People in typical SCM Scenario
• Baselines
• Configuration objects
• SCM Repository
• SCM Process
• Change Control
• Version Control
• CSR

396
What is SCM?
• Change management is commonly called software
configuration management (SCM or CM)

• Babich defines Configuration Management as:

• the art of identifying, organizing and controlling modifications to
the software being built by a programming team.

• SCM activities have been developed to manage change

throughout the life cycle of computer software

• SCM can be viewed as a quality assurance activity

397
What is SCM? (Contd.)

• SCM is a set of activities designed to manage

change by:
• Identifying s/w work products that are likely to change,
• Establishing relationships among them,
• Defining mechanisms for managing different versions of
these work products
• Controlling changes
• And auditing and reporting on the changes made

398
Difference between Software
Support and SCM
• Support is a set of s/w engg. activities that occur
after software has been delivered to the customer
and put into operation.

• While SCM is a set of tracking and control

activities that initiate when the s/w engg project
begins & end only when the s/w is taken out of
operation

399
What is software configuration?
• The items that comprise all information produced
as part of the software process are collectively
called a software configuration
• This information (also termed as output of software
process) is broadly divided into:
• Computer programs (both source level & executable
forms)
• Work products that describe the computer programs
• Data (contained within program or external to it)

400
Sources of Changes

• New business or market conditions

• dictate changes in product requirements or business rules.
• New customer needs
• demand modification of data produced by information systems,
functionality delivered by products, or services delivered by a
computer-based system.
• Reorganization or business growth/ downsizing
• causes changes in project priorities or software engg team
structure.
• Budgetary or scheduling constraints
• cause a redefinition of the system or product.

401
People involved in a typical SCM
Scenario
• Project Manager
• In charge of s/w group
• Ensures that product is developed within time frame
• Monitors development, recognizes & reacts to problems
• by generating & analyzing reports about system status
• By performing reviews
• Configuration Manager
• In charge of CM procedures
• Ensures that procedures & policies for creating, changing &
testing of code are followed
• To control changes in code, introduces mechanisms for official
change requests, their evaluation & authoriza-tion.

402
People involved in a typical SCM
Scenario
• Software engineers
• Work efficiently without interfering in each other’s code
creation, testing & supporting documentation.
• But at the same time communicate & coordinate efficiently by
• Updating each other about tasks required & tasks completed
• Propagating changes across each others work by merging files
• Customers
• Use the product
• Follows formal procedures of requesting changes and for
indicting bugs in the product

403
Baselines

• IEEE defines baseline as:

• A specification or product that has been formally reviewed and
agreed upon, that thereafter serves as basis for further development,
and that can be changed only through formal change control
procedures.
• Before a s/w configuration item becomes a baseline,
changes may be made quickly & informally.
• However, once it becomes a baseline, formal procedures
must be applied to evaluate & verify every change.
• Refer to figure 27.1 to see steps for forming a baseline.

404
 Baselined SCIs and Project DB
Modified SCIs

PROJECT DB

S/w engg tasks FTRs Approved SCIs

SCIs

Stored SCIs

SCM Controls Extracted SCIs Baselines:

405
Configuration Objects

• SCIs are organized to form configuration objects.

• A config. object has a name, attributes and is related to
other objects.
• Example:
• Test Specification – Config. Object
• It comprises of SCIs like Test Plan, Test Procedure, Test Cases
• It is related to config. object SourceCode
• Config. Objects are related to other cong. objects, the
relationship is either compositional (part-of relationship)
or interrelated.
• In related config. objects, if one of them changes, the
other one has to be changed.

406
SCM Repository

• In early days of s/w engg, software configuration items

were maintained as paper documents (or punch cards),
placed in files and folders.
• The problems of this approach were:
• Finding a configuration item when it was needed was often difficult
• Determining which items were changed, when and by whom was
often challenging
• Constructing a new version of existing program was error-prone &
time consuming
• Describing complex relationships among configuration items was
virtually impossible
• Today SCIs are maintained in a project database or
repository.
• It acts as the center for accumulation and storage of SCIs
407
Role of repository

• SCM Repository is a set of mechanisms & data structures that

allow a software team to manage change in an effective
manner.
• It provides functions of a DBMS but in addition has the
following functions:
• Data integrity
• Info sharing
• Tool integration
• Data integration
• Methodology enforcement
• Document standardization

408
SCM Process

• SCM process defines a series of tasks having 4

primary objectives
• To identify all items that collectively define the s/w
configuration.
• To manage changes to one or more of these items.
• To facilitate construction of different versions of an
application.
• To ensure that s/w quality is maintained as configuration
evolves over time.

409
Version Control

• Combines procedures and tools to manage the different

versions of configuration objects created during the software
process.
• 1. Entity Definition: The base unit or artifact managed in the SCM system. It is a logical or physical item, such as
a file, document, or software module.
• Key Idea: The core item without considering its changes or variations.
• Example: A source code file named main.c

• 2. Version Definition: A specific instance or revision of an entity that results from changes made over time.
• Key Idea: Tracks the evolution of the entity through modifications.
• Example: main.c (Entity) →
• Version 1: Initial implementation of main.c.
• Version 2: Added a new function to main.c.
• Version 3: Fixed a bug in main.c.

• 3. VariantDefinition: A variation of an entity designed to meet different requirements, environments, or

platforms while coexisting with other variants.
• Key Idea: A parallel branch of an entity with differences tailored for specific use cases.
• Example:main.c (Entity) →
• Variant 1: A version of main.c for Windows with Windows-specific system calls.
• Variant 2: A version of main.c for Linux with Linux-specific system calls.
410
Version Control System

• A number of automated tools are available for version

control.
• CVS (Concurrent version System) is an example which is
based on RCS (Revision Control System)
• CVS features
• Establishes a simple repository.
• Maintains all versions of a file in a single named file by storing only
the differences between progressive versions of the original file.
• Protects against simultaneous changes to a file by establishing
different directions for each developer

411
Change Control
• A Change request is submitted and evaluated to
assess technical merit and impact on the other
configuration objects and budget
• Change report contains the results of the evaluation
• Change control authority (CCA) makes the final
decision on the status and priority of the change
based on the change report
• Engineering change order (ECO) is generated for each
change approved (describes change, lists the
constraints, and criteria for review and audit)

412
Change Control (Contd.)

• Object to be changed is checked-out of the project

database subject to access control parameters for the
object
• Modified object is subjected to appropriate SQA and
testing procedures
• Modified object is checked-in to the project database
and version control mechanisms are used to create
the next version of the software
• Synchronization control is used to ensure that parallel
changes made by different people don't overwrite
one another

413
Software Configuration Audit

• To ensure that a change has been properly implemented,

FTRs and software config. audits are used.
• A Software Config. Audit complements the FTR by addressing
the following questions:
• Has the change specified by the ECO been made without
modifications?
• Has an FTR been conducted to assess technical correctness?
• Was the software process followed and software engineering
standards applied?
• Do the attributes of the configuration object reflect the change?
• Have the SCM standards for recording and reporting the change been
followed?
• Were all related SCI's properly updated?

414
Configuration Status Reporting

• CSR addresses the following questions:

• What happened?
• Who did it?
• When did it happen?
• What else will be affected by the change?
• CSR entries are made when:
• A SCI is assigned new or updated identification
• Change is approved by CCA
• Config. Audit is conducted
• CSR is generated on regular basis.

415
 Chapter 25

Risk Management

- Introduction
- Risk identification
- Risk projection (estimation)
- Risk mitigation, monitoring, and
management

416
(Source: Pressman, R. Software Engineering: A Practitioner’s Approach. McGraw-Hill, 2005)
Project Risks

What can go wrong?

What is the likelihood?
What will the damage be?
What can we do about it?

417
Introduction

418
What Is Software Risk?

• Risk is an expectation of loss, a potential problem

that may or may not occur in the future.
• It is generally caused due to lack of information,
control or time.
• A possibility of suffering from loss in software
development process is called a software risk.
• Loss can be anything, increase in production cost,
development of poor quality software, not being
able to complete the project on time.

419
Types of software risks

• Software risk exists because the future is uncertain and there are
many known and unknown things that cannot be incorporated in the
project plan.
• A software risk can be of two types
• (1) internal risks that are within the control of the project manager and
• (2) external risks that are beyond the control of project manager.

420
Definition of Risk
• A risk is a potential problem – it might happen and it might not
• Conceptual definition of risk
• Risk concerns future happenings
• Risk involves change in mind, opinion, actions, places, etc.
• Risk involves choice and the uncertainty that choice entails
• Two characteristics of risk
• Uncertainty – the risk may or may not happen, that is, there are no 100%
risks (those, instead, are called constraints)
• Loss – the risk becomes a reality and unwanted consequences or losses
occur

421
Risk Categorization – Approach #1

• Project risks
• They threaten the project plan
• If they become real, it is likely that the project schedule will slip and that
costs will increase
• Technical risks
• They threaten the quality and timeliness of the software to be produced
• If they become real, implementation may become difficult or impossible
• Business risks
• They threaten the viability of the software to be built
• If they become real, they jeopardize the project or the product

(More on next slide)

422
Risk Categorization – Approach #1
(continued)

• Sub-categories of Business risks

• Market risk – building asystem that no one really wants
• Strategic risk – building a product that no longer fits into the overall
business strategy for the company
• Sales risk – building a product that the sales force doesn't understand how
to sell
• Management risk – losing the support of senior management due to a
change in focus or a change in people
• Budget risk – losing budgetary or personnel commitment

423
Risk Categorization – Approach #2

• Known risks
• Those risks that can be uncovered after careful evaluation of the project plan,
the business and technical environment in which the project is being
developed, and other reliable information sources (e.g., unrealistic delivery
date)
• Predictable risks
• Those risks that are extrapolated from past project experience (e.g., past
turnover)
• Unpredictable risks
• Those risks that can and do occur, but are extremely difficult to identify in
advance

424
 Reactive vs. Proactive Risk
Strategies

• Reactive risk strategies

• "Don't worry, I'll think of something"
• The majority of software teams and managers rely on this approach
• Nothing is done about risks until something goes wrong
• The team then flies into action in an attempt to correct the problem
rapidly (fire fighting)
• Crisis management is the choice of management techniques
• Proactive risk strategies
• Steps for risk management are followed (see next slide)
• Primary objective is to avoid risk and to have a contingency plan in
place to handle unavoidable risks in a controlled and effective
manner
425
Risk Management

• Risk management is carried out to:

• Identify the risk
• Reduce the impact of risk
• Reduce the probability or likelihood of risk
• Risk monitoring

426
Steps for Risk Management

1) Identify possible risks; recognize what can go wrong

2) Analyze each risk to estimate the probability that it will occur and
the impact (i.e., damage) that it will do if it does occur
3) Rank the risks by probability and impact
- Impact may be negligible, marginal, critical, and catastrophic
4) Develop a contingency plan to manage those risks having high
probability and high impact

427
Steps for Risk Management

Identify possible risks; recognize what can go wrong

Analyze each risk to estimate the probability that it

will occur and the impact (i.e., damage) that it will
do if it does occur

Rank the risks by probability and impact

- Impact may be negligible, marginal, critical, and
catastrophic

Develop a contingency plan to manage those risks

having high probability and high impact

428
Risk Identification

429
Background
• Risk identification is a systematic attempt to specify threats to the
project plan
• By identifying known and predictable risks, the project manager takes a
first step toward avoiding them when possible and controlling them
when necessary
• Generic risks
• Risks that are a potential threat to every software project
• Product-specific risks
• Risks that can be identified only by those a with a clear understanding of
the technology, the people, and the environment that is specific to the
software that is to be built
• This requires examination of the project plan and the statement of scope
• "What special characteristics of this product may threaten our project
plan?"

430
Risk Item Checklist

Used as one way to identify risks

Focuses on known and predictable risks in

specific subcategories (see next slide)

Can be organized in several ways

•A list of characteristics relevant to each risk subcategory
•Questionnaire that leads to an estimate on the impact of each
risk
•A list containing a set of risk component and drivers and their
probability of occurrence
431
Known and Predictable Risk
Categories

432
Recording Risk Information

Project: Embedded software for XYZ system

Risk type: schedule risk
Priority (1 low ... 5 critical): 4
Risk factor: Project completion will depend on tests which require hardware
component under development. Hardware component delivery may be delayed
Probability: 60 %
Impact: Project completion will be delayed for each day that
hardware is unavailable for use in software testing
Monitoring approach:
Scheduled milestone reviews with hardware group
Contingency plan:
Modification of testing strategy to accommodate delay using
software simulation
Estimated resources: 6 additional person months beginning 7-1-96

433
Questionnaire on Project Risk
(Questions are ordered by their relative importance to project success)

1) Have top software and customer managers formally committed to

support the project?
2) Are end-users enthusiastically committed to the project and the
system/product to be built?
3) Are requirements fully understood by the software engineering
team and its customers?
4) Have customers been involved fully in the definition of
requirements?
5) Do end-users have realistic expectations?
6) Is the project scope stable?

(More on next slide) 434

Questionnaire on Project Risk
(continued)

7) Does the software engineering team have the right mix of skills?
8) Are project requirements stable?
9) Does the project team have experience with the technology to be
implemented?
10) Is the number of people on the project team adequate to do the
job?
11) Do all customer/user constituencies agree on the importance of
the project and on the requirements for the system/product to be
built?

435
 Risk Components and Drivers
• Risk Component: refer to the key elements that make up a risk, such as the risk event, its
likelihood, impact, and potential consequences.
• Risk Driver: are the underlying factors or causes that influence the occurrence or
magnitude of a risk, such as resource availability, market conditions, or technological
changes.
• The project manager identifies the risk drivers that affect the following risk components
• Performance risk - the degree of uncertainty that the product will meet its requirements and be fit
for its intended use
• Cost risk - the degree of uncertainty that the project budget will be maintained
• Support risk - the degree of uncertainty that the resultant software will be easy to correct, adapt,
and enhance
• Schedule risk - the degree of uncertainty that the project schedule will be maintained and that the
product will be delivered on time
• The impact of each risk driver on the risk component is divided into one of four impact
levels
• Negligible, marginal, critical, and catastrophic
• Risk drivers can be assessed as impossible, improbable, probable, and frequent

436
Risk Projection
(Estimation)

437
Background

• Risk projection (or estimation) attempts to rate each risk in two ways
• The probability that the risk is real
• The consequence of the problems associated with the risk, should it occur
• The project planner, managers, and technical staff perform four risk
projection steps (see next slide)
• The intent of these steps is to consider risks in a manner that leads to
prioritization
• By prioritizing risks, the software team can allocate limited resources
where they will have the most impact

438
Risk Projection/Estimation Steps

1) Establish a scale that reflects the perceived

likelihood of a risk (e.g., 1-low, 10-high)
2) Delineate the consequences of the risk
3) Estimate the impact of the risk on the project
and product
4) Note the overall accuracy of the risk projection
so that there will be no misunderstandings

439
 Contents of a Risk Table
• A risk table provides a project manager with a simple technique for risk
projection
• It consists of five columns
• Risk Summary – short description of the risk
• Risk Category – one of seven risk categories (slide 17)
• Probability – estimation of risk occurrence based on group input
• Impact – (1) catastrophic (2) critical (3) marginal (4) negligible
• RMMM – Pointer to a paragraph in the Risk Mitigation, Monitoring, and
Management Plan

Risk Summary Risk Category Probability Impact (1-4) RMMM

(More on next slide) 440

Developing a Risk Table

• List all risks in the first column (by way of the help of the risk item
checklists)
• Mark the category of each risk
• Estimate the probability of each risk occurring
• Assess the impact of each risk based on an averaging of the four risk
components to determine an overall impact value (See next slide)
• Sort the rows by probability and impact in descending order
• Draw a horizontal cutoff line in the table that indicates the risks that
will be given further attention

441
 Assessing Risk Impact

• Three factors affect the consequences that are likely if a risk does occur
• Its nature – This indicates the problems that are likely if the risk occurs
• Its scope – This combines the severity of the risk (how serious was it) with its
overall distribution (how much was affected)
• Its timing – This considers when and for how long the impact will be felt
• The overall risk exposure formula is RE = P x C
• P = the probability of occurrence for a risk
• C = the cost to the project should the risk actually occur
• Example
• P = 80% probability that 18 of 60 software components will have to be
developed
• C = Total cost of developing 18 components is $25,000
• RE = .80 x $25,000 = $20,000

442
Risk Mitigation, Monitoring,
and Management

443
Background
• An effective strategy for dealing with risk must consider three issues
(Note: these are not mutually exclusive)
• Risk mitigation (i.e., avoidance)
• Risk mitigation: refers to the process of identifying, assessing, and implementing strategies to
reduce the likelihood or impact of potential risks on a project or organization. This involves taking
proactive steps to avoid, minimize, or control risks, ensuring the project's objectives are achieved
with minimal disruption.
• Implementing regular code reviews and automated testing to catch bugs early in a software
development project. This reduces the risk of deploying faulty code into production.
• Risk monitoring
• Risk management and contingency planning
• risk contingency planning prepares for a response if the risk happens despite efforts to mitigate it.
• Having backup personnel or contractors available to step in if key team members become
unavailable due to illness, resignation, or other reasons. This ensures that the project can continue
smoothly even if someone leaves the team.
• Risk mitigation (avoidance) is the primary strategy and is achieved through
a plan
• Example: Risk of high staff turnover (see next slide)

(More on next slide)

444
Background (continued)

Strategy for Reducing Staff Turnover

Meet with current staff to determine causes for turnover (e.g., poor
working conditions, low pay, competitive job market)
Mitigate those causes that are under our control before the project starts
Once the project commences, assume turnover will occur and develop
techniques to ensure continuity when people leave
Organize project teams so that information about each development
activity is widely dispersed
Define documentation standards and establish mechanisms to ensure that
documents are developed in a timely manner
Conduct peer reviews of all work (so that more than one person is "up to
speed")
Assign a backup staff member for every critical technologist

445
Background (continued)

• During risk monitoring, the project manager monitors factors that may
provide an indication of whether a risk is becoming more or less likely
• Risk management and contingency planning assume that mitigation
efforts have failed and that the risk has become a reality
• RMMM steps incur additional project cost
• Large projects may have identified 30 – 40 risks
• Risk is not limited to the software project itself
• Risks can occur after the software has been delivered to the user

(More on next slide)

446
Background (continued)

• Software safety and hazard analysis

• These are software quality assurance activities that focus on the
identification and assessment of potential hazards that may affect software
negatively and cause an entire system to fail
• If hazards can be identified early in the software process, software design
features can be specified that will either eliminate or control potential
hazards

447
The RMMM Plan

• The RMMM plan may be a part of the software development plan or

may be a separate document
• Once RMMM has been documented and the project has begun, the risk
mitigation, and monitoring steps begin
• Risk mitigation is a problem avoidance activity
• Risk monitoring is a project tracking activity
• Risk monitoring has three objectives
• To assess whether predicted risks do, in fact, occur
• To ensure that risk aversion steps defined for the risk are being properly
applied
• To collect information that can be used for future risk analysis
• The findings from risk monitoring may allow the project manager to
ascertain what risks caused which problems throughout the project

448
Seven Principles of Risk Management
Maintain a global perspective
View software risks within the context of a system and the business problem that is intended to
solve

Take a forward-looking view

Think about risks that may arise in the future; establish contingency plans

Encourage open communication

Encourage all stakeholders and users to point out risks at any time

Integrate risk management

Integrate the consideration of risk into the software process

Emphasize a continuous process

of risk management
Modify identified risks as more becomes known and add new risks as better insight is achieved

Develop a shared product vision

A shared vision by all stakeholders facilitates better risk identification and assessment

Encourage teamwork when

managing risk
Pool the skills and experience of all stakeholders when conducting risk management activities
449
 Summary

• Whenever much is riding on a software project, common sense dictates risk

analysis
• Yet, most project managers do it informally and superficially, if at all
• However, the time spent in risk management results in
• Less upheaval during the project
• A greater ability to track and control a project
• The confidence that comes with planning for problems before they occur
• Risk management can absorb a significant amount of the project planning
effort…but the effort is worth it

450 
451