Software Testing

and
Quality Assurance
Lecture 6 – Software Testing Types and Techniques 3
 2

Dynamic White-Box Testing Techniques

2. Data Flow Testing
 3
Introduction

 A program unit, such as a function, accepts input values, performs computations while
assigning new values to local and global variables, and, finally, produces output values.

 Meaning, there is “flow” of data values between variables along a path of program
execution.

 This calls to perform a number of tests that focuses on data values.

 Data flow testing is a white-box testing technique that is used test flow of data in a given
program unit.
 4
Cont.

 The testing process is conducted with the objective to detect bugs introduced because
of the incorrect/improper usage of data variables or data values.

 Data flow testing identifies potential bugs/errors by examining the patterns in which that
piece of data is used.

 (Has every data object been initialized prior to use? Have all defined data objects
been used for something?)

 Overall, Data flow testing gives a chance to look out for inappropriate data definition, its
use in predicates, computations, and termination.
 5
Working Procedure

 Data flow testing method find the test paths of a program according to the locations of
definitions and uses of data variables in the program.

 The normal life cycle for a data object (a variable) consists of the phases:

 Declaration and Definition - Space is reserved in memory for the variable value and a
value is assigned to the variable.

 Use - The value of the variable is used.

 Destruction - The memory set aside for the value of the variable is freed for others to
use.
 As a result, a data object can be in the following states: 6
 Defined (d): when it is initialized.

 Referenced/ Usage (u): when the data object is used in different operations.

 In general, there are two forms of reference/use:

 Predicate use (p-use)

 A use of a variable is a predicate use (p-use) if the variable is in a predicate and its
value is used to decide an execution path.

 Computation use (c-use)

 A use of a variable is a computation use (c-use) if the value of the variable is used
to compute a value for defining another variable or as an output value.

 Killed/Undefined/Released (k): when the data object is destroyed.

 7

Predicate use (p-use) and Computation use (c-use)

 The normal sequence of actions on variables are illustrated below.
8
For example


9
Initially, a variable can remain in an “undefined” (U) state, meaning that just a memory
location has been allocated to the variable but no value has yet been assigned.

 A computation can be performed to define (d) the variable in the form of assigning a
value to the variable—the variable moves to a “Defined but not referenced” (D) state.

 Later, referencing (r) the value of the variable moves the variable to a “Defined and
Referenced” state (R).

 The variable remains in the R state as long as the program keeps referencing the value of
the variable.

 If the programmer assigns a new value to the variable, the variable moves back to the D
state.

 On the other hand, the programmer can take an action to kill or undefine (u) the
variable.
 10
 However, mistakes can be made by taking the wrong actions while a variable is in a
certain state.

 For example,

 If a variable is in the state U - undefined and a programmer reads (r) the variable,
then the variable moves to an abnormal (A) state.

 In such cases, a data flow anomaly is said to have occurred.

 11
Data Flow Anomalies

 Data-flow anomalies represent the patterns of data usage (a deviant or abnormal way)
which may lead to an incorrect execution of the code.

 Some of these anomalies are

 Defined and then Defined Again (Redefinition before use): to successively assign two
values to a variable without using the first value.

 Undefined but Referenced (Use before declaration and definition): to use a value of a
variable before assigning a value to the variable.

 Defined but Not Referenced: to generate a data value and never use it.

 Use after Destruction

 12
Cont.

 Such anomalies are denoted by a two-character sequence of actions.

 For example,

 ‘dk’ means a variable is defined and killed without any use, which is a potential bug.

 These anomalies could be manifestations of potential programming errors.

 Based on this insight, data flow testing suggests that

 Pick enough paths and test cases to assure that:

 Every data object has been initialized prior to its use.

 All defined objects have been used at least once.

 13

Dynamic White-Box Testing Techniques

3. Mutation Testing
 14
Mutation Testing

 Also known as fault-based testing, program mutation, error-based testing, or mutation

analysis.

 Mutation testing is a fault-based testing technique where the product under test, i.e. the
software program, is changed in controlled ways and then tested.

 Miniscule changes (modifications or faults) are introduced purposely into a program

resulting in many versions of the program. (Hence the name Mutation)

 The modified variation or version of the original program is called Mutant.

 This is done to assess and determine the effectiveness of the prepared test cases in
isolating the deviations.
 15
Cont.

 So, when the test data is run through the mutant(s), it should ideally give us different results
from the original program.

 If the original program and mutant program(s) yield different output(s), a mutant is said to
be killed.

 This means, the faults represented by that mutant have been detected by the test
cases and the test case has caused the mutant to fail.

 Otherwise, we say that the mutant lived if the test passes.

 16
Cont.

 The ideal case is that none of the test cases should pass.

 Our goal is to kill all mutants.

 The more mutants we can kill, the higher the quality of our tests.

 Thus, the main objective is to select efficient test data which have error-detection power.

 Mutants that always produce the same output as the original program are called
Equivalent Mutants.
 17
Mutation Score

 A mutation score is the metric that calculates the percentage of mutants killed over the
total of mutants that successfully were seeded in the source code.

Note - Equivalent mutants are not considered when calculating the mutation score.

 The test suite is said to be mutation adequate if its mutation score is 100%.
 18
How to Create Mutants

1) Value Mutation

 Here, we introduce a mutation by changing the parameter and/or constant values in a

program.

 Example
Initial Code Mutant Code
int mod = 1000000007; int mod = 1007;
int a = 12345678; int a = 12345678;
int b = 98765432; int b = 98765432;
int c = (a + b) % mod; int c = (a + b) % mod;
 19
How to Create Mutants

2) Statement Mutation

 A statement is deleted, duplicated or it is replaced by some other statement.

 Example

Initial Code Mutant Code

if(a < b) if(a < b)
c = 10; d = 10;
else else
c = 20; d = 20;
 20
How to Create Mutants

3) Decision Mutation

 The target here is part of the code that makes decisions, for example, value comparisons.

 So, logical or arithmetic operators in a program are changed to create mutants.

 Example
Initial Code Mutant Code
if(a < b) if(a > b)
c = 10; c = 10;
else else
c = 20; c = 20;
 21

II. Static Testing

 22
Introduction

 In contrast to dynamic testing, Static testing is a testing type where the product under test
(the software product and its documents) is NOT executed.

 It is the systematic examination of a program structure.

 The technique assesses the structural characteristics of items and thus do not require
the bulk of test cases as they do not execute the software.

 The primary objective of any static testing is to find defects in the object under testing and
hence provide information about the quality of what is being produced.
 23
Cont.

 Static testing can be performed on anything that can be read throughout the product life
cycle.

 Deliverables and documents being produced during the development and

maintenance of a product, in every phase from the earliest conception of the
product to its disposal, including documents produced by the organization can, and
should be, objects for static testing.
 A few examples of documents that can be subjected to static testing are:


24
Organizational documents, such as policies, strategies, plans, reports, sales and
marketing material

 Plans for products, projects, quality assurance, test, configuration management,

customer interface, supplier agreements

 Requirement specifications from the business and from the users, and for software,
hardware, data, network, services, and others.

 Design for product, architecture, detailed components

 The actual product, such as code, data, hardware, installation guides, manuals,
educational material

 Test specifications and test reports for component testing, integration testing, system
testing, acceptance test

 Process descriptions, templates, examples, technique descriptions

 25
Static Testing Cost/Benefit

 There are costs associated with static testing, mainly because it takes time to perform.

 A rule of thumb says that 15% of the development budget should be reserved for all the
static testing activities.

 On the other hand, the benefits of static testing lies in the early defect detection.

 We can save a lot of money by finding the defects close to their introduction.

 Studies show that the cost of correction of a defect grows with a factor 10 for each
development phase it “survives.”
 Early defect detection also gives other benefits to the organization:

26
 Management gets an earlier insight into the quality of the product, if we report
information about our early findings.

 Better productivity in development and an overall increase in efficiency.

 Dynamic testing time is decreased because we have fewer defects left to deal with.

 Performance of static testing also means that fewer defects are sent out to the
customer, and this gives a higher trustworthiness and less maintenance.

 On the softer side static testing can spring off new ideas on how to do things better,
because knowledge sharing is an important part of static testing.

 A better esprit de corps can be promoted, since more stakeholders are involved in
getting the best quality of the product from the beginning—we did this together!
 27
Outcome of Static Testing

 A static test on a particular document will have one of three possible outcomes:

(1) Accepted: If everything is fine and no defects are found, the document can be
accepted as it is. No further work is necessary.

(2) Accepted with rework: A number of defects are found.

 If the defects are not too many and not too serious compared with the quality
criteria it can then be decided that some modification is needed before the
document can be used further, but a re-static testing is not required.

 The document is accepted on the understanding that the necessary corrections

will be done.
 28
Cont.

(3) Rejected: In the case where the defects found are too numerous and/or too serious in
view of the quality criteria, the document can be rejected.

 This means that it is returned to development for re-work before it may be presented
for static testing again.
 29
Static Testing Types and Techniques

 As per the IEEE 1028 definition there are a number of different static testing types or
techniques.
 30
A. Review

 A review can be defined as:

 A process or meeting during which software element (a software product, set of

software products, or a software process) is presented to project personnel,
managers, users, customers, user representatives, auditors or other interested parties
for examination, evaluation, comment or approval.

 Reviews can be of different types.

 Informal Review

 Technical Review

 Management Review
 Informal Review
31
 The informal review is, as the name indicates, the least formal type of static test.

 This is what we all do all the time, mostly without thinking about it as a review: asking a
colleague or a friend to look at something we have produced.

 An informal review follows no formal documented process.

 The participants are normally just the author and one or two reviewers. The reviewer(s) are
usually chosen by the author.

 The author decides when he or she would like an informal review to take place.

 The reviewer reviews the document when it fits into his or her schedule after agreement
with the author.

 The author typically gets feedback in either pure verbal form or in the form of notes
scribbled on the document under testing.
 Technical Review 32

 A technical review, sometimes also known as a peer review, is a peer group review
activity that focuses on achieving consensus on the technical approach to be taken.

 Also here the objective is to find defects, to identify discrepancies from specifications and
standards and to make technical decisions and (one hopes) reach consensus about the
approach to the work.

 Technical reviews are usually planned to take place at certain times in the development
life cycle.

 Any technical, that is, non-management, document can be the object of a technical
review.
 Cont. 33

 It is important that the participants in a technical review are more or less at the same level
in the organization.

 Often a manager should not participate in a technical review.

 The reviewers are usually expected to examine the material for defects and issues before
the review meeting is held.

 A report should be written after the review meeting summarizing the findings and the
conclusions.
 Management Review 34

 Management review is a review type performed on management documents.

 The purpose of a management review is

 to monitor progress, determine the status of plans and schedules, or evaluate the
effectiveness of management approaches used to achieve fitness for purpose.

 Management reviews

 support decisions about corrective actions, changes in the allocation of resources, or

changes to the scope of the project.

 identify consistency with and deviations from plans, or adequacies and inadequacies
of management procedures.
 Cont. 35

 Management reviews are usually planned to take place at certain times in the
development life cycle, typically in connection with defined milestones, that is, transfer
from one development phase to the next.

 The reviewers are relevant stakeholders and should include management and technical
staff involved in the execution of the planned activities.

 A report should be written after the review meeting summarizing the action items defined
and the issues to be resolved, if any.
 36
B. Walk-Through

 A walk-through is a step-by-step presentation of a document by the author of the

‘document under review’ at a walk-through meeting.

 In the course of this, the defects, omissions, possible changes, improvement ideas, style
issues, and alternatives that pop up are noted and discussed.

 This is especially useful if people from outside the software discipline are present, who are
not used to, or cannot easily understand software development documents.

 Overall, the objective is to create a common understanding of the contents of the

document under testing, to gather feedback and to find defects.

 The process for walk-throughs is usually less rigorous and not very formal.
 37
C. Inspection

 Inspection is the formal and well-defined type of static test.

 It is examination of a software product to detect and identify software anomalies,

including errors and deviations from standards and specifications.

 Inspectors systematically review the artifact individually and in teams.

 The classic and most widely used inspection technique is based on following a checklist
while reading the artifact.

 A checklist contains a set of questions that help identify defects in the inspected artifact
and criteria to verify that the artifact complies with company standards.
 38
Cont.

 Length and complexity of checklists must reflect their expected use.

 We may have fairly long checklists with simple questions for simple reviews, and short
checklists with complex questions for indepth reviews.
 39

 Inspection process
 40
D. Audit

 Audit is an independent examination of a software product, software process, or set of

software processes performed by a third party to assess compliance with specifications,
standards, contractual agreements, or other criteria.

 It is by far the most formal static testing technique.

 The objective of an audit is to provide an independent evaluation of an activity’s

compliance to applicable process descriptions, contracts, regulations, and/or standards.

 Any document, technical as well as managerial, can be part of an audit.

 Audits are usually performed late in the development life cycle, just before release of the
product and/or closedown of the project.
 41
Cont.

 Auditors must have a special authority and certification to perform official audits.

 Sometimes audits may be either internal or external.

 Internal audits are performed by auditors from the organization, but external to the
project under audit.

 External audits are performed by auditors entirely external to the organization, usually
from some sort of authority organization.

 The result of an audit is a report in which the findings are summarized in the form of lists of
observations, deviations, recommendations, and corrective actions to be taken, as well
as a pass/fail statement.
 42

End