Defects and Failures

Not all software defects are errors. One common source of expensiye
caused by coding
derects is caused by requirement gaps, e.g, unrecognised requirements, that result in errors
ot omission by the programme designer. A common source of requirenments gaps is non-
Functional requirements such as testability, scalability, maintainability, usability,performance
and security.

Software faults occur through the following processes. A

programmer makes an error
(mistake), which results in a detect (fault, bug) in the software source code. If this defect is
executed, in certain situations the system will produce wrong results, causing
a failure. Not
Softaware Testing

. sSarily result in failures. For

ofects
all
will
efects w
nec

example, defects
273
defect can turn into
failure when the
a in dead code will never
failures. A
hesechanges in nuironment include the environment is changed. result
data or interactin with
software being run on new hardware Examples of
inns
source a
different software. A
wide range
in a
of failure
symptoms. single defect platform, may result
Compatibility

roent cause of software failure 1s

in
rating System, creasingly,
system, or, incre compatibility with another application, a new
web browser
operaliy, this can occur (for example...) becauseversion.
the
In the case of lack
of backward
ing their programmes for, or testing the software programmers have only considered
dAot aperating system. The unintended upon, "the latest version of" this-or-
consequence
fully compatible with earlier mixtures of
of this fact is that: their
latest work might
ot
software/hardware, it
ompatible with another important operating system. In any case, these might not be fully
Comp
or

thev might be, may have resulted in differences, whatever

significant population of conmputer users.
(unintended...) software failures, as witnessed by some
Input Combinations and Preconditions
furndamental problem with software testing is that
A very
of inputs and preconditions testing under all combinations
(initial state) is not feasible, even with a simple product. This
means that the number of defects in software
product can be very large and defects that
a
occur
infrequently difficult to find in testing. More significantly, non-functional dimensions
are
of quality (how it is
supposed to be versus what it is supposed to do-usability, scalability,
performance, compatibility, reliability-can be highly subjective; something that constitutes
Sutficient value to one person may be intolerable to another.

Static vs
Dynamic Testing
The are many approaches to software testing. Reviews, walkthroughs, or inspections
considered as static testing, whereas actually executing programmed code with a given
t Of fest cases is referred to as dynamic testing. Static testing can be (and unfortunately in
itself is used
f Oten is) omitted. Dynamic testing takes place when the programme
the first time (which is generally considered the beginning of the testing stage). Dynamic
testing
Sectdy begin before the programme is 100 complete in order
per cent particular
to test
this are either using
Strihe code (modules or discrete functions). Typical techniques for
stubs/drivers or
Program or execution from a debugger environment.
For example, Spreadsheet
extent interactively (on the fly), with
programmes
results dare,
esults are,
display
Payed
by their very nature, tested to a large
immediately after each calculation or text manipulation.

Software Testing Team

the term "softwaretester
Software test be done by software testers. Until the 1980s
used esting
can
Was used
as profession. Regarding the periods
the generally,, but
also separate
seen as a
and the differ
diff lat it
but later was
different roles have been
established: manager,
testlead, to in software testing,
t goals and test administrator.
designer, tester, automation developer,
 Software Engjiner
274
Software
Quality
Assurance
(5QA)

software testing may be

viewed as an in
ortant part of
Engine rnq
controversial, rocess specialists and
software process

audithetorss twkakeatea
Though In SQA,
nt. They exami
process.
(SQA) examine ar
quality
assurance

software and its development.

end n in
Chang
ange t
ts
of faults that end
broader view

engineering process
on
reduce the
itself to

defect rate.
amount
up the
sot wate
delivered sotwe
deli ed sotwate
the so-called
detect rate" depends on the nat. of
the
an airplanothesoft
softwa
"acceptable
What constitutes
an

video game designed to

e flying
simulate fiying an
airplane
softw would
ould Dr bFor
arcade fects than mission critical software such as
example, an

have a much higher

tolerance for
defects than
is flying!
amably
that used
airliner that realy to
functions of an
the
control SQA, testing departments often. n existindependetiy
there are close links with
Although function in some companies.
be no SQA
and there may
detects in software by contrasin.
task intended to detect
Software Testing is a
with its actual results
results
for a given set of inno Compu
By contrast, Computer
programme's expected policies and procedures intendeOntas
is the implementation of
QA (Quality Assurance)
defects from occurring
in the first place.

Testing Methods

Approach of Boxes
Software testing methods are traditionally divided into black-box testing and whiteho
to describe the point of view that a test engineer takes
testing. These two approaches are used
when designing test cases.

Black-box Testing
Black-box testing treats the software as a "black box"-without any knowledge of intermal
implementation. Black-box testing methods include: equivalence partitioning, boundary value
analysis, all-pairs testing, fuzz testing, model-based testing, traceability matrix, exploratory
testing and specification-based testing
1. Specification-based Testing: Specification-based testing aims tò test the functionality of

software according to the applicable requirements. Thus, the tester inputs data nt
and only sees the output from, the test object. This level of testing usuauy
thorough test cases to be provided to the tester, who then can simply veniy u e
a given input, the output value (or behaviour), either "is" or "is not' thesanie
expected value specified in the test case. Specification-based testing is necessa
it is insuficient to
guard against
certain risks.
2. Advantages and Disadvantages: The black box tester has no "bonds" with the code, and
ciple, "Ask
a
tester's perception is very
simple: a code must have bugs. Using do not. But
Bu
the
and you shall receive," black box testers find
bugs where
hand, black-box testing has been said to be "likeprograi
labyrinth
on the other dark
a walk in a tesi
beingtested
without a e heing
flashlight," because the tester doesn't know how the s
fienareTèstinp

was
chually
tually constructed.ru1cted. As a
result,
check there are
manytest
to
something
ing
of the bacet
parts of
that Could situations when
could have
have been (1) a
275
tester writes
or (2) some back-end are not tested by
an black-box
testing has the tested at all. only one test
tes case,
T h e r e f o r e ,

and th
advantage of "an unaffiliated
heretodisadvantage of "blind exploring"
d the other. opinion," on on the one
tite-bor Testing

ite-box testing
White-bo
is when the tester has access to the
code that implement these. internal data
the structures and
ndluding
algorithms
White-box Testing
Types of
The
following types of white-box testing exist:
D testing (application programming interface) Testing of
the application using
-

Public and Private APls.

Code coverage creating tests to satisty some criteria

designer can of code
create tests to cause all statements in coverage (e.g., the test
the programme to be executed a
least once).
Fault injection methods.
Mutation testing methods.
Static testing White-box testing includes all static testing.
Code Completeness Evaluation
White-box testing methods can also be used to evaluate the completeness of a test suite
tat was created with black-box testing methods. This allows the software team to examine
parts of a system that are rarely tested and ensures that the most important function points
have been tested.
Iwo common forms of code
coverage are
runction coverage, which reports on functions executed.

which reports on the number of lines executed

to complete the
e
test.
n t coverage,
Ihey both return a code
< coverage metric, measured as a percentage.

stey Box Testing

and algorithms for
Grey
Ptposee box testing involves having access to internal
data structures

level. Manipulating
plurposes ofi o but testing
test cases, because
at the user,
input
or black-box
the and output
ata and f o 5 e qualify as grey box,
output do not under test. This distinction
Clearly outside o
s paricu rting
the "black-box" that we are
calling the system
between two
between
modules of code
code
Cularlyi testing for test. However,
portant when conducting integration interfaces are exposed be able
itw odiff
two different developers, where only the
user would
not normally
yýing data
a
evelopers,
as grey
box, as the
repository does qualify
 Sofasotware Engineh
Grey box testing
Engmearn
under
u nder test.
test.
h e system
system
276 tside of
of the
t
be
boundary values or error so indiud
messages.
outside

instance,
for instance,
data
change
the
determine,
1e. for
to to
engineering

Integration Testing software testing, that

testing, seeks to
of software
Such integration defects canuncov
type of
is any
u type integratio
testing othe. Such ario. collisi
integrated into therise, when thhe new
to each
and then inte
Integration modules
software branches,
individual

modules
are
developed
in separate
main prie.
Regression 1Testing
change in functionality
or
either for a defece
efects, aregress
After modifying
software,
on the software toon
modified softwaremodified
to ensure that the modsion
passing tests
test
re-runs

have not
previously
unintentionally
previous functional
caused a regression
st levels.
the above test
ot ressi
regressionttests are often ang
levels. These regression
ifcatons
at or all ot ften automated
be performed any are known as sanity teci.

forms ofregressiorn festing g (which

qui
More specific
behaviour) and smoke testing
(which tests for basic ft.
checks for bizarre to ensure that tha.
Benchmarks may be employed during regresSion testing the performan
software will be at least as acceptable as the earlier
of the newly modified in he
that some real improvement has been achieved
case of code optimisation,
that to uncoversoftware
testing is any type ofsottware testing seeks
e

Regression regressionson
whenever software functionality, that was previously working
Such regression occur
occur as an unintended conTedy,
corecty,
as intended. 1ypically, regressions
stops working of the software collid of
programme changes, when the newly developed part
previously existing. Common methods of regression testing include rerunning previondrrun
tests and checking whether previously fixed faults have re-emerged. The depth of testi
depends on the phase in the release process and the risk of the added features. They can eith
be complete, for changes added late in the release or deemed to be risky, to very shallow,
consisting of positive tests on each feature, if the changes are early in the release or deemed
to be of low risk.

Aceptance Testing
Acceptance testing can mean one of two things:
1. A smoke test is used as an acceptance test prior to introducing a new build to me
main testing process,
i.e. before integration or
regression.
ACceptance testing performed by the customer, often in their lab environment ontheir
own HW, is known as user
acceptance testing (UAT).
NonFunctional Softrware
Testing
Special methods exist to test
non-functional aspects of software
Performance testing checks to see if quantities
fdata

the
or users. This is generally referred to assoftware can handle large of n t i v i t y o fn o n

Functional Software software scalability. This5

Testing is often referred to as Endurance
Spftunare1 e s t i n g
checks to see if the software can
Stability testing checks 277
Cability ceptable period.
This activity of continuously
function well in or
above
an
(or endurance)
load
non-Functional Software Testing is often
referred to as
testing.
time
e fing needed to check if the
Isability testing is
is user interface is easy to use and
is essential for understand.
ecurity testing software that processes confidential
hackers. data to prevent
system
trusion by
and calisation is needed to
I n t e r n a t i o n a l i s a
test these
which a pseudolocalisation method can be used. aspects of software, for
contrasi to functional testing, which establishes the correct operation of the software
orect
nIn in that it matches the expected behaviour defined in the design requirements), non
orTE
uncHonal
esting verifies that the sottware functions properly even when it receives invalid
unexpe inputs. Softwar fault injection, in the form of fuzzing, is an example of non-
ional
nctional testing. n-functional
Non-i testing, especially for software, is designed to establish
vhether the device under test can tolerate invalid or unexpected inputs, thereby establishing
he ro
stness of input validation routines as well as error-handling routines. Various
ammercial non-functional testing tools are linked from the Software fault injection page;
there are also numerous open-source and free software tools available that perform non-
iunctional testing.
Destructive Testing
Destructive testing attempts to cause the software or a subsystem to fail, in order to test
is robustness.
Testing Process
A common practice of software testing is performed by an independent group of testers
ter the functionality is developed before it is shipped to the customer. This practice often
suls in the testing phase being used as project buffer to compensate for project delays,
the time devoted to testing. Another practice testing
is to start software
compromising
a the same moment the project starts and it is a continuous process until the project finishes.
programming and
COunterpoint, some emerging software disciplines such as
extreme
e software development movement, adhere to a "test-driven software development"
model software engineers (often with pair
DrOe s process, unit tests are written first, by the Of course these tests fail initially;
i n the extreme programming methodology). portions of the
to. Then as code is written it passes incrementally larger
st suites.Teed
est suites The test suites are continuously updated as new failure conditions and corner cases
ere disco
regression developed. Unit
tests that are
ests à n d they are integrated with any
are source code and generally integrated
mto the maintained software
tdined along with the rest of the
build processalong tests being relegated
to a partially manual
uild acceptance (with inherently interactive
process)
tng can be done on the following els:
Unit testin g tests the minimal software component,
or module.
Each unit (basic
for the unit
Comn
comp that the detailed design
of the software is tested to verify
 278 impleniented.
In an object-orientep
environmnenSoftuware Enginer
is meerine
has been
correctly m i n i m a l unit tests include the constructo this
the
level, and tors and
the class
defects in interfaces
the interfaces and
and iinter d
Integration testing
exposes
bet.
betwe t
(modules). Progressively larger of groups tested software it uctors.
corresponding
components to elements of the architectural design are integratre Co

works as a system.
ntegOnenttatedta
the software

tests a completely integrated system to

ested unti
System testing
requirements
verifies that asystem is integrated
meets
Svstem integration testing
ed +to any external
defined in the system requirements. nal or third
party systems
final version of software, alpha and beta
Before shipping the testing are ofte
tocs:

additionally done
. Alpha testing is simulated or actual operational testing by potential
or an independent test team at the developers' site.
Alpha testing isusers,
ofte
customers

softnopwosaryed
for of-the-shelf software as a form of internal acceptance esting, before the
goes to beta testing.
.Beta testing comes after alpha testing. versions of the software, known
Lown as beta
versions, are released to a limited audience outside of the
software is released to groups of people SO that further testing can ensure team programming
has few faults or bugs. Sometimes, beta versions are made available to
the nrod
the open
to increase the feedback field to a maximal number of future users. public
Finally, acceptance testing can be conducted by the end-user, customer, or client h
validate whether or not to accept the product. Acceptance
hand-off
testing may be performed as part
process between any two phases of development.
Finding Faults
Finding Faults Early: It is commonly believed that the earlier a defect is found the cheaper
it is to fix it. The
it was found. For
following table shows the cost of
fixing the defect depending on the stag
example, if a problem in the
it would cost 10-100 times requirements is found only post-release, then
more to fix than if it had
review. already been found by the requiremenis

Time Detected
elease
Kequirements Architecture Construction System Test Post-
Time Introduction Requirements
Architecture 1x 3x 5-10x 10x 10-100x
25-100x
1x 10x 15x
Construction 1x 10x
10-25x

Testing Tools
Programme testing and fault detection can be aided significantly by testing
tools
and

debuggers. Testing/debug tools include features significanuy uy

such as:
Spftanre
esting
lesting
y f a n r eT e s

P r o g r a m m em o nonitors, permitting full or partial 279

n Set Simulator, permittin8 complete monitoring of programme code
I n s t r u c t i o n

trace facilities instruction level including:

monitoring and
Programme
me
animation, permitting step-by-step
at source level or in
machine code execution and
breakpoin
conditional
Code coverage reports
ddump
F o r m a t t e d
or
iables on error or at chosen points
Symbolic debugEing, tools
allowing inspection of programme
variabl

d
Automated functional GUI
fur testing tools are used to repeat
the GUI system-level tests through
Danchmarks, allowing rurn-time pertormance comparisons to be made
Derformance analysis (or profiling tools) that can help to
resource usage highlight hot spots and

Came of these features may be

incorporated into an Integrated Development Environment
DE)
Measuring Software Testing
Usually, quality is constrained to such
topics as correctness, completeness, security, but
an also include more technical requirements as described under the ISO
standard ISO 9126,
such as capability, reliability, efficierncy, portability,
maintainability, compatibility, and usability.
There are a number of common software measures, often called
"metrics", which are used
tb measure the state of the software or the
of the adequacy testing.
Testing Artifacts
oftware testing produce several artifacts.
process can

est Plan: A test specification is called a test plan. The developers are well aware what
n s will be executed and this information is made available to management and the
add he idea is makes them more cautious when developing their code or making8
ditional nanges.
chang Some companies have a higher-level document calleda test strategy
Traced
tomeiliy Matrix: A traceability matrix is a table that correlates requirements or design
docum to test document It is used to change tests when the source documents are
anged, or to
verify that the test results are correct.
Test Case:. references
rom a ase normally consists of a unique identifier, requirement
1o folowlesign preconditions, events, a series of steps (also
known as actions)
ollow, specification,
an input input, output, defined test case is
Output, expected result, and actual result. Clinically
a
an 'for condition x your deriveda
Tesult is y'and
,
an
expected
pected result.
resu This can be as pragmatic as
detail the input scenario and what
Smight whereas ds other test cases described in moreof (but often steps are contained
steps
testexpected. It be a series
occasionally
eparate test
Parate prc
procedure
can
that canbe exercised against multiple
test cases, as a matter of
 ected outcom The optional warfielde Engmeerimg
outo
or expected
280 expected
resu
but
with
one

of execution
number, related quirement(s), depth, ds are
o r d e r of
executior

economy) order
or
the test is
automatahla

ID, test step, whether

d o s beenPh, test a

criptions. Atest caA te

for
check boxes steps, and
case
or
prerequisitesta tes

wordated.Lshodt
a trq
author,
and
contain
also ult. These
result. I h e s e steps can be stored
test cases may
for the
actual
renoee Stored in a case
place ther common reposit
contain a or
other In aa da
also
document,
database,
spreadsheet,
test results,
who generated database
the results.
results, and proces
also be
able to see past
orate those
those results. These Dast
results. 'These past results would W
may
configuration
was
used to generate
table.
usually st e
in a separate combination of a test case, test
caco

is the
Test Script:
The test script
the product of work
createdCedure.
and procedure, a

was
derived from
comhiomated test
the term automated,
bot1erh ess
manual automated, or or aa
Initially
Today, test scripts can be combination of
test tools. of test
ferm for a collection cases io
Test Suite: The
most common a
test suite.
more detailed
goals for each
instructions or Ihe
suiteoften also contains
where the tester identifies the system conf
on of tes te
contains a section
It definitely also contain prerequisite states or sfene
of test cases may and
A
testing. group
of the tollowing
tests. descriptns
sets of values or data are used to tos
Test Data: In most cases, multiple test the same
feature. All the test values and changeable ens
functionality of a particular
components are collected in separate files and stored as test data. It is also useful to mental
ovide
with the product or a project.
this data to the client and
Test Harness: The software, tools, samples of data input and output, and configuai
test harness.
guraions
are all referred to collectively as a

Sample Testing Cycle

Although variations exist between organisations, there is a typical cycle for testing
Reguirements Analysis: Testing should begin in the requirements phase ofthesoftwae
development life cycle. During the design phase, testers work with developes
thae
defermining what aspects of a design are testable and with what parameters
tests work.
s will e
Test Planning: Test strategy, test plan, testbed creation. Since manyacuvau
carried out
during testing, a plan is needed.
st Ccrips
lest Development: Test procedures, test scenarios, test cases, test datast
to in
use
testing software. and repo
Test Execution: Testers and tests
execute the software based on the plans au
any errors found to the
Test
development
team.
and
make
hs
Ke
tinal

Reporting:
Once testing is
completed, testers generated is i s readyforr
ready o
l e t r i c s
ease

reports their test effort and whether or not the

on ted ual

software te
Test Result team
Analysis: Or Defect
along with the client, in order to Analysis, is
e byy
the b e treated, fixed,
development

(Le. found
wha decide what defects shoualt with la
aecide dealt with late:
software working properly) or deferred to