SoftwareTesting Introduction

Dr Atul Gupta
DrAtulGupta

Errors defects faults & Failures

Errors,defects,faults&Failures
Error:Amistake madebyaprogrammer
Example:Misunderstoodtherequirements.

Defect/fault/bug:Manifestationofanerrorin
a program
aprogram.
Example:
Incorrectcode:
Correct code
Correctcode:

if(a<b){foo(a,b);}
if(a>b){foo(a,b);}
if
(a>b) {foo(a b) }

Failure:
Failure:Manifestationofoneormorefaults
Manifestation of one or more faults in
in
theobservedprogrambehavior

Failure
Incorrect
Incorrectprogrambehaviorduetoafaultin
program behavior due to a fault in
theprogram.
Failurecanbedeterminedonlywithrespectto
Failure can be determined only with respect to
asetofrequirementspecifications.
Anecessary
A necessary conditionforafailuretooccuris
condition for a failure to occur is
thatexecutionoftheprogramforcethe
erroneous portion of the program to be
erroneousportionoftheprogramtobe
executed.
Sufficient condition?

T ti P bl
TestingProblem
Error-revealing
inputs cause
failure

I
Inputs
t

Program
Outputs

Erroneous
outputs indicate
failure

Testing Activities in the SDLC

TestingActivitiesintheSDLC
Requirement
Specification

Acceptance
Testing
SRS

Tested Software

System Design

System Testing

System Design

Subsystem

Detailed Design
g

Integration
g
Testing
g

Detailed Design

Tested Units

Coding

Unit Testing
Code

Levels of Testing
LevelsofTesting
Type of Testing

Performed By

Lowleveltesting
Unit(module) testing
Integration
g
testingg

Programmer/Developer
Development team
Developmentteam

Highleveltesting
System testing
Function testing
Acceptance testing

IndependentTestGroup
Independent
Test Group
IndependentTestGroup
Customer

Testing Terms
TestingTerms
Test
Testcase
case Asetof
A set of Inputs,executionpreconditions,and
Inputs, execution preconditions, and
expectedoutcomesfortestinganspecificaspectofCUT
TestSuite
Test Suite Acollection
A collection oftestcasesfortheCUT
of test cases for the CUT
TestCriterion Asetoftestrequirements
Failure
Failure Anobservation
An observation ofthebehaviorofCUTdifferentfrom
of the behavior of CUT different from
expectedone
Fault
Fault Acondition
A condition inCUTthatmaycausefailure
in CUT that may cause failure
Effectiveness Faultdetectioncapability
Efficiency Theaveragetestingcost(i.e.effort)toidentifya
fault intheprogram

A Testing Process
ATestingProcess
The Code Under Test (and
its Specifications)
Procedural
Systems

Object-Oriented
Systems

Test Case Selection

(Test Input Selection)

Test Case Selection

(Test Sequence
Selection + Test Input
Selection)

Test Case Execution

Comparison with
Expected Outcomes
(Test Oracle)

Test Execution Results

(Pass/Fail Information)

NO
Stop?
YES
Stop

ATestingProcess
g
The Code Under Test (and
its Specifications)
Procedural
Systems

Object-Oriented
Systems

Test Case Selection

(Test Input Selection)

Test Case Selection

(Test Sequence
Selection + Test Input
Selection)

Test Case Execution

Comparison with
Expected Outcomes
(Test Oracle)
YES

Fail
?

Debugging/
Bug Fixing

NO

Test Execution Results

(Pass/Fail Information)

Stop?
YES

Stop

A Testing Process
ATestingProcess
The Code Under Test (and
its Specification)
Procedural
Systems

Object-Oriented
Systems

Test Case Selection

(Test Input Selection)

Test Case Selection

(Test Sequence
Selection + Test Input
Selection)

Test Case Execution

Comparison with
Expected Outcomes
(Test Oracle)
YES

Debugging/
Bug Fixing

Test Execution Results

(Pass/Fail Information)

Fail
?
NO

Stop

NO

Stop?
YES

Stop

Unit Testing
UnitTesting
Doneonindividualunits(Class
Done on individual units (Class orsmallcluster
or small cluster
classesinOOSystemandafunction ormodules
in Procedural Systems)
inProceduralSystems)
Testunitw.r.tunitspecification
Mostlydonebydevelopers/programmers
M l d
b d l
/
requiresstubs anddrivers
FurtherPopularizedbytheavailabilityof Unit
TestingFrameworks
g

StubsandDrivers
Stub
Stub
dummyunitswhich
simulates the
simulatesthe
functionofother
units required to
unitsrequiredto
testunitundertest

Driver

e.g.
g unit
relationships

eg.
g to unit test B
in isolation
Stub for A

A
Driver

Codethatexecutes
unit under test
unitundertest
C

Stub for C

Integration Testing
IntegrationTesting
Testsagroupofunits,modules,orasubsystem
Testsubsystemstructurew.r.tdesign,subsystem
y
g ,
y
functions
Focusesonunit
Focuses on unit interfaces
Donebyone/groupofdevelopers
Undertakenbasedondependency considerations

IntegrationTestApproaches
g
pp
Nonincremental(BigBangintegration)
Non incremental (Big Bang integration )

unit test each module independently

combine
bi allll the
th units
it to
t form
f
the
th subsystem
b t
in
i
one step, and test the combination

Incremental

instead of testing each unit in isolation, the

next unit to be tested is first combined with
the set of units that have already been tested
integration approaches:approaches:

Functional System: Top-down, Bottom-up

OO System: Based on dependency considerations

Comparison
Non Incremental
NonIncremental

Incremental

requiresmorestubs,drivers

requireslessstubs,drivers

moduleinterfacingerrors
detectedlate

moduleinterfacingerrors
detectedearly

debuggingerrorsisdifficult

debuggingerrorsiseasier
resultsinmorethoroughtesting
g
g
ofmodules

E
Example:IntegrationTestinginFunctionalSystems
l I
i T i i F
i
lS

Example:IntegrationTestinginOOSystems
A

Topdown Integration
TopdownIntegration
Begin with the top module in the module call
hierarchy
Stub modules are produced
Stubs are often complicated

The next module to be tested is any module

with at least one previously tested
p
((calling)
g) module
superordinate
After a module has been tested, one of its
stubs is replaced by an actual module (the
nextt one to
t be
b tested)
t t d) and
d its
it required
i d stubs
t b

TopdownIntegrationTesting
Example:
A

Stub B

Stub C

Stub D

TopdownIntegrationTesting
Example:

Stub E

Stub F

Stub C

Stub D

BottomUpIntegration
Beginwiththeterminalmodules(thosethat
do not call other modules) of the modules call
donotcallothermodules)ofthemodulescall
hierarchy
Adrivermoduleisproducedforeverymodule
p
y
Thenextmoduletobetestedisanymodule
whosesubordinatemodules(themodulesit
calls)haveallbeentested
ll ) h
ll b
t t d
Afteramodulehasbeentested,itsdriveris
replaced by an actual module (the next one to
replacedbyanactualmodule(thenextoneto
betested)anditsdriver

Example:ModuleHierarchy
A

BottomUpIntegrationTesting
Example:
Driver E

Driver F

BottomUpIntegrationTesting
Example:

Driver A

Comparison
TopdownIntegration
Advantage
askeletalversionofthe
programcanexistearly

Disadvantage

BottomupIntegration
Disadvantage
theprogramasawholedoes
t i t til th l t
d l
notexistuntilthelastmodule
isadded

requiredstubscouldbe
q
expensive

No clear winner

System Testing
SystemTesting
Process of attempting to demonstrate that
the program or system does not meet its
original requirements
req irements and objectives
objecti es as
stated in the requirements specification
Test cases derived from
requirements specification
system objectives, user documentation

System Testing: An Example

SystemTesting:AnExample

Types of System Tests

TypesofSystemTests
Functional Testing
g
Testing Functionality to be delivered by the system

Specialized Testing (Testing NFRs)

Volume testing
to determine whether the program can handle the required volumes of data,
requests, etc.

Load/Stress testing
to identify peak load conditions at which the program will fail to handle required
processing loads within required time spans

Usability (human factors) testing

to identify discrepancies between the user interfaces of a product and the
human engineering requirements of its potential users.

Security Testing
to show that the programs security requirements can be subverted

Types of System Tests

TypesofSystemTests
Specialized Testing

Performancetesting
todeterminewhethertheprogrammeetsitsperformance
requirements (eg response times throughput rates etc )
requirements(eg.responsetimes,throughputrates,etc.)

Recoverytesting
ttodeterminewhetherthesystemorprogrammeetsits
d t
i
h th th
t
t it
requirementsforrecoveryafterafailure

Installabilitytesting
Installability testing
toidentifywaysinwhichtheinstallationproceduresleadto
incorrectresults

ConfigurationTesting
todeterminewhethertheprogramoperatesproperlywhenthe
software or hardware is configured in a required manner
softwareorhardwareisconfiguredinarequiredmanner

Types of System Tests

TypesofSystemTests
Specialized
p
Testing
g

Compatibility/conversiontesting
todeterminewhetherthecompatibilityobjectivesoftheprogram
p
y j
p g
havebeenmetandwhethertheconversionprocedureswork

Reliability/availabilitytesting
todeterminewhetherthesystemmeetsitsreliabilityand
availabilityrequirements

Resourceusagetesting
Resource usage testing
todeterminewhethertheprogramusesresources(memory,disk
space,etc.)atlevelswhichexceedrequirements

Acceptance Testing
AcceptanceTesting
Performed by the customer or end user
Compare the software to its initial
requirements and needs of its end users

Alpha and Beta Testing

AlphaandBetaTesting
Tests performed on a SW Product before
its released to a wide user community.
Alpha testing
conducted at the developers site by a User
tests conducted in a controlled environment

Beta testing
conducted
d t d att one or more User
U
sites
it by
b th
the
end user of the SW
it is a live
live use of the SW in an environment
over which the developer has no control

Regression Testing
RegressionTesting
Testing changes made to the code
To ensure that no new bugs are introduced in the
y
system
Can be performed at any level unit, integration, and
system level
In practice, mostly performed at unit and system level in
GUI testing
Challenges include

Reducing Regression Test Suite size

Identifyy Invalid test cases
Write additional test cases
Automation

When to Stop Testing ?

WhentoStopTesting?
Stop
Stopwhenthescheduledtimefortesting
when the scheduled time for testing
expires
Stopwhenallthetestcasesexecute
g
withoutdetectingerrors
bothcriteriaarenotgood

Better Test Completion Criteria

BetterTestCompletionCriteria
Make use of Coverage based Testing
MakeuseofCoveragebasedTesting
Example:Testcasesderivedfrom
1)satisfyingmulticonditioncoverageand
)
y g
g
2)boundaryvalueanalysisand
3) cause effect graphing
3)causeeffectgraphing

Better Test Completion Criteria

BetterTestCompletionCriteria
Statethecompletioncriteriaintermsof
State
the completion criteria in terms of
numberoferrorstobefound.Thisrequires:
Anestimateoftotalnumber
A
ti t f t t l
b oferrorsinthe
f
i th
program
Anestimateofthe%oferrors thatcanbe
foundthroughtesting
Anestimatesofwhatfractionoferrors
originate inparticulardesignprocesses,and
in particular design processes, and
duringwhatphasesoftesting theyget
detected.
detected.

Better Test Completion Criteria

BetterTestCompletionCriteria
Plotthenumberoferrorsfoundperunittimeduringatest
phase.
Therateoferrordetectionfallsbelowaspecifiedthreshold

#E
Errors ffound

#E
Errors found

40
403020101 2 3 4 5
W k
Week

40
403020101 2 3 4 5
W k
Week

Summary (PartII)
Summary(Part
II)
Testingisperformedatdifferentlevels
g p
UnitTestingisthefirstformaltestactivity
UnitTestingcanbehighlyefficientandcost
effective
ff i
IntegrationTestingisincremental
Regressiontestingisabouttestingchanges
R
i t ti i b t t ti
h
madetothecode
Toolsupportishighlydesirableinsoftware
Tool support is highly desirable in software
testing
y
Howcantheeffectivenessandefficiencyof
softwaretestingcanbeenhancedfurther?

R f
References
Beizer,Boris(1990).SoftwareTesting
Techniques (SecondEditioned.)
(Second Edition ed )
Myers,Glenford J.(1979).TheArtofSoftware
Testing.JohnWileyandSons
Wikipedia
Wikipedia http://en.wikipedia.org/wiki/Software_testing
http://en wikipedia org/wiki/Software testing