MCA 203 Software Engineering & Project Management Lecture 5 By: Deepak Garg, Assistant Professor, NIT Kurukshetra
MCA 203 Software Engineering & Project Management Lecture 5 By: Deepak Garg, Assistant Professor, NIT Kurukshetra
Contents
1 Goals of Software Testing ............................................................................................................... 2
1.1Short Term or Immediate Goals ................................................................................................. 2
1.2Long Term Goals ........................................................................................................................ 2
1.3Post Implementation Goals ......................................................................................................... 3
2 Testing Principles............................................................................................................................. 3
3 Test case characteristics .................................................................................................................. 4
4 Key Terms of Software Testing ...................................................................................................... 5
5. Testing Methods ............................................................................................................................... 6
5.1 White Box Testing ...................................................................................................................... 6
White Box Testing Techniques:............................................................................................. 7
5. 2 Black Box Testing ..................................................................................................................... 7
Black Box Testing Techniques: ............................................................................................ 8
5.3 Grey Box Testing ........................................................................................................................ 9
5.4 Comparison ................................................................................................................................. 9
6. Automated Software Testing ....................................................................................................... 10
6.1 Consideration during Automated Software Testing .................................................................. 10
6.2 Advantages of Automated Software Testing ............................................................................ 11
6.3 Disadvantages ........................................................................................................................... 11
6.4 Basic Tools used in Automated Software Testing .................................................................... 12
7. Coverage Criteria ......................................................................................................................... 13
8. Basis Path Testing ......................................................................................................................... 14
8.1 Importance of Basis Path Testing: ............................................................................................ 14
8.2 Steps of Basis Path Testing:...................................................................................................... 14
8.3 Key Terms Related To Basis Path Testing: .............................................................................. 14
a. Basis Set ..................................................................................................................................... 14
b. Control Flow Graph (CFG)........................................................................................................ 15
c. Cyclomatic Complexity ............................................................................................................. 15
9. Graph Matrices (Graph matrix and Connection Matrix).......................................................... 17
Graph Matrix ................................................................................................................................... 17
a. Graph matrix for finding set of all paths ................................................................................. 17
b. Connection Matrix................................................................................................................... 18
c. Cyclomatic complexity automatically. ................................................................................... 18
1
1 Goals of Software Testing
To understand the new concepts of software testing and to define it thoroughly, goals of
software testing may be classified into three major categories as shown in fig. 1.1
2
Testing should be complete in the sense that it must satisfy the user for all the specified
requirements mentioned in the user manual, as well as for the unspecified requirements which
are otherwise understood.
Risk Management
Risk is the probability that undesirable events will occur in a system. Software testing acts as
a control to manage risks (minimizing or eliminating).
Post Implementation Goals
These goals are important after the product is released. Some of post implementation goals are
discussed below .
Reduce maintenance cost
The maintenance cost of any software product is not its physical cost, as the software does not
wear out. The only maintenance cost in software products is its failure due to errors and post
release errors are costlier to fix.
Improved Software testing process
A testing process for one project may not be successful and there may be scope for
improvement. Therefore, the bug history and post-implementation results can be analyzed to
find out snags in the present testing process, which can be rectified in future projects. Thus,
the long term post implementation goal is to improve the testing process for future projects.
2 Testing Principles
Below are the basic but important principles of software testing which should be followed by
every tester, if these are not followed by tester then it may cause failure of complete testing
process.
Effective testing, not exhaustive testing: All possible combinations of tests become so
large that it is impractical to test them all. So, considering the domain of testing as infinite,
exhaustive testing is not possible.
Testing is not a single phase performed in SDLC: Testing is done at every phase of SDLC
like during requirement gathering phase, the analysis verification of requirements etc.
Destructive approach for constructive testing: Tester must have the psychology that bugs
are always present in the program and they must think about the technique of how to uncover
them. This psychology of being always suspicious about bugs is a negative/destructive
approach.
Early testing is the best policy: Testing process is not a phase after coding; rather it starts
as soon as requirement specifications are prepared. Moreover, the cost of bugs can be reduced
3
tenfold, as bugs are harder to detect in later stages if they go undetected. Thus the policy is to
start as early as possible.
Probability of existence of an error in a section of a program is proportional to the number
of errors already found in that section: The principle provides the insight that if some section
are found error-prone in testing, then our next testing effort should give priority to these error-
prone sections.
Testing strategy should start at the smallest module level and expand towards the whole
program: This principle supports the idea of incremental testing. Testing must begin at the
unit or module level, gradually progressing toward integrated module and finally the whole
system. Testing cannot be performed directly on the whole system. It must start with the
individual modules and slowly integrate the modules and test them. After this, the whole
system should be tested.
Testing should also be performed by an independent team : When programmers develop
the software, they test it at their individual modules. However, these programmers are not
good testers of their own software. They are basically constructors of the software, but testing
needs a destructive approach. Programmers always think positively that their code does not
contain bugs. Moreover they are biased towards the correct functioning of the specified
requirements and not towards detecting bugs [CHA2011].
Everything must be recorded in software testing: Testing is not an intuitive process; rather
it is a planned process. It demands that every detail be recorded and documented.
Invalid inputs and unexpected behavior have a high probability of finding an error:
Whenever software is tested, test is done for valid inputs and for the functionality that the
software is supposed to do. But thinking in a negative way, software must be tested with
invalid inputs and with the behavior which is not expected in general.
Testers must participate in specification and design reviews: If testers are not participating
in other reviews like specification and design, it may be possible that either some
specifications are not tested or some test cases are built for no specifications.
A good test is not redundant: Testing time and resources are limited. There is no point in
conducting a test that has the same purpose as another test. Every test should have a different
purpose (even if it is subtly different).
4
A good test has a high probability of finding an error: To achieve this goal, the tester must
understand the software and attempt to develop a mental picture of how the software might
fail. Ideally, the classes of failure are probed. For example, one class of potential failure in a
GUI (graphical user interface) is a failure to recognize proper mouse position. A set of tests
would be designed to exercise the mouse in an attempt to demonstrate an error in mouse
position recognition.
A good test should be “best of breed” : In a group of test that have a similar intent, time
and resource limitations may mitigate toward the execution likelihood of uncovering a whole
class of errors should be used.
A good test should be neither too simple nor too complex: Although it is sometimes
possible to combine a series of tests into one test case, the possible side effects associated
with this approach may mask errors. In general, each test should be executed separately.
Failure: Failure of a program is the mismatch between expected and actual results of the
program under test.
Fault/Defect/Bug: Fault is a condition in a program that causes failure.
Error: Error is very general term used for human mistake. Whenever a development team
member makes a mistake in any phase of SDLC errors are produced, error is the actual
location in the program where a mistake has been made that produced bug. It might be
typographical error, a misleading of a specification etc. Thus, an error causes a bug and the
bug in turn causes failure, as shown in fig. 1.2.
5
Test data: Test data is the data which have been specifically identified to use in testing
computer program.
Test case: It is a well documented procedure designed to test the functionality of a feature
in the system. Test case includes test case id, purpose, preconditions, inputs, expected outputs
etc.
Test oracle: An oracle is the means to judge the success or failure of a test, i.e. to judge the
correctness of the system for some test. This simplest oracle is comparing actual results with
expected results by hands.
Test automation: Developing software for testing the software product.
Coverage: It includes coverage of program or faults. The aim of coverage based testing
method is to „cover‟ the program with test cases that satisfy some fixed coverage criteria. The
aim of fault coverage criteria is to cover maximum fault in a program.
Path: It is sequence of nodes and edges. If path starts form entry node and end at exit node
then it is called as a complete path.
Branch predicate or condition: It is condition, in which a node that may lead to either true
path or false path.
Path predicate or decision: A path predicate is defined as the collection of branch
predicates which are required to be true in order to traverse a path.
Feasible path: The path, where there is valid input that execute in the path.
Infeasible path: The path, where there is no valid input that execute in the path.
Testware: The documents created during testing activities are known as testware. It may
include test plans, test specifications, test case design, test reports etc.
5. Testing Methods
Any engineered software product can be tested in one of the three methods white box testing,
black box testing and grey box testing. All these testing methods come under the class of
dynamic testing because all the methods that execute the code to test software come under the
dynamic testing technique.
Guarantee that all independent paths within a module have been exercised at least once.
Exercise all logical decisions on their true and false sides.
6
Execute all loops at their boundaries and within their operational bounds.
Exercise internal data structures to ensure their validity.
5.4 Comparison
Table 1.1 shows the comparison between black box, grey box and white box testing in terms
of granularity, known or unknown internal knowledge, degree of exhaustiveness, alternative
names, functionality etc .
Internals not known, neither Internal partially known. Internals fully known.
required to be known.
Not suited for algorithm Not suited for algorithm testing. Appropriate for
testing. algorithm testing.
9
Also known as Also known as translucent box Also known as
Opaque box testing testing. Glass box testing
Closed box testing Structural testing
Function testing Clear box testing
Data driven testing
Clear and reasonable expectations should be established in order to know what can and
what cannot be accomplished with automated testing in the organization.
There should be clear understanding of the requirements that should be met in order to
achieve successful automated testing. This requires that the technical personnel should use the
tools effectively.
The organization should have detailed, reusable test cases which contain exact expected
results and a standalone test environment with a restorable database.
10
Testing tools should be cost effective. The tool must ensure that test cases developed for
manual testing are also useful for automated testing.
Select a tool that allows the implementation of automated testing in a way that conforms to
the specified long term testing strategy.
Saves time: Automated execution of test cases is faster than manual execution. This saves
time. This time can also be utilized to develop additional test cases, thereby improving the
coverage of testing.
More Reliable: Manual running the tests may result in boredom and fatigueness, more
chances of human error. Automated testing overcomes all these shortcomings.
Reusable: Automatic testing is reusable while manual testing is not reusable.
Less Resource Requirement: Tests, once automated take comparatively far less resources
to execute. A manual test suite requiring 10 persons to execute it over 31 days i.e. 31*10=310
man days, may take just 10 man-days for execution, if automated.
Independency to Test Engineers: Manual testing requires the presence of test engineers but
automated test can be done independently.
No Need of High Educated Staff (testers): Automated testing doesn‟t require high educated
testers because testing is done by automatic tools.
6.3 Disadvantages
Below are disadvantages of automated software testing over manual software testing in terms
of cost, coverage of bugs, variety of bugs, time to test execution .
Costly: An automated test suite development is normally 3-5 times the cost of a complete
manual test cycle.
Not complete coverage: It achieves less than 50% statement coverage because it finds bugs
only in part of codes that are actually executable.
Uncovers limited types of bugs: It can uncover only those types of bugs that are
traversable through execution, while there is no such condition in manual software testing.
Only when executables are ready: Automatic testing can be done only when executables
are ready, while manual testing can be done before compilation.
11
6.4 Basic Tools used in Automated Software Testing
Since testing is of two types, static testing and dynamic testing. In the same way testing tools
are accordingly classified in to two types as given below:
Static Testing tools: These tools do not involve actual input and output. Rather they take a
symbolic approach to testing, i.e. they do not test the actual execution of the software. It
includes flow analyzers, path tester, coverage analyzers, interface analyzers etc.
Dynamic Testing tools: These tools test the software system with „live‟ data. Dynamic test
tools includes test drivers, test beds, emulators mutation analyzers etc.
On the whole, there are various tools for automating the software testing and that can be used
in different areas of testing. At this moment, there are many tools to assist software testing
such as capture/playback tools, tools for automated execution of tests, coverage analyzers, test
case generators, logical and complexity analyzers, code instrumentation tools, defect tracking
tools and test management tools.
Capture/playback tools can be either intrusive or non-intrusive. Intrusive tools are native
tools as they along with software under test reside on the same machine. In non intrusive
capture/playback tool software under test and testing tool resides on the different machines.
Capture/playback tools are used to record testing sessions in script files, allowing to playback
them later. It can be made for multiple tests and the results can be compared. Those tools are
useful in regression testing. One of the most boring and time consuming activity during
testing life cycle is to rerun manual tests number of time. So, capture/playback tools are of
great help to the testers.
Coverage analyzers are tools used to evaluate how much of the structure of the tested code
has been covered by the test cases. These tools are useful for identifying the code sequences
which were not covered by the tests.
Test case generations are tools based on information such as requirements, data models
and object models which are used to generate significant test cases. The main advantage of
using these tools is that the testing redundancy is eliminated by finding the test cases that
assure the best coverage for the code.
Test data generators are used to fill data in files and databases that will be used for testing.
Data are chosen randomly or based on some specific conditions. These tools are usually used
for large volume of data necessary in operational testing and load testing.
Tools for code instrumentation are tools used for the analysis of source code of the
program under test and inserting calls to specific functions in order to gather information
about program during its execution.
12
Defect tracking tools are used to manage the information regarding the detected errors,
their status and to centralize that information in order to provide information regarding the
trend of the fault. Some steps of the software development cycle can be improved on the basis
of trends obtained.
Tools for test management are tools used to assist in planning and organizing of all
elements involved in testing such as- script files, test cases, test reports and test results.
Logical and complexity analyzers are tools used to quantify the complexity of the code.
Many of these tools can show the graphical structure of the code [AGG2006]. These tools are
useful to find the necessary test cases for execution of some point s of the code in complex
modules.
7. Coverage Criteria
To measure how well the program is exercised by a test suite, one or more coverage criteria
are used or it gives the measure to what extent the software is exercised in the process of
testing by a test suite . Coverage can be applied during any stage of testing, whether it is unit
testing, integration testing or system testing etc. Test coverage can be based on functional
specification (black box testing) or an internal program structure (white box testing). Structure
coverage is more commonly used. Such testing can measure coverage at various granularities,
including statements, lines, blocks, conditions, methods and classes. It provides a way to
quantify the degree of thoroughness of white box testing. A coverage criterion is usually
defined as a rule or requirement, which a test suite needs to satisfy.
Coverage= (Number of coverage items exercised / Total no. of coverage items) * 100
There are number of coverage criteria exist, below are few ones :
Statement Coverage: Has each line of the source code been executed by test suite?
Decision Coverage: Has each branch/ decision of the source code been executed on both
true and false output by test suite?
Condition Coverage: Has each condition of the source code been executed on both true and
false output by test suite?
Function Coverage: Has each function in the source code been executed by test suite?
Path Coverage: Has every possible path in the source code been executed by test suite?
Multiple Condition coverage: Has all combination of different multiple conditions in the
source code been executed by test suite?
13
8. Basis Path Testing
Basis path testing is a white box testing technique. It was proposed by Tom McCabe. The
basis path method enables the test case designer to derive a logical complexity measure of a
procedural design and use this measure as a guide for defining a basis set of execution
paths . This set of independent paths is called basis set, that’s why it is termed as basis path
testing.
a. Basis Set
Basis set is a finite set of linearly independent paths through a standard flow graph. An
independent path is a path of a program, where at least one edge of this path never appears
in any other path in the control-flow graph (CFG). Every path in the Basis set should satisfy
these three conditions:
14
b. Control Flow Graph (CFG)
The control flow graph is a graphical representation of control structure of a program. Flow
graphs can be prepared as a directed graph. A directed graph (V, E) consists of a set of
vertices V and a set of edges E that are ordered pairs of elements of V.
Notations of CFG
Node- It represents one or more procedural statements (having a fixed input and fixed output).
The nodes are denoted by a circle. These are numbered or labeled.
Edge- They represent the flow of control in a program, denoted by an arrow on the edge
Decision Node – A node with more than one arrow leaving is called a decision node.
Regions- Areas bounded by edges and nodes are called regions.
Important points regarding CFG
Sequential statements having no conditions or loops can be merged in a single node. That is
why; the flow graph is also known as decision to decision graph or DD graph.
An edge must terminate at a node, even if the node does not represent any procedural
statements.
When counting region we include the area outside the graph as a region.
A compound condition occurs when one or more Boolean operators (logical OR, AND,
NAND, NOR) is present in a conditional statement, so a separate node is created for each of the
compound condition as shown in fig. 1.3
c. Cyclomatic Complexity
McCabe has given a measure for the logical complexity of a program by considering its
control flow graph by considering only independent paths. Cyclomatic complexity number
gives the number of independent paths (as basis set) as upper bounds for the number of
tests that must be conducted to ensure that all the statements and every condition has been
executed on its true and false sides at least once.
15
Strongly connected graph in which each node can be reached from any other node.
When counting the regions, area outside the graph is also considered as a region.
Table 1.2 gives cyclomatic complexity of any graph g including following notations:
V(g)= 3 V(g)=4-1
Counting no. of decision nodes (d) for Switch-case/multiple if else: When a decision node
has exactly two arrows leaving it, then we count it as a single decision node [CHA2011].
However, switch case and multiple if else statements having more than two arrows leaving a
16
decision node, then no. of decision nodes will be one less than no. of arrows leaving it as
shown in table 1.3.
K>2 k-1
Graph Matrix
It is a square matrix whose rows and columns are equal to the number of nodes in the flow
graph; matrix entries represent a connection between the nodes [AGG2006]. Example CFG is
shown in fig. 1.4 and its graph matrix is in fig. 1.5
1. If there is a connection from node „a‟ to node „b‟, then it does not mean that there is
connection from node „b‟ to node „a‟.
2. To represent a graph matrix, digits are used for nodes and letter symbols for edges.
a. Graph matrix for finding set of all paths: If A is a initial graph matrix of 1 link path from
one node to other node then if we want to find k link path‟s graph matrix G from every node
to other node then it can be find by following.
A=one length matrix
17
G= A1 + A2 + A3………+ A k
b. Connection Matrix
It is a square matrix, where number of rows and number of columns are equal to number of
nodes in the CFG of a program . In the simplest form when the connection exists, the link
weight is 1, otherwise 0 ( but 0 is not entered in the cell entry of matrix to reduce the
complexity).
c. Cyclomatic complexity automatically: Here is the use of connection matrix in finding
Cyclomatic complexity Number in fig. 1.6 of CFG graph in figure 1.4.
1. For each row, count the total number of 1s and write it in front of that row.
2. Subtract 1 from that count . Ignore the blank rows, if any.
3. Add the final count of each row.
4. Add 1 to the sum calculated in step 3
5. The final sum in Step 4 is the cyclomatic number of CFG.
18