Testing Process

Test and Test case

Testing is obviously concerned with errors, faults, failures,
and incidents.

Test — A test is the act of exercising software with test cases.

A test has two distinct goals: to find failures and to
demonstrate correct execution.

Test case — Test case has an identity and is associated with a

program behavior. A test case also has a set of inputs and
expected outputs

Test and Test case terms are used interchangeably. In practice,

both are same and are treated as synonyms..
 Test, Test Case and Test Suite
Test cases need to be developed, reviewed, used,
managed, and saved.

Test cases are valuable — at least as valuable as

source code.
The set of test cases is called a test suite. Hence
any combination of test cases may generate a test
suite.

3
 Test, Test Case and Test Suite
•Test cases occupy a central position in testing.
•Test cases should have an identity .
•Need to record the execution history of a test case-
• When and by whom it was run,
• Pass/fail result of each execution
• Version (of software) on which it was run.

Test case template

5
 Life Cycle model for Testing
In the development phases, three opportunities arise for
errors to be made, resulting in faults that propagate
through the remainder of the development process.

a)The first three phases are Putting Bugs IN;

b)The testing phase is Finding Bugs;
c)Last three phases are Getting Bugs OUT
d) The Fault Resolution step is another opportunity for
errors (and new faults). When a fix causes formerly
correct software to misbehave, the fix is deficient.
Life cycle model for Testing
 Testing Process

• The process of testing can be subdivided into

separate steps:
– Test Planning
– Test case Development
– Running Test cases
– Evaluating Test results.
Functional Test cases Identification
In Functional testing or Black box testing, the content
(implementation) of a black box is not known, and the
function of the black box is understood completely in
terms of its inputs and outputs

The only information used is the specification of the

software.
Functional Test cases Identification
Advantages:
•Independent of Software Implementation
•Test case development in parallel reducing the overall
project development interval.

Disadvantages:
•Significant redundancies may exist among test cases.
•Possibility of gaps of untested software.
Functional Test cases Identification
Functional methods are based on the specified behavior.

Test cases identified by two functional methods.

Method A identifies a larger set of test cases than does

Method B.
Functional Test cases Identification
Mainline approaches to functional testing –
• Boundary value analysis,
• Robustness testing,
• Worst-case analysis,
• Special value testing,
• Input (domain) equivalence classes,
• Output (range) equivalence classes,
• Decision table-based testing.
 Structural Test cases Identification
Structural testing also called white box (or even clear box)
testing.

Implementation (of the black box) is known and used to

identify test cases.

Familiarity with the concepts of linear graph theory is

essential.

In Structural Testing coverage metrics provide a way to

explicitly state the extent to which a software item has
been tested.
 Structural Test cases Identification
Structural methods are based on the Programmed
behavior.

Method A identifies a larger set of test cases than does

Method B.
 Structural Testing
Structural Testing Advantages:
■ Enables you to test the software’s logic
■ Enables you to test structural attributes, such as efficiency of code

Structural Testing Disadvantages:

■ Does not ensure that you’ve met user requirements
■ May not mimic real-world situations

15
 Functional Vs Structural Testing
Identification of test cases
Functional testing - Specification
Structural testing - program source code

If all specified behaviors have not been implemented, structural test

cases will never be able to recognize this.

If the program implements behaviors that have not been specified,

this will never be revealed by functional test cases.

Conclusion: Both approaches are needed; a judicious

combination will provide the confidence of functional
testing and the measurement of structured testing.
Functional Vs Structural Testing
 Black Box Testing
•Program is viewed as a black box.

•Unconcerned about the internal behavior and structure of the

program.

•Concentrate on finding circumstances in which the program does

not behave according to its specifications.

•Test data derived solely from the specifications (i.e., without

taking advantage of knowledge of the internal structure of the
program).

•To find all errors in the program, the criterion is exhaustive input
testing, making use of every possible input condition as a test
case. 18
 White Box Testing
•White-box or logic-driven testing, permits to examine the internal
structure of the program.

•This strategy derives test data from an examination of the program’s

logic (and often, unfortunately, at the neglect of the specification)

•Exhaustive path testing. is, if you execute, via test cases, all possible
paths of control flow through the program.

19
Quotes - The Art of Software Testing - Glenford Myers,

“Testing is the process of executing a program with the intent of finding

errors.”

“A good test case is one that has a high probability of detecting an as-
yet undiscovered error.”

”A successful test case is one that finds an error. An unsuccessful test

case is one that causes a program to produce the correct result. ”

”One should start with the assumption that the program contains errors
and then test the program to find as many of the errors as possible. ”

20
Exhaustive Testing is Impossible

 The domain of possible inputs is too large to test

 There are too many possible paths through the program to test

21
The domain of possible inputs is too large to test
We cannot test the program’s response to every possible input
 Valid inputs

Large number of inputs

 Invalid inputs

Set of Invalid Inputs is also too large to test.

22
Too many possible paths through the program to test

The number of unique logic paths through a program is too large.

[Myers].
Complete path testing does not guarantee that there will not be errors.
E.g Program does not match its specification.

23
 Domain of Possible Tests

 The domain of possible Tests is infinite. Even processing

resources and time are not sufficient for performing these tests.
 Testing needs to be performed on selected subsets that can ne
performed within the constrained resources.
 This selected group of sub sets but not whole domain of testing
makes effective software testing.

24
 Exhaustive Testing is Impossible-Example
3 Equilateral- triangle test cases for the triangle program-No way
guarantees the correct detection of all equilateral triangles.

The program could contain a special check for values 3842,3842,3842

and denote such a triangle as a scalene triangle. Since the program is a
black box, the only way to be sure of detecting the presence of such a
statement is by trying every input condition.

To test the triangle program exhaustively, you would have to create

test cases for all valid triangles up to the maximum integer size of the
development language. This in itself is an astronomical number of test
cases, but it is in no way exhaustive.

25
 Exhaustive Testing is Impossible-Example
To be sure of finding all such errors, you have to test
using not only all valid inputs, but all possible inputs. Hence, to test the
triangle program exhaustively, you would have to produce virtually an
infinite number of test cases, which, of course, is not possible.

26
 Effective Vs Exhaustive Testing
• Exhaustive or complete software testing means that every
statement in the program and every possible path combination
with every possible combination of data must be executed.

• All possible combinations of tests become so large that it is

impractical to test them all.

• Considering the domain of testing as infinite, exhaustive testing is

not possible.

• Tester’s approach should be based on effective testing to

adequately cover program logic and all conditions.

• Effective testing emphasizes efficient techniques to test the

software so that important features will be tested within the
constrained resources.
27
 How do we know that we have tested Enough?

• Limiting factors are

• Time
• Cost
• Personnel
• Customer
• Quality etc.

28
10 Software Testing Principles

29
 Software Testing : Principle 1
Principle 1: A necessary part of a test case is a definition
of the expected output or result.

•If the expected result of a test case has not been predefined,
chances are that a plausible, but erroneous, result will be interpreted
as a correct result.

•Encourage a detailed examination of all output by precisely spelling

out, in advance, the expected output of the program.

•A test case must consist of two components:

1. A description of the input data to the program.
2. A precise description of the correct output of the program
for that set of input data.
30
 Software Testing : Principle 2
Principle 2: A programmer should avoid attempting to test
his or her own program.
•After a programmer has constructively designed and coded a program,
it is extremely difficult to suddenly change perspective to look
at the program with a destructive eye.

•In addition, a programmer may subconsciously avoid finding

errors for fear of retribution from peers or from a supervisor, a client,
or the owner of the program or system being developed.

•The program may contain errors due to the programmer’s

misunderstanding of the problem statement or specification.

•Testing is more effective and successful if someone else does it.

31
 Software Testing : Principle 3
Principle 3: A programming organization should not test
its own programs.
•A project or programming organization is, with psychological problems
similar to those of individual programmers.

•A programming organization or a project manager is largely measured

on the ability to produce a program by a given date and for a certain
cost.

• Testing process, if approached with the proper definition, may be

viewed as decreasing the probability of meeting the schedule and the
cost objectives.

•It is more economical for testing to be performed by an objective,

independent party. 32
 Software Testing : Principle 4
Principle 4:Thoroughly inspect the results of each test.

•Numerous experiments show many subjects failed to detect certain

errors, even when symptoms of those errors were clearly observable on
the output listings.

•Errors that are found on later tests are often missed in the results from
earlier tests.

33
 Software Testing : Principle 5
Principle 5:Test cases must be written for input conditions
that are invalid and unexpected, as well as for those
that are valid and expected.

•Concentration on the valid and expected input conditions, at the

neglect of the invalid and unexpected conditions.

•Many errors that are suddenly discovered in production programs turn

up when the program is used in some new or unexpected way.
•
•Test cases representing unexpected and invalid input conditions seem
to have a higher error-detection yield than do test cases for valid input
Conditions.

34
 Software Testing : Principle 6
Principle 6: Examining a program to see if it does not do what
it is supposed to do is only half the battle; the other half is
seeing whether the program does what it is not supposed to do.

•Programs must be examined for unwanted side effects.

•For instance, a payroll program that produces the correct paychecks is

still an erroneous program if it also produces extra checks for
nonexistent employees or if it overwrites the first record of the
personnel file.

35
 Software Testing : Principle 7
Principle 7: Avoid throwaway test cases unless the program is
truly a throwaway program.

•Test cases represent a valuable investment.

•If test cases are not saved then whenever the program has to be tested
again the test cases must be reinvented.

•People tend to avoid Re invention. Retest of the program is rarely as

rigorous as the original test.

• Saving test cases and running them again after changes to other
components of the program is known as regression testing.

36
 Software Testing : Principle 8
Principle 8: Do not plan a testing effort under the tacit
assumption that no errors will be found.

•This is a mistake project managers often make and is a sign of the

use of the incorrect definition of testing—that is, the assumption that
testing is the process of showing that the program functions correctly.

•Once again, the definition of testing is the process of executing a

program with the intent of finding errors.

37
 Software Testing : Principle 9
Principle 9: The probability of the existence of more errors
in a section of a program is proportional to the number
of errors already found in that section.

•Errors tend to come in clusters and that, in the typical program, some
sections seem to be much more prone to errors than other sections.

•If a particular section of a program seems to be much more prone to

errors than other sections, then this phenomenon tells us that, in terms
of yield on our testing investment, additional testing efforts are best
focused against this error-prone section.

38
 Software Testing : Principle 10
Principle 10:Testing is an extremely creative and intellectually
challenging task.

•It is probably true that the creativity required in testing a large program
exceeds the creativity required in designing that program.

•It is impossible to test a program sufficiently to guarantee the absence

of all errors.

40
 References

1)Paul C. Jorgenson, Software Testing A

Craftsman’s approach, CRC Press, 1997..
2)Software Testing – Yogesh Singh (Cambridge
University Press)
3)Software Testing – Principles and Practices
Naresh Chauhan (Oxford Higher Education)
4)Glenford Myers, “The Art of Software Testing”,
Wiley-India Edition