Software Testing

Basics
LIU Weihong
School of computer science and technology
Anhui university of technology
[email protected]
 Outlines
• Basic concepts
• General testing principles
• A software tester’s jobs
• Test automation
• Test approaches
• Bug reporting
 Basic Concepts
• Software
• Software testing
• Bug
• Test Case
Basic concepts-software

Software=program ？

Software=program+document

So, software testing != program

testing
Basic concepts-software testing
There are many definitions of software testing.
Some people think: Software Testing is the
process of executing a program or system
with the intent of finding errors.

Testing is the process of evaluating a system or

its component(s) with the intent to find whether
it satisfies the specified requirements or not.
 Basic Questions on Testing
Why to test?
 testing becomes absolutely essential to make sure the software works
properly and does the work that it is meant to perform.

What to test?
 Any working product which forms part of the software application
has to be tested. Both documents and programs must be tested.

When to test?
 as early as possible

How often to test?

 When a program (source code) is modified or newly developed, it
has to be tested.

Who tests?
 Programmer, Tester and Customer
 Testing Vs Debugging
 Testing is focused on identifying the
problems in the product
 Done by Tester
 Need not know the source code sometimes

 Debugging is to make sure that the bugs are

removed or fixed
 Done by Developer
 Need to know the source Code
Basic concepts-bug
Why is the error often called bug in computer field?
• In 1947 Harvard University was
operating a room-sized computer
called the Mark II.
• A moth flew into the computer and
was zapped by the high voltage
when it landed on a relay.
• Hence, the first computer bug!
 Bugs a.k.a. …
• Defect • Failure
• Fault • Inconsistency
• Problem • Product
• Error Anomaly
• Incident • Product
• Incidence
Anomaly
• Feature :-)
• Variance
Basic concepts-bug(Cont’ d)
• Product Specification defines the product what it will
be, how it will act, what it will do, and what it won't
do”
• Formally, we say that a bug occurs when one or
more of the following four rules is true: when the
software
– doesn't do something that the product specification says it
should do.
– does something that the product specification says it
shouldn't do.
– does something that the product specification doesn't
mention.
– doesn't do something that the product specification
doesn't mention but should. These include: is difficult to
understand, hard to use, slow, and so on.
Why do Bugs Occur?

Most bugs are not because of

mistakes in the code :
Specification (~= 55%)
Design (~= 25%)
Code (~= 15%)
Other (~= 5%)
 Why does testing need to
begin as early as possible?

The later the bug is discovered, the higher the cost

of fault correction will be.
Basic concepts-test case
• A test case contains at least:
– A sequence of Steps describing actions to be performed,
– Test data to be used
– An expected response for each action performed.
• Test Cases are written based on Business and
Functional/Technical requirements, use cases and
Technical design documents.
• The level of details specified in test cases will vary
depending on Organizations, Projects and also on the
Test Case Template used OR on the Test Management
tool being used in the project.
• Test cases must be repeatable.
Good test cases are data-specific, and describe each
interaction necessary to repeat the test exactly.
 How detailed should be a test
case?
• Test Case steps should be as detailed as
possible and should not be written at high level.
Test Cases – Good Example

Each test case should contain at least test data, test steps and expected
results.
 General Testing Principles
1. All tests should be traced back to user
requirements.
2. Testing shows the presence of Defects
( it is not a ghost. It is a fact )
• We test to find Faults (as known as Defects)
• As we find more defects, the probability of
undiscovered defects remaining in a system
reduces ( decreasing nature).
• However Testing cannot prove that there are
no defects present
 General Testing Principles
3.Exhaustive Testing is Impossible!

• We have learned that we cannot test

everything (i.e. all combinations of
inputs and pre-conditions).

• So we must Prioritise our testing effort

using the test approaches.
 Why don’t we test
everything ?
System has 20 screens
Average 4 menus / screen
Average 3 options / menu
Average of 10 fields / screen
2 types of input per field
Around 100 possible values

Approximate total for exhaustive testing

20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
Test length = 1 sec then test duration = 17.7 days
Test length = 10 sec then test duration = 34 weeks
Test length = 1 min then test duration = 4 years
Test length = 10 mins then test duration = 40 years!

It is not a matter of time. But,time is money ( salary is taken by hour. So

second is valuable for software houses)
 General Testing Principles
4.Early testing
• Testing activities should start as
early as possible in the
development life cycle

• These activities should be

focused on defined objectives –
outlined in the Test Strategy

• Remember from our Definition of

Testing, that Testing doesn’t start
once the code has been written!
 General Testing Principles
5.Defect Clustering
• Defects are not evenly distributed in a system
• They are ‘clustered’
• In other words, most defects found during testing are
usually confined to a small number of modules ( 80% of
uncovered errors focused in 20% modules of the whole
application) “Pareto law”

• Similarly, most operational failures of a system are usually

confined to a small number of modules
• An important consideration in test prioritisation!
 General Testing Principles
6.Avoid testing your own programs.

5.programmers will not easily admit that they have

written errors.
6. programmers have limitations in testing ideas.
When testing, they are easily influenced by
programming ideas.
7.programmer testing is not typical.
P.S: Unit testing generally requires programmers to
do by themselves.
 General Testing Principles
7.Testing is Context (background)Dependent

• Testing is done differently in different contexts

• For example, safety-critical software is tested differently
from an e-commerce site
• Whilst, Testing can be 50% of development costs, in
NASA's Apollo program (it was 80% testing)
• 3 to 10 failures per thousand lines of code (KLOC)
typical for commercial software
• 1 to 3 failures per KLOC typical for industrial software
• 0.01 failures per KLOC for NASA Shuttle code!
• Also different industries impose different testing
standards
 What does a Software Tester
Do?
• The goal of a software tester is to
– find bugs
– find bugs and find them as early as
possible
– find bugs, find them as early as possible,
and make sure they get fixed
The software testing process

PRD: Product Requirements;

SPEC: Specification
CC: Code Completion
CF: Code Freeze
ER: Engineering Release
Four core tasks for testing engineers
Test Plan -Mgr, leader
Test case -every one
Bug--------Every one
ER report---Mgr,leader
 Main skills a qualified software
tester need to have
Test requirement analysis
capability
Test case design capability
Test case execution capability
Defect description ability
 Traits of Good Testers
• A Tester needs:

– good communication skills

– good observation skills
– people handling skills
– Curiosity ( testing activities interest )
– patience
– reliability
– Thoroughness ( carefulness)
– attention to detail
– creativity in terms of identifying likely faults
– Experience

• As with most other disciplines an effective test team will

need a mix of skills ， so it is difficult to generalise
 Test Automation
Test automation is a practice in which testers
employ a software tool to reduce or eliminate
repetitive tasks.
Testers either write scripts or use record-and-
playback to capture user interactions with the
software being tested.
This can save the testers a lot of time if many
iterations of testing will be required.
It costs a lot to develop and maintain automated
test suites, so it is generally not worth developing
them for tests that will executed only a few times.
 What to Automate ?

Simple Things Hard Things

Done Once Do it Manually Automate It

Done often Automate It Buy or Write

Software

1. Manual testing is impossible for some jobs.

2. Automated testing is no substitute for manual
testing.
Advantages & Disadvantages
of Automated Testing
• Advantages • Disadvantages

• Reliable: eliminating human • High investment is needed

error in the tools and training

• Reusable • High man power

• Better Quality Software requirement for test

preparations
• Fast
• A lot of testing areas left
• Cost Reduction
uncovered
 Test tools
• Different testing stages have different
tools.
• Different testing methods have different
tools.
 Test approaches
Static testing
Execute program?
Dynamic testing
Soft Black-box testing
ware Test techniques White-box testing
Grey-box testing
testi
Unit testing Re
ng gre
ssi
Integration Testing
Testing levels on

System tesing Tes

ting
Acceptance Testing
 Static Testing
• Instead of running the program under test, it is a process
of statically checking program code, interfaces, or
documents for possible errors. For code testing, the
main job is to check if the code is in line with the
corresponding standards and specifications.

Static testing finds bugs before you run the program.

 Example
#include <stdio.h>
max(float x,float y)
No code annotation
{float z;
Precision loss
z=x>y?x:y; Use void if no results to be
return z;} returned
main() Appropriate blank lines
{ ……
float a,b;
int c;
scanf(“%f,%f”,&a,&b);
c=max(a,b);
printf(“Max is %d\n”,c);
}
 dynamic Testing
• Run the program to check if there are
errors
 Black-box Testing
Input


Software
output

 It focuses on the external behavior of the software .

Also known as functional –based
testing ， or specification-based testing.
 White-box Testing


Also known as Structure-based testing, or
program-based testing
 Grey Box Testing

input

software +
output

 The combination of black-box testing and white-box testing.

 Unit Testing
 Programs will be tested at unit level. It
focus on an individual module .
A function or a method in a class is a unit.
 The same developer will do the test
Unit test environment
 The unit itself can not run by itself. It
depends on other parts to call and drive.
driver
-- It can drive the unit under test to run. Its jobs
include: receiving the test data, transmitting the data to
the unit under test, and outputting the results.
stub
--It is used to replace the other units that are called by
the unit under test.
Unit test environment

Test Driver results

cases
Module
under
test

Stub 1 Stub 2 Stub 3

 Example ： driver and stub
(1)Only function fun1, how to test?
void main(void)
{
int fun1(int x,int y)
{ Simulate int a,b;
return x+y; driver scanf(“%d%d”,&a,&b);
}
printf(“%d\n”,fun1(a,b));
}
(2)Only function main, How to
test?
#include <stdio.h>
void main(void) int fun1(int a,int b)
Simulate stub
{
{ return a+b;
int a=1,b=2,c; }

c=fun1(a,b);
}
Exercise ： Write the driver for the
function divide()
int divide(int a,int b)
{
int c;
if(b==0)
{
printf(“divisor can not be
0”);
return 0;
}
c=a/b;
return c;
}
 Integration Testing
• scope = set of interacting components
– Focus: correctness of component interactions
– Mostly black-box, some white-box techniques
 System Testing
• scope = entire system
– Focus: overall system correctness
– Only black-box techniques
 Acceptance testing
Identify all possible issues/bugs before
releasing the product to users or public.
• Alpha testing
Alpha testing is carried out in a lab environment and usually the testers are
internal employees of the organization. To put it as simple as possible, this
kind of testing is called alpha only because it is done early on, near the end
of the development of the software, and before beta testing.

• Beta testing
Beta Testing of a product is performed by "real users" of the software
application in a "real environment" and can be considered as a form of
external User Acceptance Testing.
 Bug reporting
• Testers report bugs to programmers
• If the report is not clear and understandable, the
bug will not get fixed
• To write a fully effective report you must:
– Explain how to reproduce the problem
– Analyze the error so that it can be described with a
minimum number of steps
– Write a report that is complete and easy to understand
 Bug Reports
• A bug report is a tool that you use to sell the
programmer on the idea of spending his/her time and
energy to fix a bug.
• Bug reports are your primary work product as a
tester. This is what people outside of the testing
group will most notice and most remember of your
work.
• The best tester isn’t the one who finds the most bugs
or who embarrasses the most programmers. The
best tester is the one who gets the most bugs fixed.
• One defect per report: No more, no less. A single
bug in a report can help to avoid duplication and
confusion. If you describe too many defects some of
them may be overlooked.
 What is a good bug report?
• Reproducible – if reproduce the bug isn't possible, then
it won't be fixed.
• Specific and Informative - do not spread problem in the
whole essay. Strive to summarize the problem in
minimum words yet in an effective way. And in any case,
don't combine multiple problems even if they seem to be
similar. Write different reports for each problem.

So the goals of a bug report are to:

explain a bug to the developer and show where it is.
help the developer fix it with the minimal time cost.
 Content of bug report?
Title: In title should be a short explanation of what the issue is. A
good title should have information about product name or error
message or steps you were doing when it failed.
For example:
How not to do: “Crash”, “Looking an error”, “Bug”
How to do: “Error 5C79 when confirming request”

Product: Specify the product name and version of which is operated.

How not to do: “Your application”
How to do: “Agfa”
 Content of bug report?
Classification: Very important to indicate is this a feature request, minor
bug or horrible bug that crash the whole application. Most of all tools ha
ve a set of these to choose from, but if they don’t, use one of these:
“New feature”, “Minor Annoyance”, “Crashing Bug”,
“Serious Bug” or “Minor Bug” for those you can work around.
How not to do: "Leave it blank"
How to do: “Serious Bug”, “Annoyance”, “Feature Request”

Platform: Will be a big plus to determine what you are using to run
the software, particularly operating system name and version, in the
case of Web Application, should tell web browser name and version.
There are a lot of different versions of systems and browsers, which
important to developers to know.
How not to do: “Windows”
How to do: “Windows 7, Internet Explorer 9”
 Content of bug report?
Summary: Try to briefly describe what you were trying to do before you
find the bug and how to react to your application. Try to use simple
natural language and write all simple sentences.
How not to do: “Application don't work”
How to do: “Clicked on ‘File / Save As…’ and the ‘Save’ dialog came
up →Clicked on the ‘OK’ button but the file did not save.”

Steps to reproduce: It’ll be great if you able to reproduce that

bug again. Because of reproducible bugs easier to fix. You
should describe how to reproduce that bug step by step, as detailed as p
ossible. Don’t forget, you must be specific.
How not to do: “
Homepage >Left Button>Click on it
Shop>try to buy smth>you can't"
How to do: “
Go to Settings > Profile (this would take user to new screen)
Tap on More Options > Delete Account”
 Content of bug report?
Expected Result: What should happen when you make an action?
How not to do: “I tried to print, but it didn't work”
How to do: “From the 'Shop' screen, click on the ‘Print’ button”

Actual Result: Here’s the result of the bug. As usual, testers are a
little bit indefinite in this part of the bug report. So, try to be concise.
How not to do: “Button does not work as expected”
How to do: “Button closes app”

Priority: It's showing to the developer how soon it has to be fixed.

Priority is generally set from 1 to 5. The lower the number, the
higher priority. 1 score - must be fixed as soon as possible. 5 score -
can fix when time permits.
 Content of bug report?
Severity:
This describes the impact of the bug.

Types of Severity:
Blocker: No further testing work can be done.
Critical: Application crash, Loss of data.
Major: Major loss of function.
Minor: Minor loss of function.
Trivial: Some UI enhancements.
Enhancement: Request for a new feature or some enhancement in
existing one.
 Content of bug report?
Status: It's showing the status of a bug report in any bug tracking
system. At the beginning status of bug report will be "New". After that
status can changing like Fixed, Verified, Reopen, Won’t Fix and etc.

Attachments: If you can take a screenshot - do it. It'll be a huge plus

for developer see what you was seen before the bug and after. Just
attach few screenshots for the bug report.
Here is an example of bug tracking system - EasyQA, which has all the
necessary fields to write bug report.