Software Testing and Quality Assurance

Meti Dejene
[email protected]
Haramaya University
Lecture 3 – Basics of Software Testing
 Introduction

From time to time Software is becoming complex and the demand for quality in
software products has increased.

This rise in customer awareness for quality increases the workload and responsibility of
the software development team and calls for implementation of software quality
assurance activities.

That’s when software testing techniques enter the picture since testing is the critical
element of quality.
 What is Testing?
Many practitioners and researchers have defined software testing in their own way.

❖“Testing is the process of executing a program with the intent of finding errors.” Myers

❖“Software testing is an empirical investigation conducted to provide stakeholders with information about
the quality of the product or service under test, with respect to the context in which it is intended to
operate.” Cem Kaner

❖“Testing is a concurrent lifecycle process of engineering, using and maintaining test ware (i.e. testing
artifacts) in order to measure and improve the quality of the software being tested.” Craig

❖“Software testing is a process that detects important bugs with the objective of having better quality
software.” Naresh Chauhan
 Cont.

According to ANSI/IEEE 1059 standard, Testing can be defined as

“ A process of analyzing a software item to detect the differences between existing

and required conditions (that is defects/errors/bugs) and to evaluate the features of
the software item. ”
 Software Testing Myths (misconceptions) and Facts
1. Myth - Testing starts after program development and it is a single phase in SDLC.
Truth - In reality, testing starts as soon as we get the requirement specifications for
the software and the testing work continues throughout the SDLC, even post-
implementation of the software.
2. Myth - Testing is easy and don't need technical skills.
Truth - Testers’ job is not easy. Modern testing often requires understanding the
code, using automation tools, participating in the development process, and
planning and executing effective testing.
3. Myth - Software development is worth more than testing.
Truth - Testing has now become an established path for career
 Cont.
4. Myth - Complete testing is possible.
Truth - In reality, it is not possible to provide all the possible inputs to test the
software, as the input domain of even a small program is too large to test.
5. Myth - The purpose of testing is to check the functionality of the software.
Truth - Ultimately, the goal of testing is also to ensure quality of the software. But
quality does not imply checking only the functionalities. It is beyond that.
6. Myth - Anyone can be a tester.
Truth - As an established process and career, software testing needs professional
knowledge regarding the testing lifecycles, techniques, tools and many more.
 Goals of Software Testing
The goals of software testing may be classified into three major categories.
 Cont.
➢ Error discovery - Testing aims to find/uncover errors and inconsistencies in the
software.
➢Bug prevention - From the behavior and interpretation of bugs discovered, the
software development team gets to learn how to code safely such that the bugs
discovered should not be repeated in later stages or future projects.
➢ Verification of requirements ensure that the software behaves as expected
according to the requirements defined by stakeholders.
➢Quality – In the long run a thorough testing and a rigorous reliability analysis ensures
high quality.
➢ Customer satisfaction - A complete testing process achieves reliability, reliability
enhances the quality, and quality in turn, increases the customer satisfaction.
 Cont.
➢ Risk management - Software testing may act as a control, which can help in
eliminating or minimizing risks. Testing as a control provide information regarding risk
factors to management so that they can better react to risk situations.

➢ Reduced maintenance cost - Post-release errors are costlier to fix. Thus, if testing
has been done rigorously and effectively during production, then the chances of failure
are minimized during operation and in turn, the maintenance cost is reduced.

➢ Improved software testing process - bug history and post-implementation results

in the present testing process help to improve the testing process for future projects.
 Exhaustive Vs Effective Software Testing
I. Exhaustive (Complete) Software Testing
➢ involves testing every statement in the program and every possible path combination
with every possible input combination and scenario for a given software system.
➢It attempts to test every aspect of the software and aims to explore all possible paths,
conditions, and states the software might encounter.
➢ But exhaustive (complete) testing is rarely feasible (often impractical or impossible).
 Cont.
Why is exhaustive testing often impractical or impossible?
1. The Domain of Possible Inputs to the Software is too Large to Test
➢ Especially for complex systems, the number of permutations for every possible
input combination can be vast (sometimes infinite) and the sheer volume of potential
test cases makes it impossible to execute all within a reasonable timeframe.

2. There are too Many Possible Paths Through the Program to Test

➢ From the start of a program to its termination the number of logic paths through a
program is too large.

➢ Exercising all these paths requires a significant amount of time and resources
because large number of possible test cases are needed to exercise all these paths.
 Cont.

When to Use Exhaustive testing?

➢ Exhaustive testing is typically used only in small, critical systems where complete
coverage is absolutely necessary, such as in safety-critical software (e.g., medical
devices, aerospace systems).
 Cont.
II. Effective Software Testing

➢ Suggests testing must be conducted on selected subsets that can be performed

within the constrained resources.

➢ Focuses on identifying the most critical areas to test, using a well-defined strategy
to maximize coverage and defect detection within practical constraints.

➢ It provides the flexibility to select only the subsets of the domain of tests based on
project priority and on the factors which are required in a particular environment.

➢ Aims to balance thoroughness with efficiency, optimizing the use of time and
resources.
 Basic Terminologies and Concepts in Software Testing
A. Test case

➢ Test case is a well-documented procedure designed to determine if a software

application behaves as expected.

➢ It is a set of inputs and corresponding expected results that exercise a component

with the purpose of causing failures, detecting faults and finding errors in the system.

➢ A test case has an identifier and is associated with a program behavior.

➢After execution, observed results are compared with expected outputs mentioned in
the test case.

➢A collection of test cases intended to test a software program are collectively called
Test Suite.
 Cont.
B. Test script: A set of instructions (code scripts) executed by automated testing tools.
C. Test ware
➢ Test ware are the documents that a test engineer produces during testing activities.
➢ It may include test plans, test specifications, test case design, test reports, etc.
D. Test oracle
➢ An oracle is the means to judge the success or failure of a test.
➢ It is a mechanism that determines the correctness of the system for some test (i.e.
whether the software executed correctly for a test case or not).
➢ The simplest oracle is comparing actual results with expected results.
 Testing Principles
1. Effective testing, not exhaustive testing.

2. Testing is not a single phase performed in SDLC.

3. Early testing is the best policy.

➢ The cost of bugs can be reduced tenfold by starting testing early, as bugs are
harder to detect in later stages if they go undetected.

4. Absence-of-Errors Fallacy: Just because a software system is free of defects doesn't

mean it is useful or meets the user's needs.

➢ Testing should ensure that the software not only works correctly but also fulfills
its intended purpose.
 Cont.
5. Probability of existence of an error in a section of a program is proportional to the
number of errors already previously found in that section.
➢ If some sections are found error-prone in testing, then our next testing effort
should give priority to these error-prone sections.
6. Testing strategy should start at the smallest module level and expand towards the
whole program.
➢ This principle supports the idea of incremental testing which states “Testing must
begin at the unit or module level, gradually progressing towards integrated modules
and finally the whole system”.
7. Testers must participate in specification and design reviews.
 Cont.
8. Testing should also be performed by an independent team.
➢ Since developers are mostly biased towards testing their own work, it is always
recommended to have the software tested by an independent testing team.
9. Everything must be recorded in software testing.
➢ The record of every test case run and the bugs reported, the inputs provided
during testing and the corresponding outputs must be recorded as a lesson for other
projects.
10. Invalid inputs and unexpected behavior have a high probability of finding an error.
➢ Whenever a software is tested, we test for valid inputs and for the functionality
that the software is supposed to do. But we must also test the software with invalid
inputs and the behavior which is not expected in general.
 Software Testing as a Process

With quality as the major driving force, Software testing has emerged as a complete
process in software engineering.

➢ This means, Testing is not just a phase in SDLC performed after coding, rather

software testing is a process which runs parallel with other SDLC activities.

Thus, we can say testing is a dual-purpose process, as it is used

➢ (1) to detect errors as well as

➢(2) to establish confidence in the quality of software.

 Cont.
An organization, to ensure better quality software, must adopt a testing process and
consider the following points:
➢ Testing process should be organized such that there is enough time for
important and critical features of the software.
➢ Testing techniques should be adopted such that these techniques detect
maximum errors.
➢ The testing process should be driven by quantified quality goals. In this way, the
process can be monitored and measured.
➢ Testing procedures and steps must be defined and documented.
 Software Testing Life Cycle (STLC)
Since we have recognized software testing as a process, like SDLC, there is need
for a well-defined series of steps to ensure successful and effective software testing.
Why ?

➢ Because systematic execution of each step will result in saving time and effort.

This testing process divided into a well-defined sequence of steps is termed as

Software Testing Life Cycle (STLC).
STLC consists of the following phases
 1. Test Planning
Objective: Develop a comprehensive test strategy and plan.

Activities:

❖ Define the scope of testing.

❖ Identify the testing approach, testing techniques and tools.

❖ Estimate testing effort, resources required, their duration, and cost.

❖ Identifying areas of risks.

❖ Defining the test entry and exit criteria.

❖ Define roles, responsibilities and reporting procedures within the testing team.

The major output of test planning is the test plan document.

 2. Test Design
Objective: Design and create detailed test cases and test scripts.

Activities:

➢ Determine the specific conditions and scenarios that need to be tested.

➢ Write test cases with specific inputs, execution steps, and expected results for each
test condition.

➢ Organize test cases into test suite based on test objectives, priority, or other criteria.

➢ Review and validate test cases to ensure coverage of requirements.

➢ Setting up the test environment and supporting tools

➢ Create automated test scripts (if applicable).

 Cont.
➢ Creating test procedure specification:
➢ This is a description of how the test case will be run.
➢ It is written in the form of sequenced steps to be actually used by the tester
at the time of execution of test cases.
All the details specified in the test design phase are documented in the test design
specification.
Major Deliverables of this phase: Test cases, test scripts, test data.
 3. Test Execution

In this phase, all test cases are executed.

It is the decision of the test team to opt for automation or manual execution.

Log defects for any deviations from expected results.

Test results are documented in the test incident reports, test logs, testing status, and
test execution report.
 4. Post-Execution/Test Review

This phase is to get feedback and analyze error-related issues so that maximum
number of errors can be removed.
After successful test execution, errors will be reported to the concerned developers.
➢ As soon as the developers get the error report, they analyze the error reported and
starts the debugging process.
➢ After fixing the bug/error, the developer reports to the testing team and the
modified portion of the software is tested once again.
In the end, the final error report and associated metrics are reviewed and analyzed to
establish whether the software meets the predefined functionality, quality and
reliability goals or not.
 5. Test Cycle Closure

Objective: Conclude testing activities and evaluate the results.

Activities:
➢ Ensure all planned test cases have been executed.
➢ Assess the quality of the software based on test results and defect density.
➢ Document lessons learned and best practices.

➢ Archive test cases, test scripts, and other artifacts for future reference.

Deliverables: Test summary and closure report.

 Software Testing Strategy

A strategy for software testing integrates software test case design and execution
methods into a well-planned series of testing steps that result in the successful
construction of software.

This test strategy provides a road map on what type of test to conduct and what
testing techniques and methods to implement.
 Cont.
The rule for development of a test strategy is that
1. Testing ‘begins from the smallest unit and progresses to enlarge’.
➢ This means the testing strategy should start at the component level and
works "outward" toward the integration of the entire system.
2. Different testing techniques are appropriate at different points in time.
➢ A strategy for software testing must accommodate low-level tests that are
necessary to verify that a small source code segment has been correctly
implemented as well as high-level tests that validate major system functions
against customer requirements.