Chapter 1: Fundamentals of Testing

Hela Lajmi
Assistant Professor
IEEE Membership Developement chair at Tunisia Section
IEEE Intelligent Transportation Systems Chair

[email protected]
References
[1] https://www.cftl.fr/wp-content/uploads/2018/10/CTFL-Syllabus-
2018-FR.pdf
[2] https://www.istqb.org/
[3] http://istqbfoundation.wikidot.com/1#toc0

2
 3
Outline and Learning Objectives

Part 1: What is Testing?

Part 2: Why is Testing Necessary?

Part 3: Seven Testing Principles

Part 4: Test Process

Part 5: The Psychology of Testing

Part 6: Quizz
 4
Overview

Chapter 1: Fundamentals of Testing The tester learns the basic principles related to

testing, the reasons why testing is required, what test objectives are, and the

principles of successful testing. The tester understands the test process, the major

activities, and work products.

Overview

 Chapter 1: Fundamentals of Testing The tester learns the basic

principles related to testing, the reasons why testing is required, what

test objectives are, and the principles of successful testing. The tester

understands the test process, the major activities, and work products.

5
 6
Outline and Learning Objectives

Part 1: What is Testing?

Learning Objectives:
LO-1.1.1 Identify typical objectives of testing (K1)
LO-1.1.2 Differentiate testing from debugging (K2)
 7
Typical Objectives of Testing

 To evaluate work products such as requirements, user stories, design,

and code
 To verify whether all specified requirements have been fulfilled
 To validate whether the test object is complete and works as the
users and other stakeholders expect
 To build confidence in the level of quality of the test object
 To prevent defects
 To find failures and defects
 8
Typical Objectives of Testing

 To provide sufficient information to stakeholders to allow them to

make informed decisions, especially regarding the level of quality of
the test object
 To reduce the level of risk of inadequate software quality (e.g.,
previously undetected failures occurring in operation)
 To comply with contractual, legal, or regulatory requirements or
standards, and/or to verify the test object’s compliance with such
requirements or standards
 9
Testing Vs Debugging

Testing Debugging

Tester Developer

 Testing is the exploration of the

system in order to find defects
Observe failure Investigate and
isolate defect
 Debugging is a process used to
identify the causes of bug in a code
Fix defect and correct them
Re-test to
confirm failure
no longer Check fix works
occurs
 10
Verification Vs Validation

Verification • Are we building the product right

Validation • Are we building the right product
 11
Verification Vs Validation Vs Accreditation
 12

Part 1: What is Testing?

Part 2: Why is Testing Necessary?

Learning Objectives:

LO-1.2.1 Give examples of why testing is necessary (K2)

LO-1.2.2 Describe the relationship between testing and quality assurance and give examples of how testing contributes
to higher quality (K2)
LO-1.2.3 Distinguish between error, defect, and failure (K2)
LO-1.2.4 Distinguish between the root cause of a defect and its effects (K2)
Why is testing necessary? 13

Some of the problems might be trivial, but others can be

costly and damaging with loss of money, time or business
reputation and even may result in injury or death [3]
Why is testing necessary? 14

Why we make mistakes:

• Lack of experience
• Don't have right information
• Misunderstanding
• Careless attitude
• Tiredness
• Under time pressure
• Dealing with complexity
Testing’s Contributions to Success 15
 16
Quality assurance Vs. Quality control (Testing)
 17
Quality control vs quality assurance

Quality Control Quality Assurance

- Focus on Product - Focus on Process

- Find defects - Prevent defects

- Achievement of appropriate levels of quality - Provide confidence that the appropriate levels of quality will be
achieved.
- Test activities are part of the overall software - Quality Audits
development or maintenance
process.
- Testing contributes to the - The use of root cause analysis to detect and remove the causes of
achievement of quality in a variety of ways defects, along with the proper application of the findings of retrospective
meetings to improve processes, are important for effective
quality assurance.
 18
Error - Fault - Failure

A person makes
an error or mistake ...

… that creates a
fault, defect or bug in the
software ...

… that can cause

a failure
in operation
 19
Error - Fault - Failure

 Error: a human action that produces an incorrect result

 Fault: a manifestation of an error in software
• also known as a defect or bug
• if executed, a fault may cause a failure
 Failure: deviation of the software from its expected
delivery or service
(found defect)

Failure is an event; fault is a state of the software, caused by an error

Defects - Root Causes - Effects 20
Defects - Root Causes - Effects 21
Outline and Learning Objectives 22

Part 1: What is Testing?

Part 2: Why is Testing Necessary?
Part 3: Main Testing Principles

Learning Objective:
LO-1.3.1 Explain the seven principles of testing (K2)
 23
Testing principles : Principle 1 – Testing shows presence of defects, not their absence

 Testing can show that defects are present, but cannot prove
that there are no defects.
 Testing reduces the probability of undiscovered defects
remaining in the software but, even if no defects are found,
it is not a proof of correctness.
 24
Testing principles : Principle 2 – Exhaustive testing is impossible (1/5)

 Testing everything (all combinations of inputs and preconditions) is not feasible

except for trivial cases.
 Instead of exhaustive testing, risk analysis and priorities should be used to focus
testing efforts.
 25
Testing principles : Principle 2 – Exhaustive testing is impossible (2/5)

Why not just "test everything"?

Avr. 4 menus
3 options / menu

system has Average: 10 fields / screen

20 screens 2 types input / field
(date as Jan 3 or 3/1)
(number as integer or decimal)
Around 100 possible values
Total for 'exhaustive' testing:
20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
If 1 second per test, 8000 mins, 133 hrs, 17.7 days
(not counting finger trouble, faults or retest)

10 secs = 34 wks, 1 min = 4 yrs, 10 min = 40 yrs

 26
Testing principles : Principle 2 – Exhaustive testing is impossible (3/5)

How much testing is enough?

 it’s never enough
 when you have done what you planned
 when your customer/user is happy
 when you have proved that the system works correctly
 it depends on the risks for your system
 risk of missing important faults
 risk of incurring failure costs
 risk of releasing untested or under-tested software
 risk of losing credibility and market share
 risk of missing a market window
 risk of over-testing, ineffective testing
 27
Testing principles : Principle 2 – Exhaustive testing is impossible (4/5)

Too little time, too much to test ..

 test time will always be limited

 use RISK to determine:

 what to test first (priority)



what to test most
how thoroughly to test each item
what not to test (this time)
} i.e. where to
place emphasis

 use RISK to:

 allocate the time available for testing by prioritising testing ...
 28
Testing principles : Principle 2 – Exhaustive testing is impossible (5/5)

Prioritise tests
so that,
whenever you stop testing,
you have done the best testing
in the time available.
 29
Testing principles : Principle 3 – Early testing

 Testing activities should start as early as possible in the

software or system development life cycle, and should be
focused on defined objectives.
 30
Testing principles : Principle 4 – Defects cluster together

 A small number of modules contain most of the defects

discovered during pre-release testing, or are responsible for
the most operational failures.

 Defect Clustering in Software Testing is based on the Pareto

principle (the 80-20 rule)

 Approximately 80% of the problems are caused by 20% of

the modules.

 Review defects and failures in order to improve processes

 31
Testing principles : Principle 5 – Beware of the pesticide paradox

 If the same tests are repeated over and over again, eventually the same set of test cases
will no longer find any new defects.
 To overcome this “pesticide paradox”, the test cases need to be regularly reviewed and
revised, and new and different tests need to be written to exercise different parts of the
software or system to potentially find more defects.

(just as pesticides are no

longer effective at killing
insects after a while.)
 32
Testing principles : Principle 6 – Testing is context dependent

 Testing is done differently in different contexts. For

example, safety-critical software is tested differently from
an e-commerce site.

 the higher the possibility of losses, the more we need to invest in testing
 33
Testing principles : Principle 7 – Absence-of-errors is a fallacy

 Finding and fixing defects does not help if the system built
is unusable and does not fulfil the users’ needs and
expectations.

 The fact that no defects are outstanding is not a good

reason to ship the software
 34
Outline and Learning Objectives

Part 1: What is Testing?

Part 2: Why is Testing Necessary?
Part 3: Testing Principles
Part 4: Fundamental Test Process

Learning Objectives:
LO-1.4.1 Explain the impact of context on the test process (K2)
LO-1.4.2 Describe the test activities and respective tasks within the test process (K2)
LO-1.4.3 Differentiate the work products that support the test process (K2)
LO-1.4.4 Explain the value of maintaining traceability between the test basis and the test work products (K2)
Test Process in Context 35

Contextual factors that could influence the test process for an organization:

 Software development lifecycle model and project methodologies being used

 Test levels and test types being considered

 Product and project risks

 Business domain
Test Process in Context 36

 Operational constraints, including but not limited to:

 Budgets and resources

 Timescales

 Complexity

 Contractual and regulatory requirements

 Organizational policies and practices

 Required internal and external standards

 …
 3734
Fundamental Test Process
 38
Fundamental Test Process
 39
Test planning and control

 Test planning is the activity of verifying the mission of testing, defining the
objectives of testing and the specification of test activities in order to meet
the objectives and mission.

 It involves taking necessary actions to meet the mission and objectives of

the project. In order to control testing, it should be monitored throughout the
project. Test planning takes into account the feedback from monitoring and
control activities.
 40
Test analysis and design

Test analysis and design is the activity where general testing objectives are transformed

into tangible test conditions and test cases.

Test analysis and design has the following major tasks:

 Reviewing the test basis (such as requirements, architecture, design, interfaces).

 Evaluating testability of the test basis and test objects.

 41
Test analysis and design

 Identifying and prioritizing test conditions based on analysis of test items, the

specification, behaviour and structure.

 Designing and prioritizing test cases.

 Identifying necessary test data to support the test conditions and test cases.

 Designing the test environment set-up and identifying any

required infrastructure and tools.

 42
Test implementation and execution (1/2)

 Developing, implementing and prioritizing test cases.

 Developing and prioritizing test procedures, creating test data and, optionally, preparing
test harnesses and writing automated test scripts.

 Creating test suites from the test procedures for efficient test execution.

 Verifying that the test environment has been set up correctly.

 Executing test procedures either manually or by using test execution tools, according to the
planned sequence.
 43
Test implementation and execution (2/2)

 Logging the outcome of test execution and recording the identities

and versions of the software under test, test tools and testware.

 Comparing actual results with expected results.

 Reporting discrepancies as incidents and analyzing them in order to establish their cause
(e.g. a defect in the code, in specified test data, in the test document, or a mistake in the
way the test was executed).
 44
Test implementation and execution (2/2)

Repeating test activities as a result of action taken for each discrepancy. For example,
reexecution of a test that previously failed in order to confirm a fix (confirmation testing),
execution of a corrected test and/or execution of tests in order to ensure that defects have
not been introduced in unchanged areas of the software or that defect fixing did not
uncover other defects (regression testing).
 45
Evaluating exit criteria and reporting

 Checking test logs against the exit criteria specified in test planning.

 Assessing if more tests are needed or if the exit criteria specified should be
changed.

 Writing a test summary report for stakeholders.

 46
Test closure activities

 Checking which planned deliverables have been delivered, the closure of incident
reports or raising of change records for any that remain open, and the documentation
of the acceptance of the system.

 Finalizing and archiving testware, the test environment and the test infrastructure for
later reuse.

 Handover of testware to the maintenance organization.

 47
Fundamental Test Process
 48
Costs according to Process Test activity [3]
 49
What is traceability in Software testing?

 Test conditions should be able to be linked back to their sources in the test basis,
this is known as traceability.

 Traceability can be horizontal through all the test documentation for a given test
level (e.g. system testing, from test conditions through test cases to test scripts)
or it can be vertical through the layers of development documentation (e.g.
from requirements to components).
 50
What does good traceability support ?

 Analyzing the impact of changes

 Making testing auditable
 Meeting IT governance criteria
 Improving the understandability of test progress reports and test summary
reports to include the status of elements of the test basis (e.g., requirements
that passed their tests, requirements that failed their tests, and requirements
that have pending tests)
 51
What does good traceability support ?

 Relating the technical aspects of testing to stakeholders in terms that they can
understand
 Providing information to assess product quality, process capability, and project
progress against business goals
 52
Outline and Learning Objectives

Part 1: What is Testing?

Part 2: Why is Testing Necessary?
Part 3: Testing Principles
Part 4: Fundamental Test Process
Part 5: The Psychology of Testing

Learning Objectives:
LO-1.5.1 Identify the psychological factors that influence the success of testing (K1)
LO-1.5.2 Explain the difference between the mindset required for test activities and the mindset required
for development activities (K2)
The Psychology of Testing 53

Testers and test managers need to have good interpersonal skills to be able to communicate effectively about
defects, failures, test results, test progress, and risks, and to build positive relationships with colleagues.

Ways to communicate :
 Start with collaboration rather than battles. Remind everyone of the common goal of better-quality systems.
 Emphasize the benefits of testing. For example, for the authors, defect information can help them
improve their work products and their skills. For the organization, defects found and fixed during testing will
save time and money and reduce overall risk to product quality.
 Communicate test results and other findings in a neutral, fact-focused way without criticizing the
person who created the defective item. Write objective and factual defect reports and review findings.
 Try to understand how the other person feels and the reasons they may react negatively to the information.
 Confirm that the other person has understood what has been said and vice versa.
 54
Outline and Learning Objectives

Part 1: What is Testing?

Part 2: Why is Testing Necessary?
Part 3: Testing Principles
Part 4: Fundamental Test Process
Part 5: The Psychology of Testing

Quizz
 55
Q1 : Keywords Chapter 1

Which one of the following is the BEST description of a test condition?

a) An attribute of a component or system specified or implied by requirements

documentation.
b) An aspect of the test basis that is relevant to achieve specific test objectives.
c) The capability of the software product to provide functions which meet stated and
implied needs when the software is used under specified conditions.
d) The percentage of all single condition outcomes that independently affect a
decision outcome that have been exercised by a test case suite.
 56
Q2 : Identify typical objectives of testing

Which of the following statements is a valid objective for testing?

a) To determine whether enough component tests were executed

within system testing.
b) To find as many failures as possible so that defects can be
identified and corrected.
c) To prove that all possible defects are identified.
d) To prove that any remaining defects will not cause any failures.
 57
Q3 : Differentiate testing from debugging

Which of the following statements correctly describes the difference

between testing and debugging?

a) Testing identifies the source of defects; debugging analyzes the

defects and proposes prevention activities.
b) Testing shows failures caused by defects; debugging finds,
analyzes, and removes the causes of failures in the software.
c) Testing removes faults; debugging identifies the causes of
failures.
d) Testing prevents the causes of failures; debugging removes the
failures.
 58
Q4 : Distinguish between error, defect and failure

Which one of the statements below describes a failure discovered during testing or in
production?

a) The product crashed when the user selected an option in a dialog box.
b) The wrong version of one source code file was included in the build.
c) The computation algorithm used the wrong input variables.
d) The developer misinterpreted the requirement for the algorithm.

Select one option.

 59
Q5 : Explain the seven testing principles

Which of the following statements CORRECTLY describes one of the seven key
principles of software testing?

a) By using automated testing it is possible to test everything.

b) With sufficient effort and tool support, exhaustive testing is feasible for all
software.
c) It is impossible to test all input and precondition combinations in a system.
d) The purpose of testing is to prove the absence of defects.

Select one option.

 60
Q6 : Describe the relationship between testing and quality assurance

In what way can testing be part of Quality assurance?

a) It ensures that requirements are detailed enough.

b) It reduces the level of risk to the quality of the system.
c) It ensures that standards in the organization are followed.
d) It measures the quality of software in terms of number of executed test cases.

Select one option.

 61
Q7 : Describe the test activities and respective tasks within the test process

Which of the below tasks is performed during the test analysis activity of the test
process?

a) Identifying any required infrastructure and tools.

b) Creating test suites from test scripts.
c) Analyzing lessons learned for process improvement.
d) Evaluating the test basis for testability.

Select one option.

 62
Q8 : Differentiate the following test work products

Differentiate the following test work products,1-4, by mapping them to the right
description, AD.

1. Test suite. A. A group of test scripts or test execution schedule.

2. Test case. B. A set of instructions for the automated execution of test procedures.
3. Test script. C. Contains expected results.
4. Test charter. D. An event that could be verified.

a) 1A, 2C, 3B, 4D.

b) 1D, 2B, 3A, 4C.
c) 1A, 2C, 3D, 4B.
d) 1D, 2C, 3B, 4A.
 63
Differentiate the following test work products ++

 Test suite : Syllabus 1.4.3 for Test implementation: Test implementation work products
also include test suites, which are groups of test scripts, as well as a test execution
schedule. (1A).

 Test case : A set of input values, execution preconditions, expected results and execution
postconditions…. (2C).

 Test script : Glossary test script, A set of instructions for the automated execution of test
procedures (3B).

 Test charter : A statement of test objectives, and possibly test ideas

about how to test. Test charters are used in exploratory testing. (4D).
 64
Differentiate the following test work products ++

 Test strategy : Test approach

 Test procedure : Document specifying a sequence of actions for the execution of a test.
Also known as test script or manual test script

 Test basis : Requirements in which the TCs are based

 Test specification : Test design specification

 Test data : Content of test specification

 Testware : Documentations / scripts / inputs / expected results / environnement.

 65
Differentiate the following test work products ++

 Test policy : high level document that describes the principles / approch and major
objectives.

 Test plan : A document describing the scope / approach / ressources and schedule of test
activities. It identifies the test items / the features to be tested / test tasks and
responsabilities / tester independence level / test environnement / test design
techniques / entry and exit criteria

 Test process : Test planning and control  …  Test closure activities

 66
Differentiate the following test work products ++

 Test Harness : Test harness execute tests, by using a test library and generates a report. It
requires that your test scripts are designed to handle different test scenarios and test
data. It provides stubs and drivers which are small programs that interact with the
software under test.