Foundations of Software Testing - IsTQB Certification
Foundations of Software Testing - IsTQB Certification
TESTING
ISTQB CERTIFICATION
Dorothy Graham
Isabel Evans
Rex Black
CONTENTS
Acknowledgements viii Preface ix
1 Fundamentals of testing 1
1.1 Why is testing necessary? 1
1.2 What is testing? 11
1.3 Testing principles 18
1.4 Fundamental test process 20
1.5 The psychology of testing 26
Chapter review 31
Sample exam questions 32
Exercise: Test psychology 33
Exercise solution 34
3 Static techniques 57
3.1 Reviews and the test process 57
3.2 Review process 59
3.3 Static analysis by tools 69
Chapter review 74
Sample exam questions 75
Glossary 209
Answers to sample exam questions 227
References 231
Authors 237
Companies 239
Index 243
CHAPTER 1
Fundamentals of testing
Ibehind
n this chapter, we will introduce you to the fundamentals of testing: why
testing is needed; its limitations, objectives and purpose; the principles
testing; the process that testers follow; and some of the psychological
factors that testers must consider in their work. By reading this chapter you'll
gain an understanding of the fundamentals of testing and be able to describe
those fundamentals.
1 Describe, with examples, the way in which a defect in software can cause
harm to a person, to the environment or to a company. (K2)
2 Distinguish between the root cause of a defect and its effects. (K2)
3 Give reasons why testing is necessary by giving examples. (K2)
4 Describe why testing is part of quality assurance and give examples of
how testing contributes to higher quality. (K2)
5 Recall the terms 'mistake', 'defect', 'fault', 'failure' and the correspon
ding terms 'error' and 'bug'. (Kl)
6 Explain the fundamental principles in testing. (K2)
1.1.1 Introduction
In this section, we're going to kick off the book with a discussion on why testing
matters. We'll describe and illustrate how software defects or bugs can cause
problems for people, the environment or a company. We'll draw important dis-
tinctions between defects, their root causes and their effects. We'll explain why
testing is necessary to find these defects, how testing promotes quality, and how
testing fits into quality assurance. In this section, we will also introduce some
fundamental principles of testing.
As we go through this section, watch for the Syllabus terms bug, defect, error,
failure, fault, mistake, quality, risk, software, testing and exhaustive testing.
You'll find these terms defined in the glossary.
You may be asking 'what is testing?' and we'll look more closely at the defi-
nition of testing in Section 1.2. For the moment, let's adopt a simple everyday-
life usage: 'when we are testing something we are checking whether it is OK'.
We'll need to refine that when we define software testing later on. Let's start by
considering why testing is needed. Testing is necessary because we all make mis-
takes. Some of those mistakes are unimportant, but some of them are expensive
or dangerous. We need to check everything and anything we produce because
things can always go wrong - humans make mistakes all the time - it is what we
do best!
Because we should assume our work contains mistakes, we all need to check
our own work. However, some mistakes come from bad assumptions and blind
spots, so we might make the same mistakes when we check our own work as we
made when we did it. So we may not notice the flaws in what we have done.
Ideally, we should get someone else to check our work - another person is more
likely to spot the flaws.
In this book, we'll explore the implications of these two simple paragraphs
again and again. Does it matter if there are mistakes in what we do? Does it
matter if we don't find some of those flaws? We know that in ordinary life, some
of our mistakes do not matter, and some are very important. It is the same with
software systems. We need to know whether a particular error is likely to cause
problems. To help us think about this, we need to consider the context within
which we use different software systems.
What is quality?
Projects aim to deliver software to specification. For the project to deliver
what the customer needs requires a correct specification. Additionally, the
delivered system must meet the specification. This is known as validation ('is
this the right specification?') and verification ('is the system correct to spec-
ification?'). Of course, as well as wanting the right software system built cor-
rectly, the customer wants the project to be within budget and timescale - it
should arrive when they need it and not cost too much.
The ISTQB glossary definition covers not just the specified requirements but
also user and customer needs and expectations. It is important that the project
team, the customers and any other project stakeholders set and agree expecta-
tions. We need to understand what the customers understand by quality and
what their expectations are. What we as software developers and testers may
see as quality - that the software meets its defined specification, is technically
excellent and has few bugs in it - may not provide a quality solution for our cus-
tomers. Furthermore, if our customers find they have spent more money than
they wanted or that the software doesn't help them carry out their tasks, they
won't be impressed by the technical excellence of the solution. If the customer
wants a cheap car for a 'run-about' and has a small budget then an expensive
sports car or a military tank are not quality solutions, however well built they
are.
To help you compare different people's expectations, Table 1.1 summarizes
and explains quality viewpoints and expectations using 'producing and buying
tomatoes' as an analogy for 'producing and buying software'. You'll see as you
look through the table that the approach to testing would be quite different
depending on which viewpoint we favor [Trienekens], [Evans].
In addition to understanding what quality feels and looks like to customers,
users, and other stakeholders, it helps to have some quality attributes to
measure quality against, particularly to aid the first, product-based, viewpoint
in the table. These attributes or characteristics can serve as a framework or
checklists for areas to consider coverage. One such set of quality attributes can
Quality is fitness for use. Quality can We will ask the users whether they The tomatoes are right for our
have subjective aspects and not just can carry out their tasks; if they are recipe,
quantitative aspects. satisfied that they can we will release
the software.
Quality is based on good manufacturing We will use a recognized software The tomatoes are organically
processes, and meeting defined development process. We will only farmed. The tomatoes have no
requirements. It is measured by testing, release the software if there are fewer blemishes and no pest
inspection, and analysis of faults and than five outstanding high-priority damage,
failures. defects once the planned tests are
complete.
Expectation of value for money. We have time-boxed the testing to The tomatoes have a good
affordability, and a value-based trade-off two weeks to stay in the project shelf life. The tomatoes are
between time, effort and cost aspects. budget. cheap or good value for
We can afford to buy this software and money,
we expect a return on investment.
Transcendent feelings - this is about the We like this software! It is fun and it's We get our tomatoes from a
feelings of an individual or group of the latest thing! So what if it has a small local farm and we get on
individuals towards a product or a few problems? We want to use it so well with the growers,
supplier. anyway...
We really enjoy working with this
software team. So, there were a few
problems - they sorted them out
really quickly - we trust them.
be found in the ISO 9126 standard. This hierarchy of characteristics and sub-
characteristics of quality is discussed in Chapter 2.
We've seen that testing helps us find defects and improve software quality. How
much testing should we do? We have a choice: test everything, test nothing or
test some of the software. Now, your immediate response to that may well be to
say, 'Everything must be tested'. We don't want to use software that has not
been completely tested, do we? This implies that we must exercise every aspect
of a software system during testing. What we need to consider is whether we
must, or even can, test completely.
Let's look at how much testing we'd need to do to be able to test exhaus-
tively. How many tests would you need to do to completely test a one-digit
numeric field? The immediate question is, 'What you mean by test completely?'
There are 10 possible valid numeric values but as well as the valid values we
need to ensure that all the invalid values are rejected. There are 26 uppercase
alpha characters, 26 lower case, at least 6 special and punctuation characters as
well as a blank value. So there would be at least 68 tests for this example of a
one-digit field.
This problem just gets worse as we look at more realistic examples. In prac-
tice, systems have more than one input field with the fields being of varying
sizes. These tests would be alongside others such as running the tests in differ-
Section 2 What is testing? 11
ent environments. If we take an example where one screen has 15 input fields,
each having 5 possible values, then to test all of the valid input value combina-
tions you would need 30 517 578 125 (515) tests! It is unlikely that the project
timescales would allow for this number of tests.
Testing our one-digit field with values 2, 3 and 4 makes our tests more thor-
ough, but it does not give us more information than if we had just tested with the
value 3.
Pressures on a project include time and budget as well as pressure to
deliver a technical solution that meets the customers' needs. Customers and
project managers will want to spend an amount on testing that provides a
return on investment for them. This return on investment includes prevent-
ing failures after release that are costly. Testing completely - even if that is
what customers and project managers ask for - is simply not what they can
afford.
Instead we need a test approach which provides the right amount of testing
for this project, these customers (and other stakeholders) and this software. We
do this by aligning the testing we do with the risks for the customers, the stake-
holders, the project and the software. Assessing and managing risk is one of the
most important activities in any project, and is a key activity and reason for
testing. Deciding how much testing is enough should take account of the level
of risk, including technical and business risks related to the product and project
constraints such as time and budget.
We carry out a risk assessment to decide how much testing to do. We can
then vary the testing effort based on the level of risk in different areas.
Additionally, testing should provide sufficient information to stakeholders
to make informed decisions about the release of the software or system
we're testing, for the next development step or handover to customers. The
effort put into the quality assurance and testing activities needs to be tai
lored to the risks and costs associated with the project. Because of the
limits in the budget, the time, and in testing we need to decide how we will
focus our testing, based on the risks. We'll look at risk assessment in
Chapter 5.
In this section, we will review the common objectives of testing. We'll explain
how testing helps us to find defects, provide confidence and information, and
prevent defects. We will also introduce additional fundamental principles of
testing.
As you read this section, you'll encounter the terms code, debugging,
development of software, requirement, review, test basis, test case, testing
and test objective.
We can use both dynamic testing and static testing as a means for
achieving similar test objectives. Both provide information to
improve both the system to be tested, and the development and
testing processes. We mentioned above that testing can have
different goals and objectives, which often include:
• finding defects;
• gaining confidence in and providing information about the level
of quality;
• preventing defects.
Over time, as we improve our whole software development life cycle and the
early static testing, we may well find that dynamic test levels find fewer
defects. A typical test improvement initiative will initially find more defects
as the testing improves and then, as the defect prevention kicks in, we see the
defect numbers dropping, as shown in Figure 1.3. The first part of the
improvement enables us to reduce failures in operation; later the improve-
ments are making us more efficient and effective in producing the software
with fewer defects in it.
As the 'hot spots' for bugs get cleaned up we need to move our focus else-
where, to the next set of risks. Over time, our focus may change from finding
coding bugs, to looking at the requirements and design documents for defects,
and to looking for process improvements so that we prevent defects in the
product. Referring to Figure 1.3, by releases 9 and 10, we would expect that the
overall cost and effort associated with reviews and testing is much lower than in
releases 4 or 7.
This principle arises from the theory of the process of scientific experimenta-
tion and has been adopted by testers; you'll see the idea in many testing books.
While you are not expected to read the scientific theory [Popper] the analogy
used in science is useful; however many white swans we see, we cannot say 'All
swans are white'. However, as soon as we see one black swan we can say 'Not
all swans are white'. In the same way, however many tests we execute without
finding a bug, we have not shown 'There are no bugs'. As soon as we find a bug,
we have shown 'This code is not bug-free'.
1.2.9 If we don't find defects does that mean the users will
accept the software?
Testing Principle - Absence of errors fallacy
Finding and fixing defects does not help if the system built is unusable and does not fulfill the
users' needs and expectations.
There is another important principle we must consider; the customers for soft-
ware - the people and organizations who buy and use it to aid in their day-to-
day tasks - are not interested in defects or numbers of defects, except when they
are directly affected by the instability of the software. The people using soft-
ware are more interested in the software supporting them in completing tasks
efficiently and effectively. The software must meet their needs. It is for this
reason that the requirements and design defects we discussed earlier are so
important, and why reviews and inspections (see Chapter 3) are such a funda-
mental part of the entire test activity.
Principle 1: Testing shows Testing can show that defects are present,
presence of defects but cannot prove that there are no
defects. Testing reduces the probability of
undiscovered defects remaining in the
software but, even if no defects are found,
it is not a proof of correctness.
Principle 5: Pesticide paradox If the same tests are repeated over and
over again, eventually the same set of test
cases will no longer find any new bugs. To
overcome this 'pesticide paradox', the test
cases need to be regularly reviewed and
revised, and new and different tests need
to be written to exercise different parts of
the software or system to potentially find
more defects.
1.4.1 Introduction
In this section, we will describe the fundamental test process and activi-
ties. These start with test planning and continue through to test closure.
For each part of the test process, we'll discuss the main tasks of each test
activity.
In this section, you'll also encounter the glossary terms confirmation testing,
exit criteria, incident, regression testing, test basis, test
condition, test coverage, test data, test execution, test log, test
plan, test strategy, test summary report and testware.
As we have seen, although executing tests is important, we also need a
plan of action and a report on the outcome of testing. Project and test plans
should include time to be spent on planning the tests, designing test cases,
preparing for execution and evaluating status. The idea of a fundamental test
process for all levels of test has developed over the years. Whatever the level
of testing, we see the same type of main activities happening, although there
may be a different amount of formality at the different levels, for example,
component tests might be carried out less formally than system tests in most
organizations with a less documented test process. The decision about the
level of formality of the processes will depend on the system and software
context and the level of risk associated with the software. So we can divide
the activities within the fundamental test process into the following basic
steps:
• planning and control;
• analysis and design;
• implementation and execution;
• evaluating exit criteria and reporting;
• test closure activities.
These activities are logically sequential, but, in a particular project, may
overlap, take place concurrently and even be repeated. This process is par-
ticularly used for dynamic testing, but the main headings of the process
can be applied to reviews as well. For example, we need to plan and
prepare for reviews, carry out the reviews, and evaluate the outcomes of
the reviews. For some reviews, such as inspections, we will have exit crite-
ria and will go through closure activities. However, the detail and naming
of the activities will be different for static testing. We'll discuss static
testing in Chapter 3.
1.4.2 Test planning and control
During test planning, we make sure we understand the goals and objectives of
the customers, stakeholders, and the project, and the risks which testing is
intended to address. This will give us what is sometimes called the mission of
testing or the test assignment. Based on this understanding, we set the goals and
objectives for the testing itself, and derive an approach and plan for the tests,
including specification of test activities. To help us we may have organization or
program test policies and a test strategy. Test policy gives rules for testing, e.g.
'we always review the design documents'; test strategy is the overall high-level
approach, e.g. 'system testing is carried out by an independent team reporting
to the program quality manager. It will be risk-based and proceeds from a
product (quality) risk analysis' (see Chapter 5). If policy and strategy are
defined already they drive our planning but if not we should ask for them to be
stated and defined. Test planning has the following major tasks, given approxi-
mately in order, which help us build a test plan:
• Determine the scope and risks and identify the objectives of testing: we con
sider what software, components, systems or other products are in scope for
testing; the business, product, project and technical risks which need to be
addressed; and whether we are testing primarily to uncover defects, to show
that the software meets requirements, to demonstrate that the system is fit
for purpose or to measure the qualities and attributes of the software.
• Determine the test approach (techniques, test items, coverage, identifying
and interfacing with the teams involved in testing, testware): we consider
how we will carry out the testing, the techniques to use, what needs testing
and how extensively (i.e. what extent of coverage). We'll look at who needs
to get involved and when (this could include developers, users, IT infrastruc
ture teams); we'll decide what we are going to produce as part of the testing
(e.g. testware such as test procedures and test data). This will be related to
the requirements of the test strategy.
• Implement the test policy and/or the test strategy: we mentioned that there
may be an organization or program policy and strategy for testing. If this is
the case, during our planning we must ensure that what we plan to do
adheres to the policy and strategy or we must have agreed with stakeholders,
and documented, a good reason for diverging from it.
• Determine the required test resources (e.g. people, test environment, PCs):
from the planning we have already done we can now go into detail; we decide
on our team make-up and we also set up all the supporting hardware and
software we require for the test environment.
• Schedule test analysis and design tasks, test implementation, execution and
evaluation: we will need a schedule of all the tasks and activities, so that we
can track them and make sure we can complete the testing on time.
• Determine the exit criteria: we need to set criteria such as coverage criteria (for
example, the percentage of statements in the software that must be executed
during testing) that will help us track whether we are completing the test activ
ities correctly. They will show us which tasks and checks we must complete for
a particular level of testing before we can say that testing is finished.
Management of any activity does not stop with planning it. We need to
control and measure progress against the plan. So, test control is an ongoing
activity. We need to compare actual progress against the planned progress, and
report to the project manager and customer on the current status of testing,
including any changes or deviations from the plan. We'll need to take actions
where necessary to meet the objectives of the project. Such actions may entail
changing our original plan, which often happens. When different groups
perform different review and test activities within the project, the planning and
control needs to happen within each of those groups but also across the groups
to coordinate between them, allowing smooth hand-offs between each stage of
testing. Test planning takes into account the feedback from monitoring and
control activities which take place through out the project. Test control has the
following major tasks:
• Measure and analyze the results of reviews and testing: We need to know
how many reviews and tests we have done. We need to track how many
tests have passed and how many failed, along with the number, type and
importance of the defects reported.
• Monitor and document progress, test coverage and exit criteria: It is important
that we inform the project team how much testing has been done, what the
results are, and what conclusions and risk assessment we have made. We must
make the test outcome visible and useful to the whole team.
• Provide information on testing: We should expect to make regular and
exceptional reports to the project manager, project sponsor, customer and
other key stakeholders to help them make informed decisions about
project status. We should also use the information we have to analyze the
testing itself.
• Initiate corrective actions: For example, tighten exit criteria for defects fixed,
ask for more effort to be put into debugging or prioritize defects for fixing
test blockers.
• Make decisions: Based on the measures and information gathered during
testing and any changes to business and project risks or our increased under
standing of technical and product risks, we'll make decisions or enable others
to make decisions: to continue testing, to stop testing, to release the software
or to retain it for further work for example.
In this section, we'll discuss the various psychological factors that influence
testing and its success. These include clear objectives for testing, the proper
roles and balance of self-testing and independent testing, clear, courteous com-
munication and feedback on defects. We'll also contrast the mindset of a tester
and of a developer.
You'll find a single Syllabus term in this section, independent testing, and the
glossary term, independence.
1.5.1 Independent testing - who is a tester?
The mindset we want to use while testing and reviewing is different from the
one we use while analyzing or developing. By this we mean that, if we are build-
ing something we are working positively to solve problems in the design and to
realize a product that meets some need. However, when we test or review a
product, we are looking for defects in the product and thus are critical of it.
Suppose you were going to cook a meal to enter in a competition for chefs.
You select the menu, collect the ingredients, cook the food, set the table, and
serve the meal. If you want to win, you do each task as well as you can. Suppose
instead you are one of the judges evaluating the competition meals. You
examine everything critically, including the menu, the ingredients, the methods
used, keeping to time and budget allowances, choice of ingredients, the ele-
gance of the table setting and the serving, and the look and taste of the meal.
To differentiate between the competition chefs, you'll praise every good aspect
of their performances but you'll also note every fault and error each chef made.
So it is with software testing: building the software requires a different mindset
from testing the software.
We do not mean that a tester cannot be a programmer, or that a program-
mer cannot be a tester, although they often are separate roles. In fact, program-
mers are testers - they test the components which they build, and the
integration of the components into the system. The good chef will be as critical
as the competition judges of his own work, in order to prevent and rectify errors
and defects before anyone notices them. So, with the right mindset, program-
mers can test their own code; indeed programmers do test their own code and
find many problems, resolving them before anyone else sees the code. Business
analysis and marketing staff should review their own requirements. System
architects should review their own designs. However, we all know it is difficult
to find our own mistakes. So, business analysts, marketing staff, architects and
programmers often rely on others to help test their work. This other person
might be a fellow analyst, designer or developer. A person who will use the soft-
ware may help test it. Business analysts who worked on the requirements and
design may perform some tests. Testing specialists - professional testers - are
often involved. In fact, testing may involve a succession of people each carrying
out a different level of testing. This allows an independent test of the system.
We'll look at the points in the software development life cycle where testing
takes place in Chapter 2. You'll see there that several stages of reviews and
testing are carried out throughout the life cycle and these may be independent
reviews and tests. Early in the life cycle, reviews of requirements and design
documents by someone other than the author helps find defects before coding
starts and helps us build the right software. Following coding, the software can
be tested independently. This degree of independence avoids author bias and is
often more effective at finding defects and failures.
Several levels of independence can be identified, listed here from the lowest
level of independence to the highest:
• tests by the person who wrote the item under test;
• tests by another person within the same team, such as another programmer;
• tests by a person from a different organizational group, such as an independ
ent test team;
• tests designed by a person from a different-organization or company, such as
outsourced testing or certification by an external body.
We should note, however, that independence is not necessarily the most
important factor in good testing. Developers who know how to test and who
are, like good chefs, self-critical, have the benefit of familiarity and the pride-
of-work that comes with true professionalism. Such developers can efficiently
find many defects in their own code. Some software development methodolo-
gies insist on developers designing tests before they start coding and executing
those tests continuously as they change the code. This approach promotes early
testing and early defect detection, which is cost effective. Remember, independ-
ent testing may be carried out at any level of testing and the choice of independ-
ence level depends on the risk in the particular context.
Question 2 According to the ISTQB Glossary, Question 6 According to the ISTQB Glossary,
the word 'bug' is synonymous with which of the regression testing is required for what purpose?
following words? a. To verify the success of corrective actions.
a. Incident b. To prevent a task from being incorrectly consid
b. Defect ered completed.
c. Mistake c. To ensure that defects have not been introduced
d. Error by a modification.
d. To motivate better unit testing by the program
Question 3 According to the ISTQB Glossary, a mers.
risk relates to which of the following?
a. Negative feedback to the tester. Question 7 Which of the following is most
b. Negative consequences that will occur. important to promote and maintain good relation-
ships between testers and developers?
c. Negative consequences that could occur.
a. Understanding what managers value about
d. Negative consequences for the test object. testing.
b. Explaining test results in a neutral fashion.
Question 4 Ensuring that test design starts
during the requirements definition phase is impor- c. Identifying potential customer work-arounds for
tant to enable which of the following test objec- bugs.
tives? d. Promoting better quality software whenever
a. Preventing defects in the system. possible.
b. Finding defects through dynamic testing.
Question 8 Which of the statements below is the
c. Gaining confidence in the system. best assessment of how the test principles apply
d. Finishing the project on time. across the test life cycle?
a. Test principles only affect the preparation for
Question 5 A test team consistently finds testing.
between 90% and 95% of the defects present in b. Test principles only affect test execution activi
the system under test. While the test manager ties.
understands that this is a good defect-detection
percentage for her test team and industry, senior c. Test principles affect the early test activities
management and executives remain disappointed such as review.
in the test group, saying that the test team misses d. Test principles affect activities throughout the
too many bugs. Given that the users are generally test life cycle.
EXERCISE: TEST PSYCHOLOGY
Read the email below, and see what clues you find to help you identify problems in the scenario described.
Categorize the clues/problems as:
• possible people, psychology and attitude problems;
• other problems, e.g. possible test management and role problems, possible product problems.
Hi there!
Well, I nearly caused a panic today because I thought I had found a mega
showstopper on the trading system we are testing. The test manager and others got
involved examining databases first on the server and then on the gateway that feeds
the clients, checking update logs from processes that ran overnight as well as
checking data passed to the client. Eventually I found the problem. I had mis-clicked
on a .bat file when running up a client and had run up the wrong client environment.
By that time the test manager was ready to say a few short words in my ear,
particularly as the development people had started to get involved and they have zero
tolerance for mistakes made by testers. The only saving grace was that I found the
mistake and not one of the developers.
It was, objectively, an interesting mistake. When you log into the server test
environments, the panels always show the environment to which you are
connected. In our case we have two test environments called Systest14 and
Systest15 and my tests were set up in Systest15. To run up the clients, we have to
run .bat files for either a 14 or 15 client. I had started two clients, that is two
exchange participants, so I could do some trading between them.
It appears I started the first client OK in environment 15 but when I started the
second, I accidentally moved the mouse a fraction so it ran the 14 .bat file that is next
to it in the Explorer file list. To make matters worse, the client screens do not show the
environment to which you are attached.
At first I felt a bit stupid having caused much hectic and wasted activity. On
reflection I thought that if I, as a reasonably competent person, can make a mistake
like this then something is wrong. On the server side when I log on to a test
environment, I have to enter the environment name and it's shown on all the panels.
On the client side, I run a client test environment by selecting a .bat file from a list of
many and have to ensure I click on the right file. There is neither a display nor the
ability to determine the client environment in which I am working.
So I am going to log this as a high priority, or even showstopper, error - the client
does not show the environment. In real life terms, it means a real user could be
connected to the production system and think he is connected to a test system and
screw up trading. I know this happened once on the equities trading system, when a
trader entered a load of test transactions into the production system by mistake and
caused mayhem.
As an addendum to this story, a couple of days later one of the testers found what
appeared to be another mega showstopper. He and the test manager spent three
hours crawling all over the system before they discovered the 'error'. A new filter had
been added to the client software to filter transactions displayed in panels by
geographical market. Unknown to them, it was set to a default of the German market,
whereas they thought they were in the UK market. Consequently, at first sight, it
appeared there were fundamental problems with the network transaction bus and the
message-broadcasting systems. Apart from the issue that they should have been
informed of this change, it raised a similar problem to the one I had experienced -the
client system does not display the market in which you are trading.
Well - I'm off for another happy day at the
office! All the best
EXERCISE SOLUTION
People, psychology and attitude problems include, for example:
• Poor relationships between the test team and the test manager, and the testers and developers, e.g. 'By
that time the test manager was ready to say a few short words in my ear, particularly as the development
people had started to get involved and they have zero tolerance for mistakes made by testers. The only
saving grace was that I found the mistake and not one of the developers.'
• Emotive use of language - understandable in the circumstances but not suitable for reporting problems,
e.g. 'Well, I nearly caused a panic today because I thought I had found a mega showstopper on the
trading system we are testing,' and 'As an addendum to this story, a couple of days later one of the testers
found what appeared to be another mega-showstopper.'
• Initial diffidence overcome by revisiting the problem - if one person can make this mistake then others
will. 'At first I felt a bit stupid having caused much hectic and wasted activity. On reflection I thought
that if I, as a reasonably competent person, can make a mistake like this then something is wrong.'
• Understandable use of sarcasm ... 'Well - I'm off for another happy day at the office!'
Other problems include test management and role problems, for example:
• Configuration management and release control - A new filter had been added to the client software to
filter transactions displayed in panels by geographical market.'
• Configuration management, relationships, communications - Apart from the issue that they should
have been informed of this change ....'
• Does the test manager really understand his role? 'He and the test manager spent three hours crawling
all over the system before they discovered the "error",' and 'The test manager and others got involved
examining databases.'
There are some product problems, although no functionality or technical problems. Not all the problems
we encounter as testers are functionality or technical problems. There are some non-functional problems
- specifically, usability - which indicate that a real user might be inconvenienced or worse by this problem:
• 'I had mis-clicked on a .bat file ...'
• 'In real life terms, it means a real user could be connected to the production system and think he is
connected to a test system and screw up trading. I know this happened once ... when a trader entered
a load of test transactions into the production system by mistake and caused mayhem.'
• 'It raised a similar problem to the one I had experienced - the client system does not display the market
in which you are trading.'
• 'There is neither a display nor the ability to determine the client environment in which I am working.'
And 'To make matters worse, the client screens do not show the environment to which you are attached.'
• 'Unknown to them, it was set to a default of the German market, whereas they thought they were in the
UK market.'
Note that we will return to this exercise at the end of Chapter 5, where we deal with writing a good
incident report.
CHAPTER 2
T esting is not a stand-alone activity. It has its place within a software development life cycle
model and therefore the life cycle applied will largely determine how testing is organized.
There are many different forms of testing. Because several disciplines, often with different interests,
are involved in the development life cycle, it is important to clearly understand and define the
various test levels and types. This chapter discusses the most commonly applied software
development models, test levels and test types. Maintenance can be seen as a specific instance of a
development process. The way maintenance influences the test process, levels and types and how
testing can be organized is described in the last section of this chapter.
The development process adopted for a project will depend on the project aims and goals. There are
numerous development life cycles that have been developed in order to achieve different required
objectives. These life cycles range from lightweight and fast methodologies, where time to
market is of the essence, through to fully controlled and documented methodologies where quality
and reliability are key drivers. Each of these methodologies has its place in modern software
development and the most appropriate development process should be applied to each project. The
models specify the various stages of the process and the order in which they are carried out.
The life cycle model that is adopted for a project will have a big impact on the testing that is carried
out. Testing does not exist in isolation; test activities
are highly related to software development activities. It will define the what,
where, and when of our planned testing, influence regression testing, and
largely determine which test techniques to use. The way testing is organized
must fit the development life cycle or it will fail to deliver its benefit. If time to
market is the key driver, then the testing must be fast and efficient. If a fully
documented software development life cycle, with an audit trail of evidence, is
required, the testing must be fully documented.
In every development life cycle, a part of testing is focused on verification
testing and a part is focused on validation testing. Verification is concerned
with evaluating a work product, component or system to determine whether it
meets the requirements set. In fact, verification focuses on the question 'Is the
deliverable built according to the specification?'. Validation is concerned with
evaluating a work product, component or system to determine whether it meets
the user needs and requirements. Validation focuses on the question 'Is the
deliverable fit for purpose, e.g. does it provide a solution to the problem?'.
2.1.1 V-model
Before discussing the V-model, we will look at the model which came before it.
The waterfall model was one of the earliest models to be designed. It has a
natural timeline where tasks are executed in a sequential fashion. We start at the
top of the waterfall with a feasibility study and flow down through the various
project tasks finishing with implementation into the live environment. Design
flows through into development, which in turn flows into build, and finally on
into test. Testing tends to happen towards the end of the project life cycle so
defects are detected close to the live implementation date. With this model it has
been difficult to get feedback passed backwards up the waterfall and there are
difficulties if we need to carry out numerous iterations for a particular phase.
The V-model was developed to address some of the problems experienced
using the traditional waterfall approach. Defects were being found too late
in the life cycle, as testing was not involved until the end of the project.
Testing also added lead time due to its late involvement. The V-model pro-
vides guidance that testing needs to begin as early as possible in the life
cycle, which, as we've seen in Chapter 1, is one of the fundamental princi-
ples of structured testing. It also shows that testing is not only an execution-
based activity. There are a variety of activities that need to be performed
before the end of the coding phase. These activities should be carried out
in parallel with development activities, and testers need to work with devel-
opers and business analysts so they can perform these activities and tasks
and produce a set of test deliverables. The work products produced by the
developers and business analysts during development are the basis of
testing in one or more levels. By starting test design early, defects are often
found in the test basis documents. A good practice is to have testers
involved even earlier, during the review of the (draft) test basis documents.
The V-model is a model that illustrates how testing activities (verification
and validation) can be integrated into each phase of the life cycle. Within
the V-model, validation testing takes place especially during the early
stages, e.g. reviewing the user requirements, and late in the life cycle, e.g.
during user acceptance testing.
Although variants of the V-model exist, a common type of V-model uses
four test levels. The four test levels used, each with their own objectives,
are:
• component testing: searches for defects in and verifies the functioning of
software components (e.g. modules, programs, objects, classes etc.) that are
separately testable;
• integration testing: tests interfaces between components, interactions to dif
ferent parts of a system such as an operating system, file system and hard
ware or interfaces between systems;
• system testing: concerned with the behavior of the whole system/product as
defined by the scope of a development project or product. The main focus of
system testing is verification against specified requirements;
• acceptance testing: validation testing with respect to user needs, require
ments, and business processes conducted to determine whether or not to
accept the system.
The various test levels are explained and discussed in detail in Section 2.2.
In practice, a V-model may have more, fewer or different levels of devel-
opment and testing, depending on the project and the software product. For
example, there may be component integration testing after component
testing and system integration testing after system testing. Test levels can be
combined or reorganized depending on the nature of the project or the
system architecture. For the integration of a commercial off-the-shelf
(COTS) software product into a system, a purchaser may perform only inte-
gration testing at the system level (e.g. integration to the infrastructure and
other systems) and at a later stage acceptance testing.
Note that the types of work products mentioned in Figure 2.2 on the left side
of the V-model are just an illustration. In practice they come under many dif-
ferent names. References for generic work products include the Capability
Maturity Model Integration (CMMi) or the 'Software life cycle processes' from
ISO/IEC 12207. The CMMi is a framework for process improvement for both
system engineering and software engineering. It provides guidance on where to
focus and how, in order to increase the level of process maturity [Chrissis et ah,
2004]. ISO/IEC 12207 is an integrated software life cycle process standard that
is rapidly becoming more popular.
Agile development
Extreme Programming (XP) is currently one of the most well-known agile
development life cycle models. (See [Agile] for ideas behind this approach.)
The methodology claims to be more human friendly than traditional develop-
ment methods. Some characteristics of XP are:
• It promotes the generation of business stories to define the functionality.
• It demands an on-site customer for continual feedback and to define and
carry out functional acceptance testing.
• It promotes pair programming and shared code ownership amongst the
developers.
• It states that component test scripts shall be written before the code is
written and that those tests should be automated.
• It states that integration and testing of the code shall happen several times
a day.
• It states that we always implement the simplest solution to meet today's
problems.
With XP there are numerous iterations each requiring testing. XP develop-
ers write every test case they can think of and automate them. Every time a
change is made in the code it is component tested and then integrated with the
existing code, which is then fully integration-tested using the full set of test
cases. This gives continuous integration, by which we mean that changes are
incorporated continuously into the software build. At the same time, all test
cases must be running at 100% meaning that all the test cases that have been
identified and automated are executed and pass. XP is not about doing extreme
activities during the development process, it is about doing known vajue-adding
activities in an extreme manner.
2.1.3 Testing within a life cycle model
In summary, whichever life cycle model is being used, there are several charac-
teristics of good testing:
• for every development activity there is a corresponding testing activity;
• each test level has test objectives specific to that level;
• the analysis and design of tests for a given test level should begin during the
corresponding development activity;
• testers should be involved in reviewing documents as soon as drafts are avail
able in the development cycle.
2 . 2 TEST LEVELS
The V-model for testing was introduced in Section 2.1. This section looks in
more detail at the various test levels. The key characteristics for each test level
are discussed and defined to be able to more clearly separate the various test
levels. A thorough understanding and definition of the various test levels will
identify missing areas and prevent overlap and repetition. Sometimes we may
wish to introduce deliberate overlap to address specific risks. Understanding
whether we want overlaps and removing the gaps will make the test levels more
complementary thus leading to more effective and efficient testing.
Regression testing
Like confirmation testing, regression testing involves executing test cases that
have been executed before. The difference is that, for regression testing, the
test cases probably passed the last time they were executed (compare this with
the test cases executed in confirmation testing - they failed the last time).
The term 'regression testing' is something of a misnomer. It would be better
if it were called 'anti-regression' testing because we are executing tests with the
intent of checking that the system has not regressed (that is, it does not now
have more defects in it as a result of some change). More specifically, the
purpose of regression testing is to verify that modifications in the software or
the environment have not caused unintended adverse side effects and that the
system still meets its requirements.
It is common for organizations to have what is usually called a regression test
suite or regression test pack. This is a set of test cases that is specifically used for
regression testing. They are designed to collectively exercise most functions (cer-
tainly the most important ones) in a system but not test any one in detail. It is
appropriate to have a regression test suite at every level of testing (component
testing, integration testing, system testing, etc.). All of the test cases in a regression
test suite would be executed every time a new version of software is produced and
this makes them ideal candidates for automation. If the regression test suite is very
large it may be more appropriate to select a subset for execution.
Regression tests are executed whenever the software changes, either as a result
of fixes or new or changed functionality. It is also a good idea to execute them
when some aspect of the environment changes, for example when a new version
of a database management system is introduced or a new version of a source code
compiler is used.
Maintenance of a regression test suite should be carried out so it evolves
over time in line with the software. As new functionality is added to a system
new regression tests should be added and as old functionality is changed or
removed so too should regression tests be changed or removed. As new tests
are added a regression test suite may become very large. If all the tests have to
be executed manually it may not be possible to execute them all every time the
regression suite is used. In this case a subset of the test cases has to be chosen.
This selection should be made in light of the latest changes that have been made
to the software. Sometimes a regression test suite of automated tests can
become so large that it is not always possible to execute them all. It may be pos-
sible and desirable to eliminate some test cases from a large regression test
suite for example if they are repetitive (tests which exercise the same condi-
tions) or can be combined (if they are always run together). Another approach
is to eliminate test cases that have not found a defect for a long time (though
this approach should be used with some care!).
2 . 4 MAINTENANCE TESTING
Once deployed, a system is often in service for years or even decades. During
this time the system and its operational environment is often corrected,
changed or extended. Testing that is executed during this life cycle phase is
called 'maintenance testing'.
Note that maintenance testing is different from maintainability testing,
which defines how easy it is to maintain the system.
The development and test process applicable to new developments does not
change fundamentally for maintenance purposes. The same test process steps
will apply and, depending on the size and risk of the changes made, several
levels of testing are carried out: a component test, an integration test, a system
test and an acceptance test. A maintenance test process usually begins with the
receipt of an application for a change or a release plan. The test manager will
use this as a basis for producing a test plan. On receipt of the new or changed
specifications, corresponding test cases are specified or adapted. On receipt of
the test object, the new and modified tests and the regression tests are executed.
On completion of the testing, the testware is once again preserved.
Comparing maintenance testing to testing a new application is merely a
matter of an approach from a different angle, which gives rise to a number of
changes in emphasis. There are several areas where most differences
occur, for example regarding the test basis. A 'catching-up' operation is
frequently required when systems are maintained. Specifications are
often 'missing', and a set of testware relating to the specifications simply
does not exist. It may well be possible to carry out this catching-up oper-
ation along with testing a new maintenance release, which may reduce the
cost. If it is impossible to compile any specifications from which test cases
can be written, including expected results, an alternative test basis, e.g. a
test oracle, should be sought by way of compromise. A search should be
made for documentation which is closest to the specifications and which
can be managed by developers as well as testers. In such cases it is advis-
able to draw the customer's attention to the lower test quality which may
be achieved. Be aware of possible problems of 'daily production'. In the
worst case nobody knows what is being tested, many test cases are execut-
ing the same scenario and if an incident is found it is often hard to trace
it back to the actual defect since no traceability to test designs and/or
requirements exists. Note that reproducibility of tests is also important
for maintenance testing.
One aspect which, in many cases, differs somewhat from the development
situation is the test organization. New development and their appropriate test
activities are usually carried out as parts of a project, whereas maintenance tests
are normally executed as an activity in the regular organization. As a result,
there is often some lack of resources and flexibility, and the test process may
experience more competition from other activities.
Planned modifications
The following types of planned modification may be identified:
• perfective modifications (adapting software to the user's wishes, for instance
by supplying new functions or enhancing performance);
• adaptive modifications (adapting software to environmental changes such as
new hardware, new systems software or new legislation);
• corrective planned modifications (deferrable correction of defects).
The standard structured test approach is almost fully applicable to planned
modifications. On average, planned modification represents over 90% of all
maintenance work on systems. [Pol and van Veenendaal]
Static techniques
S tatic test techniques provide a powerful way to improve the quality and productivity of software
development. This chapter describes static test techniques, including reviews, and provides an overview
of how they are conducted. The fundamental objective of static testing is to improve the quality of software
work products by assisting engineers to recognize and fix their own defects early in the software
development process. While static testing techniques will not solve all the problems, they are
enormously effective. Static techniques can improve both quality and productivity by impressive factors.
Static testing is not magic and it should not be considered a replacement for dynamic testing, but all
software organizations should consider using reviews in all major aspects of their work including
requirements, design, implementation, testing, and maintenance. Static analysis tools implement
automated checks, e.g. on code.
In Chapter 1, several testing terms were presented. Also testing itself was defined. The latter definition
is repeated here as a means for explaining the two major types of testing.
The definition of testing outlines objectives that relate to evaluation, revealing defects and quality. As
indicated in the definition two approaches can be used to achieve these objectives, static testing and
dynamic testing.
With dynamic testing methods, software is executed using a set of input values and its output is then
examined and compared to what is expected. During static testing, software work products are examined
manually, or with a set of tools, but not executed. As a consequence, dynamic testing can only be applied
to software code. Dynamic execution is applied as a technique to detect
defects and to determine quality attributes of the code. This testing option is
not applicable for the majority of the software work products. Among the ques-
tions that arise are: How can we evaluate or analyze a requirements document,
a design document, a test plan, or a user manual? How can we effectively pre-
examine the source code before execution? One powerful technique that can be
used is static testing, e.g. reviews. In principle all software work products can be
tested using review techniques.
Dynamic testing and static testing are complementary methods, as they tend
to find different types of defects effectively and efficiently. Types of defects that
are easier to find during static testing are: deviations from standards, missing
requirements, design defects, non-maintainable code and inconsistent interface
specifications. Note that in contrast to dynamic testing, static testing finds
defects rather than failures.
In addition to finding defects, the objectives of reviews are often also
informational, communicational and educational, whereby participants
learn about the content of software work products to help them understand
the role of their own work and to plan for future stages of development.
Reviews often represent project milestones, and support the establishment
of a baseline for a software product. The type and quantity of defects found
during reviews can also help testers focus their testing and select effective
classes of tests. In some cases customers/users attend the review meeting
and provide feedback to the development team, so reviews are also a means
of customer/user communication.
Studies have shown that as a result of reviews, a significant increase in pro-
ductivity and product quality can be achieved [Gilb and Graham, 1993], [van
Veenendaal, 1999]. Reducing the number of defects early in the product life
cycle also means that less time has to be spent on testing and maintenance. To
summarize, the use of static testing, e.g. reviews, on software work products has
various advantages:
• Since static testing can start early in the life cycle, early feedback on quality
issues can be established, e.g. an early validation of user requirements and
not just late in the life cycle during acceptance testing.
• By detecting defects at an early stage, rework costs are most often relatively
low and thus a relatively cheap improvement of the quality of software prod
ucts can be achieved.
• Since rework effort is substantially reduced, development productivity
figures are likely to increase.
• The evaluation by a team has the additional advantage that there is an
exchange of information between the participants.
• Static tests contribute to an increased awareness of quality issues.
In conclusion, static testing is a very suitable method for improving the
quality of software work products. This applies primarily to the assessed
products themselves, but it is also important that the quality improvement is
not achieved once but has a more structural character. The feedback from
the static testing process to the development process allows for process
improvement, which supports the avoidance of similar errors being made in
the future.
3 , 2 REVIEW PROCESS
Reviews vary from very informal to formal (i.e. well structured and regulated).
Although inspection is perhaps the most documented and formal review tech-
nique, it is certainly not the only one. The formality of a review process is
related to factors such as the maturity of the development process, any legal or
regulatory requirements or the need for an audit trail. In practice the informal
review is perhaps the most common type of review. Informal reviews are
applied at various times during the early stages in the life cycle of a document.
A two-person team can conduct an informal review, as the author can ask a col-
league to review a document or code. In later stages these reviews often involve
more people and a meeting. This normally involves peers of the author, who try
to find defects in the document under review and discuss these defects in a
review meeting. The goal is to help the author and to improve the quality of the
document. Informal reviews come in various shapes and forms, but all have one
characteristic in common - they are not documented.
Planning
The review process for a particular review begins with a 'request for review' by the
author to the moderator (or inspection leader). A moderator is often assigned to
take care of the scheduling (dates, time, place and invitation) of the review. On a
project level, the project planning needs to allow time for review and rework
activities, thus providing engineers with time to thoroughly participate in reviews.
For more formal reviews, e.g. inspections, the moderator always performs an
entry check and defines at this stage formal exit criteria. The entry check is
carried out to ensure that the reviewers' time is not wasted on a document that
is not ready for review. A document containing too many obvious mistakes is
clearly not ready to enter a formal review process and it could even be very
harmful to the review process. It would possibly de-motivate both reviewers and
the author. Also, the review is most likely not effective because the numerous
obvious and minor defects will conceal the major defects.
Although more and other entry criteria can be applied, the following can be
regarded as the minimum set for performing the entry check:
• A short check of a product sample by the moderator (or expert) does not
reveal a large number of major defects. For example, after 30 minutes of
checking, no more than 3 major defects are found on a single page or fewer
than 10 major defects in total in a set of 5 pages.
• The document to be reviewed is available with line numbers.
• The document has been cleaned up by running any automated checks
that apply.
• References needed for the inspection are stable and available.
• The document author is prepared to join the review team and feels confident
with the quality of the document.
If the document passes the entry check, the moderator and author decide
which part of the document to review. Because the human mind can com-
prehend a limited set of pages at one time, the number should not be too
high. The maximum number of pages depends, among other things, on the
objective, review type and document type and should be derived from prac-
tical experiences within the organization. For a review, the maximum size is
usually between 10 and 20 pages. In formal inspection, only a page or two
may be looked at in depth in order to find the most serious defects that are
not obvious.
After the document size has been set and the pages to be checked have been
selected, the moderator determines, in co-operation with the author, the com-
position of the review team. The team normally consists of four to six partici-
pants, including moderator and author. To improve the effectiveness of the
review, different roles are assigned to each of the participants. These roles help
the reviewers focus on particular types of defects during checking. This reduces
the chance of different reviewers finding the same defects. The moderator
assigns the roles to the reviewers.
Figure 3.1 shows the different roles within a review. The roles represent
views of the document under review.
Within reviews the following focuses can be identified:
• focus on higher-level documents, e.g. does the design comply to the
requirements;
• focus on standards, e.g. internal consistency, clarity, naming conventions,
templates;
• focus on related documents at the same level, e.g. interfaces between soft
ware functions;
• focus on usage, e.g. for testability or maintainability.
The author may raise additional specific roles and questions that have to be
addressed. The moderator has the option to also fulfil a role, alongside the task of
being a review leader. Checking the document improves the moderator's ability to
lead the meeting, because it ensures better understanding. Furthermore, it improves
the review efficiency because the moderator replaces an engineer that would other-
wise have to check the document and attend the meeting. It is recommended that
the moderator take the role of checking compliance to standards, since this tends to
be a highly objective role, which leads to less discussion of the defects found.
Kick-off
An optional step in a review procedure is a kick-off meeting. The goal of this
meeting is to get everybody on the same wavelength regarding the document
under review and to commit to the time that will be spent on checking. Also the
result of the entry check and defined exit criteria are discussed in case of a more
formal review. In general a kick-off is highly recommended since there is a
strong positive effect of a kick-off meeting on the motivation of reviewers and
thus the effectiveness of the review process. At customer sites, we have meas-
ured results up to 70% more major defects found per page as a result of per-
forming a kick-off, [van Veenendaal and van der Zwan, 2000]
During the kick-off meeting the reviewers receive a short introduction on the
objectives of the review and the documents. The relationships between the doc-
ument under review and the other documents (sources) are explained, espe-
cially if the number of related documents is high.
Role assignments, checking rate, the pages to be checked, process changes
and possible other questions are also discussed during this meeting. Of course
the distribution of the document under review, source documents and other
related documentation, can also be done during the kick-off.
Preparation
The participants work individually on the document under review using the
related documents, procedures, rules and checklists provided. The individual
participants identify defects, questions and comments, according to their
understanding of the document and role. All issues are recorded, preferably
using a logging form. Spelling mistakes are recorded on the document under
review but not mentioned during the meeting. The annotated document will be
given to the author at the end of the logging meeting. Using checklists during
this phase can make reviews more effective and efficient, for example a specific
checklist based on perspectives such as user, maintainer, tester or operations,
or a checklist for typical coding problems.
A critical success factor for a thorough preparation is the number of pages
checked per hour. This is called the checking rate. The optimum checking
rate is the result of a mix of factors, including the type of document, its com-
plexity, the number of related documents and the experience of the reviewer.
Usually the checking rate is in the range of five to ten pages per hour, but
may be much less for formal inspection, e.g. one page per hour. During
preparation, participants should not exceed this criterion. By collecting data
and measuring the review process, company-specific criteria for checking
rate and document size (see planning phase) can be set, preferably specific
to a document type.
Review meeting
The meeting typically consists of the following elements (partly depending on
the review type): logging phase, discussion phase and decision phase.
During the logging phase the issues, e.g. defects, that have been identified
during the preparation are mentioned page by page, reviewer by reviewer and
are logged either by the author or by a scribe. A separate person to do the
logging (a scribe) is especially useful for formal review types such as an inspec-
tion. To ensure progress and efficiency, no real discussion is allowed during the
logging phase. If an issue needs discussion, the item is logged and then handled
in the discussion phase. A detailed discussion on whether or not an issue is a
defect is not very meaningful, as it is much more efficient to simply log it and
proceed to the next one. Furthermore, in spite of the opinion of the team, a dis-
cussed and discarded defect may well turn out to be a real one during rework.
Every defect and its severity should be logged. The participant who identifies
the defect proposes the severity. Severity classes could be:
• Critical: defects will cause downstream damage; the scope and impact of the
defect is beyond the document under inspection.
• Major, defects could cause a downstream effect (e.g. a fault in a design can
result in an error in the implementation).
• Minor, defects are not likely to cause downstream damage (e.g. non-compli
ance with the standards and templates). ,
In order to keep the added value of reviews, spelling errors are not part of
the defect classification. Spelling defects are noted, by the participants, in the
document under review and given to the author at the end of the meeting or
could be dealt with in a separate proofreading exercise.
During the logging phase the focus is on logging as many defects as possible
within a certain timeframe. To ensure this, the moderator tries to keep a good
logging rate (number of defects logged per minute). In a well-led and disci-
plined formal review meeting, the logging rate should be between one and two
defects logged per minute.
For a more formal review, the issues classified as discussion items will be
handled during this meeting phase. Informal reviews will often not have a sep-
arate logging phase and will start immediately with discussion. Participants
can take part in the discussion by bringing forward their comments and rea-
soning. As chairman of the discussion meeting, the moderator takes care of
people issues. For example, the moderator prevents discussions from getting
too personal, rephrases remarks if necessary and calls for a break to cool
down 'heated' discussions and/or participants.
Reviewers who do not need to be in the discussion may leave, or stay as a
learning exercise. The moderator also paces this part of the meeting and
ensures that all discussed items either have an outcome by the end of the
meeting, or are defined as an action point if a discussion cannot be solved
during the meeting. The outcome of discussions is documented for future
reference.
At the end of the meeting, a decision on the document under review has to
be made by the participants, sometimes based on formal exit criteria. The most
important exit criterion is the average number of critical and/or major defects
found per page (e.g. no more than three critical/major defects per page). If the
number of defects found per page exceeds a certain level, the document must
be reviewed again, after it has been reworked. If the document complies with
the exit criteria, the document will be checked during follow-up by the moder-
ator or one or more participants. Subsequently, the document can leave the
review process.
If a project is under pressure, the moderator will sometimes be forced to
skip re-reviews and exit with a defect-prone document. Setting, and agreeing,
quantified exit level criteria helps the moderator to make firm decisions at
all times.
In addition to the number of defects per page, other exit criteria are used
that measure the thoroughness of the review process, such as ensuring that all
pages have been checked at the right rate. The average number of defects per
page is only a valid quality indicator if these process criteria are met.
Rework
Based on the defects detected, the author will improve the document under
review step by step. Not every defect that is found leads to rework. It is the
author's responsibility to judge if a defect has to be fixed. If nothing is done
about an issue for a certain reason, it should be reported to at least indicate that
the author has considered the issue.
Changes that are made to the document should be easy to identify during
follow-up. Therefore the author has to indicate where changes are made (e.g.
using 'Track changes' in word-processing software).
Follow-up
The moderator is responsible for ensuring that satisfactory actions have been
taken on all (logged) defects, process improvement suggestions and change
requests. Although the moderator checks to make sure that the author has
taken action on all known defects, it is not necessary for the moderator to check
all the corrections in detail. If it is decided that all participants will check the
updated document, the moderator takes care of the distribution and collects
f
the feedback. For more formal review types the moderator checks for compli-
ance to the exit criteria.
In order to control and optimize the review process, a number of measure-
ments are collected by the moderator at each step of the process. Examples of
such measurements include number of defects found, number of defects found
per page, time spent checking per page, total review effort, etc. It is the respon-
sibility of the moderator to ensure that the information is correct and stored for
future analysis.
The moderator
The moderator (or review leader) leads the review process. He or she deter-
mines, in co-operation with the author, the type of review, approach and the
composition of the review team. The moderator performs the entry check
and the follow-up on the rework, in order to control the quality of the input
and output of the review process. The moderator also schedules the
meeting, disseminates documents before the meeting, coaches other team
members, paces the meeting, leads possible discussions and stores the data
that is collected.
The author
As the writer of the document under review, the author's basic goal should
be to learn as much as possible with regard to improving the quality of the
document, but also to improve his or her ability to write future documents.
The author's task is to illuminate unclear areas and to understand the defects
found.
The scribe
During the logging meeting, the scribe (or recorder) has to record each defect
mentioned and any suggestions for process improvement. In practice it is often
the author who plays this role, ensuring that the log is readable and understand-
able. If authors record their own defects, or at least make their own notes in
their own words, it helps them to understand the log better during rework.
However, having someone other than the author take the role of the scribe (e.g.
the moderator) can have significant advantages, since the author is freed up to
think about the document rather than being tied down with lots of writing.
The reviewers
The task of the reviewers (also called checkers or inspectors) is to check any
material for defects, mostly prior to the meeting. The level of thoroughness
required depends on the type of review. The level of domain knowledge or tech-
nical expertise needed by the reviewers also depends on the type of review.
Reviewers should be chosen to represent different perspectives and roles in the
review process. In addition to the document under review, the material review-
ers receive includes source documents, standards, checklists, etc. In general, the
fewer source and reference documents provided, the more domain expertise
regarding the content of the document under review is needed.
The manager
The manager is involved in the reviews as he or she decides on the execution of
reviews, allocates time in project schedules and determines whether review
process objectives have been met. The manager will also take care of any review
training requested by the participants. Of course a manager can also be
involved in the review itself depending on his or her background, playing the
role of a reviewer if this would be helpful.
Walkthrough
A walkthrough is characterized by the author of the document under review
guiding the participants through the document and his or her thought
processes, to achieve a common understanding and to gather feedback. This is
especially useful if people from outside the software discipline are present, who
are not used to, or cannot easily understand software development documents.
The content of the document is explained step by step by the author, to reach
consensus on changes or to gather information.
Within a walkthrough the author does most of the preparation. The partici-
pants, who are selected from different departments and backgrounds, are not
required to do a detailed study of the documents in advance. Because of the
way the meeting is structured, a large number of people can participate and this
larger audience can bring a great number of diverse viewpoints regarding the
contents of the document being reviewed as well as serving an educational
purpose. If the audience represents a broad cross-section of skills and disci-
plines, it can give assurance that no major defects are 'missed' in the walk-
through. A walkthrough is especially useful for higher-level documents, such as
requirement specifications and architectural documents.
The specific goals of a walkthrough depend on its role in the creation of the
document. In general the following goals can be applicable:
• to present the document to stakeholders both within and outside the soft
ware discipline, in order to gather information regarding the topic under
documentation;
• to explain (knowledge transfer) and evaluate the contents of the docu
ment;
• to establish a common understanding of the document;
• to examine and discuss the validity of proposed solutions and the viability of
alternatives, establishing consensus.
Key characteristics of walkthroughs are:
• The meeting is led by the authors; often a separate scribe is present.
• Scenarios and dry runs may be used to validate the content.
• Separate pre-meeting preparation for reviewers is optional.
Technical review
A technical review is a discussion meeting that focuses on achieving con-
sensus about the technical content of a document. Compared to inspec-
tions, technical reviews are less formal and there is little or no focus on
defect identification on the basis of referenced documents, intended read-
ership and rules. During technical reviews defects are found by experts,
who focus on the content of the document. The experts that are needed for
a technical review are, for example, architects, chief designers and key
users. In practice, technical reviews vary from quite informal to very
formal.
The goals of a technical review are to:
• assess the value of technical concepts and alternatives in the product and
project environment;
• establish consistency in the use and representation of technical concepts;
• ensure, at an early stage, that technical concepts are used correctly;
• inform participants of the technical content of the document.
Key characteristics of a technical review are:
• It is a documented defect-detection process that involves peers and
technical experts.
• It is often performed as a peer review without management partici
pation.
• Ideally it is led by a trained moderator, but possibly also by a technical
expert.
• A separate preparation is carried out during which the product is examined
and the defects are found.
• More formal characteristics such as the use of checklists and a logging list or
issue log are optional.
Inspection
Inspection is the most formal review type. The document under inspection is
prepared and checked thoroughly by the reviewers before the meeting, compar-
ing the work product with its sources and other referenced documents, and
using rules and checklists. In the inspection meeting the defects found are
logged and any discussion is postponed until the discussion phase. This makes
the inspection meeting a very efficient meeting.
The reason for carrying out inspections can be explained by using
Weinberg's concept of egoless engineering [Weinberg, 1971]. Weinberg refers
to the human tendency to self-justify actions. Since we tend not to see evidence
that conflicts with our strong beliefs, our ability to find errors in our own work
is impaired. Because of this tendency, many engineering organizations have
established independent test groups that specialize in finding defects. Similar
principles have led to the introduction of inspections and reviews in general.
Depending on the organization and the objectives of a project, inspections
can be balanced to serve a number of goals. For example, if the time to market
is extremely important, the emphasis in inspections will be on efficiency. In a
safety-critical market, the focus will be on effectiveness.
The generally accepted goals of inspection are to:
• help the author to improve the quality of the document under inspection;
• remove defects efficiently, as early as possible;
• improve product quality, by producing documents with a higher level of
quality;
• create a common understanding by exchanging information among the
inspection participants;
• train new employees in the organization's development process;
• learn from defects found and improve processes in order to prevent recur
rence of similar defects;
• sample a few pages or sections from a larger document in order to measure
the typical quality of the document, leading to improved work by individuals
in the future, and to process improvements.
Key characteristics of an inspection are:
• It is usually led by a trained moderator (certainly not by the author).
• It uses defined roles during the process.
• It involves peers to examine the product.
• Rules and checklists are used during the preparation phase.
• A separate preparation is carried out during which the product is examined
and the defects are found.
• The defects found are documented in a logging list or issue log.
• A formal follow-up is carried out by the moderator applying exit criteria.
• Optionally, a causal analysis step is introduced to address process improve
ment issues and learn from the defects found.
• Metrics are gathered and analyzed to optimize the process.
3.2.4 Success factors for reviews
Implementing (formal) reviews is not easy as there is no one way to success and
there are numerous ways to fail. The next list contains a number of critical
success factors that improve the chances of success when implementing reviews.
It aims to answer the question, 'How do you start (formal) reviews?'.
Find a 'champion'
A champion is needed, one who will lead the process on a project or organiza-
tional level. They need expertise, enthusiasm and a practical mindset in order
to guide moderators and participants. The authority of this champion should be
clear to the entire organization. Management support is also essential for
success. They should, amongst other things, incorporate adequate time for
review activities in project schedules.
Train participants
It is important that training is provided in review techniques, especially the
more formal techniques, such as inspection. Otherwise the process is likely to
be impeded by those who don't understand the process and the reasoning
behind it. Special training should be provided to the moderators to prepare
them for their critical role in the review process.
Report results
Report quantified results and benefits to all those involved as soon as possible,
and discuss the consequences of defects if they had not been found this early.
Costs should of course be tracked, but benefits, especially when problems don't
occur in the future, should be made visible by quantifying the benefits as well as
the costs.
Just do it!
The process is simple but not easy. Each step of the process is clear, but expe-
rience is needed to execute them correctly. So, try to get experienced people to
observe and help where possible. But most importantly, start doing reviews and
start learning from every review.
C hapter 3 covered static testing, looking at documents and code, but not running the code we are
interested in. This chapter looks at dynamic testing, where the software we are interested in is run
by executing tests on the running code.
4.1.1 Introduction
Before we can actually execute a test, we need to know what we are trying to test, the inputs, the results
that should be produced by those inputs, and how we actually get ready for and run the tests.
In this section we are looking at three things: test conditions, test cases and test procedures (or scripts) -
they are described in the sections below. Each is specified in its own document, according to the Test
Documentation Standard [IEEE829].
Test conditions are documented in a Test Design Specification. We will look at how to choose test
conditions and prioritize them.
Test cases are documented in a Test Case Specification. We will look at how
to write a good test case, showing clear traceability to the test basis (e.g. the
requirement specification) as well as to test conditions.
Test procedures are documented (as you may expect) in a Test Procedure
Specification (also known as a test script or a manual test script). We will look
at how to translate test cases into test procedures relevant to the knowledge
of the tester who will be executing the test, and we will look at how to produce
a test execution schedule, using prioritization and technical and logical
dependencies.
In this section, look for the definitions of the glossary terms: test case, test
case specification, test condition, test data, test procedure
specification, test script and traceability.
The test case should also say why it exists - i.e. the objective of the test it is
part of or the test conditions that it is exercising (traceability). Test cases can
now be prioritized so that the most important test cases are executed first, and
low priority test cases are executed later, or even not executed at all. This may
reflect the priorities already established for test conditions or the priority may
be determined by other factors related to the specific test cases, such as a spe-
cific input value that has proved troublesome in the past, the risk associated
with the test, or the most sensible sequence of running the tests. Chapter 5 gives
more detail of risk-based testing.
Test cases need to be detailed so that we can accurately check the results and
know that we have exactly the right response from the system. If tests are to be
automated, the testing tool needs to know exactly what to compare the system
output to.
In this section we will look at the different types of test design technique, how
they are used and how they differ. The three types or categories are distin-
guished by their primary source: a specification, the structure of the system or
component, or a person's experience. All categories are useful and the three are
complementary.
In this section, look for the definitions of the glossary terms: black-box test
design techniques, experience-based test design techniques,
specification-based test design techniques, structure-based test
design techniques and white-box test design techniques.
4.2.1 Introduction
There are many different types of software testing technique, each with its own
strengths and weaknesses. Each individual technique is good at finding partic-
ular types of defect and relatively poor at finding other types. For example, a
technique that explores the upper and lower limits of a single input range is
more likely to find boundary value defects than defects associated with combi-
nations of inputs. Similarly, testing performed at different stages in the software
development life cycle will find different types of defects; component testing is
more likely to find coding logic defects than system design defects.
Each testing technique falls into one of a number of different categories.
Broadly speaking there are two main categories, static and dynamic. Static tech-
niques were discussed in Chapter 3. Dynamic techniques are subdivided into
three more categories: specification-based (black-box, also known as behavioral
techniques), structure-based (white-box or structural techniques) and experi-
ence-based. Specification-based techniques include both functional and non-
functional techniques (i.e. quality characteristics). The techniques covered in
the syllabus are summarized in Figure 4.1.
Notice that we have identified four partitions here, even though the specifica-
tion only mentions three. This illustrates a very important task of the tester - not
only do we test what is in our specification, but we also think about things that
haven't been specified. In this case we have thought of the situation where the
balance is less than zero. We haven't (yet) identified an invalid partition on the
right, but this would also be a good thing to consider. In order to identify where
the 7% partition ends, we would need to know what the maximum balance is for
this account (which may not be easy to find out). In our example we have left this
open for the time being. Note that non-numeric input is also an invalid partition
(e.g. the letter 'a') but we discuss only the numeric partitions for now.
We have made an assumption here about what the smallest difference is
between two values. We have assumed two decimal places, i.e. $100.00, but we
could have assumed zero decimal places (i.e. $100) or more than two decimal
places (e.g. $100.0000) In any case it is a good idea to state your assumptions -
then other people can see them and let you know if they are correct or not.
When designing the test cases for this software we would ensure that the
three valid equivalence partitions are each covered once, and we would also test
the invalid partition at least once. So for example, we might choose to calculate
the interest on balances of-$10.00, $50.00, $260.00 and $1348.00. If we hadn't
specifically identified these partitions, it is possible that at least one of them
could have been missed at the expense of testing another one several times
over. Note that we could also apply equivalence partitioning to outputs as well.
In this case we have three interest rates: 3%, 5% and 7%, plus the error
message for the invalid partition (or partitions). In this example, the output par-
titions line up exactly with the input partitions.
How would someone test this without thinking about the partitions? A naive
tester (let's call him Robbie) might have thought that a good set of tests would
be to test every $50. That would give the following tests: $50.00, $100.00,
$150.00, $200.00, $250.00, ... say up to $800.00 (then Robbie would have got
tired of it and thought that enough tests had been carried out). But look at what
Robbie has tested: only two out of four partitions! So if the system does not cor-
rectly handle a negative balance or a balance of $1000 or more, he would not
have found these defects - so the naive approach is less effective than equiva-
lence partitioning. At the same time, Robbie has four times more tests (16 tests
versus our four tests using equivalence partitions), so he is also much less effi-
cient! This is why we say that using techniques such as this makes testing both
more effective and more efficient.
Note that when we say a partition is 'invalid', it doesn't mean that it repre-
sents a value that cannot be entered by a user or a value that the user isn't sup-
posed to enter. It just means that it is not one of the expected inputs for this
particular field. The software should correctly handle values from the invalid
partition, by replying with an error message such as 'Balance must be at least
$0.00'.
Note also that the invalid partition may be invalid only in the context of crediting
interest payments. An account that is overdrawn will require some different action.
To apply boundary value analysis, we will take the minimum and maximum
(boundary) values from the valid partition (1 and 99 in this case) together with
copies to be made, from 1 to 99.
the first or last value respectively in each of the invalid partitions adjacent to the
valid partition (0 and 100 in this case). In this example we would have three
equivalence partitioning tests (one from each of the three partitions) and four
boundary value tests.
Consider the bank system described in the section about equivalence parti-
tioning.
Because the boundary values are defined as those values on the edge of a
partition, we have identified the following boundary values: -$0.01 (an invalid
boundary value because it is at the edge of an invalid partition), $0.00, $100.00,
$100.01, $999.99 and $1000.00, all valid boundary values.
So by applying boundary value analysis we will have six tests for boundary
values. Compare what our naive tester Robbie had done: he did actually hit one
of the boundary values ($100) though it was more by accident than design. So
in addition to testing only half of the partitions, Robbie has only tested one-
sixth of the boundaries (so he will be less effective at finding any boundary
defects). If we consider all of our tests for both equivalence partitioning and
boundary value analysis, the techniques give us a total of nine tests, compared
to the 16 that Robbie had, so we are still considerably more efficient as well as
being over three times more effective (testing four partitions and six bound-
aries, so 10 conditions in total compared to three).
Note that in the bank interest example, we have valid partitions next to other
valid partitions. If we were to consider an invalid boundary for the 3% interest
rate, we have -$0.01, but what about the value just above $100.00? The value of
$100.01 is not an invalid boundary; it is actually a valid boundary because it falls
into a valid partition. So the partition for 5%, for example, has no invalid
boundary values associated with partitions next to it.
A good way to represent the valid and invalid partitions and boundaries is in
a table such as Table 4.1:
Test conditions Valid partitions Invalid partitions Valid boundaries Invalid boundaries
Balance in $0.00 $100.00 < $0.00 $0.00 -$0.01
aU Unt
° $100.01-$999.99 >$Max $100.00 $Max+0.01
$1000.00
$Max
5% Non-integer
7% No interest calculated
By showing the values in the table, we can see that no maximum has been
specified for the 7% interest rate. We would now want to know what the
maximum value is for an account balance, so that we can test that boundary.
This is called an 'open boundary', because one of the sides of the partition is left
open, i.e. not defined. But that doesn't mean we can ignore it - we should still
try to test it, but how?
Open boundaries are more difficult to test, but there are ways to approach
them. Actually the best solution to the problem is to find out what the bound-
ary should be specified as! One approach is to go back to the specification to
see if a maximum has been stated somewhere else for a balance amount. If
so, then we know what our boundary value is. Another approach might be to
investigate other related areas of the system. For example, the field that
holds the account balance figure may be only six figures plus two decimal
figures. This would give a maximum account balance of $999 999.99 so we
could use that as our maximum boundary value. If we really cannot find any-
thing about what this boundary should be, then we probably need to use an
intuitive or experience-based approach to probe various large values trying
to make it fail.
We could also try to find out about the lower open boundary - what is the
lowest negative balance? Although we have omitted this from our example,
setting it out in the table shows that we have omitted it, so helps us be more
thorough if we wanted to be.
Representing the partitions and boundaries in a table such as this also
makes it easier to see whether or not you have tested each one (if that is your
objective).
Suppose we test only the valid boundary values 1 and 99 and nothing in
between. If both tests pass, this seems to indicate that all the values in between
should also work. However, suppose that one page prints correctly, but 99 pages
do not. Now we don't know whether any set of more than one page works, so
the first thing we would do would be to test for say 10 pages, i.e. a value from
the equivalence partition.
We recommend that you test the partitions separately from boundaries - this
means choosing partition values that are NOT boundary values.
However, if you use the three-value boundary value approach, then you
would have valid boundary values of 1, 2, 98 and 99, so having a separate equiv-
alence value in addition to the extra two boundary values would not give much
additional benefit. But notice that one equivalence value, e.g. 10, replaces both
of the extra two boundary values (2 and 98). This is why equivalence partition-
ing with two-value boundary value analysis is more efficient than three-value
boundary value analysis.
Which partitions and boundaries you decide to exercise (you don't need
to test them all), and which ones you decide to test first, depends on your test
objectives. If your goal is the most thorough approach, then follow the pro-
cedure of testing valid partitions first, then invalid partitions, then valid
boundaries and finally invalid boundaries. However if you are under time
pressure and cannot test everything (and who isn't?), then your test objec-
tives will help you decide what to test. If you are after user confidence of
typical transactions with a minimum number of tests, you may do valid par-
titions only. If you want to find as many defects as possible as quickly as pos-
sible, you may start with boundary values, both valid and invalid. If you want
confidence that the system will handle bad inputs correctly, you may do
mainly invalid partitions and boundaries. Your previous experience of types
of defects found can help you find similar defects; for example if there are
typically a number of boundary defects, then you would start by testing
boundaries.
Equivalence partitioning and boundary value analysis are described in most
testing books, including [Myers, 1979] and [Copeland, 2003]. Examples of types
of equivalence classes to look out for are given in [Kaner et al., 1993]
Equivalence partitioning and boundary value analysis are described in BS7925-
2, including designing tests and measuring coverage.
Next we will identify all of the combinations of True and False (see Table
4.3). With two conditions, each of which can be True or False, we will have four
combinations (two to the power of the number of things to be combined). Note
that if we have three things to combine, we will have eight combinations, with
four things, there are 16, etc. This is why it is good to tackle small sets of com-
binations at a time. In order to keep track of which combinations we have, we
will alternate True and False on the bottom row, put two Trues and then two
Falses on the row above the bottom row, etc., so the top row will have all Trues
and then all Falses (and this principle applies to all such tables).
TABLE 4.3 Decision table with input combinations
The next step (at least for this example) is to identify the correct outcome for
each combination (see Table 4.4). In this example, we can enter one or both of
the two fields. Each combination is sometimes referred to as a rule.
TABLE 4. 4 Decision table with combinations and outcomes
Actions/Outcomes
Process term Y Y
At this point, we may realize that we hadn't thought about what happens
if the customer doesn't enter anything in either of the two fields. The table
has highlighted a combination that was not mentioned in the specification
for this example. We could assume that this combination should result in an
error message, so we need to add another action (see Table 4.5). This high-
lights the strength of this technique to discover omissions and ambiguities in
specifications. It is not unusual for some combinations to be omitted from
specifications; therefore this is also a valuable technique to use when review-
ing the test basis.
TABLE 4 . 5 Decision table with additional outcomes
entered
Actions/Outcomes
Process term Y Y
Error message Y
Suppose we change our example slightly, so that the customer is not allowed
to enter both repayment and term. Now our table will change, because there
should also be an error message if both are entered, so it will look like Table 4.6.
TABLE 4 . 6 Decision table with changed outcomes
entered
Actions/Outcomes
Process term Y
Error message Y Y
You might notice now that there is only one 'Yes' in each column, i.e. our
actions are mutually exclusive - only one action occurs for each combination of
conditions. We could represent this in a different way by listing the actions in
the cell of one row, as shown in Table 4.7. Note that if more than one action
results from any of the combinations, then it would be better to show them as
separate rows rather than combining them into one row.
TABLE 4 . 7 Decision table with outcomes in one row
entered
Actions/Outcomes
The final step of this technique is to write test cases to exercise each of the
four rules in our table.
In this example we started by identifying the input conditions and then iden-
tifying the outcomes. However in practice it might work the other way around
- we can see that there are a number of different outcomes, and have to work
back to understand what combination of input conditions actually drive those
outcomes. The technique works just as well doing it in this way, and may well
be an iterative approach as you discover more about the rules that drive the
system.
In Table 4.8, the conditions and actions are listed in the left hand column.
All the other columns in the decision table each represent a separate rule,
one for each combination of conditions. We may choose to test each
rule/combination and if there are only a few this will usually be the case.
However, if the number of rules/combinations is large we are more likely to
sample them by selecting a rich subset for testing.
Note that we have put X for the discount for two of the columns (Rules 1 and
2) - this means that this combination should not occur. You cannot be both a
new customer and already hold a loyalty card! There should be an error
message stating this, but even if we don't know what that message should be, it
will still make a good test.
We have made an assumption in Rule 3. Since the coupon has a greater
discount than the new customer discount, we assume that the customer will
choose 20% rather than 15%. We cannot add them, since the coupon cannot
be used with the 'new customer' discount. The 20% action is an assumption
on our part, and we should check that this assumption (and any other
assumptions that we make) is correct, by asking the person who wrote the
specification or the users.
For Rule 5, however, we can add the discounts, since both the coupon and
the loyalty card discount should apply (at least that's our assumption).
Rules 4, 6 and 7 have only one type of discount and Rule 8 has no discount.
so 0%.
If we are applying this technique thoroughly, we would have one test for each
column or rule of our decision table. The advantage of doing this is that we may
test a combination of things that otherwise we might not have tested and that
could find a defect.
However, if we have a lot of combinations, it may not be possible or sensible
to test every combination. If we are time-constrained, we may not have time to
test all combinations. Don't just assume that all combinations need to be tested;
it is better to prioritize and test the most important combinations. Having the
full table enables us to see which combinations we decided to test and which not
to test this time.
There may also be many different actions as a result of the combinations
of conditions. In the example above we just had one: the discount to be
applied. The decision table shows which actions apply to each combination
of conditions.
In the example above all the conditions are binary, i.e. they have only two
possible values: True or False (or, if you prefer Yes or No). Often it is the case
that conditions are more complex, having potentially many possible values.
Where this is the case the number of combinations is likely to be very large, so
the combinations may only be sampled rather than exercising all of them.
where the 'coverage item' is whatever we have been able to count and see
whether a test has exercised or used this item.
There is danger in using a coverage measure. 100% coverage does not mean
100% tested! Coverage techniques measure only one dimension of a multi-dimen-
sional concept. Two different test cases may achieve exactly the same coverage but
the input data of one may find an error that the input data of the other doesn't.
One drawback of code coverage measurement is that it measures coverage
of what has been written, i.e. the code itself; it cannot say anything about the
software that has not been written. If a specified function has not been imple-
mented, specification-based testing techniques will reveal this. If a function was
omitted from the specification, then experience-based techniques may find it.
But structure-based techniques can only look at a structure which is already
there.
Types of coverage
Test coverage can be measured based on a number of different structural ele-
ments in a system or component. Coverage can be measured at component-
testing level, integration-testing level or at system- or acceptance-testing levels.
For example, at system or acceptance level, the coverage items may be require-
ments, menu options, screens, or typical business transactions. Other coverage
measures include things such as database structural elements (records, fields
and sub-fields) and files. It is worth checking for any new tools, as the test tool
market develops quite rapidly.
At integration level, we could measure coverage of interfaces or specific
interactions that have been tested. The call coverage of module, object or pro-
cedure calls can also be measured (and is supported by tools to some extent).
We can measure coverage for each of the specification-based techniques as
well:
• EP: percentage of equivalence partitions exercised (we could measure valid
and invalid partition coverage separately if this makes sense);
• BVA: percentage of boundaries exercised (we could also separate valid and
invalid boundaries if we wished);
• Decision tables: percentage of business rules or decision table columns
tested;
• State transition testing: there are a number of possible coverage measures:
- Percentage of states visited
- Percentage of (valid) transitions exercised (this is known as Chow's 0-
switch coverage)
- Percentage of pairs of valid transitions exercised ('transition pairs' or
Chow's 1-switch coverage) - and longer series of transitions, such as tran
sition triples, quadruples, etc.
- Percentage of invalid transitions exercised (from the state table)
The coverage measures for specification-based techniques would apply at
whichever test level the technique has been used (e.g. system or component
level).
When coverage is discussed by business analysts, system testers or users, it
most likely refers to the percentage of requirements that have been tested by a
set of tests. This may be measured by a tool such as a requirements manage-
ment tool or a test management tool.
However, when coverage is discussed by programmers, it most likely refers
to the coverage of code, where the structural elements can be identified using a
tool, since there is good tool support for measuring code coverage. We will
cover statement and decision coverage shortly.
Statements and decision outcomes are both structures that can be measured
in code and there is good tool support for these coverage measures. Code cov-
erage is normally done in component and component integration testing - if it
is done at all. If someone claims to have achieved code coverage, it is important
to establish exactly what elements of the code have been covered, as statement
coverage (often what is meant) is significantly weaker than decision coverage or
some of the other code-coverage measures.
Although it isn't completely correct, we have numbered each line and will regard
each line as a statement. (Some tools may group statements that would always be
executed together in a basic block which is regarded as a single statement.) However,
we will just use numbered lines to illustrate the principles of coverage of statements
(lines). Let's analyze the coverage of a set of tests on our six-statement program:
TEST SET 1 Test 1_1: A
= 2, B = 3 Test 1_2: A =
0, B = 25 Test 1_3: A =
47, B = 1
Which statements have we covered?
• In Test 1_1, the value of C will be 8, so we will cover the statements on lines
1 to 4 and line 6.
• In Test 1_2, the value of C will be 50, so we will cover exactly the same state
ments as Test 1_1.
• In Test 1_3, the value of C will be 49, so again we will cover the same state
ments.
Since we have covered five out of six statements, we have 83% statement
coverage (with three tests). What test would we need in order to cover state-
ment 5, the one statement that we haven't exercised yet? How about this one:
Test 1_4: A = 20, B = 25
This time the value of C is 70, so we will print 'Large C and we will have
exercised all six of the statements, so now statement coverage = 100%. Notice
that we measured coverage first, and then designed a test to cover the statement
that we had not yet covered.
Note that Test 1_4 on its own is more effective- (towards our goal of achiev-
ing 100% statement coverage) than the first three tests together. Just taking
Test 1_4 on its own is also more efficient than the set of four tests, since it has
used only one test instead of four. Being more effective and more efficient is
the mark of a good test technique.
What feels like reasonably thorough functional testing may achieve only
40% to 60% decision coverage. Typical ad hoc testing may cover only 20% of
the decisions, leaving 80% of the possible outcomes untested. Even if your
testing seems reasonably thorough from a functional or specification-based
perspective, you may have only covered two-thirds or three-quarters of the
decisions. Decision coverage is stronger than statement coverage. It 'sub-
sumes' statement coverage - this means that 100% decision coverage always
guarantees 100% statement coverage. Any stronger coverage measure may
require more test cases to achieve 100% coverage. For example, consider
code sample 4.1 again.
We saw earlier that just one test case was required to achieve 100% state-
ment coverage. However, decision coverage requires each decision to have had
both a True and False outcome. Therefore, to achieve 100% decision coverage,
a second test case is necessary where A is less than or equal to B. This will
ensure that the decision statement 'IF A > B' has a False outcome. So one test
is sufficient for 100% statement coverage, but two tests are needed for 100%
decision coverage. Note that 100% decision coverage guarantees 100% state-
ment coverage, but not the other way around!
1 READ A
2 READ B
3 C=A-2*B
4 IFC <0THEN
5 PRINT "C negative"
6 ENDIF
Code sample 4.3
Let's suppose that we already have the following test, which gives us 100%
statement coverage for code sample 4.3.
TEST SET 2
Test 2_1: A = 20, B = 15
Which decision outcomes have we exercised with our test? The value of C is
-10, so the condition 'C < 0' is True, so we will print 'C negative' and we have
exercised the True outcome from that decision statement. But we have not exer-
cised the decision outcome of False. What other test would we need to exercise
the False outcome and to achieve 100% decision coverage?
Before we answer that question, let's have a look at another way to represent
this code. Sometimes the decision structure is easier to see in a control flow
diagram (see Figure 4.4).
The dotted line shows where Test 2_1 has gone and clearly shows that we
haven't yet had a test that takes the False exit from the IF statement.
Let's modify our existing test set by adding another test:
TEST SET 2
Test 2_1: A = 20, B = 15
Test 2_2: A = 10, B = 2
This now covers both of the decision outcomes, True (with Test 2_1) and
False (with Test 2_2). If we were to draw the path taken by Test 2_2, it would be
a straight line from the read statement down the False exit and through the
ENDIF. Note that we could have chosen other numbers to achieve either the
True or False outcomes.
4 . 5 EXPERIENCE-BASED TECHNIQUES
1 Recall reasons for writing test cases based on intuition, experience and
knowledge about common defects. (Kl)
2 Compare experience-based techniques with specification-based testing
techniques. (K2)
In this section we will look at two experience-based techniques, why and when
they are useful, and how they fit with specification-based techniques.
Although it is true that testing should be rigorous, thorough and systematic,
this is not all there is to testing. There is a definite role for non-systematic tech-
niques, i.e. tests based on a person's knowledge, experience, imagination and
intuition. The reason is that some defects are hard to find using more system-
atic approaches, so a good 'bug hunter' can be very creative at finding those
elusive defects.
In this section, look for the definitions of the glossary terms: error guessing
and exploratory testing.
4.5.1 Error guessing
Error guessing is a technique that should always be used as a complement to
other more formal techniques. The success of error guessing is very much
dependent on the skill of the tester, as good testers know where the defects are
most likely to lurk. Some people seem to be naturally good at testing and others
are good testers because they have a lot of experience either as a tester or
working with a particular system and so are able to pin-point its weaknesses.
This is why an error-guessing approach, used after more formal techniques have
been applied to some extent, can be very effective. In using more formal tech-
niques, the tester is likely to gain a better understanding of the system, what it
does and how it works. With this better understanding, he or she is likely to be
better at guessing ways in which the system may not work properly.
There are no rules for error guessing. The tester is encouraged to think of
situations in which the software may not be able to cope. Typical conditions to
try include division by zero, blank (or no) input, empty files and the wrong kind
of data (e.g. alphabetic characters where numeric are required). If anyone ever
says of a system or the environment in which it is to operate 'That could never
happen', it might be a good idea to test that condition, as such assumptions
about what will and will not happen in the live environment are often the cause
of failures. A structured approach to the error-guessing technique is to list pos-
sible defects or failures and to design tests that attempt to produce them. These
defect and failure lists can be built based on the tester's own experience or that
of other people, available defect and failure data, and from common knowledge
about why software fails.
In this final section we will look at the factors that go into the decision about
which techniques to use when.
Which technique is best? This is the wrong question! Each technique is
good for certain things, and not as good for other things. For example, one
of the benefits of structure-based techniques is that they can find things in
the code that aren't supposed to be there, such as 'Trojan horses' or other
malicious code. However, if there are parts of the specification that are
missing from the code, only specification-based techniques will find that -
structure-based techniques can only test what is there. If there are things
missing from the specification and from the code, then only experience-
based techniques would find them. Each individual technique is aimed at
particular types of defect as well. For example, state transition testing is
unlikely to find boundary defects.
The choice of which test techniques to use depends on a number of factors,
including the type of system, regulatory standards, customer or contractual
requirements, level of risk, type of risk, test objective, documentation available,
knowledge of the testers, time and budget, development life cycle, use case
models and previous experience of types of defects found.
Some techniques are more applicable to certain situations and test levels:
others are applicable to all test levels.
This chapter has covered the most popular and commonly used software
testing techniques. There are many others that fall outside the scope of the
Syllabus that this book is based on. With so many testing techniques to choose
from how are testers to decide which ones to use?
Perhaps the single most important thing to understand is that the best testing
technique is no single testing technique. Because each testing technique is good
at finding one specific class of defect, using just one technique will help ensure
that many (perhaps most but not all) defects of that particular class are found.
Unfortunately, it may also help to ensure that many defects of other classes are
missed! Using a variety of techniques will therefore help ensure that a variety
of defects are found, resulting in more effective testing.
So how can we choose the most appropriate testing techniques to use? The
decision will be based on a number of factors, both internal and external.
The internal factors that influence the decision about which technique to use
are:
• Models used - Since testing techniques are based on models, the models
available (i.e. developed and used during the specification, design and
implementation of the system) will to some extent govern which testing
techniques can be used. For example, if the specification contains a state
transition diagram, state transition testing would be a good technique to
use.
• Tester knowledge I experience - How much testers know about the system and
about testing techniques will clearly influence their choice of testing tech
niques. This knowledge will in itself be influenced by their experience of
testing and of the system under test.
• Likely defects - Knowledge of the likely defects will be very helpful in choos
ing testing techniques (since each technique is good at finding a particular
type of defect). This knowledge could be gained through experience of
testing a previous version of the system and previous levels of testing on the
current version.
• Test objective - If the test objective is simply to gain confidence that the soft
ware will cope with typical operational tasks then use cases would be a sen
sible approach. If the objective is for very thorough testing then more
rigorous and detailed techniques (including structure-based techniques)
should be chosen.
• Documentation - Whether or not documentation (e.g. a requirements
specification) exists and whether or not it is up to date will affect the
choice of testing techniques. The content and style of the documentation
will also influence the choice of techniques (for example, if decision tables
or state graphs have been used then the associated test techniques should
be used).
• Life cycle model - A sequential life cycle model will lend itself to the use of
more formal techniques whereas an iterative life cycle model may be better
suited to using an exploratory testing approach.
The external factors that influence the decision about which technique to use
are:
• Risk - The greater the risk (e.g. safety-critical systems), the greater the need
for more thorough and more formal testing. Commercial risk may be influ
enced by quality issues (so more thorough testing would be appropriate) or
by time-to-market issues (so exploratory testing would be a more appropri
ate choice).
• Customer I contractual requirements - Sometimes contracts specify par
ticular testing techniques to use (most commonly statement or branch
coverage).
• Type of system - The type of system (e.g. embedded, graphical, financial, etc.)
will influence the choice of techniques. For example, a financial application
involving many calculations would benefit from boundary value analysis.
• Regulatory requirements - Some industries have regulatory standards or
guidelines that govern the testing techniques used. For example, the aircraft
industry requires the use of equivalence partitioning, boundary value analy
sis and state transition testing for high integrity systems together with state
ment, decision or modified condition decision coverage depending on the
level of software integrity required.
• Time and budget - Ultimately how much time there is available will always
affect the choice of testing techniques. When more time is available we can
afford to select more techniques and when time is severely limited we will be
limited to those that we know have a good chance of helping us find just the
most important defects.
CHAPTER REVIEW
Let's review what you have learned in this chapter.
From Section 4.1, you should now be able to differentiate
between a test condition, a test case and a test procedure, and
know that they are documented in a test design specification, a
test case specification and a test procedure specification
respectively. You should be able to write test cases that include
expected results and that show clear traceability to the test basis
(e.g. requirements). You should be able to translate test cases into
a test procedure specification at the appropriate level of detail for
testers and you should be able to write a test execution schedule
for a given set of test cases that takes into account priorities as
well as technical and logical dependencies. You should know the
glossary terms test cases, test case specification, test condition,
test data, test procedure specification, test script and
traceability.
From Section 4.2 (categories of test design techniques), you
should be able to give reasons why both specification-based
(black-box) and structure-based (white-box) approaches are useful,
and list the common techniques for each of these approaches. You
should be able to explain the characteristics and differences
between specification-based, structure-based and experience-based
techniques. You should know the glossary terms black-box test
design techniques, experience-based test design techniques,
specification-based test design techniques, structure-based test
design techniques and white-box test design techniques.
From Section 4.3, you should be able to write test cases from
given software models using equivalence partitioning, boundary
value analysis, decision tables and state transition diagrams. You
should understand the main purpose of each of these four
techniques, what level and type of testing could use each tech-
nique and how coverage can be measured for each of them. You
should also understand the concept and benefits of use case
testing. You should know the glossary terms boundary value
analysis, decision table testing, equivalence partitioning, state
transition testing and use case testing.
From Section 4.4, you should be able to describe the concept and
importance of code coverage. You should be able to explain the
concepts of statement and decision coverage and understand that
these concepts can also be used at test levels other than
component testing (such as business procedures at system test
level). You should be able to write test cases from given control
flows using statement testing and decision testing, and you should
be able to assess statement and decision coverage for
completeness. You should know the glossary terms code coverage,
decision coverage, statement coverage, structural testing,
structure-based testing and white-box testing.
From Section 4.5, you should be able to remember the reasons
for writing test cases based on intuition, experience and
knowledge about common defects and you should be able to
compare experience-based techniques with specification-based
techniques. You should know the glossary terms error guessing
and exploratory testing.
From Section 4.6, you should be able to list the factors that
influence the selection of the appropriate test design technique
for a particular type of problem, such as the type of system, risk,
customer requirements, models for use case modeling,
requirements models or testing knowledge.
SAMPLE EXAM QUESTIONS
Question 1 In which document described in IEEE Question 4 Why are both specification-based and
829 would you find instructions for the steps to be structure-based testing techniques useful?
taken for a test including set-up, logging, a. They find different types of defect.
environment and measurement?
b. Using more techniques is always better.
a. Test plan
c. Both find the same types of defect.
b. Test design specification
d. Because specifications tend to be unstructured.
c. Test case specification
d. Test procedure specification Question 5 What is a key characteristic of
structure-based testing techniques?
Question 2 With a highly experienced tester with a. They are mainly used to assess the structure of a
a good business background, which approach to specification.
defining test procedures would be effective and
most efficient for a project under severe time b. They are used both to measure coverage and to
pressure? design tests to increase coverage.
a. A high-level outline of the test conditions and c. They are based on the skills and experience of
general steps to take. the tester.
b. Every step in the test spelled out in detail. d. They use a formal or informal model of the
software or component.
c. A high-level outline of the test conditions with
the steps to take discussed in detail with another
Question 6 Which of the following would be an
experienced tester. example of decision-table testing for a financial
d. Detailed documentation of all test cases and application applied at the system-test level?
careful records of each step taken in the testing. a. A table containing rules for combinations of
inputs to two fields on a screen.
Question 3 Put the test cases that implement the b. A table containing rules for interfaces between
following test conditions into the best order for the components.
test execution schedule, for a test that is checking
c. A table containing rules for mortgage
modifications of customers on a database.
applications.
1 Print modified customer record.
d. A table containing rules for chess.
2 Change customer address: house number and
street name.
Question 7 Which of the following could be a
3 Capture and print the on-screen error message. coverage measure for state transition testing?
4 Change customer address: postal code. V All states have been reached.
5 Confirm existing customer is on the database by W The response time for each transaction is
opening that record. adequate.
6 Close the customer record and close the X Every transition has been exercised.
database.
Y All boundaries have been exercised.
7 Try to add a new customer with no details at all.
Z Specific sequences of transitions have been
a. 5,4, 2,1, 3, 7, 6 exercised.
b. 4,2,5,1,6,7,3 a. X,YandZ
c. 5,4,2,1,7,3,6 b. V, X, Y and Z
d. 5,1, 2, 3,4, 7, 6 c. W,XandY
d. V, X and Z
Question 8 Postal rates for 'light letters' are 25p Q Making sure that the mainstream business
up to l0g, 35p up to 50g plus an extra l0p for each processes are tested.
additional 25g up to l00g. R Finding defects in the interaction between
Which test inputs (in grams) would be selected components.
using equivalence partitioning? S Identifying the maximum and minimum values
a. 8,42,82,102 for every input field. T Identifying the
b. 4,15, 65, 92,159 percentage of statements
c. 10,50,75,100 exercised by a sets of tests.
d. 5, 20, 40, 60, 80 a. P, Q and R
b. Q,SandT
Question 9 Which of the following could be used c. P, Qand S
to assess the coverage achieved for specification-
based (black-box) test techniques? d. R, S and T
V Decision outcomes exercised
Question 12 Which of the following statements
W Partitions exercised about the relationship between statement coverage
X Boundaries exercised and decision coverage is correct?
Y State transitions exercised a. 100% decision coverage is achieved if statement
Z Statements exercised coverage is greater than 90%.
a. V,W,YorZ b. 100% statement coverage is achieved if decision
b. W,XorY coverage is greater than 90%.
c. V,XorZ c. 100% decision coverage always means 100%
statement coverage.
d. W, X, Y or Z
d. 100% statement coverage always means 100%
decision coverage.
Question 10 Which of the following structure-
based test design technique would be most
likely to be applied to? Question 13 If you are flying with an economy
ticket, there is a possibility that you may get
1 Boundaries between mortgage interest rate
upgraded to business class, especially if you hold a
bands.
gold card in the airline's frequent flier program. If
2 An invalid transition between two different you don't hold a gold card, there is a possibility
arrears^ statuses. that you will get 'bumped' off the flight if it is full
3 The business process flow for mortgage and you check in late. This is shown in Figure 4.5.
approval. Note that each box (i.e. statement) has been
4 Control flow of the program to calculate numbered.
repayments. Three tests have been run:
a. 2, 3 and 4 Test 1: Gold card holder who gets upgraded to
b. 2 and 4 business class
c. 3 and 4 Test 2: Non-gold card holder who stays in economy
d. 1, 2 and 3 Test 3: A person who is bumped from the flight
What is the statement coverage of these three tests?
Question 11 Use case testing is useful for which of a. 60%
the following? b. 70%
P Designing acceptance tests with users or c. 80%
customers.
d. 90%
W regulatory standards that apply
X the type of test execution tool that will be used
Y the documentation available
Z previous experience in the development
language
a. V,W,YandZ
b. U, V, W and Y
c. U,XandY
d. V, W and Y
EP/BVA exercise
The first thing to do is to establish exactly what the boundaries are between the full fare and saver fare.
Let's put these in a table to organize our thoughts:
We have assumed that the boundary values are: 9:29 am, 9:30 am, 4:00 pm, 4:01 pm, 7:30 pm and 7:31
pm. By setting out exactly what we think is meant by the specification, we may highlight some ambiguities
or, at least, raise some questions - this is one of the benefits of using the technique! For example:
'When does the morning rush hour start? At midnight? At 11:30 pm the previous day? At the time of
the first train of the day? If so, when is the first train? 5:00 am?'
This is a rather important omission from the specification. We could make an assumption about when it
starts, but it would be better to find out what is correct.
• If a train is due to leave at exactly 4:00 pm, is a saver ticket still valid?
• What if a train is due to leave before 4:00 pm but is delayed until after 4:00 pm? Is a saver ticket still
valid? (i.e. if the actual departure time is different to the scheduled departure time)
Our table above has helped us to see where the partitions are. All of the partitions in the table above
are valid partitions. It may be that an invalid partition would be a time that no train was running, e.g. before
5:00 am, but our specification didn't mention that! However it would be good to show this possibility also.
We could be a bit more formal by listing all valid and invalid partitions and boundaries in a table, as we
described in Section 4.3.1, but in this case it doesn't actually add a lot, since all partitions are valid.
Here are the test cases we can derive for this example:
Note that test cases 1, 4, 7 and 10 are based on equivalence partition values; test cases 2, 3, 5, 6, 8 and 9
are based on boundary values. There may also be other information about the test cases, such as precondi-
tions, that we have not shown here.
Decision table exercise
The fare types mentioned include an 'over 60s' rail card, a family rail card, and whether you are traveling
with a child or not. With three conditions or causes, we have eight columns in our decision table below.
When we come to fill in the effects, we may find this a bit more difficult. For the first two rules, for
example, what should the output be? Is it an X because holding more than one rail card should not be pos-
sible? The specification doesn't actually say what happens if someone does hold more than one card, i.e. it
has not specified the output, so perhaps we should put a question mark in this column. Of course, if
someone does hold two rail cards, they probably wouldn't admit this, and perhaps they would claim the
50% discount with their family rail card if they are traveling with a child, so perhaps we should put 50%
for Rule 1 and 34% for Rule 2 in this column. Our notation shows that we don't know what the expected
outcome should be for these rules!
This highlights the fact that our natural language (English) specification is not very clear as to what the
effects should actually be. A strength of this technique is that it forces greater clarity. If the answers are
spelled out in a decision table, then it is clear what the effect should be. When different people come up
with different answers for the outputs, then you have an unclear specification!
The word 'otherwise' in the specification is ambiguous. Does 'otherwise' mean that you always get at
least a 10% discount or does it mean that if you travel with a child and an over 60s card but not a family
card you get 10% and 34%? Depending on what assumption you make for the meaning of 'otherwise', you
will get a different last row in your decision table.
Note that the effect or output is the same (34%) for both Rules 3 and 4. This means that our third cause
(whether or not a child is also traveling) actually has no influence on the output. These columns could
therefore be combined with 'don't care' as the entry for the third cause. This 'rationalizing' of the table
means we will have fewer columns and therefore fewer test cases. The reduction in test cases is based on
the assumption we are making about the factor having no effect on the outcome, so a more thorough
approach would be to include each column in the table.
Here is a rationalized table, where we have shown our assumptions about the first two outcomes and we
have also combined Rules 6 and 8 above, since having a family rail card has no effect if you are not travel-
ing with a child.
Here are the test cases that we derive from this table. (If you didn't rationalize the table, then you will
have eight test cases rather than six.) Note that you wouldn't necessarily test each column, but the table
enables you to make a decision about which combinations to test and which not to test this time.
Note that we may have raised some additional issues when we designed the test cases. For example, does
the discount for a rail card apply only to the traveler or to someone traveling with them? Here we have
assumed that it applies to all travelers for the family rail card, but to the individual passenger only for the
over 60s rail card.
Although our example is not interested in what happens from State 4, there would be other events and
actions once we enter the payment process that could be shown by another state diagram (e.g. check valid-
ity of the credit card, deduct the amount, email a receipt, etc.).
The corresponding state table is:
All of the boxes that contain - are invalid transitions in this example. Example negative tests would
include:
• attempt to add an item from the summary and cost state (S3)
• try to remove an item from the empty shopping basket (SI)
• try to enter 'OK' while in the Shopping state (S2).
Statement and decision testing exercise
The control flow diagram is shown in Figure 4.8. Note that drawing a control diagram here illustrates
that structural testing can also be applied to the structure of general processes, not just to computer
algorithms. Flowcharts are generally easier to understand than text when you are trying to describe the
results of decisions taken on later events.
On Figure 4.9, we can see the route that Tests 1 and 2 have taken through our control flow graph.
Test 1 has gone straight down the left-hand side to select a cold drink. Test 2 has gone to the right at each
opportunity, adding both milk and sugar to a hot drink.
Every statement (represented by a box on the diagram) has been covered by our two tests, so we
have 100% statement coverage.
We have not taken the No exit from either the 'milk?' or 'sugar?' decisions, so there are two decision
outcomes that we have not tested yet. We did test both of the outcomes from the 'hot or cold?' decision,
so we have covered four out of six decision outcomes. Decision coverage is 4/6 or 67% with the two
tests.
No additional tests are needed to achieve statement coverage, as we already have 100% coverage of
the statements.
One additional test is needed to achieve 100% decision coverage:
Test 3: Hot drink, no milk, no sugar
This test will cover both of the 'No' decision outcomes from the milk and sugar decisions, so we will
now have 100% decision coverage.
CHAPTER FIVE
Test management
T esting is a complex activity. Testing is often a distinct sub-project within the larger software
development, maintenance, or integration project. Testing usually accounts for a substantial
proportion of the overall project budget. Therefore, we must understand how we should manage the
testing we do.
In this chapter, we cover essential topics for test management in six sections. The first relates to how to
organize the testers and the testing. The second concerns the estimation, planning and strategizing of the
test effort. The third addresses test progress monitoring, test reporting and test control. The fourth
explains configuration management and its relationship to testing. The fifth covers the central topic of risk
and how testing affects and is affected by product and project risks. The sixth and final section discusses the
management of incidents, both product defects and other events that require further investigation.
In this section, let's talk about organizing a test effort within a project. We'll look at the value of
independent testing, and discuss the potential benefits and risks associated with independent testing. We
will examine the various types of different test team members we might want on a test team. And we'll
familiarize ourselves with the typical tasks performed by test leaders and testers.
As we go through this section, keep your eyes open for the glossary terms tester, test leader and
test manager.
5.1.1 Independent and integrated testing
In Chapter 1 we talked about independent testing from the perspective of indi-
vidual tester psychology. In this chapter, we'll look at the organizational and
managerial implications of independence.
The approaches to organizing a test team vary, as do the places in the organ-
ization structure where the test team fits. Since testing is an assessment of
quality, and since that assessment is not always positive, many organizations
strive to create an organizational climate where testers can deliver an inde-
pendent, objective assessment of quality.
When thinking about how independent the test team is, recognize that inde-
pendence is not an either/or condition, but a continuum. At one end of the
continuum lies the absence of independence, where the programmer performs
testing within the programming team.
Moving toward independence, you find an integrated tester or group of
testers working alongside the programmers, but still within and reporting to
the development manager. You might find a team of testers who are independ-
ent and outside the development team, but reporting to project management.
Near the other end of the continuum lies complete independence. You
might see a separate test team reporting into the organization at a point equal
to the development or project team. You might find specialists in the business
domain (such as users of the system), specialists in technology (such as data-
base experts), and specialists in testing (such as security testers, certification
testers, or test automation experts) in a separate test team, as part of a larger
independent test team, or as part of a contract, outsourced test team. Let's
examine the potential benefits and risks of independence, starting with the
benefits.
An independent tester can often see more, other, and different defects than
a tester working within a programming team - or a tester who is by profession
a programmer. While business analysts, marketing staff, designers, and pro-
grammers bring their own assumptions to the specification and implementa-
tion of the item under test, an independent tester brings a different set of
assumptions to testing and to reviews, which often helps expose hidden defects
and problems related to the group's way of thinking, as we discussed in
Chapter 3. An independent tester brings a skeptical attitude of professional
pessimism, a sense that, if there's any doubt about the observed behavior, they
should ask: Is this a defect?
At the team level, an independent test team reporting to a senior or execu-
tive manager may enjoy (once they earn it) more credibility in the organization
than a test leader or tester who is part of the programming team. An independ-
ent tester who reports to senior management can report his results honestly
and without concern for reprisals that might result from pointing out problems
in coworkers' or, worse yet, the manager's work. An independent test team
often has a separate budget, which helps ensure the proper level of money is
spent on tester training, testing tools, test equipment, and so forth. In addition,
in some organizations, testers in an independent test team may find it easier to
have a career path that leads up into more senior roles in testing.
Independent test teams are not risk-free. It's possible for the testers and the
test team to become isolated. This can take the form of interpersonal isolation
from the programmers, the designers, and the project team itself, or it can take
the form of isolation from the broader view of quality and the business objec-
tives (e.g., obsessive focus on defects, often accompanied by a refusal to accept
business prioritization of defects). This leads to communication problems,
feelings of alienation and antipathy, a lack of identification with and support
for the project goals, spontaneous blame festivals and political backstabbing.
Even well-integrated test teams can suffer problems. Other project stake-
holders might come to see the independent test team - rightly or wrongly - as
a bottleneck and a source of delay. Some programmers abdicate their respon-
sibility for quality, saying, 'Well, we have this test team now, so why do I need
to unit test my code?'
Due to a desire for the benefits of an independent test team, companies
sometimes establish them, only to break them up again later. Why does that
happen? A common cause is the failure of the test manager to effectively
manage the risks of independence listed above. Some test teams succumb to
the temptation to adopt a 'no can do' attitude, coming up with reasons why
the project should bend to their needs rather than each side being flexible
so as to enable project success. Testers take to acting as enforcers of process
or as auditors without a proper management mandate and support.
Resentments and pressures build, until at last the organization decides that
the independent test team causes more problems than it solves. It's espe-
cially important for testers and test managers to understand the mission
they serve and the reasons why the organization wants an independent test
team. Often, the entire test team must realize that, whether they are part of
the project team or independent, they exist to provide a service to the
project team.
There is no one right approach to organizing testing. For each project, you
must consider whether to use an independent test team, based on the project,
the application domain, and the levels of risk, among other factors. As the size,
complexity, and criticality of the project increases, it is important to have inde-
pendence in later levels of testing (like integration test, system test and accept-
ance test), though some testing is often best done by other people such as
project managers, quality managers, developers, business and domain experts
or infrastructure or IT operations experts.
As you document the answers to these kinds of questions, the test plan
becomes a record of previous discussions and agreements between the testers
and the rest of the project team.
The test plan also helps us manage change. During early phases of the
project, as we gather more information, we revise our plans. As the project
evolves and situations change, we adapt our plans. Written test plans give us a
baseline against which to measure such revisions and changes. Furthermore,
updating the plan at major milestones helps keep testing aligned with project
needs. As we run the tests, we make final adjustments to our plans based on the
results. You might not have the time - or the energy - to update your test plans
every time a variance occurs, as some projects can be quite dynamic. In Chapter
6 [Black, 2001], we describe a simple approach for documenting variances from
the test plan that you can implement using a database or spreadsheet. You can
include these change records in a periodic test plan update, as part of a test
status report, or as part as an end-of-project test summary.
We've found that it's better to write multiple test plans in some situations.
For example, when we manage both integration and system test levels, those
two test execution periods occur at different points in time and have different
objectives. For some systems projects, a hardware test plan and a software test
plan will address different techniques and tools as well as different audiences.
However, since there might be overlap between these test plans, a master test
plan that addresses the common elements can reduce the amount of redundant
documentation.
5.2.2
5.2.3 Estimating what testing will involve and what it will cost
The testing work to be done can often be seen as a subproject within the larger
project. So, we can adapt fundamental techniques of estimation for testing. We
could start with a work-breakdown structure that identifies the stages, activities
and tasks.
Starting at the highest level, we can break down a testing project into phases
using the fundamental test process identified in the ISTQB Syllabus: planning
and control; analysis and design; implementation and execution; evaluating exit
criteria and reporting; and test closure. Within each phase we identify activities
and within each activity we identify tasks and perhaps subtasks. To identify the
activities and tasks, we work both forward and backward. When we say we work
forward, we mean that we start with the planning activities and then move
forward in time step by step, asking, 'Now, what comes next?'
Working backward means that we consider the risks that we identified during
risk analysis (which we'll discuss in Section 5.5). For those risks which you
intend to address through testing, ask yourself, 'So, what activities and tasks are
required in each stage to carry out this testing?' Let's look at an example of how
you might work backward.
Suppose that you've identified performance as a major area of risk for your
product. So, performance testing is an activity in the test execution phase. You
now estimate the tasks involved with running a performance test, how long
those tasks will take and how many times you'll need to run the performance
tests.
Now, those tests didn't just appear out of thin air: someone had to develop
them. So, performance test development entails activities in test analysis, design
and implementation. You now estimate the tasks involved in developing a per-
formance test, such as writing test scripts and creating test data.
Typically, performance tests need to be run in a special test environment that
is designed to look like the production or field environment, at least in those
ways which would affect response time and resource utilization and perform-
ance tests need special tools to generate load and check response. So, perform-
ance test environment acquisition and configuration is an activity in the test
implementation phase. You now estimate tasks involved in acquiring and con-
figuring such a test environment, such as simulating performance based on the
production environment design to look for potential bottlenecks, getting the
right hardware, software and tools and setting up hardware, software and tools.
Not everyone knows how to use performance-testing tools or to design per-
formance tests. So, performance-testing training or staffing is an activity in the
test planning phase. Depending on the approach you intend to take, you now
estimate the time required to identify and hire a performance test professional
or to train one or more people in your organization to do the job.
Finally, in many cases a detailed test plan is written for performance testing,
due to its differences from other test types. So, performance-testing planning is
an activity in the test planning phase. You now estimate the time required to
draft, review and finalize a performance test plan.
When you are creating your work-breakdown structure, remember that you
will want to use it for both estimation (at the beginning) and monitoring and
control (as the project continues). To ensure accuracy of the estimate and
precise control, make sure that you subdivide the work finely enough. This
means that tasks should be short in duration, say one to three days. If they are
much longer - say two weeks - then you run the risk that long and complex sub-
tasks are 'hiding' within the larger task, only to be discovered later. This can
lead to nasty surprises during the project.
In this section, we'll review techniques and metrics that are commonly used for
monitoring test preparation and execution. We'll focus especially on the use
and interpretation of such test metrics for reporting, controling and analyzing
the test effort, including those based on defects and those based on test data.
We'll also look at options for reporting test status using such metrics and other
information.
As you read, remember to watch for the glossary terms defect density, failure
rate, test control, test coverage, test monitoring and test report.
5.3.1 Monitoring the progress of test activities
Having developed our plans, defined our test strategies and approaches and
estimated the work to be done, we must now track our testing work as we carry
it out. Test monitoring can serve various purposes during the project, including
the following:
• Give the test team and the test manager feedback on how the testing work is
going, allowing opportunities to guide and improve the testing and the project.
• Provide the project team with visibility about the test results.
• Measure the status of the testing, test coverage and test items against the exit
criteria to determine whether the test work is done.
• Gather data for use in estimating future test efforts.
Especially for small projects, the test leader or a delegated person can gather
test progress monitoring information manually using documents, spreadsheets
and simple databases. When working with large teams, distributed projects and
long-term test efforts, we find that the efficiency and consistency of data collec-
tion is aided by the use of automated tools (see Chapter 6).
One way to gather test progress information is to use the IEEE 829 test log
template. While much of the information related to logging events can be use-
fully captured in a document, we prefer to capture the test-by-test information
in spreadsheets (see Figure 5.1).
In Figure 5.1, columns A and B show the test ID and the test case or test suite
name. The state of the test case is shown in column C ('Warn' indicates a test
that resulted in a minor failure). Column D shows the tested configuration,
where the codes A, B and C correspond to test environments described in detail
in the test plan. Columns E and F show the defect (or bug) ID number (from
the defect-tracking database) and the risk priority number of the defect
(ranging from 1, the worst, to 25, the least risky). Column G shows the initials
of the tester who ran the test. Columns H through L capture data for each test
related to dates, effort and duration (in hours). We have metrics for planned
and actual effort and dates completed which would allow us to summarize
progress against the planned schedule and budget. This spreadsheet can also be
summarized in terms of the percentage of tests which have been run and the
percentage of tests which have passed and failed.
Figure 5.1 might show a snapshot of test progress during the test execution
period, or perhaps even at test closure if it were deemed acceptable to skip
some of the tests. During the analysis, design and implementation of the tests,
such a worksheet would show the state of the tests in terms of their state of
development.
In addition to test case status, it is also common to monitor test progress
during the test execution period by looking at the number of defects found and
fixed. Figure 5.2 shows a graph that plots the total number of defects opened
and closed over the course of the test execution so far.
It also shows the planned test period end date and the planned number of
defects that will be found. Ideally, as the project approaches the planned end
date, the total number of defects opened will settle in at the predicted number
and the total number of defects closed will converge with the total number
opened. These two outcomes tell us that we have found enough defects to feel
comfortable that we're done testing, that we have no reason to think many more
defects are lurking in the product, and that all known defects have been
resolved.
Charts such as Figure 5.2 can also be used to show failure rates or defect
density. When reliability is a key concern, we might be more concerned with the
frequency with which failures are observed than with how many defects are
causing the failures. In organizations that are looking to produce ultra-reliable
software, they may plot the number of unresolved defects normalized by the
size of the product, either in thousands of source lines of code (KSLOC), func-
tion points (FP) or some other metric of code size. Once the number of unre-
solved defects falls below some predefined threshold - for example, three per
million lines of code - then the product may be deemed to have met the defect
density exit criteria.
Measuring test progress based on defects found and fixed is common and
useful - if used with care. Avoid using defect metrics alone, as it is possible to
achieve a flat defect find rate and to fix all the known defects by stopping any
further testing, by deliberately impeding the reporting of defects and by allow-
ing programmers to reject, cancel, or close defect reports without any inde-
pendent review.
That said, test progress monitoring techniques vary considerably depending
on the preferences of the testers and stakeholders, the needs and goals of the
project, regulatory requirements, time and money constraints and other factors.
In addition to the kinds of information shown in the IEEE 829 Test Log
Template, Figures 5.1 and Figure 5.2, other common metrics for test progress
monitoring include:
• The extent of completion of test environment preparation;
• The extent of test coverage achieved, measured against requirements, risks,
code, configurations or other areas of interest;
• The status of the testing (including analysis, design and implementation)
compared to various test milestones;
• The economics of testing, such as the costs and benefits of continuing test
execution in terms of finding the next defect or running the next test.
As a complementary monitoring technique, you might assess the subjective
level of confidence the testers have in the test items. However, avoid making
important decisions based on subjective assessments alone, as people's impres-
sions have a way of being inaccurate and colored by bias.
5 . 4 CONFIGURATION MANAGEMENT
5 . 6 INCIDENT MANAGEMENT
Let's wind down this chapter on test management with an important subject:
how we can document and manage the incidents that occur during test execu-
tion. We'll look at what topics we should cover when reporting incidents and
defects. At the end of this section, you'll be ready to write a thorough incident
report.
Keep your eyes open for the only Syllabus term in this section, incident
logging.
5.6.1 What are incident reports for and how do I write good
ones?
When running a test, you might observe actual results that vary from expected
results. This is not a bad thing - one of the major goals of testing is to find prob-
lems. Different organizations have different names to describe such situations.
Commonly, they're called incidents, bugs, defects, problems or issues.
To be precise, we sometimes draw a distinction between incidents on the one
hand and defects or bugs on the other. An incident is any situation where the
system exhibits questionable behavior, but often we refer to an incident as a
defect only when the root cause is some problem in the item we're testing.
Other causes of incidents include misconfiguration or failure of the test envi-
ronment, corrupted test data, bad tests, invalid expected results and tester
mistakes. (However, in some cases the policy is to classify as a defect any inci-
dent that arises from a test design, the test environment or anything else which
is under formal configuration management.) We talk about incidents to indicate
the possibility that a questionable behavior is not necessarily a true defect. We
log these incidents so that we have a record of what we observed and can follow
up the incident and track what is done to correct it.
While it is most common to find incident logging or defect reporting
processes and tools in use during formal, independent test phases, you can also
log, report, track, and manage incidents found during development and
reviews. In fact, this is a good idea, because it gives useful information on the
extent to which early - and cheaper - defect detection and removal activities are
happening.
Of course, we also need some way of reporting, tracking, and managing inci-
dents that occur in the field or after deployment of the system. While many of
these incidents will be user error or some other behavior not related to a defect,
some percentage of defects do escape from quality assurance and testing activ-
ities. The defect detection percentage, which compares field defects with test
defects, is an important metric of the effectiveness of the test process.
Here is an example of a DDP formula that would apply for calculating DDP
for the last level of testing prior to release to the field:
defects (testers)
DDP= ----------------------------------------------------------
defects (testers) + defects (field)
It is most common to find defects reported against the code or the system
itself. However, we have also seen cases where defects are reported against
requirements and design specifications, user and operator guides and tests.
Often, it aids the effectiveness and efficiency of reporting, tracking and manag-
ing defects when the defect-tracking tool provides an ability to vary some of the
information captured depending on what the defect was reported against.
In some projects, a very large number of defects are found. Even on smaller
projects where 100 or fewer defects are found, you can easily lose track of them
unless you have a process for reporting, classifying, assigning and managing the
defects from discovery to final resolution.
An incident report contains a description of the misbehavior that was
observed and classification of that misbehavior. As with any written communi-
cation, it helps to have clear goals in mind when writing. One common goal for
such reports is to provide programmers, managers and others with detailed
information about the behavior observed and the defect. Another is to support
the analysis of trends in aggregate defect data, either for understanding more
about a particular set of problems or tests or for understanding and reporting
the overall level of system quality. Finally, defect reports, when analyzed over a
project and even across projects, give information that can lead to development
and test process improvements.
When writing an incident, it helps to have the readers in mind, too. The pro-
grammers need the information in the report to find and fix the defects. Before
that happens, though, managers should review and prioritize the defects so that
scarce testing and developer resources are spent fixing and confirmation testing
the most important defects. Since some defects may be deferred - perhaps to
be fixed later or perhaps, ultimately, not to be fixed at all - we should include
work-arounds and other helpful information for help desk or technical support
teams. Finally, testers often need to know what their colleagues are finding so
that they can watch for similar behavior elsewhere and avoid trying to run tests
that will be blocked.
A good incident report is a technical document. In addition to being clear for
its goals and audience, any good report grows out of a careful approach to
researching and writing the report. We have some rules of thumb that can help
you write a better incident report.
First, use a careful, attentive approach to running your tests. You never
know when you're going to find a problem. If you're pounding on the key-
board while gossiping with office mates or daydreaming about a movie you
just saw, you might not notice strange behaviors. Even if you see the incident,
how much do you really know about it? What can you write in your incident
report?
Intermittent or sporadic symptoms are a fact of life for some defects and it's
always discouraging to have an incident report bounced back as 'irrepro-
ducible'. So, it's a good idea to try to reproduce symptoms when you see them
and we have found three times to be a good rule of thumb. If a defect has inter-
mittent symptoms, we would still report it, but we would be sure to include as
much information as possible, especially how many times we tried to reproduce
it and how many times it did in fact occur.
You should also try to isolate the defect by making carefully chosen changes
to the steps used to reproduce it. In isolating the defect, you help guide the pro-
grammer to the problematic part of the system. You also increase your own
knowledge of how the system works - and how it fails.
Some test cases focus on boundary conditions, which may make it appear
that a defect is not likely to happen frequently in practice. We have found that
it's a good idea to look for more generalized conditions that cause the failure to
occur, rather than simply relying on the test case. This helps prevent the infa-
mous incident report rejoinder, 'No real user is ever going to do that.' It also
cuts down on the number of duplicate reports that get filed.
As there is often a lot of testing going on with the system during a test period,
there are lots of other test results available. Comparing an observed problem
against other test results and known defects found is a good way to find and
document additional information that the programmer is likely to find very
useful. For example, you might check for similar symptoms observed with other
defects, the same symptom observed with defects that were fixed in previous
versions or similar (or different) results seen in tests that cover similar parts of
the system.
Many readers of incident reports, managers especially, will need to under-
stand the priority and severity of the defect. So, the impact of the problem on
the users, customers and other stakeholders is important. Most defect-tracking
systems have a title or summary field and that field should mention the impact,
too.
Choice of words definitely matters in incident reports. You should be
clear and unambiguous. You should also be neutral, fact-focused and impar-
tial, keeping in mind the testing-related interpersonal issues discussed in
Chapter 1 and earlier in this chapter. Finally, keeping the report concise
helps keep people's attention and avoids the problem of losing them in the
details.
As a last rule of thumb for incident reports, we recommend that you use a
review process for all reports filed. It works if you have the lead tester review
reports and we have also allowed testers - at least experienced ones - to review
other testers' reports. Reviews are proven quality assurance techniques and
incident reports are important project deliverables.
I E E E 829 STANDARD:
TEST INCIDENT REPO RT TEMPL AT E
Test incident report identifier
Summary
Incident description (inputs, expected
results, actual results, anomalies,
date and time, procedure step,
environment, attempts to repeat,
testers and observers)
Impact
5.6.3 What happens to incident reports after you file them?
As we mentioned earlier, incident reports are managed through a life cycle
from discovery to resolution. The incident report life cycle is often shown as a
state transition diagram (see Figure 5.3). While your defect-tracking system
may use a different life cycle, let's take this one as an example to illustrate how
an incident report life cycle might work.
In the incident report life cycle shown in Figure 5.3, all incident reports move
through a series of clearly identified states after being reported. Some of these
state transitions occur when a member of the project team completes some
assigned task related to closing an incident report. Some of these state transi-
tions occur when the project team decides not to repair a defect during this
project, leading to the deferral of the incident report. Some of these state tran-
sitions occur when an incident report is poorly written or describes behavior
which is actually correct, leading to the rejection of that report.
Let's focus on the path taken by incident reports which are ultimately fixed.
After an incident is reported, a peer tester or test manager reviews the report.
If successful in the review, the incident report becomes opened, so now the
project team must decide whether or not to repair the defect. If the defect is to
be repaired, a programmer is assigned to repair it.
Once the programmer believes the repairs are complete, the incident report
returns to the tester for confirmation testing. If the confirmation test fails, the
incident report is re-opened and then re-assigned. Once the tester confirms a
good repair, the incident report is closed. No further work remains to be done.
In any state other than rejected, deferred or closed, further work is required
on the incident prior to the end of this project. In such a state, the incident
report has a clearly identified owner. The owner is responsible for transitioning
the incident into an allowed subsequent state. The arrows in the diagram show
these allowed transitions.
In a rejected, deferred or closed state, the incident report will not be
assigned to an owner. However, certain real-world events can cause an incident
report to change state even if no active work is occurring on the incident report.
Examples include the recurrence of a failure associated with a closed incident
report and the discovery of a more serious failure associated with a deferred
incident report.
Ideally, only the owner can transition the incident report from the current
state to the next state and ideally the owner can only transition the incident
report to an allowed next state. Most defect-tracking systems support and
enforce the life cycle and life cycle rules. Good defect-tracking systems allow
you to customize the set of states, the owners, and the transitions allowed to
match your actual workflows. And, while a good defect-tracking system is
helpful, the actual defect workflow should be monitored and supported by
project and company management.
CHAPTER REVIEW
Let's review what you have learned in this chapter.
From Section 5.1, you should now be able to explain the basic
ideas of test organization. You should know why independent
testing is important, but also be able to analyze the potential
benefits and problems associated with independent test teams. You
should recognize the types of people and skills needed in a test
team and recall the tasks that a tester and a test leader will carry out.
You should know the glossary terms tester, test leader and test
manager.
From Section 5.2, you should now understand the fundamentals
of test planning and estimation. You should know the reasons for
writing test plans and be able to explain how test plans relate to
projects, test levels or phases, test targets and test execution. You
should know which parts of the test process require special
attention in test planning. You should be able to explain the
justification behind various entry and exit criteria that might relate
to projects, test levels or phases and test targets. You should be able
to distinguish the purpose and content of test plans from that of test
design specifications, test cases and test procedures, and know the
IEEE 829 outline for a test plan. You should know the factors that
affect the effort involved in testing, including especially test
strategies (approaches) and how they affect testing. You should be
able to explain how metrics, expertise and negotiation are used for
estimating. You should know the glossary terms entry criteria, exit
criteria, exploratory testing, test approach, test level, test plan, test
procedure and test strategy.
From Section 5.3, you should be able to explain the essentials of
test progress monitoring and control. You should know the common
metrics that are captured, logged and used for monitoring, as well
as ways to present these metrics. You should be able to analyze,
interpret and explain test metrics that can be useful for reporting test
status and for making decisions about how to control test progress.
You should be able to explain a typical test status report and know
the IEEE 829 test summary report and test log. You should know
the glossary terms defect density, failure rate, test control, test
coverage, test monitoring and test report.
From Section 5.4, you should now understand the basics of
configuration management that relate to testing. You should be
able to summarize how good configuration management helps us
do our testing work better. You should know the glossary terms
configuration management and version control.
From Section 5.5, you should now be able to explain how risk
and testing relate. You should know that a risk is a potential
undesirable or negative outcome and that most of the risks we are
interested in relate to the achievement of project objectives. You
should know about likelihood and impact as factors that determine
the importance of a risk. You should be able to compare and
contrast risks to the product (and its quality) and risks to the project
itself and know typical risks to the product and project. You should
be able to describe how to use risk analysis and risk management for
testing and test planning. You should know the glossary terms
product risk, project risk, risk and risk-based testing.
From Section 5.6, you should now understand incident logging
and be able to use incident management on your projects. You
should know the content of an incident report according to the
IEEE 829 standard. You should be able to write a high-quality
report based on test results and manage that report through its life
cycle. You should know the glossary term incident logging.
SAMPLE EXAM QUESTIONS
Question 1 Why is independent testing c. The test plan is the least thorough, the test
important? procedure specification is the most thorough and
a. Independent testing is usually cheaper than the test design specification is midway between
testing your own work. the two.
b. Independent testing is more effective at finding d. The test plan is finished in the first third of the
defects. project, the test design specification is finished in
the middle third of the project and the test
c. Independent testers should determine the
procedure specification is finished in the last
processes and methodologies used.
third of the project.
d. Independent testers are dispassionate about
whether the project succeeds or fails.
Question 5 Which of the following factors is an
influence on the test effort involved in most
Question 2 Which of the following is among the projects?
typical tasks of a test leader?
a. Geographical separation of tester and
a. Develop system requirements, design programmers.
specifications and usage models.
b. The departure of the test manager during the
b. Handle all test automation duties. project.
c. Keep tests and test coverage hidden from c. The quality of the information used to develop
programmers. the tests.
d. Gather and report test progress metrics. d. Unexpected long-term illness by a member of
the project team.
Question 3 According to the ISTQB Glossary,
what do we mean when we call someone a test Question 6 The ISTQB Foundation Syllabus
manager? establishes a fundamental test process where test
a. A test manager manages a collection of test planning occurs early in the project, while test
leaders. execution occurs at the end. Which of the following
b. A test manager is the leader of a test team or elements of the test plan, while specified during test
teams. planning, is assessed during test execution?
c. A test manager gets paid more than a test leader. a. Test tasks
d. A test manager reports to a test leader. b. Environmental needs
c. Exit criteria
Question 4 What is the primary difference d. Test team training
between the test plan, the test design specification,
and the test procedure specification? Question 7 Consider the following exit criteria
a. The test plan describes one or more levels of which might be found in a test plan:
testing, the test design specification identifies the I No known customer-critical defects.
associated high-level test cases and a test
II All interfaces between components tested.
procedure specification describes the actions for
executing a test. III 100% code coverage of all units.
b. The test plan is for managers, the test design IVAll specified requirements satisfied.
specification is for programmers and the test V System functionality matches legacy system for
procedure specification is for testers who are all business rules.
automating tests.
Which of the following statements is true about d. The programmers should focus their attention on
whether these exit criteria belong in an acceptance fixing the remaining known defects prior to release.
test plan?
a. All statements belong in an acceptance test plan. Question 11 In a test summary report, the
b. Only statement I belongs in an acceptance test project's test leader makes the following statement,
plan. 'The payment processing subsystem fails to accept
payments from American Express cardholders,
c. Only statements I, II, and V belong in an
which is considered a must-work feature for this
acceptance test plan. release.' This statement is likely to be found in
d. Only statements I, IV, and V belong in an which of the following sections?
acceptance test plan. a. Evaluation
b. Summary of activities
Question 8 According to the ISTQB Glossary,
what is a test level? c. Variances
a. A group of test activities that are organized d. Incident description
together.
b. One or more test design specification Question 12 During an early period of test
documents. execution, a defect is located, resolved and
confirmed as resolved by re-testing, but is seen
c. A test type. again later during subsequent test execution. Which
d. An ISTQB certification. of the following is a testing-related aspect of
configuration management that is most likely to
Question 9 Which of the following metrics would have broken down?
be most useful to monitor during test execution? a. Traceability
a. Percentage of test cases written. b. Confirmation testing
b. Number of test environments remaining to be c. Configuration control
configured. d. Test documentation management
c. Number of defects found and fixed.
d. Percentage of requirements for which a test has Question 13 You are working as a tester on a
been written. project to develop a point-of-sales system for
grocery stores and other similar retail outlets.
Question 10 During test execution, the test Which of the following is a product risk for such a
manager describes the following situation to the project?
project team: '90% of the test cases have been run. a. The arrival of a more-reliable competing product
20% of the test cases have identified defects. 127 on the market.
defects have been found. 112 defects have been b. Delivery of an incomplete test release to the first
fixed and have passed confirmation testing. Of the cycle of system test.
remaining 15 defects, project management has
decided that they do not need to be fixed prior to c. An excessively high number of defect fixes fail
release.' Which of the following is the most during re-testing.
reasonable interpretation of this test status report? d. Failure to accept allowed credit cards.
a. The remaining 15 defects should be confirmation
tested prior to release. Question 14 A product risk analysis meeting is
b. The remaining 10% of test cases should be run held during the project planning period. Which of
prior to release. the following determines the level of risk?
c. The system is now ready for release with no a. Difficulty of fixing related problems in code
further testing or development effort. b. The harm that might result to the user
c. The price for which the software is sold giving the message, "Unexpected data type: 15.
d. The technical staff in the meeting Click to continue." ' This statement is likely to be
found in which of the following sections of an IEEE
Question 15 You are writing a test plan using the 829 standard incident report?
IEEE 829 template and are currently completing a. Summary
the Risks and Contingencies section. Which of the b. Impact
following is most likely to be listed as a project risk? c. Item pass/fail criteria
a. Unexpected illness of a key team member d. Incident description
b. Excessively slow transaction-processing time
c. Data corruption under network congestion Question 19 According to the ISTQB Glossary,
d. Failure to handle a key use case what do we call a document that describes any
event that occurred during testing which requires
further investigation?
Question 16 You and the project stakeholders
develop a list of product risks and project risks a. A bug report
during the planning stage of a project. What else b. A defect report
should you do with those lists of risks during test c. An incident report
planning?
d. A test summary report
a. Determine the extent of testing required for the
product risks and the mitigation and contingency
Question 20 A product risk analysis is performed
actions required for the project risks.
during the planning stage of the test process.
b. Obtain the resources needed to completely cover During the execution stage of the test process, the
each product risk with tests and transfer test manager directs the testers to classify each
responsibility for the project risks to the project defect report by the known product risk it relates to
manager. (or to 'other'). Once a week, the test manager runs
c. Execute sufficient tests for the product risks, a report that shows the percentage of defects
based on the likelihood and impact of each related to each known product risk and to unknown
product risk and execute mitigation actions for risks. What is one possible use of such a report?
all project risks. a. To identify new risks to system quality.
d. No further risk management action is required b. To locate defect clusters in product subsystems.
at the test planning stage.
c. To check risk coverage by tests.
d. To measure exploratory testing.
Question 17 According to the ISTQB Glossary, a
product risk is related to which of the following?
a. Control of the test project
b. The test object
c. A single test item
d. A potential negative outcome
Evidence attached
• Screen shots of server and client environments 14 and 15
• Screen shots of equivalent live environments
• Screen shots of the country choice problem
• Recovery time stats for this problem, the country/market default problem and the equity system problem
referred to.
CHAPTER S I X
Y ou may be wishing that you had a magic tool that would automate all of
the testing for you. If so, you will be disappointed. However, there are a
number of very useful tools that can bring significant benefits. In this chapter
we will see that there is tool support for many different aspects of software
testing. We will see that success with tools is not guaranteed, even if an
appropriate tool is acquired - there are also risks in using tools. There are
some special considerations mentioned in the Syllabus for certain types of
tool: test execution tools, performance testing tools, static analysis tools and
test management tools.
In this section, we will describe the various tool types in terms of their general
functionality, rather than going into lots of detail. The reason for this is that, in
general, the types of tool will be fairly stable over a longer period, even though
there will be new vendors in the market, new and improved tools, and even new
types of tool in the coming years.
We will not mention any commercial tools in this chapter. If we did, this book
would date very quickly! Tool vendors are acquired by other vendors, change
their names, and change the names of the tools quite frequently, so we will not
mention the names of any tools or vendors.
Test comparators
Is it really a test if you put some inputs into some software, but never look to
see whether the software produces the correct result? The essence of testing is
to check whether the software produces the correct result, and to do that, we
must compare what the software produces to what it should produce. A test
comparator helps to automate aspects of that comparison.
There are two ways in which actual results of a test can be compared to the
expected results for the test. Dynamic comparison is where the comparison is
done dynamically, i.e. while the test is executing. The other way is post-execu-
tion comparison, where the comparison is performed after the test has finished
executing and the software under test is no longer running.
Test execution tools include the capability to perform dynamic comparison
while the tool is executing a test. This type of comparison is good for comparing
the wording of an error message that pops up on a screen with the correct wording
for that error message. Dynamic comparison is useful when an actual result does
not match the expected result in the middle of a test - the tool can be programmed
to take some recovery action at this point or go to a different set of tests.
Post-execution comparison is usually best done by a separate tool (i.e. not
the test execution tool). This is the type of tool that we mean by a test compara-
tor or test comparison tool and is typically a 'stand-alone' tool. Operating
systems normally have file comparison tools available which can be used for
post-execution comparison and often a comparison tool will be developed in-
house for comparing a particular type of file or test result.
Post-execution comparison is best for comparing a large volume of data, for
example comparing the contents of an entire file with the expected contents of
that file, or comparing a large set of records from a database with the expected
content of those records. For example, comparing the result of a batch run (e.g.
overnight processing of the day's online transactions) is probably impossible to
do without tool support.
Whether a comparison is dynamic or post-execution, the test comparator
needs to know what the correct result is. This may be stored as part of the test
case itself or it may be computed using a test oracle. See Chapter 4 for informa-
tion about test oracles.
Features or characteristics of test comparators include support for:
• dynamic comparison of transient events that occur during test execution;
• post-execution comparison of stored data, e.g. in files or databases;
• masking or filtering of subsets of actual and expected results.
Coverage measurement tools (D)
How thoroughly have you tested? Coverage tools can help answer this
question.
A coverage tool first identifies the elements or coverage items that can be
counted, and where the tool can identify when a test has exercised that cover-
age item. At component testing level, the coverage items could be lines of code
or code statements or decision outcomes (e.g. the True or False exit from an IF
statement). At component integration level, the coverage item may be a call to
a function or module. Although coverage can be measured at system or accept-
ance testing levels, e.g. where the coverage item may be a requirement state-
ment, there aren't many (if any) commercial tools at this level; there is more
tool support at component testing level or to some extent at component inte-
gration level.
The process of identifying the coverage items at component test level is
called 'instrumenting the code', as described in Chapter 4. A suite of tests is
then run through the instrumented code, either automatically using a test exe-
cution tool or manually. The coverage tool then counts the number of coverage
items that have been executed by the test suite, and reports the percentage of
coverage items that have been exercised, and may also identify the items that
have not yet been exercised (i.e. not yet tested). Additional tests can then be run
to increase coverage (the tool reports accumulated coverage of all the tests run
so far).
The more sophisticated coverage tools can provide support to help iden-
tify the test inputs that will exercise the paths that include as-yet unexercised
coverage items (or link to a test design tool to identify the unexercised
items). For example, if not all decision outcomes have been exercised, the
coverage tool can identify the particular decision outcome (e.g. a False exit
from an IF statement) that no test has taken so far, and may then also be able
to calculate the test input required to force execution to take that decision
outcome.
Features or characteristics of coverage measurement tools include support
for:
• identifying coverage items (instrumenting the code);
• calculating the percentage of coverage items that were exercised by a suite of
tests;'
• reporting coverage items that have not been exercised as yet;
• identifying test inputs to exercise as yet uncovered items (test design tool
functionality);
• generating stubs and drivers (if part of a unit test framework).
Note that the coverage tools only measure the coverage of the items that
they can identify. Just because your tests have achieved 100% statement cover-
age, this does not mean that your software is 100% tested!
Security tools
There are a number of tools that protect systems from external attack, for
example firewalls, which are important for any system.
Security testing tools can be used to test security by trying to break into a
system, whether or not it is protected by a security tool. The attacks may focus
on the network, the support software, the application code or the underlying
database.
Features or characteristics of security testing tools include support for:
• identifying viruses;
• detecting intrusions such as denial of service attacks;
• simulating various types of external attacks;
• probing for open ports or other externally visible points of attack;
• identifying weaknesses in password files and passwords;
• security checks during operation, e.g. for checking integrity of files, and
intrusion detection, e.g. checking results of test attacks.
Monitoring tools
Monitoring tools are used to continuously keep track of the status of the system
in use, in order to have the earliest warning of problems and to improve service.
There are monitoring tools for servers, networks, databases, security, perform-
ance, website and internet usage, and applications.
Features or characteristics of monitoring tools include support for:
• identifying problems and sending an alert message to the administrator (e.g.
network administrator);
• logging real-time and historical information;
• finding optimal settings;
• monitoring the number of users on a network;
• monitoring network traffic (either in real time or covering a given length of
time of operation with the analysis performed afterwards).
6.1.7 Tool support for specific application areas (Kl)
In this chapter, we have described tools according to their general functional
classifications. There are also further specializations of tools within these clas-
sifications. For example there are web-based performance-testing tools as well
as performance-testing tools for back-office systems. There are static analysis
tools for specific development platforms and programming languages, since
each programming language and every platform has distinct characteristics.
There are dynamic analysis tools that focus on security issues, as well as
dynamic analysis tools for embedded systems.
Commercial tool sets may be bundled for specific application areas such as
web-based or embedded systems.
The reason for acquiring tools to support testing is to gain benefits, by using a
software program to do certain tasks that are better done by a computer than
by a person.
Advice on introducing tools into an organization can be found in web arti-
cles, magazines and books such as [Dustin et al., 1999], [Siteur, 2005] and
[Fewster and Graham, 1999].
6.2.1 Potential benefits of using tools
There are many benefits that can be gained by using tools to support testing,
whatever the specific type of tool. Benefits include:
• reduction of repetitive work;
• greater consistency and repeatability;
• objective assessment;
• ease of access to information about tests or testing.
Repetitive work is tedious to do manually. People become bored and
make mistakes when doing the same task over and over. Examples of this
type of repetitive work include running regression tests, entering the same
test data over and over again (both of which can be done by a test execution
tool), checking against coding standards (which can be done by a static analy-
sis tool) or creating a specific test database (which can be done by a test data
preparation tool).
People tend to do the same task in a slightly different way even when they
think they are repeating something exactly. A tool will exactly reproduce what
it did before, so each time it is run the result is consistent. Examples of where
this aspect is beneficial include checking to confirm the correctness of a fix to
a defect (which can be done by a debugging tool or test execution tool), enter-
ing test inputs (which can be done by a test execution tool) and generating
tests from requirements (which can be done by a test design tool or possibly
a requirements management tool).
If a person calculates a value from the software or incident reports, they may
inadvertently omit something, or their own subjective prejudices may lead them
to interpret that data incorrectly. Using a tool means that subjective bias is
removed and the assessment is more repeatable and consistently calculated.
Examples include assessing the cyclomatic complexity or nesting levels of a
component (which can be done by a static analysis tool), coverage (coverage
measurement tool), system behavior (monitoring tools) and incident statistics
(test management tool).
Having lots of data doesn't mean that information is communicated.
Information presented visually is much easier for the human mind to take in
and interpret. For example, a chart or graph is a better way to show informa-
tion than a long list of numbers - this is why charts and graphs in spread-
sheets are so useful. Special purpose tools give these features directly for the
information they process. Examples include statistics and graphs about test
progress (test execution or test management tool), incident rates (incident
management or test management tool) and performance (performance
testing tool).
In addition to these general benefits, each type of tool has specific benefits
relating to the aspect of testing that the particular tool supports. These benefits
are normally prominently featured in the sales information available for the
type of tool. It is worth investigating a number of different tools to get a general
view of the benefits.
6.2.2 Risks of using tools
Although there are significant benefits that can be achieved using tools to
support testing activities, there are many organizations that have not achieved
the benefits they expected.
Simply purchasing a tool is no guarantee of achieving benefits, just as buying
membership in a gym does not guarantee that you will be fitter. Each type of
tool requires investment of effort and time in order to achieve the potential
benefits.
There are many risks that are present when tool support for testing is intro-
duced and used, whatever the specific type of tool. Risks include:
• unrealistic expectations for the tool;
• underestimating the time, cost and effort for the initial introduction of a
tool;
• underestimating the time and effort needed to achieve significant and con
tinuing benefits from the tool;
• underestimating the effort required to maintain the test assets generated by
the tool;
• over-reliance on the tool.
Unrealistic expectations may be one of the greatest risks to success with
tools. The tools are only software and we all know that there are many problems
with any kind of software! It is important to have clear objectives for what the
tool can do and that those objectives are realistic.
Introducing something new into an organization is seldom straightforward.
Having purchased a tool, you will want to move from opening the box to having
a number of people being able to use the tool in a way that will bring benefits.
There will be technical problems to overcome, but there will also be resistance
from other people - both need to be addressed in order to succeed in introduc-
ing a tool.
Think back to the last time you did something new for the very first time
(learning to drive, riding a bike, skiing). Your first attempts were unlikely to be
very good but with more experience you became much better. Using a testing
tool for the first time will not be your best use of the tool either. It takes time
to develop ways of using the tool in order to achieve what is possible.
Fortunately there are some short-cuts (e.g. reading books and articles about
other people's experiences and learning from them). See also Section 6.3 for
more detail on introducing a tool into an organization.
Insufficient planning for maintenance of the assets that the tool pro-
duces is a strong contributor to tools that end up as 'shelf-ware', along
with the previously listed risks. Although particularly relevant for test exe-
cution tools, planning for maintenance is also a factor with other types of
tool.
Tools are definitely not magic! They can do very well what they have been
designed to do (at least a good quality tool can), but they cannot do everything.
A tool can certainly help, but it does not replace the intelligence needed to
know how best to use it, and how to evaluate current and future uses of the tool.
For example, a test execution tool does not replace the need for good test
design and should not be used for every test - some tests are still better
executed manually. A test that takes a very long time to automate and will not
be run very often is better done manually.
This list of risks is not exhaustive. Two other important factors are:
• the skill needed to create good tests;
• the skill needed to use the tools well, depending on the type of tool.
The skills of a tester are not the same as the skills of the tool user. The tester
concentrates on what should be tested, what the test cases should be and how
to prioritize the testing. The tool user concentrates on how best to get the tool
to do its job effectively and how to give increasing benefit from tool use.
Question 1 Which tools help to support static Question 5 Which of the following are advanced
testing? scripting techniques for test execution tools?
a. Static analysis tools and test execution tools. a. Data-driven and keyword-driven
b. Review process support tools, static analysis b. Data-driven and capture-driven
tools and coverage measurement tools. c. Capture-driven and keyhole-driven
c. Dynamic analysis tools and modeling tools. d. Playback-driven and keyword-driven
d. Review process support tools, static analysis
tools and modeling tools. Question 6 Which of the following would NOT be
done as part of selecting a tool for an organization?
Question 2 Which test activities are supported by a. Assess organizational maturity, strengths and
test harness or unit test framework tools? weaknesses.
a. Test management and control. b. Roll out the tool to as many users as possible
b. Test specification and design. within the organization.
c. Test execution and logging. c. Evaluate the tool features against clear
d. Performance and monitoring. requirements and objective criteria.
d. Identify internal requirements for coaching and
Question 3 What are the potential benefits from mentoring in the use of the tool.
using tools in general to support testing?
a. Greater quality of code, reduction in the number Question 7 Which of the following is a goal for a
of testers needed, better objectives for testing. proof-of-concept or pilot phase for tool evaluation?
b. Greater repeatability of tests, reduction in a. Decide which tool to acquire.
repetitive work, objective assessment. b. Decide on the main objectives and requirements
c. Greater responsiveness of users, reduction of for this type of tool.
tests run, objectives not necessary. c. Evaluate the tool vendor including training,
d. Greater quality of code, reduction in paperwork, support and commercial aspects.
fewer objections to the tests. d. Decide on standard ways of using, managing,
storing and maintaining the tool and the test
Question 4 What is a potential risk in using tools assets.
to support testing?
a. Unrealistic expectations, expecting the tool to do
too much.
b. Insufficient reliance on the tool, i.e. still doing
manual testing when a test execution tool has
been purchased.
c. The tool may find defects that aren't there.
d. The tool will repeat exactly the same thing it did
the previous time.
MOCK EXAM
On the real exam, you will have 60 minutes to work through 40 questions of
approximately the same difficulty mix and Syllabus distribution as shown in the following
mock exam. After you have taken this mock exam, check your answers with the answer
key.
Question 8 Review the following portion of an Question 10 Which of the following encourages
incident report. objective testing?
1 I place any item in the shopping cart. a. Unit testing
2 I place any other (different) item in the shopping cart. b. System testing
3 I remove the first item from the shopping cart, but c. Independent testing
leave the second item in the cart. d. Destructive testing
4 I click the < Checkout > button.
5 I expect the system to display the first checkout Question 11 Of the following statements about
screen. Instead, it gives the pop-up error message, reviews of specifications, which statement is true?
'No items in shopping cart. Click <Okay> to a. Reviews are not generally cost effective as the
continue shopping.' meetings are time consuming and require
6 I click < Okay >. preparation and follow up.
7 I expect the system to return to the main window b. There is no need to prepare for or follow up on reviews.
to allow me to continue adding and removing c. Reviews must be controlled by the author.
items from the cart. Instead, the browser d. Reviews are a cost effective early static test on the
terminates. system.
Question 12 Consider the following list of
test process activities:
I Analysis and design
II Test closure activities
IIIEvaluating exit criteria and
reporting
IVPlanning and control
V Implementation and execution
Which of the following places these in
their logical sequence?
a. I, II, III, IV and V
b. IV, I, V, III and II.
c. IV, I, V,II and III.
d. I, IV, V HI and II. What additional tests would be needed to
achieve 100% decision coverage?
Question 13 Test objectives vary between a. A gold card holder who stays in economy
projects and so must be stated in the test and a
plan. Which one of the following test non-gold card holder who gets upgraded
objectives might conflict with the proper to
tester mindset? business class.
a. Show that the system works before we b. A gold card holder and a non-gold card
ship it. holder
who are both upgraded to business class.
b. Find as many defects as possible.
c. A gold card holder and a non-gold card
c. Reduce the overall level of product risk. holder
d. Prevent defects through early who both stay in economy class.
involvement. d. A gold card holder who is upgraded to
business
Question 14 Which test activities are class and a non-gold card holder who
supported by test data preparation tools? stays in
a. Test management and control economy class.
b. Test specification and design
Question 16 Consider the following types
c. Test execution and logging
of tools:
d. Performance and monitoring
V Test management
Question 15 If you are flying with an tools
economy ticket, there is a possibility that W Static analysis tools
you may get upgraded to business class, X Modeling tools
especially if you hold a gold card in the Y Dynamic analysis tools
airline's frequent flyer program. If you
don't hold a gold card, there is a possibility Z Performance testing tools
that you will get 'bumped' off the flight if it Which of the following of these tools is most
is full and you check in late. This is shown likely to be used by developers?
in Figure 7.1. Note that each box (i.e. a. W,XandY
statement) has been numbered. b. V Y a n d Z
Three tests have already been run: c. V,WandZ
Test 1: Gold card holder who gets upgraded d. X,YandZ
to
business class
Test 2: Non-gold card holder who stays in
economy
Test 3: A person who is bumped from the
flight
Question 17 What is a test condition? c. Functional testing is more powerful than
a. An input, expected outcome, static testing
precondition and as you actually run the system and see what
postcondition happens.
b. The steps to be taken to get the system to a d. Inspection is a form of functional testing.
given point
c. Something that can be tested Question 21 What is the purpose of
d. A specific state of the software, e.g. confirmation testing?
before a test
a. To confirm the users' confidence that
can be run the system
will meet their business needs.
Question 18 Which of the following is the b. To confirm that a defect has been fixed
most important difference between the metrics-
based approach and the expert-based approach correctly.
to test estimation? c. To confirm that no unexpected changes
a. The metrics-based approach is more have been
accurate introduced or uncovered as a result of
than the expert-based approach. changes
b. The metrics-based approach uses made.
calculations d. To confirm that the detailed logic of a
from historical data while the expert- component
based conforms to its specification.
approach relies on team wisdom.
c. The metrics-based approach can be used Question 22 Which success factors are
to verify required for good tool support within an
an estimate created using the expert- organization?
based
approach, but not vice versa. a. Acquiring the best tool and ensuring
that all
d. The expert-based approach takes longer testers use it.
than the
metrics-based approach. b. Adapting processes to fit with the use of
the tool
and monitoring tool use and benefits.
Question 19 If the temperature falls c. Setting ambitious objectives for tool
below 18 degrees, the heating is switched benefits and
on. When the temperature reaches 21 aggressive deadlines for achieving them.
degrees, the heating is switched off. What
is the minimum set of test input values to d. Adopting practices from other successful
cover all valid equivalence partitions? organizations and ensuring that initial ways
of
a. 15,19 and 25 degrees using the tool are maintained.
b. 17,18,20 and 21 degrees
c. 18,20 and 22 degrees Question 23 Which of the following best
d. 16 and 26 degrees describes integration testing?
a. Testing performed to expose faults in
Question 20 Which of these the
statements about functional testing is interfaces and in the interaction
true? between
a. Structural testing is more important integrated components.
than b. Testing to verify that a component is ready
functional testing as it addresses the for
code. integration.
b. Functional testing is useful throughout the c. Testing to verify that the test environment
life can be
cycle and can be applied by business integrated with the product.
analysts, d. Integration of automated software test
testers, developers and users. suites with
the product.
Question 24 According to the ISTQB Question 27 Consider the following decision
Glossary, debugging: table.
a. Is part of the fundamental testing process.
Given this decision table, what is the expected
b. Includes the repair of the cause of a result for the following test cases?
failure. TCI: A 26-year-old on business but with
c. Involves intentionally adding known violations or accidents on his driving
defects. record
d. Follows the steps of a test procedure. TC2: A 62-year-old tourist with a clean
driving record
Question 25 Which of the following could a. TCI: Don't supply car; TC2: Supply car
be a root cause of a defect in financial with
software in which an incorrect interest rate premium charge.
is calculated? b. TCI: Supply car with premium charge;
a. Insufficient funds were available to TC2:
pay the interest rate calculated. Supply car with no premium charge.
b. Insufficient calculations of compound c. TCI: Don't supply car; TC2: Supply car
interest with no
were included. premium charge.
c. Insufficient training was given to the d. TCI: Supply car with premium charge;
developers TC2:
concerning compound interest calculation Don't supply car.
rules.
d. Inaccurate calculators were used to Question 28 What is exploratory testing?
calculate the a. The process of anticipating or guessing
expected results. where
defects might occur.
Question 26 Assume postal rates for 'light b. A systematic approach to identifying
letters' are: specific
$0.25 up to 10 equivalent classes of input.
grams; $0.35 up to c. The testing carried out by a chartered
50 grams; $0.45 up engineer.
to 75 grams; $0.55 d. Concurrent test design, test execution,
up to 100 grams. test
Which test inputs (in grams) would be logging and learning.
selected using boundary value analysis?
a. 0,9,19,49,50,74,75, 99,100 Question 29 What does it mean if a set of
tests has achieved 90% statement coverage?
a. 9 out of 10 decision outcomes have
been
exercised by this set of tests.
b. 9 out of 10 statements have been exercised
by this
set of tests.
c. 9 out of 10 tests have been run on this
set of
software.
d. 9 out of 10 requirements statements about
the
software are correct.
b. 10,50,75,100,250,1000
c. 0,1,10,11,50,51,75,76,100,101
d. 25,26,35,36,45,46,55,56 Question 30 A test plan is written
specifically to describe a level of testing
where the primary goal is establishing
confidence in the system. Which of the III Check a test item for defects introduced by a
following is a likely name for this change
document? IVRecord and report the status of changes to test
a. Master test plan items
b. System test plan V Confirm that changes to a test item fixed a defect
c. Acceptance test plan Which of the following statements is true?
d. Project plan a. Only I is a configuration management task.
b. All are configuration management tasks.
Question 31 Requirement 24.3. A c. I, II and III are configuration management tasks.
'Postage Assistant' will calculate the amount d. I, II and IV are configuration management tasks.
of postage due for letters and small
packages up to 1 kilogram in weight. The
inputs are: the type of item (letter, book or Question 35 Consider the following state transition
other package) and the weight in grams. diagram.
Which of the following conform to the
required contents of a test case?
a. Test the three types of item to post
and three different weights [Req 24.3]