How to perform

Exploratory Testing by using

Test Charters

By

Anders Claesson
[email protected]

How to perform Anders Claesson R1.0 2007-09-06 1(41)

Exploratory Testing by using Test Carters www.enea.se

Contents

Introduction and definitions

The purpose of testing

Context driven testing

Test preparations

A test charter example

Test execution

Test reporting

When to stop testing

Experiences

Conclusions

Appendix A:
Examples of analogies and
reasoning in Exploratory Testing

How to perform Anders Claesson R1.0 2007-09-06 2(41)

Exploratory Testing by using Test Carters www.enea.se
 Introduction

This presentation will give you an overview and understanding of the concept of
Exploratory Testing.

You will also learn how to use it from the examples and templates included in this
material.

Other Test Methods are also explained to get an overall picture where Exploratory
Testing (ET) can be combined and mixed with other test methods.

ET and the test charters are used to guide the tester in their exploration, which
also have the positive effects of decreasing the required (traditional) test
documentation (TS’s) by at least 90% without losing any significant
information of what is to be tested.

Logging and test notes are very important, to know what
have been covered by the tests.

How to perform Anders Claesson R1.0 2007-09-06 3(41)

Exploratory Testing by using Test Carters www.enea.se

Exploratory Testing definition

Concurrent Test Design and Execution

The most simple definition of exploratory testing is: test design and test
execution made at the same time where the outcome from your most recent
test guide you what the next test will be.

The starting points for your exploration are the specified test ideas and
mission/purpose defined in the used test charter.

ET is the opposite of scripted testing (predefined test procedures, whether
manual or automated).

Exploratory testing is not an unstructured activity.
Test charters on a higher level are used instead of
detailed test specifications.

See www.satisfice.com for more information

How to perform Anders Claesson R1.0 2007-09-06 4(41)
Exploratory Testing by using Test Carters www.enea.se
 The purpose of testing
Testing = Questioning a product in order to evaluate it.

Find the most important problems

in the system before the customer do.

Assure all requirements, needs and expectations

are fulfilled.
Assure all important quality factors are satisfied.
Identify and mitigate all critical risks that may
threaten the value of the product.
Leading to a satisfied and happy customer.
How to perform Anders Claesson R1.0 2007-09-06 5(41)
Exploratory Testing by using Test Carters www.enea.se

Find out what the customer value the most:

User/Customer Value Analysis
Importance Customer Priority Risk/Cost
Quality Attribute
Weight % Satisfaction IW / CS of failure
1. Reliability 30 0,9 33 High Critical

2. Availability 15 0,8 18 High Critical

3. Usability 20 1,1 18 Medium High

4. Performance 20 1,1 18 High Critical

5. Service Level 10 1,0 10 High High

6. Maintainability 5 0,8 6 Medium Low

Summary: 100 % X=0,96

Supplier Customer
costs costs
Also analyze the complexity of the system X < 1: Less satisfied
including all valid and prioritized
combinations of features and platforms. X > 1: More satisfied

How to perform Anders Claesson R1.0 2007-09-06 6(41)

Exploratory Testing by using Test Carters www.enea.se
 Context driven testing [http://www.context-driven-testing.com/]

1. The value of any practice depends on its context.

2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project's
context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn't solved, the product doesn't
work.
6. Good software testing is a challenging
intellectual process.
7. Only through judgment and skill,
exercised cooperatively throughout
the entire project, are we able to do
the right things at the right times to
effectively test our products. A Black Box view of a system under test.
How to perform Anders Claesson R1.0 2007-09-06 7(41)
Exploratory Testing by using Test Carters www.enea.se

Workflow using Test Charters and Session Based Exploratory Testing

Features or Characteristics Specifications Sub-system Specifications and Implementation

Test Analysis Risk Analysis Test Goals Test Strategies

Analysis of implemented
fault codes using Test Charters Test Items Test Items List
Glocksoft resource viewer

Test Execution notes

Further exploration

Spector Log-files Incident Reports Test records Test progress reports

When exploring/testing Functions, Features or Characteristics, any of the Release

following test techniques might be efficient such as equivalence class Final Test Report
testing, boundary value analysis, domain testing, pair wise testing,
state based testing or scenario based testing depending on the nature of the system under test.
How to perform Anders Claesson R1.0 2007-09-06 8(41)
Exploratory Testing by using Test Carters www.enea.se
 Test Methods: Risk Based Testing
Objective: Risk Based Testing is a method of identifying potential problems early
(by a technical Risk Analysis) and focus testing in these areas. The risk
based approach gives the tester the opportunity to discuss the problems
before they occur and possibly prevent them. The amount of Test Cases
to run can be decided based on the current risks. The test effort can be
continuously changed and re-focused based on the main problem areas
found during the progress of the project and the test execution.
Limitations: It takes a lot of experience, skill and preparation to find the risks in the
first place. It might be difficult to get the key persons participating in the
risk analysis meeting and there might not be enough time to discuss the
potential problems in detail. After the risks have been identified people
often don’t know what to do with them, or they tend to ‘”forget” their
analysis because they are so busy trying to get their things to work in the
first place (or just continue working as they are used to).
Note that it is much better to, at least, attempt finding the problems before
they occur to be able to get the designers prevent some of them before they
hit the system and a lot of costly correction work have to be done. Testers
can also focus their test effort on the most critical areas from the beginning
of the test period. Low risk test cases can also be removed completely.
How to perform Anders Claesson R1.0 2007-09-06 9(41)
Exploratory Testing by using Test Carters www.enea.se

The Risk Based Testing process

Define the scope of the risk
1 Scope analysis, e.g. robustness.

2 Input
data
Identify and collect all input information
needed for the risk analysis.

Prep. Present all input information for

3 meeting the risk analysis participants.

Risk Identify the risks, prioritize, suggest actions

4 analysis including the name of the action responsible.
For the next iteration, start the
process over again with the TC Select test strategies, test ideas to evaluate the
new features/ functions being 5 selection risks (consider other aspects as well).
delivered from design. The
second run usually goes faster
when the risk process has been
learned by the team. 6 Follow
up
Have weekly follow-up of the progress of
the risk evaluation (testing) and mitigation.

At the end of the iteration, go through how much of the test object was actually
Evaluation covered, how many test charters/ideas that were executed, the risks that were
7 covered, how many faults were found (any stopping or severe faults), leftover
meeting Test Cases, if the initial test strategy worked or not, and why. This learning
process is very important in order to improve in the next iteration!
How to perform Anders Claesson R1.0 2007-09-06 10(41)
Exploratory Testing by using Test Carters www.enea.se
 Risk example
Prio Risk description P*C, T Actions

1 Risk: Unstable system platform. The 5*3= Risk reduction and evaluation
interconnections and interactions between the 15 strategy:
functions in the platform may not work. Ask 3PP vendors how they have tested
Why: Purchased platform components for this system T=2 their products.
developed by 10 independent, third party vendors. Perform platform tests to evaluate the
Components were not tested together by vendors. interactions between components.
Why: Application parameters are not set correctly. Test items:
How: May cause failures in the communication and Platform test of system xxx.
the functional behavior of the application functions. Test ideas:
When: Happens when the application starts up and - Test of individual important

C = Consequence
when user data is entered into the system. functions.

P = Probability

T = Testability
Where: Problems may be visible in the interface - Tests of related function groups,
between the application and the middleware. bottom up to the application
interface level.
Which part: The faults reside in the functions of the
HP Unix server operating system.

How to perform Anders Claesson R1.0 2007-09-06 11(41)

Exploratory Testing by using Test Carters www.enea.se

Exploratory test preparation - selection and prioritization of test ideas

1 Identify test items (charters covering a Use Case or a feature, function or char.)
- Identify which test items/charters to include based on the test scope and the test goals
for the current area. A Test item could be a function, feature, Use Case, characteristic, etc.
2 Identify the number of required test ideas based on your ranking below.
- How much testing is needed to cover the areas within your test charters?
3 Select the most important test ideas based on the following attributes:
No Attribute Description Ranking
1 Risk Risk level for the area (Feature, Characteristic or system part). H, M, L
2 Required Priority of the requirement (from the customer and/or product management). H, M, L
How complex the feature, product or characteristic is with regard to internal logic and
external connections to other components, systems, interfaces and/or amount of
3 Complexity H, M, L
units/features (3PP, multiple usage of features, simultaneous users, background load/traffic
mix, etc.) to be tested.
How much the customer value the current area (Feature, Characteristic or system part) in
4 Value H, M, L
their business/operation.
Importance of the system part, interface or function to achieve a stable, responsive and
5 Importance H, M, L
capable operation of the system (e.g. restart functions).
6 Failure Severity if failure occurs in the area to be covered. H, M, L
7 Probability Probability of how likely a failure is in the area. H, M, L
8 Cost Cost of failure (both internal and for the customer). H, M, L
9 Visibility Failure visibility for the user. H, M, L
10 Tolerability Tolerability of failure, i.e. how tolerant the customer/user is of a failure occurring. H, M, L
11 Human factors How usable the system is in the current area. H, M, L
For more information about how to select Test Cases, see the book: Software Testing Fundamentals by Marnie Hutcheson
http://www.ideva.com/
How to perform Anders Claesson R1.0 2007-09-06 12(41)
Exploratory Testing by using Test Carters www.enea.se
 Exploratory test preparation

Create a Bug Taxonomy: Categorize common types of faults found in
previous projects and in operation (including Root Cause Analyzes),
feedback from maintenance is very important. The taxonomy is a good input
for the identification of risks and selection of test ideas. Look for systematic
faults your designers usually do in every project.
Poor accessibility Example of problem Compromised client privacy
categories from an
Scalability e-commerce web site. Poor usability

Compliance Calculation/computation errors

How does the function fail with respect to each category?

Bad software upgrade Web server failures

Network failures Third party software failures

Accessibility Failure at ISP/ Web host

System security

Network failures

Bug Taxonomies: Use Them to Generate Better Tests

[http://www.testingeducation.org/a/bugtax.pdf]
How to perform Anders Claesson R1.0 2007-09-06
Exploratory Testing by using Test Carters www.enea.se

The green parts are the test

techniques most commonly used
for system level testing. Testing
The identified risks are the starting
Dynamic point for all testing. Exploration of your
product are encouraged by the use of
Static
any of the described techniques here.

Black box White box Analysis Review

Equivalence Statement Measure
Partitioning (complexity)
Formal Informal
Branch/ Find Faults
Boundary Decision (using the
Value compiler or Walkthrough
Analysis Error static analysis
Technical
tools)
State Guessing Review
Transition Goals for Dynamic and (Peer)
Runtime Black Box Testing
Domain Analysis Find faults, show that the Inspection
Analysis system works, is robust,
Goals for Static Testing
Scenario/ Decision Pair evaluate risks and to show the
Use Case Table Wise customer that all requirements, Verification, validation, consensus,
needs and expectations have identify Improvements, fault finding.
been met.
How to perform Anders Claesson R1.0 2007-09-06 14(41)
Exploratory Testing by using Test Carters www.enea.se
 A test charter template for Exploratory Testing
Test Charter id and unique name
Actor/Theme: <The user role you as a tester is going to play when using the system or theme when testing
system characteristics>
Purpose: <The purpose of the charter including what particular objectives the actor is supposed to achieve or
what the expected outcome of the theme exploration should be>
Setup: <What needs to be in place in order to start using the intended function or characteristic , i.e. the
precondition to be able to test (configuration and the initial system state, e.g. a start menu should
appear on the screen with login fields or what background load scenarios should be up and running)>
Priority: <High/low, based on what the priority of the requirements are, or other considerations (risk level)>
Reference: <Specification (e.g. Requirements Specification), risk or other information source>
Data: <Reference to a data file with input data to be used, e.g. an Excel sheet. Variations of data to be
made based on the initial data can also be specified, e.g. use boundary values of valid and
invalid input data for all parameters, use pairings of input data to catch combinatorial problems>
Activities: <What the actor should do with the system, e.g. Log on to the system as a super user:
- Add a new user.
Test ideas - Attempt to delete a non existing user>
Oracle notes: <Things to pay special attention to regarding system response and what is expected as an
adequate/correct outcome, e.g. Check that correct error messages are given that informs the
user about what mistake he made and how to do it correctly next time. It could also be a
reminder to the tester to check for buffer consumption or other important resource usage>
Variations: <Additional system or user events that could occur at anytime the normal activities are performed
as described above under Activities, e.g. another super user logs on to the system trying to
manipulate the same data as the first is working on, or the network connection suddenly fails>
For more information about test charters with examples see: http://www.satisfice.com/rst-appendices.pdf, http://www.satisfice.com/rst.pdf

How to perform Anders Claesson R1.0 2007-09-06 15(41)

Exploratory Testing by using Test Carters www.enea.se

The test charter

A typical test charter includes the work of a pair of testers working together
for a 90 minutes period of uninterrupted time, i.e. time boxing (cell phones
shut off, no email pop-ups active, etc.).

The test charter should suggest what to be tested, how it should be tested
and what problems to look for (OBS, not a detailed step-by-step instruction).

Chartered testing is mainly used for exploratory testing and require more
system and testing knowledge than scripted testing (but the fault finding
efficiency is 2 - 4 times higher). Try to play chess with someone using a
predefined strategy without consider changing the strategy based on what
moves your opponent makes.

Chartered testing encourage testers to react on the
response from the system and think for themselves
what a correct outcome should look like and also
use their creativity how to break the system. The
tester will also be encouraged to look for different
ways how it might fail to meet the users needs
expectations and requirements
(e.g. usability problems).
How to perform Anders Claesson R1.0 2007-09-06 16(41)
Exploratory Testing by using Test Carters www.enea.se
 The test charter

A test charter can be seen as a mirror of a Use Case. The Use Case represents the
design view, while the test charter represents the test view, i.e.:
1. What data, test environment and configuration do we want to use when testing
this Use Case or characteristic?
2. From which actor or theme do we want to give input and observe the output
when exercising the specific paths we want to cover in the Use Case, or quality
attributes for the characteristics.
3. The test ideas are the starting point for the exploration and are similar to Test
Cases, with the main difference that it is not a step-by-step description of how
to perform the test, it is a one line statement of what might be a good starting
point regarding what to do with the feature to cover the Use Case, its paths and
to find the most serious problems as fast as possible. If the tester can’t
figure out how to use the feature or evaluate the characteristic for him/herself,
how could we expect the users to understand how the feature should be used?

This way of working with exploratory and chartered testing is a completely different
approach, which might need some discussions and acceptance on management
level. Instead of reporting how many Test Cases were executed the testers report
how many charters and how much test effort have been subjected to a specific
feature or characteristic based on the current risk levels and other factors.

How to perform Anders Claesson R1.0 2007-09-06 17(41)

Exploratory Testing by using Test Carters www.enea.se

The test charter

An illustration of the two views comparing Use Cases and test charters:
A use case is a single A test charter is a description of HOW an end user is
task, performed by the end supposed to use the system in a real situation by executing
user of a system, that has specified tasks in different ways. The intention is to execute
some useful outcome. I.e. what may be useful for him/her or any other system actor,
WHAT the user should be which may show weaknesses in the way it is intended to be
able to do with the system. used considering what the user wants, needs and expects.
Role description of the actor and the theme for the
exploration, i.e. what do we want to find out about this feature,
actor or characteristic within the scope of this charter?.
Test setup, data and suggested test methods/techniques to
be used. Try to use a customer like environment to catch any
compatibility issues. Also outline an overall test strategy/
approach to be used in this charter (in the purpose section).
Activities described as test ideas how to perform the different
parts the actor should be able to do. Test ideas for negative
testing should also be included.
Things to pay special attention to when executing different
parts (known weaknesses or checking error messages for
relevant or misleading information). Find useful test oracles.
Suggested variations when executing the different parts of the
Use Case, e.g. interruptions, interactions with other Use Cases,
etc. that can happen anytime during the use of the feature.
How to perform Anders Claesson R1.0 2007-09-06
Exploratory Testing by using Test Carters www.enea.se
 Example of a test charter for a 90 minutes test session
Test Charter 1: Analyze the copy/paste function of pictures
Actor: Normal user
Purpose: To evaluate if the copy/paste function of pictures works in our word processor together with the most
commonly used word processors on the market (Word, Power Point., etc.) and other programs where
pictures can be inserted in the copy buffer for copy/paste operations in the PC. The purpose is also to
see that no information is lost or corrupted.
Setup: A Dell PC with 2 Gb memory, our word processor, the Microsoft Office package professional and the
home edition 2003 patch level xx, PDF reader version 7.1, Notepad version 4, Internet Explorer
version 7, Opera version 5, Mozilla……etc. (the setup might be common for several charters and can
therefore be described and referred to instead of repeating the same information in every charter).
Priority: High, because this function is used very frequently both within our own word processor, but also
between other word processors and programs, the user may want to copy/paste pictures with ours.
Reference: Requirement abc_1 in document Def-07:034 revision R1.1. Risk number 3 from the risk assessment
held on April 4 2007 regarding the copy function, which is documented and followed up in the document
Rsk-07:012.
Data: A variety of pictures with different resolutions, both vector graphics as well as bit mapped pictures. The
pictures could be photos or figures in web browsers for example. Complex pictures are also included
and pictures which might be copy protected in some way.
Activities: 1. Copy/paste a picture in our word processor from one location to another in the same document.
2. Copy/paste a picture to and from our word processor into/from Word, PowerPoint and Excel.
3. Copy a picture into our word processor from a variety of the mostly used Web browsers.
4. Copy/paste a picture to/from the most commonly used web site building tools such as Dreamweaver
from Adobe.
5. Copy a picture from a PDF document using Acrobat Reader into our word processor.
6. Try to copy a write protected picture from the web or other source into our word processor.
7. Try to copy/paste a variety of non readable and some readable ASCII-characters or corrupted pictures.
www.enea.se

A simple example of a test charter, continued….

Test Charter 1: Analyze the copy/paste function of pictures, continued….

Oracle notes:
Look if the size of the pasted picture changed on the screen (it should not).

Check if there is any loss in the resolution of the picture especially when you copy and paste from
other programs to or from ours.

Check which is the highest resolution picture that can be copied and how that affect the system.

Check for memory leaks and how much memory the copy/paste operation takes from the system and
how that affects the use of other programs. Other programs should not slow down or be affected in
any way.

Print pasted pictures to see if there are any differences in color, resolution or any other anomaly.
Variations:  Try out (and find) the boundaries of how large pictures are possible to copy/paste.
 Perform copy/paste with a large number of items in the copy/paste buffer in your PC.
 Try to fill the copy/paste buffer to its limit and then copy/paste a picture and see what happens.
 Try the longest file name of the picture you can type before making the copy/paste. You may use the
tool Perlclip (download from http://www.satisfice.com/tools/perlclip.zip) to generate file names with a
million characters or more if you want.

How to perform Anders Claesson R1.0 2007-09-06 20(41)

Exploratory Testing by using Test Carters www.enea.se
 Example of a Test Charter for Robustness Testing
Test Charter 2:
Theme: CEP handling (check of internal node communication).
Purpose: To trigger failures in the CEP handling. For the internal communication in the node, many changes have
been made in design which may be a high risk. Faults have been reported from the customer and
tests made by the Product Line Maintenance organization. If CEP fails in operation, it can not be found at
the customer site. Communication is interrupted. Cells can not, for example, be possible to get up and
running.
Setup: Minimum C-node configuration.
Priority: High.
Reference: IR <abc_identity> and system documents:
Xxxx Movable CEP in RNC,
Yyyy Node Module MP’s in Pool.
Data: Traffic transactions through the node (BAT). Load modules interacting with CEP handling (see below).
Activities: No Prio Description
1 H Module Main Processor (mMP) Plug In Unit (PIU) locking.
2 H Perform restarts of mMP or RPU switching during interactions with CEP handling.
3 H Restart of LmCell (Load Module for handling cell administration).
4 H Restart of UNI-SAAL (CEP resource control program).
5 H Restart of RoamFro (xx).
6 H RPU switch of RoamFro (xx).
7 H Remodule IubLinks.
How to perform Anders Claesson R1.0 2007-09-06 21(41)
Exploratory Testing by using Test Carters www.enea.se

A simple example of a test charter, continued….

Test Charter 1: Analyze the copy/paste function of pictures, continued….

Oracle notes:
Check that UNI-SAAL Termination Point (TP) moves successfully after 120 seconds to another
module xx when yyy is locked.

Check that restarts don’t disturbs traffic handling or the system performance.

Check that it is possible to perform RPU switch back after board restarted, board removed or
blocked.

Check trace logs for error messages.

Check status of the node/boards.
Variations:  Select a number of load modules that interacts with CEP handling when performing restarts or RPU
switching.
 Perform cold, warm and refresh restarts.
 Restart board, block board or remove to trigger switch to redundant board.
 Or other variations that the tester might find useful.

How to perform Anders Claesson R1.0 2007-09-06 22(41)

Exploratory Testing by using Test Carters www.enea.se
 Exploratory test execution Expect the unexpected
1. Keep your mission clearly in mind. You never know what you
will find.
2. Keep notes that help you report what you Continue on the paths that
did and why you did it, that support your evokes your curiosity until
you have reached a point
assessment of the perceived product quality. where you don’t see
anything suspicious
3. Keep track of questions and issues raised in your anymore, and you consider
exploration (constant logging, taking notes and the probability of any
debriefing after each test session). remaining faults being
reduced to a level where
4. To supercharge your testing, pair up with another there are no major risks in
that that area.
tester and test the same thing on the same
computer at the same time. You may need to back-
track to a previous stable
state if you reached a
dead end where further
testing will not bring any
added value.
Initial test idea

Evaluate the outcome

of every test attempt
before deciding what Incident Reports should include enough details to be able
to do next to recreate the test that lead to the discovered failure.
How to perform Anders Claesson R1.0 2007-09-06 23(41)
Exploratory Testing by using Test Carters www.enea.se

Exploratory test execution

The results from the tests you design The more we can make testing
and execute influence the next test intellectually rich and fluid, the more
you will choose to design and likely we will hit upon the right tests at
execute. the right time [Bach].

You build a mental model of the
product while you test it. This model
includes what the product is and how
it behaves, and how it’s supposed to
behave.

You test what you know about, and
you are alert for clues about
behaviors and aspects of the product
that you don’t yet know about. But
beware of your biases. You may only
see what you expect to see.
By writing down test scripts in advance
and following them tends to disrupt the
intellectual processes and make testers
unable to find important problems
quickly [Bach].
How to perform Anders Claesson R1.0 2007-09-06 24(41)
Exploratory Testing by using Test Carters www.enea.se
 Exploratory test execution - Taking testing notes
When performing exploratory testing it is very important to take notes of
what you are doing, seeing or suspect, which could be an anomaly or a fault.
Make your own conclusions regarding the quality of the function based on
the results from your explorations.
Category Notes
<What input data have I studied and given feedback on to improve its quality?
Test coverage Which reviews have I participated in?
How much have I covered in my test execution and how much remain>
<What have I learned so far by executing the test ideas/attempts for these specific
functions? Does it seem stable? Have I found any faults?
Evaluation notes
What test attempts am I going to do next based on my idea list and current
observations>
<Which risks have been covered and which ones remain among the most
Risk/Strategy list important ones? Which part of the initial test strategy should I follow and does it
need any changes?>
<Any specific notes I would like to highlight in addition to the recordings made
Test execution log during the test execution>
<Any strange behavior I don’t understand. Is there anything wrong with my test
Issues, Questions
idea/test attempt, test environment, test data, misunderstanding of the function,
and Anomalies test script or the system under test?>
How to perform Anders Claesson R1.0 2007-09-06 25(41)
Exploratory Testing by using Test Carters www.enea.se

Useful tools for Exploratory testing

Glocksoft Resource Viewer is intended for viewing of resources of executable files
(with the extension .exe and .dll). It displays information about program resources including
dialogs, icons, strings (such as error messages the developer has put into the code but
“forgot” to document for the user) and more, it allows you to open any executable windows or
resource file and gain access to the Windows resources that it contains. The cost of the tool is
$75 per license for one PC.
http://www.glocksoft.com/resource_viewer.htm

Spector Pro is used for monitoring and recording every detail of all PC and Internet
activities (logs everything you do, with time stamps). All details of what you do on the
computer – chats, instant messages, emails, the web sites visited, what searches are made,
posted pictures and which you have looked at, all keystrokes, all programs that have been
run on the computer and much more. And because of its advanced surveillance screen
snapshot features, you get to see not only WHAT is happening on the screen, but the EXACT
order in which it is done, step by step. The cost of the tool is $99,95 per license for one PC.
http://www.spectorsoft.com/products/SpectorPro_Windows/

How to perform Anders Claesson R1.0 2007-09-06 26(41)

Exploratory Testing by using Test Carters www.enea.se
 Testing combinatorial explosions with a Two-wise parameter selection

One of the most difficult testing problems is the demand for flexibility in many
systems where the behavior of the available Features should be possible to
change and adapt according to customers’ needs.

There could be thousands of parameters that should be possible to change.
Features could be optional and/or adapted in many ways. Also interactions
with other Features may be possible to tailor for a specific customer and
application.

When the flexibility increase the number of possible tests grow exponentially
which makes testing much more Two-wise configuration combinations
difficult (and often impossible to cover all Combination
combinations) if using traditional methods.
Fortunately there exist techniques to handle
such situations. The test technique is called
n-wise combinatorial testing.

All two-wise parameter combinations are
included in the example to the right covering
each value at least once. The application
under test are then added for each configuration.
How to perform Anders Claesson R1.0 2007-09-06 27(41)
Exploratory Testing by using Test Carters www.enea.se

Why Two-wise testing finds many logical and data dependent faults

Most faults are of the category of:
One Factor faults
A one factor fault is a consistent problem triggered by any of the
possible input parameters equivalence classes (either only one, or up to all
of the equivalence classes from one to all parameters) used by the
function. The function under test does not work, and any test using that
input value range or a specific value in the range of that function would find
the fault.

Two Factor faults
If there exists a consistent problem when specific equivalence classes with
two parameter values occur together, it is called a two factor fault. Indeed, a
two factor fault is an indication of a two-wise incompatibility or incorrect
interaction between two parameters and their selected values. It is the
pairing of this functions specific parameter value (or range of values) with
another functions specific parameter value (or range) that fails even though
all other pairings work properly.
How to perform Anders Claesson R1.0 2007-09-06 28(41)
Exploratory Testing by using Test Carters www.enea.se
 Why Two-Wise testing find many logical and data dependent faults
A two-wise coverage approach would find most of the faults (70 - 98%) when
testing possible parameter values used by a large numbers of functions and
their parameter values combined. Tests can be reduced by 99,99999…% or
more compared to exhaustive testing, which is impossible in most cases.

Smarter Testing
finds more faults
in less time.

How to perform Anders Claesson R1.0 2007-09-06 29(41)

Exploratory Testing by using Test Carters www.enea.se

A useful tool for Two-wise/Pair-wise testing

PICT The Pair-wise Independent Combinatorial Testing tool (PICT) can help you efficiently
design test cases and test configurations for software systems. With PICT, you can generate
tests that are more effective than manually generated tests and create them in a fraction of
the time required by hands-on test case design. PICT generates a compact set of parameter
value choices that represent the test cases you should use to get comprehensive
combinatorial coverage of your parameters.
http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-
8c6f76cc014b/pict33.msi

How to perform Anders Claesson R1.0 2007-09-06 30(41)

Exploratory Testing by using Test Carters www.enea.se
 Test reporting
Initial Current
Needed test Current Q
Test area Risk test W1 W2 W3 W4 W5 W6 Comments
effort risk level ass.
Level effort

Feature(s) not yet delivered

from design and integration.
Area 1 Low Low, QL1 Low None QL0 QL1
RFT not fulfilled.
No testing possible.

Area 2 Medium Medium, QL2 Low High QL0 QL1 QL1 QL2 On track, no faults.
Area 3 High High, QL4 High Blocked QL0 QL1 Crashes, IR12345

Area 4 High High, QL3 Medium Pause QL0 QL1 QL1 QL2 IR1212 under investigation.

Area 5 Medium Medium, QL2 Medium High QL0 QL1 QL2 Configuration problems

The Current Risk level is based on the results from all performed preventive actions and/or all performed tests at the
current week when the progress report was written. The Quality Level includes required test coverage, the level of
positive/negative tests, test effort and achieved stability under various usage and load conditions to be reached for a
“sufficient” system quality.

= Delayed test item according to plan. Not Ready For Test from design and system
integration.
QL1 = Planned ready date to achieve a specific Quality Level.
QL2 = QL-level achieved on time according to the test plan.
QL1 = QL-level achieved later than planned.
How to perform Anders Claesson R1.0 2007-09-06 31(41)
Exploratory Testing by using Test Carters www.enea.se

When to stop testing

Coverage Quality
All planned/required test ideas and test Testing should stop when:
charters should have been executed and
• The probability of remaining
passed, according to the current risk
faults is reduced to a level that
areas/levels, where faults are found (by
can be accepted by the
exploratory testing) and the coverage
customer [B. Beizer].
objectives stated in the test goals (e.g.
system requirements, Use Cases, etc.). • No open priority A IR’s.
• The systems’ risk level is
within acceptable limits (i.e. no
critical risks remain unsolved).
• The product value has been
demonstrated and accepted (i.e.
implicit and explicit quality
attributes are satisfied).

Time
When the agreed ship date has been
reached.

How to perform Anders Claesson R1.0 2007-09-06 32(41)

Exploratory Testing by using Test Carters www.enea.se
 When to stop testing When looking at the curves pay attention to the trends.
Extrapolate and see if you are going to be able to meet your
test plans. If not, communicate and reprioritize immediately by
Number of Total risk level considering the risks, failure rates, coverage and test case
identified Risks follow up execution progress.
[High/Medium/Low]
A risk may be Current
downgraded from
date with risk level Consider number of risks that are:
Suspected problem areas

high to medium.
and failure rate. - Open.
Resulting in
new risks. - Under investigation/testing.
High - Resolved.
- In each category of: High, Medium, Low.

Medium At the release date there shall be no

unsolved high priority risks or critical
Low Incident Reports open. Planned test
Goal coverage should have been achieved.
New problem areas
Known problems in the system

discovered during Time - Weeks

testing.
Fault intensity Feedback for improvements
[Faults/week] Internal defects External defects
Bad test case Good test case
IR - C selection selection

IR - B
Acceptable
quality IR - A
level
Testing Test stop criteria fulfilled. Time - Weeks
starts System delivered and put into service.
How to perform Anders Claesson R1.0 2007-09-06 33(41)
Exploratory Testing by using Test Carters www.enea.se

Appendix A
Example of forensic test methods used by the police
The police has just arrived to a crime scene in an office building somewhere in
Stockholm, where at least 10 computers have been stolen. The main objective for
the police is now to secure as much evidence they can. The personnel in the office
building show them around starting with the place where the thieves entered the
building. The technicians could now start their job. The first thing they do is to map
how the suspects have moved around in the building. The technicians must use a lot
of their imagination, creativity and reasoning based on their previous experiences
and knowledge. They start by thinking – where would I put my hands and feet if I
have climbed in this way? This is the way they need to think by putting themselves in
the shoes of the suspect.
On a paper lying on a desk just beneath the smashed window they found a stain of
blood. The police takes a Q-tip out of his bag and gently touch the stain to absorb
some blood. This is all that is needed to get enough DNA and connect the
perpetrator to the crime scene. If the DNA can be secured at the scene, the
probability is quite high there is a hit in the DNA register of a suspect. It is very
common that the same person has done several crimes in the past. At another crime
scene a fingerprint on a milk box led to that the perpetrator could be connected to
another 14 committed crimes in the past.
How to perform Anders Claesson R1.0 2007-09-06 34(41)
Exploratory Testing by using Test Carters www.enea.se
Appendix A
Example of forensic test methods used by the police

Entry point to the The case with different Securing evidence using the Logging of test results for
crime scene. test tools to be used for DNA-analysis test method. later analysis at the lab.
different test methods.

Forensic test methods are by its nature tools in the exploration of what really might have happened before the
investigators arrive at a crime scene. They are very thorough in order to keep all the evidence and the pieces of the
puzzle together in order to hold up in court. No clue is left unexplored, even if it doesn’t make sense by itself the
collection of all the evidence might give the whole picture which would not be possible to see by looking at each
individual evidence. Software testers should work and think in the same way, but are rarely given the time for reflection
by piecing all the individual parts together to give an accurate image of how the user may experience the product
under development. Companies that value continuous improvements and see their customer as their most valuable
input for feedback prosper because of their company culture (like Toyota) while others who listen more to their
engineers suffer from the disconnection between what the customer wants and what the engineers like to do.

How to perform Anders Claesson R1.0 2007-09-06 35(41)

Exploratory Testing by using Test Carters www.enea.se

Appendix A
Analogies with software testing
You are testing in your lab and just found something weird which may or may
not be a fault (you don’t know yet). But it intrigues you into further investigation. Your
main objective as a tester is to find as many important problems in the product as
possible before the customer do. You look at the evidence that caught your interest
and try to catch some more information if possible. By looking into your log files and
also analyze what actions you did before
reaching the current state/situation
will give you more clues. You can
use a checklist of what to look for
so you don’t miss anything (in
addition to your own creative
thinking of what information
might be useful).

How to perform Anders Claesson R1.0 2007-09-06 36(41)

Exploratory Testing by using Test Carters www.enea.se
Appendix A
Analogies with software testing
Look at your test oracles if you can find any help whether your test generated any
undesired outcome from the users’ and/or customers’ point of view or not. Proceed
by considering likely options of what to do next to get more information that supports
your most probable hypothesis – where might the designers have misunderstood the
requirements, used new untested technology, not followed the design rules, made
any logical faults, used faulty assumptions, a lack of communication among
designers regarding the interactions between their modules, skipped unit testing
partly or completely (because of time pressure), etc. You don’t know. Use available
test methods, techniques and tools that best suit the situation. When you put yourself
in the shoes of the designer (it is not forbidden to talk to the designers and ask
questions!) you may get a better perception of their
world and where they usually make mistakes.
After exploring the most likely possibilities you
either find the fault or dismiss it, considering
your initial assumption whether there was any
mistake made by the designer or not. Oracle example:
You may also have misunderstood Command buttons
how to use the function you are should be used only
for frequent or critical
testing (a testing mistake, but could immediate actions
on the other hand be a usability issue).
How to perform Anders Claesson R1.0 2007-09-06 37(41)
Exploratory Testing by using Test Carters www.enea.se

Appendix A
Analogies with software testing
Now put yourself in the shoes of the user. What kind of exploration would you do
in this case? Would it differ in any way compared to the designers view (even after
you have looked at your test oracles including user/customer aspects)? Would the
user be annoyed or irritated if your observation of the possible anomaly would go into
the final version? Does the feature you are testing really solve the users’ problems or
abilities to reach his goals in an efficient way? Is your initial observation in any way
causing the user to be mislead how to proceed, e.g. by a faulty error message or any
other usability aspect?
By using a bug taxonomy you can create your own “DNA” record of observed
problems, likely mistakes and faults made by the designers. In combination with Root
Cause Analyses of the most serious faults found to date (where the faults reported by
your customer should be the most important ones) we are in a far better position of
making better risk assessments before testing starts, select more efficient test ideas
and, when executing, quickly get the problems we see getting fixed by the
developers and hopefully reduce any debates regarding incident report priorities.
Things we know the user/customer regard as very serious should be paid special
attention to in testing (and in the prioritization of which bugs to prioritize).
How to perform Anders Claesson R1.0 2007-09-06 38(41)
Exploratory Testing by using Test Carters www.enea.se
Appendix A
Analogies with software testing
An interesting observation you may elaborate on is why we use the same (in most
cases) test techniques as we did 30 years ago (except for pair-wise testing) when
they were discovered (equivalence partitioning, boundary value analysis, etc.). There
are no scientific research published to date that tells us the efficiency of our known
and most used test techniques, which types of faults they find and which faults that
slip through. There are no silver bullets in software testing but
it would be very helpful to see in which
context a certain set of techniques are efficient
and why. The reason for such investigations
not taking place is probably that companies
don’t want to expose their way of testing,
which faults they find and how many that slip
through. But you can at least make
it in your own company.

How to perform Anders Claesson R1.0 2007-09-06 39(41)

Exploratory Testing by using Test Carters www.enea.se

Experiences

To create the test charter was made as a group activity and only
took one hour to do.

A blocking fault on the first test attempt/step halted further
execution of the charter.

All testers in the group felt it was more productive to create
charters. One problem is that they have a lot of old test
specifications and test cases. There is a resistance to change to
something new. An idea they had was to refer to old test cases,
when applicable in the list of test ideas, not to loose previous tests
that had been productive.

”System upgrade” will be the next area to create test charters for.
Risk Based Testing will then be combined with Exploratory Testing.

How to perform Anders Claesson R1.0 2007-09-06 40(41)

Exploratory Testing by using Test Carters www.enea.se
 Conclusions

Chartered testing focus on the purpose of testing to a higher
degree than scripted testing, i.e. to find the most serious problems
before the customer do.

Chartered testing is in general 2 – 4 times more efficient in finding
bugs than scripted testing.

Scripted testing doesn’t work as well as
chartered testing under pressure.
When forced to execute test cases there
is no time to explore any anomalies.

Chartered testing gives the tester more
freedom to be creative to find the faults faster.

Since ~50% of the requirements are faulty or
incomplete, scripted tests usually becomes
even worse.
How to perform Anders Claesson R1.0 2007-09-06 41(41)
Exploratory Testing by using Test Carters www.enea.se