8-5-2019

Specialisten in vooruitgang

Exploratory testing in
theory and practice
Jan Jaap Cannegieter
Principal consultant Squerist

Software testen Business Process Transformation Security testen

Exploratory testing in theory and practice

Missie: verder brengen van mens, klant en expertise

Wendbaar-
Specialisme Mens op 1
heid

1
 8-5-2019

Exploratory testing in theory and practice

Test diensten

(Agile) testen Test automation BI testen Security

Test advies Test uitvoering Test management Test opleidingen

Test partnership

Specialisten in vooruitgang

Why testing?

Software testen Business Process Transformation Security testen 4

2
 8-5-2019

Exploratory testing in theory and practice

Een short movie

Exploratory testing in theory and practice

The most expensive piece of software. Untested

declare
vertical_veloc_sensor: float;
horizontal_veloc_sensor: float;
vertical_veloc_bias: integer;
horizontal_veloc_bias: integer;
...
begin
declare
pragma suppress(numeric_error, horizontal_veloc_bias);
begin
sensor_get(vertical_veloc_sensor);
sensor_get(horizontal_veloc_sensor);
vertical_veloc_bias := integer(vertical_veloc_sensor);
horizontal_veloc_bias := integer(horizontal_veloc_sensor);
...
exception
when numeric_error => calculate_vertical_veloc();
when others => use_irs1();
end;
end irs2;

€ 6.522.600.000

3
 8-5-2019

Exploratory testing in theory and practice

TU Delft geeft gezakte

studenten een diploma
Laatste update: 8 mei 2019 07:28

I want to
prevent this Studenten van de TU Delft die
onvoldoendes voor vakken hebben
gehaald hebben toch een diploma
gekregen. Dit komt door een
softwarefout in de systemen
De fout is ontstaan door een softwarefout.
Volgens de bank komt dit door een fout bij een
leverancier. De klanten halen massaal hun
spaargeld weg bij de bank. Dit is niet nodig,
aldus een woordvoerder van de bank. “alle
klanten worden schadeloosgesteld” aldus de
woordvoerder. , NIBC probeert al jaren een
positie op de consumentenmarkt probeert te
krijgen. Dit helpt daar niet bij.

Exploratory testing in theory and practice

Exploratory testing

10:30

4
 8-5-2019

Exploratory testing in theory and practice

Main streams in testing Testing

Scripted testing Exploratory testing

Test management
Strategy &
planning
Design &
TP 3
Execution Closure
Specification
TP 4

TP 2

Test point

Exploratory testing in theory and practice

Exploratory testing is…

…a style of software testing…
…that emphasizes the personal freedom and responsibility of the individual tester…
…to continually optimize the value of her work…
…by treating test-related learning, test design, test execution, and test result
interpretation as mutually supportive activities that run in parallel throughout the
project.
Source: http://kaner.com/?p=46
and ‘Explore it!’ by Elisabeth Hendrickson

10

10

5
 8-5-2019

Exploratory testing in theory and practice

Essence of exploratory testing

TP 3

TP 4

TP 2

Test point / test idea 1 Based on RST 3.0 by James Bach

and Michael Bolton

11

11

Exploratory testing in theory and practice

Or populair said

12

12

6
 8-5-2019

Exploratory testing in theory and practice

Simultanuous working

Execution Learning

Strategy
(thinking)

13

13

Exploratory testing in theory and practice

Optional discussion
What are the possible benefits of exploratory testing compared to scripted
testing?
What are the possible backdraws of exploratory testing compared to
scripted testing?

How can we structure exploratory testing to make sure we don’t do ad hoc

testing without losing the benefits?

14

14

7
 8-5-2019

Exploratory testing in theory and practice

Possible structuring elements

Overall mission of testing
List of product risks
Insight in the structure of the product
Determine test units
Track coverage
Test charters with testpoints
Test design techniques
Checklists
ISO 25010 / 9126
Defect administration
15

15

Exploratory testing in theory and practice

Example test charter

16

16

8
 8-5-2019

Exploratory testing in theory and practice

Case PHPTravels
Go to www.phptravels.net
This is a test system!

Explore the hotel parts

Make up a test design with
Test goals
Test points

14:00

17

Exploratory testing in theory and practice

15:30

18

9
 8-5-2019

Exploratory testing in theory and practice

Session based testing

13:30

19

Exploratory testing in theory and practice

The need for more stucture

Exploratory testing without good guidance is like wandering around a city
looking for cool tourist attractions. It helps to have a guide and to understand
something about your destination. - James Whittaker

20

20

10
 8-5-2019

Exploratory testing in theory and practice

Session based testing

Session-based testing is a software test method that aims to combine
accountability and exploratory testing to provide rapid defect discovery, creative on-
the-fly test design, management control and metrics reporting
- Wikipedia -

From

To

Based on ‘Exploratory software testing’ by James Whittaker 21

21

Exploratory testing in theory and practice

Elements session based testing

Sessions (timebox)
Mission
Test charters
Test points
Session notes
Session debrief
Metrics
22

22

11
 8-5-2019

Exploratory testing in theory and practice

Session
Timebox
60 min. – 90 min. – 120 min.
Short enough to report in detail
Short enough to plan it easily
Long enough to test in depth
Long enough to debrief effectively
Don’t make a detail planning of the session
Eventually pair testing
Tester – tester
Tester – product owner
Tester – user
Tester – developer
23

23

Exploratory testing in theory and practice

Test mission

Representation of the goal of the test session

Risk based
Defined scope

My mission is to test <<the risk you want to cover>>

for <<the scope/depth/coverage>>

24

24

12
 8-5-2019

Exploratory testing in theory and practice

Test charter
Also named a session sheet
Information of the test item
Mission
Test points
Information about the test execusion (optional)
Defects (or references to the defects in the tool)
New test charters or new test points

See example
25

25

Exploratory testing in theory and practice

Session notes / session ideas

Things you want to know

Things you don’t want to miss
Things you want to test
≈ logical test cases

A good test point starts with a verb

26

26

13
 8-5-2019

Exploratory testing in theory and practice

Session notes
What did I test?
What didn’t I test?
Why did I test it?
How did I test it?
With which coverage and depth did I test it?
Which test data did I use?
On which environment did I test?
What have I found out?
What defects did I find?
What more should I test?
27

27

Exploratory testing in theory and practice

Possible follow up items

Defects
Questions
Possible new charters
Possible new points
Tests that should be automated
Tests that should be documented
Instructions for the next time this charter will be executed
28

28

14
 8-5-2019

Exploratory testing in theory and practice

Session debrief
Mission
Risks Coverage

Techniques Environment

Test session

Status Defects Obstacles

29

29

Exploratory testing in theory and practice

Important skill of an exploratory tester

Focus - defocus

Based on RST 3.0 by James Bach and Michael Bolton

30

30

15
 8-5-2019

Exploratory testing in theory and practice

It is finding the balance between

Discipline - freedom

31

31

Exploratory testing in theory and practice

Choosing the right way

to test

16:00

32

16
 8-5-2019

Exploratory testing in theory and practice

Situational testing bingo

Scripted Exploratory
Category Situation testing testing
System Calculations
User interface oriented system
Backend oriented system
Mobile app
Test goals Checking a set of req. / regulation
Value based testing
Usability
Business rules testing
Performance
Automated checking
Security
Organization Planning and preperation oriented
Young, modern startup
Hierarchical, traditional organisation
Self management, resp. low in org. 33

33

Exploratory testing in theory and practice

Scripted Exploratory
Category Situation testing testing
Documentation A lot and detailed documentation
Little documentation
Constantly changing doc./req.
System Waterval
Agile
Budget A lot of budget
Little budget
Time Early involved
Late involved
A lot of time available
Little time available
Test skills Analytical, accurate testers
Critical thinking (question everything)
Flexible
Professional testers
Non-professional testers 34

34

17
 8-5-2019

Exploratory testing in theory and practice

Testing is exploration and experimentation

Who doesn’t like that?

In case you are interested in a career in QA or testing, please contact me.

All the best with your studies
35

35

18