Exploratory Testing

Maaret Pyhäjärvi
This book is for sale at http://leanpub.com/exploratorytesting

This version was published on 2020-08-09

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing
process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and
many iterations to get reader feedback, pivot until you have the right book and build traction once
you do.

© 2015 - 2020 Maaret Pyhäjärvi

Contents

Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Why This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

The LeanPub way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

My work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
My Side Business . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
My history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Taking Learning Seriously . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Conference Speaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Idealist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
From Finland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

What and Where of Exploratory Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

What is Exploratory Testing? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

An Approach to Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Specifying It By Contrast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
The Three Scopes of Exploratory Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Listen to Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

What is Exploratory Testing - the Programmer Edition . . . . . . . . . . . . . . . . . . . . . . 12

What Testing Gives You? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Exploratory Testing, eh? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Where is Exploratory Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Place of exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
An example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Separate Tester Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Future is Here, Just Not Evenly Distributed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

The Two Guiding Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
CONTENTS

Opportunity Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

How to Explore with Intent - Exploratory Testing Self-Management . . . . . . . . . . . . . . 24

Intertwining Different Testing Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Learning To Self-manage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
You’re In Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Going beyond Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Size of heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Purpose of heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
The following chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Recall Heuristics for Test Ideation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Recall on level of change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Recall on level of feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Automation First Microheuristic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

How would you test a textfield? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Shallow examples without labels - wannabe testers . . . . . . . . . . . . . . . . . . . . . . . . 37
Seeing the world around a text box - functional testers . . . . . . . . . . . . . . . . . . . . . . 37
Tricks in the bag come with labels - more experienced functional testers . . . . . . . . . . . 38
Question askers - experienced and brave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
The three exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Exploring Gilded Rose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Introducing Gilded Rose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Priming With Information Sources and Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Getting to Work—How Would I Test this? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
The First Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
The Second Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Reading the Spec While Aiming for 100% Branch Coverage . . . . . . . . . . . . . . . . . . . 48
Tools Ease The Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Summing up to some fun pitfalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Exploring a WebUI with Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Two sessions, two results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Breakdown of activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Examples of activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Some Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
The Final Deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Python Primer for Exploring an API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

CONTENTS

Python, in the interpreter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Python like a script, in Pycharm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Python like a Class and Methods, in Pycharm . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Exploratory Testing an API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Why Is Exploratory Testing An API Relevant? . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Testing as Artifact Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Testing As a Performance, (aka Exploratory Testing) . . . . . . . . . . . . . . . . . . . . . . . 62
What Does Testing Give Us? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
An Example API with ApprovalTests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
13 Patterns To Help You Explore An API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Anchoring an Idea While Exploratory Testing an API . . . . . . . . . . . . . . . . . . . . . . . 71

Reporting and Notetaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Test Case Management Tools in Exploring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

What Zephyr Brings In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
What I Bring In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Feature and Release testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

Training and Exploratory Tester . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Dedication
I have three dedications to pass, the book wouldn’t be in existence with any of you missing.
Huib Schoots, thank you for your support and trust in me writing this book. As I started my writing
journey on Exploratory Testing, I learned a respected colleague in the theme had beat me to the
obvious name on LeanPub. He did not only pass the space on to me, but also encouraged me to fill
it. This book could not exist without you.
People paying for this book have provided me critical support in writing, and coming back to add
and improve to the written understanding. Every single email telling someone paid for work I’m
doing makes me feel valued. Thank you. While this book is what you directly pay for, indirectly
your support shows up in all the work I do for growing testing communities.
I talk exploratory testing with many people, and compare notes. I want to start mentioning some
of them by name. Alex Schladebeck, Ru Cindrea, Anne-Marie Charrett, Elizabeth Zagroba,
Kristine Korbus - you’ve shared my journey in relevant ways and inspired and taught me.
Why This Book
Exploratory testing - to me - is the way we think around software systems to find information
relevant to various stakeholders. The thinking starts with the idea that if something was easy and
straightforward to know, it could already be known. So we need to dig deeper to build systems that
provide the value.
I’ve learned something about testing in two decades that I still can’t find in the books I’m reading.
I share some of it in talks, but I have only so much time traveling around. I share some of it in
blog posts, but reading things as they are in flux is a different to a book. With this book, I want to
structure my experiences, stories and tricks into something I hope other testers will find useful.
If you have not read Elisabeth Hendrickson’s wonderful book Explore IT yet, please do so. We write
from different experiences, believing in a lot of similar things.
I think of this book as my lessons learned book on testing. I call it exploratory testing because the
way I approach testing with an exploratory mindset is different. It’s different to the artifact-oriented
mindset of the past (test cases) and artifact-oriented mindset of today (test automation). While all
testing may be exploratory, not all testing is done exploration first. Not all testing focuses on learning
as much as good exploratory testing does.
Back in the days, there was a guitar. Then someone invented an electric guitar. The electric guitar was
clearly different, but it was still a guitar. So we called the original guitar acoustic guitar. Exploratory
testing is to testing what acoustic guitar is guitars.
It’s the specialty thinking more commonly found in specialist testers, and a set of skills anyone can
choose to learn, with practice. Don’t expect it to be quick and easy though. Deep learning in layers
takes time, experimenting and reflection.
The LeanPub way
I used to have a book deal with a publisher, and the main thing that came out of that plan of a book
was a feeling of overwhelming barrier to publishing. I remember thinking nothing was ever good
enough. I knew I would learn more, I knew I needed to learn more and I expected there would be a
day when I had learned enough and I then just could get the book done.
The more I’ve learned, the more I’ve also realizes that my learning will never stop. I can proudly
show my old texts and my new texts indicating experiences that have caused a 180 turn in my
perspectives. I can only speak with the experience I have. I expect you, the reader, to read it with
the experience you have, and take everything with a grain of salt. I’m sharing lessons from my
experience.
I found LeanPub with another book to write and writing on this platform is different.
I publish pieces that could be useful as I write them, giving myself permission to iterate and
increment the book.
You will see pieces in the book that I’ve published as articles and blog posts. I’m working though
harmonizing and combining, and making the book easier to read, incrementally.
My readers can choose to read as I write or at any point later. I can make the book available for free,
and those readers who pay for the book, even a few dollars, are priceless source of motivation for
me to keep adding stuff.
This way, I discover the book. With you.
I would love to hear your feedback, comments and questions. You can always reach me by email:
[email protected] and if you want to give boost to my writing, please tweet about this book. I use the
hashtag #ExploratoryTestingBook.
Update 08/2020:
Writing bits and pieces around has been more practical than writing a coherent book. Over the years,
there has been a lot of bits and pieces.
To make progress with this book, in summer 2020 I collected together about half of the things I had
written on Exploratory Testing as Exploratory Testing Index¹. The index is giving me a baseline of
things I clean up into good chapters of this book.
To test out some of the exercises I don’t yet know how to write on, I started Exploratory Testing
Academy, where I can teach experimental pieces. The courses there are free within the limits of
my availability, and I am looking into a LeanPub style payment model. Obviously I need to be paid
by some people, but I’d like to trust the community with setting the prices based on value they are
getting from my work within their financial capabilities. Sharing is more important than a paywall.
¹https://maaretp.com/ETI.html
The LeanPub way 4

In Autumn 2020, I intend to work through various chapters, and you will see major changes.
First, I’m bringing in materials from other sources and categorizing them.
Then, I combine and fill gaps.
If some aspect that I am writing on is more important to you, reaching out with a question is the
best way to target my energies around that particular topic.
I have been doing exploratory testing for 25 years. For me, it engulfs test automation. Test automation
is one of my ways of documenting, but it is also a way of doing things I shouldn’t do without code,
like staying up testing 24/7 or adding 1000 users. I recognize I’ve become better at it, even good,
through noticing that the gap analysis exploratory testing targeting information people who came
before me missed finds problems when I perform it, and finds somewhat less after I have been
around.
If you have ideas of how we can work together for me to teach this better, my DMs are open on
twitter where I am @maaretp.
About the Author
Every piece of writing will focus on the perspectives its author finds relevant. This chapter exists to
help you understand some of the experiences I consider core to myself.

My work
I work as a principal test engineer at Vaisala. Vaisala is an awesome product company working with
measurement technology. This book is not about my work at any of the companies I have work
with. As a matter of fact, I started writing this book with a rule that has served me well: ask for
forgiveness, not a permission. There are things that I would not have done that are valuable both
to me personally but my employers at those times that could have been blocked with waiting for
permission.
As a principal test engineer, my work is about testing. Or anything else I find I can do, want to learn
and that has value for us creating awesome products. I focus in particular in exploratory testing that
for me engulfs deep understanding of how programmers write their code (to understand risks they
are introducing both short-term and long-term) and many kinds of test automation (including one
focusing on regression testing). It’s not just about what I do, it’s about what we do, together.
I’m an expert explorer. I’m also a catalyst - things happen when I’m around. I believe that is the
power of information testing provides.

My Side Business
My employers have been supportive of my aspirations of keeping options open for years, and I have a
contract that allows me to work for other companies with a limited availability. On the side, I teach
commercial courses on testing: Exploratory Testing Academy, Exploratory Testing Work Course,
Test Case Design Course and Ensemble Testing Course are ones I’ve focused on recently. I have
favored tailoring my courses towards hands-on with the company’s own software.

My history
I started testing with localization testing for Microsoft. Moving from clicking through ready test
cases to a product company requiring me to figure out the tests I would run myself, I found my
calling in testing. I realized I did not only want to test but also work to make testing more respected.
About the Author 6

I’ve been a tester, a test manager, filled in for product owners, project managers and product owners,
and worked as a researcher in testing. I came back from the management track as I realized with
agile that self-organized teams enable me to be a superb tester. I decided to value great results and
my hands-on participation over perceived career progression.
I’ve had a job where I was “too valuable to test”. I believe that is a big misconception.

Taking Learning Seriously

I believe there is magic in learning while doing. Exploratory testing has been a way for me to think
that learning is not only allowed, it is encouraged and required. If learning stops, exploring stops.
We need to learn to learn. Then we learn about our products as well as all sorts of things in software
industry, and the domain in which we’re solving problems with software.
Doing things alone requires focus on introspection. Also, being alone (with your immediate
colleagues) limits your abilities to learn.
I believe in doing with others. I’ve fallen for Ensemble Testing (and Ensemble Programming) which
is a way of whole group working together on one computer, getting the best out of everyone into the
work we’re doing. Learning and contributing are both valuable. Seeing people test differently than
you would stretches your understanding. When a group isn’t available, I would go for Strong-Style
Pairing. Strong-Style Pairing is a way of pairing in which an idea from one’s head must get to the
computer through someone else’s hands. Traditional pairing confuses me, which must mean there’s
something to learn about it that I have yet to learn.
People and discussions are my source of energy. For years, I have organized meetups to find people
to learn with. I organize to learn. I teach to learn.

Conference Speaking
I speak a lot in conferences on topics of testing to connect with people I could learn with. Conference
speaking helps me find individuals to work things out with. It usually encourages the others to take
the first step to talk to me about stuff I already spoke on, and helps me bridge my insecurities in
speaking to random people about things they may not be interested on.
A lot means on average 30 sessions a year for the last few years. I’m working on cutting down.

Idealist
I believe in the scout rule: Things should be left in better shape than they were when I came in.
This drives me to figure out ways to leave organizations in better shape when I go find new
challenges (which I have done about every 3 years) and to make sure whatever I’ve learned I will
share.
About the Author 7

It also drives me to find ways of improving our software creation community - makers and menders,
smart creatives, all of us. There’s few projects in particular that come from this:

• European Testing Conference has been my platform for awesome testing contents on testing as
both testers and programmers know it, but also on fairer treatment of speakers in the conference
circuit.
• TechVoices is a thing to allow room for new voices in testing, an effort to help new speakers
through mentoring, and I’m a serial volunteer with Them

I’m a spokesperson for diversity, believing the lack of diversity we see in conferences does not
represent the talent around but the organizer’s contacts. In addition to diversifying speakers to
include women and people of color, a special focus of diversity I take is bringing out the voices
in Europe.
One of my forms of idealism is kindness in disagreement. I’m for dialog that seeks to understand
where the other is coming from, and have mild allergies to debate that seeks any ways of
communication to win over arguments. Safety is a prerequisite of learning and I take learning
seriously.

From Finland
I’m a Finnish woman living in Finland. Finns take pride in our education system and for me
personally it has enable me to be the first in my line of family to go to a university. Some say
that the most extroverted of us Finns are still introverts by any other standards.
My culture is built-in and it won’t leave me easily. Finland is amazing place to create software in
and as a small country, I feel we have a little lower walls between different beliefs in testing. We
have needed to agree to disagree while getting along.
And the weather sucks. I don’t want to talk about the weather. So I talk testing and anything software
development.
What and Where of Exploratory
Testing
This is a start of a section of writings that try many different approaches to explaining what is
exploratory testing and where in testing you find it. This is less of a discussion of definition, and
more of providing an opportunity to hook into something as multifaceted from one of the provided
angles. Each angle itself may teach you how to do some of it better, help you frame the work you
do.
What is Exploratory Testing?
It has been 34 years since Cem Kaner coined the term to describe a style of skilled multidisciplinary
testing common in Silicon Valley. I’ve walked the path of exploratory testing for 25 years and it has
been a foundational practice in becoming the testing professional I am today. Let’s look at what it
is, to understand why it still matters—more than ever.

An Approach to Testing
Exploratory Testing is an approach to testing that centers the person doing testing by emphasizing
intertwined test design and execution with continuous learning where next test is influenced by
lessons on previous tests. As an approach, it gives a frame on how we do testing in a skilled way.
We use and grow multidisciplinary knowledge for fuller picture of empirical information testing
can provide. With the product as our external imagination, we are grounded on what is there but
inspired to seek beyond it.
We learn with every test about the application under test, ourselves as the tool doing the testing,
other tools helpful in extending our capabilities and the helpful ways we can view the world the
application lives in. We keep track of testing that has been done, needs doing and how this all
integrates with the rest of the people working on similar or interconnected themes.
What makes our activity exploratory testing over other exploratory approaches founded in curiosity
is the intent to evaluate. We evaluate, seek for information we are missing, making sure what we
know is real with empirical evidence.

Specifying It By Contrast
It’s not a surprise that some folks would like to call exploratory testing just testing. In many ways it
is the only way of testing that makes sense—incorporating active learning is central to our success
with software these days.
To contrast exploratory testing with what people often refer to as manual testing, exploratory testing
as a skilled approach encompasses use of programming for testing purposes. We use our brains,
our hands as well as programs to dig in deep while testing. Sometimes the way of including test
automation happens through means of collaboration, where ideas from exploratory testing drive
implementation of automation that makes sense.
To contrast exploratory testing with people refer to as scripted testing, exploratory testing isn’t
driven by scripts. If we create scripts from exploratory testing, we know to use them in exploratory
What is Exploratory Testing? 10

fashion remembering that active thinking should always be present even when the script supports
us in remembering a basic flow. We’ve talked a lot about scripted testing as an approach where we
separate design—deciding what to test, and execution—making the test happen, and thus lowering
our chances of active learning targeting the most recent understanding of the risk profile.
Another contrast to programming centric views to testing comes with embracing the multidisci-
plinary view of of testing where asking questions like “is my application breaking the law today
after recent changes?” is something routinely encoded into exploratory testing, but often out of
scope for a group of automators.

The Three Scopes of Exploratory Testing

As an approach, we can time and integrate this into our development efforts in many ways.
I find there are three ways of scoping: * exploratory testing as a way of extending existing test cases
* exploratory testing as a limited timebox * exploratory testing as frame of all thinking
When exploratory testing is scoped as a way of extending existing test cases, the way of working
does not look very exploratory. In places like this, you find yourself wondering why Tina is so
successful with the same tests Toni is missing problems with. The secret is that one of them actively
extends how they understand the test cases, refuses to follow exact instructions or only instructions.
They learn and find new perspectives.
When exploratory testing is scoped as a limited time box, you find people feeling they can only
be free on Friday afternoons. These are moments in project life that are structured separately,
with different expectations of how things flow and where focus is. This is a great way of scoping
exploratory testing into a process where the natural inclination is to believe we know where the
tasks begin and end, and need explicit encouragement to see if what holds true.
When exploratory testing is the frame of all thinking, it encompasses all considerations about testing.
We start with exploratory testing to identify something that needs documenting, and take time
to document it—perhaps with automation that is the modern way of documenting as executable
specifications.

Listen to Language
If you hear: “My boss asked me to test search so I searched something that was found and something
that wasn’t and reported I was done” you may be witnessing very low quality exploratory testing. It
relies on following high-level orders to an extent the tester can imagine based on their knowledge.
If you hear: “My colleague wrote me 50 cases of what to try out with search and I tried them
and reported I was done” you may be witnessing testing that isn’t exploratory. There is no hint
of learning, and owning responsibility of quality of the testing that happens.
What is Exploratory Testing? 11

If you hear: “My boss asked me to test search so I came back with 50 quick ideas of what could
be relevant and we figured out I’d just do 10 of them before we decided if going further was
worthwhile“, you are likely to be witnessing exploratory testing.
Similarly, in the automation first space, if you hear: “I got a Jira ticket saying I should automate this.
I did, and found some problems while at it, and extended existing automation because of the stuff I
learned.”, you may be seeing someone who is exploratory testing.
If you hear: “The Jira ticket said to automate A, B, and C, and I automated A and B, C could not be
automated.”, you may be witnessing testing that isn’t exploratory.
Look at who is in the center: is the tester doing the work and learning actively applying a better way
of doing the overall testing? If yes, that is exploratory testing.
As skilled approach, it is only as good as the skill of the person applying it. With focus on learning
through, skill may be a problem of today, but improved upon every day as exploratory testing is being
done. If you find yourself not learning, you most likely are not exploring. With the product as your
external imagination, you should find yourself imagining new routes through the functionalities,
new users with new perspectives, and relevant information your project teams would be happy to
make justified decisions on their take for the risk. With and without automation, in a good balance.
What is Exploratory Testing - the
Programmer Edition
In the new software world regime where programmers find themselves taking a lot more respon-
sibility of testing, we need to understand what exploratory testing is as it extends what most
programmer’s find their tests covering, and causes us talk past each other in understanding what
testing.

What Testing Gives You?

As a programmer, you know what you’re implementing. When you don’t know, you’ll learn. You
explore the problem to figure out a solution.
Back in the days some bad organizations told you that they’ve hired a group of testers. They might
even say that since they pay other people for testing, you shouldn’t be bothering yourself with that.
But you know you want to test. Because testing has direct value to you as a programmer. It gives
you four things:

• Specification
• Feedback
• Regression
• Granularity

Specification means that tests you write can be concrete examples of what the program you’re about
to write is supposed to do. No more fancy words around high level concepts—give me an example
and what is supposed to happen with it. And when moving on, you have the specification of what
was agreed. This is what we made the software do, change is fine but this was the specification you
were aware of at time of implementation.
Feedback means that as the tests are around and we run the tests—they are automated, of course—
the tests give us feedback of what is working and what not. The feedback, especially when we
work with modern agile technical practices like test-driven development, gives us a concrete goal of
what we need to make work and if it is working. They help us anchor our intent so that given the
interruptions, we still can stay on the right path. And we can figure out if the path was wrong. This
is the test that passes, yet you say it isn’t right. What are we not seeing?
Regression means that tests don’t only help us when we’re building something for the first time,
but they also help us in changes. And software that does not change is dead, because users using
What is Exploratory Testing - the Programmer Edition 13

and loving it will come back with loads of new ideas and requirements. We want to make sure that
when we change something, we make only changes we intended. And regression is the perspective
of doing more than we intended, without tests without us knowing.
Granularity comes to play when our test fail for a reason. Granularity is about knowing exactly
what is wrong and not having to spend a lot of time figuring it out. We know that small tests pinpoint
problems better. We know that automation pinpoints problems better than people. And we know
that we can ask people to be very precise on their feedback when they complain something doesn’t
work. Not having to waste time on figuring out what is wrong is valuable.
This is not exploratory testing. Exploratory testing often guides this type of testing, but this is testing
as artifact creation. Exploratory testing focuses on testing as performance—like improvisational
theatre—in a multidisciplinary way beyond computer science.

Exploratory Testing, eh?

I get all of this good stuff from testing as I know it, unit tests and test automation. What is this
exploratory testing stuff then?
Exploratory testing is basically saying that spending time thinking with your application, your
APIs, your environments as an external imagination, you will find things you did not realize you
were missing. And time thinking with something you’ve implemented is real, empirical. Not just
something you wish was true like most of our designs.
Exploratory testing is multidisciplinary, basically saying that it is not enough for you to take orders
from whoever gives you requirements, but you have to think critically on why they are asking what
they are asking and its practical implications. If someone asks you to do something that is illegal,
you need to point it out. If you don’t know if it is illegal, you need to spend time figuring out what
is and isn’t illegal, and how that would show up in the application you are creating. GDPR is a great
example of this: not caring for privacy can cost your organization a lot. And it is not just law you
need to care for. You need to care for business, users, environments your software runs in, ethics,
psychology—and many more. You want to do no harm, and be conscious about what you are doing.
Intentional, not accidental.
With the way you look at testing as programmer, what more is there that exploratory testing
promises to give you? It’s an approach that focuses on learning while evaluating what you know
and don’t know, giving yourself chances of learning the things you don’t know you don’t know but
can see while using the application like your users would.
It gives you four things:

• Guidance
• Understanding
• Models
• Serendipity
What is Exploratory Testing - the Programmer Edition 14

Guidance is not just about the specification, but general directions of what is better and what is
not. Some of it you don’t know to place in yes/no boxes, but have to clarify with your stakeholders
to turn them into something that can be a specification.
Understanding means that we know more about the place of our application in the overall world
of things, why people would find it valuable, how other’s design decisions can cause us trouble and
how what should be true if things were right always are not. It helps us put the details we’re asked
to code into a bigger picture—one that is sociotechnical and extending beyond our own organization
and powers.
Models are ways of encoding knowledge, so that we can make informed decisions, understand
things deeper and learn faster next time or with next people joining our teams.
Serendipity is the lucky accidents, running into information you did not expect to find. The lucky
accidents of new information about how things could and do go wrong when using your application
emerge given enough time and variety to use of your application. And knowing helps you not get
the escalations waking you up to critical maintenance tasks because who else would fix all of this
than the programmers?
An Approach To Testing
Exploratory testing is a approach to testing. It says whoever tests needs to be learning. Learning
needs to change what you are doing. You can’t separate designing of tests and executing them
without losing learning that influences your next tests. It is an approach that frames how we do
testing in a skilled way.
We don’t do just what is asked, but we carefully consider perspectives we may miss.
We don’t look at just what we are building, but the dependencies too, intentional and accidental.
Unsurprisingly, great programmer teams are doing exploratory testing too. Creating great automa-
tion relies on exploratory testing to figure out all the things you want to go check. While with
exploratory testing we believe that premature writing of instructions hinders intellectual processes,
we also know that writing that stuff as code that can be changed as our understanding grows, this
frees our mental capacity to think of other things. The executable test artifacts give us that peace of
mind.
Programmer teams are also doing what I would consider low quality exploratory testing with
limited ideas of what might make a difference in new information being revealed. And that is where
testers often come in—mindspace free of some of the programmer burdens, they focus their energies
elsewhere, raising the overall quality of work coming out of teams.
Finally, I want to leave you with this idea—bad testing is still testing. It just does not give much
of any of the benefits you could get with any testing. Exploratory testing and learning actively
transforms bad testing to better.
Where is Exploratory Testing
When people start learning about testing, and agile testing in particular, they quickly get to a model
of testing quadrants. Testing quadrants, popularized by Janet Gregory and Lisa Crispin with their
Agile Testing book, place a collection of testing words in four quadrants. When you go look, you can
find “exploratory testing” in the top right corner, meaning it is considered Business Facing Product
Critique.
As the term was coined 35 years ago to express a specific approach to testing, making it a technique
in one corner of the quadrants was not the intent. It expressed a style of testing unfamiliar to the
majority, that was observable in Silicon Valley product companies, a skilled multidisciplinary testing
under product development constraints.
The world moved on, and testing certifications had hard time placing a whole approach of testing
into their views of the world. With introduction of ways to manage this style of testing (session-
and thread-based exploratory testing), those seeking to place it in the technique box combined the
management style and defined their idea of using only limited time - separately defined sessions -
on this way of testing and everything it was born to be different from remained in the center.
That means that in the modern world, exploratory testing is two things:

• a technique to fill gaps that all other testing leaves

• an approach that encapsulates all other testing

As a technique, you can put it in the corner of quadrants. As a technique, you can put it on top of
your test automation pyramid and make jokes about the pyramid turning into an ice cream cone
with too much of exploratory testing on top. But as an approach, it exist for every quadrant, and for
every layer.
Due to the great confusion, questions about the other testing I do on top of exploratory testing are
quite common.

Surely Exploratory Testing is not the only testing you are doing. I see it as a “plus” on top
of feature testing, regression, all the non-functional testing. Exploratory fills the gaps.

This response today inspired me to think about this a little more.

Exploratory fills the gaps.

Where is Exploratory Testing 16

But for me, it does not fill the gaps. It is the frame in which all other testing exists. It is what
encourages short loops to learn, challenges the limits of what I already have learned, makes me pay
attention to what is real, and creates a sharp focus on opportunity cost.
I scribbled an image on paper, that I recreated for purposes of this blog post. If all these shapes
here are the other kinds of testing mentioned: feature testing, regression testing and non-functional
testing, what is the shape of exploratory testing?

Exploratory Testing Encloses All Other Testing

The shape of exploratory testing is that it fills the gaps. But it also defines the shape of all the other
tests. It’s borders are by design fuzzy. We are both right: for me it is the frame from which all the
other testing exists, even when it fills gaps.
There is such thing as non-exploratory testing. It’s the one where shape of other tests stay in place
and are not actively challenged, and where particular artifacts are important over considering their
value and opportunity cost.
Where I worked, we had two teams doing great at testing. Both teams explored and left behind
test automation as documentation. When asked what percentage they automated, their responses
were very different. One automated 100%, and it was possible by not having as many ideas of what
Where is Exploratory Testing 17

testing could be. The other automated 10%. Yet they had just as much automation as the first, but
often found problems outside the automation box. Easiest way to get get to 100% is by limiting your
ideas of what testing could be.
Seeing there’s plenty of space in between the shapes and plenty of work in defining the shapes can
be a make or break for great testing.
Place of exploration
Over the years, I’ve worked with places where release cycles grow shorter. From integrating all
changes into builds a couple of times a week, we’ve moved over to continuous integration. Each
change gets integrated to the latest system and made available for testing as soon as the build
automation finishes running. We don’t get to test the exactly same thing for a very long time, or if
we do, we spend time on something that will not be the final assembly delivered. Similarly, we’ve
moved from giving those assemblies to customers once every six months to continuous delivery and
the version in production can change multiple times a day.
In the fast-paced delivery world, we turn to look heavily at automation. As we need to be able to
run our tests again and again, and deliver the change as soon as it has been checked in and run
through our automated build and test pipeline, surely there is no place for exploratory testing? Or if
there is, maybe we just do all of our exploratory testing against a production version? Or maybe on
top of all this automation, exploratory testing is a separate activity, happening just before accepting
the change into the assembly that gets built and pushed forward? Like a time-box spent on testing
whatever risks we saw the change potentially introducing that our automation may not catch?
Think of exploratory testing as a mindset that frames all testing activities - including automation.
It’s the mindset that suggests that even when we automate, we need to think. That the automation
we are creating for continuous testing is a tool, and will be only as good as the thinking that created
it. Just like the application it tests.

An example
We were working in a small team, building the basis for a feature: managing Windows Firewall
remotely. There were four of us: Alice and Bob were the programmers assigned at the task. Cecile
specialized in end to end test automation. David could read code, but on most days chose not to and
through of themselves as the team’s exploratory testing specialist.
As the team was taking in the new feature, there was a whole group discussion. The group talked
about existing APIs to use for the task at hand, and figured out that the feature had a core. There
was the existing Windows Firewall. There was information about rules to add delivered. And those
rules needed to be applied, or there would not be the feature. After drawing some boxes on the wall,
having discussions about overall and minimal scope, the programmers started their work of writing
the application code.
It did not take long until Alice checked in the module frame, and Bob reviewed the pull request
accepting the changes making something available that was still just a frame. Alice and Bob paired
to build up the basic functionality, leading Cecile and David listening to them bouncing off ideas of
Place of exploration 19

what was the right thing to do. As they introduced functionality, they also included unit tests. And as
they figured out the next slice of functionality to add, David was noticing how much of exploratory
testing the two did between the pair. The unit tests would surprise them on a regular basis, and they
took each surprise as an invitation to explore in the context of code. Soon the functionality of adding
rules was forming, and the pull requests were accepted within the pair.
Meanwhile, Cecile was setting up possibilities to run the Windows Firewall in a multitude of
Windows operating systems. They created a script that introduced five different flavors of Windows
that were supposed to be supported for the new functionality to be run as jobs within the continuous
integration pipeline. They created libraries that allowed to drive the Windows Firewall in those
operating systems, so that one could programmatically see what rules were introduced and shown.
Since the team had agreed on the mechanism of how the rules would be delivered from the outside,
they also created mechanisms of locally creating rules through the same mechanism.
As soon as the module could be run, Alice and Bob would help out Cecile on getting the test
automation scripts running on the module. David also participated as they created the simplest
possible thing that should work: adding a rule called “1” that blocked ping and could be verified in
system context by running ping before and after. Setting up the scripts on top of Cecile’s foundation
was straightforward.
Cecile wanted to test their scripts before leaving them running triggered for an hourly repeat for a
baseline, and manually started the run on one of their operating systems, visually verifying what
was going on. They soon realized that there was a problem they had not anticipated, leaving the
list of rules in an unstable state. Visually, things were flickering when the list of rules was looked
at through the UI. That was not what happened when rules were added manually. And Cecile had
explored enough of the system to know what adding a rule through the existing user interfaces
should look like. Something was off.
Cecile, Bob and Alice figured out that the problem was related to naming the rules. If the rule name
was less than three characters, there was a problem. So Bob introduced a fix limiting rule’s minimum
length, Alice approved the change and Cecile changed the rules to have a name longer than three
characters. Cecile used Windows Firewall more to figure out different types of rules, and added
more cases by exploring what was same and different and made sure they would test things both
locally and remotely - end to end.
David had also started exploratory testing the application as soon as there were features available.
They had learned that Alice and Bob did not introduce logging right from the start, and as they
did, that the log wasn’t consistent with other existing logs in how it was named. They were aware
of things being built into the automation, and focused their search on things that would expand
the knowledge. They identified that there were other ways of introducing rules, or locking the
Windows Firewall so that rules could not be introduced through external mechanisms. They would
pay attention to rule wizard functionalities in the Windows Firewall, enforcing rules around legal
rules, and make notes of those only to realize through testing that Alice and Bob had not considered
that all combinations were not legal. Things David would find would not be bugs as the team defined
a bug - programming errors - but more of missing functionalities, for lacking information about the
execution environment.
Place of exploration 20

David would make lists of tests for Cecile to add to the test automation, and pair with Cecile to
add them. As they were pairing, the possibilities of automatically creating a lot of rules triggered
their minds and they would try introducing a thousand rules to note performance concerns. And as
adding was so much fun, obviously removing some of them would make sense too. They would also
add changing rules. And as they were playing with numbers, they realized that they had uncovered
a security issue: rules were there to protect, and timings would allow for times of unprotection.
The team built it all to a level they felt was minimal to be delivered outside the organization. Unit
tests and test automation allowed for making changes and believing those cases still worked out ok.
They could explore around every change, and every idea.
The functionality also included a bit of monitoring, allowing them to see the use of feature in
production. After having the feature running in production, monitoring provided extra ideas of
what to explore to understand the overall system better.
What this story shows: * everyone explores, not everyone calls it exploratory testing even if it is
that * we explore before be build, during building, while we automate and separately from building
and automating, as well as while in production * exploration can happen in context of code and in
context of the system we’re building as well as in context of use in the overall system * without
exploratory testing, we don’t give ourselves the best chances of knowing what we’re building

Separate Tester Role

In software development, we translate ideas to code. When we focus on the idea separately, the
translation process separately and the result of the translation process separately. To see things with
many dimensions in their full potential, quite much headspace gets occupied. In fact, our ability to
focus and think is heavily limited.
It has been a common experience that having people who use their headspace focusing on building
the solution separately from people focusing on finding it’s weak points and understanding it in
context gives us better results in a shorter timeframe. For this to work to its full potential, we need
collaboration, as closely knit as possible. Diverse views and focuses bring in the fuller picture when
everyone’s voices are valued and heard.
With limited headspace for learning, it makes sense we diversify our learning focus. From a personal
experience, I can speak on the extensive practice needed to become a tester able to do deep (as
opposed to shallow) exploratory testing. Also, I can speak on the experience of needing less of
specialized testers in teams in general, as those who do and practice deep exploratory testing can
proceed much faster in the modern ways of working where everyone tests and basic quality of
the products is not the time consuming obstacle. Where there used to be a need of a tester per
programmer, with the new division of labor portions such as 1:10 are common.
Place of exploration 21

Future is Here, Just Not Evenly Distributed

There are teams where collaboration is taken to a positive extreme and the whole team works
together, even through a single computer used to do the work in a group. We call this way of
collaboration mob programming. With Mob Programming, the deep exploratory testing as well as
a tester perspective’s typical insights are contributed as soon as they emerge, while working on the
code and solution together.
Just as there are these new closely knit teams, there are teams where people in different roles are
highly isolated. There are places in which developers barely test and the organization guides them
to leave testing work for a separate role of testers. There are places where developers try to test, but
results show that they miss a lot of relevant issues. While the team learns to build software well, the
feedback of quality typically provided by testers can be a make-or-break practice for both customer
happiness but also team effectiveness.
With places that use long delivery cycles, the reliance of existence of testers is often even higher.
With systems having specialized domain knowledge the development teams don’t have, it is essential
that the domain specialists participate in the testing efforts to make sure the system built is serving
the actual needs fulfilling expectations of why it was built in the first place.
No matter which version of software development we live in, exploratory testing plays a key role.
It just shows itself in a different packaging in its projectized vs. continuous flow formats.
The Two Guiding Principles
Whenever I need to define exploratory testing, I bow to people who have come before me.
Cem Kaner introduced me to the idea of exploratory testing with the first testing book I ever read:
Testing Computer Software. He defines exploratory testing as:
Exploratory software testing is a style of software testing that emphasizes the personal
freedom and responsibility of the individual tester to continually optimize the value of her
work by treating test-related learning, test design, test execution, and test result interpretation
as mutually supportive activities that run in parallel throughout the project.
Elisabeth Hendrickson et al. created an invaluable resource, a Cheat Sheet, to summarize some ideas
common to starting with exploratory testing. She defines exploratory testing as:
Exploratory testing is a systematic approach for discovering risks using rigorous analysis
techniques coupled with testing heuristics.
A lot of writing on the topic and techniques are part of Rapid Software Testing Methodology that
James Bach and Michael Bolton have created. They define all testing as exploratory and have
recently deprecated the term.
Exploratory testing, to me, emphasizes the difference to other testing that Julian Harty very clearly
points out: “Most of the testing I see is worthless. It should be automated, and the automation
deleted.” Exploratory testing isn’t that testing. A lot of that testing is around through.
I find myself talking about two guiding principles around exploratory testing again and again. These
two guiding principles are learning and opportunity cost.

Learning
If we run a test but don’t stop to learn and let the results of the test we just run influence our choices
on the next test, we are not exploring. Learning is a core to exploring. Exploring enables discovery
of information that is surprising, and the surprise should lead into learning.
The learning attitude shows in the testing we do so that there is testing against the risk of regression,
but a lot of times the risk isn’t best addressed by running exact same tests again and again.
Understanding the change, seeking out various perspectives in which it might have impact and
introduce problems that were not there before is the primary way an exploratory tester thinks.
Whatever I test, I approach it with the idea of actively avoiding repeating the same test. There’s so
much I can vary, and learning about what I could vary is part of the charm of exploratory testing.
When we optimize for learning and providing as much relevant information as we can with whatever
we have learned by that time, we can be useful in different ways at different phases of our learning
with the system.
The Two Guiding Principles 23

Opportunity Cost
Whatever we choose to do is a choice away from something else. Opportunity cost is the idea of
becoming aware of your choices that have always more dimensions than the obvious.
There are some choices that remove others completely. Here’s a thought experiment to clarify what
I mean: Imagine you’re married and having hard time with your spouse. You’re not exactly happy.
You come up with ideas of what could be changed. Two main ideas are on the table. One is to go
to counceling and the other is to try an open relationship. If you choose the latter and your spouse
feels strongly against this, the latter option may no longer be available. The system has changed.
There are some choices that you can do in different order and they still are both relevant options.
If you choose to test first with a specification, you will never again be the person who has never
read the specification. If you choose to test first without the specification, you will never have the
experience of what you would notice if your first experience was with a specification.
There are some choices that leave others outside scope. If you choose to use all your time in creating
automation and avoiding following the exploration ideas you generate while automating as basic
cases already require effort, you leave the information exploring could provide out of scope. If you
choose to explore and not automate, you leave repetitive work of future to be done manually or
undone.
The idea of being aware of opportunity cost emphasizes a variety of choices where there is no one
obviously correct choice in the stream of small decisions. We seek to provide information, and we
can do so with various orders of tasks.
It’s good to remember that rarely we have an endless schedule and budget. So being aware of
opportunity cost keeps us focused on doing the best testing possible with the time we have available.
How to Explore with Intent -
Exploratory Testing Self-Management
This article was published in Ministry of Testing Testing Planet in 2016. Appropriate pieces of it will
find their place as part of this book.
Exploratory testing is the wonderful idea that we can use our freedom of choice while testing, to
learn as we go on and let that learning influence the choices we make next. The system we test
is our external imagination, and it’s our responsibility to give it the chance to whisper out all the
information there is.
When we test, everyone is allowed to stretch their assigned boxes with exploration at least a little.
Even the most test case oriented organizations will ask you to think, learn, and look around while
executing your assigned tests. That’s what makes you good at testing in the organization.
For more of a stretch, these organizations will allow for a few hours of freedom from the assigned
box, to do some time-boxed exploratory testing for finding gaps your usual test case harness keeps
you from spotting.
Others, like myself, work in the exploratory testing mode full time. In this mode, test cases (if such
will exist) are an output of the process instead of an input and created at a time we know the most
about a feature or product. We’ve learned a lot by the time we’re done testing.
Regardless of whether your mode of exploratory testing is using it as technique (extending your
test cases), as a task (time-boxing exploration) or as an approach (engulfing all your thinking of
testing), there’s a critical skill of self-management you’ll need to develop. You’ll want to explore
with intent, keep track of what you know and learn, and what more there is to learn. All of this will
grow iteratively and incrementally as you do this type of testing.

Intertwining Different Testing Activities

With years of practice on skilled exploration, I find it now possible to do different activities
simultaneously. I can strategize on a testing big picture and create tasks out of the ideas. I can
execute testing on some of those ideas configuring the environments and learn from the different
types of thinking. It’s not really simultaneous, it’s intertwined into these tiny bits of tasks, allowing
my mind to wonder and categorize things into a frame of reference.
It was not always possible. Actually, it was really hard. In particular, it is really hard to intertwine
long-term (looking into future work) and short-term (looking at what is going on now) thinking,
which are very different in nature. It’s ok, because the ability to intertwine is not a requirement to get
How to Explore with Intent - Exploratory Testing Self-Management 25

started. You would do well acknowledging where your abilities are and developing them further by
practicing intertwining, but also allowing yourself time to focus on just one thing. With exploratory
testing, the formula includes you: what works for you, as you are today.

A Practical Example
Imagine learning to drive a car. You’re taking your first lessons at the driving school and after some
bits of theory you know the basic mechanics of driving but have never done any of it.
You’ve been shown the three pedals, and when you stop to think, you know which one is which.
You know the gear shifter and it’s clear without telling what the steering wheel does (as long as you
drive forward, that is). And finally comes the moment you’re actually going to drive.
The driving instructor makes you drive a couple of laps around the parking lot and then tells you
to drive out, amongst other cars. With newness of all of this, your mind blanks and you remember
nothing of the following half an hour. And if you remember something, it’s the time when your car
stopped at an embarrassing location because it was too hard to do the right combination of clutch
and gears.
All the pieces are new and doing the right combination of even two of them at the same time is
an effort. Think about it, when you looked if you could turn right, didn’t you already start turning
the wheel? And when you were stopped at the lights to turn, didn’t it take significant effort to get
moving and turn at the same time?
After years of driving, you’re able to do the details without thinking much, and you’re free to use
your energy on optimizing your route of the day or the discussion you’re having with the person
next to you. Or choosing a new scenic route without messing up your driving flow.
It’s the same with testing. There’s a number of things to pay attention to. The details of the
application you’re operating. The details of the tools you need to use. The uncertainties of
information. All your thoughts and knowledge. The information you get from others, and whether
you trust it or not. The ideas of what to test and how to test it. The ideas of what would help you test
again later. The expectations driving you to care about particular type of information. Combining
any two of these at a time seems like a stretch and yet with exploratory testing, you’re expected to
keep track of all of these in some way. And most essentially from all the details, you’re expected to
build out and communicate both a long-term and a short-term view of the testing you’ve done and
are about to do.

Learning To Self-manage
I find that a critical skill for an exploratory tester is the skill to self-manage, and to create a structure
that helps you keep track of what you’re doing. Nowadays, with some years of experience behind
me, I just create mind maps. There is a simple tool I found to be brilliant for learning the right kind
of thinking, and that tool is what I want to share with you.
How to Explore with Intent - Exploratory Testing Self-Management 26

When I say tool, I mean more of a thinking tool. The thinking tool here though has a physical
structure.
For a relevant timeframe, I was going around testing with a notebook for a very particular purpose.
Each page in the notebook represented a day of testing, and provided me a mechanism to keep track
of my days. A page was split into four sections, with invisible titles I’ve illustrated in the picture:
Mission (why am I here?), Charter (what I’m doing today?), Details (what am I keeping track of in
details?) and Other Charters (what should I be doing before I’m done?).
At the start of a day of testing, I would open a fresh page and review my status after letting earlier
learning sink in. Each of the pages would stay there to remind me of how my learning journey
developed as the application was built up, one day at a time.

Notebook illustration

Mission
In the top left corner, I would stick a note about my mission, my purpose or as I often liked to think
of it, the sandbox I was hired to play in. What did the organization expect of me as per information
I would provide, having hired me as an exploratory tester? How I could describe that in just a few
sentences?
How to Explore with Intent - Exploratory Testing Self-Management 27

For example, I was hired in an organization with ten teams, each working on a particular area of the
product. My team was specializing in installations. That little note reminded me that while I could
test anything outside the installations if I so wished, there was a sandbox that I was supposed to
cover for relevant findings and it was unlikely that others would feel the urge to dig deep into my
area.
They were likely to travel through it, but all the special things in the area, they would probably
rather avoid. If I would be digging through someone else’s area, nothing would stop me. But I might
leave mine unattended. I might feel that I used all this time, and therefore I’m done, even if I was
only shallowly covering my own area.
The mission note reminded me of the types of information the organization considered relevant,
and the area of responsibility I felt I had accepted. It served as an anchor when the whispers of the
product lead me elsewhere to explore.

Charter
In the top right corner was my note about the work of the day: the Charter. Each morning I would
imagine what I was trying to achieve today - only to learn most evenings I had done something
completely different. Charter is a framing of what I’m testing, and as I learn they change over time.
It’s acceptable to start out with one idea and end up with something completely different when you
are finished.
The note of the day was another anchor keeping me honest. With exploration, I’m not required to
stick to my own plans. But I’m required to be in control of my plans in the sense that I don’t fool
myself into believing something is done just because the time is used.
Continuing on my example with the Installations team, I might set up my charter of the day to
be 2 installations with a deep dive into what actually gets installed. Or I might set it up to be 20
installations, looking through each shallowly. Or I might decide to focus on a few particular features
and their combinations. If I saw something while testing that triggered another thought, I could
follow it. But at the end of the day, I could review my idea from the morning: did I do 20 shallow
installations like I thought I would? If I didn’t, what did I do? What am I learning for myself from
how things turned out?

Details
In the bottom right corner, I would pile up notes. At first, these were just lines of text I would write
that would often fill the page next to the one I was working on. Later, I realized, that for me there
were three things I wanted to make notes of: the bugs, the questions, the ideas for test automation
or test cases, and my notes extended to have a categorization shorthand.
With any of the detailed ideas, I could choose to stop doing the testing I was doing, and attend to
the detail right away. I could decide that instead of focusing on exploring to find new information,
I could create an automated test case from a scenario I cooked up from exploration. I could decide
How to Explore with Intent - Exploratory Testing Self-Management 28

that instead of completing what I was planning on doing today, I would write the great bug report
with proper investigation behind it. I could decide to find a product owner, a support representative,
a programmer, or my manager to get an answer for a burning question I had. Or, I could make note
of any of these with minimum effort, and stick to my idea of what I would do to test the application
before attending to the details.
I learned that people like me can generate so many questions, that if I don’t have a personal throttling
mechanism, I can block others from focusing on other things. So I realized that collecting the
questions and asking them in regular intervals was a good discipline for me. And while looking
through my questions, I would notice that I had answers to more questions myself than I first
thought.
With each detail, the choice is mine. Shall I act on this detail immediately, or could it wait? Am I
losing something relevant if I don’t get my answer right away? Is the bug I found something the
developer would rather know now, than at the end of my working day? Do I want to stop being in
exploratory mode to improve my documentation, or to pair with a developer to implement a piece
of test automation, or do I rather time-box that work for another day from the idea I had while
testing?

Other Charters
In the bottom left corner, I would make notes of exploratory testing work I realized needed doing
while I was testing. I would write down ideas small and large that I would park for future reference,
sometimes realizing later that some of those I had already covered and just forgotten. Sometimes I
would add them to my backlog of work to do, and sometimes tuning the existing backlog of work
to support choosing focus points of upcoming testing days.
Some of my ideas would require creating code for purposes of extending the reach of exploration.
Some ideas would require getting intimately familiar with the details of log files and database
structures. Each new idea would build on the learning that had happened before, making me reassess
my strategy of what information I would invest in to have available first.

You’re In Control
The tool isn’t there to control you, it’s there to give you a structure to make your work visible for
you. You get to decide what happens when you explore, and in what order. If you need to go through
a particular flow 15 times from various angles, you do that. If you find it hard to think about strategy
and importance of particular tasks when you’re deep in doing testing, you reserve time separately
for strategic thinking.
With the days passing, and notes taken, I could go back seeing what types of sessions I would
typically have. There would be days where I’d just survey a functionality, to figure out a plan of
charters without focus on details. There would be target rich functionalities, where the only detail I
could pay attention to was the bugs. Over time, I could pay attention to doing things intentionally
How to Explore with Intent - Exploratory Testing Self-Management 29

with particular focus, and intentionally intertwined. I could stop to think, how different days and
different combinations made me feel. I learned to combine things in ways that were useful for my
organization, but also maximized the fun I could have while testing in a versatile manner.
While most value was in learning to self-manage my testing work around learning, there was also
a side impact. When someone would show up to ask about what I had done and was doing, I could
just flip a page and give an account of what had been going on. Seeing the structure created trust in
those who were interested in my progress.
As an active learner, you will get better every day you spend on testing. Exploratory testing treats
test design, test execution and learning as parallel, as mutually supportive activities to find unknown
unknowns. Doing things in parallel can be difficult, and testing needs to adjust to the tester’s
personal skill level and style. Your skill to self-manage your work and your learning - making
learning and reflection a habit - is what differentiates skilled exploratory testing from randomly
putting testing activities together.
I believe that the thing that makes us, testers, to not be treated as a commodity, is learning. It’s the
same with programmers. Learners outperform the ones that don’t. Exploratory testing has learning
at it’s core.
Yesterday in a testing workshop I run 3rd time in this format online, something interesting happened.
I noticed a pattern, and I am indebted to the women who made it so evident I could not escape the
insight.
We tested two pieces of software, to have an experience on testing that enabled us to discuss what
testing is, why it is important and if it interests us. On my part, the interest is evident, perhaps even
infectious. The 30 women new to creating software to the skills in that space.
The first software we tested was the infamous Park Calculator. The insight that I picked up on
came quite late to a bubbling discussion on how many problems and what kind of problems we
were seeing, when someone framed it as a question: Why would the calculator first ask the type
of parking and then give the cost of it, when a user would usually start with the idea that they
know when they are traveling, and would benefit from a summary of all types of parking for that
timeframe? The answer seems to be that both approaches would fit some way of thinking around
the requirements, and what we have was the way who ever implemented this decided to frame
that requirement. We found a bug, that would lead to a redesign of what we had, even if we could
reuse many of the components. Such feedback would be more welcome early on, but if not early,
discussing this to be aware was still a good option.
The second software we tested was the GildedRose. A piece of code, intertwined with lovely clear
requirements, often used as a way of showing how a programmer can get code under tests without
even reading the requirements. Reading the requirements leads us to a different, interesting path
though. One of the requirements states that quality of an item can never be negative and another
one tells it is maximum 50. Given a list of requirements where these two are somewhere in the
middle, the likelihood of a tester picking these up to test first is very high - a fascinating pattern in
itself. However, what we learn from those is that there is no error message on an input or output
beyond the boundaries as we expect, instead given inputs outside boundaries the software stays on
How to Explore with Intent - Exploratory Testing Self-Management 30

the level of wrongness of input not making it worse, and given inputs barely at boundaries it blocks
the outputs from changing to wrong. This fits the requirement, but goes as a confusing design choice.
The two examples together bring out a common pattern we see in testing: sometimes what we have
is what we intended to have, and fits our requirements. However, we can easily imagine a better
way of interpreting that requirement, and would start a discussion on this as a bug we know will
stretch some folks ideas of what a bug is.
Going beyond Defaults
With a new job, comes a new mobile phone. The brand new version of iPhone X is an upgrade to
my previous iPhone 7, except for color - the pretty rose gold I come to love is no longer with me.
The change experience is fluent, a few clicks and credentials, and all I need is time for stuff to sync
for the new phone.
As I start using the phone, I can’t help but noticing the differences. Wanting to kill an application
that is stuck, I struggle when there is no button to double click to get to all running applications. I
call out for my 11-year-old daughter to rescue and she teaches me the right kind of swipes.
Two weeks later, it feels as if there was never a change.
My patterns of phone use did not change as the model changed. If there’s more power (features) to
the new version, I most likely am not taking advantage of them, as I work on my defaults. Quality-
wise, I am happy as long as my defaults work. Only the features I use can make an impact on my
perception of quality.
When we approach a system with the intent of testing it, our own defaults are not sufficient. We
need a superset of everyone’s defaults. We call out to requirements to get someone else’s model of
all the things we are supposed to discover, but that is only a starting point.
For each claim made in requirements, different users can approach it with different expectations,
situations, and use scenarios. Some make mistakes, some do bad things intentionally (especially for
purposes of compromising security). There’s many ways it can be used right (“positive testing”) and
many ways it can be used wrong (“negative testing” - hopefully leading to positive testing of error
flows).
Exploratory testing says we approach this reality knowing we, the testers, have defaults. We actively
break out of our defaults to see things in scale of all users. We use requirements as the skeleton map,
and outline more details through learning as we test. We recognize some of our learnings would
greatly benefit us in repeating things, and leave behind our insights documented as test automation.
We know we weren’t hired to do all testing, but to get all testing done and we actively seek everyone’s
contributions.
We go beyond the defaults knowing there is always more out there.
Heuristics
When we would like to have a rule of how the world works, but we quite can’t make one because
everything depends, we still have ideas of what might be helpful rule-like constructs. Exploratory
Testing is full of them, and we like to call them heuristics.
We pass heuristics forward as the folklore, wisdom of the past of what has worked.
We use heuristics elaborately sometimes knowingly and sometimes unconsciously - they help as
shortcuts in situations where there are already too many things to think of.
As far as I am concerned, I have not yet discovered a great way to categorize heuristics. But I seem
to use two dimensions.

Size of heuristics
Some heuristics are models that help us do what we need to do in testing. Models tend to grow
bigger in size, and take a form of listing things to consider.
Things we notice we are doing in our projects are smaller in size and it can be really helpful to
label them. With these types of heuristics, I follow Alex Schladebeck’s convention and call them
microheuristics. Instead of creating a theory of everything, we can appreciate the smallest pieces of
wisdom we pass on.

Purpose of heuristics
I have come across multiple applications for heuristics, and for now I focus on two major ones.

• Recall heuristics help with test ideation. We need to come up with ideas to make choices, how
do we learn to do that?
• Decision heuristics help with explaining directions we are taking for test strategy, for testing
in the moment and for conversations we have about our testing.

The following chapters

This chapter is very much work in progress. I’ve already written a piece on recall heuristics, and
one example of a microheuristic. I will be adding a lot more, and structuring this.
Recall Heuristics for Test Ideation
Good exploratory testing balances our choices what to do now so that whenever we are out of time,
we’ve done the best job testing we could in the time we were given, and are capable of having a
conversation about our ideas of risks we have not assessed. To balance choices, we need to know
there are choices and recently I have observed that the amount of choices some testers make is
limited. A lot of what we call test design nowadays is recalling information to make informed
selections. Just like they say:

If the only tool you know is a hammer, everything starts to look like a nail.

We could add an exploratory testing disillusionment corollary:

It’s not just that everything starts to look like a nail, we are only capable of noticing nails.

The most common nail of testers that I see is the error handling cases of any functionality. This
balances the idea that most common nail programmers see is the sunny day scenario of any
functionality, and with the two roles working together, we already have a little better coverage
over functionality in general.
To avoid the one ingredient recipe, we need awareness of all kinds of ingredients. We need to know
a wide selection of options for how to document our testing from writing instructional test cases, to
making freeform notes to making structural notes on individual level to making structural notes on
group level to documenting tests as automation as we are doing it. We need to know a selection of
coverage perspectives. We need to know that while we are creating programs in code, they are made
for people and a wide variety of people and societal disciplines from social sciences to economics
to legal apply. We need to know relevant ways things have failed before, being well versed in both
generally available bug folklore as well as local bug folklore, and to consider both not failing the
same way, but also not allowing our past failures to limit our future potential and drive testing by
risk, not fear.
This all comes down to the moment you sit in a team meeting, and you do backlog refinement over
the new functionality your team is about to work on. What are the tasks you ensure the list includes
so that testing gets done?
In that moment, what I find useful being put on the spot is recall heuristics. Something that helps
me remember and explain my thoughts in a team setting. We can’t make a decision in the moment,
without knowing our options.
I find I use three different levels of recall heuristics to explore what I need to recall my options in a
moment. Each level explores at a different level of abstraction:
Recall Heuristics for Test Ideation 34

change: starting from a baseline where the code worked, a lot of times what I get to test is on a level
of code commit to trunk (or about to head to trunk).

• story: starting from a supposingly vertical slice of a feature, a user story. In my experience
though people are really bad at story-based development in teams, and this abstraction is
available rarely even if it is often presented as the go-to level for agile teams.
• feature: starting from value collection in the hands of customers where we all can buy into the
idea of enabling new functionality.

For a story level recall heuristic, I really like what Anne-Marie Charrett has offered in her post here².
Simultaneously, I am in a position of not seeing much of story-based development but backlogs
around me tend to be on value items (features and capabilities) and the story format not considered
essential.

Recall on level of change

The trigger for this level of recall is a chance in code. Not a Jira ticket, but seeing lines of code change
with a comment that describes the programmer intent for the change.
Sometimes this happens in a situation of pairing, on the programmer’s computer, the two of you
working together on a change.
Sometimes this happens on a pull request, someone having made a change and asking for approval
to merge it to trunk.
Sometimes this happens on seeing a pull request merged and thus available in the test environment.
This moment of recall happens many times a day, and you thinking quickly on your feet under
unknown change is a difference in fast feedback and delayed feedback.
How I recall here:

• (I) Intent: What is supposed to be different?

• (S) Scope: How much code changed? Focused or dispersed?
• (F) Fingerprint: Whose change, what track record?
• (O) On it: How do I see it work?
• (A) Around it: How do I see other potentially connected things still work?

Recall on level of feature

The trigger for this level of recall is need of test planning on a scale of feature to facilitate
programmers carrying their share of testing but also to make space for testing.
²https://mavericktester.com/2019/12/31/heuristics-sfdipot/
Recall Heuristics for Test Ideation 35

Sometimes this happens in a backlog refinement meeting, the whole team brainstorming how we
would test a feature.
Sometimes this happens in a pair, coming up with ideas of what we’d want to see tested.
Sometimes this happens alone, thinking through the work that needs doing for a new feature when
the work list is formed by process implying “testing” happens on every story ticket and epic ticket
level without agreeing what it specifically would mean.

• (L) Learning: Where can we get more information about this: documents, domain understand-
ing, customer contacts.
• (A) Architecture: What building it means for us, what changes and what new comes in, what
stays.
• (F) Functionality: What does it do and where’s the value? How do we see value in monitoring?
• (P) Parafunctional: Not just that it works, but how: usability, accessibility, security, reliability,
performance…
• (D) Data: What information gets saved temporarily, retained, and where. How do we create
what we need in terms of data?
• (E) Environment: What does it rely on? How we get to see it in growing pieces, and where?
• (S) Stakeholders: People we hold space for. Not just users/customers but also our support, our
documentation, our business management.
• (L) Lifecycle: Features connect to processes, in time. Not just once but many times.
• (I) Integrations: Other folks’ things we rely on.

Recalling helps make choices as we are aware of our choices. It helps call in help in making those
choices.
 Automation First Microheuristic
Developers announce a new feature is available in the build and could use a second pair of eyes.
What is the first thing to do? Changing companies made me realize I have a heuristic on deciding
when I automate test cases as part of exploratory testing.
Both automating and not automating end up bringing in that second pair of eyes, that seeking of
understanding the feature and how it shows in the relevant flows. The first level of making the choice
if you start with automating is if you are capable of automating. It makes the choice available on an
individual level, and only after that it can be a choice.
When that choice is available, these things could impact choosing Automation First. * Belief that
change in the basic flow matters beyond anything else you imagine wrong with it

1 * When automating, you will visually and programmatically verify the basic flow as\
2 you are building it. Building it to a good reliable level takes longer than just lo\
3 oking at it but then remains around to see if changes in software change the status \
4 of it.

• Availability of quality dimensions (reliability, environment coverage) through automation

– If your application domain’s type issues are related to timing of use or multitudes of
environments where one works while others may not. automating first gives you a wider
scope than doing it manually ever could.
• Effort difference isn’t delaying feedback.
– With an existing framework and pipeline, extending it is an effort to consider. Without
them, having to set things up can easily become the reason why automating takes so long
it makes sense to always first provide feedback without it to ensure it can work.
• Brokenness of application
– Humans work around broken / half-baked features whereas writing automation against it
may be significantly harder.

I was thinking of this as I realized that the automated tests on my current system see very few
problems. There is no relevant environmental difference, like with my previous job. Automation
works mostly in the change dimension, unlike my previous job.
Going into the moment of making this choice, I find I still go back to my one big heuristic that
guides it all: Never be bored. First or Second does not matter as much as the idea that keeping
things varied helps keep me away from boredom. Documenting with automation makes sense to
avoid that boredom in the long run.
How would you test a textfield?
I’ve been doing tester interviews recently. I don’t feel fully in control there as there’s an established
way of asking things that is more chatty than actionable, and my bias for action is increasing. I’m
not worried that we hired the wrong people, quite the opposite. But I am worried we did not hire
all the right people, and some people would shine better given a chance of doing instead of talking.
One of the questions we’ve been using where it is easy to make step from theory to practice is How
would you test a text field? I asked it in all, around a whiteboard when not on my private computer
with all sorts of practice exercises. And I realized that the exercise tells a lot more when done on a
practice exercise.
In the basic format, the question talks of how people think of testing and how they generate
ideas. The basic format as I’m categorizing things here is heavily based on years of thinking
and observation by two of my amazing colleagues at F-Secure Tuula Posti and Petri Kuikka, and
originally inspired by discussions on some of the online forums some decades ago.

Shallow examples without labels - wannabe testers

There’s a group of people who want to become testers but yet have little idea of what they’re into,
and they usually tend to go for shallow examples without labels.
They would typically give a few examples of values, without any explanation of why that value
is of relevance in their mind: mentioning things like text, numbers and special characters. They
would often try showing their knowledge by saying that individual text fields should be tested in
unit testing, and suggest easy automation without explaining anything else on how that automation
could be done. They might go talking about hardware requirements, just to show they are aware of
environment but go too far in their idea of what is connected. They might jump into talking about
writing all this into test cases so that they can plan and execute separately, and generate metrics
on how many things they tried. They might suggest this is a really big task and suggest to set up a
project with several people around it. And they would have a strong predefined idea of their own
of what the text field looks like on screen, like just showing text.

Seeing the world around a text box - functional testers

This group of people have been testers and caught up some of the ideas and lingo, but also often over
reliance on one way of doing things. They usually see there’s more than entering text to a text box
that could go wrong (pressing the button, trying enter to send the text) and talk of user interface
more than just the examples. They can quickly list categories of examples, but also stop that list
How would you test a textfield? 38

quit quickly as if it was irrelevant question. They may mention a more varied set of ideas, and list
alphabetic, numeric, special characters, double-byte characters, filling up the field with long text,
making the field empty, copy-pasting to the field, trying to figure out the length of the field, erasing,
fitting text into the visible box vs. scrolling, and suggest code snippets of HTML or SQL, the go to
answer for security. They’ve learned there’s many things you can input, and not just basic input
into the field, but it also has dimensions.
This group of people often wants to show the depth of their existing experience by moving the
question away from what it is (the text field) to processes and emphasize experiences around how
relevant it is to report to developers through bug reports, how they may not fix things correctly and
how a lot of time goes into retesting and regression testing.

Tricks in the bag come with labels - more experienced

functional testers
This group of testers have been looking around enough to realize that there are labels for all of the
examples others just list. They start talking of equivalence partitioning and boundary values, testing
positive and negative scenarios and can list a lot of different values and even say why they consider
they’re different. When the list starts growing, they start pointing out that priority matters and not
everything can be tested, and may even approach the idea of asking why would anyone care of
this text field, where is it? But the question isn’t the first thing, the mechanic of possible values is.
They prioritization focus takes them to address use of time into testing it, and they question if it
is valuable enough to be tested more. Their approach is more diversified and they often are aware
that some of this stuff could be tested on unit level and others require it integrated. They may even
ask if seeing the code is available. And when they want to enter HTML and SQL, they frame those
not just as inputs but as ideas around security testing. The answer can end up long, and show off
quite much of knowledge. And they often mention they would talk to people to get more, and that
different stakeholders may have different ideas.

Question askers - experienced and brave

There’s a group who seems to know more even though they show less. This group realizes that
testing is a lot about asking questions, and mechanistic approach of listing values is not going to be
what it takes to succeed. They answer back with questions, and want to understand typically the
user domain but at best also the technical solution. They question everything, starting with their
understanding of the problem at hand. What are they assuming, and can that be assumed? When
not given a context of where the text field is, they may show a few clearly different ones to be able
to highlight their choices. Or if the information isn’t given, they try to figure out ways of getting to
that information.
The small group I had together started just with brainstorming the answer. But this level wasn’t
where we left off.
How would you test a textfield? 39

Group doing the exercise

After the listing of ideas (and assumptions, there was a lot of that), I opened a web page on my
computer with a text field and an ok button and had the group mob to explore, asking them to apply
their ideas on this. Many of the things they mentioned in the listing exercise just before immediately
got dropped - the piece of software and possibility to use it took people with it.

The three exercises

The first exercise was a trick exercise. I had just had them spend 10 minutes thinking how they
would test, and mostly they had not thought about the actual functionality associated with the text
field. Facing one, they started entering values and looking at output. Over time, they came up with
theories but did not follow up testing those and got quite confused. The application’s text field had
no functionality, only the button had. After a while, they realized to go into dev tools and the code.
And were still confused with what the application did. And with a few rounds of three minutes each
on the keyboard, I had us move on to the next example.
The second exercise was text box in the context of a fairly simple editor application, but one where
focusing on the text box alone without the functions immediately connected to the text box (unit
test perspective) would miss a lot of information. The group was strong on ideas, but weaker on
execution. When giving a value, what a tester has to do is to stop (very shortly) and look at what
they learned. The learning wasn’t articulated. They missed things that went wrong. Things where to
me, an experienced exploratory tester, the application is almost shouting to tell how it is broken. But
they also found things I did not remember, like the fact that copy pasting did not work. With hints
and guidance through questions, I got them to realize where the text box was connected (software
tends to save stuff somewhere) and eventually we were able to understand what we could do with
How would you test a textfield? 40

the application and what with the file it connects to. We generated ideas around automation, not
through the GUI but the file and discussed what kind of things that would enable us to test. When
asked to draw a conceptual picture of relevant pieces, they did good. There was more connections
to be found, but that takes either a lot of practice on exploring or more time to learning layers.
Again with the second exercise, I was left puzzled on what I observed. They had a lot of ideas as a
group on what to try, but less of discipline in trying that out of following what they had tried. While
they could talk of equivalence partitioning or boundaries, their actual approaches on thinking what
values are equivalent and learning more as they used the application left something to hope for.
The sources of actual data were interesting to see, “I want a long text” ended up as something they
could measure but unawareness immediately of an online tool that would help with that. They knew
some existed but did not go to get those. It could have been a priority call, but they also did not talk
about doing a priority call. When the application revealed new functionality, I was making a mental
note of new features of the text box I should test. And when that functionality (ellipsis shortening)
changed into another (scroll bars), I had a bug in mind. Either they paid no attention, or I pointed
that out too soon. Observation and reflection of the results was not as strong as idea generation.
The third exercise was a text field in a learning API, and watching that testing unfold was one of
the most interesting ones. One in the group quickly created categories of three outputs that could
be investigated separately. This one was on my list because the works right / wrong is multifaceted,
and in the perspective of where the functionality would be used and how reliable it would need to
be. Interestingly in the short timeframe we stick with data we could easily generate, and this third
one gave me a lot to think about as I made later one of my teams’s developers test it, getting them
even stronger into testing an output at a time, and insisting on never testing an algorithm without
knowing what it includes. I regularly test their algorithms of assessing if the algorithm was good
enough for the purpose of use, and found that the discussion was around “you shouldn’t do that,
that is not what testers are for”.
The session gave me a lot of food for thought. Enough so that I will turn this into a shorter session
teaching some of the techniques of how to actually be valuable. And since my conference tracks
planned are already full, I might just take an extra room for myself to try this out as the fourth
track.
Exploring Gilded Rose
There is a lot of talk around testing—who will do it, when it needs to happen, boxes it needs to fit
in—yet not enough on the actual testing. This is the first article in the series of looking at software
to test, and figuring out how to test it. This article is based on the experiences I’ve had watching
people test as I coach and teach them using these examples.

Introducing Gilded Rose

Gilded Rose³ is a refactoring Kata (practice) created by Emily Bache. The idea with Gilded Rose is
simple. There’s an inn somewhere that has an inventory system implemented. They would want
it extended for new requirements but that won’t be all straightforward. Don’t break anything that
used to work!
I’m a tester, so I don’t naturally come to the problem as it has been given, but I come with the idea
that after someone changes it I may have to test it. How would I do that?
³https://github.com/emilybache/GildedRose-Refactoring-Kata
Priming With Information Sources
and Tools
At work, things don’t come to you with the full range of sources and tools readily handed in. For
doing the exercise, I no longer drop people in cold to “just figure it out” but I give them a few starting
points. Requirements. Gilded Rose comes with a requirements specification⁴. Would that be of use?
⁴https://github.com/emilybache/GildedRose-Refactoring-Kata/blob/master/GildedRoseRequirements.txt
Priming With Information Sources and Tools 43

Requirements Specification
Priming With Information Sources and Tools 44

But there’s also other sources you can turn your attention to:

• Code. It “works in production” now and you can look at the implementation. If Java isn’t your
cup of tea even if I use it while I teach this, Emily has been nice to provide it with tons of other
languages.
• Unit test. The one unit test gives a starting point of how to execute the code so that you don’t
need to figure that out.
• IDE with visual code coverage. Code and unit test in an environment where you can start
working on it, with a code coverage tool. I use Eclipse with Code Coverage as I just like the
visual nature of it showing green, yellow and red for the branches.
• Ideas of the domain. You have past experiences of what makes sense for a shop and inventory
management system. You have ideas of what inputs are meaningful, and what could be
confusing. Everything you’ve learned about what makes software worthwhile is with you.

To make the exercise slightly more tester friendly and approachable in coaching people who never
work with code, I extracted a method out of the original unit test.

The Test

If you want to try things out before spoilers, pause here and go do the exercise. Figure out what your
test cases for it look like and why they are the way they are.

Getting to Work — How Would I Test this?

We’re approaching the exercise with exploratory testing, and all of our options are open. What makes
this exercise particularly exploratory is that I will rule out the option of going away to your cubicle
Priming With Information Sources and Tools 45

to write test cases based on the specification without running a single test. I expect you to design
your tests as you go and allow you to learn rather sooner than later.
Unlike for me right now writing this article after having done the exercise, you are now at a time you
know the least. You know there’s two information sources and either specification or code could
be your starting point. If you are a tester by trade, you are likely to lean towards the specification
and if you are a developer by trade, you are likely to lean towards the code and coverage tools.
As the exploratory tester, you are on the driver seat. You get to choose which way you go. And any
mix is possible. There is no recipe. But there is options.

Just Try It
You could just forget about the specification for now, as well as not read the code that implements
this, and start playing with values you can enter into the CheckItem-method. It takes three inputs:

• a name (String item)

• a number of days we sell the item (Integer selling)
• a number indicating how valuable it is (Integer quality)

If we didn’t look at the specification, deducting this much info out of the interface is unlikely. We
would just see it takes text and two numbers. But that is enough to play with it! This brings you to
the problem with least structure, highest chance of getting confused and highest chance of learning
something outside the spec and the code.
You could look at the problem with the simple ideas around the inputs. What if the input is really
really long? What if the number is really really big? Oo, negative numbers? Special characters? All
the things forbidden? Hey, there’s numbers that are special: what if your input is 00100, is that 100
or something different?

Read the Specification

Exploratory testing does not mean you have to jump in without considering any of the sources. It
means you are intentional about what you choose, and you combine things in ways that keep you
engaged as well as ensure you do a good job tracking coverage and meaningfulness of your work.
Reading the specification gives you one way tracking coverage.
The specification is full of claims. Some of them are clear. Some will turn out not so clear when
you’re testing and seeing what the application does. Some lines are single claims, some are have
multiple claims in them. Some lines stand on their own, others depend on other lines of text. Not all
lines are meaningful at all.
Which one do you choose to start from? How do you know? Actually, you don’t know. So you
sample something and hope to have made a selection that leads you to understanding rather than
confusion. And that if it leads you to confusion, you get out of there with later samples.
Priming With Information Sources and Tools 46

Read the Code

You could also choose to read the code. You could choose to introduce some tests that enable you
to step the logic through in a debugger so that maybe you could see some patterns in how it is
implemented. Maybe you just read it without executing so much. Read line by line, or read paying
attention to some aspects like variable names or values the code checks against.
Code is the ultimate truth, what is not in the code that now “works in production” isn’t working in
production. The spec and the code can be in sync or not, but if they disagree, the code wins until it
gets changed.

Think about the environment

The program you’re supposed to test is probably intended for some use by some people somewhere.
That somewhere most likely isn’t the test machine you’re using now, and the end user interface most
likely isn’t going to be the method you have your hands on now.
What would change in the way you look at the application of the environment was different? Would
someone try running it concurrently? Would it need to speak to an external system? Could it be
limited to run with minuscule amount of memory, or in an environment that does not allow Java
Virtual Machine to be running?

The First Test

There is no absolute choice for the first test, and having tested this with a good crowd of people both
individually and in mobs, some people still make a different choice for the first test in the setup we
test in.
For me, the first test is to see we can actually test. Running the test that has been given to us as an
example. The test that reveals our ability to run any consequent tests.

The Test

There has been days of going into doing the exercise where this test fails, because I accidentally
cleaned up more than I should after the previous run through the exercise.
Priming With Information Sources and Tools 47

Generally, this test results in a green bar.

The Test Run Status

As we are exploratory testing, we learn that the interface provided is good for us to go further. We
learn that the tool reports us with green when a test is given that passes. We might even read the test
to figure out that it says that when we start with item named “Any”, no days to sell it and quality
of zero, the quality isn’t changing anywhere.

The Second Test

With the second test, we arrive at the significant divergence of choices.

• the freeform value exploration

• the specification
• the code through measurable coverage
• the context of use

The real choice is actually to realize they are all different dimensions and for properly testing this,
some work on all might need to happen. Some tracking of coverage on all of them would need to
happen. Some discipline in not abstracting results of one to fully cover the results of another would
need to happen.
What I often see with the choice of the second test is that some people choose to just pay attention
to the code, and end up missing out on all the problems the specification leads you to uncover.
Some people choose to pay attention only to the specification and report problems that aren’t really
problems and fail to cover the code. And some people just don’t feel like they are empowered to
add anything beyond the given artifacts, which is detrimental to their ability to uncover yet another
category of problems.
Priming With Information Sources and Tools 48

Reading the Spec While Aiming for 100% Branch

Coverage
Let’s assume we intentionally, not accidentally, chose to approach this code and code coverage first,
with the help of the specification.
Here’s tests from one group⁵ to get to 100% branch coverage. 18 tests. Took two hours in a mob.
At first they tried giving good names, but knowing the least to begin with, the names were not really
good.
Halfway through they got tired of trying to think of names and just gave up thinking and trying to
understand other things and focused on getting through the coverage of specification and code.
In every single test their assert was on quality, and they never see a requirement around how sellin-
number would behave in case of legendary items.
Power of the group lead them to pay attention to the specification and they found the discrepancies
there:

• The lack of input validation implied by many requirements

• The shorthand of naming items in the specification in comparison to full names used in the
code
• The fuzziness around limits the rules defined that behavior would change at

By the time they were done, they had worked significantly. Yet they called done before it was time.
No cleaning up the names, no looking for rules they might have missed.

Tools Ease The Exploration

To contract to the 18 handpicked tests over 2 hour intensive work, I’ll show you a few minute example
of just covering the code with ApprovalTests-library.
This tool includes a possibility to pass a group of values and automatically generate combinations of
those values. The generated tests are pushed into a text file where we can visually verify and approve
them. Within the minutes version, I would use the principle of them all being correct because this is
legacy code that Works in Production. Within minutes, I generated 41616 tests to get to 100% branch
coverage, and to run them again to make sure nothing breaks it takes 1.028 seconds to run. The code
to do that is four lines and I go with the ultimate lazy of not even hand-picking relevant integer
values but using all between -1 and 100.
⁵https://gist.github.com/maaretp/e7dafe02b662ab809fbca2f76f8d4110
Priming With Information Sources and Tools 49

ApprovalTest code

If you miss the copy-pasteable example of this, I put in a gist⁶. You may notice I needed to do one
change to the original method so that the returning object would have a toString() to write to file.
What was object type GildedRose in the first example, is no object type Items in the latter as it
already had the necessary toString() defined.
With the 41616 tests, I have now a list of things I can verify with specification. Obviously I would
sample heavily over checking them all. But with then visible, I can run into problems serendipitously.
I can notice that for
[Sulfuras, Hand of Ragnaros, 27, 23] ⇒ Sulfuras, Hand of Ragnaros, 27, 23
the second value, sellin, isn’t in fact changing. I could visually and conceptually compare it to another
item
[Foo, 1, 10] ⇒ Foo, 0, 9
Seeing that for all others but the legendary item Sulfuras, as specified, “never has to be sold” in
requirements means the sellin date won’t be changing.

Code coverage
I mentioned code and coverage earlier, and for including those in my exploration, my test
environment has a coverage plugin installed.
With already an individual test, I can see coverage in percentages.

Coverage numbers

Be careful though, the percentages here are line coverage - you get to 100% line coverage with many
branches not yet covered. Thus it is handy that the tool used has another way of looking at things,
color coding lines to reflect where branches have been covered.
⁶https://gist.github.com/maaretp/2ec5eb9e38b9d9b1758f98e0bdb016ed
Priming With Information Sources and Tools 50

Coverage over code

To use coverage, you will need to read a bit of code. But reading it requires you to pick up texts that
are meaningful (the names of items you sell) and numbers that are meaningful (the values where
behavior is changing). If you feel lazy on the thinking side, a good rule for seeing a number is the
Boundary principle: choose the number, something just a little smaller and just a little larger. Adding
an extra test and seeing what happens often both makes your exploration faster, but also enables
serendipity - lucky accident of running into something your careful design isn’t leading you to, at
least not yet.

Summing up to some fun pitfalls

For a little exercise like this, it has surprising dimensions. To end this article, I wanted to share
lessons learned with one newbie tester who found out there was a lot to learn.
They did not get bit by missing the first test, because they were lucky enough to have it in a safe
and good green starting point.
They started creating their tests from the end of the specification, where many claims depend on
each other. They assumed they spotted an easy claim to verify which I find a fascinating judgement
call of what is easy and what is difficult. Defining a test around that claim lead them to learn things
that were not true, and double the time to finish the exercise to include unlearning things they had
started to firmly believe in.
The lessons were:

• First test was a poor choice leading to long-term confusion, we could either choose differently
or pay attention to what we really know more actively.
• Any coverage will do, some here some there and testing is done! That is true, but tracking
how much done are you is a big part of testing. Even the code coverage can fool you because
the number shows line coverage and the colors show branch coverage. Calling it done on line
coverage left out lines leading to new branches.
Priming With Information Sources and Tools 51

• Naming coverage dimensions isn’t a thing everyone does, they didn’t. Conceptualizing code,
specification, environment and risk coverage is a necessary thinking model.
• Code coverage can be achieved without proper asserts and gives a very false sense of security.
You don’t even need to assert values of quality for the 16 tests above to have them still run to
100% branch coverage.
• Problems against specification were all assumed distinct problems, no grouping around the fact
that there was full areas of “I expect input validation” that were not implemented anywhere.
• Specification having shorthand like saying “Sulfuras” instead of “Sulfuras, the Hand of
Ragnarok” isn’t a major problem with the specification even if it bit them in the exercise. Best
of specifications are helpful, yet incomplete. Best of testers using specifications don’t need them
to be complete to do complete testing.
• When results did not match the specification, quickly jumping to conclusion that we are testing
a buggy software which wasn’t the case. The models to pinpoint whether problem could be
in my tests that I have control over were not in place. We practiced many rounds of “is there
another test we could do that would give us a second data point to verify we understood the
requirement”.
• One test one requirement isn’t a thing. They are a messy network of dependencies.

There is no easy recipe for testing any application. Stop to think. Approach from different angles.
Check and double-check. And don’t hide in your corner, maximize your chances of getting to the
right information by working with the others.

Exploring a WebUI with Automation

Open space conferences like Socrates UK Digital Summer provide a great platform for making a little
progress on finding ways to teach about exploratory testing in writing. For purposes of writing, I
run an ensemble testing session to compare notes of what I did in prep alone vs. where the group
ends up. Putting the two together could provide useful lessons for those who did not get to join.
For these sessions, I picked up a new test target. Eviltester posted some of his testing apps and games
a while back, and EPrimer ended up as my choice as it promised * Not heavy on bugs - could actually
focus on testing instead of bug reporting * Completely unknown domain: proper English language
writing style “eprime” I had never heard of. * WebUI with beautiful IDs
At this point, I encourage you to follow the link⁷ to the app and stop reading what I say before you
tried it out yourself. If you did not follow my encouragement, I suggest that after reading this, pick
up another of the eviltester test targets and apply what you learned here.

Session Charter: Explore EPrimer focusing on two kinds of documentation as output: test
automation you can run (using e.g Robot Framework) and a mindmap. Time: 1 hour plus
learning time for test automation tool if you have no experience and no expert available
answering your questions in the moment.
⁷https://eviltester.github.io/TestingApp/apps/eprimer/eprimer.html
Priming With Information Sources and Tools 52

Two sessions, two results

As expected, the two session provided very different results that complement one another.
Session one produced ∼30 tests one can run again, spread over 7 test suites, each named on the type
of collection of data it was testing and a mindmap on realization that all tests were on single function
while there were multiple but covered the domain description as specification well, identifying
multiple problems against specification.
Session two produced 5 tests one can run again, all in 1 test suite where a bit of commenting out
is necessary to get the tests to run later. The coverage of functions was significantly better and the
session identified 2 bugs. No mindmap was created and better function coverage came from choosing
to understand everything a little and not diving systematically into specification. Single created test
covered more ground.

Breakdown of activities
Whenever we are doing exploratory testing, we get to make choices of where we use our limited time
based on the best information available at the time of testing. We are expected to intertwine various
activities, and when learning, it may be easier to learn one activity at a time before intertwining
them.
If you think back to learning to drive (while stick gear was a thing), you probably have ended up
in an intersection, about to move forward and your car shutting down as intertwining your actions
with the gear and pedals were not quite as they should. You slowed down, made space for each
activity and got the car moving again. Exploratory testing is like that, you control the pace and
those who have practiced long will be intertwining activities in a way that appears magical.
For this testing target, we had multiple activities we needed to intertwine (learn / design / execute):
* Quickly acquiring domain knowledge: no one knew what eprime is, and we had our choice of
reading about it. * Acquiring functional knowledge: using the application and figuring out what it
does. * Creating simple scripts with multiple inputs and outputs: using same test as template for data-
driven helps repeat similar cases in groups. * Identifying css selectors: if you wanted test automation
scripts, you needed to figure out what to click and verify and how to refer to those from the scripts.
* Controlling scope of tests: see it yourself, see it blink with automation, repeat all, repeat only the
latest. * Creating an invisible or visible model: Seeing SFDPOT (Structure, Function, Data, Platform,
Operations, Time) to understand coverage in selected or multiple dimensions * Cleaning up test
automation: Improving naming and structuring to make more sense than what was created in the
moment. * Using the application: Making space for chances to see problems beyond the immediate
test * Identifying problems: Recognizing problems with the application. * Documenting problems:
Writing down problems in either test automation or otherwise. * Working to systematic coverage:
Pick a dimension, and systematically cover it learning more on it. * Reading the code: We had the
code and we could read it. That could add to our understanding.
Priming With Information Sources and Tools 53

Taking another two hours on top of the two hours on cleaning up the results. I summarized final
results like this:

After 2 hours of #ExploratoryTesting documented with automation, it took another

2 hours to clean it up. 52 tests total, 38 passed, 14 failed. Quite many bugs as per
specification. Time on app has been maybe 30 minutes of this 4 hours.

The app isn’t completely tested, as the exercise setting biased us towards documentation.

Examples of activities
Quickly acquiring domain knowledge. Reading the specification of eprime. Focusing on examples
of eprime. Refreshing knowledge of English grammar around the verb “to be” e.g. 5 basic forms of
verbs and 6 different types of verbs, or 6 categories of verbs - all things I googled for as I am writing
this. While the specification tells what knowledge was probably used to create the application, there
is domain knowledge beyond what people choose to write down in specification.
Acquiring functional knowledge. Using the application. Asking questions about what is visible, par-
ticularly the concepts that are no obvious: ‘What is Possible Violations?”. Seeking data demonstrating
it could work. Seeking large data to demonstrate functions through serendipity.
Creating simple scripts with multiple inputs and outputs. Writing test automation that allows for
giving multiple input and output values as parameters. Getting into the tool and into using the tool.
Identifying css selectors. Getting to various values with code, understanding what is there in different
functions to click and check. Feeling joy systematic ID use making the work easier. Recognizing
conflicts in UI language and selector language.
Controlling scope of tests. Moving tests to separate files. Commenting out tests that already work.
Running tests one by one.
Creating an invisible or visible model. Ensuring we see all things work once before we dig deeper in
any individually. Creating a map of learning in the last minutes of the session. Writing down notes
of what we are seeing as either test automation or other types of documents.
Cleaning up test automation. Rename everything named foo at time when we knew the least.
Comment out things to focus on the next thing getting done efficiently. Using domain concepts
as names of collections.
Using the application. Spending time using the application to allow for serendipity. Observing look
and feel. Observing selected terminology.
Identifying problems. Seeing things that don’t work. Like visual of it that is very bare-bones. Or line
breaks turning valid positives into false negatives.
Documenting problems. Writing these down in test automation. Figuring out it we want to leave it
passing (documenting production behavior) or failing (documenting bugs). Remembering issues to
mention. Writing them down as notes. Writing a proper bug report.
Priming With Information Sources and Tools 54

Working to systematic coverage. Stopping to compare models to how well those are covered. Creating
a visual model. Covering everything in the specification. Covering all visible functionality.
Reading the code. Closing the window that has the code as it gets in the way of moving between
windows. Reading the code to see how concepts were implemented.

Some Reflections
Every time I teach exploratory testing, I feel I should find ways of teaching each activity separately.
There is a lot going on at the same time, and part of its effectiveness is exactly that.
In the group, someone suggested we could split the activity so that we first only document as
test automation, not caring about any of the information other than what is true right now in the
application. Then we could later review it against specifications and domain knowledge. That could
work. It would definitely work as one of the many mixes when exploring - change is the only
constant. This split is what approval testing is founded on, yet I find that I see different things when
I use the application and create documentation intertwined, than receiving documentation that I
could review. One night in between the actions is enough for me to turn into a different person.

The Final Deliverables

In the last minutes of one of the sessions, I cooked up a mindmap of what was in my head on the
application. I had only covered a small portion, focusing on counting Discouraged words.
Priming With Information Sources and Tools 55

Mindmap of Exploration

The robot tests from the two sessions combined with cleanup are available on github.⁸
⁸https://github.com/maaretp/exploring-robot-framework/tree/master/eprime
Python Primer for Exploring an API
With first of our release, I taught the most straightforward way I could to test an API for my summer
trainee. I gave them a URL (explaining what a URL is), showed different part of it indicated where
you connected and what you were asking for and ended up leaving office for four hours letting
them test for the latest changes just as other people in the team wanted to get out of office for their
summer vacation. They did great with just that in my absence, even if they felt the responsibility of
releasing was weighing on them.
No tools. No postman. Just a browser and an address. Kind of like this: http://api.zippopotam.us/us/90210

API Responds in Browser

The API we were testing returned a lot more values. We were testing 20000 items as the built-in limit
for that particular release, and it was clear that the approach to determine correctness was sampling.
Two weeks later, today we returned to that API, with the idea that it was time to do something more
than just looking at results in the browser.

Python, in the interpreter

We started off by opening a command line, and starting python.
Python Primer for Exploring an API 57

API responds in interpreter

As we were typing in import requests, I explained that we’re taking a library into use. Similarly
I explained print(requests.get("http://api.zippopotam.us/us/90201")), forgetting the closing
parenthesis at first and adding it on a line after.
With the 200 response, I explained the idea of this code meaning it was ok, but we’d need more to
see the message we had earlier seen in a browser, and that while we could also use this for testing,
we’d rather move to writing our code to a file in an IDE.

Python like a script, in Pycharm

As we opened Pycharm and created a .py file to write things in, the very first lines were exactly the
same ones we had been running from command line. We created two files. First requirements.txt
in which we only wrote requests and second file ended up with name experiments.py. As the two
lines were in, Pycharm suggested installing what requirements.txt defined and we ensured it was
still running just the same. At first we found the Run menu in IDE, later the little green play buttons
started to seem more appealing as well as the keyboard shortcut for doing this one often.
We replaced the print with a variable that could keep our response to explore it further response
= requests.get("http://api.zippopotam.us/us/90210") typing in response. and ctrl+space, we
could see options of what to do with it and settled with print(response.text) At this point, we
could see the same text we had seen before in browser, visually verify it just as much as with the
browser and were ready to move on.
Next we started working on the different pieces of the URL, as we wanted to test same things in
different environments, and our API had a few more options than this one I use for educational
purposes here.
We pulled out the address into a variable, and the rest of it into another, and concatenated them
together. for the call.
import requests address = "http://api.zippopotam.us/" rest_of_it ="us/90210" whole_thing
= address + rest_of_it response = requests.get(whole_thing) print(response.text)

The API we were playing with had a lot more pieces. With environments, names, id’s, dates, limits
and their suffixes in the call we had a few more moving parts to pull out with the very same pattern.
Python Primer for Exploring an API 58

As we were now able to run this for one set of values, our next step was to see it run for another
set of values. On our API, we’re working on a data-specific bug that ends up giving us a different
status code of 500, we wanted to move for the idea of seeing that here.
Making the status code visible with
print(response.status_code)

we started our work to have calls of the whole_thing where it wasn’t what we started with but had
multiple options.
#rest_of_it ="us/90210" rest_of_it = "fi/00780"

Every option we would try got documented, but the state of changing one into a comment and
another into the one we would was not what we’d settle for.
We wanted two things: * a method that would take in the parts and form the whole_thing for us *
a way of saving the results of calls
We started with keeping a part of the results introducing pytest writing that into requirements.txt
as second line.
requests pytest

Again we clicked an ok adding what our environment was missing as Pycharm pinged us on that,
and saved the response code codifying it into an assert. We remembered to try other values to see it
fail to trust it in the first place.
assert response.status_code == 200

Us still wanting the two things above, I interrupted our script creation to move us a step in a different
direction.

Python like a Class and Methods, in Pycharm

We googled for “pytest class example” under my instructions, and after not liking the first glance of
the first hits, we ended up on a page: https://code-maven.com/slides/python-programming/pytest-
class
We copied the example as experiments_too.py file contents on our IDE.
We hit a mutual momentary hiccup, to figure out three things: * We needed to set pytest as our
default test runner from File | Settings | Tools | Python integrated tools | Default test runner. * The
file must have Test in name for it to be recognized as tests * We could run a single test from the
green play button next to it
The original example to illustrate setup and teardown had a little bit too much noise, so we cleaned
that up before starting to move our script in to the structure. “‘ class TestClass():
 Python Primer for Exploring an API 59

1 def setup_class(self):
2 pass
3
4 def teardown_class(self):
5 pass
6
7 def setup_method(self):
8 pass
9
10 def teardown_method(self):
11 pass
12
13 def test_one(self):
14 assert True

1 We moved everything from the script we had created inside `test_one()`

def test_one(self): import requests address = “http://api.zippopotam.us/” # rest_of_it =”us/90210”

rest_of_it = “fi/00780” whole_thing = address + rest_of_it response = requests.get(whole_thing)
print(response.text) assert response.status_code == 200 “‘ And we moved the import from inside
the test to beginning of the file to have it available for what we expected to be multiple tests. With
every step, we run the tests to see they were still passing.
Next, I asked the trainee to add a line right after def test_one(self) that would be like we imagined
what we’d like to call to get our full address. We ended up with
define_address("foo", "bar")

representing us giving two pieces of text that would end up forming the changing parts of the
address.
A little red bulb emerged on the IDE next to our unimplemented method (interjecting TDD here!)
and we selected Define function from the little menu of options on the light bulb. IDE created us a
method frame. def define_address(param, param1): pass
We had already been through the idea of Refactor | Rename coming up with even worse names and
following the “let’s rename every time we know a name that is better than what we have now”
principle. I wouldn’t allow just typing in a new name, but always go through Refactor to teach the
discipline that would be benefiting from the tooling. Similarly, I would advice against typing whole
words but allowing IDE to complete what it can.
We moved the piece of concatenating two parts together into the method (ours had a little more
parts than the example). def define_address(part1, part2): whole_thing = part1 + part2
return whole_thing`
 Python Primer for Exploring an API 60

and were left with a test case where we had to call the method with relevant parts of the
address def test_one(self): # rest_of_it ="us/90210" response = requests.get(define_-
address("http://api.zippopotam.us/", "fi/00780")) print(response.text) assert response.status_-
code == 200 The second test we’d want as comment in the first became obvious, and we created a
second test. def test_two(self): response = requests.get(define_address("http://api.zippopotam.us/",
"us/90210")) assert response.status_code == 200

Verifying that response.text

Now that we had established the idea of test cases in a test class and structure of a class over writing
just a script with a hint of TDD, we moved our attention to saving results of the calls we were
making. Seeing “200 success” isn’t quite what we’d look for.
In the final step of the day, we introduced approvaltests into requirements.txt file.

1 approvaltests
2 pytest-approvaltests

We edited two line of our file, adding

from approvaltests.approvals import verify
and changing print to verify verify(response.text)
We run the tests from terminal once to see them fail (as we saw them be ignored without this step
on the usual run)

1 pytest --approvaltests-use-reporter='PythonNative' TestClass.py`

We saw a file TestClass.test_one.received.txt emerge in our files, and after visually veri-
fying it captured what we had seen printed before, we renamed the file as TestClass.test_-
one.approved.txt. We run the tests again from the IDE to now see them pass, edited the approved-
file to see it fail and corrected it back to verifying our results match.
As finalization of the day, we added verification on our second test, again visually verifying and
keeping the approved file around.

1 def test_one(self):
2 response = requests.get(define_address("http://api.zippopotam.us/", "fi/00780"))
3 verify(response.text)
4 assert response.status_code == 200`

And finally, we defined approvaltests_config.json file to include information where the files
approvaltests create should go ‘ { “subdirectory”: “approved_files” } “‘
These steps give us what we could do in a browser, and allow us to explore. They also help us save
results for future with minimal effort, and introduce a baseline from which we can reuse things
we’ve created.
Exploratory Testing an API
This article was published in Ministry of Testing Testing Planet in 2016. Appropriate pieces of it will
find their place as part of this book.
As an exploratory tester I have honed my skills in testing products and applications through a
graphical user interface. The product is my external imagination and I can almost hear it whispering
to me: “Click here… You want to give me a different input… Have you checked out the log file I’m
producing?”
Exploratory testing is a systematic approach to uncovering risks and learning while testing. The
whispers I imagine are heuristics, based on years of experience and learning of how I could model
the product in relevant ways to identify information relevant to stakeholders. While the product is
my external imagination when I explore, I am my programmer’s external imagination when they
explore. They hear the same, unspoken whispers: “You’d want me to do this… I guess I should then.”
and they then become better testers.
I’ve only recently started applying this skillset on APIs - Application Programming Interfaces. An
application programming interface is a set of routines, protocols, and tools for building software and
applications. What triggered this focus of exploration effort was an idea to show at a conference how
testing something with a code interface is still very similar to testing something with a GUI.
With an API call, I can just fill in the blanks and figure out how the API sits in the bigger picture.
There should be nothing that stops me from exploring through an API, but why haven’t I done
it before? And then as I started exploring an API with a testing mindset, I started seeing APIs
everywhere, and finding more early opportunities to contribute.

Why Is Exploratory Testing An API Relevant?

The way I look at the world of testing, I see two ways we look at good testing - testing as creating
artifacts and testing as a performance. I learned this working side by side with Llewellyn Falco, a
test-infected developer. Whenever he would explain the testing as he knew it, it was clear he was
not speaking of testing as I knew it. We both talked about good testing, just very different ideas to
it.

Testing as Artifact Creation

People with automation first ideas to testing see testing as activity around creating artifacts. During
the previous times before agile and extensive automation, the same testing was supposed to be
Exploratory Testing an API 62

covered by detailed test cases. I’ve lived with Agile ideals long enough to have moved to the idea
that whatever is worth documenting in detail, it is probably worth documenting as test automation.
This way of looking at testing tends to focus on what we know.
When we approach testing as artifact creation, our focus is primarily on solving the problem of
creating right artifacts: what kinds of things would be useful automated? Where are the low-hanging
fruit and what kind of automation would help us drive and support the development?
The test automation artifacts at best give us:

• Spec - we know what we’re building

• Feedback - we know when it’s built as we specified
• Regression - we guard against things staying true over time
• Granularity - we can pinpoint what went wrong when the tests fail.

Testing As a Performance, (aka Exploratory Testing)

People with an exploratory testing background would be more inclined to see testing as a
performance, like a show with improvisation, improving with practice and revealing new layers
to how the performance goes. There’s nothing that stops you from creating automation from that
approach, but it starts with valuing different things. It starts with a focus on the things we don’t
know, and illusions we’re holding true without empirical evidence.
In contrast to testing as artifact creation, exploratory testing gives us:

• Guidance - not just yes/no, but a direction to better.

• Understanding - being able to place the information into a bigger picture,
• Models - ways to learn faster with supporting ideas or documents,
• Serendipity - the lucky accidents that none thought of that emerge given enough time and
variety to any application.
Exploratory Testing an API 63

What Does Testing Give Us?

What Testing Gives Us

We need both sides of the coin. Exploratory testing is a process of discovery, and it is entirely possible
to discover information we did not have from extensive test automation using APIs as our external
imagination.
There’s inherently nothing in exploratory testing that requires we must have a user interface or
have finalized features available. Still often I hear people expressing surprise at the idea that you
can explore an API.
In addition, exploring an API is thought as something that is intended for programmers. And an even
more specific misconception is that exploratory testing could not use programming / automation as
part of the exploration - that there would be something inherently manual required for exploration.
We must, as software testers, help team members understand that we can explore software in a
variety of manual, automated and technical ways.

An Example API with ApprovalTests

After a some time and research on considering a REST API or even some old Cobol APPC API or
a framework/library programmers typically could use, I ended up with ApprovalTests. It is created
Exploratory Testing an API 64

by a developer friend with significant reliance on the greatness of his unit tests, and I welcomed the
challenge to find problems through the means of exploratory testing.
ApprovalTests is a library for deciding if your tests pass or not, with a mechanism of saving a
result to a file and then comparing to the saved result. It also offers mechanisms of digging into
the differences on failure. It has extensive unit tests, and a developer who likes to brag about how
awesome his code is. The developer is a friend of mine and has a great approach to his open source
project: he pairs with people who complain to fix things together.
ApprovalTests have a couple of main connecting points.

• There are the Approvals that are specific to technology your testing. For example, my company
used ExcelApprovals that packaged a solution to problem of having different yet same results
with every run.
• And then there are the Reporters that are a way of saying how you want to analyze your tests
if they fail.

I personally know enough about programming to know to appreciate an IDE tool and the automatic
word completion feature. The one in Visual Studio is called Intellisense. It’s as if there is a GUI:
I write a few letters, and the program already suggests me options. That’s a user interface I can
explore, just as much as any other! The tools shows what the API includes.

Picture 1

Using the IDE word completion, I learn that Approvals has a lot of options in its API. Here is an
example where to test specific technologies with Approvals, you would want to make different
selections. Documentation reveals that Approvals.Verify() would be a basic scenario.
Exploratory Testing an API 65

Picture 2

I look at Reporters with the same idea of just opening up a menu, and find it hard to figure out what
in the list are reporters.
I later learn that it’s because of the word before, and that naming everything ReportWith would
help make the reporters more discoverable.
I also learn that the names can improve to include better their intent, for example some are supposed
to be silent - to run with continuous integration.

Picture 3

I go for online examples, and learn that they are images - not very user friendly. And I try to look for
in-IDE documentation, and learn it’s almost non-existent. I run existing unit tests to uncover they
don’t work at first run, but the developer fixes them quickly. And browsing through the public API
with the tool, I note a typo that gets fixed right away.
The API has dependencies to other tools, specifically test runners (e.g. nUnit and MSTest) and I want
my environment to enable exploring the similarities and differences with the two. Serendipitous
Exploratory Testing an API 66

order of installing the pieces reveals a bug in combination of using two runners in combination
with a delivery channel (Nuget). Over the course of testing, I draw a map of the environment I’m
working with, around ApprovalTests. The map is a source of test ideas on the dependencies.
I don’t only rely on the information available online, I actively ask for information. I ask the
developer what Approvals and Reporters do, to get a long list of things that some do and some
don’t - this becomes a great source for more exploratory testing. Like a checklist of claims, and a
great source for helping him tune up his user documentation.
Even a short exploration gave me ideas of what to dig in deeper, and issues to address for the
developer. Some additional sessions with groups in conferences revealed more problems, and showed
the power of exploratory testing on an extensively automation tested API.

13 Patterns To Help You Explore An API

It’s OK if you are unfamiliar with the concepts above in the specific example, but I think the patterns
below will help. They sum up my lessons learned on exploring an API myself, coming from a GUI-
focused viewpoint.

1 - Focus: Working with limited understanding

As an exploratory tester, your goal could be to provide value with a limited amount of effort. As you
are learning in layers, you need to choose the layers intelligently. There’s no right or wrong answer
on what to test first, but there are some usual candidates.
Some testers start looking at input fields. It’s like “fill in the blank” or like testing Google search.
What are the relevant things I could enter here?
How do I find the APIs to explore in the first place? Asking around, picking something people
call an API or a library? Using tools like Fiddler that track particular technologies API calls from
applications?
Others choose to start with documentation and collecting claims.
And others try to figure out a representative use case to get a basic feeling of what it would look
like when this works.
Notes are useful here. Park some ideas that you could make use of later. Choose some to take action
on. Continue to add more as you progress.
Regardless of what you do, you need to focus. Choose something. Learn. Choose something different.
Learn more. Continue until you’re satisfied with what you’ve tested or until you run out of time. The
idea is that when time runs out, you’ve already done the best testing you could in that timeframe.
Exploratory Testing an API 67

2 - Finding your building blocks

Tweak around a little, and you’ll see it. There are calls and operations. There are inputs and outputs.
There’s requests and responses. There are exceptions. There are dependencies. And all exist to serve
a purpose. Understand what your are the dials in the API that you can turn while testing.
You could play with the building blocks without understanding the purpose. You might discover the
purpose through the building blocks. But the building blocks are your blanks to fill in: what you call,
with what values, and what you expect to get back. Make a model of that.
You might call your inputs requests and your outputs responses. Find your vocabulary.
Also, look around for extending, could you be building something on top of these blocks?

3 - The environment it depends on

Your API probably does not exist alone. It depends on an environment. There’s a specific syntax
coming from the language in play. There’s things it uses, and things that use it.
Understand what’s in your scope (“your API”) and what is outside it (“ecosystem in which your API
provides value”). Draw a picture. Extend that picture as you’re learning more.

4 - Specific user with a specific purpose

APIs have users. Often, the users are developers. And developers are people with specific habits and
expectations.
They expect things to be language-wise idiomatic, to follow the conventions they are used to living
with. They expect particular styles of documentation. They expect discoverability.
Some of the best APIs are things where time to hello world (seeing it could work with some example)
is short and do not include stepping out of your IDE.
Talk to developers to learn some of their quirks. Listen to what they say about APIs they use and
don’t use. Listen to what they say about how they extend the APIs or work around their limitations.
In addition to intended use, there’s also misuse. What could go wrong if people misused your API?
Also, pay attention to names. If your API has something called ReadStuff() and it ends up creating
a lot of new data, that might be against what you expect. People use this. What could come in the
way of that and what should come in the way of that?

5 - Usability of an API
There’s great material out there on what makes a good API. It’s really a usability discussion!
When using the different commands/methods, what if they would be consistent in naming and in
parameter ordering so that programmers using them would make fewer mistakes?
Exploratory Testing an API 68

What if the method signatures didn’t repeat many parameters of same type so programmers using
them would get mixed up in the order?
What if using the API incorrectly would fail compile-time, not only run-time to give fast feedback?
What if the methods would follow a conservative overloading strategy, an idea that two methods
with same name would never have the same amount of arguments so that users can confuse the
inputs? I had never heard of the concept before exploring an API, and run into the idea googling for
ideas of what people say that make APIs more usable.
There’s even a position in the world called Developer Experience (DX), applying the user experience
(UX) concepts to the APIs developers use and focusing on things like Time to Hello world (see it run
on your environment) and Time to Working Application (using it for something real). These ideas
come naturally with an exploratory testing mindset.

6 - Why would anyone use this?

APIs exist for a purpose. Even the free open-source APIs exist for a purpose. Many open source
libraries hope to see more developers using them. Some may even be puzzled on why something
this excellent isn’t getting more traction. Sometimes with a lot of choice, you wonder why would
you choose something that is difficult or does not serve your exact needs?
It’s important that an exploratory tester asks, even faced with a programming interface the question
of purpose:

• Why does this exist?

• Who finds this valuable and in what way?
• What might come in between expected and received value?

7 - Think Lifecycle
You’re probably testing a version you have just kindly received into your hands. There were probably
versions before and there will be versions after. The only software that does not change is dead. How
does this make you think about testing?

• How would change work on this API?

• Is there a way for the user to recognize it’s version?
• How about removing functionality or replacing API calls, how long would you keep things
marked deprecated in your interface?
• And does that stop you from correcting even simple typos?
Exploratory Testing an API 69

8 - Google for concepts

The world is full of sources. If you see a word you don’t understand, Google is your friend. I ended up
googling “Conservative Overloading Strategy” after I learned it’s something you consider a feature
of a good API.
Answers are usually only a few clicks away.

9 - Collaborate: Fact-track your understanding

You’re not alone. Find someone to pair with. Find someone to ask questions from. Even if you
start testing all on your own, you don’t have to stay that way. Especially working on your own
organizations APIs, try pairing with a programmer.
I often specifically recommend Strong-Style Pairing. If your paired partner has better ideas on how
to test, you take the keyboard. Then speak to get ideas from the others head through your hands
into the computer.
It’s not just hands-on testing you could collaborate on. You could collaborate on coming up with
things you should know to test about. Mindmap creation together is a wonderful way of building
that information to feed exploratory testing.

10 - Documentation matters a lot for APIs

For programmers to be able to use an API, they look at documentation with examples. If you’re
examples are in many languages and all over the place, you’re not making it easier to find the basic
stuff.
If there’s no documentation, the usability and discoverability of your API should be good. Then
again, it could be that your users just have no options on what API to use for their purposes. That
seems popular too.

11 - Explore to create documentation

Programmers might not find the documentation creation their strong point, and while exploring,
you might become a subject matter expert in the use of the API you’re testing.
Turn the things you’ve learned into things that make the product better.

• Find the core of a new user’s experience and describe that clearly
• Add the deeper understanding of what (and why) the functionalities do into API documenta-
tion
• Clean up and remove things from automatically created API documentation
Exploratory Testing an API 70

12 - Some patterns become visible with repetition

I find myself doing almost or exactly same things many times before I understand what the API
does for a specific type of case. Sometimes repetition helps me see subtle differences that should be
tested for separately. Sometimes repetition helps me see how things are the same. I find that while
exploring, I need to have patience to do things more than once, with my mind paying attention to
all the subtle clues the API can give me. Learning is the key. When no longer learning, move on. But
give yourself time, as repetition is necessary.
My two favorite quotes on this are:

• It’s not that I’m so smart, it’s I just that I stay with the problems longer (Albert Einstein)
• The more I practice, the luckier I get (Arnold Palmer)

I find a lot more insights digging in deeper than what is first visible. Manual repetition may be key
for this insight, even on APIs.

13 - Disposable test automation

When you create these tests, they may end up being automated. Many times you have some level
of automation to access your API. You have requests and responses, and you may have code around
those. Some APIs you can’t run without code. Your code might not include automatic results
checking if you are just driving the API, not checking the results. But your code may also include
theories about ideas that must hold true, like exploring if a rare error message over large data sample
is really rare.
Most of all, with this automation you’re creating, you need to critically ask yourself:

• Could it be disposable, one time use?

• The value of it is in the learning it provided now?
• Are these tests that you want to see yourself maintaining?

Your set of automated tests will require maintenance. When tests fail, you will look into those.
Choose wisely. Choose when you learn what wise means.

Summary
Exploring APIs gives you the power of early feedback and easy access to new skills closer to code.
I suggest you give it a chance. Volunteer to work on something you wouldn’t usually work on as a
non-programmer. You’ll be surprised to see how much of your knowledge is transferrable to “more
technical” environment.
Anchoring an Idea While Exploratory
Testing an API
One of the things we get to test is a customer oriented API. It’s particularly lovely test target for
multiple reasons:

• Read-only: It only gets data, and does not allow us to change data. Makes it simpler!
• Time-constrained on API level: You can tell dates as input and it does freeze time for test
automation purposes. You don’t have to play with concepts of today() and now().
• Limited and understandable UI level edits to data: There are some things we can change from
GUI that impact the API but they are fairly straightforward.

The main reason it brought us joy for testing today is that we found a bug on it a few weeks back
where particular combination returns 500 error code (Server error) where it should not, and we got
to start creating some tests back then to create a nice baseline for the time that bug would be fixed.
The long awaited message of bug fix arrived today, and the first thing we’d do is pull out the tests
we had automated the last round (asserts and approvals, I wrote about those earlier as we set the
project up). We ran the tests, expecting to see a fail for the assert for getting 500 for that bug. The
results surprised us.
We still had that test passing, but now we also had another test failing with 500. Instead of going
forward with the fix, we had momentarily gone backwards.
Not long after, we got to try again with a new version. This time it was just as we expected. Within
30 seconds of realizing the version was available, we knew that on the level we automated our tests
before, those were now matching today’s expectations.
For those of you concerned on the tests not running in CI, it is about the same time to go check they
are blue as we did not place these tests as ones blocking the pipeline. These tests weren’t designed
for the pipeline, they were designed as an entry point for exploratory testing where we could leave
some of them behind for the pipeline or for other purposes.
We quickly drafted our idea of what we would test and change today:

• Capturing and reviewing for correctness for the combination that we previously documented
as receiving the 500 response for that bug
• Ensuring we could see latest data after the most recent changes
• Having easily configurable control over dates and times we had not needed in our tests before
Anchoring an Idea While Exploratory Testing an API 72

Making some of the tests approval files smaller in size as long as they did not lose the idea of what we
were testing with them What turned out to be the most fun thing to test was the latest data. Starting
with that idea, we found multiple other ideas of what to test, including things around changing more
values on the data, and things around multiple overlapping limits. We needed to remind ourselves,
multiple times, that we still have not seen our starting idea in action, even if we had seem many
other ideas.
As a conclusion of today, we came to the importance of anchor, and remembering that anchor. If
writing it down helps, write it down. If having a pair that keeps you honest helps, have a pair.
Whatever works for you. But a lot of times, when we do some testing, we end up forgetting what
was the thing we set out to do in the first place. Anchoring an idea allows us to discover while we
explore, and still stay true to what we originally set out to do.
We ended up refactoring our test code a bit to make it more flexible for the ideas we had today,
and we discovered one test we wanted to keep for future. It started off with one name and concept,
yet though exploring we learned that what we wanted to keep for future was different to what we
wanted and needed to do today.

Great chance of pairing on API #ExploratoryTesting with my trainee as there was a major
change on the logic behind the API. Got to experience the ideas of tests we throw away,
tests we want to keep for later but for reasons other than we start with, and relying on
past tests.

Truth is, we always throw some away, and that is where I recognize learning and thinking is going
on. Can keep and should keep are two different things.
Reporting and Notetaking
Exploratory Testing. The wonderful transformation machinery that takes in someone skilled and
multidisciplinary and time, crunches whatever activities necessary and provides an output of
someone more skilled, whatever documentation and numbers we need and information about
testing.

Input and Output

What happens in the machinery can be building tools and scripts, it can be operating and observing
the system under test but whatever it is, it is chosen under the principle of opportunity cost and
focused on learning.
When the world of exploratory testing meets the world of test automation, we often frame same
activities and outputs differently. When a test automation specialist discusses reporting, it seems
they often talk about what an exploratory tester would describe as note taking.
Note taking is when we keep track of the steps we do and results we deliver and explain what we
tested to other people who test. Reporting is when we summarize steps and results over a timeframe
and interpret our notes to people who were not doing the testing.
Note taking produces: * freeform or structured text * screenshots and conceptual images * application
logs * test automation that can be thrown away or kept for later
Reporting produces something that says what quality is, what coverage is, how long what we say
now stays true, and what we recommend we would do about that.
Automation can take notes, count the passes and fails, summarize use of time. The application under
test takes notes (we call that logging) and it can be a central source of information on what happened.
When automation does reporting, that usually summarizes numbers. I find our managers care very
little for these numbers. Instead, the reporting they seek is releases and their contents.
When automation does log of test case execution, we can call that a report. But it is a report in scope
very different than what I mean by a report from exploratory testing - including automation.
Test Case Management Tools in
Exploring
Zephyr, in case you did not know, is a Jira Test Management extension. I dislike Jira, and I dislike
Zephyr. But what I like and don’t like does not change (well, immediately) the whole organization,
and I play within general bounds of organizational agreements. In this case, it means an agreement
that tests are documented in Zephyr - for some definition of tests.
This chapter is about how I play within those bounds, enabling exploratory testing.

What Zephyr Brings In

Zephyr as a Jira plugin enables some very rudimentary test specific concepts: Ticket reuse. When
the jira ticket is a test, it can be run many times, like for example for each build we test. Normal Jira
tickets are more straightforward in their lifecycle.

• Steps. For some reason people still think tests have steps with expected values. If you don’t
know better, you might use these. DON’T.
• Mapping tests to releases. You can tell what test ticket connects with a particular Jira release.
It shows the structure of how testing usually progresses in relation to changes.
• Grouping. You can group tests inside releases into test suites. You have many reasons you might
want to group things. Zephyr calls mapping and grouping cycles.
• Run-time checklists. You can keep track of passes and fails, things in progress. You can do it
either on level of a group of tests or on an individual test. You have a whole own view to
making notes while testing on a particular test case, execution view. It seems to imagine all
your test needs in one place: bug reporting, steps, notes.

What I Bring In
When I document my plans of testing, I create a few kinds of tests: * [Explore] <write a one line
summary here> These tests can be for the whole application like “Gap analysis exploration - learn
all the problems they don’t yet know”, or a particular purpose like “Release”, or an area of particular
interest like “Use for people with disabilities”. If I can get away with it, I have only one test case
titled “[Explore] Release” and I only write notes on it at time of making a release. What this assumes
though is that release is something more continuously flowing rather than one final act in the end
- agile as if we meant it. * [Scenario] <write a one line summary here> These tests are for very high
Test Case Management Tools in Exploring 75

level splitting of stakeholder perspectives I want to hold space for. They are almost like the ones
I mark [Explore] expect that they all together try to summarize remembering the most important
stakeholders and their perspective in the product lifecycle. These are in the system context, regardless
of what my team thinks their component delivery responsibility has been limited to.
* [Feature] <write a one line summary here> These tests I use when I have bad or non-existent
documentation on what we promise the software will do. These tests all together try to summarize
what features we have and try to get to remain, but as a high level checklist, not going into details of
it. These are in the context of the system, but more towards the application my team is responsible
for.
I use states of these tests to indicate scope ahead of me.
If a test is Open (just like a regular Jira ticket), it is something I know we expect to deliver by a major
milestone like a marketing release all the little releases work towards, but I have not seen a version
in action we could consider for the major milestone scope. It reminds me to ask if we have changed
our mind on these.
If a test is Closed, it is still alive and used. but it is something where we have delivered all the way
to production some version of it and we intend to keep it alive there.
If I can get away with one test case, that is all I would do. There are many reasons for me not to be
able to get away with it: a newer colleague we need a shared checklist with, me needing a checklist
and creating it here with minimal extras, or auditing process that would not be fulfilled with just
that one ticket of [Explore] Release.
The updating of test status is part of release activities for me. Someone needs to create a release in
Jira, which usually happens when the previous release is out. For that release, I add at most two
Cycles: * Pre-Release Testing * Release Testing
Again, if I can get away with it, I have only one: Release Testing and within in, I have only one test:
[Explore] Release that I mark passed and write notes if I have something useful to say. Usually the
useful thing for me to say is “release notes, including scope of changes is available here <link>”.
The way testing works for me is that I see every pull request and nothing changes outside pull
requests. I test selected bits and pieces of changes, assessing risk in the moment. I also have a set of
test automation that is supposed to run blue/green (pick your color for ‘pass’) that hunts down need
of attending to some detail. And I grow the set of automation. If you need ‘proof’ of passing for a
particular release, we could in theory get that out of version control but why would you really want
that?
The Pre-Release Testing Cycle, if it exists, I fill it when I think though what happened since last
release and what still needs to happen before the next one and I drag in existing tests from all
three categories [Explore], [Scenario] and [Feature] to be a checklist. What this cycle contains tells
about themes and features I found myself limiting to. And when a Pass on the cycle isn’t sufficient
documentation, I can always comment the test ticket.
My use of Zephyr is very different to my colleagues. Perhaps also to your use?
Process
This section collects together topics around process. How to frame software development in general.
Feature and Release testing
Back in the day, we used to talk about system testing. System testing was the work done by testers,
with an integrated system where hardware and software were both closer to whatever we would
imagine having in production. It usually came with the idea that it was a phase after unit and
integration testing, and in many projects integration testing was same testing as system testing but
finding a lot of bugs, where system testing was to find a little bugs and acceptance testing ended
up being the same tests but now by the customer organization finding more bugs that what system
testing could find.
I should not say “back in the day”, as for the testing field certification courses, these terms are still
being taught as if they were the core of smartassery testers need. I’m just feeling very past the terms
and find them unhelpful and adding to the confusion.
The idea that we can test our software as isolated units of software and in various degrees of
integrated units towards a realistic production environment is still valid. And we won’t see some of
the problems unless things are integrated. We’re not integrating only our individual pieces, but 3rd
party software and whatever hardware the system runs on. And even if we seek problems in our
own software, the environment around matters for what the right working software for us to build
is.
With introduction of agile and continuous integration and continuous delivery, the testing field very
much clung to the words we have grown up with, resulting in articles like the ones I wrote back
when agile was new to me showing that we do smaller slices of the same but we still do the same.
I’m calling that unhelpful now.
While unit-integration-system-acceptance is something I grew up with as tester, it isn’t that helpful
when you get a lot of builds, one from each merge to master, and are making your testing way
through this new kind of jungle where the world around you won’t stop just so that you’d get
through testing that feature you are on on that build you’re on, that won’t even be the one that
production will see.
We repurposed unit-integration-system-acceptance to test automation, and I wish we didn’t. Giving
less loaded names to things that are fast to run or take a little longer to run would have helped us
more.
Instead of system testing I found myself talking about feature/change testing (anything you could
test for a change or a group of changes comprising a feature that would see the customer’s hands
when we were ready) and release testing (anything that we needed to still test when we were making
a release, hopefully just run of test automation but also a check of what is about to go out).
For a few years, I was routinely making a checklist for release testing:
Feature and Release testing 78

• Minimize the tests needed now, get to running only automation and observing automation
running as the form of visual, “manual” testing
• Split into features in general and features being introduced, shining special light to features
being introduced by writing user oriented documentation on what we were about to introduce
to them

From tens of scenarios that the team felt that needed to be manually / visually confirmed to running
a matrix test automation run on the final version, visually watching some of it and confirming
the results match expectations. One automated test more at a time. One taken risk at a time, with
feedback on its foundation.
Eventually, release testing turned into the stage where the feature/change testing that was still
leaking and not completed was done. It was the moment of stopping just barely enough to see
that the new things we are making promises on were there.
I’m going through these moves again. Separating the two, establishing what belongs in each box
and how that maps into the work of “system testers”. That’s what a new job gives me - appreciation
of tricks I’ve learned so well I took them for granted.
Training and Exploratory Tester
This summer gives me the perfect possibility - a summer intern with experience of work life outside
software and I get to train them into being a proper Exploratory Tester.
Instead of making a plan of how to do things, I do things from a vision, and adapt as I learn about
what the product team needs (today) and what comes easy for trainee trusted into my guidance.
Currently my vision is that by end of the summer, the trainee will: * Know how to work effectively
in scope of a single team as tester inside that team * Understand the core a tester would work from
and regularly step away from that core to developer and product owner territory * Know how to
see versatile issues and prioritize what issues make sense to report, as each report creates a response
in the team * Know that best bug reports are code but it’s ok to learn skills one by one to get
to that level of reporting ability - being available is second best thing * Understand how change
impacts testing and guide testing by actual change in code bases in combination of constraints
communicated for that change * Write test automation for WebUI in Jest + Puppeteer and Robot
Framework and take part in team choice of going with one or the other * Operate APIs for controlling
data creation and API-based verifications using Java, Python and JavaScript. * Understand how their
testing and test automation sits in the context of environments it runs in: Jenkins, Docker and the
environment the app runs in: Docker, Kubernetes and CI-Staging-Prod for complex set of integrated
pieces * Communicate clearly the status of their testing and advocate for important fixes to support
‘zero bugs on product backlog’ goal in the team * Control their own balance of time to learning
vs. contributing that matches their personal style to not require task management but leading the
testing they do on their own * Have connections outside the company in the community to solve
problems in testing that are hard to figure out internally
We got started this week, are are one week into the experience. So far they have:

• Reported multiple issues they recognized are mostly usability and language. I jumped on the
problems with functionality and reported them, demoing those enforced the idea that they are
seeing only particular categories now.
• Navigated command line, filesystem, Git, and IDE in paired setting and shown they pick things
up from examples they experience, repeating similar moves a day later from learning the
concepts.
• Skipped reporting for a language bug and fixed it with PR instead.
• Covered release testing with a provided one-liner checklist for the team’s first release.
• Provided observations on their mentors (mine) models of how I train them, leading me to an
insight that I both work hard to navigate on higher level (telling what they should get done,
and only after digging into exactly how to do it if they already don’t know that) and respond
questions with questions to reinforce they already know some of the stuff.
Training and Exploratory Tester 80

• Taken selective courses from Test Automation University on keywords they pick up as I explain,
as well as reading tool-specific examples and guidelines.
• Explained to me how they currently model unit - service - UI tests and mixed language set the
team has.
• Presented a plan of what they will focus on achieving next week with Jest-Puppeteer 1st case
with our application.

After the week, I’m particularly happy to see the idea of self-management and you leading your own
work but radiating intent is catching up. Them recognizing they can’t see all types of bugs yet is
promising as is their approach to learning.
Every step, I prepare them for the world where I won’t be there to guide them but they know how
to pull in help when they need it - inside the company and outside.