Exploratory Testing Manual

A basic overview of the test IO workflow:

● TestGuide
● Test Types
● Features
● Exploratory Test Setup
● Bug Severities
● Bug Review
● Information Request and Comments

Hundreds of global brands trust test IO with their QA:

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Introducing TestGuide
TestGuide is test IO’s proven method for implementing
crowdtesting fluidly into development workflows
test IO’s TestGuide is a methodology for integrating crowdtesting into a
team's software release process to create optimal business results. It
grew organically from best practices and data derived from hundreds of
successful engagements with customers. Using TestGuide, our Customer
Success Team creates an implementation plan and roadmap for your
evolving testing efforts. Since every product and team is unique — QA
setup, development workflow, degree of automation, available personnel,
and speed of changes — we help you understand how crowdtesting can
best fit and adapt to the specific needs and characteristics of your team.

After you've satisfied basic testing needs, you may want to run more
advanced tests, like persona or interaction-based testing. This is
where we use TestGuide to help us evolve to those changing needs.
We’re not just in the business of finding bugs; we’re your expert advisors
for optimizing the efficiency and efficacy of your entire human-driven
software testing operation.

The right type of testing for you

test IO offers different types of tests for different situations. But software
teams under pressure to ship don’t always run the most appropriate test
prior to release, or they simply may not be aware of the optimal test to run
in a given situation. We can help to specify and monitor testing in temporary,
staging, or production environments as needed. Maybe you’re focused on a
specific user flow, such as payments, with the goal of ensuring a successful
customer checkout, or maybe you’re looking to achieve interaction testing
between apps, where users test on two different apps interacting at the same
time; we can guide you to find the best fit.

Optimal testing cadence

Which test to run is only the first part of the equation; when to run it is just
as important. Matching your testing and release rhythm has proven hugely
successful for many customers. We can help you find the best times to test
based upon your particular needs, whether that be over the weekend so that
your team can triage bugs on Monday before a recurring production release
TestGuide

on Tuesdays, or a scheduled sanity test in production to make sure

repeated pushes from a fast-moving team haven’t caused a problem that
slipped through your automated checks? What if your development team
does a pull request and needs fresh eyes on the product within the same
day? Or, maybe you simply want weekly, biweekly, and monthly “feel-good”
tests that keep your team abreast of issues in a branch of your code that’s
not ready for release?

Which devices to choose

We don’t want you to have to rethink your workflow when partnering with us,
so we work around yours. We integrate into the tools you’re most
comfortable with, such as Jira and Slack, so that you don’t miss a beat. We
help you find the blind spots between the various forms of automated and
internal testing you already have in place. Moreover, this integration extends
past tools to the organization, working with internal teams to grow and scale
their own efforts, supporting internal leads with upward trajectories.

How?
1. First, our integration experts analyze the status quo of your development
and testing processes: existing development workflow, applied
development methodologies, tools in place, and team structure.

2. Next, working alongside you and your team, we help define and refine
your development objectives for every product you have to see how
crowdtesting can best support your specific needs.

3. Then, based on collected data points and reformulated release objectives,

our integration experts develop a customized strategy that makes it easy
for you to integrate crowdtesting into your workflow. The strategy entails
recommendations on which test types to run, what your testing cadence
should be, which devices to test on, how to integrate into your existing
tools, and how to modify your testing processes over time.

Which tests Optimal testing Which devices How to integrate

to run cadence to choose into existing
tools
Test Types
test IO offers several types of functional tests, as well as usability testing, all
carried out by carefully selected and trained human testers. These tests are
designed to get you the kinds of results you need, in the time frame that you
need them in.

Rapid Test
Rapid tests are fast and can finish in as soon as 2 hours. They’re
designed to catch high-priority bugs. When should you run a rapid
test? They’re perfect for when you’re about to commit a new build
or make a pull request. Rapid tests give you the peace of mind that
your app does what it’s supposed to and that it works on
thedevices it’s been tested on. When you run a rapid test, you’re
looking for a green light or a “sanity test.”

Focused Test
When performing a focused test, testers delve deep into a specific
feature or a section of your app. It’s the equivalent of putting a part
of your software under the microscope. Many teams choose “to run
focused tests when they’ve developed a new feature. It’s the way to
find bugs big and small in new functionality, including edge cases
no one has considered yet. When you run a focused test, you’re
looking for all the potential problems in a segment of your
software.

Coverage Test
The coverage test is deals with one of the main challenges of
software development: whether your app will work on phones
and tablets of varying screen sizes, different browsers, on multiple
versions of iOS, or on different flavors of Android. You can specify
what types of devices to test on and multiple product areas. It’s a
great way to make sure that your software will work on a variety
of real devices before release or after major changes. In particular,
it comes in handy when most of your developers work on a single
platform like Chrome or the latest version of iOS.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Test Types

Usability Test
From early-stage feedback to pre-release testing to finding usability
issues in production, this test gets you feedback from diverse group
of usability testers. For better results and more useful suggestions,
we recommend adding user stories or scenarios, though you can
also request an evaluation of individual Feature or of the overall
user experience.

Custom Test
This is the most versatile and broad type of test that gives you
a lot of room for any custom requirements and allows you to
precisely define the test scope exactly the way you designed it.
However, if you have an “unconventional” test setup in mind
and you are not sure how to go about it, please contact your
CSM to work out a solution.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Features: Best Practice
Creating a new product
A product is basically a unit that is going to be tested repeatedly in the
following test cycles. It may be a whole website, an Android or iOS App.
For big platforms with many QA and development teams involved it makes
sense to break the environment down to several units (products) depending
on how many teams are involved.

To structure and facilitate the testing process a product needs to be broken

down into features (flows).

Managing Features
A well written Feature can yield high quality results for your tests. All the
bugs found will be assigned to a respective Feature - a well defined Feature
structure may save you some time when reviewing the test results.

When describing how to find the part of the app, explain as if describing it
to someone who has never seen the product before. The Feature description
is the only way for testers to know how to go about the problem and what
issues may be relevant for you, especially when it comes to non-obvious
functionalities, where a certain level of user adoption is needed.

Features may include

● where a given feature can be found
● what is the expected behavior/user flow to be tested
● what is out of scope within this feature/user flow
● known issues
● acceptance criteria and other relevant information

Different sections don’t have to be 1:1 the described Features of an app.

They can also be areas that are present in different processes or “parts”
of the product. For example, sharing, filtering, or search might be present
in different parts of the software, but all share a common code base.

You can also use Features to define user-scenarios or flows to be tested if

that is your preferred approach. Including your internal acceptance criteria
(if available) might be a way to structure your test setup.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Features Best Practices

General recommendations on how to set up Features

● Preferably, one Feature should be broad enough to allow you to run a test
with a single Feature in scope.
● Features should be reusable for the future test cycles within a given
product. However, you can always edit a Feature on a test level therefore
the Feature content shouldn’t always be final.

Editing Features
● Editing Features when reviewing the created test cycle will not affect the
existing Features on the dashboard.
● Updating your existing Features on the dashboard will also not have any
impact on past or already running tests, nor their duplicates.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Steps: Exploratory Test Setup
Step 1: Test Environment
● Choose the test environment that you want the testers to evaluate. In
the drop-down you can see any previously tested builds or you can create
a new one by clicking add new. Name your test environment something
easy to remember, like “Beta 3.2” or “Staging”.
● Choose the type of environment, either the URL of a website/app or upload
an .IPA or .APK file. Also be sure to enter credentials to access
the environment if needed, including use of a proxy server.

For additional information security access requirements to non-public

environment, please contact your CSM to discuss other solutions, including
setup of a VPN tunnel from test IO servers.

Step 2: Features
● Pick the Features you want to get covered in this test cycle.. Depending
on the Test Type, you may have the option to select a variety of bug types
in scope for this test, as well as a different amount of features.
At least one feature should be in scope to be able to run a test. For Rapid
and Focused tests there is a limit of 4 features per cycle.

Editing Features
Editing feature content when creating a new test will not affect the global
list of features which can be managed vie “Manage Features” in the panel
on the left.

The changes you’ve made will only be saved for this given test cycle and
its duplicates.

Step 3: Instructions
Though its name might be misleading, this section can make the difference
between a good and a great test cycle!

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Steps: Exploratory Test Setup

Goal of this test

This section should be the north star for our testers. Think about the
desired results of the test and what information the tester needs in order
to discover these types of issues. Common examples include release of
a new feature, a sanity test for “Critical functional issues only”, changes
to a common workflow, etc. It’s also common to include a brief description
of your company and product, so the tester has some context about the
software they’re testing. In other words, keep in mind that tests instructions
is the only way for you to communicate your needs and expectations to
a crowd of testers who might have never had experience with your
site/platform/app before.

Out of scope
This section should explicitly state any features or functionality out of
scope for this test. It’s also common to state:
● undesired functional severities or bug types in this section
● informing the tester not to take particular steps in the flow (make final
purchases, contact support, send out applications etc.)
● Known issues in a given Feature

Additional requirements
This section should be used to provide any additional information the tester
should know to achieve the desired results. Examples include:
● Test accounts
● Test payment information
● Any special requirements to the environment access / bug report format /
device usage / access settings

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Steps: Exploratory Test Setup

Step 4: Attachments
Consult with your CSM for applicable scenarios. A common use case is
if screenshots should be provided to the tester as a reference for testing,
be it a module, newsletter or page layout.

Any test guidelines, access settings, user stories or additional details,

as well as payment data can be attached to the instructions for testers
to download.

Step 5: Test Devices

You can define device scope by editing device type, device model, OS and/or
browsers. Customers typically leverage the crowd for device coverage alone.

Please keep in mind that any “unconventional” or outdated device, OS and

browser combination may lead to irrelevant device specific bugs.

Step 6: Bug Report Language

Bug Language is defaulted to English.

When choosing the bug language, please keep in mind, that testers get
invited based on the report language; therefore, the instructions/Features
and bug report languages should be consistent.

Step 7: Test Launch Date and Duration

Select your desired test date and start time. Note that due to our Team Lead
review and invitation process, the earliest a test can start is the next full hour
+1 (e.g. current time: 9:40am / test time: 11:00am, current time: 5:05pm / test
time: 7:00pm). Duration ranges vary based on Test Type, from 2 hours (Rapid
Test) up to 48 hours.

Step 8: Review
The final step in the process is to review all test information and make any
adjustments as needed. Once all information has been confirm you can
Submit the Test.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Steps: Exploratory Test Setup

Editing the Test Cycle

To provide an opportunity for Team Leader review, the option to edit the
test it is not available once the test has been submitted. If you notice that
there is a mistake in the test setup, please reach out to your CSM or test IO
team, but please keep in mind that the test environment and instructions
can only be edited by a CSM within one hour before the test starts. After
the test has started, no further changes can be made. If you feel the test will
not yield appropriate results based on the current criteria, please contact
your CSM to cancel the test and work with you on resubmission.

Test Rejection
If the test cannot be accepted for any reason (problems with the test
environment, inconsistent instructions, etc), you will get notified via
email. Please correct or provide clarity based on the test rejection reason
included in the email then re-submit the test.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Cheat Sheet: Bug Severities
Different test types allow including different severity levels in the test scope.
Sometimes the bug severities you define internally may vary from our
guidelines and how testers access the issue severity.* To unify the approach
here’s how we classify bugs and issues.

Functional Bugs
Critical Preventing a function of the app or website,
causes a potential loss of income for the company
running the app or website - e.g. an app crash or
‘not able to login.’

High Serious impact on user experience, but doesn’t

prevent the function of the app or website.

Low Minimal impact on user experience.

Non-Functional Bugs

Usability suggestions Improvements to existing features

and functions that would make the product easier and
more intuitive to use.

Visual The user can accomplish a task, but the interface

looks wrong, typically because of responsive design, CSS,
HTML, or layout framework problems.

Content bugs relate to missing data, images, or broken

links.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Cheat Sheet: Bug Review
Bug Review
After a test cycle is completed all the logged bugs should be reviewed -
in other words -- accepted and exported, or rejected. This is a necessary
step for you to filter out the bugs you want to fix as soon as possible and at
the same time it provides us valuable data on what kind of issues are most
relevant for you so we can tailor the test setup according to your needs.

These are the options available when reviewing the bugs:

● accept the bug and export (optional)
● change the bug priority and export (optional)
● reject the bug

Regardless whether an issue is accepted or rejected there are

following options available:
● mark a bug as “known”
● change the priority
● ask the tester for additional information
● write comments - please keep in mind there is no dialogue function, the
comment function only serves as a “log” to export into your bugtracker

Known Bugs
If some of the bugs found in a test cycle are already known (for example,
from internal testing) they can be labeled as such in test IO interface to
prevent these bugs from being reported again in the following tests.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Cheat Sheet: Bug Review

Labeling Bugs as known

This option is available in the bug list view in the column known . Once a bug
is marked as known it is added to the known bug list and gets assigned to the
Feature it was submitted in - whenever this Feature gets tested again, all
related “known” bugs will be displayed to the testers.
● If the bug is marked as Known, it is visible to the testers and will not
be reported in subsequent tests.
● Unchecking the box removes the bug from the list, so the crowd can
report it again should the issue re-appear.

Known Bug List and Navigation

● A Global Known Bug List containing all Known Bugs for a given Product
can be quickly accessed via the main Product navigation.
● The Known Bugs List tab under a given test cycle now only contains
Known Bugs for that test, simplifying Known Bug management within
a single test run.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Information Request vs.
Bug Comments
Information Request
Although every submitted bug is carefully reviewed by a Team Leader before
being forwarded to you there is a chance you might need an extra piece of
information from a tester: browser version, Session ID, UDID, etc.

Tobe able to get in touch with a tester please only use the Information
Request Feature which you can find on the bar below in the bug detail view:

Comment Function
Sometimes you are able to see the tester comment thread and yet
a dialogue does not start whenever you add a new comment. It
means, the tester or team leader will not be notified if you add a
bug comment. This feature serves for documentation purposes only,
which means bug comments, as well as attachments, get exported
into your bug tracking system.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859
Additional Questions?

This manual is only meant as a reference to familiarize you with the terms
and common workflows within test IO; the following channels should be
used for any questions that come up following a kickoff call with your
Customer Success Manager.

1. Check out our Customer Help Center

Most questions and common answers for support-related inquiries may be
found here.

2. Contact your Customer Success Manager

Your Customer Success Manager is your problem solver at test IO. With
knowledge of your specific use case, they are the best person to help
optimize your use of test IO. Leverage them for anything beyond what
our FAQs cover.

3. Email [email protected], or use in-app chat.

For emergencies and urgent requests, please use this channel to escalate
your request for assistance.

©2019 test IO | 535 Mission St. San Francisco CA, 94105 | test.io | (415) 937-6859