Dscout Usability Testing Guide Ebook Final
Dscout Usability Testing Guide Ebook Final
Usability
TABLE OF CONTENTS
3 Usability: An introduction
For example, with a clothing company, a user would go to the website with the
expectation of being able to purchase clothing. But just because a user can buy clothing,
doesn’t mean it’s an easy or satisfactory experience.
1. Effectiveness:
Whether a user can accurately complete tasks and an overarching goal.
2. Efficiency:
How much effort and time it takes for the user to complete tasks,
and an overarching goal accurately.
3. Satisfaction:
How comfortable and satisfied a user is with completing the tasks and goal.
Usability testing, whether you use metrics or run a qualitative usability test, looks at these
three factors to determine whether or not a product or service is usable.
Again, one of the main goals of a clothing website would be to purchase clothing.
You can have smaller goals within that larger goal, such as comparing clothing.
Then, you can break this down into the tasks people must do to achieve those goals.
For instance:
These are all tasks associated with the larger goal. With usability testing, we ask people
to do these critical tasks to assess whether or not they can achieve them and the larger
goal in an efficient, effective, and satisfactory way.
If someone can do these tasks, they can get to their expected outcome. However,
if efficiency, effectiveness, or satisfaction suffer during this process, they may get
frustrated and give up or go to a different website.
We’ve all encountered this—an infuriating user experience that made us rage-click,
throw our phones (against a soft surface, of course), and give up on a product or service.
• Whether or not a user can use a product for its intended function
• Whether or not a product allows a user to reach their goals
• How a user uses a product, separate from how we think a user should use a product
• How a product functions when placed in front of a human
• Where bugs and complicated user experiences lie within a product
Now, let’s dive into some goals for usability testing because the best thing you can do is
start with goals and a research plan.
plan
• Learn about people’s current pain points, frustrations, and barriers about
[current process/current tools] and how they would improve it
• Uncover the current tools people use to [achieve goal] and their experience with
those tools. Uncover how they would improve those tools
• OR Evaluate how people are currently interacting with a [product/ website/
app/service]
For usability tests, ask yourself the following questions, with some examples from above,
to understand who your target users would be:
One note about the sample size for usability testing: a general idea for evaluative
research is testing with five people. While this could be correct, it isn’t a hard and
fast rule.
For the shopping example above, if you pick five random people off the street to
test your product, you likely won’t find 85% of the issues. The fine print behind “five
people per usability study” is that it means five users per segment.
Concept testing
• Moderated: Recruit at least eight participants per segment
• Unmoderated: Recruit at least 15 participants per segment, in case you get messy data
Card sorting
• Recruit about 20-30 participants per segment
Benchmarking
• Recruit 25 or more participants per segment, since we’re looking at quantitative data
If you plan to create a rolling usability program, for example, consistency across outputs
might necessitate a panel sampling method. If, on the other hand, you are managing
many requests from multiple stakeholder teams, honing in on the 3-5 key screening
criteria is key to keeping ahead.
Want to learn more about dscout’s recruiting options? See our platform in action.
Start with what you want the user’s end goal to be, not the goal of the task. What does
the user need (or want) to accomplish at the end of the action? What is their goal for
using this product?
Instead of throwing participants into action with no relevant information, give them
context on why they need to use the product. You can also consider the context and
background information for why they would use the product in the real world.
Since you’re recording metrics, you don’t want to be vague in your instructions. If users
need to input dates, locations, or particular data in a form, give them that information.
You don’t want the user to guess what you want them to do, resulting in skewed data.
If you’re trying to get someone to accomplish a task, make sure they can. There
should be a reachable “end,” which satisfies the participant and helps you record if the
participants could complete the task.
Once you’ve brainstormed this information, it’s time to write your task scenario. Don’t
shy away from creating questions that are open-ended, qualitative, or media-based.
You can use different data types to triangulate your findings, adding rigor to your
recommendations.
* Think aloud questions – Participants can articulate why they are making the choices
they are. This can be more accessible and inclusive for some participants who find typing
tedious. As a researcher, it also gives you a rich set of data to triangulate, especially if you
have access to screen recordings.
After writing down your task scenarios, it can be extremely beneficial to try a dry run
with internal employees, friends, family, etc. This will allow you to practice the test flow,
make sure the tasks make sense, and indicate whether there are too many.
In terms of how many tasks should be in one usability test, it depends on the complexity
of the tasks and how much time you have with the participant.
Pro tip: For a 45-60 minute session, five to seven tasks are an appropriate amount.
By including a dry run in your process, you can know how many tasks you can fit into
the session.
A potential task example for the above clothing company might look like this:
Winter is coming up, and you’re looking for a new winter coat to keep you warm that is
under $150.
Learn how dscout makes task writing and follow up quick and easy with Express.
Before we dive into the actual metrics, keep in mind the three cornerstones of usability,
because the metrics you collect will be measuring these things:
Effectiveness: Whether or not a user can accurately complete a task that allows them
to achieve their goals. Can a user complete a task? Can a user complete a task without
making errors?
Efficiency: The amount of cognitive resources it takes for a user to complete tasks. How
long does it take a user to complete a task? Do users have to expend a lot of mental
energy when completing a task?
Effectiveness
• Task Success: This simple metric tells you if a user can complete a given task
(0=Fail, 1=Pass). You can get fancier with this by assigning more numbers that
denote the difficulty users had with the task, but you need to determine the levels
with your team before the study.
• Number of Errors: This task gives you the number of errors a user committed while
trying to complete a task. You can also gain insight into common mistakes users run
into while attempting to complete the task. If any of your users seem to want to
complete a task differently, a common trend of errors may occur.
• Single Ease Question (SEQ): The SEQ is one question (on a seven-point scale)
measuring the participant’s perceived task ease. Ask the SEQ after each completed
(or failed) task.
Efficiency
• Time on Task: This metric measures how long it takes participants to complete
or fail a given task. This metric can give you a few different options to report on,
where you can provide the data on average task completion time, average task
failure time, or overall average task time (of both completed and failed tasks)
• Subjective Mental Effort Question (SMEQ): The SMEQ allows the users to rate
how mentally tricky a task was to complete.
Satisfaction
• System Usability Scale (SUS): The SUS has become an industry standard and
measures the perceived usability of user experience. Because of its popularity,
you can reference published statistics (for example, the average SUS score is 68).
Overarching metrics
• SUM: This measurement will enable you to take completion rates, ease, and time
on task and combine it into a single metric to describe the usability and experience
of a task.
This can proceed in the other direction, too, where a few repeated examples of friction or
delight points might create the opportunity to trend-spot with a larger sample.
Here, going from a 15-person usability test to a 1500-person survey adds confidence you
can weave into your recommendations, especially for future-proofing the product.
• Schedule 1-1 interviews with a few of your most engaged participants to better
understand their feedback and gather additional insights.
• Ask a handful of your most engaged participants to join a series of studies to
understand their product usage habits over time.
• To gut-check a potential trend, conduct another usability test with a wider audience.
After each usability session, it can help to do a quick debrief and split the global tags into
four quadrants and note what happened during that interview concerning each quadrant.
For example, if you were usability testing the clothing website, a session debrief (one
participant) might look like this:
Goals
• Purchase a new weatherproof winter jacket to keep warm during snowstorms and in
below-freezing weather.
• Get the highest quality for the lowest price by comparing different products on
the website.
Needs
• Weatherproof and durable jacket, knowing this is the case through a description
• Lasts for more than five years
• Warm enough for below-freezing weather
• Under $250
Pain points
• Not knowing if a coat is waterproof or weatherproof for the necessary conditions
• Understanding fit and size with additional layers
• Knowing the length of the coat
• Understanding other peoples’ experiences with the coat in similar situations
Completing this small synthesis session makes for easier work at the end of the study.
Once you complete all the sessions and each debrief, bring all participants together to
assess patterns and trends (the same thing three or more people are saying).
Streamline your analysis process and quickly view completion, time on task,
average ease rating and other metrics with dscout. Learn more.
more
A stoplight report:
• Conveys whether or not a user has passed a usability testing task
• Includes how severe the problem is in a task
• Shows the amount of time spent on a task, by task and participant
• Highlights, on average, how many participants passed/failed a given task
• Summarizes how the usability test went through visuals
The most valuable part of the stoplight approach is how visual it is. It can quickly provide
a stakeholder with a holistic overview of how the usability test went.
A stoplight chart includes the following components (but you don’t have to include all of them!):
• Each participant has a column, with a participant summary at the bottom
• Each task has a row, with an average task summary
• The three colors indicate:
• Whether a participant succeeded (green)
• Whether a participant struggled with a task (orange)
• Whether a participant failed the task (red)
The time for each task is recorded within the task participant bubble and averaged per
task. To try it out yourself, grab a copy of our Usability Quant Testing Template.
Template
Now that you’ve got the basics on conducting and analyzing a usability test, we’ll go
through a handy usability testing checklist and offer specific use cases showing how
other prominent companies have approached this method.
The challenge:
People use Dropbox in browser to host and share files—relegating it to more passive,
background usage. So when the company began the development of their new desktop
app, they strived to create a single workspace for users to organize their content,
connect their tools, and bring their teammates together.
But when you build a product for more flexible and frequent usage—you have to be sure
that it’ll work as intended. And to be confident that it’ll work as intended, the insights that
you need are often more extensive than what you could glean from a typical usability test.
“I looked at dscout because we wanted to do a longitudinal study,” she says. “It was the
first time we had people using the product outside of interviewing and concept testing.
So we really wanted to get a sense for their day-to-day: How are they interacting with
this app, and what are their attitudes towards it?”
“We were hoping to get some in-context feedback; we needed participants to submit
surveys in the moment they were doing things. That made a major impact in our attempts
to understand what the real issues were.”
“We kept it pretty open-ended in the beginning,” Meghan says, “We didn’t want people
to feel like they were doing something right or wrong. We really just wanted to be a fly
on the wall and understand what was going on.”
The second half of the study was more straightforward. Participants were sent specific
survey-like questions asking about different parts of the product. And as the study
concluded, Meghan conducted a Live mission—pulling specific users in for 1:1 interviews
about their experience.
“The interviews were nice because we already had so much context,” Meghan says.
“They were really efficient. You can get straight to the heart of things after having heard
from this person on dscout for the past four weeks.”
This allowed Dropbox to address those issues, and address areas in which they could
really impact their users’ needs.
“We identified some key problems pertaining to the new functionality that we’re
adding. We’re adding features to help people collaborate and work with each other
better. And there were some pretty key blockers to people being able to do that.
“So identifying that has informed our design direction and understanding what we need
to do in order to help people collaborate more in Dropbox.”
Dive deeper into the study and see a few of Meghan’s tips for ensuring a successful
project in our Field Report.
READ MORE
“It was great that I was able to share the results from the analyze page
with someone else on my team. Huge time-saver! Otherwise, I spend
a lot of time taking the results and re-summarizing it somewhere else.”
Mary Mascari, UX Researcher at a Leading U.S. Airline