THE UX INSIGHTS COMPANY
When Unmoderated Remote
Usability Testing is the Best Choice
Contents
In-lab Usability Testing and its value 4
Unmoderated Remote Usability Testing (URUT)
and its value 5
When would you use unmoderated remote
usability testing for your research? 7
To collect data quickly and efficiently 8
To quantify usability 9
To get the right participants 10
To conduct studies on a limited budget 11
To compare designs 12
To identify the areas of a website that need the most
improvement 13
To conduct competitive benchmark studies 14
To test users in their natural context 15
To understand users’ behavior 16
To validate or define lab-based research 17
To test internationally without travelling 17
2
Unmoderated remote usability testing (URUT) has grown a lot in recent years and has become a choice of many UX
professionals. Despite the cumbersome name, URUT is a low cost and effective technique for collecting task based
usability data.
In the 2009 UPA Survey, which gathers salary and other job-related information from UPA members, 18% of
respondents reported using unmoderated remote testing. The results showed the growth of this research method of
around 28% (and it wasn’t even listed as a research method in 2007).
According to the recent 2011 UPA Survey, the number of UX Professionals who use URUT has increased to 23%.
Thus, unmoderated remote usability testing is here to stay. The proof is in the increased interest from the UX
community and the explosion of remote usability tools.
A few other highlights from the 2011 UPA Survey:
82% 52% 50% 23%
of UX Professionals use SOME sort of are conducting are conducting are conducting
usability testing (lab, unmoderated, testing in a lab remote, moderated remote, unmoderated
moderated, heuristic, etc.) testing testing
Until recently, when we talked about usability testing, we meant labs with a
team of researchers observing users completing predesigned tasks. Even
though lab usability testing is a great research method, it’s not always
feasible. Unmoderated remote
usability testing and in-lab
What if you need to conduct a usability testing with people from different usability testing are NOT
countries? Or, what if you need to quantify your research? In such cases, to replace one another,
remote usability testing is a more viable option. but rather, to meet a
specific need and
To understand the difference between the two methods and the value in complement each other.
adding URUT to your research methods toolkit, let’s first break apart each
method at a high level and understand the value each brings to the table.
3
In-lab Usability Testing
In a traditional lab-based research study, between 6-10 participants (varying according to needs and perspective) are
brought into a “lab” environment to run through a series of tasks. Participants work on a pre-configured PC or Mac in a
pre-configured environment, while being observed in a separate room either via a monitor or through a one-way mirror.
During the study, participants are given tasks and asked to perform them with a researcher sitting next to them or in the
other room. If using a think aloud protocol, participants are asked to express their thoughts out loud and the researcher
can probe or ask further questions while the participants are walking through their task and after. Alternatively, in order
to gauge the time spent on the task, participants can walk through their task with no interruptions, and questions can be
left for probing after the task or after the study. There are variations to this method; however, we are defining the
method in its most traditionally used form.
What’s the value?
The value in this method comes from being able to probe
users while they are walking through their tasks, gathering
visual cues that include facial expression and body language,
providing assists to stumped participants, and being able to
change your question set or tasks mid-way through your
research study. There are many other benefits depending on
the variations of the method.
4
Unmoderated Remote Usability
Testing (URUT)
In URUT, hundreds of participants simultaneously participate in a study from their own computer and in their own
environment. During the study, participants are provided tasks by the URUT tool and are asked to walk through the
tasks as they normally would. The participants are then asked to provide feedback after the task completion via likert
scale, open-ended, multiple-choice and single-choice questions (just to name a few). Participants act and respond
naturally as they are participating in their own time and when it is convenient for them. Participants are given the tasks
in the same format and in the same manner - virtually eliminating moderator bias.
What’s the value?
The value in this method is the ability to have participants participate from their natural context, to have a
cross-representation of population across the country or internationally, and being able to gather statistically significant
data. Additionally, URUT is a very cost-effective research method.
5
Value to a researcher
Ŗ Unmoderated remote usability testing tools aggregate usability metrics that can be downloaded to a variety of file
formats, such as PowerPoint, Word, Excel etc.
Ŗ Get feedback on short tasks or small changes that may be an overkill to test in a lab.
Ŗ Collect quantititave data to convince the executive staff.
These are just a few of the important benefits; there are many more according to the specific type of URUT that you are
undertaking.
In outlining the differences and values in each method, you can see that one is not supposed to replace the other.
Rather, each method meets a specific need and complements the other. Now that we understand the differences
between and the value in each method, we are ready to explore why and when to use URUT to gain the most value and
benefit for your research.
6
So, when would you use
unmoderated remote usability
testing for your research?
To collect data quickly and efficiently
To quantify usability
To get the right participants
To conduct studies on a limited budget
To compare designs
To identify the areas of a website that need the most improvement
To conduct competitive benchmark studies
To test users in their natural context
To understand user behavior
To validate or define your lab-based research
To test internationally without travelling
7
1. To collect data quickly and efficiently
In an ideal world, you have weeks of time to prepare for your research and to recruit the participants. However,
in reality, this is not always the case. Sometimes design teams want to get feedback on the prototypes they are
working on in a matter of days.
With a lab study, you need at least a couple of weeks to prepare. It takes time to find the right participants who
can come to your location. In addition, you need to spend a couple of days in the lab.
With remote usability testing, you can set up a study within a couple of hours. Once the study is ready, all you
need to do is to send a web-link to your participants.
This research method is especially valuable when you work on an agile timeline. Scrum/agile teams use this
method because researchers can obtain feedback from a sample of participants, pause the study, analyze the
results, iterate the designs, resume the study and repeat the process until the team is satisfied with the design.
8
2. To quantify usability
Usability professionals generally agree that for usability testing, you don’t
need large samples of users to identify most design flaws. However, this
is not the case when it comes to measuring, benchmarking, and using
metrics and key performance indicators to make solid strategic decisions.
With URUT, you can test large samples of geographically dispersed
audiences. While users perform tasks, the software collects several kinds
of UX data, including effectiveness ratios (the percentage of users able to
complete the tasks successfully), efficiency ratios (time, number of clicks,
and problems encountered), and satisfaction ratios (how users feel about
the interactive experience). On top of these measures, you can collect
click streams, the navigation paths chosen to complete tasks, and
heatmaps.
After hearing a qualitative usability study presentation from the marketing team of one major international
airline, the marketing director said, “We just can’t make this kind of decision [redesigning the flight search and
check-out process] based exclusively on research with just ten users. We’d need to combine it with
quantitative data and test it with at least 250 users or so.” He also added, “I’d like to see some industry
benchmarks, both for the current website design and for the future version we’ll be launching. So for this, I’d
need to quantify usability and see if we are doing good or bad compared to the industry standards.” He did
not feel comfortable going up to the CEO to request a redesign budget based on evidence from a small
sample of users.
9
3. To get the right participants
Imagine the following scenario: you need to recruit 100 people who own a parrot, who buy parrot food online,
and who have never visited your website before. Remote usability testing enables you to recruit anyone who
has access to the Internet. It is easy to recruit 6 – 12 people (varying according to your needs) for a lab study;
however, when you need to recruit large numbers of participants, it is a lot easier to do it online.
When recruiting participants online, you have the following options:
Ŗ Intercept real visitors from your site
Ŗ Email an invite link to your private mailing list
Ŗ Recruit from a panel
Ŗ Post a link to your study on social media sites
10
4. To conduct studies on
a limited budget
FĹěĬ±ÆƉ ƋåŸƋĜĹčƉ ĜŸƉ åƻŞåĹŸĜƴåţƉ {±ųƋĜÏĜޱĹƋŸƉ ±ųåƉ ƋƼŞĜϱĬĬƼƉ ޱĜÚƉ ±ƋƉ Ĭ屟ƋƉ âŎLjLjƉ üŅųƉ ƚĹÚåųƋ±ĩĜĹčƉ ƋĘåƉ ŸƋƚÚƼţƉ ĬŸŅØƉ Ĭ±ÆƉ
moderators need to be paid for several hours of work.
With remote usability testing, you can test at a much lower cost. There is no need to travel and to oversee the
participants, and user reimbursements are typically much lower.
Additionally, multiple research projects can be conducted simultaneously by one researcher versus needing a
team of researchers.
Ŗ%QUV'HHKEKGPE[
Ŗ Unmoderated testing can cost 50% less than traditional moderated
testing (primiraly due to reduction in effort hours spent in a lab)
One of UserZoom’s
customers presented a Ŗ6KOGNKPG
“URUT vs. Lab Testing” Ŗ Reduced the 3 week lead time for recruiting to less than 1 week
case study on the “How
to Conduct Remote Ŗ8CNKFCVKQP6GUVKPI
Usability Testing” Ŗ Retest within days, instead of weeks
webinar. Main points Ŗ Moderated test: 180 hours vs. Unmoderated test: 40 hours
highlighted in the case Ŗ Remote Unmoderated Testing performed for 20% the cost of benchmark
study were: ÎƉåŸƋƉŎƉŠaŅÚåų±ƋåÚšƉ÷ƉâŎLjØLjLjLj
ÎƉåŸƋƉƖƉŠĹĵŅÚåų±ƋåÚšƉ ÷ƉâƖLjLjLj
Ŗ+ORTQXGF%QPHKFGPEG+PVGTXCNU
11
5. To compare designs
When you have more than one design alternative, you should let users decide which one is better. Sometimes
you might think that the choice is obvious, but users might not agree. It is hard to argue with hundreds of
participants who clearly indicate which design they like better.
You can also test wireframes and protototypes using URUT. It is important to test your prototypes with real
users, as it is the point in time when critical design decisions set the stage for all that will follow. Most likely, you
will want to do all the testing quickly and cost-‐effectively at this stage, and URUT gives you that flexibility.
12
6. To identify the areas of a website
that need the most improvement
Task success rates and completion times will often reveal the areas of
a website that visitors have the most difficulty with.
A traditional lab study can also help identify issues with a website.
However, with a small number of participants, it is hard to know the
magnitude of the problem.
Competitive user experience benchmarking is the process of
comparing your website’s performance to that of your competitors’
websites. In user experience benchmarking, a website’s performance is
judged using various user experience metrics such as efficiency,
effectiveness, and satisfaction. The practice of user experience
benchmarking is excellent for determining how your website stacks up
against the competition.
By utilizing statistically significant quantitative data, which can be easily
achieved with remote usability testing, you can determine the relative
quality of your website and the areas for improvement.
You can also conduct benchmark studies to obtain statistically
significant usability metrics on how a website performs
vs. the other versions of the site.
13
7. To conduct competitive
benchmark studies
Competitive user experience
benchmarking is the process of
comparing your website’s performance
to that of your competitors’ websites. In
user experience benchmarking, a
website’s performance is judged using your company
various user experience metrics such as competitor 1
competitor 2
efficiency, effectiveness, and satisfaction. competitor 3
The practice of user experience
benchmarking is excellent for
determining how your website stacks up
against the competition.
By utilizing statistically significant
quantitative data, which can be easily Look & Feel Organization Clarity Brand Perception
achieved with remote usability testing,
you can determine the relative quality of
your website and the areas for
improvement.
You can also conduct benchmark studies
to obtain statistically significant usability
metrics on how a website performs vs.
the other versions of the site.
14
8. To test users in
their natural context
Testing participants in their natural context accounts for
different systems, configurations, and setups. Since
participants are not being observed, the data you gather
not only accounts for a mix of these various
environments and setups, but also encourages
participants to act as they normally would.
When participants are in a lab, they have to complete
the study on a preconfigured computer. This computer
and its configuration can be totally different from what
they have at home.
15
9. To understand
user behavior
You want to understand why users come to
your site and what they do once they come
there. Remote usability testing uses a
combination of web analytics (where users
go) and surveys (why they go there) to
create a complete picture of the user
behavior. This data is valuable for you in
designing the best user experience for your
site.
Figure 1. Heatmap
Figure 2. Clickstream
16
10. To validate or define
your lab-based research
Have you ever presented findings from a lab usability test only to receive push back on the data, because only
5–10 people participated in your study? Sometimes, you need greater numbers to give stakeholders the
warm-and-fuzzy feeling they need to make million-dollar design decisions. URUT allows you to ensure that the
research you are currently conducting is valid and is a true representation of reality.
11. To test internationally
without travelling
International research is very expensive and at times put
aside due to the cost and the time commitment. URUT
allows you the flexibility to conduct a study in many
international locations without taking a step out of your
home. It removes the barriers that have traditionally
impeded this very critical research.
To summarize, user experience design is critical in today’s
highly competitive online marketplace. This is driving user
experience designers, researchers, and marketers to focus
on the customer and their online experience. There is a
need to gather more concrete statistical and geographical
data which can be readily obtained through remote
research methods.
17
About UserZoom
UserZoom is the #1 platform for cost-effective and agile usability testing and user experience analytics. We offer a
worry-free, all-in-one solution: software + services + recruitment.
1 Software Platform 2 UX Research Services
Unmoderated Remote Usability Our UX Research Team
Testing ensures your success:
Live Web & Mobile Task-based
Surveys
for
m 1 UX
1. Strategizing
2. Initial training & support
UX Benchmarking
at 3. Setting up studies
Competitive Research Pl 4. Managing all recruiting tasks
Re
re
International Research 5. Launching, monitoring
se
wa
arc
6. Analyzing results, reporting
Soft
h Services
3
Information Architecture &
Design Validation Recruiting Solutions
Card Sorting THE UX INSIGHTS COMPANY
Tree Testing We recruit in 3 ways:
Click Testing 1. Using a Panel
Prototype Testing
3 2 Strict quality controls
National and international reach
CX Measurement / VOC
Re 2. Customized invitation layer
c ru Intercept studies
itin g
Live Web & Mobile True
Intent Studies (Intercepts) S o l u ti o n s 3. Invitation links
Online Surveys Private email list
Web Analytics Integration Social media
The best way to understand the power of UserZoom is to see it for yourself
Request a Free Demo 18
THE UX INSIGHTS COMPANY
www.userzoom.com
San Jose, CA (USA) Madrid (Spain) London (UK) München (DE)
10 Almaden Blvd #250, C/ O’Donnell 12, 2ª planta, Thomas House, Kronstadterstrasse 4,
San Jose, CA 95113 28009, Madrid 81677 München
Phone: 866-599-1550 Phone: +(34) 91 111 48 82 84 Eccleston Square Phone: +49 (0)89 45 35 35 25
London, SW1VPX
Phone: 0203 948 1360
Denver, CO (USA) Barcelona (Spain) Manchester (UK)
1630 Welton Street #500C Av. Diagonal, 618 3ºD 4-6 Bank Square Wilmslow,
Denver, CO 80202 E08021 Barcelona Cheshire SK9 1AN
Phone: +34 93 414 7554 Phone: + 44 (0) 1625 525 650
