International Journal of Human–Computer Interaction

ISSN: 1044-7318 (Print) 1532-7590 (Online) Journal homepage: http://www.tandfonline.com/loi/hihc20

A Systematic and Generalizable Approach to the

Heuristic Evaluation of User Interfaces

David Alonso-Ríos, Eduardo Mosqueira-Rey & Vicente Moret-Bonillo

To cite this article: David Alonso-Ríos, Eduardo Mosqueira-Rey & Vicente Moret-Bonillo (2018): A
Systematic and Generalizable Approach to the Heuristic Evaluation of User Interfaces, International
Journal of Human–Computer Interaction, DOI: 10.1080/10447318.2018.1424101

To link to this article: https://doi.org/10.1080/10447318.2018.1424101

Published online: 24 Jan 2018.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=hihc20
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION
https://doi.org/10.1080/10447318.2018.1424101

A Systematic and Generalizable Approach to the Heuristic Evaluation of User

Interfaces
David Alonso-Ríos , Eduardo Mosqueira-Rey , and Vicente Moret-Bonillo
Department of Computer Science, Universidade da Coruña, A Coruña, Spain

ABSTRACT KEYWORDS
Heuristic evaluation is one of the most actively used techniques for analyzing usability, as it is quick and Case study; context-of-use
inexpensive. This technique is based on following a given set of heuristics, which are typically defined as taxonomy; Heuristic
broad rules of thumb. In this paper, we propose a systematic and generalizable approach to this type of evaluation; mobile apps;
evaluation based on using comprehensive taxonomies as a source for the heuristics. This approach Nielsen’s heuristics; usability
taxonomy
contrasts with other typical approaches, such as following (or adapting) Jakob Nielsen’s heuristics or
creating ad hoc heuristics (formally or informally). The usefulness of our approach is investigated in two
ways. Firstly, we carry out an actual heuristic evaluation of a mobile app in this manner, which we
describe in detail. Secondly, we compare our approach and Nielsen’s. Additionally, we identify some
limitations in Nielsen’s heuristics and some inconsistencies between them and established usability
models, including Nielsen’s own.

1. Introduction blemishes of their products and services to the public, prefer-

ring instead to keep them ‘in house’” (Lewis, 2014, p. 663).
Heuristic evaluation (Nielsen and Molich, 1990) is currently
Secondly, we compare our approach and Nielsen’s heuris-
one of the most actively used and researched (Hollingsed and
tics. Moreover, we also examine the relationship between
Novick, 2007) usability techniques. It consists in assessing
Nielsen’s heuristics and popular usability models, including
usability according to a set of heuristics (i.e., good practices)
Nielsen’s own. For example, Nielsen’s definition of usability
that are basically broad rules of thumb, as opposed to detailed
assumes that users need to memorize things, whereas his
usability guidelines. The main pioneer and most influential
heuristics demand intuitiveness and, only as a last resort,
author in this area is Jakob Nielsen, who has proposed a set of
guidance. Nielsen’s heuristics also neglect subjective satisfac-
ten usability heuristics (Nielsen, 1995) for user interface
tion, which is historically one of the main attributes of usabil-
design.
ity, and its subjective component. We argue that these
Instead of using Nielsen’s heuristics, the approach taken in
problems arise because the relationship between Nielsen’s
this paper is based on comprehensive and general-purpose
heuristics and the concept of usability is never made explicit.
taxonomies. The goal is to retain the usefulness, ease of use,
In contrast, we suggest that usability heuristics should be
and generalizability of heuristic evaluation while adding depth
strongly connected to a usability model.
and structure.
This connection with a usability model makes our
More specifically, the methodology is based on usability
approach more generalizable to other domains and ensures
and context-of-use taxonomies (Alonso-Ríos et al., 2009,
that we are in fact evaluating usability. This is opposed to the
Alonso-Ríos et al., 2010) that are “instantiated” with the
ad hoc approaches (formally or informally), which pose sev-
particular characteristics of the system under study. This
eral risks to the evaluation: redundancy, inconsistency, being
means that it is also necessary to decompose said system
too specific to a given context, and so on.
into its constituent parts. The target of our particular case
This paper is structured as follows: Section 2 explains the
study is a web application for mobile devices, so we have also
background about heuristic evaluation and usability models.
produced a tentative taxonomy of user interface elements that
Section 3 presents our methodology for the heuristic evalua-
can be used to describe any simple, HTML-based web
tion explaining the taxonomies for usability and the context of
interface.
use and also a taxonomy for the user interface elements of a
The usefulness of the methodology is investigated in two
web application used for the actual study. Section 4 includes a
ways. Firstly, in practical terms. We test the methodology by
description of the case study used to analyze the methodology
conducting an actual heuristic evaluation, which we describe
in practical terms, characterizing the system and its context of
in depth. This can also be useful to researchers in the sense
use and presenting the results of its heuristic evaluation.
that detailed descriptions of usability studies are still relatively
Section 5 analyzes the validity of our approach, comparing it
rare, as companies “are reluctant to expose the usability

CONTACT Eduardo Mosqueira-Rey [email protected]

Color versions of one or more of the figures in the article can be found online at www.tandfonline.com/hihc.
© 2018 Taylor & Francis Group, LLC
2 D. ALONSO-RÍOS ET AL.

with the one based on Nielsen’s heuristics and, finally, we end Other authors decided to adapt Nielsen’s heuristics to very
the paper commenting on the conclusions. specific fields. For example, Sutcliffe and Gault (Sutcliffe and
Gault, 2004) to virtual reality, Magoulas et al. (Magoulas et al.,
2003) to adaptive learning environments, Mankoff et al.
(Mankoff et al., 2003) to ambient displays, Levi and Conrad
2. Background (Levi and Conrad, 1996) to a World Wide Web prototype, or
2.1. Heuristic evaluation Bertini et al. (Bertini et al., 2006), Yáñez et al. (Yáñez Gómez
et al., 2014), and Al-Khalifa et al. (Al-Khalifa et al., 2016) to
Heuristic evaluation was first introduced by Nielsen and mobile devices.
Molich (Nielsen and Molich, 1990) in 1990. This technique However, simply adding new, ad hoc heuristics to old ones
is part of Nielsen’s “discount” usability philosophy (Nielsen, poses several risks: redundancy, inconsistency, and making
1994) and consists in evaluating usability based on heuristics, the new heuristics so context-specific that they are almost
which Nielsen defines as “broad rules of thumb and not guidelines.
specific usability guidelines” (Nielsen, 1995). This means Finally, some authors have proposed sets of heuristics that
that, on the one hand, heuristic evaluation is mostly informal, are completely unrelated to Nielsen’s, such as Baker et al. on
but, on the other hand, it has the advantage of being quick groupware (Baker et al., 2001), Desurvire et al. (Desurvire
and inexpensive. According to Dumas: “Nielsen’s work on et al., 2004) and Amaro et al. (Amaro et al., 2016) on
heuristic evaluation freed expert reviews from the burden of games, Mariage and Vanderdonckt (Mariage and
the hundreds of guidelines that characterized earlier inspec- Vanderdonckt, 2001) on electronic newspapers, or Silva
tions” (Dumas, 2007, p. 55). See, for comparison, the exhaus- et al. (Silva et al., 2014) on mobile applications targeting
tive list of guidelines for designing user interfaces proposed by older adults. De Lima et al. (De Lima Salgado et al., 2016)
Smith and Mosier (Smith and Mosier, 1986) in 1986. observed an important increase in the number of researchers
Nielsen and Molich initially compiled a list of nine usabil- and studies that apply heuristics for evaluation, in their case
ity heuristics for user interfaces, providing a brief description for mobile usability.
for each. This list was later revised to, firstly, include a new In general, many different sets of heuristics have been
heuristic about help and documentation and, secondly, to proposed by many researchers, but there is still little scientific
change the names of the heuristics while essentially retaining evidence on the preferability of one over the others in terms
their meanings and descriptions. of validity, thoroughness, effectiveness, efficiency, or inter-
Table 1 shows the correspondences between the initial evaluator reliability (Bastien and Scapin, 1995, Hvannberg
1990 version (Nielsen and Molich, 1990) and the 1995 version et al., 2007). Some authors suggested that heuristic evaluation
(Nielsen, 1995) that is currently available on the Nielsen works best when the results of several evaluators are aggre-
Norman Group website. gated, as these typically only find some of the actual usability
Nielsen’s heuristics have been successfully used by many problems (J. Nielsen, 1992). Other authors have proposed
researchers and practitioners. However, some authors found it utility inspection methods (Johannessen and Hornbæk,
necessary to modify them in some way. 2014) to complement heuristic evaluation and find more
For example, some authors decided to complement problems.
Nielsen’s list with other general-purpose heuristics. Zhang Another different approach to developing usability heuris-
et al. (Zhang et al., 2003) merged Nielsen’s heuristics with tics is following a methodology for creating them. Quiñones
Shneiderman’s “eight golden rules” (Shneiderman, 2009). and Rusu (Quiñones and Rusu, 2017) described such metho-
Muller et al. (Muller et al., 1998) proposed new heuristics of dology as a formal and systematic process that includes steps
their own to “support (rather than replace) human skills”, that are formally defined to create heuristics. In an extensive
“provide a pleasurable and respectful working experience”, study developed by these authors they also notice that most
“help the user to provide quality results to her or his clients”, studies do not specify whether the process followed is infor-
and “aid the user in protecting the privacy of all persons mal, formal, or if the authors used a methodology.
involved in her or his work”. Authors like Hvannberg et al. These authors found several methodologies in the litera-
(Hvannberg et al., 2012) have argued that a patchwork of ture (such as Rusu et al., 2011, Greunen et al., 2011,
heuristics might be more useful than a single list. Hermawati and Lawson, 2015, Lechner et al., 2013). They
are very dissimilar, in the sense that they include different
phases and processes to obtain the heuristics. But they more
Table 1. Jakob Nielsen’s heuristics.
or less follow a common scheme of collecting information,
Nielsen and Molich (1990) Nielsen (1995)
creating heuristics, applying them and, finally, validating their
Simple and natural dialogue Aesthetic and minimalist design
Speak the user’s language Match between system and the real world application. They also are mainly proposed to establish heur-
Minimize the user’s memory load Recognition rather than recall istics for specific domains. For example, Rusu et al. (Rusu
Be consistent Consistency and standards et al., 2011) applied their methodology to search for heuristics
Provide feedback Visibility of system status
Provide clearly marked exits User control and freedom in grid computing applications, interactive television, and
Provide shortcuts Flexibility and efficiency of use virtual worlds.
Good error messages Help users recognize, diagnose, and
recover from errors Apart from analyzing existing heuristics and using meth-
Prevent errors Error prevention odologies, Quiñones and Rusu (Quiñones and Rusu, 2017)
- Help and documentation identified also other sources for creating heuristics such as:
 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 3

literature reviews; usability problems; mixing processes; Following this approach some researchers constructed
guidelines, principles, or design recommendations; interviews; their own comprehensive usability models, trying to encom-
and theories. pass all the existing interpretations of the term. Lewis (Lewis,
2014) identified this trend as “expanded models of usability”,
referring to the work of authors like Bevan (Bevan, 2009),
2.2. Usability models
Seffah et al. (Seffah et al., 2006), Winter et al. (Winter et al.,
Concurrently to all the work on heuristics, the concept of 2008), and Alonso-Ríos et al. (Alonso-Ríos et al., 2009).
usability itself was also being formally defined by researchers Consistently with this trend, and as mentioned by Bevan
and standard bodies. et al. (Bevan et al., 2015), the new draft proposal for the ISO
Some of the best-known usability models are those of the 9241–11 standard makes it clear that usability applies to all
International Organization for Standardization (ISO) (ISO/ aspects of use, including learnability, regular use, error pro-
IEC 9126-1, 2001, ISO 9241-11, 1998). The latest revisions tection, accessibility, and maintainability.
of the ISO usability model appear in the ISO FDIS 9241–210
(ISO 9241-210, 2010) and ISO/IEC 25010 (ISO/IEC 25010,
3. Methodology for the heuristic evaluation
2011) standards. The definition of usability for both is very
similar to the one in ISO-9241–11, and essentially consists of: As mentioned in the literature review, heuristic evaluation
often requires creating new lists of heuristics, either infor-
(1) Effectiveness. Accuracy and completeness with which mally or following a methodology, and the implementations
users achieve specified goals. for either approach vary greatly from one researcher to
(2) Efficiency. Resources expended in relation to the accu- another.
racy and completeness with which users achieve goals. The novelty of our approach is that it is not a methodology
(3) Satisfaction. Freedom from discomfort and positive for creating ad hoc heuristics for specific domains, instead it is
attitudes toward the use of the product. a methodology to guide a heuristic evaluation that tries to be
both systematic and generalizable. Our methodology is based
Jakob Nielsen was also a pioneer in this field and proposed on comprehensive, general-purpose, and hierarchical taxo-
his own usability model in 1993, which consists of five attri- nomies for the attributes of usability, the attributes of the
butes (Nielsen, 1993, p. 26): context of use, and the elements of a user interface (UI).
The aim is to retain the usefulness, ease of use, and general-
(1) Learnability. The system should be easy to learn so izability of heuristic evaluation while adding depth and
that the user can rapidly start getting some work structure.
done with the system. These generic taxonomies must be complemented with a
(2) Efficiency. The system should be efficient to use, so characterization of the system under study and its context of
that once the user has learned the system, a high level use. This characterization allows us to “instantiate” the taxo-
of productivity is possible. nomies with the specific characteristics of the system and the
(3) Memorability. The system should be easy to remem- context of use. Finally the heuristic evaluation consists then in
ber, so that the casual user is able to return to the assessing whether or not the elements of the system and the
system after some period of not having used it, with- tasks satisfy the relevant usability attributes guided by the
out having to learn everything all over again. taxonomies.
(4) Errors. The system should have a low error rate, so The generalizability of our methodology means that it can
that users make few errors during the use of the be applied to widely different domains, something that is
system, and so that if they do make errors they can especially useful in domains for which little research or stu-
easily recover from them. Further, catastrophic errors dies exist. As an example, we have used our methodology on
must not occur. usability studies of hardware devices (Alonso-Ríos et al., 2014)
(5) Satisfaction. The system should be pleasant to use, so and multiple-device applications (Mosqueira-Rey et al., 2017).
that users are subjectively satisfied when using it; they This does not mean that our methodology cannot also be
like it. used as a methodology for obtaining new heuristics for spe-
cific domains. For example, the work about multiple-device
Usability models are traditionally schematic and do not applications (Mosqueira-Rey et al., 2017) ended up suggesting
decompose the attributes of usability into further subattri- new heuristics for second-screen applications.
butes. This can lead to ambiguities that make it difficult to
carry out a usability analysis. For example, an attribute such
3.1. Taxonomies for usability and the context of use
as efficiency may refer to efficiency in human effort, efficiency
in task execution time, efficiency in tied up resources (both The usability model we employed is the usability taxonomy by
material and human), or efficiency in economic costs. Alonso-Ríos et al. (Alonso-Ríos et al., 2009). Of all the
Subdividing an attribute into multiple subattributes and expanded models of usability mentioned in Section 2, this is
providing precise definitions of each of them allows avoiding the most comprehensive one that happens to have been pub-
ambiguities and redundancies, thereby simplifying the usabil- lished in full. The taxonomy is a synthesis of the usability
ity study, ensuring greater thoroughness, and ultimately guar- models from ISO 9241–11 (ISO 9241-11, 1998), ISO/IEC
anteeing that results will not be contradictory. 9126–1, (ISO/IEC 9126-1, 2001), Nielsen (Nielsen, 1993),
4 D. ALONSO-RÍOS ET AL.

Preece et al. (Preece et al., 1994), Quesenbery (Quesenbery, distinction between user safety, third party safety, and
2001), Abran et al. (Abran et al., 2003), and Seffah et al. environment safety. The first two are further subdivided
(Seffah et al., 2006). The correspondences between the taxon- into physical safety, legal safeguarding, confidentiality,
omy and the other models are explicitly mapped out in and safety of assets.
Alonso-Ríos et al. (2009). ● Subjective satisfaction: The capacity of the system to
The usability taxonomy consists of six main attributes, that produce feelings of interest in users and to be aestheti-
are subsequently broken down into several subattributes, cally pleasant to them (in each of the five senses: visual,
which are in turn hierarchically structured into levels (see acoustic, tactile, olfactory, and gustatory).
Figure 1):
As ISO 9241–11 states, usability is always relative to the
● Knowability: The property by means of which the user context of use. The usability taxonomy above is complemen-
can understand, learn, and remember how to use the ted with a context-of-use taxonomy (Alonso-Ríos et al., 2010)
system. This attribute is subdivided into clarity, consis- that follows the same principles. The context-of-use taxonomy
tency, memorability, and helpfulness. The first three is composed of three main attributes which are again subdi-
apply to formal (e.g., visual, acoustic, etc.) and concep- vided into several levels of subattributes (see Figure 2):
tual aspects, and to the functioning of user and system
tasks. ● User. A user is a person who interacts directly or indir-
● Operability: The capacity of the system to provide users ectly with the system. It is further subdivided in: role
with the necessary functionalities and to permit users (how the user functions in the interaction with the
with different needs to adapt and use the system. This system), experience (practical skills and knowledge of
attribute is subdivided into completeness, precision, uni- the user in relation to the system), education (knowledge
versality (e.g., accessibility and cultural universality), and the user has acquired not through using the system but
flexibility (e.g., controllability and adaptiveness). through formal education or from the social, cultural, or
● Efficiency: The capacity of the system to produce appro- organizational environment), attitude to the system (feel-
priate results in return for the resources that are ings or emotions experienced by the user when operat-
invested. The taxonomy draws a distinction between ing the system), physical characteristics (characteristics
efficiency in human effort, in task execution time, in of the users body that have an impact on usability), and
tied up resources, and in economic costs, with each cate- cognitive characteristics (mental characteristics of the
gory further decomposed into more subattributes (e.g. user).
physical or mental effort, material or human resources, ● Task. A task is a piece of work that the user carries out
etc.). by interacting with the system. This category is further
● Robustness: The capacity of the system to resist error divided in: choice in system use (the extent to which
and adverse situations. The taxonomy draws a distinc- users may choose whether or not to use the system),
tion between robustness to internal error, to improper complexity (the degree to which completion of the task is
use, to third party abuse, and to environment problems. difficult for the user), temporal characteristics (refers to
● Safety: The capacity to avoid risk and damage derived both task duration and task frequency), demands (the
from the use of the system. The taxonomy draws a resources necessary to complete the task successfully),

Usability

Subjective
Knowability Operability Efficiency Robustness Safety
Satisfaction

In human To internal
Clarity Completeness User safety Interest
effort error

In task
To improper Third party
Consistency Precision execution Aesthetics
use safety
time

In tied up To third party Environment

Memorability Universality
resources abuse safety

To
In economic
Helpfulness Flexibility environment
costs
problems

Figure 1. Usability taxonomy.

 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 5

Context of
Use

User Task Environment

Attitude to Physical Cognitive

Role Experience Education
the system characteristics characteristics

Choice in Temporal Workflow

Complexity Demands Safety Criticality
system use characteristics controllability

Physical Social Technical

environment environment environment

Figure 2. Context-of-use taxonomy.

workflow controllability (to what extent the task can be Figure 3 shows a simplified view of the taxonomy, focusing
controlled by the user during implementation), safety on those elements that are more relevant to our case study.
(the degree to which the task as implemented does not Only the subattributes related to input controls are shown
cause damage or risk to users, third parties, the envir- expanded in the figure. Other categories, like structured text,
onment, or the system itself), and criticality (the extent can be easily expanded to include entities from the HTML
to which performance of the task is decisive). specification like paragraphs, block quotes, and so on.
● Environment. The environment consists of the external All the elements in the taxonomy possess in turn attributes
factors that affect the use of the system. We distinguish whose usability must be evaluated. Some examples are shown
between the physical environment (the surroundings and in Table 2 (components in general) and Table 3 (text).
space in which the user operates the system), the social Furthermore, the tasks that can be carried out with a UI
environment (the people with whom the user interacts and might be also generalized into taxonomies. The tasks in our
who effect the users interaction with the system), and the case study were simple, so we did not need to create a whole
technical environment (the technical equipment and infra- new model to identify and classify the tasks. For case studies
structures that support the functioning of the system). with more complex tasks, we can recommend authors like
Byrne et al. (Byrne et al., 1999), who proposed a “taskonomy”
of WWW use that consists of use information, locate on page,
go to page, provide information, configure browser, and react to
3.2. Taxonomy of UI elements and their properties
environment. This “taskonomy” has been used by automated
While the theoretical research on HCI has been mostly tools such as Web TANGO (Ivory, 2000). The development of
focused on the concepts of usability and the context of use, a full taxonomy of tasks is left as future work.
we argue that it is also possible to generalize the elements of
simple UIs into a taxonomy. For example, Han et al. (Han
3.3. Steps in the methodology
et al., 2000) proposed a framework for human interface ele-
ments that divides a consumer electronics product into phy- The methodology for heuristic evaluation is then composed of
sical components (e.g., display, button, and body) and logical the following steps:
components (e.g., menu, metaphor, and message). Each com-
ponent possesses a set of usability-relevant properties, which (1) Characterization of the system. This is a character-
they classify into three categories: individual (e.g., shape, size, ization of the features that the system must perform,
color), integration (e.g., structure, ordering, layout), and inter- identifying all the elements, user tasks and system
action (e.g., response, feedback, range of motion). tasks needed. This can be done with the help of
Following a similar approach, we produced a taxonomy of taxonomies for categorizing UI elements and their
UI elements for simple web applications like the one in our properties and also by “taskonomies” for categorizing
case study. The taxonomy is mainly based on the HTML 4.01 the tasks.We can start this characterization using the
specification (Raggett et al., 1999), the CSS 2.1 specification description of the features offered by the system. This
(Bos et al., 2005), and UI frameworks like Swing (Oracle, way we can easily connect our usability study with the
2017). For example, the controls are directly taken from the requirements analysis, integrating therefore the
HTML specification, except for combo boxes, which are not usability into the life cycle of development of the
natively supported by HTML. Similarly, dialog boxes are product.
mainly invoked via Javascript, and tab panels can be imple- (2) Characterization of the context of use. The charac-
mented with CSS. terization of the context of use consists in analyzing
6 D. ALONSO-RÍOS ET AL.

UI Elements

Container Control Text Images Dialog box

Information
Window Button Label Simple image
dialog box

Structured Input dialog

Panel Input Map image
text box

Tab panel

Text input Checkbox Radio button List

Drop-down
Text field Text area Password List box Combo box
list

Figure 3. UI elements taxonomy.

Table 2. Usability-relevant attributes for the UI elements in general.

example, several attributes in the usability taxonomy
Component Attributes Subattributes
are subdividing according to each of the five senses
Individual elements (visual) Shape–
Size Width (e.g., formal clarity, aesthetics, etc.), but many appli-
Height cations may not include some of these aspects. So, if a
Orientation –
Color Hue
web application does not include sound, all the cor-
Saturation responding subattributes must be removed.The next
Lightness task is the instantiation itself, in which we instantiate
Individual elements (conceptual) Shape Literal meaning
Symbolism specific attributes of the taxonomies with the charac-
Color Symbolism teristics of the system and the context of use, always
Relations between individual elements Ordering –
Focus – from the point of view of the user. For example, an
Positioning Horizontal alignment abstract usability attribute like visual clarity of ele-
Vertical alignment ments can be instantiated as visual clarity of the text
Horizontal spacing
Vertical spacing (or legibility), and text has specific properties that
Color contrast Hue determine its legibility, such as size, color, font family,
Saturation
Lightness and so on. Therefore, we need to pay attention to the
legibility of the font family, the colors, and so on.The
same process is repeated for the context-of-use tax-
Table 3. Usability-relevant attributes for the text. onomy. For example, the technical environment may
Attribute Subattributes Example values consist of hardware (e.g., a desktop computer) and
Font Font family Serif, Sans-Serif, Condensed, Monospace. . . software (e.g., the operating system or the browser).
Font weight Normal, bold, light. . .
Font style Normal, italic. . . (4) Heuristic evaluation. This activity consists in asses-
Decoration Underline. . . sing the usability of the system. The inputs for this
Capitalization Uppercase, lowercase. . .
Content – – process are the UI element, tasks, and taxonomy
attributes that were identified as relevant in previous
steps. The goal is to check how the individual ele-
the properties of the target audience, the functional- ments in the context of the system and its particular
ities of the system, and the setting. This is done by context of use satisfy the specific usability subattri-
following the attributes of the context-of-use taxon- butes. For example, the size of the font may not be
omy, which are classified into users, tasks, and envir- adequate depending on the size of the interface or the
onments.If the characteristics of the context of use are device and the specific characteristics of the user. The
explicitly taken into account, the results of the heur- results of the heuristic evaluation are strong and weak
istic evaluation can be interpreted more accurately. usability points and suggestions for improvement.
(3) Instantiation of the taxonomies. The first task in Although the steps are presented in order, there is
this step is to remove all the branches of the taxo- some flexibility. For example, you may perform the
nomies that are not relevant to the product. For characterization of the context of use before the
 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 7

characterization of the system, or you may perform The goal of the heuristic evaluation was to identify and fix
the instantiation jointly with the heuristic evaluation. usability problems in the UI as soon as possible, in order to
make the user tests more realistic and their results more
In the next sections we apply this methodology to a case relevant (Alonso Ríos, 2015). The results of the user tests
study and present the results. and the subsequent questionnaires and interviews were pub-
lished in Alonso-Ríos et al. (2014), Raneburger et al. (2013,
and Raneburger et al. (2013).
4. Applying the methodology
4.1. Case study 4.2. Characterization of the system

The system in our usability study was a set of UIs composed The flight booking application consists of four screens: (1)
of automatically generated web pages. The UIs were created origin, destination, and departure date; (2) available flights;
by an automated multi-device UI generator (Raneburger, (3) billing information; and (4) confirmation. As mentioned,
2014) that takes a high-level model of the interface and the items in these screens might be rearranged depending on
produces different versions of the same UI. The generator the type of layout. Figure 5 shows screenshots for the tabbed
uses artificial intelligence techniques to determine the best layout (T-UI) of the three first screens (the last one it is very
ways to arrange the elements of the UI (Raneburger, 2014), simple as it only shows the booking reference). The other
with the possible UIs constituting a search space. This way, types of layout would require scrolling on the tabbed screens.
different types of layouts may be obtained and, for example, Most of the elements in the UIs are standard HTML 4.01
unnecessary scrolling can be avoided. The process can be components, such as text labels, text fields, buttons, radio but-
customized according to certain parameters, and the resulting tons, and drop-down lists. Non-HTML elements would include
UIs can be also modified manually. The UIs are typically sets the tabs (which were implemented with CSS) and the dialog
of web pages, but other technologies may also be employed. boxes (which are invoked via Javascript). Therefore, the pre-
The heuristic evaluation was performed by the authors of viously discussed taxonomy of UI elements and their properties
this paper, who have a background in computer science and can be directly used to characterize the system.
usability. The results of the heuristic evaluations were dis- The tasks were identified after testing the application and
cussed in situ with the developers of the UIs. discussing the functionalities with the developers.
The UIs in our case study simulated a simplified applica- Firsly, we identified two user tasks:
tion for flight booking that was deployed as a web application
● Navigation. The users move between different screens
running on a web server. The UIs ultimately consisted of
HTML, CSS, and Javascript code. The users’ hardware con- and keep track of where they are.
● Data input. The users enter text, tap on radio buttons,
sisted in mobile devices (see Figure 4).
The users were asked to test different types of UI layouts and so on.
for the same application and decide which one they liked best. Secondly, we identified two system tasks (i.e., tasks that
Specifically, users tried layouts that could be tab-based (T-UI), are automatically performed by the system):
● Data validation. The system checks that required fields
horizontally-scrolling (H-UI), or vertically-scrolling (V-UI).
The purpose of the layouts was that, when the information are filled. If they are not, an error dialog box is shown.
● Refreshing the UI after the user’s input. The system
did not fit on the screen, you could either arrange the infor-
mation in tabs of let the information “spill over” and enable responds to user interactions (e.g., selecting an item
scrolling. Double scrolling was left out of this study because it from a drop-down list) by updating the contents of the
was considered impractical. display.
The next subsections are focused on the heuristic evalua-
tion, which was conducted before the user tests took place.
4.3. Characterization of the context of use
The characterization of the context of use is done by following
the attributes of the context-of-use taxonomy, which are clas-
sified into users, tasks, and environments.
The most relevant user characteristics for this case
study are:

● Experience with similar systems and knowledge of the

system’s domain and language. The application
assumes a certain familiarity with the technology being
used (e.g., the hardware and web browser), as well as the
domain of the application (i.e., flight booking) and the
language in which it is implemented (i.e., English).
● Physical characteristics. In particular, aptitudes and
Figure 4. The flight booking application on an iPod Touch. disabilities related to sight (as the only type of feedback
8 D. ALONSO-RÍOS ET AL.

Figure 5. Screenshots of screens 1 to 3 for the tabbed layout (T-UI).

is visual), hand movements, and touch (especially in 4.4. Instantiation of the taxonomies
terms of precision). Cognitive aspects such as memory
The instantiation begins by removing all the branches of the
are less relevant here.
taxonomies that are not relevant to the product. The most
The tasks were previously identified as navigation, data
noteworthy aspects are that we disregarded all acoustic ele-
input, data validation, and refreshing the UI after the
ments (since the interfaces are purely visual and tactile) and
user’s input. The context-of-use taxonomy was again
all economic aspects (since the application is a prototype
used to focus on their most relevant characteristics,
running on a browser). More specifically, the following
namely:
branches of the usability taxonomy were pruned:
● Complexity. The individual user tasks are meant to be
simple. They should be easy to understand and require
● For Knowability: All the formal aspects that are not
little effort. Clarity and efficiency are expected to be
good. related to visual or tactile issues.
● ● For Operability: All the accessibility and controllability
Duration (Temporal characteristics). The tasks are
divided into short steps and should not take much aspects that are not related to visual or tactile issues.
● For Efficiency: All the aspects related to economic costs.
time. Again, efficiency should be a strong point.
● ● For Safety: All the physical safety aspects that are not
Frequency (Temporal characteristics). The individual
tasks are repetitive, but the whole process (i.e., flight related to visual or tactile issues.
● For Subjective satisfaction: All the aesthetic aspects that
booking) is supposed to be sporadic. Clarity and mem-
orability should be prioritized so users can work without are not related to visual or tactile issues.
having to learn and remember everything all over again.
● Workflow controllability. Navigation, the steps in The following branches of the context-of-use-taxonomy were
which information is entered, and going back and also removed:
forth between screens to double-check data should be
carried out with flexibility and robustness. ● For User: All the physical characteristics that are not
The environment is classified into three categories in the related to visual or tactile issues.
context-of-use taxonomy. The most relevant character- ● For Task: All the physical characteristics in demands,
istics were: safety, and son on that are not related to visual or tactile
● Physical environment. Sensorial, atmospheric, and spa- issues.
tial conditions are particularly important. Mobile appli- ● For Physical Environment: All the sensorial characteris-
cations can be operated in all kinds of physical tics that are not related to visual or tactile issues.
environments, with variable characteristics.
● Social environment. Using mobile devices means that The next step is the instantiation itself. We correlate the
social factors like interruptions or other people’s support specific usability attributes with the components whose
or feedback are relevant here. usability must be examined from the point of view of the
● Technical environment. The fact that the UI runs on user. For example, the attribute cultural universality can be
specific types of technology means that these can applied to specific elements like language, terminology, and
become significant bottlenecks (e.g., display size). The date formats. Together with the context of use, this results in
technical environment is characterized in terms of phy- the following aspects to be evaluated:
sical equipment (i.e., hardware), logical equipment (i.e.,
operating system and browser), and consumption mate- ● Is the language of the UI an obstacle for non-native
rials (i.e., battery or power, and internet connection). users?
 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 9

● Is the terminology in the UI (e.g., the text in the con- the equal sign a neutral one. The first column of Table 4 shows
trols) understandable to all kinds of users? the relevant attributes of the usability taxonomy, whereas the
● Are date formats specified? second column shows the heuristic evaluation for them.
This methodology was then repeated for each element in
This process is repeated for all the usability attributes and the UI taxonomy. What follows is a brief summary of the
all the elements and tasks in the UI. results for these remaining elements, grouped by type of
element. Again, positive, negative, and neutral assessments
are indicated in parentheses.
4.5. Heuristic evaluation Containers:
Once we have instantiated the usability taxonomy with the
characterizations of the system and the context of use, the (1) The window’s title is too short, and identical for all
heuristic evaluation consists in assessing each UI element and screens. (–)
task in terms of clarity, consistency, and so on. The results of (2) The tab structure is not cluttered. (+)
this heuristic evaluation appear below. Bear in mind that (3) Width, spacing, and horizontal alignment are not
some of these problems are more easily solved than others. consistent between tab panels. (–)
For example, some of them are a result of the limitations of (4) The background color of the tab panels is not very
HTML and may be fixed using other technologies, such as consistent throughout the application. (–)
JavaScript. However, the heuristic evaluation must concern (5) Tab panels have the same meaning throughout the
itself with the usability of the system as it appears to the users, application. (+)
rather than the limitations of the programming language. (6) The “Credit Card Data” tab and the “Passenger Data”
Positive aspects should be maintained in future versions of tab could be merged. (=)
the UI, negative aspects should be corrected, and neutral (7) It is always clear which tab is selected. (+)
aspects are not really usability problems, but they can include (8) Color contrasts inside the tab panels could be higher.
suggestions for improvement. (–)

Heuristic evaluation of the UI elements Controls:

Table 4 shows an excerpt of the heuristic evaluation of the global
components of the UIs, prior to evaluating more specific ele- (1) The “Submit” text in the buttons is too close to
ments like containers, controls, and so on. The plus sign indi- computer jargon. (–)
cates a positive assessment, the minus sign a negative one, and (2) It is not clear in advance which fields are required
and which ones are not. (–)
(3) Text fields representing dates might be improved by
Table 4. Heuristic evaluation of the global components of the UIs. adding calendar controls. (=)
Usability attribute Heuristic Evaluation (4) The “expires” field does not specify the date format.
Knowability (–)
Visual clarity + Size is good enough.
+ Color is adequate. (5) The non-editable drop-down lists in the application
– Occasionally, some UI elements may not be are less flexible than editable combo boxes. (–).
visible, depending on layout.
Conceptual clarity + Meaning is clear due to standardized
technologies. Text:
Visual consistency + Visual consistency is generally good due to
standardized technologies.
= The shape of some UI elements may vary (1) The simultaneous use of serif and sans serif fonts
between different platforms or browsers. seems inconsistent or arbitrary. (–)
– Horizontal and vertical spacing between UI (2) The “From” and “To” labels in screen 1 could be
elements is occasionally inconsistent.
Conceptual consistency + Meaning is consistent due to standardized substituted with more descriptive terms like
technologies. “Origin” and “Destination”. (–)
Memorability + Memorability is good due to standardized
technologies.
(3) The “Name” label on the “credit card” tab could be
Helpfulness – No interactive help is provided, just alerts. changed to something less ambiguous like “Name on
Operability Credit Card”. (–)
Completeness + HTML, JavaScript, and CSS are enough for the (4) The meaning of the “CVC” label may not be clear to
desired UI elements.
Configurability – Application is not configurable. the user. (–)
Adaptiveness + Adaptiveness to different hardware/OS (5) The text is in English only. (–)
environments is guaranteed.
Robustness
Robustness to internal + Robustness to internal error is enhanced due to
Dialog boxes:
error standardized technologies.
Robustness to – UI is vulnerable to Internet connection (1) The title of all the dialog boxes is just the URL (see
environment problems.
problems Figure 6). This is an anti-phishing feature that may be
– UI is vulnerable to browser problems. confusing for some users. (–)
Subjective satisfaction (2) Dialog boxes tend to have very brief and not very
Visual aesthetics = UI has a simple and unadorned look.
descriptive messages. (–)
10 D. ALONSO-RÍOS ET AL.

Navigation:

(1) Going back and forth between screens can be diffi-

cult. The browser’s “back” and “forward” buttons are
unreliable. (–)
(2) Users have no way of keeping track of where they are. (–)
(3) Users need to go all the way up to the top of the
screen in order to switch tabs. (–)
(4) There are no actions available to the user on the
finish Screen. This is inconsistent with the other
screens. (–)

Data input:

(1) Tapping on a text field always brings up the default

keyboard. For numeric fields, it would be preferable
to show the numeric keyboard. (–)
(2) There is no easy way to clear all the fields in a screen. (–)
(3) The application offers some freedom in tasks, as data
can be entered in any order within a screen. (+)
(4) Efficiency would be improved by automatically set-
ting the focus on the first field. (–)
(5) Time and effort would be saved by allowing radio
buttons to be selected when you tap on their asso-
Figure 6. Error dialog box. ciated text. (–)

Data validation:
(3) There are inconsistencies in terminology between the
dialog boxes and the rest of the application. (–) (1) The only type of validation is checking that the fields
are not blank. (–)
(2) It would be advisable to take standard anti-hacking
Heuristic evaluation of the tasks precautions such as validating parameters, preventing
The tasks were previously characterized as navigation, data SQL injection, and so on. (–)
input, data validation, and refreshing the UI after the user’s
input. A summary of the most relevant results for the first Finally, Figure 7 shows an example of how the heuristic
three appears below, with positive, negative, and neutral evaluation led to concrete improvements of the UI. For
assessments again in parentheses. example:

Figure 7. Changes in Screen 1 after the heuristic evaluation.

 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 11

● The text is less vague (i.e., the header is now “Origin and usability model was one of the sources for the usability tax-
destination selection”; “From” and “To” have been onomy, we decided to use both Nielsen’s model and his
changed to “Origin” and “Destination”; “SUBMIT” has heuristics as a reference model for our validation.
been changed to “Search flighs”).
● The fonts are more consistent and legible (i.e.,
“Departure Date” has been changed from Serif to Sans- 5.1. Comparing the usability taxonomy and Nielsen’s
Serif). heuristics
● The title indicates the current screen and the number of
Table 5 maps Nielsen’s 1995 heuristics to the attributes in the
screens in parentheses (i.e., “(1/4)”).
usability taxonomy. Each row is headed by a Nielsen heuristic,
which is immediately followed by Nielsen’s description of it—
quoted verbatim—and the corresponding attributes or subat-
tributes from the usability taxonomy. Please note that each of
5. Validity of the approach
Nielsen’s descriptions contains multiple sentences, so most of
So far, the validity of the approach has been illustrated by them correspond to several attributes in the taxonomy.
example, showing how the taxonomies have been used to The main findings of this analysis are as follows:
identify, model, and classify real-world usability information.
This does not necessarily prove the validity of the approach, ● All of Nielsen’s heuristics could be mapped to one or
but failures in the process would point to inherent limitations more attributes in the usability taxonomy. This would
in it. The conditions for failure would be that the taxonomies suggest that the taxonomy is complete enough to repre-
do not represent what they aim to represent, that the concepts sent this kind of information at this level of detail.
are difficult to understand, or that the results it yields seem Conversely, it also shows that Nielsen’s heuristics can
arbitrary or not relevant. be connected to a comprehensive usability model that
Another way to investigate the validity of the approach has been derived from the usability literature.
would be to compare it with the work of other authors, as ● There is little redundancy in the mappings. Different
proposed by the methodology for creating heuristics by Rusu Nielsen heuristics tend to map to different attributes in
et al. (Rusu et al., 2011). More specifically, these authors used the taxonomy, with little repetition. This would suggest
Nielsen’s heuristics as a reference model. As Nielsen’s that both the usability taxonomy and Nielsen’s heuristics

Table 5. Correspondences between Nielsen’s heuristics (Nielsen, 1995) and the Usability Taxonomy (Alonso-Ríos et al., 2009).
Nielsen’s description Usability taxonomy
Visibility of system status
The system should always keep users informed about what is going on, through appropriate Clarity in functioning (system tasks). Efficiency in task
feedback within reasonable time. execution time (system response).
Match between system and the real world
The system should speak the users’ language, with words, phrases and concepts familiar to the Clarity of elements (conceptual). Clarity of structure
user, rather than in system-oriented terms. Follow real-world conventions, making information (conceptual).
appear in a natural and logical order.
User control and freedom
Users often choose system functions by mistake and will need a clearly marked “emergency exit” to Freedom in tasks. Reversibility.
leave the unwanted state without having to go through an extended dialogue. Support undo and
redo.
Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same Consistency of elements (conceptual). Consistency in
thing. Follow platform conventions. functioning (user tasks).
Error prevention
Even better than good error messages is a careful design which prevents a problem from Robustness to internal error. Robustness to improper use.
occurring in the first place. Either eliminate error-prone conditions or check for them and
present users with a confirmation option before they commit to the action.
Recognition rather than recall
Minimize the user’s memory load by making objects, actions, and options visible. The user should Efficiency in human effort (mental). Clarity of elements
not have to remember information from one part of the dialogue to another. Instructions for (formal). Suitability of documentation content.
use of the system should be visible or easily retrievable whenever appropriate.
Flexibility and efficiency of use
Accelerators—unseen by the novice user—may often speed up the interaction for the expert user Efficiency in task execution time (user). Freedom in tasks.
such that the system can cater to both inexperienced and experienced users. Allow users to Technical configurability.
tailor frequent actions.
Aesthetic and minimalist design
Dialogues should not contain information that is irrelevant or rarely needed. Every extra unit of Clarity of structure (formal).
information in a dialogue competes with the relevant units of information and diminishes their
relative visibility.
Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, Clarity of elements (conceptual). Helpfulness.
and constructively suggest a solution.
Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to Suitability of documentation content.
provide help and documentation. Any such information should be easy to search, be focused on
the user’s task, list concrete steps to be carried out, and not be too large.
12 D. ALONSO-RÍOS ET AL.

are able to convey fine-grained distinctions. However, attention to visual consistency. The taxonomies pointed
some redundancy is to be expected, as we are dealing us to the consistency of colors, fonts, width, spacing, and
with natural language and complex concepts. The only alignment.
attributes in the taxonomy that are repeated (once) are ● Completeness. Users’ expectations about the function-
conceptual clarity of elements, freedom in tasks, and alities an application should include are also neglected
suitability of documentation content. In this regard, the by Nielsen’s heuristics. However, today’s users are much
only suggestion we would make to improve Nielsen’s more active participants and would expect features like
heuristics is a minor one. His sentence “Instructions for validations or reset buttons in our application.
use of the system should be visible or easily retrievable ● Universality. An important drawback of the application
whenever appropriate” should be under his help and was that the text was only in English. Nielsen’s heuristics
documentation heuristic—instead of recognition rather were conceived before multi-language applications
than recall—in order to avoid redundancy. became widespread, and his famous “speak the users’
● Nielsen’s heuristics do not cover the full range of language” phrase refers to terminology.
usability. The usability taxonomy includes a number ● Safety. Our evaluation also detected the lack of standard
of subattributes, and even first-level attributes, that do anti-hacking precautions, whereas Nielsen’s heuristics
not map to any of Nielsen’s heuristics. However, this do not take safety into account.
fact needs to be examined carefully, because it is not ● Visual aesthetics. As mentioned above, subjective aes-
always a problem. Firstly, we must say that Nielsen’s thetics play no role in Nielsen’s heuristics.
heuristics are focused on software, whereas the taxon- ● The influence of the technological enviroment. On the
omy is aimed at any type of system, so it is natural that one hand, the application’s dependecy on the browser
the latter would include more attributes. Secondly, and the internet connection (which are both part of the
Nielsen’s heuristics were never intended to be exhaus- context of use) can lead to unexpected usability pro-
tive, as he explicitly defined them as broad rules of blems. On the other hand, the standardized nature of
thumb. Thirdly, both Nielsen’s heuristics and the con- HTML helps to achieve adaptiveness, robustness, and
cept of usability itself have been subsequently revised visual consistency. None of this can be derived from
and updated by different authors in ways that Nielsen Nielsen’s heuristics, which were formulated when appli-
could not have predicted. Nielsen’s own observation on cations were more self-contained, and portability and
the evolution of usability is that old guidelines tend to be connectivity were not essential software features.
remarkably stable, although he acknowledges the impor-
tance of technological improvements and behavioral
adaptations (Nielsen, 2017). That said, Nielsen’s heuris-
5.3. Comparing Nielsen’s heuristics and his own usability
tics surprisingly do not consider subjective satisfaction,
model
which is historically one of the main attributes of usabil-
ity. In fact, it is the purely subjective component of Comparing Nielsen’s heuristics with his own usability model
usability, as opposed to measurable attributes like effi- shows some disconnect between them. Firstly, and as men-
ciency. There is a heuristic called aesthetic and minim- tioned before, an important attribute of his model, namely,
alist design, but, as his description shows, it has nothing subjective satisfaction, is neglected by his heuristics.
to do with (subjective) aesthetics. Conversely, Nielsen’s heuristics suggest good practices that
do not seem to be derived from his usability model. For
example, Nielsen’s heuristics include configurability, user
5.2. Comparing the results of the heuristic evaluation
freedom, and reversibility—all of them covered by the usabil-
and Nielsen’s heuristics
ity taxonomy—whereas his model does not. Moreover, his
The next part of our analysis consists in taking the results of heuristics prioritize recognition rather than recall (“the user
our heuristic evaluation and trying to connect them to should not have to remember information”) or, alternatively,
Nielsen’s heuristics. We found that most of our results could help and documentation (“even though it is better if the
be connected to his heuristics but the correspondences were system can be used without documentation”), whereas his
not always straightforward, as his heuristics tend to be broad. usability model assumes that users need to memorize things.
More importantly, there is some usability information that In sum, Nielsen’s heuristics and his own model convey some-
could have been missed if one simply relied on his heuristics. what different views of usability, and this is because the
Other authors have identified similar limitations in Nielsen’s relationship between them is never made explicit. In contrast,
heuristics, but their solution is typically to propose new heur- we argue that usability heuristics should be strongly and
istics to complement Nielsen’s own (Bertini et al., 2006, Yáñez explicitly connected to a usability model to ensure that they
Gómez et al., 2014, Al-Khalifa et al., 2016). are evaluating the same thing.
The results of our heuristic evaluation that cannot be easily
connected to Nielsen’s heuristics refer to aspects like:
6. Conclusions
● Consistency. Nielsen is only concerned with what we This paper has presented a systematic and generalizable
call conceptual consistency (i.e., consistency in meaning). approach to heuristic evaluation that has been applied to a
However, our results show that one must also pay usability study of mobile UIs.
 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 13

Heuristic evaluation is traditionally based on a set of broad Alonso-Ríos, D., Mosqueira-Rey, E., & Moret-Bonillo, V. (2014). A
rules of thumb, such as Jakob Nielsen’s heuristics, which are taxonomy-based usability study of an intelligent speed adaptation
device. International Journal of Human-Computer Interaction, 30(7),
typically complemented by ad hoc judgments or deep knowl-
585–603.
edge of guidelines. In contrast, the most novel aspect of our Alonso-Ríos, D., Raneburger, D., Popp, R., Kaindl, H., & Falb, J. (2014).
approach is the integration of general-purpose taxonomies of A user study on tailoring GUIs for smartphones. In Proceedings of the
usability attributes, context-of-use characteristics, and UI ele- 29th Annual ACM Symposium on Applied Computing, ACM, pp. 186–
ments into the technique. The goal is to retain the usefulness, 192.
Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E., & Moret-
ease of use, and generalizability of heuristic evaluation while
Bonillo, V. (2009). Usability: A critical analysis and a taxonomy.
adding depth and structure. International Journal of Human-Computer Interaction, 26(1), 53–74.
We put our methodology into practice by conducting a Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E., & Moret-
heuristic evaluation of a flight booking application for mobile Bonillo, V. (2010). A context-of-use taxonomy for usability studies.
devices. The taxonomies facilitated the task and helped to International Journal of Human-Computer Interaction, 26(10), 941–
970.
identify usability problems related to aspects like clarity, con-
Amaro, A. C., Veloso, A. I., & Oliveira, L. (2016). Social games and
sistency, completeness, universality, safety, robustness, or different generations: A heuristic evaluation of candy crush saga. In
aesthetics. 2016 1st International Conference on Technology and Innovation in
We also investigated the validity of our approach by com- Sports, Health and Wellbeing (TISHW), Dec, pp. 1–8.
paring the usability taxonomy with Nielsen’s heuristics. This Baker, K., Greenberg, S., & Gutwin, C. (2001). Heuristic evaluation of
groupware based on the mechanics of collaboration. In Engineering for
also helped us to identify some deficiencies in Nielsen’s heur-
human-computer interaction (pp. 123–139). London, UK: Springer-
istics. For example, Nielsen’s heuristics do not cover the full Verlag.
range of usability, which means that his heuristics neglect Bastien, J. C., & Scapin, D. L. (1995). Evaluating a user interface with
important usability attributes like aesthetics, visual consistency, ergonomic criteria. International Journal of Human-Computer
or universality. Other researchers have reached similar con- Interaction, 7(2), 105–121.
Bertini, E., Gabrielli, S., & Kimani, S. (2006). Appropriating and assessing
clusions, but their solution is typically to create new heuristics
heuristics for mobile computing. In Proceedings of the working con-
that are context-dependent and difficult to reuse. Our meth- ference on Advanced visual interfaces, ACM, pp. 119–126.
odology, on the other hand, is easily generalizable to other Bevan, N. (2009). Extending quality in use to provide a framework for
domains because it is based on generic taxonomies. usability measurement. Berlin, Germany: Springer.
We also argue that heuristics should always be explicitly Bevan, N., Carter, J., & Harker, S. (2015). Iso 9241-11 revised: What have
we learnt about usability since 1998? In International Conference on
connected to a usability model in order to ensure that they are
Human-Computer Interaction, Springer, pp. 143–151.
in fact evaluating usability. For example, we found a certain Bos, B., Çelik, T., Hickson, I., & Lie, H. W. (2005). Cascading style sheets
disconnect between Nielsen’s usability model and his own level 2 revision 1 (CSS 2.1) specification. W3C working draft, W3C,
heuristics, which means that they convey somewhat different June.
views of usability. Byrne, M. D., John, B. E., Wehrle, N. S., & Crow, D. C. (1999). The
tangled web we wove: A taskonomy of WWW use. In Proceedings of
the SIGCHI conference on Human Factors in Computing Systems,
ACM, pp. 544–551.
Acknowledgments De Lima Salgado, A., Rodrigues, S. S., & Fortes, R. P. M. (2016). Evolving
heuristic evaluation for multiple contexts and audiences: Perspectives
This work has been supported by Xunta de Galicia (Spain) under project
from a mapping study. In Proceedings of the 34th ACM International
GRC2014/035. Also financial support from the Xunta de Galicia (Centro
Conference on the Design of Communication, SIGDOC ‘16, New York,
singular de investigación de Galicia accreditation 2016-2019, REF.
NY, USA, ACM, pp. 19: 1–19:8.
ED431G/01) and the European Union (European Regional
Desurvire, H., Caplan, M., & Toth, J. A. (2004). Using heuristics to
Development Fund—ERDF) is gratefully acknowledged.
evaluate the playability of games. In CHI’04 extended abstracts on
human factors in computing systems (pp. 1509–1512). New York,
NY: ACM.
ORCID Dumas, J. (2007). The great leap forward: The birth of the usability
profession (1988-1993). Journal of Usability Studies, 2(2), 54–60.
David Alonso-Ríos http://orcid.org/0000-0003-2147-3010 Greunen, D. V., Yeratziotis, A., & Pottas, D. (2011). A three-phase
Eduardo Mosqueira-Rey http://orcid.org/0000-0002-4894-1067 process to develop heuristics for specific application domains. In
Vicente Moret-Bonillo http://orcid.org/0000-0002-9435-3151 Proceedings of the 13th ZA-WWW conference, Johannesburg, South
Africa.
Han, S. H., Hwan Yun, M., Kim, K.-J., & Kwahk, J. (2000). Evaluation of
References product usability: Development and validation of usability dimensions
and design elements based on empirical models. International Journal
Abran, A., Khelifi, A., & Suryn, W. (2003). Usability meanings and of Industrial Ergonomics, 26(4), 477–488.
interpretations in ISO standards. Software Quality Journal, 11, 325– Hermawati, S., & Lawson, G. (2015). A user-centric methodology to
338. establish usability heuristics for specific domains. In Proceedings of
Al-Khalifa, H. S., Al-Twaim, B., & AlHarbi, B. (2016). A heuristic checklist the International Conference on Ergonomics & Human Factors, pp.
for usability evaluation of saudi government mobile applications. In 80–85.
Proceedings of the 18th International Conference on Information Hollingsed, T., & Novick, D. G. (2007). Usability inspection methods
Integration and Web-based Applications and Services, ACM, pp. 375–378. after 15 years of research and practice. In Proceedings of the 25th
Alonso Ríos, D. (2015). Development of usability and context-of-use annual ACM international conference on Design of communication,
taxonomies, integration with techniques for the study of usability and ACM, pp. 249–255.
application to real-world intelligent systems.A Coruña: Universidade Hvannberg, E. T., Halldórsdóttir, G., & Rudinsky, J. (2012). Exploitation
da Coruña. of heuristics for virtual environments. In Proceedings of the 7th Nordic
14 D. ALONSO-RÍOS ET AL.

Conference on Human-Computer Interaction: Making Sense Through Nielsen, J. Change vs. stability in web usability guidelines. Retrieved May
Design, ACM, pp. 308–317. 18, 2017, from https://www.nngroup.com/articles/usability-guidelines-
Hvannberg, E. T., Law, E. L.-C., & Lérusdóttir, M. K. (2007). Heuristic change/, June 2007
evaluation: Comparing ways of finding and reporting usability pro- Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces.
blems. Interacting with Computers, 19(2), 225–240. In Proceedings of the SIGCHI conference on Human factors in comput-
ISO 9241-11. (1998). 1998 - Ergonomic requirements for office work with ing systems, ACM, pp. 249–256.
visual display terminals (VDTs) – Part 11: Guidance on usability. Oracle. Swing APIs and developer guides. Retrieved May 18, 2017, from
Geneva, Switzerland: ISO. http://docs.oracle.com/javase/8/docs/technotes/guides/swing/index.
ISO 9241-210. (2010). 2010 - Ergonomics of human-system interaction – html
Part 210: Human-centred design for interactive systems. Geneva, Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T.
Switzerland: ISO. (1994). Human-computer interaction. Essex, UK: Addison-Wesley.
ISO/IEC 25010. (2011). Systems and software engineering – Systems and Quesenbery, W. (2001). What does usability mean: Looking beyond “ease
software quality requirements and evaluation (SQuaRE) – System and of use”. In Proceedings of the 18th Annual Conference Society for
software quality models. Geneva, Switzerland: ISO. Technical Communications.
ISO/IEC 9126-1. (2001). 2001 - Software engineering – Product quality – Quiñones, D., & Rusu, C. (2017). How to develop usability heuristics: A
Part 1: Quality model. Geneva, Switzerland: ISO. systematic literature review. Computer Standards & Interfaces, 53, 89–
Ivory, M. Y. (2000). Web TANGO: Towards automated comparison of 122.
information-centric web site designs. In CHI’00 extended abstracts on Raggett, D., Hors, A. L., Jacobs, I. (1999). HTML 4.01 specification. W3C
human factors in computing systems (pp. 329–330). New York, NY: ACM. Recommendation, from https://www.w3.org/TR/html401/.
J. Nielsen. (1992). Finding usability problems through heuristic evalua- Raneburger, D. (2014). Interactive Model-Driven Generation of Graphical
tion. In Proceedings of the SIGCHI conference on Human factors in User Interfaces for Multiple Devices. PhD thesis, Institute of Computer
computing systems, ACM, pp. 373–380. Technology, Vienna University of Technology.
Johannessen, G. H. J., & Hornbæk, K. (2014). Must evaluation methods Raneburger, D., Alonso-Ríos, D., Popp, R., Kaindl, H., & Falb, J. (2013).
be about usability? Devising and assessing the utility inspection A user study with GUIs tailored for smartphones. In Human-
method. Behaviour & Information Technology, 33(2), 195–206. Computer Interaction–INTERACT 2013 (pp. 505–512).
Lechner, B., Fruhling, A. L., Petter, S., & Siy, H. P. (2013). The chicken Berlin, Germany: Springer.
and the pig: User involvement in developing usability heuristics. In Raneburger, D., Popp, R., Alonso-Ríos, D., Kaindl, H., & Falb, J. (2013).
19th Americas Conference on Information Systems, AMCIS 2013, A user study with GUIs tailored for smartphones and tablet PCs. In
Chicago, Illinois, USA, August 15-17, 2013. 2013 IEEE International Conference on Systems, Man, and Cybernetics
Levi, M. D., & Conrad, F. G. (1996). A heuristic evaluation of a world (SMC), IEEE, pp. 3727–3732.
wide web prototype. Interactions, 3(4), 50–61. Rusu, C., Roncagliolo, S., Rusu, V., & Collazos, C. (2011). A methodology
Lewis, J. R. (2014). Usability: Lessons learned. . . and yet to be learned. to establish usability heuristics. In ACHI 2011-4th International
International Journal of Human-Computer Interaction, 30(9), 663– Conference on Advances in Computer-Human Interactions, pp. 59–62.
684. Seffah, A., Donyaee, M., Kline, R. B., & Padda, H. K. (2006). Usability
Magoulas, G. D., Chen, S. Y., & Papanikolaou, K. A. (2003). Integrating measurement and metrics: A consolidated model. Software Quality
layered and heuristic evaluation for adaptive learning environments. Journal, 14, 159–178.
In Proceedings of the Second Workshop on Empirical Evaluation of Shneiderman, B. (2009). Designing the user interface: Strategies for effec-
Adaptive Systems, held at the 9th International Conference on User tive human-computer interaction. 5th edition. Boston, MA: Pearson
Modeling UM2003, Pittsburgh, pp. 5–14. Addison-Wesley.
Mankoff, J., Dey, A. K., Hsieh, G., Kientz, J., Lederer, S., & Ames, M. (2003). Silva, P. A., Jordan, P., & Holden, K. (2014). Something old, something
Heuristic evaluation of ambient displays. In Proceedings of the SIGCHI new, something borrowed: Gathering experts’ feedback while per-
conference on Human factors in computing systems, ACM, pp. 169–176. forming heuristic evaluation with a list of heuristics targeted at older
Mariage, C., & Vanderdonckt, J. (2001). A comparative usability study of adults. In Proceedings of the 2014 Workshops on Advances in
electronic newspapers (pp. 325–337). London, UK: Springer London. Computer Entertainment Conference, Funchal, Portugal, pp. 19:1–19:8.
Mosqueira-Rey, E., Alonso-Ríos, D., Prado-Gesto, D., & Moret-Bonillo, Smith, S. L., & Mosier, J. N. (1986). Guidelines for designing user interface
V. (2017). Usability evaluation and development of heuristics for software. Bedford, MA: Mitre Corporation.
second-screen applications. In Proceedings of the 32th Annual ACM Sutcliffe, A., & Gault, B. (2004). Heuristic evaluation of virtual reality
Symposium on Applied Computing, ACM, pp. 569–571. applications. Interacting with Computers, 16(4), 831–849.
Muller, M. J., Matheson, L., Page, C., & Gallup, R. (1998). Methods & Winter, S., Wagner, S., & Deissenboeck, F. (2008). A comprehensive
tools: Participatory heuristic evaluation. Interactions, 5(5), 13–18. model of usability. In Engineering interactive systems (pp. 106–122).
Nielsen, J. (1993). Usability Engineering. Boston, MA: Academic Press. Berlin, Germany: Springer.
Nielsen, J. Guerrilla HCI: Using discount usability engineering to pene- Yáñez Gómez, R., Cascado Caballero, D., & Sevillano, J.-L. (2014).
trate the intimidation barrier. Retrieved May 18, 1994, from https:// Heuristic evaluation on mobile interfaces: A new checklist. The
www.nngroup.com/articles/guerrilla-hci/ Scientific World Journal, 2014. doi: 10.1155/2014/434326
Nielsen, J. (1995). 10 usability heuristics for user interface design. Zhang, J., Johnson, T. R., Patel, V. L., Paige, D. L., & Kubose, T. (2003).
Retrieved May 18, 2017, from https://www.nngroup.com/articles/ten- Using usability heuristics to evaluate patient safety of medical devices.
usability-heuristics/ Journal of Biomedical Informatics, 36(1), 23–30.