See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220422205

Usability Engineering Methods For Software Developers

Article  in  Communications of the ACM · January 2005

DOI: 10.1145/1039539.1039541 · Source: DBLP

CITATIONS READS
608 5,080

1 author:

Andreas Holzinger
Medical University of Graz
507 PUBLICATIONS   6,934 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Acrossing View project

TuGroVis - Tumor Growth Visualization View project

All content following this page was uploaded by Andreas Holzinger on 10 November 2014.

The user has requested enhancement of the downloaded file.

 USABILITY
ENGINEERING
METHODS FOR
SOFTWARE
DEVELOPERS
The human-computer interaction community aims to
increase the awareness and acceptance of established methods
among software practitioners. Indeed, awareness of the basic
usability methods will drive an Information Society for all.

By Andreas Holzinger

U
sability is most often defined as the
ease of use and acceptability of a
system for a particular class of users
carrying out specific tasks in a specific environ-
ment. Ease of use affects the users’ performance
and their satisfaction, while acceptability affects
whether the product is used [1]. Thus, it is of
great importance that every software practitioner
not only be aware of various usability methods,

COMMUNICATIONS OF THE ACM January 2005/Vol. 48, No. 1 71

but be able to quickly determine Inspection Methods Test Methods
which method is best suited to
every situation in a software Heuristic Cognitive Action Thinking Field
Evaluation Walkthrough Analysis Aloud Observation Questionnaires
project.
Applicably all all design design final testing all
One of the basic lessons we in Phase
have learned in human-computer Required low medium high high medium low
interaction (HCI) is that usabil- Time
ity must be considered before Needed Users
none none none 3+ 20+ 30+
prototyping takes place. There
3+ 3+ 1-2 1 1+ 1
are techniques (such as usability Required
Evaluators
context analysis) intended to Required low low low high medium low
facilitate such early focus and Equipment
commitment [11]. When usabil- Required medium high high medium high low
Expertise
ity inspection, or testing, is first
no no no yes yes no
carried out at the end of the Intrusive
design cycle, changes to the inter- Comparison of Usability Evaluation Techniques
face can be costly and difficult to
implement, which in turn leads to usability recom- Comparison of These methods include heuristic
usability evaluation
mendations. These are often ignored by developers techniques. evaluation, cognitive walk-
who feel, “We don’t have usability problems.” The throughs, and action analysis.
earlier critical design flaws are detected, the more Heuristic evaluation (HE) is
likely they can be corrected. Thus, user interface the most common informal
design should more properly be called user interface method. It involves having usability specialists judge
development, analogous to software development, whether each dialogue or other interactive element
since design usually focuses on the synthesis stages, follows established usability principles [8]. The origi-
and user interface components include metaphors, nal approach is for each individual evaluator to
mental models, navigation, interaction, appearance, inspect the interface alone. Only after all the evalua-
and usability [6]. tions have been completed are the evaluators allowed
It is generally accepted that the following five to communicate and aggregate their findings. This
essential usability characteristics should be part of any restriction is important in order to ensure indepen-
software project: learnability, so the user can rapidly dent and unbiased evaluations. During a single evalu-
begin working with the system; efficiency, enabling a ation session, the evaluator goes through the interface
user who has learned the system to attain a high level several times, inspects the various interactive ele-
of productivity; memorability, allowing the casual user ments, and compares them with a list of recognized
to return to the system after a period of non-use with- usability principles (for example, Nielsen’s Usability
out having to relearn everything; low error rate, so Heuristics [7]). There are different versions of HE
users make fewer and easily rectifiable errors while currently available; for example, some have a cooper-
using the system, and no catastrophic errors occur; ative character. The heuristics must be carefully
and satisfaction, making the system pleasant to use. selected so they reflect the specific system being
There are trade-offs among these criteria, and some inspected, especially for Web-based services where
are more important than others, although this rank- additional heuristics become increasingly important.
ing depends on the situation. For example, long-term Usually 3–5 expert evaluators are necessary (increas-
efficiency may be sufficiently important for develop- ing the cost of this technique); less-experienced peo-
ers to be willing to sacrifice rapid learnability [10]. ple can perform an HE, but the results are not as
To ensure a software project has these essential good. However, HE using non-experts is appropriate
usability characteristics, we use methods we divide at times, depending on who is available to participate.
into inspection methods (without end users) and test Advantages include the application of recognized
methods (with end users). The accompanying figure and accepted principles; intuitiveness; usability early
details these characteristics. in the development process; effective identification of
major and minor problems; rapidity; and usability
Usability Inspection Methods throughout the development process.
This is a set of methods for identifying usability Disadvantages include separation from end users;
problems and improving the usability of an interface inability to identify or allow for unknown users’
design by checking it against established standards. needs; and unreliable domain-specific problem iden-

72 January 2005/Vol. 48, No. 1 COMMUNICATIONS OF THE ACM

tification. Also, HE does not necessarily evaluate the rapidly. Such representations can often only be inter-
complete design, since there is no mechanism to preted by those who conducted the analysis.
ensure the entire design is explored, and evaluators Advantages include precise prediction of how long
can focus too much on one section; and the validity of a task will take, and a deep insight into users’
Nielsen’s guidelines has been questioned [9]. behavior.
A cognitive walkthrough (CW) is a task-oriented Disadvantages of action analysis include it is very
method by which the analyst explores the system’s time-consuming and requires high expertise.
functionalities; that is, CW simulates step-by-step
user behavior for a given task. CW emphasizes cogni- Usability Test Methods
tive issues, such as learnability, by analyzing the men- Testing with end users is the most fundamental
tal processes required of the users. This can be usability method and is in some sense indispensable.
achieved during the design by making the repertory of It provides direct information about how people use
available actions salient, providing an obvious way to our systems and their exact problems with a specific
undo actions, and offering limited alternatives [5]. interface. There are several methods for testing
The background is derived from exploratory learning usability, the most common being thinking aloud,
principles. Several versions of CW exist, including field observation, and questionnaires.
pluralistic walkthroughs wherein end users, software Thinking aloud (THA) [7] may be the single most
developers, and usability experts go through the sys- valuable usability engineering method. It involves
tem, discussing every single dialogue element. having an end user continuously thinking out loud
Advantages include independence from end users while using the system. By verbalizing their thoughts,
and a fully functioning prototype, helping designers the test users enable us to understand how they view
to take on a potential user’s perspective, effective iden- the system, which makes it easier to identify the end
tification of problems arising from interaction with users’ major misconceptions. By showing how users
the system, and the ability to help to define users’ interpret each individual interface item, THA facili-
goals and assumptions. tates a direct understanding of which parts of the dia-
Disadvantages of CW include possible tediousness logue cause the most problems. In THA the time is
and the danger of an inherent bias due to improper very important, since the contents of the users’ work-
task selection, emphasis on low-level details, and non- ing memory contents are desired. Retrospective
involvement of the end user. reports are much less useful, since they rely on the
The action analysis method is divided into formal users’ memory of what they had been thinking some
and back-of-the-envelope action analysis; in both, the time ago. A variant of THA called constructive inter-
emphasis is more on what the practitioners do than action involves having two test users use a system
on what they say they do. The formal method requires together (co-discovery learning). The main advantage
close inspection of the action sequences a user per- is that the test situation is much more natural than
forms to complete a task. This is also called keystroke- standard THA with single users working alone, since
level analysis [2]. It involves breaking the task into people are used to verbalizing their thoughts when
individual actions such as move-mouse-to-menu or trying to solve a problem together. Therefore, users
type-on-the-keyboard and calculating the times may make more comments when engaged in con-
needed to perform the actions. Back-of-the-envelope structive interaction than when simply thinking aloud
analysis is less detailed and gives less precise results, for the benefit of an experimenter.
but it can be performed much faster. This involves a Advantages of THA include revealing why users do
similar walkthrough of the actions a user will perform something; providing a close approximation to how
with regard to physical, cognitive, and perceptual individuals use the system in practice; provision of a
loading. To understand this thoroughly we must keep wealth of data, which can be collected from a fairly
in mind that goals are external, and we achieve goals. small number of users; user comments of often con-
Tasks are those processes applied through some device tain vivid and explicit quotes; preference and perfor-
in order to achieve the goals, and we perform tasks. mance information can be collected simultaneously;
Actions are tasks with no problem-solving and no THA helps some users to focus and concentrate; and
internal control structure. We do actions. The main early clues can help to anticipate and trace the source
problem of task analysis [3] is the difficulty in accom- of problems to avoid later misconceptions and confu-
modating complicated tasks completed by more than sion in the early stage of design.
one individual. Furthermore, the representation of a Disadvantages include a failure to lend itself well to
task analysis is complex, even when a simple task is most types of performance measurement; the differ-
studied, and tends to become very unwieldy very ent learning style is often perceived as unnatural, dis-

COMMUNICATIONS OF THE ACM January 2005/Vol. 48, No. 1 73

 tracting, and strenuous by the users; nonanalytical The form of the interview can be adjusted to respond
learners generally feel inhibited; and this method is to the user and encourage elaboration.
time-consuming since briefing the end users is a nec- Advantages include that subjective user prefer-
essary part of the preparation. ences, satisfaction, and possible anxieties can be easily
Causing users to focus and concentrate is both an identified; and questionnaires can be used to compile
advantage and a disadvantage since it results in less- statistics.
than-natural interactions at times and causes THA to Disadvantages include that indirect methods result
be faster due to the users’ focus. in low validity (discrepancies between subjective and
Field observation is the simplest of all methods. It objective user reactions must be taken into account);
involves visiting one or more users in their work- this method needs sufficient responses to be signifi-
places. Notes must be taken as unobtrusively as possi- cant (we are of the opinion that 30 users is the lower
ble to avoid interfering with their work. Noise and limit for a study); and it identifies fewer problems
disturbance can also lead to false results. Ideally, the than the other methods.
observer should be virtually invisible to ensure nor- Usability inspection needs to be combined with
mal working conditions. Sometimes video is used to usability test methods. For example, a cognitive walk-
make the observation process less obtrusive, but it is through can be supplemented with a task-independent
rarely necessary. Observation focuses on major usabil- method, such as heuristic evaluation. Indirect usability
ity catastrophes that tend to be so glaring they are tests, such as questionnaires or interviews, must be sup-
obvious the first time they are observed and thus do plemented with direct usability tests; thinking aloud or
not require repeated perusal of a recorded test session. observation would be suitable. An absolute must is
Considering the time needed to analyze a videotape is understanding the user’s task, culture, and capabilities;
approximately 10 times that of a user test, the time is involving the users in the design early on; and testing
better spent testing more subjects or testing more iter- and iterating, with or without users. c
ations of the design. Video is, however, appropriate in
some situations. For example, a complete record of a References
series of user tests can be used to perform formal 1. Bevan, N. Measuring usability as quality of use. Softw. Quality J. 4
(1995), 115–130.
impact analysis of usability problems [4]. 2. Card, S.K., Moran, T.P. and Newell, A. The Psychology of Human-
Another means of electronic observation is data Computer Interaction. Erlbaum, Hillsdale, NJ, 1983.
3. Carroll, J.M. Making use is more than a matter of task analysis. Inter-
logging, which involves statistics about the detailed act. Comput. 14, 5 (2002), 619–627.
use of a system. Data logging can provide extensive 4. Holzinger, A. Application of rapid prototyping to the user interface devel-
timing data, which is generally important in HCI and opment for a virtual medical campus. IEEE Softw. 21, 1 (Jan. 2004).
5. Lewis, C. and Wharton, C. Cognitive Walkthroughs. Handbook of
usability. Normally, logging is used to collect infor- Human-Computer Interaction, 2nd ed. M. Helander, Ed. Elsevier, Ams-
mation about the field use of a system after release, terdam, 1997, 717–732.
6. Marcus, A. Dare we define user-interface design? interactions 9, 5
but it can also be used as a supplementary method of (2002), 19–24.
collecting more detailed data during user testing. Typ- 7. Nielsen, J. Usability Engineering. Morgan Kaufmann, San Francisco,
ically, an interface log will contain statistics about the 1994.
8. Nielsen, J. and Mack, R.L. Usability Inspection Methods. Wiley, New
frequency with which each user has used each feature York, 1994.
in the program and the frequency with which various 9. Sears, A.L. Heuristic walkthroughs: Finding problems without the
noise. Int. J. Human-Comput. Interact. 9, 3 (1997), 213–234.
events of interest (such as error messages) have 10. Shneiderman, B. Designing the User Interface, 3rd ed.. Addison-Wesley,
occurred. Reading, MA, 1997.
Many aspects of usability can best be studied by 11. Thomas, C. and Bevan, N. (Eds.). Usability Context Analysis: A Practi-
cal Guide. National Physical Laboratory, Teddington, UK, 1996.
querying the users. This is especially true for issues
related to the subjective satisfaction of the users and
For more literature and pointers, see www.basiswissen-multimedia.at.
their possible anxieties, which are difficult to measure
objectively. Questionnaires are useful for studying ANDREAS HOLZINGER ([email protected]) is an
how end users use the system and their preferred fea- associate professor of information processing at Graz University,
tures, but need some experience to design. They are Austria, and is a member of the European Research Consortium for
Informatics and Mathematics (ERCIM) Working Group “User
an indirect method, since this technique does not Interfaces for All” (UI4ALL).
study the actual user interface: it only collects the
opinions of the users about the interface. One cannot Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
always take user statements at face value. Data about for profit or commercial advantage and that copies bear this notice and the full cita-
tion on the first page. To copy otherwise, to republish, to post on servers or to redis-
people’s actual behavior should have precedence over tribute to lists, requires prior specific permission and/or a fee.
people’s claims of what they think they do.
A simpler form of questionnaire is the interview. © 2005 ACM 0001-0782/05/0100 $5.00

74 January 2005/Vol. 48, No. 1 COMMUNICATIONS OF THE ACM

View publication stats