Usability Analysis

and Testing
De Castro, Edzel Gamlanga, Prince
Del Rosario, Niell Gonzales, Jean
Esguerra, Reiner Hipolito Jr, Willy Bhoy
Fabrigas, Diomedes Quiroz, Gabriel
Gaddi, Marc Yanga, Jonas
Introduction
Usability analysis and testing evaluate the efficiency and effectiveness of products for
different groups of people. Efficiency is measured by time and effort required, while
effectiveness refers to how well users achieve their goals. Usability studies also evaluate
satisfaction and performance, making usability measures closely related to product quality.
However, they primarily focus on user interfaces, rather than internal functions, affecting
product functionality.
Focusing on the interface is crucial as poorly designed interfaces can cause
usability problems, even if the product functions correctly. Usability
analysis and testing can identify issues with both functionality and the
interface. This approach allows usability analysis and testing to be
conducted in parallel with other design activities during product
development, ensuring a seamless user experience. For example, Web-
based and telephone interfaces for electronic banking systems can be
developed parallel to the transaction processing database management
system (DBMS), allowing the interface designer to evaluate usability
without needing to know all DBMS function details.

What is a INTERFACE?
> a point where two systems, subjects,
organizations, etc. meet and interact

> a device or program enabling a user to

communicate with a computer
Introduction
Ergonomists and usability engineers use various approaches to evaluate product usability,
including task analysis. These methods identify and analyze tasks and subtasks for each
product function, with some being analytical, while others focus on observation or
empirical data collection.
Advantages and Disadvantages of Existing Usability Evaluation Techniques
Method Example Tools/Techniques General Line Advantages Disadvantages

Analytic/ Cognitive task analysis GOMS Used early in usability design life Useful in making accurate design Narrow in focus lack of specific
Theory based cycle for production of expert user decisions early in the usability life diagnostic output to guide design;
performance cycle without the need for a broad assumption on user’s
prototype or costly user testing experience (expert) and cognitive
processes; results may differ based
on the evaluator’s interpretation of
the task

Expert Design walk-throughs Used early in design life cycle Strongly diagnostic; can focus Even the best evaluations can
Evaluation Heuristic evaluations to identify theoretical problems on entire system; high potential mis significant usability issues;
Process/system checklists that mat pose actual practical return in terms of number of results are subject to evaluator
Group Evaluation problems usability issues identified; can bias; does not capture real user
assist in focusing observational behavior
evaluation

Observational Direct observation Used in iterative design stage for Quickly highlights usability issues; Observation can affect user
Evaluation Video Verbal protocols problem identification verbal protocols provide performance with the system;
Computer logging significant insights; provides rich analysis of data can be time and
Thinking aloud techniques Field qualitative data resource consuming
evaluation Ethnographic studies
Facilitated free play
Advantages and Disadvantages of Existing Usability Evaluation Techniques
Method Example Tools/Techniques General Line Advantages Disadvantages

Survey Questionnaires Used any time in the design life Provides insights into user’s User experience important;
Evaluation Structured interviews cycle to obtain information on options and understanding of the possible user response bias
Ergonomic checklists user’s preferences and system; can be diagnostic; rating (e.g., only dissatistifed users
Focus groups perception of a system scales can provide quantitative respond); response rates can
data; can gather data from large below; possible interviewer
subject people bias; analysis of data can be
time and resource consuming;
evaluator may not be using
appropriate checklist to suit the
situation

Experimental Quantitative measures Used for competitive analysis in Powerful and prescriptive method; Experiment is generally time and
evaluation Alternative design comparison final testing of the system provides quantitative data; can resource consuming; focus can be
provide a comparison of narrow; tasks and evaluative
alternatives; reliability and validity environment can be contrived;
generally good results difficult to generalize
Scenario 1
Mr. Smith is sitting next to Mr. Jones, watching him work a high-fidelity prototype of a Web
browser for personal digital assistants (PDAs). Mr. Jones is the third person that Mr. Smith has
watched performing these tasks with this version of prototype. Mr. Smith is not constantly
reminding Mr. Jones to talk while he works but is counting on his proximity to Mr. Jones to
encourage verbal expressions when Mr. jones encounters any difficulty in accomplishing his
current task. Mr. Smith takes written faltering in his use of the application (e.g. exploring
menus in search of a desired function). Later that day he will notes to develop problem reports
and, in consultation with the development team, will work on recommendation for product
changes that should eliminate or reduce the impact of the reported problems. When a new
version of the prototype is ready, he will resume testing.
Scenario 2
Dr. White is watching Mr. Adams work with a new version of a word-processing application. Mr. Adams is
working alone in a test cell that look almost exactly like an office, except for the large mirror on one wall and the
two video cameras overhead. He has access to a telephone and a number to call if he encounters a difficulty that he
cannot overcome. If he places such a call, Dr. White will answer and provide help modeled on the types of help
provided at the company’s call centers. Dr. White can see Mr. Adams through the one-way glass as she coordinates
the test. She has one assistant working the video cameras for maximum effectiveness and another who is taking
time-stamped notes on a computer (coordinated with the video time stamps) as different members of the teams
notice and describe different aspects of Mr. Adams’s task performance. Software monitors Mr. Adams’s computer,
recording all keystrokes and mouse movements. Later that day, Dr. White and her associates will put together a
summary of the task performance measurements for the tested version of the application, nothing where the
performance measurements do not meet the test criteria. They will also create a prioritized list of problems and
recommendations, along with video clips that illustrate key problems, for presentation to the development team at
their weekly status meeting.
Usability testing involves problem discovery and resolution in scenario 1
and task performance measurement in scenario 2. Dr. White's teams
recognize that examining a list of problems is insufficient to determine if
they have met usability performance goals. They also need to provide
guidance to product developers on how to redesign the product (Chapnis,
1981; Norman, 1983). J. Karat (1997, p.693) emphasizes that the end goal
of evaluation is a redesigned system that meets usability objectives,
ensuring users achieve their goals and are satisfied with the product.
Goals of Usability Testing
Usability testing aims to enhance the usability of products by helping developers create more usable ones.
It can be categorized into formative and summative goals. Formative goals involve identifying and
rectifying usability deficiencies in computer-based and electronic equipment and their support materials
before release. Summative goals involve setting specific, quantitative usability goals for the product early
in the process and designing to meet those goals.

The goals of different tests can vary in practice, with a focus on measurement leading to more formal
testing and problem discovery resulting in less formal testing. There are two common types of
measurement tests: comparison against objectives and comparison of products.
Task Analytic Methods
The task analytic method uses cognitive task analysis to identify potential usability issues by decomposing
tasks into subtasks and analyzing performance sequences, typically focusing on (1) documenting the
knowledge and information required to do the task, (2) documenting how and when the needed information
is provided, and (3) predicting the time required

Cognitive task analysis methods like GOMS, • GOMS= Goals, Operators, Methods
NGOMSL, and MHP are used in product and Selection rules

usability-related applications to develop • NGOMSL= Natural GOMS

hierarchical descriptions of user goals, Language

emphasizing goal attainment. • MHP= Model Human Processors

Hierarchical descriptions of knowledge required for tasks can predict time and learning transfer between
products, while GOMS and NGOMSL descriptions quantitatively predict the learning needed for operating
a product. GOMS or NGOMSL analysis enables the evaluation of interface designs before system
development through various qualitative measures. These include the following:

1 Naturalness of the design- Do the goals and subgoals make sense to a new user of the system?

2 Completeness of the design-Is there a method for each goal or subgoal?

3 Cleanliness of the design-If there is more than one method for attaining a goal, is there a clear and easily
stated selection rule for choosing an appropriate method?

4 Consistency of design-Are methods for attaining similar goals consistent?

5 Efficiency of design-Are the most important goals accomplished by quick methods?

Task Analytic Methods
The MHP is a theoretically grounded approach that provides time and expected ranges for
simple cognitive tasks early in system design. It allows for early prediction without testing
or physical prototypes, unlike GOMS or NGOMSL which do not directly show problem
existence or fix solutions.

The analyst must interpret and test the GOMS or NGOMSL descriptions, as they often align
with a skilled user's task. Expert evaluation or user testing can be more efficient in
identifying errors and usability issues for less skilled users.