Tests

The design of graphical systems and Web pages, and their screens, is a complicated process. As has been
shown, in both a host of factors must be considered. In graphical systems among the many design
elements are the types of windows used, the way the windows are organized, what controls are
selected to collect and present information, and the way the controls are organized within one window
and between several windows. Web page design factors include the proper integration of text, graphics,
navigation links, and controls, page size, writing for simplicity and clarity, the characteristics of browsers
and monitors, and accessibility requirements. In both design processes numerous design trade-offs will
be made. Also, some design decisions may be based on skimpy data and reflect the most educated guess
possible at the moment. Finally, the implications for some design decisions may not be fully appreciated
until the results can be seen. To wait until after a system has been implemented to uncover and correct
any system usability deficiencies can be aggravating, costly, and time-consuming for both users and
developers. Indeed, after implementation many problems may never be corrected because of time
constraints and costs. To minimize these kinds of problems, and ensure usability, interfaces must be
continually tested and refined before they are implemented. What follows is an overview of the testing
process and the role it plays in design. Its purpose is to provide an awareness of the testing procedures
and methods, and to summarize some basic testing guidelines. Testing steps to be reviewed are:
Identifying the purpose and scope of testing.

 Understanding the importance of testing. Developing a prototype

 Developing the right kind of test plan.
 Designing a test to yield relevant data.
 Soliciting, selecting, and scheduling users to participate.
 Providing the proper test facility.
 Conducting tests and collecting data.
 Analyzing the data and generating design recommendations.
 Modifying the prototype as necessary.
 Testing the system again. Evaluating the working system.

The Importance of Usability Testing

A thorough usability testing process is important for many reasons, including all of the following.
Developers and users possess different models. As discussed earlier, developers and users have different
expectations and levels of knowledge. Specialized knowledge possessed by the developers enables them
to deal with complex or ambiguous situations on the basis of context cues not visible to the users.
Developers also frequently use terminology that does not always match that of the users. Developer’s
intuitions are not always correct.The intuition of designers, or anyone for that matter, no matter how
good or bad they may be at what they do, is error prone. This is illustrated by the previously reported
Tullis and Kodimer (1992) study evaluating several screen-based controls. They found that
programmers’ predictions of control usage speed correlated only .07 with actual measured speeds. They
There is no average user.We all differ—in looks, feelings, motor abilities, intellectual abilities, learning
abilities and speeds, device-based control preferences, and so forth. In a keyboard data entry task, for
example, the best operators will probably be twice as fast as the poorest and make 10 times fewer
errors. The design must permit people with widely varying characteristics to satisfactorily and
comfortably learn and perform the task or job. It’s impossible to predict usability from appearance.Just
as it is impossible to judge a person’s personality from his or her looks, it’s impossible to predict a
system’s usability from its appearance. Design standards and guidelines are not sufficient.Design
standards and guidelines are an important component of good design, laying the foundation for
consistency. But design standards and guidelines often fall victim to trade-offs. They also cannot address
all the countless design element interactions that occur within a completed system. Informal feedback is
inadequate. Informal feedback is a hit-and-miss proposition. Parts of the system may be completely
overlooked; significant problems in other parts may never be documented. Products’ built-in pieces
almost always have system-level inconsistencies. This is a normal and expected result when different
developers work on different aspects of a system. We might also say that developers differ—there is no
average developer. Problems found late are more difficult and expensive to fix. Unless they’re really
severe, they may never be fixed. Problems fixed during development mean reduced support costs later.
Support costs are directly proportional to the usability problems that remain after development. The
more problems, the higher the support costs. Advantages over a competitive product can be achieved.
Many products can do something. The most successful products are those that permit doing something
easily. also found that programmers’ predictions of user control preferences correlated only .31 with
actuality. Intuition is too shallow a foundation on which to base design decisions.

Scope of Testing

Testing should begin in the earliest stages of product development and continue throughout the
development process. It should include as many of the user’s tasks, and as many of the product’s
components, as reasonably possible. Always involve all members of the design team in the testing to
ensure a common reference point for all. Involving all also permits multiple insights into the test results
from the different perspectives of team members.

Prototypes

A prototype is primarily a vehicle for exploration, communication, and evaluation. Its purpose is to
obtain user input in design, and to provide feedback to designers. Its major function is the
communicative role it plays, not accuracy or thoroughness. A prototype enables a design to be better
visualized and provides insights into how the software will look and work. It also aids in defining tasks,
their flow, the interface itself, and its screens

Hand Sketches and Scenarios

■ Description: — Screen sketches created by hand. — Focus is on the design, not the interface
mechanics. — A low-fidelity prototype. ■ Advantages: — Can be used very early in the development
process. — Suited for use by entire design team. — No large investment of time and cost. — No
programming skill needed. — Easily portable. — Fast to modify and iterate. — A rough approximation
often yields more substantive critical comments. — Easier to comprehend than functional specifications.
— Can be used to define requirements. ■ Disadvantages: — Only a rough approximation. — Limited in
providing an understanding of navigation and flow. — A demonstration, not an exercise. — Driven by a
facilitator, not the user.

Advantages.Hand-drawn sketches of screens can be easily developed and used very early in the
development process. Many usability problems can then be identified and corrected quickly. Sketches
are also suitable for use by the entire development team, giving everyone a sense of what the real
design issues are. Sketches require no large investment of time and money, and they are portable,
placing few restrictions on where the testing may occur. Sketches can also be quickly modified and
iterated, as many times as necessary. Because there has been no emotional investment in “code” and
the status quo, there is no necessity for team members to defend something already created from hard
work. Screen sketches are rough approximations, and rough approximations often yield more
substantive suggestions or critical comments than actual screen-drawn versions. Their “draft” or
“unpolished” look greatly softens the attitude that everything is cast in concrete. Sketches can also be
used to define a system’s requirements.

Disadvantages. Since hand-drawn sketches are rough approximations, they can only suggest the final
layout of the interface. They are limited in helping understand system navigation and flow, and are a
demonstration device driven by a facilitator, with the user assuming a more passive role. They are
usually restricted to the most common user tasks. As a result, they possess limited usefulness for
usability testing.

Kinds of Tests

A test is a tool that is used to measure something. The “something” may be: Conformance with a
requirement. Conformance with guidelines for good design. Identification of design problems. Ease of
system learning. Retention of learning over time. Speed of task completion. Speed of need fulfillment.
Error rates. Subjective user satisfaction. A test is usually formal; it is created and applied intentionally
and with a purpose. It is usually based upon some kind of criteria, an understanding of what a good
result would be. Several testing techniques, at varying levels of sophistication and cost, are available to
exercise the system.

Guidelines Review

■ Description: — A review of the interface in terms of an organization’s standards and design guidelines.
■ Advantages: — Can be performed by developers. — Low cost. — Can identify general and recurring
problems — Particularly useful for identifying screen design and layout problems. ■ Disadvantages: —
May miss severe conceptual, navigation, and operational problems.

Description.Aguidelines reviewis an inspection of an interface’s navigation and screen design and layout
in the context of an organization’s standards and design guidelines. A checklist summarizing a system’s
standard or guideline document is prepared and is used as the basis for the comparison. Failure to
comply with a guideline or standard indicates that a design modification may be necessary. Advantages.
Software developers can perform this kind of test. Its advantages include its low cost and its ability to
identify recurring and general problems. It is particularly useful in identifying screen design and layout
problems. Disadvantage. Because this review tends to be static in nature, it may miss severe conceptual,
navigation, and operational problems.

Heuristic Evaluation

■ Description: — A detailed evaluation of a system by interface design specialists to identify problems. ■

Advantages: — Easy to do. — Relatively low cost. — Does not waste user’s time. — Can identify many
problems. ■ Disadvantages: — Evaluators must possess interface design expertise. — Evaluators may
not possess an adequate understanding of the tasks and user communities. — Difficult to identify
systemwide structural problems. — Difficult to uncover missing exits and interface elements. — Difficult
to identify the most important problems among all problems uncovered. — Does not provide any
systematic way to generate solutions to the problems uncovered.

Heuristic Evaluation Effectiveness

One of the earliest papers addressing the effectiveness of heuristic evaluations was by Nielsen (1992).
He reported that the probability of finding a major usability problem averaged 42 percent for single
evaluators in six case studies. The corresponding probability for uncovering a minor problem was only
32 percent. He also found that the actual number of located minor problems exceeded the number of
major problems uncovered by about a 3:1 ratio (152 minor problems versus 59 major problems). Minor
design problems are normally more prevalent in design (for example, inconsistent use of typefaces) and,
consequently, show up in greater numbers in this kind of evaluation. Minor problems, such as
inconsistencies, are more susceptible to identification by inspection, and may not be noticed in testing
with actual users, who are focusing on a task or procedure. Nielsen suggests that severity ratings are
especially useful in prioritizing minor problems

0 = I don’t agree that this is a usability problem at all.

1 = A cosmetic problem only. Need not be fixed unless extra time is available.

2 = A minor usability problem. Fixing should be given a low priority.

3 = A major usability problem. Important to fix and should be given a high priority.

4 = A usability catastrophe. Imperative to fix before the product can be released.