Got Data, Now What?

Analyzing Usability Study Results

Lynn Silipigni Connaway

June 26, 2005

Presented at the ALA 2005 Annual Conference

Chicago, IL
LAMA/MAES
Using Measurement Data for Library Planning
and Assessment Committee
Usability Testing: Why?

“Probably the best reason to test for usability is to eliminate

those interminable arguments about the right way to do
something. With human-factors input and testing, however,
you can replace opinion with data. Real data tend to make
arguments evaporate and meeting schedules shrink.”
(Fowler, 1998, Appendix, p. 283)
Usability Testing: Definition
 Degree to which a user can successfully learn and use a
product to achieve a goal
 Research methodology
• Evaluation
• Experimental design
 Observation and analysis of user behavior while users use a
product or product prototype to achieve a goal (Dumas and
Reddish, 1993, p.22)
 “User-centered design” process involving user from initial
design to product upgrade (Norlin and Winters, 2002)
 Approach is to be a servant to the users of a system NOT to
be subservient to technology (Gluck, 1998)
 Goal is to identify usability problems and make
recommendations for fixing and improving the design
(Rubin, 1994)
Usability Testing: Background
 Relatively new methodology (Norlin and Winters, 2002)
• Origins in aircraft design
• Traced back to marketing
• Development of a product
• Popular in 1980s with widespread access to computers
• Initiation of human computer interface usability studies
• Evolved from human ethnographic observation,
ergonomics, and cognitive psychology
• Qualitative and quantitative data
Usability Testing: Purpose

 Evaluation tool
 Identify problem areas
 “Determine the fit of the design to the intended users”
(Norlin and Winters, 2002, p. 5)
Usability Testing: Suitable Questions

 What is the best layout for a web page?

 How can you optimize reading from PDAs and small screen
interfaces?
 Which online fonts are the best?
 What makes an e-commerce site difficult to use?
 Can individual personality or cognitive skills predict Internet
use behavior?
 How can library collection holdings and library data be
represented geographically?
Usability Testing: Principles

 Keep the end user in mind

 Achieve superiority through simplicity
 Improve performance through design
 Refine and iterate (Norlin and Winters, 2002, p.10)
Usability Testing: Web Design Criteria

 Links must be consistent and predictable

 Group like things on the same page
 Be consistent with language
 Most important information should be on the first screen
 Provide keywords for quick reading/scanning
 Do not use animation or sounds
 Make links look like links
 Distinguish text from graphics
 Avoid jargon (Spool, 1999)
Usability Testing: Web Design Criteria

 Ten Usability Heuristics (Nielsen)

• Visibility of system status
• Match between system and the real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognize, diagnose, and recover from errors
• Help and documentation
Usability Testing: Web Design Criteria

 Goals for user-centered design

• Enable users to
• Achieve their particular goals and meet their needs
• Move quickly and with few errors
• Create a site that users like
• More likely to perform well on a product that provides
satisfaction
Usability Testing: Methodology

 Artificial environment (laboratory)

• Maintain more control
• May provide more specific data on a particular feature

 Natural environment
• Better holistic representation of real people doing real
work
Usability Testing: Methodology

 Four types of usability tests (Rubin, 1994, p. 31-46)

• Exploratory test – early product development
• Assessment test – most typical, either early or midway
in the product development
• Validation test – verification of product’s usability
• Comparison test – compare two or more designs; can be
used with other three types of tests
Usability Testing: Methodology

 Develop problem statements, objectives, and/or hypotheses

 Identify and select participants who represent target population
• May or may not be randomly selected
 Select test monitor/administrator
• Empathetic
• Impartial
• Good communicator
• Good memory
• Able to follow test structure
• Able to react spontaneously to situations that cannot be
anticipated
• Allow user time for task
• Don’t rescue the user
• Continue with the plan if mistakes occur
Usability Testing: Methodology

 Design test materials

• Screening questionnaire
• Provides user profile
• Ascertains pretest attitudes and background
information
• Provides information about participants’ previous
knowledge and experience
• Orientation script
• Describes the test to participants
• Aids in understanding the participants’ performance
• Data logger materials
• Data collection instrument for categorizing
participants’ actions
• Can note time to match with videotape recording
Usability Testing: Methodology

 Design test materials

• Non-disclosure and tape consent forms for legal
protection
• Task list
• List of actions participants will execute
• Desired end results
• Motives for performing task
• Actual observations monitor will record
• State of system
Usability Testing: Methodology

 Design test materials

• Posttest questionnaire
• All participants asked the same questions
• Gather qualitative information and precision
measurements
• Debriefing guide
• Structure and protocols for ending the session
• Participants explain things not apparent in actions
• Motive
• Rationale
• Points of confusion
Usability Testing: Methodology

 Test materials and equipment

 Conduct the test
• Represent the actual work environment
• Users are asked to think aloud
• Observe users while using or reviewing the product
• Probe
• Controlled and extensive questioning
• Collect quantitative and qualitative data and measures
• Record comments or questions about the product
• Observe and document users’ behaviors
Usability Testing: Methodology

 Debrief
 Analyze the data
• Diagnose and recommend corrections
• Categorize and identify problems with the product
• Identify solutions
• Qualitative analysis
• Textual notes from debriefing
• Read responses
• Summarize findings
Usability Testing: Methodology

 Analyze the data

• Quantitative analysis
• Questionnaires
• Screening
• Posttest
• Triangulation to validate findings
• Data from questionnaires, observations, screen
tracking software, comments, and open-ended
questions
Usability Testing: Interpret Data

 Interpret the data

• Five factors for benchmarking the usability of an
interface (Shneiderman and Plaisant, 2004)
• Time to learn
• Speed of performance
• Rate of errors
• Retention over time
• Subjective satisfaction
Usability Testing: Interpret Data

 Interpret the data

• Prioritize severity of problems
• Severity ratings (Zimmerman and Akerelrea, 2004)
• Time required to complete task
• Number of users who encountered problem
• Negative impact on users’ perception of the product
• Difficult if 70% of users cannot perform task
• Error criticality = Severity + Probability of Occurrence
(Rubin, 1994)
Usability Testing: Interpret Data

 Usable Web site: (Rubin, 1994)

• Usefulness
• Establish whether it does what the user needs it to do
• Effectiveness
• Ease of use to achieve the desired task
• Learnability
• Ease of learning application and moving from being a
novice to a skilled user
• User satisfaction
• User’s attitude about the site—how enjoyable it is to
use
Usability Testing: Report Results

 Executive summary
 Report
• Describe methodology
• Who, what, when, where, and how
• Describe how tests were conducted
• Profile users and describe sampling
• Detail data collection methods
• Succinctly explain the analysis
• Provide screen captures
• Include tables and graphs
• Provide examples
• Identify strengths and weaknesses
• Recommend improvements
Usability Testing: Making the Data Work

 Read report
 Determine what worked and what did not work
 Redesign product/system based upon findings
 May be necessary to conduct another usability test
Usability Testing: Limitations

 Two major limitations (Wheat)

• Reliability
• Testing of users who are not atypical users
• Individual variation within the test population
• Validity
• Test tasks, scenarios of the search processes, and
testing environment are not accurate
• Results not generalizable to the entire user
population
• Testing is always artificial (Rubin, 1994, p.27)
OCLC WorldMapTM
 Research prototype
• Test geographical representation of WorldCat holdings
• By country and date of publication
• For library collection assessment and comparison
• Complement the AAU/ARL Global Resources Network
project
• Geographically represent library statistical data from
UNESCO, ARL, Bowker, and others
• Number of libraries by type
• Expenditures by library type
• Number of volumes and titles
• Number of librarians
• Number of users
Usability Testing: OCLC WorldMapTM

 Review sample handouts

• Screening questionnaire
• Task list
• Posttest questionnaire
• Executive summary
Usability Testing: OCLC WorldMapTM

 Conducted informal usability tests

 Currently redesigning the interface
 Conduct second group of formal usability tests
 Make revisions prior to making publicly available
Questions and Discussion

[email protected]