* HCI Evaluations

* Ethnographic research
* Participatory design
* Focus group research
* Surveys
* Walk throughs
* Open and closed card sorting
* Paper prototyping
* Expert or Heuristic Evaluations
* Usability Testing
* Follow-up studies

*Techniques for
building in usability?
* Qualitative method

* Observing users where they would use the

product

* Gather data about target (user, tasks, goals)

* Personas, profiles, scenarios, task descriptions

*Ethnographic
Research
* Users are on board with the design team
tapping user’s knowledge, skill set and
emotional reactions to design

* Workshops are arranged where users,

designers, and developers work together on an
aspect of design

*Participatory Design
* Used at the early stages of a project to
evaluate preliminary concepts with rep. users
(brain storming sessions)

* “Proof of Concept” preview

* Paper & pencil drawings, story boards, plastic

models are used as tools

*Focus Group
Research
* Used at initial stage of product implementation

* Checking preferences of a broad base of users

about an existing or potential product

* Open ended / closed ended questions

* Quantitative data

*Surveys
* Often used at prototype stage

* Explore how user might fare with a product by

envisioning user’s route through an early
concept or prototype

* Used to develop work guides

* Participants assume specific roles

*Walk throughs
* Used to design “findability” of content or
functionality

* Participants are given cards showing content

without titles or categories and have the users
do the naming (open)

* Participants use preexisting categories and ask

users to sort content or functions into those

*Open and closed

card sorting
* Users are shown aspect of a product on paper
and ask questions about it

* Expert mock up responses on a graph paper

*Paper Prototyping
* Review of a product or a system, usually by an
expert

* “double specialist”

*Expert or Heuristic
Evaluations
* Techniques to collect empirical data while
observing end users using the product

* Formal tests as true experiments 

confirm/refute specific hypothesis

* Iterative cycles of tests  expose usability

deficiencies and gradually shaping the product
in question

*Usability Testing
* Usability Testing
* Informing Design

* Eliminating Design Problems and Frustration

* Improving Profitability

*Why testing?
(goals)
* Hypothesis must be formulated
* Randomly chosen participants must be assigned
to experimental conditions
* Tight controls
* Control groups
* Sample must be of sufficient size to measure
statistically significant differences

*Basics of
methodology
(classical approach)
* Development of research questions or test objectives rather than
hypothesis

* Use of a representative sample of end users which may or may not be

randomly chosen

* Representation of the actual work environment

* Observation of the end users who either use or review a representation of

the product

* Controlled and sometimes extensive interviewing and probing of the

participants by the test moderator

* Collection of quantitative and qualitative performance and preference

measures

* Recommendation of improvements to the design of the product

*Basic Elements of
Usability Testing
* Testing is always an artificial situation

* Test results do not prove that a product works

* Participants are rarely fully representative of

the target population

* Testing is not always the best technique to use

*Limitations
* Exploratory (formative)

* Assessment (summative)

* Validation (verification)

* Comparison (integral part of above three)

*Types of test
* When: Quite early in development cycle
* Objective: to examine the effectiveness of
preliminary design
* Example: Web designer interested in identifying how
well the interface:
* Supports users’ tasks within a goal
* Communicates the intended workflow
* Allows the user to navigate from screen to screen and
within a screen

*Exploratory or
Formative Study
* Task-oriented user perspective:
* Overall organization of subject matter
* Whether to use a graphic or verbal approach
* How well the proposed format supports
findability
* Anticipated points of assistance and messaging
* How to address reference information

*Exploratory or
Formative Study(II)
* Typical user oriented questions:
* What do users conceive and think about using the product?
* Does the product’s basic functionality have value to the user?
* How easily do users make inferences about how to use this
user interface, based on their previous experience?
* What type of prerequisite information does a person need to
use the product?
* Which functions of the product are “walk up and use” and
which will probably require either help of written
documentation?
* How should the table of contents be organized to
accommodate both novice and experienced users?

*Exploratory or
Formative Study(III)
* Overview of methodology
* Prototypes (horizontal representation)
* Walk through (vertical representation)

*Exploratory or
Formative Study(IV)
* When: either early or midway into the
product development cycle,
usually after the fundamental or
high level design.
* Objective: to expand the findings of the
exploratory test by evaluating
usability of lower level operations
and aspects of the product.
* Example: how well a user can actually
perform a full blown realistic task.

*Assessment or
Summative Test
* Methodology:
* The user will always perform tasks rather than
simply walking through and commenting upon
screens, pages and so on.
* The test moderator will lessen his or her
interaction with the participant because there is
less emphasis on thought processes and more on
actual behaviors.
* Quantitative measures will be collected.

*Assessment or
Summative Test(II)
* When Late in deployment cycle

* Objective Measure usability of a product

against established benchmarks.
In case of verification test, to
confirm that problems
discovered earlier have been
removed and new ones have not been
introduced.
* Example Comparison against established
benchmarks.

*Validation or
Verification Test
* Methodology
* Prior to test, standards or benchmarks for the
tasks are either developed or identified.
* Participants are given tasks to perform with
either very little or no interaction with the test
moderator.
* Collection of quantitative data is the central
focus, although reasons for substandard
performance are identified.

*Validation or
Verification Test (II)
* When not associated with any specific
point in the product life cycle.

* Objective Used in conjunction of the first

three tests. It is used to
compare two or more designs.

* Example Compare two different interface

styles, or the current design of
manual, etc.

*Comparison Test
* Methodology
* The design team is forced to stretch its
conceptions of what will work rather than just
continuing along in predictable pattern.

* During the test, the participant is forced to

really consider why one design in better and
which aspects make it do so.

*Comparison Test (ii)