Chapter 4

COGNITIVE ASPECTS
 Overview
• What is cognition?
• Why it is important to understand in HCI
• Describe how cognition has been applied
to interaction design
• Explain what are mental models and how
to elicit them
• Cover relevant theories of cognition
www.id-book.com
 What is cognition?
• Thinking, remembering, learning,
daydreaming, decision-making, seeing,
reading, talking, writing…
• Ways of classifying cognition at a higher
level:
 Experiential vs. reflective cognition (Norman,
1993)
 Fast vs slow thinking (Kahneman, 2011)

www.id-book.com
 Which involves fast vs slow thinking?

• 2+2=
• 21 × 29 =
• What color eyes do you have?
• How many colors are there in the rainbow?
• How many months in the year have 31 days?
• What is the name of the first school you
attended?
www.id-book.com
How can understanding cognition help?

• Provides knowledge about what users can

and cannot be expected to do
• Identifies and explains the nature and
causes of problems that users encounter
• Provides theories, modeling tools,
guidance, and methods that can lead to
the design of better interactive products

www.id-book.com
 Cognitive processes
• Attention
• Perception
• Memory
• Learning
• Reading, speaking and listening
• Problem-solving, planning, reasoning and
decision-making
www.id-book.com
 Attention
• Selecting things on which to concentrate at a point in
time from the mass of stimuli around us
• Allows us to focus on information that is relevant to
what we are doing
• Involves audio and/or visual senses
• Focused and divided attention
 Enables us to be selective in terms of the mass of competing
stimuli, but limits our ability to keep track of all events
• Design recommendation
 Information at the interface should be structured to capture
users’ attention, for example, use perceptual boundaries
(windows), color, reverse video, sound, and flashing lights
www.id-book.com
Activity: Find the price for a double room
at the Quality Inn in Pennsylvania

www.id-book.com
Activity: Find the price of a double room at the
Holiday Inn in Columbia

www.id-book.com
 Activity
• Tullis (1987) found that the two screens produced
quite different results
 1st screen: Took an average of 5.5 seconds to search
 2nd screen: Took 3.2 seconds to search

• Why, since both displays have the same density of

information (31percent)?

• Spacing
 In the 1st screen, the information is bunched up
together, making it hard to search
 In the 2nd screen, the characters are grouped into
vertical categories of information making it easier

www.id-book.com
 Multitasking and attention
• Is it possible to perform multiple tasks without one
or more of them being detrimentally affected?
• Multitasking can cause people to lose their train of
thought, make errors, and need to start over
• Ophir et al. (2009) compared heavy vs light
multitaskers
 Heavy multitaskers were more prone to being
distracted than those who infrequently multitask
 Heavy multitaskers are easily distracted and find it
difficult to filter irrelevant information
www.id-book.com
 Multitasking experiment
• Lotteridge et al. (2015) conducted another
study involving writing an essay under two
conditions: relevant or irrelevant
information
 Heavy multitaskers were easily distracted but
able to put this to good use if the distracting
sources were relevant to the task in hand
 Irrelevant information was found to impact
task performance negatively

www.id-book.com
 Multitasking at work
It is increasingly common for workers to
multitask
• For example, hospital workers have to attend
to multiple screens in an operating room that
provide new kinds of real-time information
• This requires clinician’s constant attention to
check if any data is unusual or anomalous
• Need to develop new attention and scanning
strategies

www.id-book.com
Is it OK to use a phone when driving?

www.id-book.com
 No!
• Driving is very demanding
• Drivers are prone to being distracted
• There is a significant chance of causing accidents 
• Drivers’ reaction times are longer to external events when talking
on the phone in a car (Caird et al., 2018)
• Drivers using their phones rely more on their expectations about
what is likely to happen next as conducting a conversation takes
up their attention
• Response time is slower to unexpected events (Briggs et al., 2018)
• Drivers often try to imagine what the other person’s face is like−
the person to whom they are speaking
 Doing so competes with the processing resources needed to enable
them to notice and react to what is in front of them

www.id-book.com
Are hands-free phones safer to use when
driving?
• No, as same type of cognitive processing is happening when
talking
• The same thing happens when talking with front seat
passenger
 But both can stop in mid-sentence if a hazard is spotted allowing the
driver to switch immediately to the road
 So, it’s less dangerous talking to a front seat passenger than a
remote person
 A remote person on the end of a phone is not privy to what the driver
is seeing and will carry on the conversation when there is a hazard
 This makes it difficult for the driver to switch all their attention to the
road

www.id-book.com
Design implications for attention
• Context: Make information salient when it needs
to be attended to at a given stage of a task
• Use techniques to achieve this:
 For example, color, ordering, spacing, underlining,
sequencing, and animation
• Avoid cluttering visual interfaces with too much
information
• Consider designing different ways to support
effective switching and returning to an interface
www.id-book.com
 Perception
• How information is acquired from the
world and transformed into experiences
• Obvious implication is to design
representations that are readily
perceivable, for instance:
 Text should be legible
 Icons should be easy to distinguish and
read

www.id-book.com
Is color contrast good? Find Italian

www.id-book.com
Are borders and white space better?
Find French

www.id-book.com
 Activity
• Weller (2004) found people took less time to
locate items for information that was grouped
 Using a border (2nd screen) compared with using
color contrast (1st screen)

• Some argue that too much white space on

web pages is detrimental to search process
 Makes it hard to find information

• Do you agree?
www.id-book.com
Activity: Which is the easiest to
read and why?

What is the time? What is the time?

What is the time? What is the time?

What is the time?

www.id-book.com
 Design implications
• Icons should enable users to distinguish their
meaning readily
• Bordering and spacing are effective visual ways
of grouping information
• Sounds should be audible and distinguishable
• Research proper color contrast techniques
when designing an interface:
 Yellow on black or blue is fine
 Yellow on green or white is a no-no
• Haptic feedback should be used judiciously
www.id-book.com
 Memory
• Involves recalling various kinds of knowledge that allow
people to act appropriately
 For example, recognizing someone’s face or remembering
someone’s name
• First encode and then retrieve knowledge
• We don’t remember everything−it involves filtering and
processing what is attended to
• Context is important as to how we remember (that is,
where, when, how, and so on)
• We recognize things much better than being able to recall
things
• We remember less about objects that we have
photographed than when we observe them with the naked
eye (Henkel, 2014)
www.id-book.com
 Processing in memory
• Encoding is first stage of memory
 Determines which information is attended to in the environment
and how it is interpreted
• The more attention paid to something…
• The more it is processed in terms of thinking
about it and comparing it with other knowledge…
• The more likely it is to be remembered
 For example, when learning about HCI, it is much better to
reflect upon it, carry out exercises, have discussions with others
about it, and write notes than just passively read a book, listen to
a lecture or watch a video about it

www.id-book.com
 Context is important
• Context affects the extent to which information
can be subsequently retrieved
• Sometimes it can be difficult for people to recall
information that was encoded in a different
context:
 “You are on a train and someone comes up to you
and says hello. You don’t recognize him for a few
moments, but then realize it is one of your neighbors.
You are only used to seeing your neighbor in the
hallway of your apartment building, and seeing him
out of context makes him difficult to recognize initially”

www.id-book.com
 Activity
• Try to remember the dates of your grandparents’
birthday
• Try to remember the cover of the last two books you
read
• Which was easiest? Why?
• People are very good at remembering visual cues
about things
 For instance, the color of items, the location of objects and
marks on an object
• They find it more difficult to learn and remember
arbitrary material
 For example, birthdays and phone numbers
www.id-book.com
 Recognition versus recall
• Command-based interfaces require users to
recall from memory a name from a possible
set of 100s of names
• Graphical interfaces provide visually-based
options (menus, icons) that users need only
browse through until they recognize one
• Web browsers provide tabs and history lists
of visited URLs that support recognition
memory
www.id-book.com
The problem with the classic ‘7,+
or − 2’
• George Miller’s (1956) theory of how much
information people can remember
• People’s immediate memory capacity is very
limited to 7, + or − 2
• Has been applied in interaction design when
considering how many options to display
• But is it a good use of a theory in HCI?
• Is it helpful?
www.id-book.com
 When creating an interface, should the
designer…
• Present only 7 options on a menu
• Display only 7 icons on a tool bar
• Have no more than 7 bullets in a list
• Place only 7 items on a pull down menu
• Place only 7 tabs on the top of a website page?
• Not necessarily…

www.id-book.com
 The reason is…
• People can scan lists of bullets, tabs, and menu
items for the one they want
• They don’t have to recall them from memory,
having only briefly heard or seen them
• So you can have more than nine at the interface
 For instance, history lists of websites visited
• Sometimes a small number of items is good
 For example, smart watch displays
• Depends on task and available screen estate

www.id-book.com
 Personal Information management
Is a growing problem for many users:
• They accumulate a vast numbers of documents,
images, music files, video clips, emails,
attachments, bookmarks, and so forth
• Where and how to save them all; then remembering
what they were called and where to find them again
• Naming most common means of encoding them
• But can be difficult to remember, especially when
you have 10,000s
• How might such a process be facilitated taking into
account people’s memory abilities?

www.id-book.com
Personal Information management
• Bergman and Whittaker, three interdependent processes
model (2016) to help people manage their stuff:
I. How to decide what stuff to keep
II. How to organize it when storing
III. Which strategies to use to retrieve it later

• Most common approach is to use folders and naming

• Strong preference for scanning across and within
folders when looking for something
• Search engines only helpful if you know the name of
the file
• Smart search engines help with listing relevant files
for partial name or when type in first letter

www.id-book.com
Apple’s Spotlight search tool

www.id-book.com
 Memory load
• Online/mobile and phone banking now require users to provide
multiple pieces of information to access their account
 For instance, ZIP code, birthplace, a memorable date, first school
attended
 Known as multifactor authentication (MFA)
• Why?
 Increased security concerns
• Password managers, such as LastPass, have been developed
that require only one master password
 Reduces stress and memory load on users
• Passwords could become extinct with the widespread use of
biometrics and computer vision algorithms

www.id-book.com
 Digital Forgetting
• When might you wish to forget something that is
online?
 When you break up with a partner
 Emotionally painful to be reminded of them through shared
photos, social media, and so on.

• Sas and Whittaker (2013) suggest ways of harvesting

and deleting digital content
 For example, making photos of ex into an abstract collage
 Helps with closure

www.id-book.com
 Memory aids
• SenseCam, developed by Microsoft Research
Labs (now Autographer)
 A wearable device that intermittently takes photos
without any user intervention while worn
 Digital images taken are stored and revisited using
special software
 Has been found to improve people’s memory,
especially those suffering from dementia
• Other aids include RemArc, which triggers
long-term memory using old BBC materials

www.id-book.com
SenseCam

www.id-book.com
 Design implications
• Reduce cognitive load by avoiding long and
complicated procedures for carrying out tasks
• Design interfaces that promote recognition
rather than recall
• Provide users with various ways of labelling
digital information to help them easily identify
it again
 For example, folders, categories, color, flagging,
and time stamping

www.id-book.com
 Learning
• Involves the accumulation of skills and knowledge
involving memory
• Two main types:
 Incidental learning (for example, recognizing people’s faces, what you did
today)
 Intentional learning (for instance, studying for an exam, learning to cook)
 Intentional learning is much harder!
 Many technologies have been developed to help (for example, multimedia,
animations, VR)

• People find it hard to learn by following instructions in a

manual
• People prefer to learn by doing

www.id-book.com
 Design implications
• Design interfaces that encourage
exploration
• Design interfaces that constrain and guide
learners
• Dynamically linking concepts and
representations can facilitate the learning
of complex material

www.id-book.com
 Reading, speaking, and listening
The ease with which people can read, listen, or
speak differs:
• Many prefer listening to reading
• Reading can be quicker than speaking or listening
• Listening requires less cognitive effort than
reading or speaking
• Dyslexics have difficulties understanding and
recognizing written words

www.id-book.com
 Applications
• Voice user interfaces allow users to interact with them
by asking questions
 For example, Google Voice, Siri, and Alexa

• Speech-output systems use artificially-generated

speech
 For instance, written text-to-speech systems for the visually
impaired
• Natural-language systems enable users to type in
questions and give text-based responses
 Such as, chatbots

www.id-book.com
 Design implications
• Speech-based menus and instructions should
be short
• Accentuate the intonation of artificially
generated speech voices
 They are harder to understand than human voices

• Provide opportunities for making text large on

a screen

www.id-book.com
 Problem-solving, planning,
reasoning, and decision-making
• All these processes involve reflective
cognition
 For example, thinking about what to do, what the options
are, and the consequences
• Often involves conscious processes,
discussion with others (or oneself), and the
use of artifacts
 Such as maps, books, pen and paper
• May involve working through different
scenarios and deciding which is best option
• Weighing up alternatives
www.id-book.com
 Design implications
• Provide information and help pages that
are easy to access for people who wish to
understand more about how to carry out
an activity more effectively (for example,
web searching)
• Use simple and memorable functions to
support rapid decision-making and
planning

www.id-book.com
 Dilemma
• The app mentality is making it worse for people to
make their own decisions because they are
becoming risk averse (Gardner and Davis, 2013)
 Instead, they now rely on a multitude of apps
 This makes them increasingly anxious
 They are unable to make decisions by themselves
 They need to resort to looking up info, getting other’s
opinions on social media, and comparing notes
• Do you agree?
• Did it happen to you when deciding which
university/school to attend?
www.id-book.com
 Cognitive frameworks
• These are used to explain and predict user
behavior at the interface
 Based on theories of behavior
 Focus is on mental processes that take place
 Also use of artifacts and representations
• Most well known are:
 Mental models
 Gulfs of execution and evaluation
 Distributed cognition
 External and embodied cognition
www.id-book.com
 Mental models
• Users develop an understanding of a system
through learning about and using it
• Knowledge is sometimes described as a
mental model:
• How to use the system (what to do next)
• What to do with unfamiliar systems or
unexpected situations (how the system works)
• People make inferences using mental
models of how to carry out tasks
www.id-book.com
 More mental models
• Craik (1943) described mental models as:
 Internal constructions of some aspect of the
external world enabling predictions to be made
• Involves unconscious and conscious
processes
 Imagery and analogies are activated
• Deep versus shallow models
 For example, how to drive a car and how it works

www.id-book.com
 Everyday reasoning and mental
models
(a) You arrive home on a cold winter’s night to a cold
house. How do you get the house to warm up as
quickly as possible? Set the thermostat to be at
its highest or to the desired temperature?
(b) You arrive home starving hungry. You look in the
fridge and find all that is left is an uncooked
pizza. You have an electric oven. Do you warm it
up to 375 degrees first and then put it in (as
specified by the instructions) or turn the oven up
higher to try to warm it up quicker?

www.id-book.com
 Heating up a room or oven that is
thermostat-controlled
• Many people when asked (a) choose the first option
• Why?
 They think it will heat the room up quicker
 General valve theory, where ‘more is more’ principle is
generalized to different settings (for instance, gas pedal, gas
cooker, tap, radio volume)
 But it is a wrong mental model for thermostats based on on-off
switch model
• Many people when asked (b) choose the first option
 Electric ovens work on the same principle as thermostats
• Most of us have erroneous mental models (Kempton,
1996)

www.id-book.com
 Erroneous mental models
• Lots of people hit the button for elevators
and pedestrian crossings at least twice
 Why? Think it will make the lights change faster or
ensure that the elevator arrives!

• What kinds of mental models do users have for

understanding how interactive devices work?
 Poor, often incomplete, easily confusable, based on
inappropriate analogies and superstition (Norman, 1983)

www.id-book.com
 How can UX be designed to help people
build better mental models?
• Clear and easy to use instructions
• Appropriate tutorials and contextual sensitive
guidance
• Provide online videos and chatbot windows
when needing help
• Transparency: to make interfaces intuitive to
use
• Affordances of what actions an interface allows
 For example, swiping, clicking, or selecting

www.id-book.com
 Gulfs of execution and evaluation
• The ‘gulfs’ explicate the gaps that exist between the user
and the interface
• The gulf of execution
 The distance from the user to the physical system

• The gulf of evaluation

 The distance from the physical system to the user

• Bridging the gulfs can reduce cognitive effort required to

perform tasks
• Can reveal whether interface increases or decreases
cognitive load and whether it is obvious what to do next
(Norman, 1986; Hutchins et al, 1986)
www.id-book.com
Bridging the gulfs

www.id-book.com
 Information processing
• Conceptualizes human performance in
metaphorical terms of information
processing stages

www.id-book.com
 Limitations
• Based on modeling mental activities that
happen exclusively inside the head
• Do not adequately account for how people
interact with computers and other devices
in real world

www.id-book.com
 Distributed cognition
• Concerned with the nature of cognitive
phenomena across individuals, artifacts,
and internal and external representations
(Hutchins, 1995)
• Describes these in terms of propagation
across representational state
• Information is transformed through different
media (computers, displays, paper, heads)

www.id-book.com
A cognitive system for ATC

www.id-book.com
 What’s involved
• The distributed problem-solving that takes
place
• The role of verbal and non-verbal behavior
• The various coordinating mechanisms that are
used (for example, rules and procedures)
• The communication that takes place as the
collaborative activity progresses
• How knowledge is shared and accessed
www.id-book.com
 External cognition
• Concerned with explaining how we interact with
external representations (such as maps, notes,
and diagrams)

• What are the cognitive benefits and what

processes involved

• How they extend cognition

• What technologies can we develop to help

people carry out complex tasks (for example,
learning, problem solving, and decision-
making)?
www.id-book.com
 Externalizing to reduce memory
load
• Examples include the use of diaries, reminders,
calendars, notes, shopping lists, to-do lists
 Written to remind us of what to do
• Post-its, piles, marked emails are used to:
 Where placed indicates priority of what to do
• External representations:
 Remind us that we need to do something (for example,
to buy something for mother’s day)
 Remind us of what to do (for instance, buy a card)
 Remind us when to do something (for example, send a
card by a certain date)

www.id-book.com
 Computational offloading
• When a tool is used in conjunction with an
external representation to carry out a
computation (for instance, pen and paper)
• Try doing the two sums below (a) in your
head, (b) on a piece of paper, and (c) with a
calculator.
234 ×456 = ??
CCXXXIIII×CCCCXXXXXVI = ???
• Which is easiest and why? Both are identical
sums

www.id-book.com
 Annotation and cognitive tracing
• Annotation involves modifying existing
representations through making marks
 For example, crossing off, ticking, and underlining

• Cognitive tracing involves externally

manipulating items into different orders or
structures
 For instance, playing Scrabble or cards

www.id-book.com
 Design implication
• Provide external representations at the
interface that can reduce memory load
and facilitate computational offloading
 For example, information visualizations
have been designed to allow people to
make sense and rapid decisions about
masses of data

www.id-book.com
 Embodied Interaction
• The practical engagement with the social and physical
environment (Dourish, 2001)
• Creating, manipulating and making meaning through our
interaction with things
• How our bodies and active experiences shape how we
perceive, feel, and think (Hornecker et al., 2017)
• They enable us to develop a sense of the world at both a
concrete and abstract level
• Can provide new ideas about interaction and better design
principles
 For example, we think with our bodies not through them (Kirsh, 2013)

www.id-book.com
 Summary
• Cognition involves many processes including
attention, memory, perception, and learning
• The way an interface is designed can greatly
affect how well users can perceive, attend, learn,
and remember how to do their tasks
• Theoretical frameworks, such as mental models
and external cognition, provide ways of
understanding how and why people interact with
products
• This can lead to thinking about how to design
better products

www.id-book.com
 In-depth activity
Write down how you think a contactless card or
smartphone app like Apple Pay works
• What information is sent between the card/smartphone and the card
reader when it is placed in front of it?
• What is the maximum amount you can pay for something using a
contactless card, Apple Pay or Google Pay?
• Why is there an upper limit?
• How many times can you use a contactless card or Apple/Google
Pay in a day?
• What happens if you have two contactless cards in the same
wallet/purse?
• What happens when your contactless card is stolen and you report
it to the bank? What does the bank do?

www.id-book.com