Voice User Interfaces

• Involves a person talking with a spoken language app

– search engine, travel planner, phone service or chatbot

• Used most for inquiring about specific information

– flight times, weather

• Or for issuing command to machine

– smart TV, smart speaker

• Speech-to-text
• Call routing
• Also used by people with visual impairments
▪ speech recognition word processors, page scanners, web readers, and
home control systems
www.id-book.com 1
Have speech interfaces come of age?

www.id-book.com 2
 Modeling human conversations
• People often interrupt each other in a
conversation
▪ when ordering in a restaurant, rather than let the
waiter go through all of the options
• Speech technology has a similar feature called
‘barge-in’
▪ Users can choose an option before the system has
finished listing all of the options available

www.id-book.com 3
 Structuring VUI dialogs
• Directed dialogs are where the system is in control of the
conversation
▪ Where it asks specific questions and requires specific responses
• More flexible systems allow the user to take the
initiative:
▪ For example, “I’d like to go to Paris next Monday for two weeks.”
– but more chance of error, since caller might assume that the
system is like a human

• Guided prompts can help callers back on track

▪ For example, “Sorry I did not get all that. Did you say you
wanted to fly next Monday?”

www.id-book.com 4
 Voice assistants
• Have become popular in many homes
• Allow all to use rather than being single-use
• Can encourage social and emotional bonding
– Support families playing games
• Issues:
– limited mainly to answering questions and
responding to requests
– difficult to recognize children's speech
– don't always recognize who is talking in a group

www.id-book.com 5
 Research and design considerations
• How to design systems that can keep conversation
on track
▪ Help people navigate efficiently through a menu system
▪ Enable them to recover easily from errors
▪ Guide those who are vague in their requests for
information or services using prompts

• Type of voice actor (male, female, neutral, or dialect)

▪ Do people prefer to listen to and are more patient with a
female or male voice, a northern or southern accent?

www.id-book.com 6
 Pen-based devices
• Enable people to write, draw, select, and
move objects at an interface using light pens
or styluses
▪ Capitalize on the well-honed drawing skills
developed from childhood
• Smartpen uses a combination of ordinary ink
pen with a digital camera that digitally records
everything written with the pen on special
paper
www.id-book.com 7
 Smartpen - LiveScribe Echo 2

Source:
uk.livescribe.com/collections/smartpens/products/echo-2
www.id-book.com 8
 Advantages
• allows users to annotate existing documents
quickly and easily
• taking notes during meetings
• used to fill in paper-based forms that can
readily be converted to a digital record using
standard typeface
• used by remote teams to communicate and
work on the same documents
www.id-book.com 9
 Touchscreens
• Single touchscreens
– Select options by tapping on the screen
• Multi-touch surfaces
– more dynamic finger-tip actions
– they do so by registering touches at multiple
locations using a grid
– respond to more than one touch at the same time
supporting a variety of actions

www.id-book.com 10
A multi-touch surface

www.id-book.com 11
 Research and design considerations
• Provides fluid and direct styles of interaction for
certain tasks
– freehand and pen-based gestures
• Core design concerns: what types of interaction
techniques to use?
• choose from options
• typing on keyboard
• Gestures need to be learned for multi-touch, so a
small set of gestures for common commands is
preferable
• No tactile feedback

www.id-book.com 12
 Gesture-based systems
• Gestures involve moving arms and hands to
communicate
• Uses camera recognition, sensor, and
computer vision techniques
▪ Recognize people’s arm and hand gestures in a
room
▪ Gestures need to be presented sequentially to be
understood (compare with the way sentences are
constructed)
www.id-book.com 13
 Gestures used in the operating room

Recognizes core gestures for manipulating MRI or CT images using Microsoft Kinect
www.id-book.com 14
 Research and design considerations
• How does computer recognize user’s
gestures?
▪ Start and end points?
▪ Difference between deictic and hand
waving
• How realistic must the mirrored graphical
representation of the user be in order for
them to be believable?
www.id-book.com 15
 Haptic interfaces
• Provide tactile feedback
▪ applying vibration and forces to a person’s body
▪ using actuators that are embedded in their clothing or a
device they are carrying (smartphone)
• Vibrotactile feedback can be used to simulate the
sense of touch between remote people who want to
communicate
• Guide people when learning a musical instrument
• Ultrahaptics creates the illusion of touch in midair
using ultrasound to make the illusion of 3D shapes

www.id-book.com 16
 Realtime vibrotactile feedback
• Provides nudges when
playing violin incorrectly
• Uses motion capture to
sense arm movements that
deviate from model
• Nudges are short vibrations
on arms and hands

www.id-book.com 17
Exoskeleton with artificial muscles that
uses bubble haptic feedback

www.id-book.com 18
 Research and design considerations
• Where best to place actuators on body
• Whether to use a single or sequence of ‘touches’
• When to buzz, how intense and how often
• What kind of new smartphone/smartwatch apps
can use vibrotactile creatively?
▪ For example, slow tapping to feel like water drops
meant to indicate that it is about to rain, and heavy
tapping to indicate a thunderstorm is looming

www.id-book.com 19
 Multimodal Interfaces
• Provide enriched user experiences
▪ By multiplying the way information is experienced
▪ detected using different modalities (touch, sight, sound, and speech)
▪ Support more flexible, efficient, and expressive means of human-
computer interaction
▪ Most common is speech and vision
• Can be combined with multi-sensor input to enable
other aspects of the human body to be tracked
▪ For example, eye gaze, facial expression, and lip movements
▪ Provides input for customizing user interfaces

www.id-book.com 20
 Tracking a person’s movements

• Kinect camera can detect multimodal input in real time using RGB camera for facial
recognition and gestures, depth camera for movement tracking, and microphones
for voice recognition
• Used to build model of person and represented as avatar on display programmed to
move just like them

www.id-book.com 21
 Research and design considerations
• Need to recognize and analyze user behavior,
– speech, gesture, handwriting, or eye gaze
• Much harder to calibrate these than single-
modality systems
• What is gained from combining different
inputs and outputs
• Is talking and gesturing, as humans do with
other humans, a natural way of interacting
with a computer?

www.id-book.com 22
 Shareable interfaces
Designed for more than one person to use:
• Provide multiple inputs and sometimes allow
simultaneous input by co-located groups
• Large wall displays
• SmartBoards

• Interactive tabletops where small groups interact

with information using their fingertips
• DiamondTouch, Smart Table, and Surface
www.id-book.com 23
 A smartboard and an interactive
tabletop interface

Source: (a) Used courtesy of SMART Technologies Inc.

(www.smarttech.com/en/business) (b) Mitsubishi Electric Research
Labs

www.id-book.com 24
 Benefits
• Provide a large interactional space that can
support flexible group working
• Can be used by multiple users
▪ can point to and touch information being
displayed
▪ simultaneously view the interactions
▪ have the same shared point of reference
• Can support more equitable participation
compared with groups using single PC

www.id-book.com 25
 Research and design considerations
• Whether it can enhance collaboration?
• Whether size, orientation, and shape of the display have an
effect on collaboration
– Horizontal surfaces support more turn-taking and
collaboration
– Providing larger-sized tabletops does not improve group
working but encourages more division of labor
• Having both personal and shared spaces enables groups to
work on their own and in a group

www.id-book.com 26
 Tangible Interfaces
• Sensor-based interaction
• Physical objects (bricks, balls, and cubes) are
coupled with digital representations
• Manipulation of the physical object causes a
digital effect to occur
– sound, animation, or vibration
• Digital effects
– take place in several media and places
– can be embedded in the physical object
www.id-book.com 27
 Examples
• Flow Blocks – education & learning
▪ Depict changing numbers and lights embedded in the
blocks
▪ Vary depending on how they are connected
• Urp – urban planning
▪ Physical models of buildings moved around on tabletop
▪ Used in combination with tokens for wind and shadows
Digital shadows surrounding them to change over time

• MagicCubes
▪ Connect physical electronic components and sensors to
make digital events occur
www.id-book.com 28
 Flow block

Flowness + FlowBlocks: Uncovering the

Dynamics of Everyday Life through Playful
Modeling — MIT Media Lab

www.id-book.com 29
Learning to code and create with the
tangible MagicCubes

www.id-book.com 30
 Benefits
• Physical objects and digital representations can be
positioned, combined, and explored in creative ways
– be held in one or both hands and combined and
manipulated in ways not possible using other interfaces
▪ allow for more than one person to explore the interface
together
▪ objects can be placed on top of each other, beside each
other, and inside each other

• People can see and understand situations differently

▪ lead to greater insight, learning, and problem-solving

www.id-book.com 31
 VoxBox
A tangible system that gathers opinions through a playful and
engaging interaction (Goldsteijn et al., 2015)

https://www.researchgate.net/publication/269278475_VoxBox_A_Tangible_
Machine_that_Gathers_Opinions_from_the_Public_at_Events
www.id-book.com 32
 Research and design considerations
• What kind of coupling to use between the physical action and
digital effect
▪ If it is to support learning, then an explicit mapping between action
and effect is critical
▪ If it is for entertainment, then it can be better to design it to be more
implicit and unexpected
• What kind of physical artifact to use
▪ Bricks, cubes, and other component sets are most commonly used
because of flexibility and simplicity
▪ Stickies and cardboard tokens can also be used for placing material
onto a surface
• With what kinds of digital outputs should tangible interfaces
be combined?

www.id-book.com 33
 Augmented Reality
• Augmented reality: virtual representations are
superimposed on physical devices and objects
• Pokémon Go
▪ Used smartphone camera and GPS to place virtual
characters onto objects in the environment as if
they really were there
• Many other applications including medicine,
navigation, air traffic control and everyday
exploring
www.id-book.com 34
Gyarados in the augmented reality
mobile game Pokémon Go

© Dinah Pulver
www.id-book.com 35
 Examples
• In medicine
– x-rays and scans are overlaid on part of a patient’s
body
• In air traffic control
– dynamic information about aircraft overlaid on a
video screen of real planes
• AR-based instructions for building or repairing
complex equipment

www.id-book.com 36
Augmented reality overlay
on a car windshield

www.id-book.com 37
AR that uses forward facing camera
• Enables virtual try-ons (for example,
Snapchat filters)
• AR mirrors set up in retail stores for trying
on make-up, sunglasses, jewelry
▪ Convenient, engaging, and easy to compare
more choices
▪ But cannot feel the weight, texture, or smell of
what is being tried on
www.id-book.com 38
Singers trying on the virtual look of two
characters from the opera Akhnaten

www.id-book.com 39
 AR vs VR

Augmented Reality In Healthcare: 8 Examples -

The Medical Futurist www.id-book.com 40
 Research and design considerations

• What kind of digital augmentation?

▪ When and where in physical environment?
▪ Needs to stand out but not distract from ongoing task

▪ Needs to be able to align with real world objects

▪ How much digital content to overlay

▪ What happens if the AR is slightly off?

www.id-book.com 41
 Wearables
• First developments were head- and eye
cameras
– record what was seen and to access digital
information
• Since then, jewelry, caps, smart fabrics,
glasses, shoes, and jackets have all been used
• Recent applications focused on how to
combine textiles, electronics, and haptic
technologies

www.id-book.com 42
 Google Glass (2014)

Why was there so much excitement and concern about people

filming what they could see right in front of them?

www.id-book.com 43
 Research and design considerations
• Comfort
▪ Needs to be light, small, not get in the way, fashionable,
and preferably hidden in the clothing
• Hygiene
▪ Is it possible to wash or clean the clothing once worn?
• Ease of wear
▪ How easy is it to remove the electronic gadgetry and
replace it?
• Usability
▪ How does the user control the devices that are embedded
in the clothing?

www.id-book.com 44
 Robots
Main types
• Remote robots used in hazardous settings
▪ Can be controlled to investigate bombs and other dangerous
materials
• Domestic robots helping around the house
▪ Can pick up objects and do daily chores
• Pet robots as human companions
▪ Have therapeutic qualities, helping to reduce stress and
loneliness
• Sociable robots that work collaboratively with humans
▪ Encourage social behaviors

www.id-book.com 45
 Social robots: Mel and Paro
• Cute and cuddly
• Can open and close eyes and make sounds and
movements

Source: Images courtesy of Mitsubishi Electric Research Labs.

www.id-book.com 46
 Drones
• Unmanned aircraft controlled remotely and used in a
number of contexts
▪ Entertainment, such as carrying drinks and food to people
at festivals and parties
▪ Agricultural applications, such as flying them over
vineyards and fields to collect data about crops
▪ Helping to track poachers in wildlife parks in Africa
• Can fly low and stream photos to a ground station
where images can be stitched together into maps
• Can be used to determine the health of a crop, or
when it is the best time to harvest the crop

www.id-book.com 47
Drone being used to survey
the state of a vineyard

www.id-book.com 48
 Research and design considerations

• Whether it is acceptable to create robots that exhibit

behaviors that humans will consider to be human- or
animal-like

• Whether such anthropomorphism should be

encouraged

• The appropriation of drone technology

www.id-book.com 49
 Brain-computer interfaces
• Provide a communication pathway between a
person’s brain waves and an external device (a cursor
on a screen)
• Person is trained to concentrate on the task (moving
the cursor)
• BCIs work through detecting changes in the neural
functioning in the brain
• BCIs apps:
▪ Games (for example, Brain Ball)
▪ Enable people who are paralyzed to control robots

www.id-book.com 50
A brain-computer interface being used by a
woman who is paralyzed to select letters
on the screen

www.id-book.com 51
 Smart interfaces
• Smart: phones, speakers, watches, cars, buildings, cites
• Smart refers to having some intelligence and connect to
the internet and other devices
• Context-aware
▪ Understand what is happening around them and
execute appropriate actions (smart Nest thermostat)
• Human-building
▪ Buildings are designed to sense and act on behalf of
the inhabitants
▪ but also allow them to have some control and
interaction with the automated systems
www.id-book.com 52
 Which interface?
• Which interface to use will depend on task, users, context,
cost, robustness, and so on
• Is multimedia better than tangible interfaces for learning?
• Is speech as effective as a command-based interface?
• Is a multimodal interface more effective than a mono-modal
interface?
• Will wearable interfaces be better than mobile interfaces for
helping people to find information in foreign cities?
• Are virtual environments the ultimate interface for playing games?
• Are shareable interfaces better at supporting communication and
collaboration compared with using networked desktop PCs?

www.id-book.com 53
 Summary
• Many innovative interfaces have emerged in last 30
years, including speech, wearable, mobile, brain, and
tangible
• This raises many design and research questions as to
decide which to use
▪ For example, how best to represent information to the
user so that they can carry out ongoing activity or task
• New smart interfaces that are context-aware and
monitor people
▪ Raising new ethical issues concerned with what data is
being collected and what it is used for

www.id-book.com 54