Human-Computer Interaction

HCI Research Methods

Prof. Beat Signer

Department of Computer Science

Vrije Universiteit Brussel

beatsigner.com

2 December 2005
 Human-Computer Interaction
Human-computer interaction is a discipline concerned
with the design, evaluation and implementation of
interactive computing systems for human use and with
the study of major phenomena surrounding them.
ACM SIGCHI Curricula for Human-Computer Interaction

▪ Human-Computer Interaction is a multidisciplinary field

▪ Computer Science
▪ Design
▪ Cognitive Science
▪ Psychology
▪ …

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 2

 HCI Research Contributions
▪ Empirical contributions
▪ collected quantitative or qualitative data
▪ Artefact contributions (systems research)
▪ development and evaluation of new artefacts
▪ interfaces, toolkits, mock-ups, …

▪ Methodological contributions
▪ new method or modification of existing method, new metrics
▪ Theoretical contributions
▪ concepts and models as vehicles for thought
▪ frameworks, design spaces, conceptual models

▪ Dataset contributions
▪ corpus for the benefit of the research community
November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 3
 HCI Research Contributions …
▪ Literature survey contributions
▪ review and synthesis of work done in a specific area
▪ Opinion contributions
▪ trying to persuade the readers to change their minds

▪ HCI research as well as HCI research methods have

changed over time
▪ web interfaces, user-generated content, touch screens,
collaboration, …
▪ usability engineering, eye tracking, crowdsourcing, …

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 4

 HCI Research
▪ Most HCI researchers must collect their own data
▪ in other domains (e.g. sociology or economics) data often
collected by large entities or government agencies
▪ Studies with many participants (big data) can help us
determine correlations
▪ Smaller studies might provide us with a deeper
understanding of the meaning of data
▪ Longitudinal studies in HCI are rare
▪ technology and tools change rapidly → often does not make
sense to compare over multiple years or decades
▪ often seen as a shortcoming

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 5

 Measurement
▪ What to measure?
▪ In early days of HCI (early 1980s) mainly human perfor-
mance for individual tasks (micro-HCI) measured in labs
▪ task correctness
▪ time performance
▪ error rate
▪ time to learn
▪ user satisfaction
▪ Broader level (macro-HCI) such as motivation, collabo-
ration or trust not easy to measure with existing metrics
▪ use of multimethod approaches
- case studies, observations, interviews, data logging, …

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 6

 Different Types of Research
Type of Research Focus General Claims Typical Methods
Descriptive Describe a X is happening Observations,
situation or a set field studies,
of events focus groups,
interviews
Relational Identify relations X is related to Y Observations,
between multiple field studies
variables surveys
(correlation)
Experimental Identify causes of X is responsible Controlled
a situation or a for Y experiments
set of event
(causality)

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 7

 Experimental Research

Hypothesis Study
Data

▪ Null hypothesis and alternative hypothesis (see lecture 6)

▪ Independent variables
▪ technology, e.g. typing vs. voice input, mouse vs. joystick, …
▪ different types of design, e.g. pull-down vs. pop-up menu, font
size, colours, …
▪ user related, e.g. age, gender, culture, computer experience, …
▪ context, e.g. noise, lighting, temperature, …

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 8

 Experimental Research
▪ Dependent variables
▪ efficiency
- time to complete task, speed (e.g. words per minute)
▪ accuracy
- error rate
▪ subjective satisfaction
- normally collected via Likert scale ratings (e.g. via questionnaires)
▪ ease of learning or memorability and retention rate
- important for adoption of information technology
▪ physical or cognitive demand
- how long can we interact without significant fatigue

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 9

 Experimental Procedure
1. Identify a research hypothesis
▪ how many independent variables?
▪ how many values for each independent variable?
2. Specify experimental design (see lecture 6)
▪ between-subjects design, within-subjects design
and pair-wise design
3. Run a pilot study
▪ test design and study instruments
4. Recruit participants
5. Run the data collection sessions
6. Analyse the data (statistical analysis)
7. Report the results
November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 10
 Statistical Analysis
▪ Preprocessing of data
▪ cleaning up data and fixing errors if possible
- in anonymous studies we might not be able to contact participants for
clarification
- if error cannot be fixed, we have to remove the data item and treat it as
missing value in the analysis
▪ coding data
- e.g. gender: “male” → 1 and “female” → 0
- e.g. degree: “high school” → 1, “university college” → 2 and “university” → 3
▪ organising data
- bring data in layout / format for specific data processing software
(e.g. SPSS, available via the VUB software shop)

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 11

 Descriptive Statistical Tests
▪ Distribution of data points,
▪ means, medians, variances, standard deviations and ranges
▪ Comparing means
▪ cannot just compare the means but also have to compute some
statistical significance tests (e.g. t tests or ANOVA)
▪ Example
▪ independent-samples t test for data on the next slide results in
a t value of 2.169
- higher than the t value (2.131) for the specific degree of freedom (df =15) at
the 95% confidence interval
- “An independent-samples t test suggests that there is significant difference in
the task completion time between the group who used the standard word-
processing software and the group who used word-processing software with
word prediction functions (t(15) = 2.169, p < 0.05).”

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 12

 Data for Independent-Samples t Test

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 13

 Data for Paired-Samples t Test

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 14

 Identifying Relationships
▪ Identify whether there is a relationship (correlation)
between various variables
▪ compute Pearson’s product moment correlation coefficient
- SPSS will compute a correlation matrix between all variables
▪ Pearson’s r value ranges from -1 to 1
- r =-1.0 means that there is a perfect negative linear relationship between two
variables → specific increase in one variable perfectly predicts decrease in the
other variable
- r =1.0 means that there is a perfect positive linear relationship between two
variable → specific increase in one variable perfectly predicts increase in the
other variable
- r =0 means that there is no linear relationship between the two variables

▪ Note that correlation does not imply a causal relationship

▪ might also be based on hidden (“intervening”) variable

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 15

 Limitations of Experimental Research
▪ Requires well-defined testable hypotheses with a limited
number of independent and dependant variables
▪ many problems not clearly defined or involve a large number of
potential variables
▪ Need strict control of factors that might influence the
dependent variables
▪ not always possible to control these factors
▪ Lab experiments might not be a good representation of a
user’s typical interaction behaviour
▪ study participants might behave differently in lab-based
experiments
- stress of being observed, different environment, …

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 16

 Questionnaires / Surveys
▪ Closed as well as open questions (see lecture 3)
▪ Pilot test to ensure validity and reliability
▪ ensure that questions are clear, unambiguous and unbiased
▪ Need higher response rates than surveys in interaction
design to get statistically relevant results
▪ Data analysis
▪ clean the data
▪ statistical analysis of closed questions
- often just descriptive statistics (percentages etc.)
- inferential statistics by understanding the relationships between variables

▪ Might be combined with other research methods

▪ e.g. interviews, focus groups or diaries
November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 17
 Diaries
▪ Individual maintaining regular recordings
▪ fills the gap between observations in naturalistic settings
and lab studies
▪ Advantages
▪ good for understanding the “why” of user interaction with
technology
▪ useful for technology that is used on the go
- difficult to do in lab setting or via observation

▪ Disadvantages
▪ participants might not record a sufficient number of entries
▪ data analysis (mix of qualitative and quantitative) might take a
long time

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 18

 Interviews
▪ Open-ended (unstructured), semi-structured or
structured interviews (see lecture 3)
▪ Content analysis
▪ usually relies on qualitative methods for coding data
- try to find common structures and themes from qualitative data
- frequency of terms etc. (e.g. use MAXQDA)
▪ if validity is a particular concern, then multiple researchers should
independently analyse the interviews
- individual analysts might have some bias

▪ Might best be conducted as complements to other data

collection approaches

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 19

 Grounded Theory

Study Data Theory

▪ For news topic with limited literature to build on

▪ no established theories to develop coding categories
▪ Use emergent coding approach based on the notion of
grounded theory
▪ qualitative research method that seeks to develop theory that is
“grounded in data systematically gathered and analysed”
▪ might have multiple rounds of data collection and analysis to
allow the underlying theory to fully emerge
November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 20
 Ethnography
▪ Combination of observation, interviews and participation
▪ has its roots in anthropological studies
▪ Examples
▪ home settings
- country, culture and religion have a great impact on how technology is used in
homes
▪ work settings
- e.g. London underground control centre (video ethnography)
- non-office based settings (e.g. vineyards and use of sensors)
▪ educational settings
- use of technologies in learning activities

▪ Virtual ethnography
▪ use of web cams or videos

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 21

 Ethnography …
▪ Very useful in understanding the context of technology
usage
▪ Often used as a first step to understand a group of users,
their problems, challenges, norms and processes
▪ eventual goal of building some type of technology for them
or with them
▪ More recently also ethnographic investigations of
ubiquitous computing environments
▪ e.g. navigation needs of firefighters to find their way out of
hazardous, smoke-filled environments

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 22

 Automated Data Collection Methods
▪ Website access log analysers
▪ Activity logging software
▪ Custom software with reporting features

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 23

 Measuring the Human
▪ Eye tracking
▪ Muscular and skeletal position sensing
▪ Wii remote
▪ smartwatches
▪ Microsoft Kinect

▪ Motion tracking
▪ e.g. for interaction with large wall-sized displays
▪ Physiological data
▪ heart rate and blood volume / pressure
▪ galvanic skin response
▪ respiration
▪ brain activity (EEG)

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 24

 Online HCI Research
▪ Observational online studies
▪ usability and think-aloud studies
▪ remote screen sharing
▪ web cam live feed (audio/video)
▪ remote keyboard/mouse control
▪ potential recording of different streams
▪ A / B testing for Internet business

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 25

 Human Computation
▪ Humans are often better in tasks requiring
detailed interpretation of complex input
▪ Computers can be used to augment humans
(see NLS project in lecture 1), but humans can also be
used to augment computers → human computation
▪ Task that is hard for a computer but easy for humans
▪ ask multiple humans to complete small pieces of the task
▪ e.g. reCAPTCHA

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 26

 Conducting Human Computation Studies
▪ Use crowdsourcing ser-
vices for large pool of
inexpensive study partici-
pants
▪ e.g. Amazon’s Mechanical
Turk with specific APIs
▪ potentially less bias and
increased validity
- participants do not directly
interact with researchers

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 27

 Main HCI Conferences
▪ CHI: ACM Conference on Human Factors in
Computing Systems
▪ https://dl.acm.org/conference/chi/
▪ UIST: ACM Symposium on User Interface Software
and Technology
▪ https://dl.acm.org/conference/uist/
▪ CSCW: ACM Conference on Computer-supported
Cooperative Work and Social Computing
▪ https://dl.acm.org/conference/cscw/
▪ IUI: Annual Conference on Intelligent User Interfaces
▪ https://dl.acm.org/conference/iui/

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 28

 Main HCI Conferences …
▪ TEI: International Conference on Tangible,
Embedded, and Embodied Interaction
▪ https://dl.acm.org/conference/tei/
▪ EICS: International Conference on Engineering
Interactive Computing Systems
▪ https://dl.acm.org/conference/eics/
▪ DIS: ACM Conference on Designing Interactive Systems
▪ https://dl.acm.org/conference/dis/
▪ ICMI: International Conference on Multimodal Interaction
▪ https://dl.acm.org/conference/icmi/

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 29

 Exercise 7
▪ Evaluation (Usability / User Experience)

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 30

 Further Reading
▪ Major parts of this lecture are based on
the book Research Methods in
Human-Computer Interaction
▪ chapter 1–14

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 31

 References
▪ Research Methods in Human-Computer
Interaction, Jonathan Lazar, Jinjuan Heidi Feng and
Harry Hochheiser, Morgan Kaufmann (2nd edition),
May 2019, ISBN-13: 978-0128053904
▪ Technology in Action (Learning in Doing: Social,
Cognitive and Computational Perspectives),
Christian Heath and Paul Luff, Cambridge Univer-
sity Press, June 2000, ISBN-13: 978-0521568692
▪▪ SPSS (Statistical Package for the Social Sciences)
▪ https://www.ibm.com/spss
▪ available via VUB via Software Webshop

November 10, 2023 Beat Signer - Department of Computer Science - [email protected] 32

Next Lecture
Use Cases and Course Review

2 December 2005