Vietnam National University of HCMC

University of Information Technology

Birmingham City University

★ Week 09 - Data Collection Techniques ★

Dr Vi Chi Thanh - [email protected]

https://vichithanh.github.io
 Last week’s recap

We have learnt:
• Introduction to qualitative and quantitative research.
• Introduction to usability testing (task-based testing).

WEDNESDAY, 27 DECEMBER 2023 2

 Last week’s recap

We have learnt:
• Introduction to qualitative and quantitative research.
• Introduction to usability testing (task-based testing).

WEDNESDAY, 27 DECEMBER 2023 3

 Last week’s recap
Define data collection techniques:

Qualitative methods
Think-aloud method
Semi-structured Interviews
Open questionnaires
Define tasks And more…
for usability test
Quantitative methods
Scale-based questionnaires (e.g. SUS)
Task completion times
Success rates
Error rates
And more…

WEDNESDAY, 27 DECEMBER 2023 4

 Today’s agenda

• Introduction to user testing and evaluation methods.

• Introduction to data collection techniques for usability tests

WEDNESDAY, 27 DECEMBER 2023 5

 Semi-structured interviews

• Semi-structured interviews present some structure, as the name suggests,

but leaves room for discussion.
• The UX researcher creates a discussion guide or script that helps lead the
conversation in the direction that will produce valuable insights for the UX
design team.
• This type of interview also allows for the UX researcher to “probe” the user
to take some action or explain their thoughts.
• Important: Semi-structured interviews are usually conducted after the
user completes the usability tasks. This is called “post-interaction
interview”.

WEDNESDAY, 27 DECEMBER 2023 6

 UX surveys (questionnaires)

• Surveys (questionnaires) can gather insights about people’s

attitudes, perceptions, intents, habits, experiences, and
characteristics, at significant moments both in time and over time.
• Survey research seems straight-forward: it is easy to write questions,
and there are many inexpensive tools. However, there is a significant
gap between quick-and-dirty surveys and surveys that are properly
planned, constructed, and analysed.
• Example of standardised questionnaires: SUS, AttrakDiff, many
others.

WEDNESDAY, 27 DECEMBER 2023 7

 System Usability Scale (SUS) questionnaire

• The System Usability Scale (SUS) provides reliable tool for measuring
the usability.
• It consists of a 10-item questionnaire with five response options for
respondents, from Strongly agree to Strongly disagree.
• Originally created by John Brooke in 1986, it allows UX researchers
to evaluate a wide variety of products and services, including
hardware, software, mobile devices, websites and applications.

WEDNESDAY, 27 DECEMBER 2023 8

 System Usability Scale (SUS) questionnaire
When a SUS is used, participants are asked to score the following 10 items with one of five responses that
range from Strongly Agree to Strongly disagree:
1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

WEDNESDAY, 27 DECEMBER 2023 9

 System Usability Scale (SUS) questionnaire

• Interpreting scoring can be complex. The participant’s scores for

each question are converted to a new number, added together and
then multiplied by 2.5 to convert the original scores of 0-40 to 0-100.
• Though the scores are 0-100, these are not percentages and should
be considered only in terms of their percentile ranking.
• Based on research, a SUS score above a 68 would be considered
above average and anything below 68 is below average, however
the best way to interpret your results involves “normalising” the
scores to produce a percentile ranking.

WEDNESDAY, 27 DECEMBER 2023 10

 Cognitive Walkthrough

• A cognitive walkthrough is a structured approach to evaluating

usability of a product.
• It involves the tester, who is not a user, asking four simple questions
about the way a specific user journey is conducted whilst observing
the user.
• The tester will record the outcomes of these questions, in their
opinion, and use these observations to improve the product further.

WEDNESDAY, 27 DECEMBER 2023 11

 Cognitive Walkthrough

• Will the user try and achieve the right outcome?

• Will the user notice that the correct action is available to them?
• Will the user associate the correct action with the outcome they
expect to achieve?
• If the correct action is performed, will the user see that progress is
being made towards their intended outcome?

WEDNESDAY, 27 DECEMBER 2023 12

WEDNESDAY, 27 DECEMBER 2023 https://www.youtube.com/watch?v=Edqjao4mmxM 13
 First-click testing

• First-click testing examines what a test participant would click on first

on the interface in order to complete their intended task.
• It can be performed on a functioning website, application, a
prototype or a wireframe.
• Successful websites and apps take users’ tasks into account upfront.
First- click testing allows UX researchers to evaluate the effectiveness
of the linking structure of the product, including the navigation, to
see if users how to get around the site and complete their intended
task.

WEDNESDAY, 27 DECEMBER 2023 14

 First-click testing

• Make sure you know and have documented the correct path to
compete each task, both for yourself and for your observers. This will
simplify note taking and transcription. Track each click.
• Time how long it takes the user to make this click. Taking a long time
to make that first click may indicate an issue with navigation that will
need to be monitored or address.
• After each task, assess whether the participants feel they were able
to find the correct information using a satisfaction or confidence
scale (e.g. SUS questionnaire).

WEDNESDAY, 27 DECEMBER 2023 15

 A/B testing

• A/B testing, also known as split testing, refers to a randomised

experimentation process where two or more versions of a product
(web page, app page) are shown to different users at the same time
to determine which version provides a superior user experience and
drive business metrics.

WEDNESDAY, 27 DECEMBER 2023 16

WEDNESDAY, 27 DECEMBER 2023 https://www.youtube.com/watch?v=OtUWbsvCujM 17
 Planning and conducting a usability test

• Prior to conducting a usability test, make sure you have all your
materials, consents and documentation prepared and checked.
• It is important to pilot test equipment and materials with a volunteer
participant.
• Run the pilot test 1-2 days prior to the first test session so that you
have time to deal with any technical issues or changes if necessary.

WEDNESDAY, 27 DECEMBER 2023 18

WEDNESDAY, 27 DECEMBER 2023 https://www.youtube.com/watch?v=NdqTbpyvaIg 19
 Usability test checklist

1. Is my high-fidelity prototype ready to be tested?

2. Did I prepare a list of tasks to be part of my usability test?
3. Did I prepare consent forms before approaching potential test
participants?
4. Am I going to conduct the test in-person or remotely?
5. Am I going to use my phone to open the prototype or am I going
to use the participant’s phone?

WEDNESDAY, 27 DECEMBER 2023 20

 Usability test checklist

6. Did I define both the qualitative and quantitative data collection

methods?
7. Did I prepare all materials necessary for the test?
8. Did I conduct a pilot testing session? Is everything working?
9. Am I going to record the session for later analysis or am I taking notes
during the session?
10. Did I collect demographic data about my participants? Such as age,
gender, education/professional background, familiarity with technology
and so on?

WEDNESDAY, 27 DECEMBER 2023 21

WEDNESDAY, 27 DECEMBER 2023 https://www.youtube.com/watch?v=7_sFVYfatXY 22
 This week’s practical activity

1. Define how data will be collected during your usability test.

2. Define your qualitative and quantitative research techniques.
3. Define how and where you are going to conduct the test.
4. Define how you are going to record the session and take notes.
5. Prepare your ethics application.
You need to collect data with at least 5 participants!

WEDNESDAY, 27 DECEMBER 2023 23

 Ethics application documents

1. Information Sheet & Consent Form

2. Debriefing Sheet
3. Task Explanation for Participant
4. Participant Recruitment Poster (if any)
5. Email Recruitment Template (if any)
6. Payment Confirmation (if any)

WEDNESDAY, 27 DECEMBER 2023 24

 Vietnam National University of HCMC
International University
School of Computer Science and Engineering

THANK YOU

Dr Vi Chi Thanh - [email protected]

https://vichithanh.github.io