Week 10

Identifying Needs & Establishing

Requirements
 Aims
• Discuss how to plan and run a successful
data gathering program.
• Enables you to plan and run an interview.
• Empowers you to design a simple
questionnaire.
• Enables you to plan and carry out an
observation.
www.id-book.com 2
 Five key issues
1. Setting goals
• Decide how to analyze data once collected
2. Identifying participants
• Decide from whom to gather data
• How many participants are needed
3. Relationship with participants
• Clear and professional
• Informed consent when appropriate
4. Triangulation
• Look at data from more than one perspective
• Collect more than one type of data, for instance, qualitative
data from experiments and qualitative data from interviews
5. Pilot studies
• Small trial of main study

www.id-book.com 3
 Data recording
• Notes, audio, video, and
photographs can be used
individually or in combination:
§ Notes plus photographs
§ Audio plus photographs
§ Video
• Different challenges and
advantages with each type of data
recording
4
www.id-book.com
 Interviews
Unstructured: Not directed by a script. Rich but
not replicable.
Structured: Tightly scripted, often like a
questionnaire. Replicable but may lack
richness.
Semi-structured: Guided by a script, but
interesting issues can be explored in more
depth. Can provide a good balance between
richness and replicability.
Focus groups: A group interview

www.id-book.com 5
 Interview questions
• Two types:
§ ‘Closed questions’ have a predetermined answer format, for
example, ‘yes’ or ‘no’
§ ‘Open questions’ do not have a predetermined format

• Closed questions are easier to analyze

• Avoid:
§ Long questions
§ Compound sentences — split them into two
§ Jargon and language that the interviewee may not understand
§ Leading questions that make assumptions, for example, why
do you like …?
§ Unconscious biases, for instance, gender stereotypes

www.id-book.com 6
 Running the interview
Introduction: Introduce yourself, explain the goals of
the interview, reassure about the ethical issues, ask to
record, and present the informed consent form.
Warm-up: Make first questions easy and non-
threatening.
Main body: Present questions in a logical order
A cool-off period: Include a few easy questions to
defuse tension at the end
Closure: Thank interviewee, signal the end, for
example, switch recorder off.
www.id-book.com 7
 Other forms of interviews

Digital conferencing systems such as Skype, Zoom,

email, and smartphones can be used to conduct
interviews. Some advantages are:
• Participants are in their own environment so are
more relaxed
• Participants don’t need to travel
• Participants don’t need to worry about what to wear
• For interviews involving sensitive issues, it is easier
for interviewees to be anonymous

www.id-book.com New 8
 Enriching the interview process
Props: Devices for prompting interviewee, for example, use
a prototype, scenario

www.id-book.com 9
 Questionnaires
• Questions can be closed or open
• Closed questions are easier to analyze, and
may be distributed and analyzed by computer
• They can be administered to large populations
• Disseminated by paper, email and the web
• Sampling can be a problem when the size of a
population is unknown as is common online
evaluation

www.id-book.com 10
 Questionnaire design
• The impact of a question can be influenced by
question order.
• You may need different versions of the
questionnaire for different populations.
• Provide clear instructions on how to complete the
questionnaire.
• Strike a balance between using white space and
keeping the questionnaire compact.
• Avoid very long questions and questionnaires
• Decide on whether phrases will all be positive, all
negative, or mixed.
www.id-book.com 11
 Question and response format
• ‘Yes’ and ‘No’ checkboxes
• Checkboxes that offer many options
• Rating scales
§ Likert scales
§ Semantic scales
§ 3, 5, 7 or more points

• Open-ended responses
www.id-book.com
12
 Encouraging a good response
• Make sure that the purpose of study is clear
• Promise anonymity
• Ensure that questionnaire is well designed
• Offer a short version for those who do not have
time to complete a long questionnaire
• If mailed, include a stamped, addressed envelope
• Follow-up with emails, phone calls, or letters
• Provide an incentive
• 40 percent response rate is good, 20 percent is
often acceptable
www.id-book.com 13
 Advantages of online
questionnaires
• Relatively easy and quick to distribute
• Responses are usually received quickly
• No copying and postage costs
• Data can be collected in database for
analysis
• Time required for data analysis is reduced
• Errors can be corrected easily
14
www.id-book.com
Example of an online questionnaire

Ques&onnaire shows check boxes, radio boxes, and pull-down menus

15
www.id-book.com
Problems with online questionnaires
• Sampling is problematic if population size
is unknown
• Preventing individuals from responding
more than once can be a problem
• Individuals have also been known to
change questions in email questionnaires

www.id-book.com 16
Deploying online questionnaires
• Plan the timeline
• Design offline
• Program/complete online template
• Test the survey to make sure that it behaves
as you would expect
• Test it with a group that will not be part of the
survey to check that the questions are clear
• Recruit participants
www.id-book.com 17
 Observation
• Direct observation in the field
§ Structuring frameworks
§ Degree of participation (insider or outsider)
§ Ethnography
• Direct observation in controlled environments
• Indirect observation: tracking users’ activities
§ Diaries
§ Interaction logging
§ Video and photographs collected remotely by drones
or other equipment
18
www.id-book.com
 Observation

Figure 8.8 Mars Explora&on Rover

Source: Reproduced by permission of NASA Jet Propulsion Laboratory (NASA-JPL).

19
www.id-book.com
Structuring frameworks to guide observation
• Three easy-to-remember parts:
The person: Who?
The place: Where?
The thing: What?
• A more detailed framework (Robson, 2014):
Space: What is the physical space like and how is it laid out?
Actors: What are the names and relevant details of the people
involved?
Activities: What are the actors doing and why?
Objects: What physical objects are present, such as furniture
Acts: What are specific individual actions?
Events: Is what you observe part of a special event?
Time: What is the sequence of events?
Goals: What are the actors trying to accomplish?
Feelings: What is the mood of the group and of individuals?
www.id-book.com 20
 Planning and conducting
observation in the field
• Decide on how involved you will be: from
passive observer to active participant
• How to gain acceptance
• How to handle sensitive topics, for example,
culture, private spaces, and so on
• How to collect the data:
§ What data to collect
§ What equipment to use
§ When to stop observing

www.id-book.com 21
 Ethnography
• Ethnography is a philosophy with a set of techniques
that include participant observation and interviews
• Debate about differences between participant
observation and ethnography
• Ethnographers immerse themselves in the culture that
they study
• A researcher’s degree of participation can vary
• Analyzing video and data logs can be time-consuming
• Collections of comments, incidents, and artifacts are
made
www.id-book.com
 More on Ethnography
• Co-operation of people being observed
is required
• Informants are useful
• Data analysis is continuous
• Interpretivist technique
• Questions get refined as understanding
grows
• Reports usually contain examples
23
www.id-book.com
 More on Ethnography (continued)

Figure 8.9 (a) The situation before MERboard; (b) a scientist using MERboard to
present information
Source: J. Trimble, R. Wales and R. Gossweiler (2002): “NASA position paper for the
CSCW 2002 workshop on Public, Community and Situated Displays MERBoard.

24
www.id-book.com
 Online Ethnography
• Virtual, Online, Netnography
• Online and offline activity
• Interaction online differs from face-to-
face
• Virtual worlds have a persistence that
physical worlds do not have
• Ethical considerations and
presentation of results are different
25
www.id-book.com
 Observations and materials that
might be collected (Crabtree, 2003)
• Activity or job descriptions
• Rules and procedures that govern particular activities
• Descriptions of activities observed
• Recordings of the talk taking place between parties
• Informal interviews with participants explaining the detail of
observed activities
• Diagrams of the physical layout, including the position of
artifacts
• Other information collected when observing activities:
§ Photographs of artifacts (documents, diagrams, forms,
computers, and so forth)
§ Videos of artifacts
§ Descriptions of artifacts
§ Workflow diagrams showing the sequential order of tasks
§ Process maps showing connections between activities
26
www.id-book.com
Direct observation in a controlled environment
• Direct observation
§ Think aloud techniques

• Indirect observation – tracking users’

activities
§ Diaries
§ Interaction logs
§ Web analytics

• Video, audio, photos, and notes are

used to capture data in both types of
observations
27
www.id-book.com
Choosing and combining techniques

Depends on the:
• Focus of the study
• Participants involved
• Nature of the technique(s)
• Resources available
• Time available

28
www.id-book.com
 Summary
• Data gathering sessions should have clear goals
• An informed consent may be needed
• Five key issues of data gathering are: goals, choosing
participants, triangulation, participant relationship, pilot
• Data may be recorded using handwritten notes, audio or
video recording, a camera, or any combination of these
• Interviews may be structured, semi-structured, or
unstructured
• Focus groups are group interviews
• Questionnaires may be on paper, online, or telephone
• Observation may be direct or indirect, in the field, or in
controlled settings
• Techniques can be combined depending on the study
focus, participants, nature of technique, and available
resources and time
29
www.id-book.com