QUANTITATIVE RESEARCH METHODS

3. Data collection methods

Features (variables) of research elements

Theoretical variable

Conceptual model LATENT

Indicating

Indicator

Operationalizing

Operational definition MANIFEST

Indicating. Transforming or translating a theoretical variable (feature at abstract/theoretical level;

not possible to observe) into an empirical observable variable. For example, (a) educational level
of the mother in social status or (b) child behavior checklist in behavioral problems.
Operationalizing. How the indicator can be observed (in concrete terms). For example, (a)
educational level of the mother: what is the highest obtained degree (= question): lower,
secondary or higher education (= score) or (b) child behavior checklist: sum of 20 questions.

28
 QUANTITATIVE RESEARCH METHODS

1.1. OBSERVATIONAL UNITS (WHO IS MEASURED)

Observational units are often the research elements, but not always. This might cause doubt
regarding the validity of the information. For example, if we ask teachers to judge their own
teaching performance could cause behavioural problems for 2 year old children (parents as proxy).
The research elements are couples/families but only one member is measured.

We could use measurement instruments:

• Scales and tests:
- Test: cognitive and motor skills.
- Scale: emotions, feelings, motives, attitudes.
- Existing ones are often validated well.
• Questions: (standardized): questionnaire, interview.
• (Structured) observation. remark: quality of the measurement instrument.
• Validity and reliability (see later).

Standardized measurement instrument which goal is obtain the result of a test is independent of
who administered the test, or the type of questions used. E.g., the Dubbel blind method used by
Clever Hans. (escuchar esta parte de la clase).
To achieve the desired independence, we can use standardized measuring instruments (in the test
manual). They should have standardized instructions and scores that tell us when the answer is
correct, how many points are received for each correct answer, and how many extra points will be
given if a quick answer is given.
Examples of standardized test instruments
– IQ: wisc, wais
– Psychopathology (depression): mmpi-2
– Personality: big 5 personality test
– Student style questionnaire
– Memory
– Attention
– Mathematical abilities
– Social adaptive behavior

29
 QUANTITATIVE RESEARCH METHODS

The methods of measurement that consist of asking questions are questionnaires or interviews.
To apply them, we need to collect information in a controlled and structured way.
We can distinguish two types of this questioning method, which we must choose depending on
which one best suit our research question:
- Written: questionnaire / survey
- Oral: interview (face to face or by phone)
When choosing between a questionnaire and an interview, we must take into account factors such
as:
a) Costs. The questionnaire is cheaper in money and time (via mail, Internet).
b) Sample size. In general, a questionnaire allows you to analyze a much larger sample,
which in general tends to be more representative of the population than a small one.
c) Response rate. It will be bigger with an interview and people will be more involved.
d) Suitability for all populations. The questionnaires assume language and reading skills,
but the interview is best for children and people with few opportunities.
e) Anonymity. In an interview there isn't. So we have to be careful in the way we
communicate, especially when dealing with sensitive topics. In fact, for some issues, the
questionnaire may be better.
f) Control effect (less with questionnaires). E.g., the effect of appearance of interviewer on
respondents (costume or not).
g) Biased viewpoint effect (less with questionnaires). Interviewer may focus more on some
questions or may (non intentionally) suggest some answers
h) Honesty and commitment (better with interview). In questionnaires: it is not always clear
who the respondent is and therefore we must try to communicate properly when we talk
about sensitive topics.
i) Completeness and quality of the data. Not answering or not understanding questions
in questionnaire: the measure is not good.
j) Number of questions. More questions may be asked in an interview (it can last 2 hours).
On the other hand, if a questionnaire is too long, people may stop filling it out.
k) Analyze the data. In questionnaires, we must stick to closed-ended questions, which are
usually easier to analyze (quantitative methods). On the contrary, in the interviews we
obtain textual data that are more complicated to analyze (qualitative methods).
Exercise 1
Good or bads questions
a) Do you think that stratification is a good sampling mechanism to improve the efficiency
of a point estimator as compared to simple random sampling without replacement)? Mala
porque es demasiado profesional.
b) Are you beating your child? Mala porque pregunta acerca de un comportamiento que no
está socialmente aceptado y nadie lo va a reconocer en un cuestionario.
c) Do you also think that this teacher is a bad teacher? Mala porque no se va a decir eso
delante del propio profesor.

30
 QUANTITATIVE RESEARCH METHODS

d) Do you go to the cinema sometimes? Mala porque a veces puede ser demasiado
ambigua.
e) How did your parents and friends react when you told them that you were going to
study this master? Mala porque se deberían separar padres y amigos, porque la relación
que se tiene con cada uno es diferente y pueden diferir los pensamientos de ambos.

4.1. QUESTIONNAIRES

Question formulation

There are many important aspects you must consider in order to get a valid measurement (what
a person really thinks).
a) Questions should be comprehensible and not too difficult.
b) (social) Acceptability.
c) Neutral (no suggestion; “do you also think that …?”).
d) Unambiguous. everybody should understand the question in the same way and no words
may be unclear (no double negations). E.g., “how much do you disagree with the
statement that not voting for Obama is not good for the economy of the US?”.
e) One-dimensional. Only ask one aspect at a time.

Exercise 2

f) Question order should be logical.

- General questions should precede more specific questions. More people say “yes” to
the general question when the specific question is presented first and people using
drugs/soft drugs should be punished.
- More factual questions regarding a topic should precede questions pertaining to
attitudes regarding a topic.
- Halo-effect: earlier questions influence answer on later questions. E.g., What was your
grade for this course? What do you think of the teaching style of the teacher?
Also, we have to be careful with response set using the same formulation style for many
questions. May trigger the same answer (without rea-ding the question) and the solution
is to mix different formulations (positive and negative).

31
 QUANTITATIVE RESEARCH METHODS

Type of questions

Closed questions (quantitative tradition) Open questions (qualitative tradition)

Limited set of possible answers: Textual information (more complex analysis).

number/categories.
Goal is often more explorative. May be a first
Goal is often to test a particular hypothesis. step to formulate closed questions (and test
hypotheses). E.g, big five questionnaire for
personality.

CLOSED QUESTIONS
1) Dichotomous (binary) questions. Only two answer categories: yes/no, absent/present,
agree/not agree.
Advantages Disadvantages
Easy. Not possible to differentiate answer, can be
different for different groups, and people have
Can be used as a filter (“if yes go to question
a positive bias to agree. E.g., when in Asch
10”).
experiment discomfort not conforming to the
group.

2) Multiple choice. Multiple answers (categories). E.g., banning alcohol is: (a) a good idea,
(b) realistic, c) a waste of time, d) useless (belongs to personal freedom).
Advantages Disadvantages
Easy. Answers (categories) should be exhaustive (no
category is forgotten) and should be mutually
More differentiated answer.
exclusive (no overlap).

3) Rating scales (x-point Likert scale). Good for measurings opinions. E.g., This course is
interesting: totally agree, agree, (not agree nor disagree), disagree or totally disagree.
Advantages Disadvantages
Easy to differentiate the answer. It depends on Answer tendencies for neutral answers. Avoid
the number of categories: 5-7-9, <5: too little extreme categories (watch out for a lack of
to differentiate, and >9: too much. Midpoint variability in the answers).
(odd number) or not (even number) we must
force them to choose or not.

4) Ordering questions. What is most important to make it in life (order from most to least
important): work hard, having a good education, having good relations with people,
being competitive.

32
 QUANTITATIVE RESEARCH METHODS

Disadvantages
– Hard when there are many categories.
– Need for complex data analysis procedures (multidimensional scaling).
5) Values. How old are you?
Advantage
– Easy.
– Accurate, but not always (=disadvantage: people do not know the correct
answer).
For your information: when you are looking for a book on the topic of constructing good
questions (this book is NOT a part of the course material)
Foddy, W. (1993). Constructing questions for interviews and questionnaires. Theory and practice
in social research. Cambridge: Cambridge University Press

4.2. MEASURING BY OBSERVATION

Example. Measuring by observation study about instant gratification: impulse control.

Can a child resist eating a candy? Relation with learning performance later.
Aspects of observation: eating the candy or not, how long it takes and strategies used to resist
the temptation.
Observation is better suited (than interview/questionnaire) for (1) measuring actual behavior (not
motives, intentions, opinions); (2) automatic/less conscious behavior (example: parenting
behavior), observation can only be used for behavior in the present; (3) measuring children
(questionnaire requires certain language and reading capabilities); and (4) studying interactions.
With observations, one has to watch out for two-fold influence of the researcher (subjectivity).
The presence of the researcher may influence the behavior that is being observed and the
researcher has to code the behavior. Also, costs (not good for large groups).

Types *

The observational system is based on a theory or constructed by the researcher (may be based
on an earlier explorative phase)
– exhaustive and mutually exclusive
– example (educational sciences): teacher styles
1 Gives instructions 5 Accepts feelings
2 Asks a question 6 Accepts behavior
3 Asks an open question 7 Rejects ideas
4 Accepts ideas

33
 QUANTITATIVE RESEARCH METHODS

*
Structured observation (quantitative) Unstructured (qualitative)
Like closed questions. Therefore, need for a Researcher as measurement instrument. Its
structured observation system (behavior like open questions. There are not categories
under study fixed in advance). In advance a list in advance (more explorative), inductive
of possible behaviors (categories) is (startpoint is what we observe), may be the
constructed and their (time of) occurrence is basis for a categorization, flexible (observing
observed, should be exhaustive and mutually naturally occurring behavior), and risk of
exclusive categories and the analysis is easy: subjectivity: which aspects do you observe and
frequency/duration of behavior. which ones do you not observe?
Non-participatory observation Participatory behavior (qualitative)
(quantitative)
Observer does not take part in the observed Goal is to understand a phenomenon from
interaction/behavior inside. Example: studying the relation between
members of a particular cultur (etnographic
research in anthropology). Also, observing in a
naturalistic setting opportunity to obtain
information that is otherwise very difficult to
access.
Different risks: biased viewpoint: hard to observe all aspects, may lead to selectiveness and bias
and control effect: observer is influencing the behavior under study.

Overt Covert
People know they are observed. Danger: social People don’t know that they are observed.
desirability effects/not spontaneous. Important: ethical considerations
Naturalistic Artificial
Where the behavior naturally takes place In a different environment than normally
E.g., in school, at home E.g. In experiment; child in playroom of
university
Can also be confronting people with an
artificial situation and observing their
behavior.

How to use the observational system?

a) Time sampling (interval system)

Divide the observation period in small intervals, in each interval the behavior can only occur one
time (intervals should be small enough) and for each interval it is indicated whether the behavior
occurs or not.
Time (seconds)
0 3 6 9 12
1 Gives instructions │ │

2 Asks a question │

34
 QUANTITATIVE RESEARCH METHODS

3 Asks an open question

4 Accepts ideas
5 Accepts feelings
6 Accepts behavior │

7 Rejects ideas

Information about how often the behavior occurs, duration of the behavior, time course of the
behavior or alternation of behaviors. It may be combined with information regarding students (to
study the teacher-student interaction).
b) Event sampling (frequency system)
Counting the number of times an observation occurs during the observation period.
Frequency
1 Gives instructions ││││ ││││

2 Asks a question
3 Asks an open question ││

4 Accepts ideas │

5 Accepts feelings
6 Accepts behavior ││││││││││││

7 Rejects ideas

Counting the number of times an observation occurs during the observation period. Also
recording the sequence of behaviours provides information about the time course and is a
combination of frequency and interval system.

35