Educational Research

Methods
Dr. K. A. Korb
University of Jos

Introduction

There is no point to analyzing data from a study

that was not properly designed to answer the
research question under investigation. In fact,
theres a real point in refusing to analyze such
data less faulty results be responsible for
implementing a program or policy contrary to
whats really needed.
Two of the valuable things a researcher can
possess are:
1. Knowledge of the principle of good study design
2. The courage to refuse to cut corners

(Dallal, 1998)

Dr. K. A. Korb
University of Jos

Scientific Method
1: Ask
Question

6: Share
Findings

2:
Design
Study

5: Reach
Conclusion
s

3:
Collect
Data

4:
Analyze
Results

Dr. K. A. Korb
University of Jos

1. Ask a Question
What is your general research
question?
What is the independent variable in
the research question?
What is the dependent variable in
the research question?

Dr. K. A. Korb
University of Jos

2. Design the Study

Research Design: Structure the research
to show how the major parts of the
research project work together to address
the central research questions
Participants
What is the population of people I am interested in?

Instruments
How will I accurately measure my independent and
dependent variables?

Procedure
How will I collect data in an appropriate manner?

Dr. K. A. Korb
University of Jos

Outline
Types of Research Methodologies
Descriptive
Causal Comparative
Correlation
Experimental
Quasi-Experiment

Threats to valid Research Design

Dr. K. A. Korb
University of Jos

Descriptive
Purpose: Carefully describe a
naturally occurring educational
phenomenon through systematic
observation
Key characteristic: Describes one
(sometimes more) variable within a
particular population
Descriptive research designs are
sometimes also called Survey
Research Designs.

Dr. K. A. Korb
University of Jos

Descriptive
Research hypotheses are impossible for
descriptive designs. Only research
questions should be posed.
There are no independent or dependent variables,
simply the key psychological variables that the
researchers is interested in.

Example questions:
What are teachers attitude towards a new moral
educational program?
What would counselors recommend be included in an
HIV/AIDS curriculum?
How many students engage in exam malpractice?

Dr. K. A. Korb
University of Jos

Descriptive
1. Select participants: Define the
group that possess the variables
you want to study
2. Data collection: Administer valid
measures of the variables of
interest
3. Data analysis: Compute
descriptive statistics

Dr. K. A. Korb
University of Jos

Descriptive

Dr. K. A. Korb
University of Jos

Causal Comparative
Purpose: Determine a cause/effect
relationship where the independent
variable cannot be manipulated
Key characteristic: Compare two or
more naturally-occurring groups on
the dependent variable
Causal Comparative designs can also
be called ex post facto designs

Dr. K. A. Korb
University of Jos

Causal Comparative
Null hypothesis: There is no significance
difference between GROUPS on
DEPENDENT VARIABLE.
Independent variable: Group category
Dependent variable: What the groups are
hypothesized to differ on.

Examples:
There is no significant differences between boys and girls
on interest in mathematics.
There is no significant differences between children from
low and high socioeconomic status in the number of
books read in a month.
There is no significance differences between Nigeria and
British children in academic self efficacy.

Dr. K. A. Korb
University of Jos

Causal Comparative
1. Select participants: Obtain a critical
number of participants in each group
2. Data collection: Administered valid
measures of your variables to all groups of
participants
3. Data analysis: Calculate the mean score for
each group on the dependent variable.

However, any difference in the mean score

between groups might be due to error.
Therefore, inferential statistics are necessary to
determine if the difference are significant.

Dr. K. A. Korb
University of Jos

Inferential Statistics
The inferential statistics to be depends on how
many groups are being compared
Two groups: t-test
Three or more groups: ANOVA

If the value is greater than.05, retain the null

hypothesis.
There is no significant difference between boys and girls
on interest in math.

If the p-value is less than .05, reject the null

hypothesis.
There is a significant difference between boys and girls
on interest in math.
Look at the mean score to determine which group has the
stronger interest.

Dr. K. A. Korb
University of Jos

Causal Comparative

Dr. K. A. Korb
University of Jos

Correlational
Purpose: Quantity the extent to which
two variables are associated
Key characteristic: The same group of
participants are given measures of both
key variables in order to calculate the
correlation coefficient
Because of the directionality and third
variable problems, there are no
independent or dependent variables in
correlational designs

Dr. K. A. Korb
University of Jos

Correlational
Null hypothesis: There is no significant
relationship between VARIABLE 1 and
VARIABLE 2.
Examples
There is no significant relationship between
number of books a child reads at home and
reading ability.
There is no significant relationship between
WAEC scores and university GPA.
There is no significant relationship between the
amount of time spent in lecture and frequency
of exam malpractice.

Dr. K. A. Korb
University of Jos

Correlational
1. Select participants: A
homogeneous sample
2. Data collection: Administer valid
measures of the variables
3. Data Analysis: Calculate the
correlation coefficient and the
significance test for correlation

Dr. K. A. Korb
University of Jos

Dr. K. A. Korb
University of Jos

Interpreting Correlations
Nature:

Strength:
-1

Negative

Positive

+1

Nature
Positive: Two variables increase or decrease
together
Negative: As one variable increases, the other
decreases

Strength
Closer to -1 or +1 is stronger relationship
0 is no relationship

Dr. K. A. Korb
University of Jos

Correlation = .78

Dr. K. A. Korb
University of Jos

Correlation = -.86

Dr. K. A. Korb
University of Jos

Correlation: .04

Dr. K. A. Korb
University of Jos

Research Example
The effect of regular leisure reading on
reading achievement in primary school.
Students in a primary school will be
randomly assigned to either a treatment
or a control group.
The treatment group will spend 10 minutes
everyday reading a book with an adult.
In an attempt to keep the treatment and control
groups as similar as possible, the control group
will also spend 10 minutes with an adult
everyday, but will instead do math problems.

Dr. K. A. Korb
University of Jos

Correlational

Correlation does not prove causati

Interest

Amount
of time
spent
studying

Academic
Achievement

Interest
Interest

Academic
Achievement

Academic
Achievement

Directionality Problem

Third Variable Problem

Dr. K. A. Korb
University of Jos

Experimental
Purpose: Establish cause and effect
Key characteristic: Statement about
how one variables affects (causes) another
Three essential components of an
experiment
Treatment
Control
Random assignment to treatment and control
groups

Dr. K. A. Korb
University of Jos

Experimental
Null hypothesis: There is no significant effect of
TREATMENT on DEPENDENT VARIABLE.
Independent variable: Treatment
Dependent variable: What the treatment should change

Examples
There is no significant effect of a counseling intervention
on homework completion.
Treatment: Counseling

There is no significance effect of reading a book a day

on reading achievement.
Treatment: Reading a book a day

There is no significant effective of an anti-exam

malpractice campaign on the frequency of exam
malpractice.
Treatment: Anti-exam malpractice campaign

Dr. K. A. Korb
University of Jos

Experimental
Goal: Keep the
experiences of the
treatment and
control groups as
identical as possible
except for the
treatment in order to
conclude that the
change in the
dependent variable
is the result of the
treatment.

Experimental Group
Read a
book
every
day

Reading
Achievement

Reading
Class

Control Group
Reading
Class

Reading
Achievement

Dr. K. A. Korb
University of Jos

Experimental
Random
Assignment: Each
subject has an
equal chance of
being assigned to
either the treatment
or control group
Minimize individual
differences in extra
variables that might
influence the
dependent variable

Interest

Experimental Group
Read a
book
every
day

Reading
Achievement

Reading
Class

Control Group
Reading
Class

Reading
Achievement

Dr. K. A. Korb
University of Jos

Random Assignment
Draw names out of
a hat
Random number
table

S/No

Random
Number

Group

Treatment

Treatment

Control

Treatment

Treatment

Control

Treatment

Control

Control

10

Control

Dr. K. A. Korb
University of Jos

Experiment

Pretest, Post-test Procedure

1. Random assignment of participants to
experimental and control groups
2. Administered valid measure of DV as pretest to
all groups
3. Administer treatment to experimental groups(s)
4. Administer identical (or conceptually identical)
measure as posttest
5. Statistical Analysis

Descriptive statistics of pretest and post-test score for

experiment and control groups separately
Significance tests to determine significance between
differences (Analysis of Covariance: ANCOVA)

Dr. K. A. Korb
University of Jos

Experimental

Dr. K. A. Korb
University of Jos

Experimental

Post-test Only Procedure

1. Random assignment of participants to
experimental and control groups
2. Administered treatment to experimental group(s)
3. Administer valid measure of DV post-test
4. Statistical Analysis

Descriptive statistics of post-test score for

experimental and control groups separately
Significance tests to determine significance between
differences (Analysis of Variance ANOVA or t-test)

The post-test only procedure is typically

recommended because of the possibility
that the pretest may have an effect on the
experiment treatment

Dr. K. A. Korb
University of Jos

Experimental

Dr. K. A. Korb
University of Jos

Quasi-Experiment
Purpose: Conduct an experiment
when random assignment is not
possible
Key characteristic: Treatment and
control groups, but no random
assignment
The null hypotheses are exactly the
same as the Experimental method

Dr. K. A. Korb
University of Jos

Quasi-Experiment

Pretest-Posttest Procedure
1. Administer valid measure of DV as pretest
2. Administer treatment to experimental
group(s)
3. Administer identical (or conceptually
identical) measure as posttest
4. Statistical Analysis

Descriptive statistical of pretest and posttest score

for experiment and control groups separately
Significance tests to determine significance
between difference (ANCOVA)

Dr. K. A. Korb
University of Jos

Review: Research Designs

Descriptive: Carefully describe a naturally

occurring educational phenomenon through
systematic observation
Causal Comparative: Compare two or more
groups on a dependent variable
Correlational: Quantify the extent to which
two variables are related
Experiment: Establish cause and effect

Three requirement:

1. Treatment Groups(s)
2. Control Group
3. Random Assignment

Quasi-Experiment: Experiment where

random assignment is not possible

Dr. K. A. Korb
University of Jos

Research Designs
Conclusion
Testing the effectiveness of a
counseling or teaching intervention
MUST either use an experimental or
quasi-experimental design
Simply administering surveys
CANNOT establish the quality of a
counseling or teaching intervention

Dr. K. A. Korb
University of Jos

Research Example
The effect of regular leisure reading on reading
achievement in primary school
Students in a primary school will be randomly
assigned to either a treatment or a control group.
The treatment group will spend 10 minutes everyday
reading a book with an adult.
In an attempt to keep the treatment and control groups
as similar as possible, the control group will also spend
10 minutes with an adult everyday, but will instead do
math problems.

After one month of the treatment, both groups will

be tested on a reading achievement test.

Dr. K. A. Korb
University of Jos

Research Example
IV:
Regular
Leisure
Reading

DV:
Reading
Achievement

Experimental Group
Reading
Achievement

Read Book
Every Day
with Adult

Reading
Achievement

Control Group
Reading
Achievement

Do Maths
Every Day
with Adult

Reading
Achievement

Dr. K. A. Korb
University of Jos

Construct Validity
Construct Validity: Establishing valid
operational measures for the concepts
being studied
This relates to:
Independent Variable
Dependent Variable

Definitions of Variable:
Construct Definition: General explanation
of the construct
Operational Definition: Statement of
specifically how the construct will be
measured or implemented in the study

Dr. K. A. Korb
University of Jos

Construct vs. Operational Definitions

Construct

Operational Definition
WAEC Scores

Academic
Achievement

Final Course Exam

NECO Scores
State Exams

Economic
Status

Self-Report of Parents
Yearly Income
Public/Private School

Dr. K. A. Korb
University of Jos

Research Example
Construct validity focuses on:
IV:
Regular
Reading

DV:
Reading
Achievement

Experimental Group
Reading
Achievement

Read a Book
Every Day

Reading
Achievement

Control Group
Reading
Achievement

Reading
Achievement

Dr. K. A. Korb
University of Jos

Construct Validity
Research studies begin with hypotheses
about psychological construct.
There is no significant effect of regular leisure
reading on reading achievement scores.
Independent variable: Regular leisure reading
Dependent variable: Reading achievement score

The psychological constructs are then

operationalized into manipulatable, measurable
terms.
Regular leisure reading: Reading a book with an
adult everyday for 10 minutes
Reading achievement score: Classroom exam
scores

Dr. K. A. Korb
University of Jos

Construct Validity
When finished analyzing the data, the result
will be translated back into psychological
constructs.
The conclusion should not be limited to Reading a
book for 10 minutes everyday with an adult leads to
higher score on a classroom reading exams.
The conclusion should be Regular leisure reading will
lead to higher reading achievement scores in general.

However, in order to translate the results back

to psychological construct, the study must
have construct valid measures of the
independent and dependent variables.

Dr. K. A. Korb
University of Jos

Construct Validity
When you develop your research project, you start with a
theory.
For example, regular leisure reading will lead to increased reading
achievement scores.

Then you operationalize your theory into your research study.

You translate regular leisure reading into how it will be
implemented in your study reading a book with an adult
everyday for 10 minutes.
You translate reading achievement scores into how it will be
measured in your study classroom exam scores.

When you finish collecting data, you hope to be able to move

back to theory.
You want to be able to say that not only did reading a book for 10
minutes everyday with an adult lead to higher classroom exam
scores on reading achievement, but also that regular leisure
reading will lead to higher reading achievement scores in general.
However, in order to do that, you must have construct valid
measures of your independent and dependent variables.

Dr. K. A. Korb
University of Jos

Research Example
Constructs:

IV:
Regular
Reading

Read book
Operationalized:
every day with
an adult for
10 min.

Constructs:

IV:
Regular
Reading

DV:
Reading
Achievement

Scores on
reading section
of classroom
exams

DV:
Reading
Achievement

Dr. K. A. Korb
University of Jos

Construct Validity

Construct Validity of Research

Designs: Allows generalization from the
operationalized treatment and measures
to the general psychological constructs
Threats to Construct Validity

Poor construct definitions in the paper

Flawed matching of operationalization to
constructs

The measures do not align with the constructs

they were designed to measure.

Dr. K. A. Korb
University of Jos

Construct Validity
Research studies begin with hypotheses
about psychological construct.
There is no significant effect of regular leisure
reading on reading achievement scores.
Independent variable: Regular leisure reading
Dependent variable: Reading achievement score

The psychological constructs are then

operationalized into manipulatable, measurable
terms.
Regular leisure reading: Reading a book with an
adult everyday for 10 minutes
Reading achievement score: Classroom exam
scores

Dr. K. A. Korb
University of Jos

Construct Validity
When finished analyzing the data, the result
will be translated back into psychological
constructs.
The conclusion should not be limited to Reading a
book for 10 minutes everyday with an adult leads to
higher score on a classroom reading exams.
The conclusion should be Regular leisure reading will
lead to higher reading achievement scores in general.

However, in order to translate the results back

to psychological construct, the study must
have construct valid measures of the
independent and dependent variables.

Dr. K. A. Korb
University of Jos

Construct Validity
When you develop your research project, you start with a
theory.
For example, regular leisure reading will lead to increased reading
achievement scores.

Then you operationalize your theory into your research study.

You translate regular leisure reading into how it will be
implemented in your study reading a book with an adult
everyday for 10 minutes.
You translate reading achievement scores into how it will be
measured in your study classroom exam scores.

When you finish collecting data, you hope to be able to move

back to theory.
You want to be able to say that not only did reading a book for 10
minutes everyday with an adult lead to higher classroom exam
scores on reading achievement, but also that regular leisure
reading will lead to higher reading achievement scores in general.
However, in order to do that, you must have construct valid
measures of your independent and dependent variables.

Dr. K. A. Korb
University of Jos

Internal Validity
Internal validity: Extent to which variables other
than the treatment provide plausible explanations to
the experimental results
Only relevant for experimental designs
Primary consideration for interventions

Extraneous variable: Any variable other than treatment

variable that, if not controlled, can affect the experimental
outcome
Hold constant or eliminate all extraneous variables that
might affect the posttest
Goal of Research Design: Create set of conditions so any
observed changes in your dependent variable can be
attributed to experimental treatment instead of extraneous
variables

Do variables other than the treatment provide

plausible explanations to the experimental results?

Dr. K. A. Korb
University of Jos

Research Example
IV:
Regular
Reading

DV:
Reading
Interest
Achievement

Maturation

Experimental Group
Reading
Achievement

Read a Book
Every Day

Reading
Achievement

Control Group
Reading
Achievement
Dr. K. A. Korb
University of Jos

Reading Reading
Adult
Achievement
Ability
Attention

Threats to Internal Validity

History: If treatment lasts over time, other
events may influence dependent variable
To prevent: Keep the control group equivalent in
all aspects but treatment

Maturation: Physical or psychological

changes may influence the dependent
variable
To prevent: Use a control group

Selection Bias: Treatment and control

groups are different on an important
extraneous variable
To prevent: Random assignment OR Use a pretest

Dr. K. A. Korb
University of Jos

Threats to Internal Validity

Testing: Giving a pretest may increase
performance on the post-test
To prevent: Dont use a pre-test OR Use different
pre- and post-tests

Instrumentation: Nature of measuring

instrument has changed between pre- and posttest
To prevent: Do not use a pre-test OR Use the same
for pre- and post-test

Many of these solutions to prevent threats to

internal validity contradict.
Consider the threats that are most relevant to a
particular study and then develop a strategy for
overcoming the most important threats.

Dr. K. A. Korb
University of Jos

External Validity
External Validity: Establishing the
group of people to which the
research findings can be generalized

Dr. K. A. Korb
University of Jos

Research Example

Population
Experimental
Group
Reading
Achievement

Read a Book
Every Day

Reading
Achievement

Sample

Reading
Achievement

Control
Group

Reading
Achievement

Dr. K. A. Korb
University of Jos

Types of External Validity

Population Validity: Generalizing the results from
an experimental sample to a defined population
To establish: Give a thorough explanation of the
constitution of the constitution of the sample and how the
sample was selected.

Ecological Validity: Generalizing the results of a

study from the artificial conditions created by
researcher to real-life conditions
To establish:
Give an adequate description of treatment
Give an adequate description of the measure for the
dependent variable
Threats to Ecological Validity:
Novelty: The treatment was different than typical treatment
Experimenter Effect: The particular person delivering the
treatment affects results

Dr. K. A. Korb
University of Jos

Reliability
Reliability: Demonstrating that the
operations of the study can be
repeated with the same results
Conduct research as if someone were
always looking over your shoulder
Report the procedure of your study in
explicit detail

Dr. K. A. Korb
University of Jos

Other Issues in Valid Research

Design
Experimenter Bias: Researchers
expectations about the outcome of
experiment influences participants response
To prevent: Use research assistants who are
blind to the study (aka do not know the purpose
of the study)

Treatment Fidelity: Extent to which

treatment conditions are implemented
according to the researchers specifications

To support:
Carefully train research assistants
Periodically check up on research assistants
unexpectedly to determine if they are following
guidelines

Dr. K. A. Korb
University of Jos

Other Issues in Valid Research

Design
Strong Experimental Treatments:
Developing a treatment that will make a
robust effect on the dependent variable
To support:
Develop a thorough understand of the
dependent variable and how it can be
influenced.
Spend much thoughtful time developing
your treatment.

Dr. K. A. Korb
University of Jos

Revision
Describe the five types of educational
research designs.
What is construct validity as related to
research designs? How can it be
supported?
What is internal validity? What are
threats to internal validity?
What is external validity? What are
types of external validity?
What are some other issues to consider
when conducting educational research?

Dr. K. A. Korb
University of Jos