RESEARCH METHODS

Answers to Questions
Counterfactual
Meaning
• adjective
• adjective: counterfactual; adjective: counter-factual
• relating to or expressing what has not happened or is not the case.
• noun
• noun: counterfactual; plural noun: counterfactuals; noun: counter-
factual; plural noun: counter-factuals
• a counterfactual conditional statement (e.g. If kangaroos had no tails,
they would topple over ).
Impact Evaluation
• Suppose to increase penetration in rural areas, you introduce an educative
program. This is known as the intervention
• After sometime, you wish to know if the educative program was effective
• You do an impact evaluation to assess the effectiveness of the program
• You do this by comparing insurance purchase between people who did the
educative program and those who did NOT
• The 'counterfactual' measures what would have happened to beneficiaries in
the absence of the intervention, and impact is estimated by comparing
counterfactual outcomes to those observed under the intervention.
• 3 types of designs – experimental, quasi-experimental and non-experimental
EXPERIMENTAL DESIGN
• You have two groups; one that received the intervention. This is called the treatment
group.
• The group that did not receive the intervention is called the control group.
• Now , suppose in rural areas we have people who have gone to school and those who
do not. These people who have gone to school might have a little knowledge about
insurance and end up purchasing insurance,. They do not purchase insurance
BECAUSE of our intervention. Now, suppose we put these educated people in the
intervention group, we might end up thinking that they bought it because of our
intervention.
• The problem with this is that we might think our intervention has been very
successful when in reality it might not have been very successful!
• This is called confounding. Education background is confounded with the intervention.
CONT…
• Selection bias is when for example in the treatment group, we have only educated
people. We end up thinking that our intervention was successful but it turns out
that people bough insurance because they had learnt about it in school.
• In RCT (Randomized Control trial), we RANDOMLY select the people who will receive
intervention and those who do not. Such that both groups have both educated and
no educated people. So that we can measure the effectiveness of our intervention.
• Blinding can be used. E.g. blinding the researcher – she should not know anything
about the subject before selecting the group to receive intervention.
• Blinding can also be done in medicinal experiment. 2 group – one receives real
medicine and one receives placebo. Here, the subjects are blinded. This is done so
that they do not report fake results. If a subject knows he is on the real medicine, he
might think the medicine is working when in reality it is not!
Quasi-experimental design

• Gets rid of selection bias

• methods include matching, differencing, instrumental variables and
the pipeline approach
• Matching – for example in both intervention and control group you
have ONLY educated people. You match characteristics.
• Difference in differences (or double differences) - use data collected at
baseline and end-line for intervention and comparison groups, can be
used to account for selection bias
• https://www.youtube.com/watch?v=IpWQlHapifU
Difference in Differences
Example
• Card and Krueger compared employment in the fast food sector in
New Jersey and in Pennsylvania, in February 1992 and in November
1992, after New Jersey's minimum wage rose from $4.25 to $5.05 in
April 1992
• Observing a change in employment in New Jersey only, before and
after the treatment, would fail to control for omitted variables such as
weather and macroeconomic conditions of the region. By including
Pennsylvania as a control in a difference-in-differences model, any
bias caused by variables common to New Jersey and Pennsylvania is
implicitly controlled for,
Cont’d
Before Treatment - After Treatment - Difference before and
February November after treatment
Treatment Group - New 20.44 21.03 0.59
Jersey
Control Group - 23.33 21.17 -2.16
Pennsylvania
Difference within period -2.89 -0.14 2.75
Interpretation
• Pennsylvania's change in employment can be interpreted as the
change New Jersey would have experienced, had they not increased
the minimum wage, and vice versa.
• the increased minimum wage did not induce a decrease in
employment in New Jersey
• the $0.80 minimum wage increase in New Jersey led to a 2.75 FTE
increase in employment
• FTEs (or full-time equivalents)
Non-experimental design

• they do not involve a comparison group that does not have access to
the intervention
• he method used in non-experimental evaluation is to compare
intervention groups before and after implementation of the
intervention.
Conceptual Framework (shows the relationship
between independent and dependent variables)
Example
Theoretical Framework
• The Andersen healthcare utilization model is a theoretical model aimed at
demonstrating the factors that lead to the use of health services. According to
the model, usage of health services (including inpatient care, physician visits,
dental care etc.) is determined by three dynamics: predisposing factors, enabling
factors, and need. Predisposing factors can be characteristics such as race, age,
and health beliefs. For instance, an individual who believes health services are an
effective treatment for an ailment is more likely to seek care. Examples of
enabling factors could be family support, access to health insurance, one's
community etc. Need represents both perceived and actual need for health care
services. The original model was developed by Ronald M. Andersen, a health
services professor at UCLA, in 1968. The original model was expanded through
numerous iterations and its most recent form models past the use of services to
end at health outcomes and includes feedback loops.
Case-control vs. Cohort Study
Cont’d
• Case–control studies are often used to identify factors that may
contribute to a medical condition by comparing subjects who have
that condition/disease (the "cases") with patients who do not have
the condition/disease but are otherwise similar (the "controls")
• The case–control is a type of epidemiological observational study. An
observational study is a study in which subjects are not randomized to
the exposed or unexposed groups, rather the subjects are observed in
order to determine both their exposure and their outcome status and
the exposure status is thus not determined by the researcher.
Cont’d
• For example, in a study trying to show that people who smoke (the
attribute) are more likely to be diagnosed with lung cancer (the
outcome), the cases would be persons with lung cancer, the controls
would be persons without lung cancer (not necessarily healthy), and
some of each group would be smokers. If a larger proportion of the
cases smoke than the controls, that suggests, but does not
conclusively show, that the hypothesis is valid.
• Controls should come from the same population as the cases
Prospective vs. retrospective
cohort studies
• A prospective study watches for outcomes (outcome has not yet
happened), such as the development of a disease, during the study
period
• A retrospective study, on the other hand, looks backwards and
examines exposures to suspected risk or protection factors in relation
to an outcome that is established at the start of the study
Longitudinal Vs Cross Sectional
• A longitudinal study (or longitudinal survey, or panel study) is a
research design that involves repeated observations of the same
variables (e.g., people) over short or long periods of time
• E.g. repeatedly checking BSAS students from yr1 to yr3 to see who
drops out
• Cross-sectional data, or a cross section of a study population, in
statistics and econometrics is a type of data collected by observing
many subjects (such as individuals, firms, countries, or regions) at the
one point or period of time.
• E.g. Checking BSAS students only once to see who dropped out
Cohort Study
• A cohort study is a particular form of longitudinal study that samples
a cohort (a group of people who share a defining characteristic,
typically those who experienced a common event in a selected
period, such as birth or graduation)
• Cohort studies differ from clinical trials in that no intervention,
treatment, or exposure is administered to participants in a cohort
design; and no control group is defined. Rather, cohort studies are
largely about the life histories of segments of populations, and the
individual people who constitute these segments.