Impact Evaluation Workshop

IRSA Annual Conference, Solo, 2018

Randomized Controlled Trial

Dr. Sarah Dong

Fellow, Indonesia Project
Australian National University
[email protected]
Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 2
The intuition
• Comes from medical sciences
• Randomly divide the sample into treatment
group and control group
• Treatment group and control group are similar in
both observed and unobserved characteristics
• Therefore the two groups are comparable
• The average difference between the two groups
in outcome gives us treatment effect

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 3

 In terms of counterfactuals
Individual 1: treatment effect1 = Outcome A1: Outcome B1:

Treated …
Individual N1: treatment effectN1 = Outcome AN1 Outcome BN1

Outcome Outcome
Individual N1+1: treatment effectN1+1 = BN1+1
AN1+1
Not
Treated …
Individual N: treatment effectN = Outcome AN Outcome BN

Average(Outcome A1,…, Outcome AN1)=Average(Outcome AN1+1, …, Outcome AN)

Average(Outcome B1,…, Outcome BN1)=Average(Outcome BN1+1, …, Outcome BN)
Therefore:
Average treatment effect
= Average(Outcome A1,…, Outcome AN1) - Average(Outcome BN1+1, …, Outcome BN)
Observed Also observed!
Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 4
 Procedure

Calculating
Randomization Intervention treatment
effect

Power Check
Stratification
Calculation Balance

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 5

Randomization
• Power calculation
– Procedure to decide about the size of the sample
needed to detect a treatment effect
• Stratification
– Helps with making sure treatment and control are
comparable in observed characteristics
• Check for balance
– Check that baseline characteristics are similar
between control and treatment groups

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 6

Power calculation
• Given
– expected treatment effect
– expected standard deviation
• What is
– sample size in treatment group
– sample size in control group
• Stata: power

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 7

Stata exercise: power calculation
• Calculate the sample size of the following scenario:
– Expected increase in household per capita consumption three
years after receiving PKH is from 25000 Rp/day to 28000 Rp/day
– Expected variation in per capita consumption is 25000 Rp in
terms of standard deviation
– Treatment and control are of the same sample size
• How large should treatment and control groups be?
– What if we change some of our expectations?
– What if we want a larger control or larger treatment group?

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 8

Stratification
• Decide about the characteristics to stratify along
– eg. gender, age group, education group, provinces,
income group, etc.
• Make sure each cell has enough observations
– big groups, age groups (10-25, 26-45, 46-65) instead
of age (10,11,12,…,65) for example

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 9

Stata: randtreat
• Use stata dataset RCT1.dta
• Try to randomly assign these individuals into
three treatment groups and a control group, of
the same size
• Try stratify along baseline characteristics
• Check for the size of each strata

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 10

Check for balance
• Make sure the treatment and control groups are
similar in terms of observed characteristics
• Can run:
• X = α Treat + ε
• Treat = Ω X + π
• Simply compare averages between treatment and
control in baseline characteristics

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 11

Stata exercise: check for balance
• Using the treatment assignment from the last
exercise
• Check for balance in age, religion and years of
school, using
• X = α Treat + ε
• Treat = Ω X + π
• Simply compare averages between treatment and
control in baseline characteristics

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 12

Case study: Behavioral Therapy and Crime

Blattman and others, 2017, “Reducing Crime and

Violence: Experimental Evidence from Cognitive
Behavioral Therapy in Liberia”. American
Economic Review, 107(4)

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 13

Summary
• Can we change adults’ non-cognitive skills to reduce their criminal
activity?
• Location: Monrovia, capital of Liberia
• Approached 1500 men perceived to be high-risk of conducting crime
• 999 agreed to enter the study
• Randomly assigned men twice into two treatments:
– 8 weeks of group behavior therapy
– $200 grant
• Survey these men before and after the intervention repeatedly to
ask about their economic status and crime activity
• Outcomes: antisocial behaviour, economic performance

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 14

 Estimated equation: Intent to treat effects

Treatment variable are assignments, not actual treatment, different due to non-
perfect compliance

Xi: baseline characteristics

Γj : fixed effect of randomization block

Yij: average of the two proximate survey rounds (2 and 5 weeks, 12 and 13
months)

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 15

Baseline summary and test of balance

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 16

Program impacts: ultimate outcomes and index of
intermediary outcomes (12 to 13 months not weeks)

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 17

Stata exercise: estimate the results
• Use RCT2.dta
• Check for balance of baseline chars
• Estimate the intent to treat effect by
– Not controlling for any baseline chars
– Controlling for baseline chars
– Try estimate the treatment effect for certain strata

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 18

Conceptual framework: mechanism
• A model on the choice between leisure, consumption, legal work and crime activity
• Comparative statics (mechanisms):
– Cash windfall only affects the credit-constrained, and increases legal work, as
crime activity does not require capital investment
– Therapy can
• Increase preference against crime, and reduce crime
• Increase noncognitive skill (like self-control) and induce allocation away from
crime and into legal work, since noncognitive skill pays off better for legal
work
• A smaller time discount factor (change in time-preference) will
– increase business investment and earnings if credit-constrained, since
future consumption becomes more important
– and will decrease criminal behavior, since consequences in the future
are more important
– The effect of cash and therapy can be complementary and bigger than just the
sum of the two

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 19

Compliance
• Grants
– 98% assigned to grant received the grant
• Therapy
– 5% of men assigned to therapy attended none
– Another 5% dropped out of therapy within the
first three weeks
– 2/3 attended at least 80% of therapy
– Those who dropped out early are lower risk
compared with those who stayed

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 20

Phased implementation

• Three phases
– Pilot phase of 100 men make sure treatments
cause no harm, assess statistical power, and
allow for refinement of protocols
– Second and third phase to different
neighborhoods with minor changes to
interventions and protocols

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 21

Spillovers?
• General equilibrium effects?
– Avoided by recruiting less than 1% of men in
each neighborhoods, and less than 15% of
high-risk men on the street
• Interaction between treatment and control
groups, and within treatment group?
– Cannot be tested

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 22

Generalizability
Authors argue is it is high generalizable because:
• The therapy is adapted from US-based similar programs
• They kept the intervention low-cost and developed a
publicly available manual and guidelines to facilitate
replication
• Should be easy to fine better qualified therapy facilitators
in other countries
• The results are very similar with comparable US
programs

Sarah Dong (ANU) IRSA Workshop July 21-22, 2018 23