ECON0019: Quant Econ and Econometrics

Potential Outcomes and Experiments

Prof. Áureo de Paula

: @PaulaAureo

Department of Economics
University College London

de Paula ECON0019: Quantitative Economics and Econometrics

Practicalities

§ Instructors: de Paula (W21-W25); Lewis (W27-W31).

§ Office Hours:
W21-W25 (de Paula): Fri, 14:00-15:00 or by appt; 228
Drayton House + Zoom
W26-W30 (Lewis): Thu, 16:00-17:00, 202 Drayton House

§ Textbooks:
Wooldridge: Introductory Econometrics: A Modern
Approach
Stock and Watson: Introduction to Econometrics

de Paula ECON0019: Quantitative Economics and Econometrics

Practicalities
The fight is won or lost far away from witnesses
– behind the lines, in the gym, and out there on
the road, long before I dance under those lights.
(Muhammad Ali)
§ Homeworks (= the “gym”):
Three problem sets (. . . but there are many exercises in the
books as well!).

§ Exam (= the “lights”):

Empirical project + 3 hr written exam for whole module in
Term 3.

§ Empirical Project:
Material from Terms 1 and 2. 20% of final marks. Group
assignment with up to 4 students per group. Details: Week
26. Discussion: tutorials, Week 27. Due date TBD.
de Paula ECON0019: Quantitative Economics and Econometrics
It works!

de Paula ECON0019: Quantitative Economics and Econometrics

Practicals

§ Practical sessions will take place on Mondays,

17:00-18:00.

§ The practical sessions will be led by Ertian Chen.

§ Practicals are scheduled for Feb 3, 10 and 24, Mar 10 and

24 at TBC.

de Paula ECON0019: Quantitative Economics and Econometrics

Road Map
Week 21 (17/01): PO and Experiments (W Ch.2-7a and 3-7e + SW
Ch.13.1-13.3)

Week 22 (24/01): IV, LATE (W Ch.15.1-15.6 or SW Ch.12 + SW Ch13.6 and

App 13.2)

Week 23 (31/01): Simultaneous Equations Models (W Ch.16.1-16.4)

Week 24 (07/02): LDV (W Ch.17.1-3 plus SW Ch.11)

Week 25 (14/02): LDV (W Ch.17.4-5)

Week 27 (28/02): Reg with TS (W Ch. 10, 11 + SW Ch. 15, 16)

Week 28 (0703): Reg with TS (W Ch. 10, 11 + SW Ch. 15, 16)

Week 29 (14/03): Forecasting and Prediction (SW Ch. 14, 15)

Week 30 (21/03): Dynamic Causal Effects (SW Ch. 16)

Week 31 (28/03): Assessing the Validity of Econometric Studies (SW Ch. 9)

de Paula ECON0019: Quantitative Economics and Econometrics

Ask!

de Paula ECON0019: Quantitative Economics and Econometrics

. . . but remember:

de Paula ECON0019: Quantitative Economics and Econometrics

Outline: Potential Outcomes and Experiments

Part I: Econometrics of Causality and Experiments

§ Formalization of causality
§ When can regression estimate causal effects?
§ Application: Lalonde (1986) on causal effects of job
training
§ These slides follow Ch. 2-7e and 3-7e from Wooldridge

Part II: Randomization Based on Covariates

Part III: Pros and Cons of Experiments in Economics

de Paula ECON0019: Quantitative Economics and Econometrics

Why regression?
Regressions estimate associations between variables. Why are
economists interested?

1. To predict Y from X1 , . . . , Xk .
‚ e.g., forecasting and prediction (Week 29 with D Lewis).

2. To estimate parameters of “structural” equations, e.g.

demand elasticity in a demand equation.
‚ We will discuss this further in Week 23 (SEM).

3. To estimate causal effects of policies / interventions/

treatments, e.g. of a smaller class size on test scores; or of
a job training programme on future wages.
‚ But what does “causal” mean?
‚ When/why is regression appropriate? With which controls?

de Paula ECON0019: Quantitative Economics and Econometrics

Defining causal effects
§ What it means for A to cause B is not a simple
philosophical question (see this lecture series for
example)!
§ Assume treatment status X is binary: 1 = treated (e.g. a
person goes through job training), 0 “ not treated.
§ One can define causal effects by an abstraction called
“potential outcomes”:
- Y1 “ outcome with intervention;
- Y0 “ outcome without intervention;
- Y1 ´ Y0 “ causal effect (generally a random variable, may
vary across units).
§ Both potential outcomes are defined for each unit, but only
one is observed:
#
Y1 if X “ 1
Y “ ô Y “ XY1 ` p1 ´ X qY0
Y0 if X “ 0

(For alternative perspectives, see Dawid [2000] and this blog entry.)
de Paula ECON0019: Quantitative Economics and Econometrics
de Paula ECON0019: Quantitative Economics and Econometrics
Example: Lalonde (1986)
National Support Work (NSW):
§ A USA government programme in 1970 s for groups with
weak labour-force attachment (e.g. ex-offenders).
§ Guaranteed a job for 9 ´ 18 months and paid for it. Very
expensive: $7, 000 ´ 9, 000 per person.
§ How does it affect future wage earnings (in 1978)?
§ Y0i “ worker i ’s potential wage had she not gone through
training.
§ Y1i “ worker i ’s potential wage had she gone through
training.
§ Y1i ´ Y0i “ causal effect of NSW or worker i ’s wage.
§ Xi “ whether she actually goes through training.
§ Yi “ observed wage.

de Paula ECON0019: Quantitative Economics and Econometrics

(Some) estimands of interest
Average treatment/causal effect (ATE): E rY1 ´ Y0 s
§ E.g. how much one earns if they participated in the job
training programme relative to not participating, on average
across all people
§ Comparison is for the same person across possible worlds
(under treatment and under control) then averaged across
the population
§ This is not necessarily equal to a comparison of averages
for those that participate (X “ 1) and those that do not
(X “ 0)!
§ Challenging to measure because for each person in the
sample we can only observe either Y0 or Y1 , but not both.
Average treatment effect on the treated (ATT):
E rY1 ´ Y0 | X “ 1s
§ Average causal effect among those who chose to
participate.
de Paula ECON0019: Quantitative Economics and Econometrics
What does a regression estimate?
The OLS (linear) regression of Y on binary X estimates the
difference in means:
OLS “ ErY | X “ 1s ´ ErY | X “ 0s
ELYELX
(in population or in a plim of large samples). Cor(x Y) E(xY)
-

=
,

§ Demonstration (see also W Ch.2.7): ElYXIX)) ElEMIXI)E(x)

=
-

∞ + (l-P)ExYx(X
0) =

(x R
N PENYX
=

i“1 pXi ´X qpYi ´Y q

- Remember that ˆOLS “
=

∞N 2
. PElY(x1)[-(H) EYIXE)
i“1 pXi ´X q EIX)
-

P
- Under (SLR.2)+(SLR.3), its probability limit is
CovpX ,Y q 1
OLS ” VarpX q . PElY(X D-PRElYIX
=

= =

- For a binary X with PpX “ 1q “ p, we have P(IDEY (x= 0

VarpX q “ pp1 ´ pq.
-

- By LIE, for binary X ,

CovpX , Y q “ pp1 ´ pq ˆ pErY | X “ 1s ´ ErY | X “ 0sq
§ Note that we have not imposed a linear model
Y “ 0 ` 1 X ` U (SLR.1) (nor SLR.4 or SLR.4’) from
Term 1!

§ Does OLS ever equal to ATE or ATT?

de Paula ECON0019: Quantitative Economics and Econometrics
Causation vs. selection
Recall that when X “ 1, Y “ Y1 ; and when X “ 0, Y “ Y0 .
Therefore:

OLS “ ErY | X “ 1s ´ ErY | X “ 0s

“ E rY1 | X “ 1s ´ E rY0 | X “ 0s (Difference in means)
“ E rY1 | X “ 1s ´E rY0 | X “ 1s (ATT)
`E rY0 | X “ 1s ´ E rY0 | X “ 0s (Selection bias)

§ First term: average causal effect on the treated (ATT).

§ Second term: “selection bias” - the difference in Y0
between treatment and control groups.
§ E.g. negative if those self-selecting into the programme
would have earned less than others in the absence of the
programme.

de Paula ECON0019: Quantitative Economics and Econometrics

Experimental assignment makes regression work

OLS “E rY1 ´ Y0 | X “ 1s ` looooooooooooooooooooomooooooooooooooooooooon

looooooooooomooooooooooon pE rY0 | X “ 1s ´ E rY0 | X “ 0sq
”ATT ”Selection Bias

§ If Y0 is mean-independent of X , i.e.
E rY0 | X “ 1s “ E rY0 | X “ 0s “ E rY0 s, then
E rY0 | X “ 1s ´ E rY0 | X “ 0s “ 0 ñ OLS “ ATT .

§ If Y1 ´ Y0 is also mean-independent of X , then

E rY1 ´ Y0 | X “ 1s “ E
looooooooooomooooooooooon rY1 ´ Y0 s .
looooomooooon
”ATT ”ATE

§ Both conditions hold when X is randomly assigned in an

experiment, which guarantees X KK pY0 , Y1 q: treatment
and control groups are similar on their potential outcomes.
§ Note that X is not independent of the observed Y !
de Paula ECON0019: Quantitative Economics and Econometrics
NSW example (cont.)
Can a comparison of wages for NSW participants and
non-participants tell us the average causal effect?

Scenario 1: programme participation Xi is assigned in an

experiment:
§ No selection bias and ATE = ATT ñ simple
difference-in-means identifies ATE.

Scenario 2: workers decide whether to sign up to the

programme based on their (partial) knowledge of costs and
benefits.
§ Likely E rY0i | Xi “ 1s ‰ E rY0i | Xi “ 0s : e.g. workers with
lower job opportunities (low Y0i ) are more likely to sign up;
§ And E rY1i ´ Y0i | Xi “ 1s ‰ E rY1i ´ Y0i | Xi “ 0s : e.g.
workers with higher benefits from this programme are more
likely to sign up.
de Paula ECON0019: Quantitative Economics and Econometrics
Control variables after an experiment
In an experiment, a simple regression of Y on X estimates
E rY1 ´ Y0 s

But what about a regression on X and another variable W ?

Case 1: W is a pre-treatment variable. Then X KK pY0 , Y1 , W q.

§ This regression still identifies E rY1 ´ Y0 s.
§ Proof by partialling out (Week 2 in Term 1): population
regression of X on W has a slope of zero, so its residuals
are just X (minus a constant ErX s);
§ But controlling for W can make the estimator more efficient
by reducing residual variance, when Y0 is related with W .
Case 2: W is causally affected by X ñ X K { pY0 , Y1 q | W
K
§ “Overcontrolling:” we no longer get the average causal
effect.

de Paula ECON0019: Quantitative Economics and Econometrics

NSW example (cont.)
Actual design of NSW:
§ Qualified applicants were randomly assigned to treatment
or control groups.

Lalonde (1986) compared estimates based on experimental

and observational data: NSW treatment group vs. . . .
§ NSW control group;
§ CPS-1: all males under age 55 from the Current
Population Survey;
§ CPS-3: those unemployed in 1976 and below poverty line
in 1975.
. . . via difference-in-means and regressions with various
controls

de Paula ECON0019: Quantitative Economics and Econometrics

Effects on 1978 earnings with Different Specifications

Experimental estimates are very stable; non-experimental ones

are not!
Full Samples
Specification
NSW CPS-1 CPS-3
Raw Difference 1,794 ´8, 498 -635
p633q p712q p657q
Demographic controls 1,670 ´3, 437 771
p639q p710q p837q
1975 Earnings 1,750 -78 -91
p632q p537q p641q
Demographics, 1975 Earnings 1,636 623 1,010
(638) (558) (822)
Demographics, 1974 and 1975 Earnings 1,676 794 1,369
p639q p548q p809q

(Version from Angrist and Pischke "Mostly Harmless Econometrics," Table 3.3.3)

de Paula ECON0019: Quantitative Economics and Econometrics

Connection to the linear regression model
Let’s try to derive the regression model from our primitives:

Yi “ Y1i Xi ` Y0i p1 ´ Xi q “ Y0i ` pY1i ´ Y0i q Xi

“ 0 ` 1i Xi ` Ui ,
where 0 “ E rY0i s ; Ui “ Y0i ´ E rY0i s p with E rUi s “ 0q; and
1i “ Y1i ´ Y0i .

The causal effect 1i generally varies in the population.

§ If causal effects are constant, 1i ” 1 , we get standard
SLR
§ . . . and our assumption for OLS unbiasedness (SLR.4 =
E rUi | Xi s “ 0) means E rY0i | Xi s “ E rY0i s.
§ But we also derived the conditions for OLS to estimate
average causal effects in the general case with
heterogeneous effects.

de Paula ECON0019: Quantitative Economics and Econometrics

Outline: Potential Outcomes and Experiments

Part I: Econometrics of Causality and Experiments

Part II: Randomization Based on Covariates

§ These slides have some overlap with Ch. 13.1 and
Appendix 13.3 of Stock and Watson but go beyond them

Part III: Pros and Cons of Experiments in Economics

de Paula ECON0019: Quantitative Economics and Econometrics

Control variables in experiments
So far we haven’t seen the importance of control variables with
experiments:
§ Either they are not needed for consistency, only helping
efficiency (case 1);
§ Or they introduce selection bias (case 2).

But they are also important when randomisation is based on

covariates W :
§ E.g. to estimate the effect of in-person attendance on
marks Y , we randomly assign students to attend in person
pX “ 1q
§ ... but due to capacity constraints, invite 10% of first-years,
50% of second-years, and 70% of finalists ( W “ student’s
year).
§ Assume for simplicity W “ 1, . . . , K is discrete, defining the
relevant groups.
de Paula ECON0019: Quantitative Economics and Econometrics
Bias of difference-in-means and unconfoundness

Regression of Y on X suffers from selection bias:

§ Y0 needs not be mean-independent of X , as both may be
related to W in similar ways.
§ E.g. if marks tend to differ across student cohorts and
different cohorts are assigned treatment at different rates,
this would bias estimation of SLR.

But “unconfoundness” holds: X KK pY0 , Y1 q | W

§ Running regressions group-by-group would identify
group-specific average causal effects: E rY1 ´ Y0 | W s.

de Paula ECON0019: Quantitative Economics and Econometrics

Regression with controls

Another solution: controlling for all non-experimental

determinants of treatment Xi
§ i.e. include group dummies (a.k.a. group “fixed effects” or
group-specific intercepts) as controls:
K
ÿ
Yi “ 0k 1 rWi “ ks ` 1 Xi ` Ui ,
k “1

where 1 rWi “ k s “ 1 when Wi “ k , and 0 otherwise.

de Paula ECON0019: Quantitative Economics and Econometrics

Weights of OLS with group fixed effects

Let PpW “ k q “ pk and PpX “ 1 | W q “ qpwq, i.e.

PpX “ 1 | W “ k q “ qpk q for each k “ 1, . . . , K .
- pk measures the size of each group.
- qpk q measures the probability of treatment in each group.

Proposition: 1 corresponds to a particular weighted average

of group-specific average causal effects, E rY1 ´ Y0 | W “ k s
across k “ 1, . . . , K :
ÿ pk qpk qp1 ´ qpk qq
1 “ ∞ 1 1
ˆ E rY1 ´ Y0 | W “ k s
k k 1 pk 1 qpk qp1 ´ qpk qq

Proof

de Paula ECON0019: Quantitative Economics and Econometrics

Weights of OLS with group fixed effects

Thus, 1is not the population average causal effect, which is

ÿ
E rY1 ´ Y0 s “ E rE rY1 ´ Y0 | W ss “ pk ¨E rY1 ´ Y0 | W “ k s .
k

Larger groups (i.e., higher pk ) tend to get higher weights in both

cases.

But groups with qpk q « 0 or qpk q « 1 get lower OLS weight. In

such cases there is almost no experimentation in this group
and OLS ignores it.

de Paula ECON0019: Quantitative Economics and Econometrics

Back to the example

What if random 10% of first-year students, 50% of second-year,

and 70% of finalists get assigned to in-person teaching?

§ OLS with year-of-study fixed effects estimates a weighted

average of causal effects.
§ But overweighs second-years, relative to their cohort size:
qp1q ¨ p1 ´ qp1qq “ 0.1 ¨ 0.9 “ 0.09
qp2q ¨ p1 ´ qp2qq “ 0.25
qp3q ¨ p1 ´ qp3qq “ 0.21

(Notice that the coefficients on group fixed effects and controls

more generally do not have a causal interpretation.)

de Paula ECON0019: Quantitative Economics and Econometrics

Outline: Potential Outcomes and Experiments

Part I: Econometrics of Causality and Experiments

Part II: Randomization Based on Covariates

Part III: Pros and Cons of Experiments in Economics

§ These slides roughly follow Ch. 13.2 of Stock and Watson

de Paula ECON0019: Quantitative Economics and Econometrics

Pros and cons of experiments
Recall the benefits of random assignment of treatment,
X KK pY0 , Y1 q :
§ No selection bias: E rY0 | X “ 1s “ E rY0 | X “ 0s;
§ Same average effects on treated and untreated:

E rY1 ´ Y0 | X “ 1s “ E rY1 ´ Y0 s

§ OLS identifies average causal effects for the population

under study and it is thus “internally valid” (see SW Key
Concept 9.1).

Next questions:
1. What can go wrong in practice, violating internal validity?
2. Are the results “externally valid” i.e. can they be
generalized to other settings?
3. Why don’t we always run experiments?
de Paula ECON0019: Quantitative Economics and Econometrics
Threats to internal validity
1. Failure to randomise
§ Enikolopov et al. [2013] measure the effect of independent
observers on the vote share for the ruling party and
opposition in the 2011 Russian parliamentary election.
§ Independent observers were assigned pXi “ 1q to 156 out
of 3,164 Moscow polling stations i,
§ Assignment protocol (for transparency reasons): in each of
125 city districts, polling stations #1, 26, and 51 were
selected.
§ What if stations #1 are unusual (i.e., E rY0 | #1 in a
district s ‰ E rY0 s)?
§ Also, sampling probabilities are unequal: e.g. districts with
26 stations are oversampled compared to districts with 25
stations.

de Paula ECON0019: Quantitative Economics and Econometrics

Tests for randomisation
Regress X on a set of predetermined variables W and test
whether all coefficients are zero (e.g., via F-stat). Alternatively,
regress predetermined variables on X in a placebo test:

de Paula ECON0019: Quantitative Economics and Econometrics

Enikolopov et al. [2013] findings

de Paula ECON0019: Quantitative Economics and Econometrics

Threats to internal validity

2. Partial compliance with the treatment protocol

§ Imagine observers assigned to distant or high-crime areas
chose not to go.

§ This will tend to attenuate the differences between the

treatment and control groups.

§ If compliance is observed though, this can be solved by

Instrumental Variables (next topic!).

de Paula ECON0019: Quantitative Economics and Econometrics

Threats to internal validity

3. Spillover effects
§ In defining pY0i , Y1i q we assumed that the outcome may
only depend on Xi , but not the values for X for other units.

§ But e.g. observers in a polling station may lead to

reallocation of fraud to a neighboring control station.

§ With spillovers:
- the difference-in-means is not a “clean” difference between
treatment and control.
- causal effects need to be defined differently.

de Paula ECON0019: Quantitative Economics and Econometrics

Threats to internal validity

4. Attrition
§ Some units may leave the sample!

§ E.g. most able people leave the job training programme

because they get out-of-town jobs.

§ Selection bias if the decision to leave is related to potential

outcomes: E rY0 | X “ 1 , Stay s ‰ E rY0 | X “ 0 , Stay s.

de Paula ECON0019: Quantitative Economics and Econometrics

External validity
Experimental results apply to a given treatment and a given
population.

This may not be enough for policy making, especially in scaling

up the experiment:
§ If the population changes: e.g. from a small town to a
whole country;
§ If the treatment changes: e.g. quality control becomes
weaker;
§ If equilibrium spillover effects become larger: e.g. when
there are many alumni of the job training program, wages
for non-treated workers may fall.

Experiments may not be sufficiently informative about why

(instead of whether) something works ñ variations on the
treatment may not work as well (see, e.g., When the Limeys
Get Scurvy (Cautionary Tales by T. Harford))
de Paula ECON0019: Quantitative Economics and Econometrics
Why are experiments not everywhere?
Experiments can be:

§ Costly!
§ May raise ethical/political economy concerns.
§ Not fully informative on the effects of the policy of interest:
- Experiments not always deliver policy relevant treatment
parameters. In economic theory and policy, selection on
comparative advantage is often at play and the relevant
parameter is the ATT with policy assignment rules in place.
- Randomisation may affect outcomes affecting the
interpretation and external validity (e.g., individuals may
decide not to apply if they know they will be subject to
randomisation) (see discussion on the “Hawthorne effect” in
SW 13.2 and Heckman 2020] and related issues).

de Paula ECON0019: Quantitative Economics and Econometrics

Appendix

de Paula ECON0019: Quantitative Economics and Econometrics

Weights of OLS with group fixed effects
§ Let PpW “ k q “ pk and PpX “ 1 | W q “ qpwq, i.e.
PpX “ 1 | W “ k q “ qpk q for each k “ 1, . . . , K .
- pk measures the size of each group.
- qpk q measures the probability of treatment in each group.
§ Regression of X on group fixed effects yields qpk q as
coefficients.
§ Therefore, its residual is X ´ qpW q, and by the partialling
out result (Week 2 in Term 1):
CovpY , X ´ qpW qq
1 “ .
VarpX ´ qpW qq
“ ‰
§ Using the LIE, VarrX ´ qpW qs ” E pX ´ qpW qq2 can be
seen to be:
” ” ıı ÿ
E E pX ´ qpW qq2 | W “ pk qpk qp1 ´ qpk qq.
k
§ Similarly, it can beÿshown that
CovpY , X ´ qpW qq “ pk qpk qp1 ´ qpk qqE rY1 ´ Y0 | W “ k s .
k
de Paula ECON0019: Quantitative Economics and Econometrics
Weights of OLS with group fixed effects

This implies that

∞
k pk qpk qp1 ´ qpk qq ˆ E rY1 ´ Y0 | W “ k s
1 “ ∞
k pk qpk qp1 ´ qpk qq
ÿ p qpk qp1 ´ qpk qq
“ ∞ k 1 1
ˆ E rY1 ´ Y0 | W “ k s
k k 1 pk 1 qpk qp1 ´ qpk qq

is a weighted average of group-specific average causal effects,

with weights proportional to pk qpk qp1 ´ qpk qq.

Back

de Paula ECON0019: Quantitative Economics and Econometrics