Differencing estimators and event studies

Empirical Methods and Topics in Environmental Economics Lecture 3

Patrick Baylis (UBC)

1
OBSERVATIONAL RESEARCH

3
Causal inference and observational research designs
• Randomized experiments are not always possible or practical
• Regression discontinuity designs require speci7c thresholds
• Now we move to next viable alternatives: observational research
designs and quasi-experiments (admittedly this is a bit of a loose
term)
• First quasi-experiment design: differencing estimators
• To understand what quasi-experiments solve, 7rst we’ll discuss
selection on observables designs, which are generally not considered
quasi-experiments

4
Selection on observables
Techniques

1. Regression adjustment
2. Non-parametric regression
3. Matching estimators
4. Propensity score estimators
5. Combinations of the above

5
Selection on observables
Conditional unconfoundedness

• Treatment assignment is “ignorable” after we account for observable

covariates
• Conditonal unconfoundedness: treatment is independent of outcomes
a f t e r c o n d i t i o n i n g o n o b s e r va b l e c ova r i a t e s X i

Y i1 , Y i0 ⊥ D i |X i

• Compare to unconditional independence in experimental context:

Y i1 , Y i0 ⊥ D i

6
Selection on observables
When does selection on observables fail?

• In what settings might selection on observables fail to produce

unbiased estimates?
• Maybe we can’t observe determinants of mechanism that assigns
t re a t m e n t
→ I.e., units select into treatment, or a more likely to be treated
because they live in some location or during some time period
• Maybe there are unobservable characteristics that cause people who
are similar to live together
• Whatever the reason, inference is affected if treatment remains
correlated in spite of our attempts to include relevant controls/match/
use PScores

7
Selection on observables
When does selection on observables fail?

• Arceneaux, Gerber, and Green (2006) demonstrate that even in cases

where selection on observables might be appropriate, it can fail badly
→ Comparing Experimental and Matching Methods Using a Large-Scale
Voter Mobilization Experiment
• Idea: compare true experimental estimate to OLS with covariates and
to matching

8
Selection on observables
Voter mobilization experiment

“Effects” of voter encouragement (Source: Arceneaux, Gerber, and Green 2006)

9
Selection on observables
Alternatives

• Pure selection on observables can be appropriate in some cases

• But in many observational settings, we might prefer techniques that can
account for unobserved confounders as well
• These settings are called “selection on unobservables” designs, or, more
commonly, “qquasi-experiments” or “**natural experiments*”

10
Quasi-experiments
De>nition

A real world occurrence that includes “as-good-as-random” variation in treatment assignment

• In some cases, we may be able to 7nd a basis for causal inference

despite lack of control over assignment mechanism
• Key idea: want to 7nd a way to demonstrate that mechanism is “a
as
g o o d a s ra n d o m ”
• Analogous to a randomized experiment in the sense that, like
randomization, these designs do not need to measure confounders
directly to rule them out
• Plausibly exogenous sources of variation: natural, political, institutional
accident
11
Quasi-experiments
Types

• Regression discontinuity (last lecture)

• Differencing estimators (this lecture)
• Fixed effects
• Instrumental variables

12
DIFFERENCING ESTIMATORS

14
Differencing estimators
Roadmap

1. Introduction to differencing estimators

2. Currie and Walker (2011)
3. Event studies
4. Hollingsworth and Rudik (2021)
5. Triple differences
(. Walker (2013)

15
Differencing estimators
Our comparisons so far

So far we’ve discussed three kinds of comparisons to estimate treatment

effects:

1. Comparing two groups with random assignment to treatment (R

RCT)
2. Comparing two groups with as-good-as-random assignment
determined by being on either side of a cutoff (R
RD)
3. Comparing two groups with non-random assignment to treatment,
controlling for potential confounders (S
Selection on observables)

In all of these we are comparing groups in the cross-section: there is no

concept of time, before and after a policy was enacted, etc

16
Differencing estimators
What temporal comparisons deliver

• Key assumptions: the treated group would have looked the same as the
control group in the absence of treatment (after including controls)
• This assumption is often hard to defend in non-randomized settings
• One way to test is via progressive inclusion of controls: if point estimates
are sensitive to available controls, suggests they may also be sensitive
to controls you don’t observe
→ This is actually the idea that motivates Oster (2018), which
describes a method to bound bias from unobservables
• Alternatively, we can relax this assumption by exploiting temporal
comparisons alongside the cross-sectional comparison

17
Difference-in-differences
Single vs. double differences

• One way to describe our comparisons thus far is that they are single
d i f fe r e n c e s
• The estimated effect of a policy is simply the difference in expected
outcomes between treated and control groups:

τ = E[Y i1 |D i = 1] − E[Y i0 |D i = 0]

• If D i was randomly assigned, difference in expected outcomes is

a v e ra g e t r e a t m e n t e f f e c t
• This time, without random assignment we can’t expect control group to
be a perfect counterfactual for treatment group
• But, suppose we assumed that outcomes of treated and control would
have same trend were in not for treatment?
18
Difference-in-differences
Comparison of comparisons

What DID does is take the difference of two comparisons in three steps:

1. Take the difference in mean outcomes between treated and control

before treatment
2. Take the difference in mean outcomes between treated and control after
treatment
3. Take the difference between 1 and 2

The name comes from the fact that we are taking the difference (3)
between two differences (1 and 2)

19
Difference-in-differences
Why use DID?

• The identifying assumption required is less strict than for difference

approaches
• DID assumption: the treatment and control group would have followed
parallel trends in the absence of treatment
→ i.e. the difference in outcomes would have remained constant
• Much less stringent than requiring the outcomes to have been the same
in the absence of treatment

20
Conservation example
With state-speci>c and time-speci>c effects

Example: Suppose we want to understand the effect of a conservation

policy passed in New York on biodiversity (again)

• Suppose also that:

• The effect of the New York policy is given by B
• Each state has its own 7xed determinants of biodiversity (e.g. land cover,
average temperature, etc) given by NY, PA, MA, etc
• Each period has its own determinants of biodiversity, common across all
states (e.g. federal policy, global climate change) given by T 0 , T 1 , where
0 is year before the policy is passed, and 1 is after

21
Conservation example
Possible comparisons

When we observe data on biodiversity we see the combination of all

determinants: B + NY + T, not just B

We want to 7nd a way to recover only B

There are two ways you could think about trying to estimate B using
differences:

1. Compare New York to another state after the policy is passed

2. Compare New York to itself, before and after the policy is passed

22
Conservation example
The cross-sectional difference

• Let’s compare New York to another state, Pennsylvania

• With differencing, we could estimate B with:

B + (NY + T 1 ) − (P A + T 1 ) = B + NY − P A

• This is not B. Why?

• Confounded by cross-sectional variation: difference across states can’t
separate B from NY - PA

23
Conservation example
The time series difference

• Alternatively, we could compare New York to itself, before (NY + T 0 )

and after (NY + T 1 ) the policy

(B + NY + T 1 ) − (NY + T 0 ) = B + T 1 − T 0

• This is not B! Why?

• Confounded by other time series variation: difference across time can’t
separate B from T 1 − T 0

24
Conservation example
Difference-in-differences

With DID we combine these two differences

After B e f o re T i m e S e r i e s D i f fe r e n c e
New York B + NY + T 1 NY + T 0 B + T1 − T0
Pennsylvania P A + T 1 P A + T0 T1 − T0
DID Est.: B

• The time series differences lets us control for all Nxed determinants
within a state
• The cross-sectional difference lets us control for all period-speci7c
determinants common across all states
• Combining these two differences addresses both and lets us recover B,
the true effect of the policy!
25
Difference-in-differences
Not magic

• DID only can address determinants of biodiversity that are either:

→ Time-invariant
→ Time-varying, but common across all states
• If there is a determinant of biodiversity that is varying over time and
differentially across states, DID will fail to correctly estimate B
→ State climate trends, state pollution trends, etc.

26
Difference-in-differences
Add a differential trend

• Imagine another determinant of biodiversity of biodiversity C 1NY , C 0NY

→ only occurs in New York and varies over time
→ e.g., climate in New York relative to Pennsylvania
• Our DID is then:
After B e f o re T i m e S e r i e s D i f fe r e n c e
New York B + NY + T 1 + C 1NY NY + T 0 + C 0NY B + T 1 − T 0 + C 1NY − C 0NY
Pennsylvania P A + T 1 P A + T0 T1 − T0
DID Estimate B + C 1NY − C 0NY

• DID cannot isolate the effect of B versus C 1NY − C 0NY

27
Difference-in-differences
Key identi>cation assumption

• With this understanding of the structure of DID, you now have the
framework to assess basically any DID-style design
• Ask: are there are any unobserved correlated, time-varying drivers of
both outcome and treatment?
→ Still workable, if we can control for those drivers in some way
• This is what we assume away with the parallel trends assumption

28
Difference-in-differences
Regression framework

Y it = α + βD i + ξPost t + δ(D i × Post t ) + ε it

• D i is indicator for treatment group

• Post i indicator for post-period
• α: pre-program mean for comparison
• β Difference between treatment and control in pre-period
• ξ Difference between post and pre-period in control group
• δ parameter of interest

29
Difference-in-differences
Visual parallel trends

Parallel trends
30
Difference-in-differences
Defending parallel trends

• Fundamentally untestable, but suggestive evidence possible in some

settings
→ Compelling pre- and post-period graphs
→ Falsi7cation test or a direct test with panel data
→ Control directly for time trends (but, you had better get functional
forms of time trends correct)
• If only two periods, diecult to provide convincing support for parallel
trends assumption
→ Spillovers from treatment to control: double the SUTVA!
→ Lagged effect could be challenging to identify
• If treatment timing varies across units, we need a dynamic DID
31
Dynamic DID
Time-varying treatment

First, allow for a time-varying treatment

Y it = α + βD i + ϕ t + δT it + ε it

• Replaced Post t with ϕ t , time 7xed effects

• Replaced (D i × Post t ) with T it , time-varying treatment
• D i still indicates if unit is ever treated

32
Dynamic DID
Adding unit FE

Logical extension: replace D i with unit FE ϕ i :

Y it = α + ϕ i + ϕ t + δT it + ε it

• This is now a two-way Nxed effects model (next lecture topic)

• Line between DID and FE can get blurry here: i.e., you can estimate a DID
(7xed time treatment) with unit FEs if you have the statistical power
• Currie and Walker (2011) implements a DID with some dynamic
elements

33
CURRIE AND WALKER (2011)

35
Motivation
• Road congestion creates well-documented time externalities
(congestion), but what about health externalities?
• Idling cars create SO2, could impact health (esp. natal health)
• What is real-world impact of reduction in congestion on health for
children born nearby?
• How can we test this? (Ethically)

36
Currie and Walker (2011)

Paper abstract (Source: Currie and Walker 2011)

37
Overview of the paper
• Context: Automated toll collecting (E-ZPass) in NJ, PA reduced
congestion dramatically
• Data
→ Natal health data (low bw, gestation period, mother characteristics)
→ E-ZPass locations
→ Home sales
• Research design: Examine health outcomes before / after rollout and
near / far from E-ZPass

38
E-ZPass

Tolling under E-ZPass

39
Treatment and control groups

Treatment in purple, control in blue (Source: Currie and Walker 2011)

40
Descriptive evidence

LBW by distance (Currie and Walker 2011)

41
DID speciKcation
Outcome it = a + b 1 E − ZP ass it + b2Close it + b3P laza it +
b4E − ZP ass it × Close it + b5Y ear + b6Month+
b7X it + b8Distance it + ε it

• i is mother, t is birth
• E-ZPass not installed at same time everywhere, so this isn’t a “p
pure” DID,
but timing was mostly between 1997 and 2000

42
DID checks on mother characteristics

Mother differences (Source: Currie and Walker 2011)

43
DID results

Birth outcome effects (Source: Currie and Walker 2011)

44
Mother FE results

Mother FE estimates (Source: Currie and Walker 2011)

45
What do we learn?
• E-ZPass lead to signiNcant and socially impor tant improvements in
infant health
• Prematurity fell 6.7–9.1%, low birth weight fell 8.5-11.3%
• Research design gives high con7dence in estimate
• By contrast, a selection on observable approach would have had a hard
time accounting for correlations between pollution, housing location
choice, and infant health

46
What do we not learn?
• Measurement of pollution impact on maternal health
→ Only one pollution monitor near these E-ZPass locations, didn’t
monitor CO
→ More recent work focuses more on estimating this relationship
directly
• Impacts on populations that don’t live near highways
• What if we were interested in how effects evolve over time? Or
pretrends?
• Then we need an event study

47
EVENT STUDIES

49
Event study
• Event study is just another tweak on the DID/FE setup
• Allow treatment effect to vary over time by including indicator for time
until/since treatment occurred interacted with treatment indicator
• If we return to a 7xed treatment time (suppose 3 periods), can estimate
as follows:

1
Y it = α + βD i + ξPost t + ∑ δ k T tk + ε it
k=−1

• De7ne T tk = 1 if treatment occurred k periods ago/since, 0 otherwise

• Plot the δ k

50
Event study matrix goggles
i t D i P ost t T T −1 T 0 T 1
A 1 1 0 0 1 0 0
A 2 1 1 1 0 1 0
A 3 1 1 1 0 0 1
B 1 1 0 0 1 0 0
B 2 1 1 1 0 1 0
B 3 1 1 1 0 0 1
C 1 0 0 0 0 0 0
C 2 0 1 0 0 0 0
C 3 0 1 0 0 0 0

51
Event studies with time-varying treatment
• Many event studies don’t assume a 7xed treatment period
→ In these cases, we use 7xed effects and a time-varying indicator for
treatment: T itk

1
Y it = α + ϕ i + ϕ t + ∑ δ k T itk + ε it
k=−1

• Two main bene7ts of using an event study: 1) examining pre-trends to

see if they’re parallel and 2) estimating dynamic effects

52
Parallel pre-trends
• In an event study we may have multiple periods before treatment
• We can estimate the effect of being in the treatment group in each of
these (pre-)periods
• If there is no trend in the effect, the pre-trends are parallel
• This gives us some comfort that the trends were likely to be parallel in
the post-period in the absence of treatment
• We never observe what would have actually happened though, parallel
pre-trends is just supporting evidence that the parallel trends
assumption holds
• Hollingsworth and Rudik (2021) use an event study design to examine
the effect of lead on elderly mortality

53
Dynamic effects
• Reminder: event study is change in outcome over time due to policy
• In this case, if lead emissions in year 1 only kill in year 1, should only see
effects in year 1
• If lead emissions in year 1 also kill in years 2, 3, etc., should also see
effects in those years as well
• Summing these coeecients can yield cumulative effects
• Hollingsworth and Rudik (2021) provide a nice example of a paper that
studies health effects of lead with an event study

54
HOLLINGSWORTH AND
RUDIK (2021)

56
NASCAR, lead, and (bad) health
“The effect of leaded gasoline on elderly mortality: Evidence from regulatory
exemptions”

• Authors: Alex Hollingsworth and Ivan Rudik (2021)

Bubba Wallace and other NASCAR Drivers

57
What is the paper about?
• What is Hollingsworth and Rudik (2021) (H
HR) about?
• HR aims to estimate the causal effect of lead on health outcomes
• Why is this important? We know lead is bad

1. There is little causal evidence for any effects of lead

2. Almost zero causal evidence for effects of lead on adults in any way
3. Having an accurate measure of effects/costs is vital for implicit or
explicit BCA involved in policymaking

58
Research inRuencing policy

EPA Considering Tighter Lead Standards

59
Approach
Quasi-experiment

• HR estimate the causal effect of lead by exploiting a quasi-experiment

• Randomly assigning lead exposure to different groups is unethical, but
we can learn from situations where real world exposure was as good as
random
• The quasi-experiment HR exploits is the sudden removal of lead from
ra c i n g g a s o l i n e i n 2 0 0 7
• Places that happened to have racetracks in 2007 had a signi7cant
decrease in lead emissions relative to places without racetracks

60
Approach
Map of treatment and control counties
B e f o re After
Treated Areas near NASCAR racetracks Areas near NASCAR
before 2007 racetracks after 2007
Control Areas far from NASCAR racetracks Areas far from NASCAR
before 2007 racetracks after 2007

• They are comparing areas close vs far from racetracks, before vs after
deleading in 2007

61
Approach
Map of treatment and control counties

Blue: track/treated counties, green: border/weak treated counties, white:

control counties

County map
62
Approach
Event study

• Here they estimate the effect of being in the treated group, relative to
some baseline period
• In the 2x2 DD this baseline period was “before”
• Conducting an event study has two bene7ts for the paper:
→ They can look at dynamic effects of lead over time (likely, since
lead does not disappear quickly)
→ It provides **supporting evidence(()) for the parallel trends
assumption: treated counties didn’t have declining mortality rates
before removal of lead from gasoline

63
Results
Event study >gures

Figure 7
64
Paper takeways:
What can we take away from HR (2021)

• Lead has a causal effect on elderly mortality

→ It appears to be through cardiovascular channels, perhaps deaths
of despair as well (Case and Deaton, 2015)
• The mortality costs are large
→ The external mortality costs imposed by NASCAR are larger than
the value of all NASCAR teams combined
→ The social cost of lead is at least $1,000 per gram
◦ This is very, very, very big
→
→ Lead is still used in aviation and for some off-road vehicles

65
TRIPLE DIFFERENCES

67
Triple differences (DDD)
• Like DID, but add one more layer of variation
• For example, if DID was attainment vs non-attainment counties before
and after Clean Air Act
• Add comparison between regulated and unregulated 7rms
• Source of variation is the difference in differences of differences (hence
triple differences)
• Another way to think about this: second DID is placebo test

68
DDD estimation
Intuition: Difference between two DID estimates

De7ne mean outcomes for treatment, control across time 0 and 1

• Y¯C0 = E[Y it |D i = 0, t = 0]; Y¯C1 = E[Y it |D i = 0, t = 1]

• Y¯T0 = E[Y it |D i = 1, t = 0]; Y¯T1 = E[Y it |D i = 1, t = 1]

Let Z i be some other source of variation that also determines treatment

status (so actual treated only if D i = 1, Z i = 1)

0 0
δ^DID
Z=0 = ( ¯
Y T
1
− ¯
Y T ) − ( ¯
Y 1
C − ¯
Y C)
0 0
δ^DID
Z=1 = ( ¯
Y T
1
− ¯
Y T ) − ( ¯
Y 1
C − ¯
Y C)
δ^DDD = δ^DID − δ^DID
Z=1 Z=0 69
Sample DDD in regression form
• t = 0, t = 1: Before and after clean air act
• D i : Attainment status
• Z i : Regulated and unregulated

y it = α + β 1 D i + β 2 Z i + β 3 After t +
β 4 [D i × Z i ] + β 5 [D i × After t ] + β 6 [Z i × After t ]+
δ[D i × Z i × After t ] + ε it

• As with DID, regression allows for inclusion of covariates, more complex

inference (standard errors), and other bene7ts
• Note: some difference models substitute 7xed effects in place of
D i , Z i , After t , or some combo of all three
• We can see this in Walker 2018, which uses a DDD
70
WALKER 2013

72
Environmental regulations and job transitions

Walker (2013)
73
Motivation and RQ
• Environmental (and other) regulations create winners and losers.
• Air quality regs:
→ Winners: bene7ciaries from improved AQ, companies with cleaner
production processes.
→ Losers: 7rms and workers whose costs increase as a result of
regulation.
• Simple models assume job transitions are costless.
• But this doesn’t match reality: workers who lose employment will take
time to 7nd new work and train to do it.
• Walker (2013) uses restricted-access census data to identify
transitional costs of CAAA regs.

74
Attainment areas

Attainment vs. nonattainment areas

75
Source of variation

Counties that changed attainment status after 1990

76
Estimation
• Triple difference uses three sources of variation:
→ Newly designated attainment status N c
→ Sector P s : Polluting vs. non-polluting sector
→ Time period 1(τ t > 0): Before/after 1990

10
Y cst = ∑ η k1 [N c × P s × 1(τ t > 0)] + ρ cs + λ t + n ct + p st + ε c
k=0

• η k1 are the event study estimates, showing how employment among pollutin
Nrms in newly-regulated counties changes in 1990.
• FEs stand in for 7rst and second-order terms
• Also estimates “cohor t” effects to directly measure costs of the policy in
terms of lost earnings and transitional periods
77
Results
Sector employment (levels)

Employment trends (Source: Walker 2013)

78
Sector employment (differences)

Differences in employment trends (Source: Walker 2013)

79
Cohort differences

Cohort effects (Source: Walker 2013)

80
Wrapping up Walker (2013)
• Paper also tests for size of GE effects within-county (small), attempts to
identify mechanisms (workers who separate from employers are most
affected)
• Economic loss to society: nearly $8b
• Compare to transition assistance funding: $50m
• Compare to bene7ts of CAAA (estimated by EPA): $160b to 1.6t

81
Differencing/event study wrap up
• Differencing estimators are very common in policy evaluation
• Identifying assumptions weaker than selection on observables
techniques (weaker assumptions are preferred)
• But (, as always,) context and source of variation matters
• Parallel trends assumption can be supported, but is fundamentally
untestable
• Event studies allow non-simultaneous treatment / dynamic effects over
time
• Next lecture: Fixed effects and the impacts of climate change

82
References
Currie, Janet, and Reed Walker. 2011. “Tra]c Congestion and Infant Health: Evidence from E-ZPass.”
American Economic Journal: Applied Economics 3 (1): 65–90. https://doi.org/10.1257/app.3.1.65.
Hollingsworth, Alex, and Ivan Rudik. 2021. “The Effect of Leaded Gasoline on Elderly Mortality: Evidence
from Regulatory Exemptions.” American Economic Journal: Economic Policy 13 (3): 345–73. https://
doi.org/10.1257/pol.20190654.
Walker, W. Reed. 2013. “The Transitional Costs of Sectoral Reallocation: Evidence From the Clean Air
Act and the Workforce*.” The Quarterly Journal of Economics 128 (4): 1787–1835. https://
doi.org/10.1093/qje/qjt022.

83