Differences in Differences

• Juan Ponce
• FLACSO-Ecuador
References

• Bruce D. Meyer, W. Kip Viscusi and David L. Durbin. 1995. Workers' Compensation and Injury Duration: Evidence from a Natural
Experiment. The American Economic Review, Vol. 85, No. 3 (Jun., 1995), pp. 322-340
• Callaway Brantly and Pedro H. C. Sant'Anna. 2021. Difference-in-Differences with multiple time periods. Journal of Econometrics.
Vol 225. Issue 2. Pages 200-230
• Carey Colleen, Sarah Miller, and Laura R. Wherry. 2020. The Impact of Insurance Expansions on the Already Insured: The
Affordable Care Act and Medicare. American Economic Journal: Applied Economics 2020, 12(4): 288–318
• Currie Janet, Henrik Kleven, and Esmée Zwiers. 2020. EMPIRICAL PRACTICE IN ECONOMICS: CHALLENGES AND OPPORTUNITIES.
Technology and Big Data Are Changing Economics: Mining Text to Track Methods. AEA Papers and Proceedings 2020, 110: 42–48.
• Goodman-Bacon Andrew. 2021. Difference-in-differences with variation in treatment timing. Journal of Econometrics. Vol 225.
Issue 2. Pages 254-277
• Marcus Michelle and Pedro H. C. Sant’Anna. 2021. The Role of Parallel Trends in Event Study Settings: An Application to
Environmental Economics. Journal of the Association of Environmental and Resource Economists. JAERE, volume 8, number 2.
• Roth Jonathan, Pedro H. C. Sant’Anna, Alyssa Bilinski and John Poe. 2023. What’s Trending in Difference-in-Differences? A
Synthesis of the Recent Econometrics Literature. Journal of Econometrics 235: 2218-2244
Introduction
Introduction

• This method compares the changes in outcomes over time between a population that is
enrolled in a program (the treatment group) and a population that is not (the comparison
group).
• We compare the before and after changes for a treatment group with the before and after
for a control group.
• The difference in the before and after outcomes for the enrolled in the program (the first
difference) controls for factors that are constant over time in that group, since we are
comparing the same group to itself. But we are still left with the outside time varying.
Introduction

• One way to capture those time-varying factors is to measure the before and after change
in outcomes for a group that did not enroll in the program but was exposed to the same
set of environmental conditions (the second difference).
• If we “clean” the first difference of other time varying factors that affect the outcome of
interest by substracting the second difference, then we have eliminated the main source
of bias that worried us in the simple before and after comparisons.
• The diff in diff approach thus combines the two counterfeit counterfactuals (before and
after comparisons and comparisons between those who choose to enroll and those who
choose not to enroll) to produce a better estimate of the counterfactual.
Identification Strategy

• It is important to note that the counterfactual being estimated here is the change in
outcomes for the comparison group.
• The treatment and comparison groups do not necessarily need to have the same
preintervention conditions.
• For the DD to be valid, the comparison group must accurately represent the change in
outcomes that would have been experienced by the treatment group in the absence of
treatment.
• To apply DD, all that is necessary is to measure outcomes in the group that receives the
program (the treatment group) and the group that does not (the comparison group) both
before and after the program. The method does not require us to specify the rules by
which the treatment is assigned.
Graph
Impact estimate

• The DD method computes the impact estimate as follows:

• We calculate the difference in the outcomes (Y) between the before and after
situation for the treatment group (B-A).

• We calculate the difference in the outcomes (Y) between the before and after
situation for the comparison group (D_C).

• Then we calculate the difference between the difference in outcome for the
treatment group (B-A) and the difference for the comparison group (D-C), or DD= (B-
A)-(D-C). This difference in differences is our impact estimate.
Dealing with unobservables

• The unobserved characteristics of individuals, which cannot be controlled by comparing only

treatment and control group, are controlled to the extent that many characteristics of units or
individuals can reasonably be assumed to be constant over time (or time-invariant).
• It is plausible that many intrinsic characteristics of a person (example, intelligence, motivation,
etc) would not change over time.
• When the same individual is observed before and after a program and we compute a simple
difference in outcome for that individual, we cancel out the effect of all of the characteristics
that are unique to that individual and that do not change over time.
• Interestingly, we are canceling out (or controlling for) not only the effect of observed time-
invariant characteristics but also the effect of unobserved time-invariant characteristics.
Type of data

• In dealing with a Diff in Diff strategy, we can use:

• Panel Data

• Repeated Cross-section
The canonical Diff in Diff estimator
The DiD etimator
Graphs
Graphs
Potential Outcomes
Canonical 2x2 setup
Canonical 2x2 setup
Missing Data
Parameters of Interest
Parameters of Interest
Assumptions. SUTVA
SUTVA
No-Anticipation
Paralell Trends Assumption
The ATT estimator
The ATT estimator
The ATT estimator
Estimation
Estimating the ATT
Estimating ATT
Using Regression: TWFE estimator
TWFE estimator
TWFE estimator
Cluster?
Types of data
Types of Data: Panel Data
Panel Data
Types of Dara: Repeated Cross
Section
Repeated Cross Section
Repeated Cross Section
Covariates
Covariates and Conditioned PT
Strong overlap
Using Regression
Regression with covariates
Regression with covariates
Regression with covariates
Regression with covariates
How to include covariates?
Event studies and staggered design
Event study via TWFE specifications
Example of event study
Staggered Design
Example. Medicaid expansion
Example
Example
Medicaid expansion. Circa 2019
OLS estimate of β
Three groups
Treated in 2014 vs. Never-Treated
Treated in
2014 vs.
Treated in
2019 (t <
2019)
Treated in
2019 vs.
Treated in
2014 (t ≥
2014)
OLS estimate of β
Bacon Decomposition
Alternative estimators

• de Chaisemartin, Clément and Xavier D’Haultfœuille, “Two-Way Fixed Effects Estimators

with Heterogeneous Treatment Effects,” American Economic Review, 2020, 110 (9), 2964–
2996
• Sun, Liyan and Sarah Abraham, “Estimating Dynamic Treatment Effects in Event Studies
with Heterogeneous Treatment Effects,” Journal of Econometrics, 2021, 225 (2).
• Callaway, Brantly and Pedro H. C. SantAnna, “Difference-in-Differences with multiple time
periods,” Journal of Econometrics, 2021, 225 (2), 200–230
Using never treated as comparison
group
Using not yet treated as comparison
group
Validation and falsification
Testing the ‘equal trends’ assumption

• Compare changes in outcomes for the treatment and comparison groups before the
program is implemented. If the outcomes moved in tandem before the program started,
we gain confidence that outcomes would have continued to move in tandem in the
posintervention period.
• To check for the equality of pre-intervention trends, we need at least two serial
observations on the treatment and comparison groups before the start of the program.
• This means that the evaluation would require three serial observations; two pre-
intervention observations to assess the preprogram trends and at least one post-
intervention observation to assess impact with the DD formula.
Testing the ‘equal trends’ assumption

• To perform a placebo test.

• To perform an additional DD estimation using a ‘fake’ treatment group, that is, a group
that you know was not affected by the program. You have to find zero impact. If you don’t,
then the impact that you find come from some underlying difference in trends between
treatment and control groups.
• A placebo test can be performed not only with a fake treatment group but also with a fake
outcome (that you know is not affected by the intervention).
Testing the ‘equal trends’ assumption

• Perform the DD estimation using different comparison groups. If both groups are valid
comparison groups, you would find that the estimated impact is approximately the same
in both calculations.
“Ashenfelter’s Dip”

• Ashenfelter (1978) was first to note that enrollment in a training program is more likely if
temporary dip in earnings occurs just before start of program.
• As a consequence earnings growth after enrollment is likely different for participants even
without treatment.
Examples
Papers using DD

• Card and Krueger (1994). What is the effect of increase in minimum wage on employment?
• Prediction economic theory: a rise in the minimum wage leads perfectly competitive employers to cut
employment.
• Card and Krueger investigate effect of increase in minimum wage from $4.25 to $ 5.05 in New Jersey on April 1,
1992.
• Data on 410 fast-food restaurants (Burger King, Wendy’s,...):
• in New Jersey (treatment group)
• and Pennsylvania (control group)
• in February/March 1992 (before)
• and in November/December 1992 (after).
• Counter-intuitive result: Employment increased as consequence of increase in minimum wage (significant at 5%
level). (US$ 2.75).
Bergolo and Cruces (2011).

• This paper studies the incentive effects of social security benefits on labor market informality
following a policy reform in Uruguay.
• The reform extended health benefits to dependent children of private sector salaried workers,
and thus altered the incentive structure of holding formal jobs within the household.
• The identification strategy of the reform’s effects relies on a comparison between workers with
children(affected by the reform) and those without children (unaffected by the reform).
• Difference in differences estimates indicate a substantial effect of this expansion of coverage
on informality rates, which fell significantly by about 1.3 percentage points (a 5 percent
change) among workers in the treatment group with respect to those in the control group.
Galiani, Gertler and Schargrodsky (2005).

• In the 1990s Argentina embarked on one of the largest privatization campaigns in the
world, including the privatization of local water companies covering approximately 30
percent of the country’s municipalities.
• Using the variation in ownership of water provision across time and space generated by
the privatization process, we find that child mortality fell 8 percent in the areas that
privatized their water services and that the effect was largest (26 percent) in the poorest
areas.
Galiani, Gertler and Schargrodsky (2005).
Difference-in-difference-in-differences.
• See Carrillo et al. (2014)