Lecture Notes: Artificial Intelligence

Lecture Notes: Artificial Intelligence

The Value Added of Machine Learning to Causal Inference

Andrea A. Naghi1

1 Department of Econometrics

Erasmus University Rotterdam

& Tinbergen Institute

1/48
Lecture Notes: Artificial Intelligence
Introduction

Motivation

I A new and rapidly growing econometric literature is making

advances in the problem of using ML methods for causal
inference questions
I This literature adapts standard, off-the-shelf ML methods and
tailors them to causal inference
I Overcome difficulties: goodness-of-fit in ML vs. causal
inference; test sample evaluation - impossible in causal
inference; regularization bias
I However, the value added of these new tools relative to
traditional methods remains unclear

2/48
Lecture Notes: Artificial Intelligence
Introduction

This Paper

I Investigates with concrete empirical examples when and how

ML methods can improve economic causal analysis
I We revisit some well known and standard empirical economics
papers from a variety of topic in economics and reanalyze
their main results with causal ML methods
I We highlight the type of additional gains that causal machine
learning methods can bring to the table relative to the
traditional econometric approaches used for causal inference

3/48
Lecture Notes: Artificial Intelligence
Introduction

This Paper - Contribution

I Focus on:
I Revisit main results on ATE
I Extend papers on HTEs
I Causal Machine Learning (CML) Methods used:
I Double Machine Learning (DML) (Chernozhukov et al., 2018a)
I Causal Random Forest (Wager and Athey, 2018) and
Generalized Random Forest (Athey et al.,2019)
I Generic ML method (Chernozhukov et al., 2018b)

⇒ appropriately combining predictive methods with causal

questions/traditional causal methods can help improve the
credibility of causal analysis and gain additional economic
insights
I Monte Carlo simulations tailored to our empirical examples
support our main findings 4/48
Lecture Notes: Artificial Intelligence
Introduction

Literature
I Econometrics Literature on CML methods:
I ATE: Chernozhukov et al. (2018, ET and AER P&P), Athey,
Imbens and Wager (2018, JASA), Farrell et al. (2021, ECTA),
Colangelo and Lee (2019)
I HTE: Wager and Athey (2018, JASA), Athey et al. (2019,
Annals of Stat.), Chernozhukov et al. (2019), Semenova et al.
(2018), Oprescu et al. (2019)
I Statistics and ML Literatures: Hill (2011), Imai et al. (2013),
Van der Laan and Rose (2011), Suet al. (2009), Zeileis et al.
(2008), among others.
I Early applications: Davis and Heller (2017) and Davis and
Heller (2017, AER P&P), Strittmatter (2019), Knaus et al.
(2020 JHR), Bertrand et al. (2017), Deryugina et al. (2019,
AER)
5/48
Lecture Notes: Artificial Intelligence
Introduction

Outline

1. Introduction

2. ATE: Methodology and Results

3. HTE: Methodology and Results

4. Simulations

6/48
Lecture Notes: Artificial Intelligence
Methodology: ATE

Methodology - ATE - DML

I In observational studies, we often estimate TEs, by

conditioning on confounders. We do this by making functional
form assumptions
I If we misspecify this functional form, we end up with biased
estimates of TEs, even in the absence of unmeasured
confounders
I DML method – systematic framework for learning the
functional form from the data
I In RCTs, controlling for confounding factors is not needed,
but may produce more precise estimates, improves efficiency,
the standard errors will be lower

7/48
Lecture Notes: Artificial Intelligence
Methodology: ATE

Methodology - ATE - DML

DML - Partially Linear Regression:

Y = Dθ0 + g0 (X ) + U (1)

D = m0 (X ) + V (2)

I Naive approach: Estimate g0 with some ML tool and then use

OLS for estimating θ0
⇒ Bias from Regularization
⇒ Bias from Overfitting

8/48
Lecture Notes: Artificial Intelligence
Methodology: ATE

DML - Naive Approach

I Naive Estimator
!−1
1X 2 1X
θ̂0 = Di Di (Yi − ĝ0 (Xi ))
n n
i∈I i∈I
I Estimation Error
!−1
√ 1X 2 1 X
n(θ̂0 − θ0 ) = Di √ Di Ui
n n
i∈I i∈I
| {z }
:=a
!−1
1X 2 1 X
+ Di √ Di (g0 (Xi ) − ĝ0 (Xi ))
n n
i∈I i∈I
| {z }
:=b
9/48
Lecture Notes: Artificial Intelligence
Methodology: ATE

DML - Orthogonalization for Regularization Bias

I 1. Fit a ML model for m0 . Partial out the effect of X from D

and get V̂ = D − m̂0 (X )
I 2. Obtain an estimate of g0 as in the naive approach

I 3. Regress the residuals Y − ĝ0 (X ) on residuals V̂ and obtain:

!−1
1X 1X
θ̌0 = V̂i Di V̂i (Yi − ĝ0 (Xi ))
n n
i∈I i∈I

10/48
Lecture Notes: Artificial Intelligence
Methodology: ATE

DML - Sample Splitting for Overfitting Bias

I
√
n(θ̌0 − θ0 ) = a∗ + b ∗ + c ∗

1 X
b ∗ = (EV 2 )−1 √ (m0 (X ) − m̂0 (X ))(g0 (Xi ) − ĝ0 (Xi ))
n
i∈I

1 X
c∗ = √ Vi (g0 (Xi ) − ĝ0 (Xi ))
n
i∈I

I Partition the data: Auxiliary Sample (obtain ĝ0 and m̂0 ) and
Main Sample (obtain θ̌0 ); Cross-fitting

11/48
Lecture Notes: Artificial Intelligence
Results: ATE

Revisited Papers
I ATE:
I Djankov et al. AEJ: Macro (2010) ”The Effect of Corporate
Taxes on Investment and Entrepreneurship”
I Alesina et al. QJE (2013) ”On the Origins of Gender Roles:
Women and the Plough”
I Nunn and Trefler, AEJ: Macro (2010) ”The Structure of
Tariffs and Long-Term Growth”
I HTE:
I Della Vigna and Kaplan QJE (2007) ”The Fox News Effect:
Media Bias and Voting”
I Loyalka et al. AEJ: Applied (2019) ”Does Teacher Training
Actually Work? Evidence from a Large-Scale Randomized
Evaluation of a National Teacher Training Program”
12/48
Lecture Notes: Artificial Intelligence
Results: ATE

The Effect of Corporate Taxes on Investment and

Entrepreneurship
I Djankov et al. AEJ: Macro (2010), 1058 citations
I Research question: effect of corporate taxes on investment
and entrepreneurship
I Data: cross country study, 50-61 countries; negative effect
I Baseline regression (OLS):

yc = α + βtaxesc + Xc Γ + c
I Revisit the ”kitchen sink” regression, with all controls:
other taxes (other taxes payable, VAT tax, sales tax, and the highest rate on
personal income tax); log no. of tax payments made, index of tax evasion, and
no. of procedures to start a business; institutional variables (indices for
property rights, rigidity of employment laws, openness to trade, log of GDP pc);
inflation (avg. inflation in the past ten years, and seigniorage)
13/48
Lecture Notes: Artificial Intelligence
Results: ATE

Results: Effect of Corporate Taxes on Investment

(1) (2) (3) (4) (5) (6) (7) (8)
Lasso Trees Boosting Forest Nnet Ensemble Best OLS

Panel A: Investment 2003-2005

Statutory corporate tax rate -0.074 -0.069 -0.068 -0.07 -0.056 -0.066 -0.071 -0.064
(0.075) (0.065) (0.069) (0.077) (0.087) (0.078) (0.081) (0.098)
First-year effective tax rate -0.114 -0.129 -0.154 -0.144 -0.122 -0.13 -0.133 -0.117
(0.075) (0.078) (0.08) (0.083) (0.082) (0.081) (0.08) (0.106)
Five-year effective tax rate -0.187* -0.182* -0.211* -0.21* -0.217* -0.216* -0.207* -0.189
(0.079) (0.079) (0.086) (0.084) (0.083) (0.084) (0.083) (0.118)

Panel B: FDI 2003-2005

Statutory corporate tax rate -0.148 -0.157* -0.153 -0.14 -0.085 -0.133 -0.114 -0.030
(0.075) (0.078) (0.078) (0.08) (0.068) (0.078) (0.077) (0.066)
First-year effective tax rate -0.141 -0.194* -0.178* -0.157* -0.136* -0.161* -0.137* -0.1
(0.074) (0.075) (0.075) (0.06) (0.059) (0.068) (0.059) (0.071)
Five-year effective tax rate -0.147* -0.177** -0.167* -0.165* -0.139* -0.157* -0.14* -0.095
(0.067) (0.065) (0.068) (0.068) (0.062) (0.067) (0.063) (0.081)

No observations 61 61 61 61 61 61 61 61
Raw covariates 12 12 12 12 12 12 12 12

Notes: Reanalysis of Table 5D from Djankov et al. (2010). DML results are based on 100 splits. Point estimates
are calculated as the median over splits. Standard errors are reported in parentheses. Median standard errors across
splits are reported for the DML estimates. *, ** and *** indicate significance at the 10%, 5% and 1% levels.
14/48
Lecture Notes: Artificial Intelligence
Results: ATE

Results: Effect of Corporate Taxes on Entrepreneurship

(1) (2) (3) (4) (5) (6) (7) (8)
Lasso Trees Boosting Forest Nnet Ensemble Best OLS

Panel C: Business density

Statutory corporate tax rate -0.062 -0.092 -0.069 -0.07 -0.056 -0.066 -0.06 -0.034
(0.053) (0.061) (0.054) (0.055) (0.062) (0.057) (0.058) (0.083)
First-year effective tax rate -0.104 -0.156* -0.124* -0.122 -0.105 -0.114 -0.1 -0.068
(0.062) (0.067) (0.062) (0.062) (0.07) (0.063) (0.063) (0.092)
Five-year effective tax rate -0.091 -0.139* -0.122* -0.107 -0.115 -0.114 -0.104 -0.070
(0.058) (0.068) (0.06) (0.06) (0.073) (0.065) (0.064) (0.103)

No observations 60 60 60 60 60 60 60 60

Panel D: Average entry rate 2000-2004

Statutory corporate tax rate -0.112* -0.147* -0.141* -0.127* -0.067 -0.112 -0.106 -0.029
(0.055) (0.055) (0.056) (0.056) (0.063) (0.057) (0.058) (0.086)
First-year effective tax rate -0.13* -0.144* -0.143* -0.125* -0.131 -0.126* -0.117 -0.083
(0.057) (0.058) (0.058) (0.059) (0.07) (0.061) (0.063) (0.094)
Five-year effective tax rate -0.154* -0.153* -0.164* -0.164** -0.191** -0.168* -0.167* -0.133
(0.06) (0.063) (0.062) (0.061) (0.071) (0.066) (0.065) (0.103)

No observations 50 50 50 50 50 50 50 50
Raw covariates 12 12 12 12 12 12 12 12
Notes: Reanalysis of Table 5D from Djankov et al. (2010). DML results are based on 100 splits. Point estimates
are calculated as the median over splits. Standard errors are reported in parentheses. Median standard errors across
splits are reported for the DML estimates. *, ** and *** indicate significance at the 10%, 5% and 1% levels.
15/48
Lecture Notes: Artificial Intelligence
Results: ATE

Lasso interactions coefficients: m̂(·)

Average entry rate and first-year rate

16/48
Lecture Notes: Artificial Intelligence
Results: ATE

Lasso interactions coefficients: ĝ (·)

Average entry rate and first-year rate

17/48
Lecture Notes: Artificial Intelligence
Results: ATE

Results: (also) with the Causal Random Forest

(1) (2) (3) (4)

Investment FDI Business density Average entry rate
Statutory corporate tax rate -0.074 -0.131 -0.070 -0.148**
(0.066) (0.085) (0.054) (0.056)
First-year effective tax rate -0.158* -0.189** -0.119* -0.129**
(0.087) (0.081) (0.064) (0.054)
Five-year effective tax rate -0.222** -0.165** -0.112* -0.152**
(0.090) (0.073) (0.062) (0.062)
Observations 61 61 60 50
Raw covariates 12 12 12 12

Notes: Analysis of Table 5D of Djankov et al. (2010) using the causal random forest. Standard errors are reported
in parentheses. The number of covariates does not include the treatment variable. ***, ** and * indicate
significance at the 1%, 5% and 10% levels respectively.

18/48
Lecture Notes: Artificial Intelligence
Results: ATE

Important Remarks: Corporate Taxes on Investment and

Entrepreneurship

I Example illustrates a typical trade-off of the applied

researcher; CML data-driven approach to systematically and
flexibly select confounders vs ”kitchen sink” approach
I Robustness to Tuning Parameter choices
I Results supported by Monte Carlo simulations (see later)
I Sparsity Issues

I DML results ⇒ strengthen empirical conclusion of original

paper

19/48
Lecture Notes: Artificial Intelligence
Results: ATE

On the Origins of Gender Roles: Women and the Plough

I Alesina et al. QJE (2013); 1660 citations

I Research question: do societies that traditionally practiced
plough agriculture have less equal gender norms today?
I Data: cross country regression, 123-165 countries
I Revisit baseline, robustness and IV regressions
I Baseline OLS regression:

yc = α + βplough usec + XcH Γ + XcC Π + c

I Instruments: 1) suitability for ”plough-positive” cereal crops
(wheat, barley, rye); 2) suitability for ”plough-negative” cereal crops
(foxtail millet, pearl millet, sorghum)

20/48
Lecture Notes: Artificial Intelligence
Results: ATE

Controls: Women and the Plough

I Baseline controls: historical (ancestral suitability for agriculture, fraction

of ancestral land that was tropical or subtropical, ancestral domestication
of large animals, ancestral settlement patterns, ancestral political
complexity) and contemporary (ln of real per capita GDP, and square)

I Robustness controls: baseline plus measures of the characteristics of the

ancestors of the current population living in a country, and other
contemporary variables, plus continent fixed effects

I IV controls: baseline plus geo-climatic characteristics added flexibly

(terrain slope, soil depth, average temperature, average precipitation),
plus continent fixed effects

21/48
Lecture Notes: Artificial Intelligence
Results: ATE

Results: Women and the Plough; Baseline

(1) (2) (3) (4) (5) (6) (7) (8)
Lasso Trees Boosting Forest Nnet Ensemble Best OLS

Panel A: Female labour force participation 2000

Traditional plough use -10.434*** -9.287*** -11.948*** -11.243*** -11.271*** -11.501*** -11.388*** -12.401***
(3.195) (2.817) (3.129) (3.181) (3.227) (3.182) (3.18) (2.964)
No. observations 165 165 165 165 165 165 165 165

Panel B: Share of firms with female ownership

Traditional plough use -13.168*** -12.316*** -13.114**** -12.769** -14.556*** -13.287** -13.54** -15.241***
(3.954) (3.772) (3.945) (4.488) (3.828) (4.128) (4.301) (4.06)
No. observations 123 123 123 123 123 123 123 123

Panel C: Share of political positions held by women 2000

Traditional plough use -5.196** -3.617* -4.29* -5.41** -4.48* -4.817** -5.029** -4.821**
(1.946) (1.707) (1.745) (1.777) (1.853) (1.804) (1.825) (1.782)
No. observations 144 144 144 144 144 144 144 144

Raw covariates 7 7 7 7 7 7 7 7

Notes: Reanalysis of Table 4 from Alesina et al. (2013), columns 1, 3, 5. DML results are based on 100 splits.
Median standard errors across splits are reported in brackets. The number of covariates reported does not include
the treatment variable. * significant at 10 percent; ** significant at 5 percent; *** significant at 1 percent.
22/48
Lecture Notes: Artificial Intelligence
Results: ATE

Results: Women and the Plough; Robustness and IV

(1) (2) (3) (4) (5) (6) (7) (8)
Lasso Reg. Tree Boosting Forest Neural Net. Ensemble Best OLS

Panel A: DML, Partially linear Model. Outcome: Female labour force participation

Traditional plough use -5.922 -6.744 -6.445 -5.812 -5.682 -5.925 -5.224 -9.234**
(4.754) (4.086) (4.194) (4.236) (4.664) (4.272) (4.319) (4.301)
No. observations 142 142 142 142 142 142 142 142
Raw covariates 36 36 36 36 36 36 36

Lasso Reg. Tree Boosting Forest Neural Net. Ensemble Best 2SLS

Panel B: DML-IV. Outcome: Female labour force participation

Traditional plough use -38.345* -36.85** -39.429** -36.961** -20.725 -33.645* -38.712* -28.516***
(14.966) (13.161) (13.574) (13.063) (18.414) (16.626) (19.116) (7.559)
No. observations 160 160 160 160 160 160 160 160
Raw covariates 17 17 17 17 17 17 17

Notes: Reanalysis of the main robustness checks of Alesina et al. (2013). DML results are based on 100 splits.
Median standard errors across splits are reported in brackets. The number of covariates reported does not include
the treatment variable. * significant at 10 percent; ** significant at 5 percent; *** significant at 1 percent.

23/48
Lecture Notes: Artificial Intelligence
Results: ATE

Important Remarks: Women and the Plough

I Combining quasi-experimental methods such as IV with causal ML

can improve the credibility of causal analysis
I Causal ML methods as a tool to implement supplementary analysis
systematically
I Tuning Parameter choices and Causal Random Forest results

I DML-IV results ⇒ an even larger effect of the plough adoption on

female labour force participation.

24/48
Lecture Notes: Artificial Intelligence
Results: ATE

The Structure of Tariffs and Long-Term Growth

I Nunn and Trefler, AEJ: Macro (2010)

I Research question: do skill-biased tariffs lead to long-term
growth?
I Cross country and industry-level regressions; positive
relationship
I Revisit main results including all controls
I Country-level OLS regression:

ln yc1 /yc0 = α + βSB SBτc0 + Xc0 βX + c ,

Controls: log average level of tariffs, log GDP per capita, log human capital,
log investment-to-GDP ratio, cohort fixed effects, region fixed effects, log of
output in skill-intensive and in unskilled-intensive industries

25/48
Lecture Notes: Artificial Intelligence
Results: ATE

Results: Structure of Tariffs and Growth; country-level

(1) (2) (3) (4) (5) (6) (7) (8)
Lasso Reg. Tree Boosting Forest Neural Net. Ensemble Best OLS

Panel A: Skill tariff correlation

Skill tariff correlation 0.018* 0.016 0.016 0.015 0.014 0.017 0.016 0.035***
(0.010) (0.011) (0.012) (0.011) (0.013) (0.012) (0.011) (0.01)

Panel B: Tariff differential (low cut-off)

Tariff differential (low cut-off) 0.009* 0.006 0.007 0.008 0.013 0.009 0.008 0.016***
(0.005) (0.005) (0.005) (0.005) (0.008) (0.006) (0.005) (0.005)

Panel C:Tariff differential (high cut-off)

Tariff differential (high cut-off) 0.011* 0.008 0.009 0.009 0.005 0.009 0.009 0.02***
(0.005) (0.005) (0.006) (0.006) (0.008) (0.006) (0.006) (0.004)

Observations 63 63 63 63 63 63 63 63
Raw covariates 17 17 17 17 17 17 17 17

Notes: Standard errors are reported in parentheses. Standard errors adjusted for variability across splits using the
median method are reported for the DML estimates. The number of covariates does not include the treatment
variable. ***, ** and * indicate significance at the 1%, 5% and 10% levels respectively.
industry
26/48
Lecture Notes: Artificial Intelligence
Results: ATE

Important Remarks: Structure of Tariffs and Growth

I The correlation between skill-biased tariffs and long-term

economic growth is not robust to controlling for an unknown
function of the average tariff level, country characteristics,
initial production structure, cohort and region fixed effects.
I Our findings are consistent with the alternative mechanism
described in Nunn and Trefler (2010), i.e. the existence of a
causal relationship between institutions and economic growth.

I Tuning Parameter choices and Causal Random Forest results.

27/48
Lecture Notes: Artificial Intelligence
Results: ATE

Results: (also) with the Causal Random Forest; country

level

(1) (2) (3)

Skill tariff correlation Tariff differential (low cut-off) Tariff differential (high cut-off)
Skilled tariff 0.014 0.010 0.004
(0.011) (0.006) (0.004)
Observations 63 63 63
Raw covariates 17 17 17

Notes: Analysis of Table 4 (columns 1, 2, 4) of Nunn and Trefler (2010) using the causal random forest. Standard
errors are reported in parentheses. The number of covariates does not include the treatment variable. ***, ** and
* indicate significance at the 1%, 5% and 10% levels respectively.

28/48
Lecture Notes: Artificial Intelligence
Methodology: HTE

HTE: Revisited Papers

I Causal Forest: Della Vigna and Kaplan QJE (2007) ”The

Fox News Effect: Media Bias and Voting”

I Generic Method: Loyalka et al. AEJ: Applied (2019) ”Does

Teacher Training Actually Work? Evidence from a Large-Scale
Randomized Evaluation of a National Teacher Training
Program”

29/48
Lecture Notes: Artificial Intelligence
Methodology: HTE

Methodology - HTEs - CRF

I Causal Random Forest (CRF) - adaptation of the random

forest algorithm of Breiman (2001) to the problem of
heterogeneous treatment effects estimation
I Key difference: the objective function that is optimized when
determining the split to be implemented; Goal: accurately
estimate treatment effects within leaves while preserving
heterogeneity across leaves
I Criterion function – two terms. First term: rewards a split
that increases the variance of treatment effects across leaves;
Second term: penalizes a split that increases within-leaf
variance

30/48
Lecture Notes: Artificial Intelligence
Methodology: HTE

Methodology - HTEs - CRF

I Sample split in: Training Sample, Estimation Sample,

Out-of-Bag Sample
I Repeat for a number of B trees to form a causal forest
(estimates from a single tree can have a high variance)
I The predicted treatment effect for each unit will be the
average of predictions for that particular observation across
the trees
I Honesty property
I Consistency, Asymptotic normality and valid confidence
intervals

31/48
Lecture Notes: Artificial Intelligence
Results: HTE

The Fox News Effect: Media Bias and Voting

I DellaVigna and Kaplan QJE (2007); 1907 citations

I Research question: does media affect voting behaviour? Effect
of introduction of Fox News on Republican votes in U.S.
I Data: town level analysis, positive effect
I Regression model (OLS):

rep votek,2000 −rep votek,1996 = βfox newsk,2000 +Xk Γ+θc +c

Controls: town-level in 2000 and as change 1996-2000 (population, employment,
share of African Americans and of Hispanics, share of males, share of the pop.
with some college, share of college grad., share of high school grad., urban share
of town, marriage rate, unemployment, income); cable company (deciles in no.
of channels provided and potential subscribers); district fixed effect

32/48
Lecture Notes: Artificial Intelligence
Results: HTE

The Fox News Effect: Media Bias and Voting

Analysis of HTE of Fox News, extending DellaVigna and Kaplan

(2007):

I Analyze whether treatment effect is heterogeneous

I Which covariates moderate treatment effect?

I Contrast causal ML results with heterogeneity analysis from
original paper, where
I effect modifiers: no. available cable channels; share of urban
population; and whether the town is in Republican district

33/48
Lecture Notes: Artificial Intelligence
Results: HTE

The Fox News Effect: Overall Heterogeneity Test

I Original paper’s ATE estimates: 0.004 and 0.007

(1) (2)
District dummies Cluster-robust
Fox News effect (ATE) 0.0065*** 0.0065**
(0.0016) (.0027)
Fox News effect above median 0.013*** 0.0072**
(0.0024) (0.0028)
Fox News effect below median -0.0033 0.0044
(0.0021) (0.0048)
95% CI for the difference (0.01009, 0.02255) (-0.00806, 0.01374)
Observations 9256 9256

Notes: This table reports the estimated average treatment effect and a test for overall heterogeneity using the
causal forest. Standard errors are reported in parentheses. ***, ** and * indicate significance at the 1%, 5% and
10% levels respectively.
2nd approach
34/48
Lecture Notes: Artificial Intelligence
Results: HTE

The Fox News Effect: Determinants of Heterogeneity

(1) (2) (3)

CATE below median CATE above median p-value difference
Panel A: District dummies
Employment rate, diff. btw. 2000 and 1990 0.00928 0.00064 0.00656
(0.00244) (0.00203)
Share high school degree 2000 0.00805 -7e-05 0.00884
(0.00226) (0.00213)
No. cable channels available 0.00877 -0.0044 6e-05
(0.00192) (0.00264)
Panel B: Cluster-robust
Employment rate, diff. btw. 2000 and 1990 0.00938 2e-04 0.06885
(0.00254) (0.00436)
Share high school degree 2000 0.00859 -0.00179 0.05296
(0.00303) (0.00442)
No. cable channels available 0.00857 -0.00495 0.02033
(0.00289) (0.00506)
Notes: This table reports the effect of Fox News on the Republican vote share for towns with values below (column
1) and above (column 2) the median of each variable. Column 3 presents the p-value for the null of no difference
between the estimates in columns 1 and 2. Standard errors are reported in parentheses.

35/48
Lecture Notes: Artificial Intelligence
Results: HTE

HTE in DellaVigna and Kaplan (2007)

36/48
Lecture Notes: Artificial Intelligence
Results: HTE

Important Remarks: The Fox News Effect

I Pointwise asymptotic normality for the causal forest is

provided for cases where the number of covariates is relatively
low; ⇒ Robustness checks with double forest
I Tuning Parameters - optimized via cross-validation
I We implement the orthogonalization as discussed Athey et al.
(2019)
I Uniformly valid standard errors for the causal forest is still an
open question

37/48
Lecture Notes: Artificial Intelligence
Methodology: HTE

Methodology - HTEs - Generic ML

I Generic ML method
I allows detection of heterogeneity in the treatment effect (BLP)
I computes the treatment effect for different groups of
observations (least affected/most affected) (GATES)
I describes which covariates are correlated the most with the
heterogeneity (CLAN)
I Approach based on data sampling
I Auxiliary Sample: a ML estimator for the CATE is constructed;
possibly biased and consistency is not required
I Main Sample: based on the ML proxies, can compute the
estimates of interest: BLP, GATE and CLAN
⇒ Inference: relies on many splits of data; take medians of p−values
and medians of CIs bounds
38/48
Lecture Notes: Artificial Intelligence
Methodology: HTE

Methodology - HTEs - Generic ML

I Conditional average treatment effect (CATE):

s0 (Z ) = E [Y (1)|Z ] − E [Y (0)|Z ]
I Auxiliary sample: obtain ML estimators for the CATE; Main sample:
compute the predicted treatment effects, S(Z), for each unit
I BLP regression:
Y = α0 X1 + β1 (D − p(Z )) + β2 (D − p(Z ))(S(Z ) − S(Z )) + 
⇒ If β2 is different from zero, there is heterogeneity in the
treatment effects
I GATES: Y = α0 X1 + K
P
k=1 γk (D − p(Z )) · 1(Gk ) + ν

I CLAN: δ1 = E [g (Y , Z )|G1 ]; δ2 = E [g (Y , Z )|GK ]

39/48
Lecture Notes: Artificial Intelligence
Results: HTE

Does Teacher Training Actually Work?

I Loyalka et al., AEJ: Applied (2019)

I Research question: evaluate teacher training intervention
(RCT) in China; main outcome: student math scores
I Data: 10,006 students in 201 schools. Randomization at
school level
I Regression model (OLS):

Yi,j = α0 + α1 Dj + Xij α + τk + i,j ,

Controls: student (age, gender,parent educational attainment, household
wealth; math self concept, math anxiety, motivation for math, time spent
studying math), class size, and teacher (gender, age, experience, education ,
rank, a teacher certification dummy, teacher majored in math; instructional
practices, teacher care, classroom management, communication)

40/48
Lecture Notes: Artificial Intelligence
Results: HTE

Teacher Training: Overall Heterogeneity Test

I Original paper’s ATE estimate: no significant ATE (0.01 SDs)

(1) (2)
ATE HET
Estimate 0.002 0.651
90% Confidence Interval (-0.068,0.072) (0.312,0.990)
p-value 1.000 0.0003
Observations 10006 10006

Notes: The estimates are obtained using neural network to produce the proxy predictor S(Z). The values reported
correspond to the medians over 100 splits.

comparison

41/48
Lecture Notes: Artificial Intelligence
Results: HTE

Teacher Training: GATES

42/48
Lecture Notes: Artificial Intelligence
Results: HTE

Teacher Training: CLAN

(1) (2) (3)
20% most affected 20% least affected p-value for the difference
Teacher college degree 0.039 0.800 0.000
(0.019,0.059) (0.780,0.820)
Teacher training hours 2.447 1.684 0.000
(2.399,2.494) (1.636,1.731)
Teacher ranking 0.666 0.405 0.000
(0.635,0.697) (0.374,0.437)
Student age 14.18 13.73 0.000
(14.11,14.25) (13.65,13.80)
Teacher experience (years) 16.18 13.16 0.000
(15.60,16.76) (12.58,13.74)
Student female 0.417 0.555 0.000
(0.385,0.449) (0.523,0.587)
Teacher age 37.51 35.01 0.000
(37.02,38.00) (34.52,35.50)
Student math score at baseline -0.029 0.169 0.005
(-0.088,0.031) (0.110,0.229)
Student baseline math anxiety 0.298 -0.219 0.000
(0.236,0.360) (-0.281,-0.157)
Class size 52.87 64.37 0.000
(51.82,53.93) (63.32,65.43)

Notes: The estimates are obtained using neural network to produce the proxy predictor S(Z). 90% confidence
intervals are reported in parenthesis. The variables Student math score at baseline and Student baseline math
anxiety are normalized. The values reported correspond to the medians over 100 splits.

43/48
Lecture Notes: Artificial Intelligence
Results: HTE

Simulation Study

I Perform MC simulations to asses the finite sample properties

of the DML estimator and compare it with OLS
I DGP: Partially linear model
I Sample size: 100
I No. relevant covariates smaller than no. covariates available

I Vary the nuisance functions: linear or nonlinear (interactions,

squares, logarithms, inverse and exponential).
I Vary the no. of available covariates: 20, 50, 90
details

44/48
Lecture Notes: Artificial Intelligence
Results: HTE

Simulation Study: Linear Case

Mean MAE Var MSE OLS hit rate
k=20
OLS 2.030347 0.073340 0.011994 0.012795 –
DML (Lasso) 2.204096 0.308212 0.271943 0.310879 0.800000
DML (Reg. Tree) 1.689803 0.308441 0.121442 0.216449 0.850000
DML (Boosting) 1.980974 0.123810 0.035727 0.035732 0.630000
DML (Forest) 2.017441 0.195251 0.063020 0.062694 0.820000
DML (Neural Net.) 2.179827 0.295220 0.149802 0.180642 0.810000
DML (Elastic Net) 2.023434 0.190474 0.065182 0.065079 0.850000
k=50
OLS 1.998316 0.096766 0.020967 0.020760 –
DML (Lasso) 2.305637 0.361159 0.230212 0.321324 0.840000
DML (Reg. Tree) 1.599136 0.465340 0.120046 0.279538 0.870000
DML (Boosting) 1.964103 0.161710 0.076277 0.076803 0.710000
DML (Forest) 2.001251 0.194066 0.064021 0.063382 0.710000
DML (Neural Net.) 2.161565 0.277444 0.120533 0.145431 0.800000
DML (Elastic Net) 1.995786 0.208867 0.068792 0.068122 0.720000
k=90
OLS 2.015636 0.232447 0.126762 0.125739 –
DML (Lasso) 2.152435 0.248923 0.150110 0.171845 0.520000
DML (Reg. Tree) 1.644803 0.358840 0.101281 0.226434 0.610000
DML (Boosting) 1.979060 0.172211 0.057695 0.057556 0.430000
DML (Forest) 1.980332 0.181217 0.057536 0.057348 0.400000
DML (Neural Net.) 2.086499 0.238486 0.099034 0.105526 0.500000
DML (Elastic Net) 1.974706 0.169177 0.054517 0.054611 0.400000

45/48
Lecture Notes: Artificial Intelligence
Results: HTE

Simulation Study: Nonlinear Case

Mean MAE Var MSE OLS hit rate
k=20
OLS 1.987438 0.074142 0.018767 0.018737 –
DML (Lasso) 1.973458 0.072649 0.024766 0.025223 0.560000
DML (Reg. Tree) 1.933508 0.086160 0.030411 0.034529 0.570000
DML (Boosting) 1.980180 0.084764 0.023779 0.023934 0.540000
DML (Forest) 1.977051 0.073242 0.018031 0.018378 0.450000
DML (Neural Net.) 1.976888 0.067966 0.022784 0.023090 0.540000
DML (Elastic Net) 1.971941 0.061446 0.017436 0.018049 0.520000
k=50
OLS 2.036816 0.106510 0.027996 0.029072 –
DML (Lasso) 2.006489 0.082303 0.032754 0.032469 0.410000
DML (Reg. Tree) 1.953161 0.092469 0.041107 0.042890 0.440000
DML (Boosting) 2.070185 0.121154 0.035497 0.040068 0.540000
DML (Forest) 2.017207 0.077788 0.028629 0.028639 0.370000
DML (Neural Net.) 2.010202 0.081012 0.033766 0.033533 0.430000
DML (Elastic Net) 2.004967 0.081342 0.029084 0.028818 0.370000
k=90
OLS 1.962616 0.266024 0.167484 0.167207 –
DML (Lasso) 2.008953 0.090226 0.027975 0.027775 0.150000
DML (Reg. Tree) 1.960935 0.102387 0.034753 0.035931 0.230000
DML (Boosting) 2.044449 0.117992 0.037492 0.039092 0.240000
DML (Forest) 2.005692 0.079685 0.025495 0.025273 0.160000
DML (Neural Net.) 2.011866 0.080616 0.027288 0.027156 0.140000
DML (Elastic Net) 1.996920 0.075328 0.024669 0.024431 0.190000

46/48
Lecture Notes: Artificial Intelligence
Results: HTE

Conclusion and Takeaways

I Why ML helps improve causal inference?

1. Can improve credibility of the unconfoundedness assumption

and deal with the many potential covariates problem
2. Powerful tools in using data to recover complex interactions
among variables and flexibly estimate the relationship between
the outcome, the treatment and covariates
3. Systematic in model selection – Reproducibility, More Robust
Sensitivity Analysis
4. Heterogeneous Treatment Effects: systematic search, large
number of potential effect modifiers, multiple hypothesis
testing

47/48
Lecture Notes: Artificial Intelligence
Results: HTE

Other Related Work

I Uncovering Heterogeneity in the Effectiveness of Lung Cancer

Screening: A Causal Random Forest Approach
(with Kevin ten Haaf, Harry de Koning and Max Welz)

I Atlantic Causal Inference Competition Revisited: A Monte

Carlo Evaluation
(with Christian Wirths)

48/48
Lecture Notes: Artificial Intelligence

Appendix

1/9
Lecture Notes: Artificial Intelligence

Results: Structure of Tariffs and Growth; no IIT

(1) (2) (3) (4) (5) (6) (7) (8)
Lasso Reg. Tree Boosting Forest Neural Net. Ensemble Best OLS

Panel A: Skill tariff correlation

Skill tariff correlation 0.026 0.019 0.188* 0.080 0.035 0.153 0.146 0.064***
(0.053) (0.072) (0.103) (0.124) (0.045) (0.115) (0.128) (0.02)

Panel B: Tariff differential (low cut-off)

Tariff differential (low cut-off) 0.011 0.013 0.078* 0.044 0.022 0.058 0.058 0.032***
(0.034) (0.028) (0.042) (0.071) (0.022) (0.072) (0.078) (0.01)

Panel C: Tariff differential (high cut-off)

Tariff differential (high cut-off) 0.017 0.011 0.055 0.050 0.018 0.063 0.058 0.040***
(0.036) (0.031) (0.035) (0.064) (0.030) (0.058) (0.065) (0.009)

Observations 1004 1004 1004 1004 1004 1004 1004 1004

Raw covariates 36 36 36 36 36 36 36 36

Notes: Standard errors are reported in parentheses. Standard errors adjusted for variability across splits using the
median method are reported for the DML estimates. The number of covariates does not include the treatment
variable. ***, ** and * indicate significance at the 1%, 5% and 10% levels respectively.
back
2/9
Lecture Notes: Artificial Intelligence

Results: Structure of Tariffs and Growth; with IIT

(1) (2) (3) (4) (5) (6) (7) (8)
Lasso Reg. Tree Boosting Forest Neural Net. Ensemble Best OLS

Panel A: Skill tariff correlation

Skill tariff correlation 0.046 0.020 0.164* 0.086 0.035 0.142 0.103 0.066***
(0.045) (0.063) (0.091) (0.109) (0.051) (0.105) (0.111) (0.019)

Panel B: Tariff differential (low cut-off)

Tariff differential (low cut-off) 0.026 0.015 0.069* 0.048 0.022 0.073 0.055 0.033***
(0.024) (0.033) (0.037) (0.061) (0.026) (0.059) (0.059) (0.01)

Panel C: Tariff differential (high cut-off)

Tariff differential (high cut-off) 0.023 0.013 0.068* 0.044 0.019 0.063 0.048 0.039***
(0.021) (0.029) (0.040) (0.059) (0.021) (0.051) (0.056) (0.009)

Observations 1004 1004 1004 1004 1004 1004 1004 1004

Raw covariates 37 37 37 37 37 37 37 37

Notes: Standard errors are reported in parentheses. Standard errors adjusted for variability across splits using the
median method are reported for the DML estimates. The number of covariates does not include the treatment
variable. ***, ** and * indicate significance at the 1%, 5% and 10% levels respectively.
back
3/9
Lecture Notes: Artificial Intelligence

Fox News - Causal Forest: 2nd approach for heterogeneity:

Best Linear Predictor
(1) (2)
District dummies Cluster-robust
Mean forest prediction 1.168*** 1.117**
(0.316) (0.518)
Differential forest prediction 2.52473*** 0.453
(0.460) (0.638)
Observations 9256 9256

Notes:This table reports the tests for the average treatment effect (mean forest prediction) and for heterogeneity
(differential forest prediction), based on the approach of Chernozhukov et al. (2018), using the causal forest.
Standard errors are reported in parenthesis. * significant at 10 percent; ** significant at 5 percent; *** significant
at 1 percent.

back

4/9
Lecture Notes: Artificial Intelligence

Fox News - Causal Forest: ATE and test for heterogeneity,

Double Forest
(1) (2)
District dummies Cluster-robust
Fox News effect (ATE) 0.0064*** 0.0068**
(0.0016) (0.0027)
Fox News effect above median 0.013*** 0.0074**
(0.0024) (0.0028)
Fox News effect below median -0.0044 0.0042
(0.0021) (0.0051)
95% CI for the difference (0.0111, 0.0236) (-0.0083, 0.0146)
Observations 9256 9256

Notes: This table reports the estimated average treatment effect and a test for overall heterogeneity using the
causal forest. Standard errors are reported in parentheses. ***, ** and * indicate significance at the 1%, 5% and
10% levels respectively.

back

5/9
Lecture Notes: Artificial Intelligence

Fox News - Causal Forest: HTE analysis, Double Forest

(1) (2) (3)
CATE below median CATE above median p-value difference
Panel A: District dummies
No. cable channels 2000 0.00883 (0.00683) 0.00639 (0.00148) 0.72696
No. potential cable subscribers 0.00757 (0.00239) 0.00593 (0.00168) 0.57511
Empl. rate, diff. btw. 2000 and 1990 0.00924 (0.00244) 0.00021 (0.00204) 0.00448
Share African American 2000 0.00186 (0.00271) 0.00739 (0.00173) 0.08565
Share married 2000 0.00683 (0.00229) 0.0055 (0.00229) 0.68062
Unempl. rate, diff. btw. 2000 and 1990 0.00334 (0.00238) 0.00872 (0.00225) 0.10075
Decile 10 in no. cable channels 0.00857 (0.00192) -0.00475 (0.00264) 5e-05
Unempl. rate 2000 0.005 (0.00215) 0.00753 (0.00235) 0.42731
Share high school degree, diff. btw. 2000 and 1990 0.00839 (0.00202) 0.00307 (0.00268) 0.113
Panel B: Cluster-robust
No. cable channels 2000 0.00696 (0.00834) 0.00681 (0.00255) 0.98641
No. potential cable subscribers 2000 0.00769 (0.00636)- 0.00614 (0.00293) 0.82479
Share with some college 2000 0.00566 (0.00341) 0.00815 (0.00385) 0.62783
Unempl. rate, diff. btw. 2000 and 1990 0.00332 (0.00303) 0.00918 (0.0032) 0.18415
Decile 9 in no. cable channels 0.00571 (0.0028) 0.01411 (0.00633) 0.22478
Decile 10 in no. cable channels 0.00879 (0.00302) -0.00426 (0.00516) 0.02902
Empl. rate, diff. btw. 2000 and 1990 0.0099 (0.0026) -0.00017 (0.00459) 0.05626
Share African American 2000 0.00158 (0.00526) 0.00771 (0.00274) 0.30138
Median Income 2000 0.00574 (0.00403) 0.00672 (0.00268) 0.84033

Notes: This table reports the effect of Fox News on the Republican vote share for towns with values below (column
1) and above (column 2) the median of each variable. Column 3 presents the p-value for the null of no difference
between the estimates in columns 1 and 2. Standard errors are reported in parentheses. back
6/9
Lecture Notes: Artificial Intelligence

Generic ML method: comparison of ML methods

(1) (2) (3)

Elastic net Neural network Random forest
Best BLP 0.012 0.014 0.011
Best GATES 0.115 0.121 0.099

Notes: The table compares the performance of the three ML methods used to produce the proxy predictors. The
performance measures Best BLP and Best GATES are computed as medians over 100 splits.

back

7/9
Lecture Notes: Artificial Intelligence

Generic ML Method Tuning Parameters

Parameter Value
Minimum number of treated and control units 5.00
Number of covariates considered for a split 5.00
Number of trees 1000

Notes: This table shows the values of the tuning parameters used in the generic machine learning method for the
random forest.

back

8/9
Lecture Notes: Artificial Intelligence

Simulation Study: Details

I No. covariates in outcome equation: 10

I No. covariates in treatment equation: 7
I Common covariates: 4
I True value of the parameter of interest is 2.
I Values of other covariates: {-5, -3, -1, 1, 3, 5}
I Nonlinear functions include interactions, squares, logarithms,
inverse and exponential.
I ML Methods: Lasso, Reg, Trees, Boosting, Forest, Neural
Network and Elastic Net
back

9/9