Econometric Data Science:

A Predictive Modeling Approach

Francis X. Diebold
University of Pennsylvania

Version 2024.08.04
 Econometric Data Science:
A Predictive Modeling Approach
Econometric Data Science

Francis X. Diebold
 Copyright © 2013-2024
by Francis X. Diebold.

This work is freely available for your use, although preliminary and al-
ways evolving. It is licensed under the Creative Commons Attribution-
NonCommercial-NoDerivatives 4.0 International License. (Briefly: I retain
copyright, but you can use, copy and distribute non-commercially, so long as
you give me attribution and do not modify. To view a copy of the license,
go to http://creativecommons.org/licenses/by-nc-nd/4.0/.) In return
I ask that you please cite the book whenever appropriate, as: “Diebold, F.X.
(2024), Econometric Data Science: A Predictive Modeling Approach, Depart-
ment of Economics, University of Pennsylvania, http://www.ssc.upenn.
edu/~fdiebold/Textbooks.html.”
 To my undergraduates,
who continually surprise and inspire me
Brief Table of Contents

About the Author xix

About the Cover xxi

Preface xxix

I Beginnings 1

1 Introduction to Econometrics 3

2 Graphics and Graphical Style 13

II Cross Sections 29

3 Regression Under Ideal Conditions 31

4 Non-Normality 73

5 Group Heterogeneity and Indicator Variables 95

6 Nonlinearity 103

7 Heteroskedasticity 121

8 Limited Dependent Variables 133

9 Causal Estimation 143

ix
x BRIEF TABLE OF CONTENTS

III Time Series 163

10 Trend and Seasonality 165

11 Serial Correlation 191

12 Forecasting 237

13 Structural Change 243

14 Vector Autoregression 247

15 Dynamic Heteroskedasticity 265

IV Econometrics and Machine Learning 281

16 Misspecification and Model Selection 283

V Appendices 293

A Probability and Statistics Review 295

B Construction of the Wage Datasets 303

C Some Popular Books Worth Encountering 305

Bibliography 307

Index 307
Detailed Table of Contents

About the Author xix

About the Cover xxi

Preface xxix

I Beginnings 1

1 Introduction to Econometrics 3
1.1 Welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.1 Who Uses Econometrics? . . . . . . . . . . . . . . . . . 3
1.1.2 What Distinguishes Econometrics? . . . . . . . . . . . 5
1.2 Types of Recorded Economic Data . . . . . . . . . . . . . . . 5
1.3 Online Information and Data . . . . . . . . . . . . . . . . . . 6
1.4 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Tips on How to use this book . . . . . . . . . . . . . . . . . . 8
1.6 Exercises, Problems and Complements . . . . . . . . . . . . . 10
1.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Graphics and Graphical Style 13

2.1 Simple Techniques of Graphical Analysis . . . . . . . . . . . . 13
2.1.1 Univariate Graphics . . . . . . . . . . . . . . . . . . . 14
2.1.2 Multivariate Graphics . . . . . . . . . . . . . . . . . . 14
2.1.3 Summary and Extension . . . . . . . . . . . . . . . . . 17
2.2 Elements of Graphical Style . . . . . . . . . . . . . . . . . . . 19
2.3 U.S. Hourly Wages . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4 Concluding Remark . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Exercises, Problems and Complements . . . . . . . . . . . . . 22

xi
xii DETAILED TABLE OF CONTENTS

2.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.7 Graphics Legend: Edward Tufte . . . . . . . . . . . . . . . . . 27

II Cross Sections 29

3 Regression Under Ideal Conditions 31

3.1 Preliminary Graphics . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Regression as Curve Fitting . . . . . . . . . . . . . . . . . . . 33
3.2.1 Bivariate, or Simple, Linear Regression . . . . . . . . . 33
3.2.2 Multiple Linear Regression . . . . . . . . . . . . . . . . 36
3.2.3 Onward . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3 Regression as a Probability Model . . . . . . . . . . . . . . . . 38
3.3.1 A Population Model and a Sample Estimator . . . . . 38
3.3.2 Notation, Assumptions and Results: The Ideal Condi-
tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
A Bit of Matrix Notation . . . . . . . . . . . . . . . . . 39
Assumptions: The Ideal Conditions (IC) . . . . . . . . 40
Results Under the IC . . . . . . . . . . . . . . . . . . . 41
3.4 A Wage Equation . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4.1 Mean dependent var 2.342 . . . . . . . . . . . . . . . . 45
3.4.2 S.D. dependent var .561 . . . . . . . . . . . . . . . . . 45
3.4.3 Sum squared resid 319.938 . . . . . . . . . . . . . . . . 46
3.4.4 Log likelihood -938.236 . . . . . . . . . . . . . . . . . . 46
3.4.5 F statistic 199.626 . . . . . . . . . . . . . . . . . . . . 48
3.4.6 Prob(F statistic) 0.000000 . . . . . . . . . . . . . . . . 49
3.4.7 S.E. of regression .492 . . . . . . . . . . . . . . . . . . 49
3.4.8 R-squared .232 . . . . . . . . . . . . . . . . . . . . . . 50
3.4.9 Adjusted R-squared .231 . . . . . . . . . . . . . . . . . 51
3.4.10 Akaike info criterion 1.423 . . . . . . . . . . . . . . . . 51
3.4.11 Schwarz criterion 1.435 . . . . . . . . . . . . . . . . . . 52
3.4.12 A Bit More on AIC and SIC . . . . . . . . . . . . . . . 52
3.4.13 Hannan-Quinn criter. 1.427 . . . . . . . . . . . . . . . 53
3.4.14 Durbin-Watson stat. 1.926 . . . . . . . . . . . . . . . . 53
3.4.15 The Residual Scatter . . . . . . . . . . . . . . . . . . . 53
3.4.16 The Residual Plot . . . . . . . . . . . . . . . . . . . . . 53
3.5 Least Squares and Optimal Point Prediction . . . . . . . . . . 55
DETAILED TABLE OF CONTENTS xiii

3.6 Optimal Interval and Density Prediction . . . . . . . . . . . . 58

3.7 Regression Output from a Predictive Perspective . . . . . . . . 59
3.8 Multicollinearity . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.8.1 Perfect and Imperfect Multicollinearity . . . . . . . . . 61
3.9 Beyond Fitting the Conditional Mean:
Quantile regression . . . . . . . . . . . . . . . . . . . . . . . . 63
3.10 Exercises, Problems and Complements . . . . . . . . . . . . . 64
3.11 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.12 Regression’s Inventor: Carl Friedrich Gauss . . . . . . . . . . . 71

4 Non-Normality 73
4.0.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.1 Assessing Normality . . . . . . . . . . . . . . . . . . . . . . . 74
4.1.1 QQ Plots . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.1.2 Residual Sample Skewness and Kurtosis . . . . . . . . 75
4.1.3 The Jarque-Bera Test . . . . . . . . . . . . . . . . . . 75
4.2 Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.2.1 Outlier Detection . . . . . . . . . . . . . . . . . . . . . 76
Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Leave-One-Out and Leverage . . . . . . . . . . . . . . 76
4.3 Robust Estimation . . . . . . . . . . . . . . . . . . . . . . . . 77
4.3.1 Robustness Iteration . . . . . . . . . . . . . . . . . . . 77
4.3.2 Least Absolute Deviations . . . . . . . . . . . . . . . . 78
4.4 Wage Determination . . . . . . . . . . . . . . . . . . . . . . . 79
4.4.1 W AGE . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.4.2 LW AGE . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.5 Exercises, Problems and Complements . . . . . . . . . . . . . 92

5 Group Heterogeneity and Indicator Variables 95

5.1 0-1 Dummy Variables . . . . . . . . . . . . . . . . . . . . . . . 95
5.2 Group Dummies in the Wage Regression . . . . . . . . . . . . 97
5.3 Exercises, Problems and Complements . . . . . . . . . . . . . 99
5.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.5 Dummy Variables, ANOVA, and Sir Ronald Fischer . . . . . . 102
xiv DETAILED TABLE OF CONTENTS

6 Nonlinearity 103
6.1 Models Linear in Transformed Variables . . . . . . . . . . . . 104
6.1.1 Logarithms . . . . . . . . . . . . . . . . . . . . . . . . 104
Log-Log Regression . . . . . . . . . . . . . . . . . . . . 104
Log-Lin Regression . . . . . . . . . . . . . . . . . . . . 105
Lin-Log Regression . . . . . . . . . . . . . . . . . . . . 105
6.1.2 Box-Cox and GLM . . . . . . . . . . . . . . . . . . . . 106
Box-Cox . . . . . . . . . . . . . . . . . . . . . . . . . . 106
GLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.2 Intrinsically Non-Linear Models . . . . . . . . . . . . . . . . . 107
6.2.1 Nonlinear Least Squares . . . . . . . . . . . . . . . . . 107
6.3 Series Expansions . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.4 A Final Word on Nonlinearity and the IC . . . . . . . . . . . 110
6.5 Selecting a Non-Linear Model . . . . . . . . . . . . . . . . . . 110
6.5.1 t and F Tests, and Information Criteria . . . . . . . . 110
6.5.2 The RESET Test . . . . . . . . . . . . . . . . . . . . 111
6.6 Non-Linearity in Wage Determination . . . . . . . . . . . . . . 111
6.6.1 Non-Linearity in Continuous and Discrete Variables Si-
multaneously . . . . . . . . . . . . . . . . . . . . . . . 113
6.7 Exercises, Problems and Complements . . . . . . . . . . . . . 115

7 Heteroskedasticity 121
7.1 Consequences of Heteroskedasticity for Estimation, Inference,
and Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.2 Detecting Heteroskedasticity . . . . . . . . . . . . . . . . . . . 122
7.2.1 Graphical Diagnostics . . . . . . . . . . . . . . . . . . 122
7.2.2 Formal Tests . . . . . . . . . . . . . . . . . . . . . . . 123
The Breusch-Pagan-Godfrey Test (BPG) . . . . . . . . 123
White’s Test . . . . . . . . . . . . . . . . . . . . . . . . 124
7.3 Dealing with Heteroskedasticity . . . . . . . . . . . . . . . . . 126
7.3.1 Adjusting Standard Errors . . . . . . . . . . . . . . . . 126
7.3.2 Adjusting Density Forecasts . . . . . . . . . . . . . . . 127
7.4 Exercises, Problems and Complements . . . . . . . . . . . . . 127

8 Limited Dependent Variables 133

8.1 Binary Response . . . . . . . . . . . . . . . . . . . . . . . . . 133
8.2 The Logit Model . . . . . . . . . . . . . . . . . . . . . . . . . 135
DETAILED TABLE OF CONTENTS xv

8.2.1 Logit . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

8.2.2 Ordered Logit . . . . . . . . . . . . . . . . . . . . . . . 136
8.2.3 Complications . . . . . . . . . . . . . . . . . . . . . . . 137
Marginal Effects . . . . . . . . . . . . . . . . . . . . . . 137
R2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
8.3 Classification and “0-1 Forecasting” . . . . . . . . . . . . . . . 138
8.4 Exercises, Problems and Complements . . . . . . . . . . . . . 139

9 Causal Estimation 143

9.1 Predictive Modeling vs. Causal Estimation . . . . . . . . . . . 144
9.1.1 Predictive Modeling and P-Consistency . . . . . . . . . 145
9.1.2 Causal Estimation and T-Consistency . . . . . . . . . . 145
9.1.3 Correlation vs. Causality, and P-Consistency vs. T-
Consistency . . . . . . . . . . . . . . . . . . . . . . . . 145
9.2 Reasons for Failure of IC2 . . . . . . . . . . . . . . . . . . . . 147
9.2.1 Omitted Variables (“Confounders”) . . . . . . . . . . . 147
9.2.2 Misspecified Functional Form . . . . . . . . . . . . . . 148
9.2.3 Measurement Error . . . . . . . . . . . . . . . . . . . . 148
9.2.4 Simultaneity . . . . . . . . . . . . . . . . . . . . . . . . 149
9.2.5 Sample Selection . . . . . . . . . . . . . . . . . . . . . 150
9.3 Confronting Failures of IC2 . . . . . . . . . . . . . . . . . . . 150
9.3.1 Controling for Omitted Variables . . . . . . . . . . . . 150
9.3.2 Instrumental Variables . . . . . . . . . . . . . . . . . . 151
9.3.3 Randomized Controlled Trials (RCT’s) . . . . . . . . . 152
9.3.4 Regression Discontinuity Designs (RDD’s) . . . . . . . 153
9.3.5 Propensity-Score Matching . . . . . . . . . . . . . . . . 154
9.3.6 Differences in Differences (“Diff in Diff”) and Panel Data155
9.4 Internal and External Validity . . . . . . . . . . . . . . . . . . 156
9.4.1 Internal Validity and its Problems . . . . . . . . . . . . 156
9.4.2 External Validity . . . . . . . . . . . . . . . . . . . . . 158
9.4.3 A Final Thought . . . . . . . . . . . . . . . . . . . . . 158
9.5 Exercises, Problems and Complements . . . . . . . . . . . . . 158

III Time Series 163

10 Trend and Seasonality 165

xvi DETAILED TABLE OF CONTENTS

10.1 Linear Trend . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

10.2 Non-Linear Trend . . . . . . . . . . . . . . . . . . . . . . . . . 167
10.2.1 Quadratic Trend . . . . . . . . . . . . . . . . . . . . . 167
10.2.2 Exponential Trend . . . . . . . . . . . . . . . . . . . . 168
10.2.3 Non-Linearity in Liquor Sales Trend . . . . . . . . . . 170
10.3 Seasonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
10.3.1 Seasonal Dummies . . . . . . . . . . . . . . . . . . . . 174
10.3.2 More General Calendar Effects . . . . . . . . . . . . . 176
10.4 Trend and Seasonality in Liquor Sales . . . . . . . . . . . . . . 177
10.5 Exercises, Problems and Complements . . . . . . . . . . . . . 179

11 Serial Correlation 191

11.1 Characterizing Serial Correlation (in Population, Mostly) . . . 192
11.1.1 Covariance Stationary Time Series . . . . . . . . . . . 193
11.1.2 Estimation and Inference for the Mean, Autocorrela-
tion and Partial Autocorrelation Functions . . . . . . . 199
Sample Mean . . . . . . . . . . . . . . . . . . . . . . . 199
Sample Autocorrelations . . . . . . . . . . . . . . . . . 200
Sample Partial Autocorrelations . . . . . . . . . . . . . 203
11.2 Modeling Serial Correlation (in Population) . . . . . . . . . . 204
11.2.1 White Noise . . . . . . . . . . . . . . . . . . . . . . . . 204
11.2.2 The Lag Operator . . . . . . . . . . . . . . . . . . . . 210
11.2.3 Autoregression . . . . . . . . . . . . . . . . . . . . . . 211
The AR(1) Process . . . . . . . . . . . . . . . . . . . . 212
The AR(p) Process . . . . . . . . . . . . . . . . . . . . 218
11.3 Modeling Serial Correlation (in Sample) . . . . . . . . . . . . 220
11.3.1 Detecting Serial Correlation . . . . . . . . . . . . . . . 220
Residual Scatterplots . . . . . . . . . . . . . . . . . . . 221
Durbin-Watson . . . . . . . . . . . . . . . . . . . . . . 223
The Breusch-Godfrey Test . . . . . . . . . . . . . . . . 224
11.3.2 The Residual Correlogram . . . . . . . . . . . . . . . . 226
11.3.3 Estimation with Serial Correlation . . . . . . . . . . . 228
11.4 Exercises, Problems and Complements . . . . . . . . . . . . . 232

12 Forecasting 237
12.1 *** . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
12.2 Exercises, Problems and Complements . . . . . . . . . . . . . 241
DETAILED TABLE OF CONTENTS xvii

13 Structural Change 243

13.1 Gradual Parameter Evolution . . . . . . . . . . . . . . . . . . 243
13.2 Abrupt Parameter Breaks . . . . . . . . . . . . . . . . . . . . 244
13.2.1 Exogenously-Specified Breaks . . . . . . . . . . . . . . 244
13.2.2 The Chow test with Endogenous Break Selection . . . 244
13.3 Dummy Variables and Omitted Variables, Again and Again . 245
13.3.1 Dummy Variables . . . . . . . . . . . . . . . . . . . . . 245
13.3.2 Omitted Variables . . . . . . . . . . . . . . . . . . . . 245
13.4 Recursive Analysis and CUSUM . . . . . . . . . . . . . . . . . 246
13.5 Structural Change in Liquor Sales Trend . . . . . . . . . . . . 246
13.6 Exercises, Problems and Complements . . . . . . . . . . . . . 246

14 Vector Autoregression 247

14.0.1 Event Studies and Synthetic Controls . . . . . . . . . . 247
14.1 Predictive Causality . . . . . . . . . . . . . . . . . . . . . . . 249
14.2 Application: Housing Starts and Completions . . . . . . . . . 250
14.3 Exercises, Problems and Complements . . . . . . . . . . . . . 264

15 Dynamic Heteroskedasticity 265

15.1 The Basic ARCH Process . . . . . . . . . . . . . . . . . . . . 266
15.2 The GARCH Process . . . . . . . . . . . . . . . . . . . . . . . 269
15.3 Extensions of ARCH and GARCH Models . . . . . . . . . . . 273
15.3.1 Asymmetric Response . . . . . . . . . . . . . . . . . . 273
15.3.2 Exogenous Variables in the Volatility Function . . . . . 274
15.3.3 Regression with GARCH disturbances and GARCH-M 274
15.3.4 Component GARCH . . . . . . . . . . . . . . . . . . . 275
15.3.5 Mixing and Matching . . . . . . . . . . . . . . . . . . . 275
15.4 Estimating, Forecasting and Diagnosing GARCH Models . . . 275
15.5 Exercises, Problems and Complements . . . . . . . . . . . . . 276
15.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

IV Econometrics and Machine Learning 281

16 Misspecification and Model Selection 283

16.0.1 Information Criteria (Hard Thresholding) . . . . . . . 283
16.1 Cross Validation (Hard Thresholding) . . . . . . . . . . . . . . 287
xviii DETAILED TABLE OF CONTENTS

16.2 Stepwise Selection (Hard Thresholding) . . . . . . . . . . . . . 288

16.2.1 Forward . . . . . . . . . . . . . . . . . . . . . . . . . . 288
16.2.2 Backward . . . . . . . . . . . . . . . . . . . . . . . . . 288
16.3 Bayesian Shrinkage (Soft Thresholding) . . . . . . . . . . . . . 288
16.4 Selection and Shrinkage (Mixed Hard and Soft Thresholding) . 289
16.4.1 Penalized Estimation . . . . . . . . . . . . . . . . . . . 289
16.4.2 The Lasso . . . . . . . . . . . . . . . . . . . . . . . . . 289
16.5 Distillation: Principal Components . . . . . . . . . . . . . . . 291
16.5.1 Distilling “X Variables” into Principal Components . . 291
16.5.2 Principal Components Regression . . . . . . . . . . . . 291
16.6 Exercises, Problems and Complements . . . . . . . . . . . . . 292

V Appendices 293

A Probability and Statistics Review 295

A.1 Populations: Random Variables, Distributions and Moments . 295
A.1.1 Univariate . . . . . . . . . . . . . . . . . . . . . . . . . 295
A.1.2 Multivariate . . . . . . . . . . . . . . . . . . . . . . . . 296
A.2 Samples: Sample Moments . . . . . . . . . . . . . . . . . . . . 297
A.2.1 Univariate . . . . . . . . . . . . . . . . . . . . . . . . . 297
A.2.2 Multivariate . . . . . . . . . . . . . . . . . . . . . . . . 299
A.3 Finite-Sample and Asymptotic Sampling Distributions of the
Sample Mean . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
A.3.1 Exact Finite-Sample Results . . . . . . . . . . . . . . . 299
A.3.2 Approximate Asymptotic Results (Under Weaker As-
sumptions) . . . . . . . . . . . . . . . . . . . . . . . . 300
A.4 Exercises, Problems and Complements . . . . . . . . . . . . . 300

B Construction of the Wage Datasets 303

C Some Popular Books Worth Encountering 305

Bibliography 307

Index 307
About the Author

Francis X. Diebold is Paul F. Miller, Jr. and E. Warren Shafer Miller Pro-
fessor of Social Sciences, and Professor of Economics, Finance, and Statistics,
University of Pennsylvania.
His research focuses on predictive modeling of financial asset markets,
macroeconomic fundamentals, and the interface. He has made well-known
contributions to the measurement and modeling of asset-return volatility,
business cycles, yield curves, and network connectedness. He has published
more than 150 scientific papers and 8 books, and he is regularly ranked among
globally most-cited economists.
His academic research is firmly linked to practical matters: He has served
as Research Economist at the Board of Governors of the Federal Reserve
System, Executive Director at Morgan Stanley Investment Management, and
Chairman of the Federal Reserve System’s Model Validation Council.
He has won both undergraduate and graduate economics “teacher of the
year” awards, and his academic “family” includes thousands of undergraduate
students and approximately 75 Ph.D. students.

xix
About the Cover

The colorful painting is Enigma, by Glen Josselsohn, from Wikimedia Com-

mons. As noted there:

Glen Josselsohn was born in Johannesburg in 1971. His art has

been exhibited in several art galleries around the country, with a
number of sell-out exhibitions on the South African art scene ...
Glen’s fascination with abstract art comes from the likes of Picasso,
Pollock, Miro, and local African art.

I used the painting mostly just because I like it. But econometrics is indeed
something of an enigma, part economics and part statistics, part science and
part art, hunting faint and fleeting signals buried in large amounts of noise.
Yet it often succeeds.

xxi
List of Figures

1.1 Resources for Economists Web Page . . . . . . . . . . . . . . . 7

1.2 R Homepage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Python Homepage . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1 1-Year Goverment Bond Yield, Levels and Changes . . . . . . 15
2.2 Histogram of 1-Year Government Bond Yield . . . . . . . . . . 16
2.3 Bivariate Scatterplot, 1-Year and 10-Year Government Bond
Yields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 Scatterplot Matrix, 1-, 10-, 20- and 30-Year Government Bond
Yields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.5 Distributions of Wages and Log Wages . . . . . . . . . . . . . 22
2.6 Tufte Teaching, with a First Edition Book by Galileo . . . . . 27
3.1 Distributions of Log Wage, Education and Experience . . . . . 32
3.2 (Log Wage, Education) Scatterplot . . . . . . . . . . . . . . . 34
3.3 (Log Wage, Education) Scatterplot with Superimposed Re-
gression Line . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.4 Regression Output . . . . . . . . . . . . . . . . . . . . . . . . 42
3.5 Wage Regression Residual Scatter . . . . . . . . . . . . . . . . 54
3.6 Wage Regression Residual Plot . . . . . . . . . . . . . . . . . 55
3.7 Carl Friedrich Gauss . . . . . . . . . . . . . . . . . . . . . . . 71
4.1 OLS Wage Regression . . . . . . . . . . . . . . . . . . . . . . 81
4.2 OLS Wage Regression: Residual Plot . . . . . . . . . . . . . . 82
4.3 OLS Wage Regression: Residual Histogram and Statistics . . . 83
4.4 OLS Wage Regression: Residual Gaussian QQ Plot . . . . . . 84
4.5 OLS Wage Regression: Leave-One-Out Plot . . . . . . . . . . 85
4.6 LAD Wage Regression . . . . . . . . . . . . . . . . . . . . . . 86
4.7 OLS Log Wage Regression . . . . . . . . . . . . . . . . . . . . 87

xxiii
xxiv LIST OF FIGURES

4.8 OLS Log Wage Regression: Residual Plot . . . . . . . . . . . . 88

4.9 OLS Log Wage Regression: Residual Histogram and Statistics 89
4.10 OLS Log Wage Regression: Residual Gaussian QQ Plot . . . . 90
4.11 OLS Log Wage Regression: Leave-One-Out Plot . . . . . . . . 91
4.12 LAD Log Wage Regression . . . . . . . . . . . . . . . . . . . . 92
5.1 Histograms for Wage Covariates . . . . . . . . . . . . . . . . . 96
5.2 Wage Regression on Education and Experience . . . . . . . . . 98
5.3 Wage Regression on Education, Experience and Group Dummies 98
5.4 Residual Scatter from Wage Regression on Education, Expe-
rience and Group Dummies . . . . . . . . . . . . . . . . . . . 99
5.5 Sir Ronald Fischer . . . . . . . . . . . . . . . . . . . . . . . . 102
6.1 Basic Linear Wage Regression . . . . . . . . . . . . . . . . . . 112
6.2 Quadratic Wage Regression . . . . . . . . . . . . . . . . . . . 113
6.3 Wage Regression on Education, Experience, Group Dummies,
and Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.4 Wage Regression with Continuous Non-Linearities and Inter-
actions, and Discrete Interactions . . . . . . . . . . . . . . . . 115
6.5 Regression Output . . . . . . . . . . . . . . . . . . . . . . . . 120
7.1 Final Wage Regression . . . . . . . . . . . . . . . . . . . . . . 123
7.2 Squared Residuals vs. Years of Education . . . . . . . . . . . 124
7.3 BPG Test Regression and Results . . . . . . . . . . . . . . . . 125
7.4 White Test Regression and Results . . . . . . . . . . . . . . . 125
7.5 Wage Regression with Heteroskedasticity-Robust Standard Er-
rors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.6 Regression Weighted by Fit From White Test Regression . . . 130
9.1 Example of Difference in Difference Parallel Assumption . . . 157
10.1 Various Linear Trends . . . . . . . . . . . . . . . . . . . . . . 166
10.2 Various Quadratic Trends . . . . . . . . . . . . . . . . . . . . 168
10.3 Various Exponential Trends . . . . . . . . . . . . . . . . . . . 170
10.4 Log-Quadratic Trend Estimation . . . . . . . . . . . . . . . . 171
10.5 Residual Plot, Log-Quadratic Trend Estimation . . . . . . . . 172
10.6 Liquor Sales Log-Quadratic Trend Estimation with Seasonal
Dummies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
LIST OF FIGURES xxv

10.7 Residual Plot, Liquor Sales Log-Quadratic Trend Estimation

With Seasonal Dummies . . . . . . . . . . . . . . . . . . . . . 174
10.8 Liquor Sales . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
10.9 Log Liquor Sales . . . . . . . . . . . . . . . . . . . . . . . . . 178
10.10Linear Trend Estimation . . . . . . . . . . . . . . . . . . . . . 179
10.11Residual Plot, Linear Trend Estimation . . . . . . . . . . . . . 180
10.12Estimation Results, Linear Trend with Seasonal Dummies . . 181
10.13Residual Plot, Linear Trend with Seasonal Dummies . . . . . 182
10.14Seasonal Pattern . . . . . . . . . . . . . . . . . . . . . . . . . 183
11.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
11.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
11.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
11.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
11.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
11.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
11.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
11.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
11.9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
11.10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
11.11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
11.12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
11.13BG Test Equation, 4 Lags . . . . . . . . . . . . . . . . . . . . 225
11.14BG Test Equation, 8 Lags . . . . . . . . . . . . . . . . . . . . 226
11.15Residual Correlogram From Trend + Seasonal Model . . . . . 227
11.16Trend + Seasonal Model with Four Autoregressive Lags . . . 229
11.17Trend + Seasonal Model with Four Lags of y, Residual Plot . 230
11.18Trend + Seasonal Model with Four Autoregressive Lags, Resid-
ual Scatterplot . . . . . . . . . . . . . . . . . . . . . . . . . . 231
11.19Trend + Seasonal Model with Four Autoregressive Lags, Resid-
ual Autocorrelations . . . . . . . . . . . . . . . . . . . . . . . 231
14.1 Housing Starts and Completions, 1968 - 1996 . . . . . . . . . . 251
14.2 Housing Starts Correlogram . . . . . . . . . . . . . . . . . . . 251
14.3 Housing Starts Autocorrelations and Partial Autocorrelations 252
14.4 Housing Completions Correlogram . . . . . . . . . . . . . . . . 253
xxvi LIST OF FIGURES

14.5 Housing Completions Autocorrelations and Partial Autocorre-

lations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
14.6 Housing Starts and Completions Sample Cross Correlations . . 254
14.7 VAR Starts Model . . . . . . . . . . . . . . . . . . . . . . . . 255
14.8 VAR Starts Residual Correlogram . . . . . . . . . . . . . . . . 256
14.9 VAR Starts Equation - Sample Autocorrelation and Partial
Autocorrelation . . . . . . . . . . . . . . . . . . . . . . . . . . 257
14.10VAR Completions Model . . . . . . . . . . . . . . . . . . . . . 258
14.11VAR Completions Residual Correlogram . . . . . . . . . . . . 259
14.12VAR Completions Equation - Sample Autocorrelation and Par-
tial Autocorrelation . . . . . . . . . . . . . . . . . . . . . . . . 260
14.13Housing Starts and Completions - Causality Tests . . . . . . . 261
14.14Housing Starts Forecast . . . . . . . . . . . . . . . . . . . . . 262
14.15Housing Starts Forecast and Realization . . . . . . . . . . . . 262
14.16Housing Completions Forecast . . . . . . . . . . . . . . . . . . 263
14.17Housing Completions Forecast and Realization . . . . . . . . . 263
16.1 Lasso and Ridge Comparison . . . . . . . . . . . . . . . . . . 290
List of Tables

2.1 Yield Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . 25

xxvii
xxviii LIST OF TABLES
Preface

Econometric Data Science: A Predictive Modeling Approach should be

useful to students in a variety of fields – in economics, of course, but also
statistics, business, finance, public policy, and engineering – as well as in a
variety of emerging fields like data science and machine learning. The “pre-
dictive modeling” perspective, emphasized throughout, connects students to
modern and evolving themes, including the interface between econometrics
and machine learning.
I have used the material successfully for many years in my undergraduate
econometrics course at Penn, as background for various other undergraduate
courses, and in master’s-level executive education courses given to profession-
als in economics, business, finance and government. It is directly accessible
at the undergraduate and master’s levels; the only prerequisite is an intro-
ductory probability and statistics course.
Many people have contributed to the development of this book. One
way or another, all of the following deserve thanks: Xu Cheng, University
of Pennsylvania; Barbara Chizzolini, Bocconi University; Frank Di Traglia,
University of Oxford; Carlo Favero, Bocconi University; Bruce Hansen, Uni-
versity of Wisconsin; Frank Schorfheide, University of Pennsylvania; Jim
Stock, Harvard University; Mark Watson, Princeton University.
I am especially grateful to the University of Pennsylvania, which for many
years has provided an unparalleled intellectual home, the perfect incubator

xxix
xxx PREFACE

for the ideas that have congealed here. Related, I am grateful to an army of
energetic and enthusiastic Penn graduate and undergraduate students, who
read and improved much of the manuscript and code over many years.
Finally, I apologize and accept full responsibility for the many errors and
shortcomings that undoubtedly remain – minor and major – despite ongoing
efforts to eliminate them.

Francis X. Diebold
Philadelphia

Sunday 4th August, 2024

Econometric Data Science
 Part I

Beginnings

1
Chapter 1

Introduction to Econometrics

1.1 Welcome

1.1.1 Who Uses Econometrics?

Econometrics is important — it is used constantly in business, finance, eco-

nomics, government, consulting and many other fields. Econometric models
are used routinely for tasks ranging from data description to policy analysis,
and ultimately they guide many important decisions.
To develop a feel for the tremendous diversity of econometric applications,
let’s explore some of the areas where they feature prominently, and the cor-
responding diversity of decisions supported.
One key field is economics (of course), broadly defined. Governments,
businesses, policy organizations, central banks, financial services firms, and
economic consulting firms around the world routinely use econometrics.
Governments, central banks and policy organizations use econometric mod-
els to guide monetary policy, fiscal policy, as well as education and training,
health, and transfer policies.
Businesses use econometrics for strategic planning tasks. These include
management strategy of all types including operations management and con-
trol (hiring, production, inventory, investment, ...), marketing (pricing, dis-
tributing, advertising, ...), accounting (budgeting revenues and expenditures),

3
4 CHAPTER 1. INTRODUCTION TO ECONOMETRICS

and so on.
Sales modeling is a good example. Firms routinely use econometric models
of sales to help guide management decisions in inventory management, sales
force management, production planning, and new market entry.
More generally, business firms use econometric models to help decide what
to produce (What product or mix of products should be produced?), when to
produce (Should we build up inventories now in anticipation of high future
demand? How many shifts should be run?), how much to produce and how
much capacity to build (What are the trends in market size and market share?
Are there cyclical or seasonal effects? How quickly and with what pattern will
a newly-built plant or a newly-installed technology depreciate?), and where
to produce (Should we have one plant or many? If many, where should we
locate them?). Firms also use forecasts of future prices and availability of
inputs to guide production decisions.
Econometric models are also crucial in financial services, including asset
management, asset pricing, mergers and acquisitions, investment banking,
and insurance. Portfolio managers, for example, are keenly interested in
the empirical modeling and understanding of asset returns (stocks, bonds,
exchange rates, commodity prices, ...).
Econometrics is similarly central to financial risk management. In recent
decades, econometric methods for volatility modeling have been developed
and widely applied to evaluate and insure risks associated with asset portfo-
lios, and to price assets such as options and other derivatives.
Finally, econometrics is central to the work of a wide variety of consulting
firms, many of which support the business functions already mentioned. Lit-
igation support, for example, is also a very active area, in which econometric
models are routinely used for damage assessment (e.g., lost earnings), “but
for” analyses, and so on.
Indeed these examples are just the tip of the iceberg. Surely you can think
1.2. TYPES OF RECORDED ECONOMIC DATA 5

of many more situations in which econometrics is used.

1.1.2 What Distinguishes Econometrics?

Econometrics is much more than just “statistics using economic data,” al-
though it is of course very closely related to statistics.

ˆ Econometrics has special focus on prediction. In many respects the goal

of econometrics is to help agents (consumers, firms, investors, policy
makers, ...) make better decisions, and good forecasts are key inputs to
good decisions.

ˆ Econometrics must confront the special issues and features that arise
routinely in economic data, such as heteroskedasticity and serial corre-
lation. (Don’t worry if those terms mean nothing to you now.)

ˆ Econometrics must confront the special problems arising due to its largely
non-experimental nature: Model mis-specification, structural change,
etc.

With so many applications and issues in econometrics, you might fear

that a huge variety of econometric techniques exists, and that you’ll have to
master all of them. Fortunately, that’s not the case. Instead, a relatively
small number of tools form the common core of much econometric modeling.
We will focus on those underlying core principles.

1.2 Types of Recorded Economic Data

Several aspects of economic data will concern us frequently.

One issue is whether the data are continuous or binary. Continuous
data take values on a continuum, as for example with GDP growth, which in
principle can take any value in the real numbers. Binary data, in contrast,
6 CHAPTER 1. INTRODUCTION TO ECONOMETRICS

take just two values, as with a 0-1 indicator for whether or not someone
purchased a particular product during the last month.
Another issue is whether the data are recorded over time, over space, or
some combination of the two. Time series data are recorded over time, as
for example with U.S. GDP, which is measured once per quarter. A GDP
dataset might contain quarterly data for, say, 1960 to the present.
Cross sectional data, in contrast, are recorded over space (at a point in
time), as with yesterday’s closing stock price for each of the U.S. S&P 500
firms. The data structures can be blended, as for example with a time series
of cross sections. If, moreover, the cross-sectional units are identical over
time, we speak of panel data, or longitudinal data. An example would
be the daily closing stock price for each of the U.S. S&P 500 firms, recorded
over each of the last 30 days.

1.3 Online Information and Data

Much useful information is available on the web. The best way to learn about
what’s out there is to spend a few hours searching the web for whatever inter-
ests you. Here we mention just a few key “must-know” sites. Resources for
Economists, maintained by the American Economic Association, is a fine por-
tal to almost anything of interest to economists. (See Figure 1.1.) It contains
hundreds of links to data sources, journals, professional organizations, and so
on. FRED (Federal Reserve Economic Data) is a tremendously convenient
source for economic data. The National Bureau of Economic Research site
has data on U.S. business cycles, and the Real-Time Data Research Center
at the Federal Reserve Bank of Philadelphia has real-time vintage macroeco-
nomic data.
1.4. SOFTWARE 7

Figure 1.1: Resources for Economists Web Page

1.4 Software

Econometric software tools are widely available. Two good and time-honored
high-level environments with extensive capabilities are Stata and EViews.
Stata has particular strength in cross sections and panels, and EViews
has particular strength in time series. Both reflect a balance of generality
and specialization well-suited to the sorts of tasks that will concern us. If you
feel more comfortable with another environment, however, that’s fine, as none
of our discussion is wed to Stata or EViews (or any computing environment)
in any way.
There are also many flexible and more open-ended “mid-level” environ-
ments in which you can quickly program, evaluate, and apply new tools
8 CHAPTER 1. INTRODUCTION TO ECONOMETRICS

Figure 1.2: R Homepage

and techniques. R is one popular such environment, with special strengths

in modern statistical methods and graphical data analysis. (See Figure 1.2.)
Other notable and increaingly-popular environments include Python (see Fig-
ure 1.3) and Julia.

1.5 Tips on How to use this book

As you navigate through the book, keep the following in mind.

ˆ Hyperlinks to internal items (table of contents, index, footnotes, etc.)

appear in red.

ˆ Hyperlinks to bibliographic references appear in green.

ˆ Hyperlinks to the web appear in cyan.1

ˆ Hyperlinks to external files (e.g., video) appear in blue.

1
Obviously web links sometimes go dead. I attempt effort to keep them updated. If you’re encountering
an unusual number of dead links, you’re probably using an outdated edition of the book.
1.5. TIPS ON HOW TO USE THIS BOOK 9

Figure 1.3: Python Homepage

ˆ Many images in the digital version of this book are clickable to reach
related material.

ˆ Key concepts appear in bold, and they also appear in the book’s (hy-
perlinked) index.

ˆ Additional related materials appear on the book’s web page. These may
include book updates, presentation slides, datasets, and computer code.

ˆ Facebook group: Diebold Econometrics.

ˆ Additional relevant material sometimes appears on Twitter @Francis-

Diebold and on the No Hesitations blog.

ˆ The data that we use in the book from national income accounts, firms,
people, financial and other markets, etc. – are fictitious. Sometimes
they data are based on real data for various real countries, firms, etc.,
and sometimes they are artificially constructed. Ultimately, however,
any resemblance to particular countries, firms, etc. should be viewed as
coincidental and irrelevant.
10 CHAPTER 1. INTRODUCTION TO ECONOMETRICS

ˆ The end-of-chapter “Exercises, Problems and Complements” sections

are of central importance and should be studied carefully. Exercises
are generally straightforward checks of your understanding. Problems,
in contrast, are generally significantly more involved, whether analyti-
cally or computationally. Complements generally introduce important
auxiliary material not covered in the main text.

1.6 Exercises, Problems and Complements

1. (No empirical example is definitive)

Recall that, as mentioned in the text, most chapters contain at least one
extensive empirical example. At the same time, those examples should
not be taken as definitive or complete treatments – there is no such
thing. A good idea is to think of the implicit “Problem 0” at the end of
each chapter as “Obtain the relevant data for the empirical modeling in
this chapter, and produce a superior analysis”.

2. (Nominal, ordinal, interval and ratio data)

We emphasized time series, cross-section and panel data, whether con-
tinuous or discrete, but there are other complementary categorizations.
In particular, distinctions are often made among nominal data, ordi-
nal data, interval data, and ratio data. Which are most common
and useful in economics and related fields, and why?

3. (Software differences and bugs: caveat emptor)

Be warned: no software is perfect. In fact, all software is highly im-
perfect. The results obtained when modeling in different software en-
vironments may differ – sometimes a little and sometimes a lot – for
a variety of reasons. The details of implementation may differ across
packages, for example, and small differences in details can sometimes
1.7. NOTES 11

produce large differences in results. Hence, it is important that you

understand precisely what your software is doing (insofar as possible,
as some software documentation is more complete than others). And
of course, quite apart from correctly-implemented differences in details,
deficient implementations can and do occur: there is no such thing as
bug-free software.

1.7 Notes

R is available for free as part of a large and highly-successful open-source

project. RStudio provides a fine R working environment, and, like R, it’s
free. A good R tutorial, first given on Coursera and then moved to YouTube,
is here2 . R-bloggers is a massive compendium with all sorts of information
about all things R.
Python and Julia are also free. Fine Python and Julia tutorials are avail-
able at QuantEcon.

2
https://blog.revolutionanalytics.com/2012/12/coursera-videos.html
12 CHAPTER 1. INTRODUCTION TO ECONOMETRICS
Chapter 2

Graphics and Graphical Style

It’s almost always a good idea to begin an econometric analysis with graphical
data analysis. When compared to the modern array of econometric methods,
graphical analysis might seem trivially simple, perhaps even so simple as to
be incapable of delivering serious insights. Such is not the case: in many
respects the human eye is a far more sophisticated tool for data analysis
and modeling than even the most sophisticated statistical techniques. Put
differently, graphics is a sophisticated technique. That’s certainly not to
say that graphical analysis alone will get the job done – certainly, graphical
analysis has its limitations of its own – but it’s usually the best place to
start. With that in mind, we introduce in this chapter some simple graphical
techniques, and we consider some basic elements of graphical style.

2.1 Simple Techniques of Graphical Analysis

We will segment our discussion into two parts: univariate (one variable) and
multivariate (more than one variable). Because graphical analysis “lets the
data speak for themselves,” it is most useful when the dimensionality of
the data is low; that is, when dealing with univariate or low-dimensional
multivariate data.

13
14 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE

2.1.1 Univariate Graphics

First consider time series data. Graphics is used to reveal patterns in time
series data. The great workhorse of univariate time series graphics is the
simple time series plot, in which the series of interest is graphed against
time.
In the top panel of Figure 2.1, for example, we present a time series plot
of a 1-year Government bond yield over approximately 500 months. A num-
ber of important features of the series are apparent. Among other things,
its movements appear sluggish and persistent, it appears to trend gently up-
ward until roughly the middle of the sample, and it appears to trend gently
downward thereafter.
The bottom panel of Figure 2.1 provides a different perspective; we plot
the change in the 1-year bond yield, which highlights volatility fluctuations.
Interest rate volatility is very high in mid-sample.
Univariate graphical techniques are also routinely used to assess distri-
butional shape, whether in time series or cross sections. A histogram, for
example, provides a simple estimate of the probability density of a random
variable. The observed range of variation of the series is split into a number
of segments of equal length, and the height of the bar placed at a segment
is the percentage of observations falling in that segment.1 In Figure 2.2 we
show a histogram for the 1-year bond yield.

2.1.2 Multivariate Graphics

When two or more variables are available, the possibility of relations be-
tween the variables becomes important, and we use graphics to uncover the
existence and nature of such relationships. We use relational graphics to
1
In some software packages (e.g., Eviews), the height of the bar placed at a segment is simply the
number, not the percentage, of observations falling in that segment. Strictly speaking, such histograms are
not density estimators, because the “area under the curve” doesn’t add to one, but they are equally useful
for summarizing the shape of the density.
2.1. SIMPLE TECHNIQUES OF GRAPHICAL ANALYSIS 15

Figure 2.1: 1-Year Goverment Bond Yield, Levels and Changes

16 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE

Figure 2.2: Histogram of 1-Year Government Bond Yield

display relationships and flag anomalous observations. You already under-

stand the idea of a bivariate scatterplot.2 In Figure 2.3, for example, we
show a bivariate scatterplot of the 1-year U.S. Treasury bond rate vs. the
10-year U.S. Treasury bond rate, from 1960.01 to 2005.03. The scatterplot
indicates that the two move closely together; in particular, they are positively
correlated.
Thus far all our discussion of multivariate graphics has been bivariate.
That’s because graphical techniques are best-suited to low-dimensional data.
Much recent research has been devoted to graphical techniques for high-
dimensional data, but all such high-dimensional graphical analysis is subject
to certain inherent limitations.
One simple and popular scatterplot technique for high-dimensional data –
and one that’s been around for a long time – is the scatterplot matrix, or
multiway scatterplot. The scatterplot matrix is just the set of all possible
bivariate scatterplots, arranged in the upper right or lower left part of a
2
Note that “connecting the dots” is generally not useful in scatterplots. This contrasts to time series
plots, for which connecting the dots is fine and is typically done.
2.1. SIMPLE TECHNIQUES OF GRAPHICAL ANALYSIS 17

Figure 2.3: Bivariate Scatterplot, 1-Year and 10-Year Government Bond Yields

matrix to facilitate comparisons. If we have data on N variables, there are

N 2 −N
2 such pairwise scatterplots. In Figure 2.4, for example, we show a
scatterplot matrix for the 1-year, 10-year, 20-year, and 30-year U.S. Treasury
Bond rates, 1960.01-2005.03. There are a total of six pairwise scatterplots,
and the multiple comparison makes clear that although the interest rates are
closely related in each case, with a regression slope of approximately one, the
relationship is more precise in some cases (e.g., 20- and 30-year rates) than
in others (e.g., 1- and 30-year rates).

2.1.3 Summary and Extension

Let’s summarize and extend what we’ve learned about the power of graphics:

a. Graphics helps us summarize and reveal patterns in univariate time-series

data. Time-series plots are helpful for learning about many features of
time-series data, including trends, seasonality, cycles, the nature and lo-
cation of any aberrant observations (“outliers”), structural breaks, etc.

b. Graphics helps us summarize and reveal patterns in univariate cross-section

18 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE

Figure 2.4: Scatterplot Matrix, 1-, 10-, 20- and 30-Year Government Bond Yields
2.2. ELEMENTS OF GRAPHICAL STYLE 19

data. Histograms are helpful for learning about distributional shape.

c. Graphics helps us identify relationships and understand their nature, in

both multivariate time-series and multivariate cross-section environments.
The key graphical device is the scatterplot, which can help us to begin
answering many questions, including: Does a relationship exist? Is it
linear or nonlinear? Are there outliers?

d. Graphics helps us identify relationships and understand their nature in

panel data. One can, for example, examine cross-sectional histograms
across time periods, or time series plots across cross-sectional units.

e. Graphics facilitates and encourages comparison of different pieces of data

via multiple comparisons. The scatterplot matrix is a classic example
of a multiple comparison graphic.

We might add to this list another item of tremendous relevance in our age
of big data: Graphics enables us to summarize and learn from huge datasets.

2.2 Elements of Graphical Style

In the preceding sections we emphasized the power of graphics and introduced

various graphical tools. As with all tools, however, graphical tools can be
used effectively or ineffectively, and bad graphics can be far worse than no
graphics. In this section you’ll learn what makes good graphics good and bad
graphics bad. In doing so you’ll learn to use graphical tools effectively.
Bad graphics is like obscenity: it’s hard to define, but you know it when
you see it. Conversely, producing good graphics is like good writing: it’s
an iterative, trial-and-error procedure, and very much an art rather than a
science. But that’s not to say that anything goes; as with good writing, good
graphics requires discipline. There are at least three keys to good graphics:
20 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE

a. Know your audience, and know your goals.

b. Show the data, and only the data, within the bounds of reason.

c. Revise and edit, again and again (and again). Graphics produced using
software defaults are almost never satisfactory.

We can use a number of devices to show the data. First, avoid distorting
the data or misleading the viewer, in order to reveal true data variation rather
than spurious impressions created by design variation. Thus, for example,
avoid changing scales in midstream, use common scales when performing
multiple comparisons, and so on. The sizes of effects in graphics should match
their size in the data.
Second, minimize, within reason, non-data ink (ink used to depict any-
thing other than data points). Avoid chartjunk (elaborate shadings and
grids that are hard to decode, superfluous decoration including spurious 3-D
perspective, garish colors, etc.)
Third, choose a graph’s aspect ratio (the ratio of the graph’s height, h,
to its width, w) to maximize pattern revelation. A good aspect ratio often
makes the average absolute slope of line segments connecting the data points
approximately equal 45 degrees. This procedure is called banking to 45
degrees.
Fourth, maximize graphical data density. Good graphs often display lots
of data, indeed so much data that it would be impossible to learn from them
in tabular form.3 Good graphics can present a huge amount of data in a
concise and digestible form, revealing facts and prompting new questions, at
both “micro” and “macro” levels.4
3
Conversely, for small amounts of data, a good table may be much more appropriate and informative
than a graphic.
4
Note how maximization of graphical data density complements our earlier prescription to maximize the
ratio of data ink to non-data ink, which deals with maximizing the relative amount of data ink. High data
density involves maximizing as well the absolute amount of data ink.
2.3. U.S. HOURLY WAGES 21

Graphs can often be shrunken greatly with no loss, as with sparklines

(tiny graphics, typically time-series plots, meant to flow with text) and the
sub-plots in multiple comparison graphs, increasing the amount of data ink
per unit area.

2.3 U.S. Hourly Wages

We now begin our examination of Current Population Survey (CPS) wage

data, which we will use extensively. Here we use the CPS-I hourly wage
data; for a detailed description see Appendix B. Figure 5.1 has four panels;
consider first the left panels. In the upper left we show a histogram of hourly
wage for the 1000+ people in the dataset. The distribution is clearly skewed
right, with a mean around $12/hour. In the lower left panel we show a density
estimate (basically just a smoothed histogram) together with the best fitting
normal distribution (a normal with mean and variance equal to the sample
mean and sample variance of the wage data). Clearly the normal distribution
fits poorly.
The right panels of Figure 5.1 have the same structure, except that we now
work with (natural) logarithms of the wages rather than the original “raw”
wage data.5 The log is often used as a “symmetrizing” transformation for data
with a right-skewed distribution, because the log transformation compresses
things, pulling in long right tails. Sometimes taking logs can even produce
approximate normality.6 Inspection of the log wage histogram in the upper
right panel reveals that the log wage does indeed appear more symmetrically
distributed, and comparison of the density estimate to the best-fitting normal
in the lower-right panel indicates approximate normality of the log wage.
5
Whenever we say “log” in this book, we mean “natural log”.
6
Recall the famous lognormal density: A random variable x is defined to be lognormal if log(x) is normal.
Hence if the wage data is approximately lognormally distributed, then, log(wage) will be approximately
normal. Of course lognormality may or may hold – whether data are lognormal is entirely an empirical
matter.
22 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE

Figure 2.5: Distributions of Wages and Log Wages

2.4 Concluding Remark

Ultimately good graphics proceeds just like good writing, so if good writing
is good thinking, then so too is good graphics. So the next time you hear an
ignorant person blurt out something along the lines of “I don’t like to write; I
like to think,” rest assured, his/her writing, thinking, and graphics are likely
all poor.

2.5 Exercises, Problems and Complements

1. (NBER recession bars: A useful graphical device)

In U.S. time-series situations it’s often useful to superimpose “NBER
Recession Bars” on time-series plots, to help put things in context. You
can find the dates of NBER expansions and contractions at http://
www.nber.org/cycles.html.

2. (Empirical warm-up)
2.5. EXERCISES, PROBLEMS AND COMPLEMENTS 23

(a) Obtain time series of quarterly real GDP and quarterly real con-
sumption for a country of your choice. Provide details.
(b) Display time-series plots and a scatterplot (put consumption on the
vertical axis).
(c) Convert your series to growth rates in percent, and again display
time series plots.
(d) From now on use the growth rate series only.
(e) For each series, provide summary statistics (e.g., mean, standard
deviation, range, skewness, kurtosis, ...).
(f) For each series, perform t-tests of the null hypothesis that the pop-
ulation mean growth rate is 2 percent.
(g) For each series, calulate 90 and 95 percent confidence intervals for
the population mean growth rate. For each series, which interval is
wider, and why?
(h) Regress consumption on GDP. Discuss.

3. (Simple vs. partial correlation)

The set of pairwise scatterplots that comprises a multiway scatterplot
provides useful information about the joint distribution of the set of vari-
ables, but it’s incomplete information and should be interpreted with
care. A pairwise scatterplot summarizes information regarding the sim-
ple correlation between, say, x and y. But x and y may appear highly
related in a pairwise scatterplot even if they are in fact unrelated, if
each depends on a third variable, say z. The crux of the problem is
that there’s no way in a pairwise scatterplot to examine the correlation
between x and y controlling for z, which we call partial correlation.
When interpreting a scatterplot matrix, keep in mind that the pairwise
scatterplots provide information only on simple correlation.
24 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE

4. (Graphics and Big Data)

Another aspect of the power of statistical graphics comes into play in
the analysis of large datasets, so it’s increasingly more important in our
era of “Big Data”: Graphics enables us to present a huge amount of
data in a small space, and hence helps to make huge datasets coherent.
We might, for example, have supermarket-scanner data, recorded in five-
minute intervals for a year, on the quantities of goods sold in each of
four food categories – dairy, meat, grains, and vegetables. Tabular or
similar analysis of such data is simply out of the question, but graphics
is still straightforward and can reveal important patterns.

5. (Color)
There is a temptation to believe that color graphics is always better
than grayscale. That’s often far from the truth, and in any event, color
is typically best used sparingly.

a. Color can be (and often is) chartjunk. How and why?

b. Color has no natural ordering, despite the evident belief in some quar-
ters that it does. What are the implications for “heat map” graphics?
Might shades of a single color (e.g., from white or light gray through
black) be better?
c. On occasion, however, color can aid graphics both in showing the data
and in appealing to the viewer. One key “show the data” use is in
annotation. Can you think of others? What about uses in appealing
to the viewer?
d. Keeping in mind the principles of graphical style, formulate as many
guidelines for color graphics as you can.

6. (Principles of tabular style)

2.5. EXERCISES, PROBLEMS AND COMPLEMENTS 25

Table 2.1: Yield Statistics

Maturity
(Months) ȳ σ̂y ρ̂y (1) ρ̂y (12)
6 4.9 2.1 0.98 0.64
12 5.1 2.1 0.98 0.65
24 5.3 2.1 0.97 0.65
36 5.6 2.0 0.97 0.65
60 5.9 1.9 0.97 0.66
120 6.5 1.8 0.97 0.68

Notes: We present descriptive statistics for end-of-month yields at various maturities. We show sample
mean, sample standard deviation, and first- and twelfth-order sample autocorrelations. Data are from the
Board of Governors of the Federal Reserve System. The sample period is January 1985 through December
2008.

The power of tables for displaying data and revealing patterns is very
limited compared to that of graphics, especially in this age of Big Data.
Nevertheless, tables are of course sometimes helpful, and there are prin-
ciples of tabular style, just as there are principles of graphical style.
Compare, for example, the nicely-formatted Table 2.1 (no need to worry
about what it is or from where it comes...) to what would be produced
by a spreadsheet such as Excel.
Try to formulate a set of principles of tabular style. (Hint: One principle
is that vertical lines should almost never appear in tables, as in the table
above.)

7. (More on graphical style: Appeal to the viewer)

Other graphical guidelines help us appeal to the viewer. First, use clear
and modest type, avoid mnemonics and abbreviations, and use labels
rather then legends when possible. Second, make graphics self-contained;
a knowledgeable reader should be able to understand your graphics with-
out reading pages of accompanying text. Third, as with our prescriptions
for showing the data, avoid chartjunk.
26 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE

8. (The “golden” aspect ratio, visual appeal, and showing the data)
A time-honored approach to visual graphical appeal is use of an aspect
ratio such that height is to width as width is to the sum of height and
width. This turns out to correspond to height approximately sixty per-
cent of width, the so-called “golden ratio.” Graphics that conform to
the golden ratio, with height a bit less than two thirds of width, are
visually appealing. Other things the same, it’s a good idea to keep the
golden ratio in mind when producing graphics. Other things are not
always the same, however. In particular, the golden aspect ratio may
not be the one that maximizes pattern revelation (e.g., by banking to
45 degrees).

9. (Google Charts and more)

Check out Google Charts at https://developers.google.com/chart/.
Poke around. What’s good? What’s bad? Can you use it to do
sparklines?
Check out www.zevross.com.

2.6 Notes

R implements a variety of sophisticated graphical techniques. The R library

ggplot2 breaks graphs into semantic components, such as scales and layers
and allows customization of practically anything in a graph. In many re-
spects, it represents the cutting edge of statistical graphics software.
A popular Python data visualization library is seaborn. It offers simple
plot syntax and a good control of figure aesthetics.
2.7. GRAPHICS LEGEND: EDWARD TUFTE 27

2.7 Graphics Legend: Edward Tufte

Figure 2.6: Tufte Teaching, with a First Edition Book by Galileo

This chapter has been heavily influenced by Tufte (1983), as are all modern
discussions of statistical graphics.7 Tufte’s book is an insightful and enter-
taining masterpiece on graphical style, and I recommend enthusiastically. Be
sure to check out his web page and other books, which go far beyond his 1983
work.

7
Photo details follow.
Date: 7 February 2011.
Source: http://www.flickr.com/photos/roebot/5429634725/in/set-72157625883623225.
Author: Aaron Fulkerson.
Originally posted to Flickr by Roebot at http://flickr.com/photos/40814689@N00/5429634725. Reviewed
on 24 May 2011 by the FlickreviewR robot and confirmed to be licensed under the terms of the cc-by-sa-2.0.
Licensed under the Creative Commons Attribution-Share Alike 2.0 Generic license.
28 CHAPTER 2. GRAPHICS AND GRAPHICAL STYLE
 Part II

Cross Sections

29
Chapter 3

Regression Under Ideal Conditions

You have already been introduced to probability and statistics, but chances
are that you could use a bit of review before plunging into regression, so
begin by studying Appendix A. Be warned, however: it is no substitute
for a full-course introduction to probability and statistics, which you should
have had already. Instead it is intentionally much more narrow, reviewing
some material related to moments of random variables, which we will use
repeatedly. It also introduces notation, and foreshadows certain ideas, that
we develop subsequently in greater detail.

3.1 Preliminary Graphics

In this chapter we’ll be working with cross-sectional data on log wages, ed-
ucation and experience. We already examined the distribution of log wages.
For convenience we reproduce it in Figure 3.1, together with the distributions
of the new data on education and experience.

31
32 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

Figure 3.1: Distributions of Log Wage, Education and Experience

3.2. REGRESSION AS CURVE FITTING 33

3.2 Regression as Curve Fitting

3.2.1 Bivariate, or Simple, Linear Regression

Suppose that we have data on two variables, y and x, as in Figure 3.2, and
suppose that we want to find the linear function of x that best fits y, where
“best fits” means that the sum of squared (vertical) deviations of the data
points from the fitted line is as small as possible. When we “run a regression,”
or “fit a regression line,” that’s what we do. The estimation strategy is called
least squares, or sometimes “ordinary least squares” to distinguish it from
fancier versions that we’ll introduce later.
The specific data that we show in Figure 3.2 are log wages (LWAGE, y)
and education (EDUC, x) for a random sample of nearly 1500 people, as
described in Appendix B.
Let us elaborate on the fitting of regression lines, and the reason for the
name “least squares.” When we run the regression, we use a computer to fit
the line by solving the problem
N
X
min (yi − β1 − β2 xi )2 ,
β
i=1

where β is shorthand notation for the set of two parameters, β1 and β2 . We

denote the set of fitted parameters by β̂, and its elements by β̂1 and β̂2 .
It turns out that the β1 and β2 values that solve the least squares problem
have well-known mathematical formulas. (More on that later.) We can use
a computer to evaluate the formulas, simply, stably and instantaneously.
The fitted values are
ŷi = β̂1 + β̂2 xi ,

i = 1, ..., N . The residuals are the difference between actual and fitted values,

ei = yi − ŷi ,
34 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

Figure 3.2: (Log Wage, Education) Scatterplot

3.2. REGRESSION AS CURVE FITTING 35

Figure 3.3: (Log Wage, Education) Scatterplot with Superimposed Regression Line

i = 1, ..., N .
In Figure 3.3, we illustrate graphically the results of regressing LWAGE on
EDUC. The best-fitting line slopes upward, reflecting the positive correlation
between LWAGE and EDUC.1 Note that the data points don’t satisfy the
fitted linear relationship exactly; rather, they satisfy it on average. To predict
LWAGE for any given value of EDUC, we use the fitted line to find the value
of LWAGE that corresponds to the given value of EDUC.
1
Note that use of log wage promostes several desiderata. First, it promotes normality, as we discussed
in Chapter 2. Second, it enforces positivity of the fitted wage, because W\ AGE = exp(LW \ AGE), and
exp(x) > 0 for any x.
36 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

Numerically, the fitted line is

LW
\ AGE = 1.273 + .081EDU C.

We conclude with a brief comment on notation. A standard cross-section

notation for indexing the cross-sectional units is i = 1, ..., N . A standard
time-series notation for indexing time periods is t = 1, ..., T . Much of our
discussion will be valid in both cross-section and time-series environments, but
we generally attempt to use the more standard notation in each environment.

3.2.2 Multiple Linear Regression

Everything generalizes to allow for more than one RHS variable. This is
called multiple linear regression.
Suppose, for example, that we have two RHS variables, x2 and x3 . Before
we fit a least-squares line to a two-dimensional data cloud; now we fit a least-
squares plane to a three-dimensional data cloud. We use the computer to
find the values of β1 , β2 , and β3 that solve the problem
N
X
min (yi − β1 − β2 x2i − β3 x3i )2 ,
β
i=1

where β denotes the set of three model parameters. We denote the set of
estimated parameters by β̂, with elements β̂1 , β̂2 , and β̂3 . The fitted values
are
ŷi = β̂1 + β̂2 x2i + β̂3 x3i ,

and the residuals are

ei = yi − ŷi ,

i = 1, ..., N .
3.2. REGRESSION AS CURVE FITTING 37

For our wage data, the fitted model is

LW
\ AGE = .867 + .093EDU C + .013EXP ER.

Extension to the general multiple linear regression model, with an arbi-

trary number of right-hand-side (RHS) variables (K, including the constant),
is immediate. The computer again does all the work. The fitted line is

ŷi = β̂1 + β̂2 x2i + β̂3 x3i + ... + β̂K xKi ,

which we sometimes write more compactly as

K
X
ŷi = β̂k xki ,
k=1

where x1i = 1 for all i.

3.2.3 Onward

Before proceeding, two aspects of what we’ve done so far are worth noting.
First, we now have two ways to analyze data and reveal its patterns. One is
the graphical scatterplot of Figure 3.2, with which we started, which provides
a visual view of the data. The other is the fitted regression line of Figure 3.3,
which summarizes the data through the lens of a linear fit. Each approach
has its merit, and the two are complements, not substitutes, but note that
linear regression generalizes more easily to high dimensions.
Second, least squares as introduced thus far has little to do with statistics
or econometrics. Rather, it is simply a way of instructing a computer to
fit a line to a scatterplot in a way that’s rigorous, replicable and arguably
reasonable. We now turn to a probabilistic interpretation.
38 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

3.3 Regression as a Probability Model

We work with the full multiple regression model (simple regression is of course
a special case). Collect the RHS variables into the vector x, where x′i =
(1, x2i , ..., xKi ).

3.3.1 A Population Model and a Sample Estimator

Thus far we have not postulated a probabilistic model that relates yi and xi ;
instead, we simply ran a mechanical regression of yi on xi to find the best
fit to yi formed as a linear function of xi . It’s easy, however, to construct
a probabilistic framework that lets us make statistical assessments about
the properties of the fitted line. We assume that yi is linearly related to
an exogenously-determined xi , and we add an independent and identically
distributed zero-mean (iid) Gaussian disturbance:

yi = β1 + β2 x2i + ... + βK xKi + εi

εi ∼ iidN (0, σ 2 ),

i = 1, ..., N . The intercept of the line is β1 , the slope parameters are the
other β’s2 , and the variance of the disturbance is σ 2 . Collectively, we call the
β’s (and σ) the model’s parameters.
We assume that the the linear model sketched is true in population; that is,
it is the data-generating process (DGP). But in practice, of course, we don’t
know the values of the model’s parameters, β1 , β2 , ..., βK and σ 2 . Our job is
to estimate them using a sample of data from the population. We estimate
the β’s precisely as before, using the computer to solve minβ N 2
P
i=1 εi .
2
We speak of the regression intercept and the regression slope.
3.3. REGRESSION AS A PROBABILITY MODEL 39

3.3.2 Notation, Assumptions and Results: The Ideal Conditions

The discussion thus far was intentionally a bit loose, focusing on motivation
and intuition. Let us now be more precise about what we assume and what
results we obtain.

A Bit of Matrix Notation

It will be useful to arrange all RHS variables into a matrix X. X has K

columns, one for each regressor. Inclusion of a constant in a regression
amounts to including a special RHS variable that is always 1. We put that in
the leftmost column of the X matrix, which is just ones. The other columns
contain the data on the other RHS variables, over the cross section in the
cross-sectional case, or over time in the time-series case. Notationally, X is
a N × K matrix.  
1 x21 x31 . . . xK1
 
1 x22 x32 . . . xK2 
X=
 ...
.

 
1 x2N x3N . . . xKN
One reason that the X matrix is useful is because the regression model
can be written very compactly using it. We have written the model as

yi = β1 + β2 x2i + ... + βK xKi + εi , i = 1, ..., N.

Alternatively, stack yi , i = 1, ..., N into the vector y, where y ′ = (y1 , y2 , ..., yN ),

and stack βj , j = 1, ..., K into the vector β, where β ′ = (β1 , β2 , ..., βK ), and
stack εi , i = 1, ..., N , into the vector ε, where ε′ = (ε1 , ε2 , ..., εN ). Then we
can write the complete model over all observations as

y = Xβ + ε. (3.1)
40 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

In addition,
εi ∼ iid N (0, σ 2 )

becomes
ε ∼ N (0, σ 2 I). (3.2)

This concise representation is very convenient.

Indeed representation (3.1)-(3.2) is crucially important, not simply be-
cause it is concise, but also because key results for estimation and inference
may be stated very simply withing it, and because the various assumptions
that we need to make to get various statistical results are most naturally and
simply stated on X and ε in equation (3.1). We now proceed to discuss such
assumptions.

Assumptions: The Ideal Conditions (IC)

1. The data-generating process (DGP) is:

yi = β1 + β2 x2i + ... + βK xKi + εi

εi ∼ iidN (0, σ 2 ),

and the fitted model matches it exactly.

2. εi is independent of (x1i , ..., xKi ), for all i

IC1 has many important sub-conditions embedded. For example:

1. The fitted model is correctly specified

2. The disturbances are Gaussian

3. The coefficients (β’s) are fixed (whether over space or time, depending
on whether we’re working in a time-series or cross-section environment)

4. The relationship is linear

3.3. REGRESSION AS A PROBABILITY MODEL 41

5. The εi ’s have constant variance σ 2

6. The εi ’s are uncorrelated (whether over space or time, depending on

whether we’re working in a time-series or cross-section environment)

IC2 also has many important sub-conditions embedded:

1. E(εi xik ) = 0, for all i, k (εi is uncorrelated with the xik ’s)

2. E(εi | xi1 , ..., xiK ) = 0, for all i (εi is conditional mean independent of
the xik ’s)

3. var(εi | xi1 , ..., xiK ) = σ 2 , for all i (εi is conditional variance independent
of the xik ’s)

IC2 is subtle, and it may seem obscure at the moment, but it is very important
in the context of causal estimation, which we will discuss in chapter 9.
The IC’s are surely heroic in many contexts, and much of econometrics
is devoted to detecting and dealing with various IC failures. But before we
worry about IC failures, it’s invaluable first to understand what happens
when they hold.3

Results Under the IC

The least squares estimator is

β̂LS = (X ′ X)−1 X ′ y,

and under the IC it is (among other things) consistent, asymptotically effi-

cient, and asymptotically normally distributed. We write

a
β̂LS ∼ N (β, V ) .
3
Certain variations of the IC as stated above can be entertained, and in addition we have ommitted some
technical details.
42 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

Figure 3.4: Regression Output

We consistently estimate the covariance matrix V using V̂ = s2 (X ′ X)−1 ,

where s2 = N 2
P
i=1 ei /(N − K).

3.4 A Wage Equation

Now let’s do more than a simple graphical analysis of the regression fit. In-
stead, let’s look in detail at the computer output, which we show in Figure 5.2
for a regression of log wages (LW AGE) on an intercept, education (EDU C)
and experience (EXP ER). We run regressions dozens of times in this book,
and the output format and interpretation are always the same, so it’s im-
portant to get comfortable with it quickly. The output is in Eviews format.
Other software will produce more-or-less the same information, which is fun-
damental and standard.
Before proceeding, note well that the IC may not be satisfied for this
dataset, yet we will proceed assuming that they are satisfied. As we proceed
through this book, we will confront violations of the various assumptions –
3.4. A WAGE EQUATION 43

indeed that’s what econometrics is largely about – and we’ll return repeatedly
to this dataset and others. But we must begin at the beginning.
The software output begins by reminding us that we’re running a least-
squares (LS) regression, and that the left-hand-side (LHS) variable is the log
wage (LWAGE), using a total of 1323 observations.
Next comes a table listing each RHS variable together with four statistics.
The RHS variables EDUC and EXPER are education and experience, and the
C variable refers to the earlier-mentioned intercept. The C variable always
equals one, so the estimated coefficient on C is the estimated intercept of the
regression line.4
The four statistics associated with each RHS variable are the estimated
coefficient (“Coefficient”), its standard error (“Std. Error”), a t statistic,
and a corresponding probability value (“Prob.”). The standard errors of
the estimated coefficients indicate their likely sampling variability, and hence
their reliability. The estimated coefficient plus or minus one standard error is
approximately a 68% confidence interval for the true but unknown population
parameter, and the estimated coefficient plus or minus two standard errors
is approximately a 95% confidence interval, assuming that the estimated
coefficient is approximately normally distributed, which will be true if the
regression disturbance is normally distributed or if the sample size is large.
Thus large coefficient standard errors translate into wide confidence intervals.
Each t statistic provides a test of the hypothesis of variable irrelevance:
that the true but unknown population parameter is zero, so that the corre-
sponding variable contributes nothing to the regression and can therefore be
dropped. One way to test variable irrelevance, with, say, a 5% probability
of incorrect rejection, is to check whether zero is outside the 95% confidence
interval for the parameter. If so, we reject irrelevance. The t statistic is
just the ratio of the estimated coefficient to its standard error, so if zero is
4
Sometimes the population coefficient on C is called the constant term, and the regression estimate is
called the estimated constant term.
44 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

outside the 95% confidence interval, then the t statistic must be bigger than
two in absolute value. Thus we can quickly test irrelevance at the 5% level
by checking whether the t statistic is greater than two in absolute value.5
Finally, associated with each t statistic is a probability value, which is the
probability of getting a value of the t statistic at least as large in absolute
value as the one actually obtained, assuming that the irrelevance hypothesis
true. Hence if a t statistic were two, the corresponding probability value
would be approximately .05. The smaller the probability value, the stronger
the evidence against irrelevance. There’s no magic cutoff, but typically prob-
ability values less than 0.1 are viewed as strong evidence against irrelevance,
and probability values below 0.05 are viewed as very strong evidence against
irrelevance. Probability values are useful because they eliminate the need for
consulting tables of the t distribution. Effectively the computer does it for us
and tells us the significance level at which the irrelevance hypothesis is just
rejected.
Now let’s interpret the actual estimated coefficients, standard errors, t
statistics, and probability values. The estimated intercept is approximately
.867, so that conditional on zero education and experience, our best forecast
of the log wage would be 86.7 cents. Moreover, the intercept is very precisely
estimated, as evidenced by the small standard error of .08 relative to the
estimated coefficient. An approximate 95% confidence interval for the true
but unknown population intercept is .867 ± 2(.08), or [.71, 1.03]. Zero is
far outside that interval, so the corresponding t statistic is huge, with a
probability value that’s zero to four decimal places.
The estimated coefficient on EDUC is .093, and the standard error is again
small in relation to the size of the estimated coefficient, so the t statistic is
large and its probability value small. The coefficient is positive, so that
5
If the sample size is small, or if we want a significance level other than 5%, we must refer to a table of
critical values of the t distribution. We also note that use of the t distribution in small samples also requires
an assumption of normally distributed disturbances.
3.4. A WAGE EQUATION 45

LWAGE tends to rise when EDUC rises. In fact, the interpretation of the
estimated coefficient of .09 is that, holding everything else constant, a one-
year increase in EDUC will produce a .093 increase in LWAGE.
The estimated coefficient on EXPER is .013. Its standard error is also
small, and hence its t statistic is large, with a very small probability value.
Hence we reject the hypothesis that EXPER contributes nothing to the fore-
casting regression. A one-year increase in EXP ER tends to produce a .013
increase in LWAGE.
A variety of diagnostic statistics follow; they help us to evaluate the ad-
equacy of the regression. We provide detailed discussions of many of them
elsewhere. Here we introduce them very briefly:

3.4.1 Mean dependent var 2.342

The sample mean of the dependent variable is

N
1 X
ȳ = yi .
N i=1

It measures the central tendency, or location, of y.

3.4.2 S.D. dependent var .561

The sample standard deviation of the dependent variable is

s
PN 2
i=1 (yi − ȳ)
SD = .
N −1

It measures the dispersion, or scale, of y.

46 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

3.4.3 Sum squared resid 319.938

Minimizing the sum of squared residuals is the objective of least squares

estimation. It’s natural, then, to record the minimized value of the sum of
squared residuals. In isolation it’s not of much value, but it serves as an
input to other diagnostics that we’ll discuss shortly. Moreover, it’s useful for
comparing models and testing hypotheses. The formula is
N
X
SSR = e2i .
i=1

3.4.4 Log likelihood -938.236

The likelihood function is the joint density function of the data, viewed as a
function of the model parameters. Hence a natural estimation strategy, called
maximum likelihood estimation, is to find (and use as estimates) the param-
eter values that maximize the likelihood function. After all, by construction,
those parameter values maximize the likelihood of obtaining the data that
were actually obtained. In the leading case of normally-distributed regres-
sion disturbances, maximizing the likelihood function (or equivalently, the
log likelihood function, because the log is a monotonic transformation) turns
out to be equivalent to minimizing the sum of squared residuals, hence the
maximum-likelihood parameter estimates are identical to the least-squares
parameter estimates. The number reported is the maximized value of the
log of the likelihood function.6 Like the sum of squared residuals, it’s not of
direct use, but it’s useful for comparing models and testing hypotheses.
Let us now dig a bit more deeply into the likelihood function, maximum-
likelihood estimation, and related hypothesis-testing procedures. A natural
estimation strategy with wonderful asymptotic properties, called maximum
likelihood estimation, is to find (and use as estimates) the parameter val-
6
Throughout this book, “log” refers to a natural (base e) logarithm.
3.4. A WAGE EQUATION 47

ues that maximize the likelihood function. After all, by construction, those
parameter values maximize the likelihood of obtaining the data that were
actually obtained.
In the leading case of normally-distributed regression disturbances, max-
imizing the likelihood function turns out to be equivalent to minimizing the
sum of squared residuals, hence the maximum-likelihood parameter estimates
are identical to the least-squares parameter estimates.
To see why maximizing the Gaussian log likelihood gives the same pa-
rameter estimate as minimizing the sum of squared residuals, let us derive
the likelihood for the Gaussian linear regression model with non-stochastic
regressors,

yi = x′i β + ε
εi ∼ iidN (0, σ 2 ).

The model implies that

yi ∼ iidN (x′i β, σ 2 ),

so that the density function of yi is

−1 −1 ′ 2
f (yi ) = (2πσ 2 ) 2 e 2σ2 (yi −xi β) .

Hence f (y1 , ..., yN ) = f (y1 )f (y2 ) · · · f (yN ) (by independence of the yi ’s). In
particular, the likelihood of the sample, denoted by L, is
N
−1 −1 ′ 2
Y
L= (2πσ 2 ) 2 e 2σ2 (yi −xi β)
i=1
48 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

so
N
 −N
 1 X
ln L = ln (2πσ ) 2 2 − 2 (yi − x′i β)2
2σ i=1
N
N N 1 X
= − ln(2π) − ln σ 2 − 2 (yi − x′i β)2 .


2 2 2σ i=1

Note in particular that the β vector that maximizes the likelihood (or log
likelihood – the optimizers must be identical because the log is a positive
monotonic transformation) is the β vector that minimizes the sum of squared
residuals.
The log likelihood is also useful for hypothesis testing via likelihood-ratio
tests. Under very general conditions we have asymptotically that:

−2(ln L0 − ln L1 ) ∼ χ2d ,

where ln L0 is the maximized log likelihood under the restrictions implied

by the null hypothesis, ln L1 is the unrestricted log likelihood, and d is the
number of restrictions imposed under the null hypothesis.
t and F tests are likelihood ratio tests under a normality assumption.
That’s why they can be written in terms of minimized SSR’s rather than
maximized ln L’s.

3.4.5 F statistic 199.626

We use the F statistic to test the hypothesis that the coefficients of all vari-
ables in the regression except the intercept are jointly zero.7 That is, we
test whether, taken jointly as a set, the variables included in the forecasting
model have any explanatory value. This contrasts with the t statistics, which
7
We don’t want to restrict the intercept to be zero, because under the hypothesis that all the other
coefficients are zero, the intercept would equal the mean of y, which in general is not zero. See Problem 6.
3.4. A WAGE EQUATION 49

we use to examine the explanatory value of the variables one at a time.8 If

no variable has explanatory value, the F statistic follows an F distribution
with k − 1 and T − k degrees of freedom. The formula is

(SSRres − SSR)/(K − 1)
F = ,
SSR/(N − K)

where SSRres is the sum of squared residuals from a restricted regression that
contains only an intercept. Thus the test proceeds by examining how much
the SSR increases when all the variables except the constant are dropped. If
it increases by a great deal, there’s evidence that at least one of the variables
has explanatory content.

3.4.6 Prob(F statistic) 0.000000

The probability value for the F statistic gives the significance level at which
we can just reject the hypothesis that the set of RHS variables has no pre-
dictive value. Here, the value is indistinguishable from zero, so we reject the
hypothesis overwhelmingly.

3.4.7 S.E. of regression .492

If we knew the elements of β and predicted yi using x′i β, then our prediction
errors would be the εi ’s, with variance σ 2 . We’d like an estimate of σ 2 ,
because it tells us whether our prediction errors are likely to be large or small.
The observed residuals, the ei ’s, are effectively estimates of the unobserved
population disturbances, the εi ’s. Thus the sample variance of the e’s, which
we denote s2 (read “s-squared”), is a natural estimator of σ 2 :
PN 2
2 i=1 ei
s = .
N −K
8
In the degenerate case of only one RHS variable, the t and F statistics contain exactly the same infor-
mation, and F = t2 . When there are two or more RHS variables, however, the hypotheses tested differ, and
F ̸= t2 .
50 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

s2 is an estimate of the dispersion of the regression disturbance and hence

is used to assess goodness of fit of the model, as well as the magnitude of
prediction errors that we’re likely to make. The larger is s2 , the worse the
model’s fit, and the larger the prediction errors we’re likely to make. s2
involves a degrees-of-freedom correction (division by N − K rather than by
N − 1, reflecting the fact that K regression coefficients have been estimated),
which is an attempt to get a good estimate of the out-of-sample prediction
error variance on the basis of the in-sample residuals.
The standard error of the regression (SER) conveys the same information;
it’s an estimator of σ rather than σ 2 , so we simply use s rather than s2 . The
formula is s
√
PN 2
i=1 ei
SER = s2 = .
N −K
The standard error of the regression is easier to interpret than s2 , because
its units are the same as those of the e’s, whereas the units of s2 are not. If
the e’s are in dollars, then the squared e’s are in dollars squared, so s2 is in
dollars squared. By taking the square root at the end of it all, SER converts
the units back to dollars.

3.4.8 R-squared .232

If an intercept is included in the regression, as is almost always the case,

R-squared must be between zero and one. In that case, R-squared, usually
written R2 , is the percent of the variance of y explained by the variables in-
cluded in the regression. R2 measures the in-sample success of the regression
equation in predicting y; hence it is widely used as a quick check of goodness
of fit, or predictibility, of y based on the variables included in the regression.
Here the R2 is about 23% – well above zero but not great. The formula is
PN 2
2 i=1 ei
R = 1 − PN .
2
i=1 (yi − ȳ)
3.4. A WAGE EQUATION 51

We can write R2 in a more roundabout way as

1
PN 2
2 N i=1 ei
R =1− 1
PN ,
2
N i=1 (yi − ȳ)

which makes clear that the numerator in the large fraction is very close to
s2 , and the denominator is very close to the sample variance of y.

3.4.9 Adjusted R-squared .231

The interpretation is the same as that of R2 , but the formula is a bit different.
Adjusted R2 incorporates adjustments for degrees of freedom used in fitting
the model, in an attempt to offset the inflated appearance of good fit if many
RHS variables are tried and the “best model” selected. Hence adjusted R2
is a more trustworthy goodness-of-fit measure than R2 . As long as there is
more than one RHS variable in the model fitted, adjusted R2 is smaller than
R2 ; here, however, the two are extremely close (23.1% vs. 23.2%). Adjusted
R2 is often denoted R̄2 ; the formula is
1
PN 2
N −K i=1 ei
R̄2 = 1 − 1
P N
,
(y − ȳ)2
N −1 i=1 i

where K is the number of RHS variables, including the constant term. Here
the numerator in the large fraction is precisely s2 , and the denominator is
precisely the sample variance of y.

3.4.10 Akaike info criterion 1.423

The Akaike information criterion, or AIC, is effectively an estimate of the

out-of-sample forecast error variance, as is s2 , but it penalizes degrees of
freedom more harshly. It is used to select among competing models. The
formula is: PN 2
i=1 ei
AIC = e( )
2K
N ,
N
52 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

and “smaller is better”. That is, we select the model with smallest AIC. We
will discuss AIC in greater depth in Chapter 16.

3.4.11 Schwarz criterion 1.435

The Schwarz information criterion, or SIC, is an alternative to the AIC with

the same interpretation, but a still harsher degrees-of-freedom penalty. The
formula is: PN 2
i=1 ei
SIC = N ( N )
K
,
N
and “smaller is better”. That is, we select the model with smallest SIC. We
will discuss SIC in greater depth in Chapter 16. Schwarz criterion is also
known as the Bayesian Information Criteria, or BIC.

3.4.12 A Bit More on AIC and SIC

The AIC and SIC are tremendously important for guiding model selection
in a ways that avoid data mining and in-sample overfitting.
You will want to start using AIC and SIC immediately, so we provide
a bit more information here. Model selection by maximizing R2 , or equiv-
alently minimizing residual SSR, is ill-advised, because they don’t penalize
for degrees of freedom and therefore tend to prefer models that are “too big.”
Model selection by maximizing R̄2 , or equivalently minimizing residual s2 ,
is still ill-advised, even though R̄2 and s2 penalize somewhat for degrees of
freedom, because they don’t penalize harshly enough and therefore still tend
to prefer models that are too big. In contrast, AIC and SIC get things just
right. SIC has a wonderful asymptotic optimality property when the set of
candidate models is viewed as fixed: Basically SIC “gets it right” asymptoti-
cally, selecting either the DGP (if the DGP is among the models considered)
or the best predictive approximation to the DGP (if the DGP is not among
the models considered). AIC has a different and also-wonderful asymptotic
3.4. A WAGE EQUATION 53

optimality property, known as “efficiency,” when the set of candidate models

is viewed as expanding as the sample size grows. In practice, the models
selected by AIC and SIC rarely disagree.

3.4.13 Hannan-Quinn criter. 1.427

Hannan-Quinn is yet another information criterion for use in model selection.

We will not use it in this book.

3.4.14 Durbin-Watson stat. 1.926

The Durbin-Watsion (DW) statistic is used in time-series contexts, and we

will study it later.

3.4.15 The Residual Scatter

The residual scatter is often useful in both cross-section and time-series sit-
uations. It is a plot of y vs ŷ. A perfect fit (R2 = 1) corresponds to all
points on the 45 degree line, and no fit (R2 = 0) corresponds to all points on
a vertical line corresponding to y = ȳ.
In Figure 3.5 we show the residual scatter for the wage regression. It is
not a vertical line, but certainly also not the 45 degree line, corresponding to
the positive but relatively low R2 of .23.

3.4.16 The Residual Plot

In time-series settings, it’s always a good idea to assess visually the adequacy
of the model via time series plots of the actual data (yi ’s), the fitted values
(ŷi ’s), and the residuals (ei ’s). Often we’ll refer to such plots, shown together
in a single graph, as a residual plot.9 We’ll make use of residual plots through-
out this book. Note that even with many RHS variables in the regression
9
Sometimes, however, we’ll use “residual plot” to refer to a plot of the residuals alone. The intended
meaning should be clear from context.
54 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

Figure 3.5: Wage Regression Residual Scatter

model, both the actual and fitted values of y, and hence the residuals, are
simple univariate series that can be plotted easily.
The reason we examine the residual plot is that patterns would indicate
violation of our iid assumption. In time series situations, we are particularly
interested in inspecting the residual plot for evidence of serial correlation
in the ei ’s, which would indicate failure of the assumption of iid regression
disturbances. More generally, residual plots can also help assess the overall
performance of a model by flagging anomalous residuals, due for example to
outliers, neglected variables, or structural breaks.
Our wage regression is cross-sectional, so there is no natural ordering of
the observations, and the residual plot is of limited value. But we can still
use it, for example, to check for outliers.
In Figure 3.6, we show the residual plot for the regression of LWAGE on
EDUC and EXPER. The actual and fitted values appear at the top of the
3.5. LEAST SQUARES AND OPTIMAL POINT PREDICTION 55

Figure 3.6: Wage Regression Residual Plot

graph; their scale is on the right. The fitted values track the actual values
fairly well. The residuals appear at the bottom of the graph; their scale is
on the left. It’s important to note that the scales differ; the ei ’s are in fact
substantially smaller and less variable than either the yi ’s or the ŷi ’s. We
draw the zero line through the residuals for visual comparison. No outliers
are apparent.

3.5 Least Squares and Optimal Point Prediction

The linear regression DGP under the ideal conditions implies the conditional
mean function,

E(yi | x1i = 1, x2i = x∗2i , ..., xKi = x∗Ki ) = β1 + β2 x∗2i + ... + βK x∗Ki

or E(yi | xi = x∗i ) = x∗i ′ β .

56 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

And as also already noted much earlier in Chapter 1, a major goal in

econometrics is predicting y. The question is “If a new person arrives with
characteristics x∗ , what is my minimum mean squared error (MSE) prediction
of her y? It turns out, very intuitively, that the answer is E(y|x = x∗ ) = x∗i ′ β.
That is, “the conditional mean is the minimum MSE (point) predictor”. (In-
deed if it were anything else you’d surely be suspicious.) The non-operational
version (i.e., pretending that we know β) is E(yi |xi = x∗i ) = x∗i ′ β, and the op-
|xi = x∗ ) = x∗ ′ β̂LS .
erational version (using β̂LS ) is E(yi\ i i
Let’s now introduce a very basic and powerful result. Notice that the β’s in
the conditional mean expression give the weights on the various x’s for form-
ing the optimal predictor. Hence under the IC, consistency of OLS ensures
that asymptotically the operational point prediction (based on β̂LS ) will use
the right weights (based on β). That is, under the IC, LS is consistent for the
right predictive weights. Now here’s the really amazing thing (although it’s
obvious when you think about it): Under great generality, in particular even
if the IC fail, LS is still consistent for the right predictive weights, simply by
virtue of the MSE-optimization problem that it solves directly. The bottom
line: Forecasting is of central importance in economics, and LS regression
delivers optimal forecasts under great generality.
If LS provides optimal forecasts even without the IC, you might wonder
why we introduced the IC. There are two key sets of reasons. First, even
for standard forecasting situations of the form “If a new person arrives with
characteristics x∗ , what is my minimum-MSE prediction of her y,” once we
drop the IC, so that the fitted model does not necessarily match the true
DGP, there is a crucial issue of what model to use. Many questions arise.
Which x’s should we include, and which should we exclude? Is a linear
model really adequate, or should we incorporate some non-linearlity? And so
on. For any given model, LS will deliver the optimal parameter configuration
for forecasting, but again, a crucial issue is what features a “good” or “the
3.5. LEAST SQUARES AND OPTIMAL POINT PREDICTION 57

best” model should incorporate.

Second, what we’ve considered so far is called “non-causal” prediction.
It exploits correlation between y and x to generate forecasts, but there is
no presumption (or need) that x truly causes y in a deep scientific sense.
(Remember, correlation does not imply causation!) But there is a causal
form of prediction that differs from the one sketched thus far. In particular,
thus far we’ve considered “If a new person arrives with characteristics x∗ ,
what is my minimum-MSE prediction of her y?”, but we might alternatively
be interested in predicting the effects of an active treatment, or intervention,
or policy, along the lines of “If I randomly select someone and change her
characteristics in some way, what is my minimum-MSE prediction of the
corresponding change in her y?” It turns out that LS does not always perform
well for such “causal prediction” questions. So when does LS perform well for
causal prediction? Under the IC! Effectively LS solves both the non-causal
and causal prediction problems under the IC (or, put differently, the two
problems are identical under the IC), but when the IC fail LS continues to
solve the non-causal prediction problem but fails for the causal prediction
problem.
Summarizing, here’s what true:

1. Non-causal prediction is important in economics

2. LS succeeds for non-causal prediction under great generality

3. Causal prediction is important in economics

4. LS fails for causal prediction unless the IC hold, so credible causal pre-
diction is much harder.

Given the combination of 1 and 2 above, it makes obvious sense to start

with with non-causal prediction and treat it extensively, reserving 4 for sep-
arate treatment (which we do in Chapter 9). That has been the successful
58 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

strategy of econometrics for many decades, and it is very much at the center
of modern “data science” and “machine learning”.

3.6 Optimal Interval and Density Prediction

Prediction as introduced thus far is so-called point prediction (a single best

– i.e., minimum MSE – guess).
Forecasts stated as confidence intervals (“interval forecasts”) are also of
interest. The linear regression DGP under the IC implies the conditional
variance function
var(yi | xi = x∗i ) = σ 2 ,

which we can use to form interval forecasts. The non-operational version is

yi ∈ [x∗i ′ β ± 1.96 σ] w.p. 0.95,

and the operational version is

yi ∈ [x∗i ′ β̂LS ± 1.96 s] w.p. 0.95.

Finally full density forecasts are of interest. The linear regression DGP
under the IC implies the conditional density function

yi | xi = x∗i ∼ N (x∗i ′ β, σ 2 ).

Hence a non-operational density forecast is

yi | xi = x∗i ∼ N (x∗i ′ β, σ 2 ),

with operational version

yi | xi = x∗i ∼ N (x∗i ′ β̂LS , s2 ).

3.7. REGRESSION OUTPUT FROM A PREDICTIVE PERSPECTIVE 59

Notice that the interval and density forecasts rely for validity on more parts
of the IC than do the point forecasts: Gaussian disturbances and constant
disturbance variances – which makes clear in even more depth why violations
of the IC are generally problematic even in non-causal forecasting situations.

3.7 Regression Output from a Predictive Perspective

In light of our predictive emphasis throughout this book, here we offer some
predictive perspective on the regression statistics discussed earlier.
The sample, or historical, mean of the dependent variable, ȳ, an estimate
of the unconditional mean of y, is a benchmark forecast. It is obtained by
regressing y on an intercept alone – no conditioning on other regressors.
The sample standard deviation of y is a measure of the in-sample accuracy
of the unconditional mean forecast ȳ.
The OLS fitted values, ŷi = x′i β̂, are effectively in-sample regression pre-
dictions.
The OLS residuals, ei = yi − ŷi , are effectively in-sample prediction errors
corresponding to use of those in-sample regression predictions.
OLS coefficient signs and sizes relate to the weights put on the various x
variables in forming the best in-sample prediction of y.
The standard errors, t statistics, and p-values let us do statistical inference
as to which regressors are most relevant for predicting y.
SSR measures “total” in-sample accuracy of the regression predictions. It
is closely related to in-sample M SE:
N
1 1 X 2
M SE = SSR = e
N N i=1 i

(“average” in-sample accuracy of the regression predictions)

The F statistic effectively compares the accuracy of the regression-based
60 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

forecast to that of the unconditional-mean forecast. It helps us assess whether

the x variables, taken as a set, have predictive value for y. That contrasts
with the t statistics, which assess predictive value of the x variables one at a
time.
s2 is just SSR scaled by N − K, so again, it’s a measure of the in-sample
accuracy of the regression-based forecast. It’s like MSE, but corrected for
degrees of freedom.
R2 and R̄2 effectively compare the in-sample accuracy of conditional-mean
(x′i β̂) and unconditional-mean (ȳ) forecasts. R2 is not corrected for d.f. and
has M SE on top: PN
1 ′ 2
2 N i=1 (yi − xi β̂)
R =1− 1
PN .
(y − ȳ)2
N i=1 i

In contrast, R̄2 is corrected for d.f. and has s2 on top:

1
PN ′ 2
2 N −K i=1 (yi − xi β̂)
R̄ = 1 − 1
PN .
(y − ȳ)2
N −1 i=1 i

Residual plots are useful for visually flagging neglected things that im-
pact forecasting. Residual correlation (in time-series contexts) indicates that
point forecasts could possibly be improved. Non-constant residual volatility
indicates that interval and density forecasts could be possibly improved.

3.8 Multicollinearity

Collinearity and multicollinearity don’t really involve failure of the ideal

conditions, but they nevertheless are someimes issues and should be men-
tioned.
Collinearity refers to two x variables that are highly correlated. But even if
all pairwise correlations are small an x variable could nevertheless be highly
correlated with a linear combination of other x variables. That raises the
3.8. MULTICOLLINEARITY 61

idea of multicollinearity, where an x variable is highly correlated with a linear

combination of other x variables. Collinearity is of course a special case of
multicollinearity, so henceforth we will simply speak of multicollinearity.

3.8.1 Perfect and Imperfect Multicollinearity

There are two types of multicollinearity, perfect and imperfect.

Perfect multicollinearity refers to perfect correlation among some re-
gressors, or linear combinations of regressors. Perfect multicollinearity is
indeed a problem; the X ′ X matrix is singular, so (X ′ X)−1 does not exist,
and the OLS estimator cannot even be computed!10 Perfect multicollinearity
is disastrous, but it’s unlikely to occur unless you do something really silly,
like entering the same regressor twice.11 In any event the solution is trivial:
simply drop one of the redundant variables.
Imperfect multicollinearity, in contrast, occurs routinely but is not
necessarily problematic, although in extreme cases it may require some at-
tention. Imperfect collinearity/multicollinearity refers to (imperfect) correla-
tion among some regressors, or linear combinations of regressors. Imperfect
multicollinearity is not a “problem” in the sense that something was done
incorrectly, and it is not a violation of the IC. Rather, it just reflects the
nature of economic and financial data. But we still need to be aware of it
and understand its effects. Telltale symptoms are large F and R2 , yet small
t’s (large s.e.’s), and/or coefficients that are sensitive to small changes in
sample period. That is, OLS has trouble parsing individual influences, yet
it’s clear that there is an overall relationship. OLS is in some sense just what
the doctor ordered – orthogonal projection.
10
For this reason people sometimes view non-singular X ′ X as part of the IC.
11
A classic and more sophisticated example involves the “dummy variable trap,” in which we include as
regressors a full set of dummy variables and an intercept. We will define dummy variables and note the
dummy variable trap in Chapters 5 and 10.
62 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

It can be shown, and it is very intuitive, that

 

var(β̂k ) = f  σ 2 , σx2k , Rk2 

|{z} |{z} |{z}
+ − +

where Rk2 is the R2 from a regression of xk on all other regressors. In the

limit, as Rk2 → 1, var(β̂k ) → ∞, because xk is then perfectly “explained”
by the other variables and is therefore completely redundant. Rk2 is effec-
tively a measure of the “strength” of the multicollinearity affecting βk . We
often measure the strength of multicollinearity by the “variance inflation
factor”,
1
V IF (β̂k ) = ,
1 − Rk2
which is just a transformation of Rk2 .
3.9. BEYOND FITTING THE CONDITIONAL MEAN: QUANTILE REGRESSION 63

3.9 Beyond Fitting the Conditional Mean:

Quantile regression

Recall that the OLS estimator, β̂OLS , solves:

N
X N
X
2
minβ (yi − β1 − β2 x2t − ... − βK xKt ) = minβ ε2i
i=1 i=1

As you know, the solution has a simple analytic closed-form expression,

β̂OLS = (X ′ X)−1 X ′ y), with wonderful properties under the IC (unbiased, con-
sistent, Gaussian, MVUE). But other objectives are possible and sometimes
useful. So-called quantile regression (QR) involves an objective function lin-
ear on each side of 0 but with (generally) unequal slopes. QR estimator β̂QR
minimizes “linlin loss,” or “check function loss”:
N
X
minβ linlin(εi ),
i=1

where:

a|e| if e ≤ 0
linlin(e) =
b|e| if e > 0

= a|e| I(e ≤ 0) + b|e| I(e > 0).

I(·) stands for “indicator” variable where I(x) = 1 if x is true, and I(x) = 0
otherwise. “linlin” refers to linearity on each side of the origin.
QR is not as simple as OLS, but it is still simple (solves a linear program-
ming problem).
A key issue is what, precisely, quantile regression fits. QR fits the d · 100%
quantile:
quantiled (y|X) = xβ
64 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

where
b 1
d= =
a + b 1 + a/b
This is an important generalization of regression (e.g., How do the wages
of people in the far left tail of the wage distribution vary with education and
experience, and how does that compare to those in the center of the wage
distribution?)

3.10 Exercises, Problems and Complements

1. (Regression with and without a constant term)

Consider Figure 3.3, in which we showed a scatterplot of y vs. x with a
fitted regression line superimposed.

a. In fitting that regression line, we included a constant term. How can

you tell?
b. Suppose that we had not included a constant term. How would the
figure look?
c. We almost always include a constant term when estimating regres-
sions. Why?
d. When, if ever, might you explicitly want to exclude the constant term?

2. (Interpreting coefficients and variables)

Let yi = β1 + β2 xi + β3 zi + εi , where yi is the number of hot dogs sold
at an amusement park on a given day, xi is the number of admission
tickets sold that day, zi is the daily maximum temperature, and εi is a
random error. Assume the IC.

a. State whether each of yi , xi , zi , β1 , β2 and β3 is a coefficient or a

variable.
3.10. EXERCISES, PROBLEMS AND COMPLEMENTS 65

b. Determine the units of β1 , β2 and β3 , and describe the physical mean-

ing of each.
c. What do the signs of the a coefficients tell you about how the var-
ious variables affects the number of hot dogs sold? What are your
expectations for the signs of the various coefficients (negative, zero,
positive or unsure)?
d. Is it sensible to entertain the possibility of a non-zero intercept (i.e.,
β1 ̸= 0)? β2 > 0? β3 < 0?

3. (Scatter plots and regression lines)

Draw qualitative scatter plots and regression lines for each of the follow-
ing two-variable datasets, and state the R2 in each case:

a. Data set 1: y and x have correlation 1

b. Data set 2: y and x have correlation -1
c. Data set 3: y and x have correlation 0.

4. (Desired values of regression diagnostic statistics)

For each of the diagnostic statistics listed below, indicate whether, other
things the same, “bigger is better,” “smaller is better,” or neither. Ex-
plain your reasoning. (Hint: Be careful, think before you answer, and
be sure to qualify your answers as appropriate.)

a. Coefficient
b. Standard error
c. t statistic
d. Probability value of the t statistic
e. R-squared
f. Adjusted R-squared
66 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

g. Standard error of the regression

h. Sum of squared residuals
i. Log likelihood
j. Mean of the dependent variable
k. Standard deviation of the dependent variable
l. Akaike information criterion
m. Schwarz information criterion
n. F statistic
o. Probability-value of the F statistic

5. (Regression semantics)
Regression analysis is so important, and used so often by so many people,
that a variety of associated terms have evolved over the years, all of which
are the same for our purposes. You may encounter them in your reading,
so it’s important to be aware of them. Some examples:

a. Ordinary least squares, least squares, OLS, LS.

b. y, LHS variable, regressand, dependent variable, endogenous variable
c. x’s, RHS variables, regressors, independent variables, exogenous vari-
ables, predictors, covariates
d. probability value, prob-value, p-value, marginal significance level
e. Schwarz criterion, Schwarz information criterion, SIC, Bayes infor-
mation criterion, BIC

6. (Regression when X Contains Only an Intercept)

Consider the regression model (3.1)-(3.2), but where X contains only an
intercept.
3.10. EXERCISES, PROBLEMS AND COMPLEMENTS 67

a. What is the OLS estimator of the intercept?

b. What is the distribution of the OLS estimator under the ideal condi-
tions?
c. Does the variance-covariance matrix of the OLS estimator under the
ideal conditions depend on any unknown parameters, and if so, how
would you estimate them?

7. (Dimensionality)
We have emphasized, particularly in Chapter 2, that graphics is a pow-
erful tool with a variety of uses in the construction and evaluation of
econometric models. We hasten to add, however, that graphics has its
limitations. In particular, graphics loses much of its power as the dimen-
sion of the data grows. If we have data in ten dimensions, and we try to
squash it into two or three dimensions to make graphs, there’s bound to
be some information loss.
Thus, in contrast to the analysis of data in two or three dimensions,
in which case learning about data by fitting models involves a loss of
information whereas graphical analysis does not, graphical methods lose
their comparative advantage in higher dimensions. In higher dimen-
sions, graphical analysis can become comparatively laborious and less
insightful.

8. (Wage regressions)
The relationship among wages and their determinants is one of the most
important in all of economics. In the text we have examined, and will
continue to examine, the relationship for 1995 using a CPS subsample.
Here you will thoroughly analyze the relationship for 2004 and 2012,
compare your results to those for 1995, and think hard about the mean-
ing and legitimacy of your results.
68 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

(a) Obtain the relevant 1995, 2004 and 2012 CPS subsamples.
(b) Discuss any differences in the datasets. Are the same people in each
dataset?
(c) For now, assume the validity of the ideal conditions. Using each
dataset, run the OLS regression W AGE → c, EDU C, EXP ER.
(Note that the LHS variable is W AGE, not LW AGE.) Discuss and
compare the results in detail.
(d) Now think of as many reasons as possible to be skeptical of your
results. (This largely means think of as many reasons as possible
why the IC might fail.) Which of the IC might fail? One? A few?
All? Why? Insofar as possible, discuss the IC, one-by-one, how/why
failure could happen here, the implications of failure, how you might
detect failure, what you might do if failure is detected, etc.
(e) Repeat all of the above using LW AGE as the LHS variable.

9. (Parallels between the sampling distribution of the sample mean un-

der simple random sampling, and the sampling distribution of the OLS
estimator under the IC)
Consider first the sample mean under Gaussian simple random sampling.

(a) What is a Gaussian simple random sample?

(b) What is the sample mean, and what finite-sample properties does
it have under Gaussian simple random sampling?
(c) Display and discuss the exact distribution of the sample mean.
(d) How would you estimate and plot the exact distribution of the sam-
ple mean?

Now consider the OLS regression estimator under the IC.

(a) What are the IC?

3.10. EXERCISES, PROBLEMS AND COMPLEMENTS 69

(b) What is the OLS estimator, and what finite-sample properties does
it enjoy?
(c) Display and discuss the exact distribution of the OLS estimator.

Under what conditions, if any, do your “sample mean answers” and

“OLS answers” precisely coincide?

10. (OLS regression residuals sum to zero)

Assertion: As long as an intercept is included in a linear regression, the
OLS residuals must sum to precisely zero. The intuition is simply that
non-zero residual mean (residual “constant term”) would automatically
be pulled into the residual constant term, hence guaranteeing a zero
residual mean.

(a) Prove the assertion precisely.

(b) Evaluate the claim that the assertion implies the regression fits per-
fectly “on average,” despite the fact that it fits imperfectly point-
by-point.

11. (Simulation algorithm for density prediction)

(a) Take R draws from N (0, σ̂ 2 ).

(b) Add x∗ ′ β̂ to each disturbance draw.
(c) Form a density forecast by fitting a density to the output from step
11b.
(d) Form an interval forecast (95%, say) by sorting the output from step
11b to get the empirical cdf, and taking the left and right interval
endpoints as the the .025% and .975% values, respectively.

As R → ∞, the algorithmic and analytic results coincide.

Note: This simulation algorithm may seem roundabout, but later we
will drop normality.
70 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS

12. (Quantile regression empirics)

For the 1995 CPS subsample (see EPC 8) re-do the regression LW AGE →
c, EDU C, EXP ER using 20%, 50% and 80% quantile regression instead
of OLS regression.

13. (More on IC2)

IC2 as stated in this chapter (full independence of εi and (x1i , ..., xKi ),
for all i) is a very strong assumption, and it implies IC2.1, IC2.2, and
IC2.3. It is almost always a stronger assumption than we will need for
reliable causal estimation. For example, if IC2.2 and IC2.3 hold, then
lots of good things will happen, even if εi and (x1i , ..., xKi ) are not fully
independent. We discuss causal estimation in chapter 9.

3.11 Notes

Dozens of software packages implement linear regression analysis. Most auto-

matically include an intercept in linear regressions unless explicitly instructed
otherwise. That is, they automatically create and include a C variable.
The R command for ordinary least squares regression is lm(...) that
stands for “linear model”. It’s already pre-loaded into R as the default pack-
age for estimating linear models. It uses standard R format for such models,
where you specify formula, data, and various estimation options. It returns
a model estimated by OLS including coefficients, residuals, and fitted val-
ues. You can also easily calculate summary statistics using the summary()
function.
The standard R quantile regression package is quantreg, written by Roger
Koenker, the inventor of quantile regression. The command rq(...) func-
tions similarly to lm. It takes as input a formula, data, the quantile to be
estimated, and various estimation options.
3.12. REGRESSION’S INVENTOR: CARL FRIEDRICH GAUSS 71

3.12 Regression’s Inventor: Carl Friedrich Gauss

Figure 3.7: Carl Friedrich Gauss

This is a photographic reproduction of a public-domain artwork, an oil paint-

ing of German mathematician and philosopher Carl Friedrich Gauss by G.
Biermann (1824-1908). Date: 1887 (painting). Source Gauß-Gesellschaft
Göttingen e.V. (Foto: A. Wittmann).
72 CHAPTER 3. REGRESSION UNDER IDEAL CONDITIONS
Chapter 4

Non-Normality

Here we consider another violation of the IC, non-normal disturbances.

Non-normality and outliers, which we introduce in this chapter, are
closely related, because deviations from Gaussian behavior are often charac-
terized by fatter tails than the Gaussian, which produce outliers. It is impor-
tant to note that outliers are not necessarily “bad,” or requiring “treatment.”
Every data set must have some most extreme observation, by definition! Sta-
tistical estimation efficiency, moreover, increases with data variability. The
most extreme observations can be the most informative about the phenom-
ena of interest. “Bad” outliers, in contrast, are those associated with things
like data recording errors (e.g., you enter .753 when you mean to enter 75.3)
or one-off events (e.g., a strike or natural disaster).

4.0.1 Results

To understand the properties of OLS without normality, it is helpful first to

consider the properties of the sample mean without normality.
As reviewed in Appendix A, for a non-Gaussian simple random sample,

yi ∼ iid(µ, σ 2 ), i = 1, ..., N,

we have that the sample mean is consistent, asymptotically normal, and

73
74 CHAPTER 4. NON-NORMALITY

asymptotically efficient, with

σ2
 
a
ȳ ∼ N µ, .
N

This result forms the basis for asymptotic inference. It is a Gaussian central
limit theorem. We consistently estimate σ 2 using s2 .
Now consider the linear regression under the IC except that we allow non-
Gaussian disturbances. OLS remains consistent, asymptotically normal, and
asymptotically efficient, with

a
β̂OLS ∼ N (β, V ) .

We consistently estimate the covariance matrix V using s2 (X ′ X)−1 .

4.1 Assessing Normality

There are many methods, ranging from graphics to formal tests.

4.1.1 QQ Plots

We introduced histograms earlier in Chapter 2 as a graphical device for learn-

ing about distributional shape. If, however, interest centers on the tails of
distributions, QQ plots often provide sharper insight as to the agreement or
divergence between the actual and reference distributions.
The QQ plot is simply a plot of the quantiles of the standardized data
against the quantiles of a standardized reference distribution (e.g., normal). If
the distributions match, the QQ plot is the 45 degree line. To the extent that
the QQ plot does not match the 45 degree line, the nature of the divergence
can be very informative, as for example in indicating fat tails.
4.2. OUTLIERS 75

4.1.2 Residual Sample Skewness and Kurtosis

Recall skewness and kurtosis, which we reproduce here for convenience:

E(y − µ)3
S=
σ3
E(y − µ)4
K= .
σ4
Obviously, each tells about a different aspect of non-normality. Kurtosis, in
particular, tells about fatness of distributional tails relative to the normal.
A simple strategy is to check various implications of residual normality,
such as S = 0 and K = 3, via informal examination of Ŝ and K̂.

4.1.3 The Jarque-Bera Test

The Jarque-Bera test (JB) effectively aggregates the information in the

data about both skewness and kurtosis to produce an overall test of the joint
hypothesis that S = 0 and K = 3 , based upon Ŝ and K̂. The test statistic
is  
N 1
JB = Ŝ 2 + (K̂ − 3)2 .
6 4
Under the null hypothesis of independent normally-distributed observations
(S = 0, K = 3), JB is distributed in large samples as a χ2 random variable
with two degrees of freedom.1

4.2 Outliers

Outliers refer to big disturbances (in population) or residuals (in sample).

Outliers may emerge for a variety of reasons, and they may require special
1
We have discussed the case of an observed time series. If the series being tested for normality is the
residual from a model, then N can be replaced with N − K, where K is the number of parameters estimated,
although the distinction is inconsequential asymptotically.
76 CHAPTER 4. NON-NORMALITY

attention because they can have substantial influence on the fitted regression
line.
On the one hand, OLS retains its magic in such outlier situations – it
is Best Linear Unbiased Estimator (BLUE) regardless of the disturbance
distribution. On the other hand, the fully-optimal (MVUE) estimator may
be highly non-linear, so the fact that OLS remains BLUE is less than fully
comforting. Indeed OLS parameter estimates are particularly susceptible to
distortions from outliers, because the quadratic least-squares objective really
hates big errors (due to the squaring) and so goes out of its way to tilt the
fitted surface in a way that minimizes them.
How to identify and treat outliers is a time-honored problem in data anal-
ysis, and there’s no easy answer. If an outlier is simply a data-recording
mistake, then it may well be best to discard it if you can’t obtain the correct
data. On the other hand, every dataset, even a perfectly “clean” dataset,
has a “most extreme observation,” but it doesn’t follow that it should be dis-
carded. Indeed the most extreme observations are often the most informative
– precise estimation requires data variation.

4.2.1 Outlier Detection

Graphics

One obvious way to identify outliers in bivariate regression situations is via

graphics: one xy scatterplot can be worth a thousand words. In higher di-
mensions, the residual ŷy scatterplot remains invaluable, as does the residual
plot of y − ŷ.

Leave-One-Out and Leverage

Another way to identify outliers is a “leave-one-out” coefficient plot, where

we use the computer to sweep through the sample, leaving out successive
4.3. ROBUST ESTIMATION 77

observations, and examining differences in parameter estimates with obser-

vation various observations “in” vs. “out”. That is, in an obvious notation,
(−i)
we examine and plot β̂OLS − β̂OLS , i = 1, ..., N .
It can be shown, however, that the change in β̂OLS is

(−i) 1
β̂OLS − β̂OLS = − (X ′ X)−1 x′i ei ,
1 − hi

where hi is the i-th diagonal element of the “hat matrix,” X(X ′ X)−1 X ′ .
(−i) 1
Hence the estimated coefficient change β̂OLS − β̂OLS is driven by 1−hi . hi is
called the observation-i leverage. hi can be shown to be in [0, 1], so that
(−i)
the larger is hi , the larger is β̂OLS − β̂OLS . Hence one really just needs to
examine the leverage sequence, and scrutinize carefully observations with
high leverage.

4.3 Robust Estimation

Robust estimation provides a useful middle ground between completely dis-

carding allegedly-outlying observations (“dummying them out”) and doing
nothing. Here we introduce outlier-robust approaches to regression. The
first involves OLS regression, but on weighted data, an the second involves
switching from OLS to a different estimator.

4.3.1 Robustness Iteration

Fit at robustness iteration 0:

ŷ (0) = X β̂ (0)

where " #
N
X
β̂ (0) = arg min (yi − x′i β)2 .
β
i=1
78 CHAPTER 4. NON-NORMALITY

Robustness weight at iteration 1:

!
(0)
(1) ei
ρi = S (0)
6 med|ei |

where
(0) (0)
ei = yi − ŷi ,

and S(z) is a function such that S(z) = 1 for z ∈ [−1, 1] but downweights
outside that interval.
Fit at robustness iteration 1:

ŷ (1) = X β̂ (1)

where " N #
(1) 2
X
β̂ (1) = arg min ρi (yi − xi ′ β) .
β
i=1

Continue as desired.

4.3.2 Least Absolute Deviations

Recall that the OLS estimator solves

N
X
min (yi − x′i β)2 .
β
i=1

Now we simply change the objective to

N
X
min |yi − x′i β|.
β
i=1

or
N
X
min |εi |
β
i=1

That is, we change from squared-error loss to absolute-error loss. We call

4.4. WAGE DETERMINATION 79

the new estimator “least absolute deviations” (LAD) and we write β̂LAD .2
By construction, β̂LAD is not influenced by outliers as much as β̂OLS . Put
differently, LAD is more robust to outliers than is OLS.
Of course nothing is free, and the price of LAD is a bit of extra compu-
tational complexity relative to OLS. In particular, the LAD estimator does
not have a tidy closed-form analytical expression like OLS, so we can’t just
plug into a simple formula to obtain it. Instead we need to use the computer
to find the optimal β directly. If that sounds complicated, rest assured that
it’s largely trivial using modern numerical methods, as embedded in modern
software.3
It is important to note that whereas OLS fits the conditional mean func-
tion:
mean(y|X) = Xβ,

LAD fits the conditional median function (50% quantile):

median(y|X) = Xβ

The conditional mean and median are equal under symmetry and hence under
normality, but not under asymmetry, in which case the median is a better
measure of central tendency. Hence LAD delivers two kinds of robustness to
non-normality: it is robust to outliers and robust to asymmetry.

4.4 Wage Determination

Here we show some empirical results that make use of the ideas sketched
above. There are many tables and figures appearing at the end of the chapter.
We do not refer to them explicitly, but all will be clear upon examination.
2
Note that LAD regression is just quantile regression for d = .50.
3
Indeed computation of the LAD estimator turns out to be a linear programming problem, which is
well-studied and simple.
80 CHAPTER 4. NON-NORMALITY

4.4.1 W AGE

We run W AGE → c, EDU C, EXP ER. We show the regression results,

the residual plot, the residual histogram and statistics, the residual Gaussian
QQ plot, the leave-one-out plot, and the results of LAD estimation. The
residual plot shows lots of positive outliers, and the residual histogram and
Gaussian QQ plot indicate right-skewed residuals.

4.4.2 LW AGE

Now we run LW AGE → c, EDU C, EXP ER. Again we show the re-
gression results, the residual plot, the residual histogram and statistics, the
residual Gaussian QQ plot, the leave-one-out plot, and the results of LAD
estimation. Among other things, and in sharp contrast to the results for
WAGE and opposed to LWAGE, the residual histogram and Gaussian QQ
plot indicate approximate residual normality.
4.4. WAGE DETERMINATION 81

Figure 4.1: OLS Wage Regression

82 CHAPTER 4. NON-NORMALITY

Figure 4.2: OLS Wage Regression: Residual Plot

4.4. WAGE DETERMINATION 83

Figure 4.3: OLS Wage Regression: Residual Histogram and Statistics

84 CHAPTER 4. NON-NORMALITY

Figure 4.4: OLS Wage Regression: Residual Gaussian QQ Plot

4.4. WAGE DETERMINATION 85

Leave−One−Out Plot

1.23
●
Coefficient (Education)
● ●
● ●
●
● ● ● ● ● ●
● ● ●
● ● ●●
● ● ● ● ● ●
●
● ● ●●
● ● ● ● ● ●● ● ● ● ● ● ● ●
●● ● ● ● ● ●
●
● ●●● ● ●● ● ●● ●● ● ● ●● ●●● ● ● ● ● ●● ● ●
●● ●●● ● ●● ●
●
●
●●●●● ●● ●● ● ●●
● ● ●● ● ●● ● ●
●● ● ● ●
● ●● ●
●
● ●● ● ● ●●●●● ● ● ● ●● ●●●● ●●● ● ●●●●●● ● ●●
●●●●● ● ●●
●●●● ●●● ●● ●● ● ●●● ● ● ●● ● ● ● ●●
● ● ● ● ● ● ●● ● ● ●●●
● ●
● ●
● ●●● ●● ● ●
●
●● ●
●●●●●
●●● ●●●
● ●●●●●● ● ●●● ●●●● ● ●● ●●● ● ●●● ●
●●●●●● ● ●
● ●●
●● ●●●●●
●●● ●● ●●
● ●● ●● ● ●
● ● ●
●
●● ●● ● ●
● ●●
●● ●●●●
● ●
● ●●
●●●● ●
● ●
●●● ●●
●●●●●●● ● ●●
●●
● ●
●
●●●
●
●●
●
●●
●
●●●
●
●●
●
● ●
●●●●
●
●●
●
●
●●
●●
●●
●●
●●
●
● ●
●
●
●●
●
●
●●
●●●●
●
●●●
●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●●●
●
● ●
●●
●
●
●●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●●
●
●●
●
●
●●
●
●
●●●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●●
●
●●
●
●
●●●●
●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●
●●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●●●●
●
●
●●
●
●●
●
●●●
●●●
●
●
●●
●
●●●
●●
●
●●●
●
●●
●
●●
●
●●
●●
●●
●●
●
●
●●
●
●●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●●
●
●
●●●
●
●●
●
●
●
●
●●
●●●●
●
●●
●
●●
●●
●●
●
●
●●
●
●●
●●
● ●
●
●●●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●●
●
●
●
●
●
●●●
●●
●
●●
●
●
●
●●
●●
●
●●
●
●
●
●●
●
●●
●
●●
●
●●
●
●●
● ●
●
●
●●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●●
● ●
●
●●
●
●
●
●●
●
●
●●
●●
●
●
●●
●
●●
●
●
●
●
●
●
● ●● ●●●
● ●●
●●●●
●●●●●
● ● ●
●●
●
●●
●●
●●
●● ●●● ●●
●●
●●
●
●●●●
●●●
●●
●●●
●●●
●●●●● ●
● ●
●●●
●●
●●
●●●
●
●●●●● ●
●●●●
●● ●●
●●●
●●●
●●
●●● ●● ●
● ●●● ● ●
●
●● ●
●●●●●●
●●
●●●●●
● ●
●●●
●●
●●
●
●●
●
●●
●
● ●
●●●
●●●
●●●●●
●
●●
●●●
●
●
●
●●
●●
●
●
●●●●●
●●●●●
●●●
● ●●
●●●●
● ●
●●●●
●● ●●●●
●●
●●●● ● ●● ●●
●
●●
●
● ●
● ●
●● ●
● ● ● ●● ● ●●●●
● ● ●● ● ●
● ●
●
●
● ● ● ●● ●●● ● ●●
●●
●●● ● ●● ● ● ● ●●●● ● ● ●
● ● ● ● ●●
● ● ● ●● ● ● ●● ●● ● ● ● ●
● ● ● ● ●● ● ●
● ● ●
1.21

●
● ● ●
● ● ● ● ●
● ●● ●
● ● ●
● ●
●
● ●

●
● ●
1.19

0 200 400 600 800 1000 1200

Leave t out

Figure 4.5: OLS Wage Regression: Leave-One-Out Plot

86 CHAPTER 4. NON-NORMALITY

Figure 4.6: LAD Wage Regression

4.4. WAGE DETERMINATION 87

Figure 4.7: OLS Log Wage Regression

88 CHAPTER 4. NON-NORMALITY

Figure 4.8: OLS Log Wage Regression: Residual Plot

4.4. WAGE DETERMINATION 89

Figure 4.9: OLS Log Wage Regression: Residual Histogram and Statistics
90 CHAPTER 4. NON-NORMALITY

Figure 4.10: OLS Log Wage Regression: Residual Gaussian QQ Plot

4.4. WAGE DETERMINATION 91

Figure 4.11: OLS Log Wage Regression: Leave-One-Out Plot

92 CHAPTER 4. NON-NORMALITY

Figure 4.12: LAD Log Wage Regression

4.5 Exercises, Problems and Complements

1. (Taleb’s The Black Swan)

Nassim Taleb is a financial markets trader turned pop author. His book,
The Black Swan (Taleb (2007)), deals with many of the issues raised
in this chapter. “Black swans” are seemingly impossible or very low-
probability events – after all, swans are supposed to be white – that
occur with annoying regularity in reality. Read his book. Where does
your reaction fall on the spectrum from A to B below?
A. Taleb offers crucial lessons for econometricians, heightening awareness
in ways otherwise difficult to achieve. After reading Taleb, it’s hard to
stop worrying about non-normality, model uncertainty, etc.
B. Taleb belabors the obvious for hundreds of pages, arrogantly “inform-
ing”’ us that non-normality is prevalent, that all models are misspecified,
4.5. EXERCISES, PROBLEMS AND COMPLEMENTS 93

and so on. Moreover, it takes a model to beat a model, and Taleb offers
little.

2. (Additional ways of quantifying “outliers”)

(a) Consider the outlier probability,

P |y − µ| > 5σ

(there is of course nothing magical about our choice of 5). In practice

we use a sample version of the population object.
(b) Consider the “tail index” γ, such that

P (y > y ∗ ) = ky ∗ −γ .

In practice we use a sample version of the population object.

3. (“Leave-one-out” coefficient plots)

Leave-one-out coefficient plots are more appropriate for cross-section
data than for time-series data. Why? How might you adapt them to
handle time-series data?
94 CHAPTER 4. NON-NORMALITY
Chapter 5

Group Heterogeneity and Indicator

Variables

From one perspective we continue working under the IC. From another we
now begin relaxing the IC, effectively by recognizing RHS variables that were
omitted from, but should not have been omitted from, our original wage
regression.

5.1 0-1 Dummy Variables

A dummy variable, or indicator variable, is just a 0-1 variable that

indicates something, such as whether a person is female, non-white, or a
union member. We use dummy variables to account for such “group effects,”
if any. We might define the dummy UNION, for example, to be 1 if a person
is a union member, and 0 otherwise. That is,

1 if observation i corresponds to a union member
U N IONi =
0 otherwise.

In Figure 5.1 we show histograms and statistics for all potential determi-
nants of wages. Education (EDUC) and experience (EXPER) are standard
continuous variables, although we measure them only discretely (in years);

95
96 CHAPTER 5. GROUP HETEROGENEITY AND INDICATOR VARIABLES

Figure 5.1: Histograms for Wage Covariates

we have examined them before and there is nothing new to say. The new vari-
ables are 0-1 dummies, UNION (already defined) and NONWHITE, where

1 if observation i corresponds to a non-white person
N ON W HIT Ei =
0 otherwise.

Note that the sample mean of a dummy variable is the fraction of the
sample with the indicated attribute. The histograms indicate that roughly
one-fifth of people in our sample are union members, and roughly one-fifth
are non-white.
We also have a third dummy, FEMALE, where

1 if observation i corresponds to a female
F EM ALEi =
0 otherwise.

We don’t show its histogram because it’s obvious that FEMALE should be
approximately 0 w.p. 1/2 and 1 w.p. 1/2, which it is.
5.2. GROUP DUMMIES IN THE WAGE REGRESSION 97

Sometimes dummies like UNION, NONWHITE and FEMALE are called

intercept dummies, because they effectively allow for a different intercept
for each group (union vs. non-union, non-white vs. white, female vs. male).
The regression intercept corresponds to the “base case” (zero values for all
dummies) and the dummy coefficients give the extra effects when the respec-
tive dummies equal one. For example, in a wage regression with an intercept
and a single dummy (UNION, say), the intercept corresponds to non-union
members, and the estimated coefficient on UNION is the extra effect (up or
down) on LWAGE accruing to union members.
Alternatively, we could define and use a full set of dummies for each cate-
gory (e.g., include both a union dummy and a non-union dummy) and drop
the intercept, reading off the union and non-union effects directly.
In any event, never include a full set of dummies and an intercept. Doing
so would be redundant because the sum of a full set of dummies is just a unit
vector, but that’s what the intercept is. If an intercept is included, one of
the dummy categories must be dropped.

5.2 Group Dummies in the Wage Regression

Recall our basic wage regression,

LW AGE → c, EDU C, EXP ER,

shown in Figure 5.2. Both explanatory variables are highly significant, with
expected signs.
Now consider the same regression, but with our three group dummies
added, as shown in Figure 5.3. All dummies are significant with the expected
signs, and R2 is higher. Both SIC and AIC favor including the group dum-
mies. We show the residual scatter in Figure 5.4. Of course it’s hardly the
forty-five degree line (the regression R2 is higher but still only .31), but it’s
98 CHAPTER 5. GROUP HETEROGENEITY AND INDICATOR VARIABLES

Figure 5.2: Wage Regression on Education and Experience

Figure 5.3: Wage Regression on Education, Experience and Group Dummies

5.3. EXERCISES, PROBLEMS AND COMPLEMENTS 99

Figure 5.4: Residual Scatter from Wage Regression on Education, Experience and Group
Dummies

getting closer.

5.3 Exercises, Problems and Complements

1. (Slope dummies)
Consider the regression

yi = β1 + β2 xi + εi .

The dummy variable model as introduced in the text generalizes the

intercept term such that it can change across groups. Instead of writing
100 CHAPTER 5. GROUP HETEROGENEITY AND INDICATOR VARIABLES

the intercept as β1 , we write it as β1 + δDi .

We can also allow slope coefficients to vary with groups. Instead of
writing the slope as β2 , we write it as β2 + γDi . Hence to capture slope
variation across groups we regress y not only on an intercept and x, but
also on D ∗ x.
Allowing for both intercept and slope variation across groups corresponds
to regressing on an intercept, D, x, and D ∗ x.

2. (Dummies vs. separate regression)

Consider the simple regression, y → c, x.

(a) How is inclusion of a group G intercept dummy related to the idea

of running separate regressions, one for G and one for non-G? Are
the two strategies equivalent? Why or why not?
(b) How is inclusion of group G intercept and slope dummies related
to the idea of running separate regressions, one for G and one for
non-G? Are the two strategies equivalent? Why or why not?

3. (Analysis of variance (ANOVA) and dummy variable regression)

[You should have learned about analysis of variance (ANOVA) in
your earlier statistics course. In any event there’s good news: If you
understand regression on dummy variables, you understand analysis of
variance (ANOVA), as any ANOVA analysis can be done via regression
on dummies. So here we go.]
You treat each of 1000 randomly-selected farms that presently use no
fertilizer. You either do nothing, or you apply one of four experimental
fertilizers, A, B, C or D. Using a dummy variable regression setup:

(a) How would you test the hypothesis that none of the four new fertil-
izers is effective?
5.4. NOTES 101

(b) Assuming that you reject the null, how would you estimate the im-
provement (or worsening) due to using fertilizer A, B, C or D?

5.4 Notes

ANOVA traces to Sir Ronald Fischer’s 1918 article, “The Correlation Be-
tween Relatives on the Supposition of Mendelian Inheritance,” and it was
featured prominently in his classic 1925 book, Statistical Methods for Re-
search Workers. Fischer is in many ways the “father” of much of modern
statistics.
102 CHAPTER 5. GROUP HETEROGENEITY AND INDICATOR VARIABLES

5.5 Dummy Variables, ANOVA, and Sir Ronald Fis-

cher

Figure 5.5: Sir Ronald Fischer

Photo credit: From Wikimedia commons. Source: http://www.swlearning.com/quant/kohler/stat/

biographical_sketches/Fisher_3.jpeg Rationale: Photographer died ¿70yrs ago =¿ PD. Date: 2008-05-

30 (original upload date). Source: Transferred from en.wikipedia. Author: Original uploader was Bletchley

at en.wikipedia. Permission (Reusing this file): Released under the GNU Free Documentation License; PD-

OLD-70. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU

Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation;

with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included

in the section entitled GNU Free Documentation License.

Chapter 6

Nonlinearity

In general there is no reason why the conditional mean function should be lin-
ear. That is, the appropriate functional form may not be linear. Whether
linearity provides an adequate approximation is an empirical matter.
Non-linearity is related to non-normality, which we studied in chapter 4.
In particular, in the mutivariate normal case, the conditional mean function
is linear in the conditioning variables. But once we leave the terra firma
of multivariate normality, anything goes. The conditional mean function
and disturbances may be linear and Gaussian, non-linear and Gaussian, linear
and non-Gaussian, or non-linear and non-Gaussian.
In the Gaussian case, because the conditional mean is a linear function
of the conditioning variable(s), it coincides with the linear projection. In
non-Gaussian cases, however, linear projections are best viewed as approxi-
mations to generally non-linear conditional mean functions. That is, we can
view the linear regression model as a linear approximation to a generally non-
linear conditional mean function. Sometimes the linear approximation may
be adequate, and sometimes not.

103
104 CHAPTER 6. NONLINEARITY

6.1 Models Linear in Transformed Variables

Models can be non-linear but nevertheless linear in non-linearly-transformed

variables. A leading example involves logarithms, to which we now turn. This
can be very convenient. Moreover, coefficient interpretations are special, and
similarly convenient.

6.1.1 Logarithms

Logs turn multiplicative models additive, and they neutralize exponentials.

Logarithmic models, although non-linear, are nevertheless “linear in logs.”
In addition to turning certain non-linear models linear, they can be used
to enforce non-negativity of a left-hand-side variable and to stabilize a dis-
turbance variance. (More on that later.)

Log-Log Regression

First, consider log-log regression. We write it out for the simple regression
case, but of course we could have more than one regressor. We have

ln yi = β1 + β2 ln xi + εi .

yi is a non-linear function of the xi , but the function is linear in logarithms,

so that ordinary least squares may be applied.
To take a simple example, consider a Cobb-Douglas production function
with output a function of labor and capital,

yi = ALαi Kiβ exp(εi ).

Direct estimation of the parameters A, α, β would require special techniques.

Taking logs, however, yields

ln yi = ln A + α ln Li + β ln Ki + εi .
6.1. MODELS LINEAR IN TRANSFORMED VARIABLES 105

This transformed model can be immediately estimated by ordinary least

squares. We simply regress ln yi on an intercept, ln Li and ln Ki . Such log-log
regressions often capture relevant non-linearities, while nevertheless main-
taining the convenience of ordinary least squares.
Note that the estimated intercept is an estimate of ln A (not A, so if you
want an estimate of A you must exponentiate the estimated intercept), and
the other estimated parameters are estimates of α and β, as desired.
Recall that for close yi and xi , (ln yi − ln xi ) is approximately the percent
difference between yi and xi . Hence the coefficients in log-log regressions give
the expected percent change in E(yi |xi ) for a one-percent change in xi , the
so-called elasticity of yi with respect to xi .

Log-Lin Regression

Second, consider log-lin regression, in which ln yi = βxi +εi . We have a log

on the left but not on the right. The classic example involves the workhorse
model of exponential growth:

yt = Aert εt

It’s non-linear due to the exponential, but taking logs yields

ln yt = ln A + rt + εt ,

which is linear. The growth rate r gives the approximate percent change in
E(yt |t) for a one-unit change in time (because logs appear only on the left).

Lin-Log Regression

Finally, consider lin-log Regression:

yi = β ln xi + εi
106 CHAPTER 6. NONLINEARITY

It’s a bit exotic but it sometimes arises. β gives the effect on E(yi |xi ) of a
one-percent change in xi , because logs appear only on the right.

6.1.2 Box-Cox and GLM

Box-Cox

The Box-Cox transformation generalizes log-lin regression. We have

B(yi ) = β1 + β2 xi + εi ,

where
ytλ − 1
B(yi ) = .
λ
Hence
E(yi |xi ) = B −1 (β1 + β2 xi ).

Because
yλ − 1
 
lim = ln(yi ),
λ→0 λ
the Box-Cox model corresponds to the log-lin model in the special case of
λ = 0.

GLM

The so-called “generalized linear model” (GLM) provides an even more flex-
ible framework. Almost all models with left-hand-side variable transforma-
tions are special cases of those allowed in the generalized linear model
(GLM). In the GLM, we have

G(yi ) = β1 + β2 xt + εi ,

so that
E(yi |xi ) = G−1 (β1 + β2 xi ).
6.2. INTRINSICALLY NON-LINEAR MODELS 107

Wide classes of “link functions” G can be entertained. Log-lin regression,

for example, emerges when G(yi ) = ln(yi ), and Box-Cox regression emerges
yiλ −1
when G(yi ) = λ .

6.2 Intrinsically Non-Linear Models

Sometimes we encounter intrinsically non-linear models. That is, there

is no way to transform them to linearity, so that they can then be estimated
simply by least squares, as we have always done so far.
As an example, consider the logistic model,
1
yi = + εi ,
a + brxi
with 0 < r < 1. The precise shape of the logistic curve of course depends on
the precise values of a, b and r, but its “S-shape” is often useful. The key
point for our present purposes is that there is no simple transformation of y
that produces a model linear in the transformed variables.

6.2.1 Nonlinear Least Squares

The least squares estimator is often called “ordinary” least squares, or OLS.
As we saw earlier, the OLS estimator has a simple closed-form analytic ex-
pression, which makes it trivial to implement on modern computers. Its
computation is fast and reliable.
The adjective “ordinary” distinguishes ordinary least squares from more
laborious strategies for finding the parameter configuration that minimizes
the sum of squared residuals, such as the non-linear least squares (NLS)
estimator. When we estimate by non-linear least squares, we use a computer
to find the minimum of the sum of squared residual function directly, using
numerical methods, by literally trying many (perhaps hundreds or even thou-
sands) of different β values until we find those that appear to minimize the
108 CHAPTER 6. NONLINEARITY

sum of squared residuals. This is not only more laborious (and hence slow),
but also less reliable, as, for example, one may arrive at a minimum that is
local but not global.
Why then would anyone ever use non-linear least squares as opposed to
OLS? Indeed, when OLS is feasible, we generally do prefer it. For example,
in all regression models discussed thus far OLS is applicable, so we prefer it.
Intrinsically non-linear models can’t be estimated using OLS, but they can
be estimated using non-linear least squares. We resort to non-linear least
squares in such cases.
Intrinsically non-linear models obviously violate the linearity assumption
of the IC. But the violation is not a big deal. Under the remaining IC (that
is, dropping only linearity), β̂N LS has a sampling distribution similar to that
under the IC.

6.3 Series Expansions

There is really no such thing as an intrinsically non-linear model. In the

bivariate case we can think of the relationship as

yi = g(xi , εi )

or slightly less generally as

yi = f (xi ) + εi .

First consider Taylor series expansions of f (xi ). The linear (first-order)

approximation1 is
f (xi ) ≈ β1 + β2 xi
1
Around x0 = 0.
6.3. SERIES EXPANSIONS 109

and the quadratic (second-order) approximation is

f (xi ) ≈ β1 + β2 xi + β3 x2i .

In the multiple regression case, Taylor approximations also involves inter-

action terms. Consider, for example, a function of two regressors, f (xi , zi ).
The second-order Taylor approximation is:

f (xi , zi ) ≈ β1 + β2 xi + β3 zi + β4 x2i + β5 zi2 + β6 xi zi .

The final term picks up interaction effects. Interaction effects are also rele-
vant in situations involving dummy variables. There we capture interactions
by including products of dummies.2
The ultimate point is that even so-called “intrinsically non-linear” models
are themselves linear when viewed from the series-expansion perspective. In
principle, of course, an infinite number of series terms are required, but in
practice nonlinearity is often quite gentle (e.g., quadratic) so that only a few
series terms are required. From this viewpoint non-linearity is in some sense
really an omitted-variables problem.
One can also use Fourier series approximations:

f (xi ) ≈ β1 + β2 sin(xi ) + β3 cos(xi ) + β4 sin(2xi ) + β5 cos(2xi ) + ...,

and one can also mix Taylor and Fourier approximations by regressing not
only on powers and cross products (“Taylor terms”), but also on various sines
and cosines (“Fourier terms”). Mixing may facilitate parsimony.
2
Notice that a product of dummies is one if and only if both individual dummies are one.
110 CHAPTER 6. NONLINEARITY

6.4 A Final Word on Nonlinearity and the IC

It is of interest to step back and ask what parts of the IC are violated in our
various non-linear models.
Models linear in transformed variables (e.g., log-log regression) actually
don’t violate the IC, after transformation. Neither do series expansion mod-
els, if the adopted expansion order is deemed correct, because they too are
linear in transformed variables.
The series approach to handling non-linearity is actually very general and
handles intrinsically non-linear models as well, and low-ordered expansions
are often adequate in practice, even if an infinite expansion is required in
theory. If series terms are needed, a purely linear model would suffer from
misspecification of the X matrix (a violation of the IC) due to the omitted
higher-order expansion terms. Hence the failure of the IC discussed in this
chapter can be viewed either as:

1. The linearity assumption (E(y|X) = X ′ β) is incorrect, or

2. The linearity assumption (E(y|X) = X ′ β) is correct, but the assumption

that X is correctly specified (i.e., no omitted variables) is incorrect, due
to the omitted higher-order expansion terms.

6.5 Selecting a Non-Linear Model

6.5.1 t and F Tests, and Information Criteria

One can use the usual t and F tests for testing linear models against non-
linear alternatives in nested cases, and information criteria (AIC and SIC)
for testing against non-linear alternatives in non-nested cases. To test linear-
ity against a quadratic alternative in a simple regression case, for example,
we can simply run y → c, x, x2 and perform a t-test for the relevance of x2 .
And of course, use AIC and SIC as always.
6.6. NON-LINEARITY IN WAGE DETERMINATION 111

6.5.2 The RESET Test

Direct inclusion of powers and cross products of the various x variables in

the regression can be wasteful of degrees of freedom, however, particularly
if there are more than just one or two right-hand-side variables in the re-
gression and/or if the non-linearity is severe, so that fairly high powers and
interactions would be necessary to capture it.
In light of this, a useful strategy is first to fit a linear regression yi → c, xi
and obtain the fitted values ŷi . Then, to test for non-linearity, we run the
regression again with various powers of ŷi included,

yi → c, xi , ŷi2 , ..., ŷim .

Note that the powers of ŷi are linear combinations of powers and cross prod-
ucts of the x variables – just what the doctor ordered. There is no need
to include the first power of ŷi , because that would be redundant with the
included x variables. Instead we include powers ŷi2 , ŷi3 , ... Typically a small
m is adequate. Significance of the included set of powers of ŷi can be checked
using an F test. This procedure is called RESET (Regression Specification
Error Test).

6.6 Non-Linearity in Wage Determination

For convenience we reproduce in Figure 6.1 the results of our current linear
wage regression,
LW AGE → c, EDU C, EXP ER,

F EM ALE, U N ION, N ON W HIT E.

The RESET test from that regression suggests neglected non-linearity; the
p-value is .03 when using ŷt2 and ŷt3 in the RESET test regression.
Non-Linearity in EDU C and EXP ER: Powers and Interactions
112 CHAPTER 6. NONLINEARITY

Figure 6.1: Basic Linear Wage Regression

Given the results of the RESET test, we proceed to allow for non-linearity.
In Figure 6.2 we show the results of the quadratic regression

LW AGE → EDU C, EXP ER

EDU C 2 , EXP ER2 , EDU C ∗ EXP ER,

F EM ALE, U N ION, N ON W HIT E

Two of the non-linear effects are significant. The impact of experience is

decreasing, and experience seems to trade off with education, insofar as the
interaction is negative.
Non-Linearity in F EM ALE, U N ION and N ON W HIT E: Interactions
Just as continuous variables like EDU C and EXP ER may interact (and
we found that they do), so too may discrete dummy variables. For example,
the wage effect of being female and non-white might not simply be the sum
of the individual effects. We would estimate it as the sum of coefficients on
6.6. NON-LINEARITY IN WAGE DETERMINATION 113

Figure 6.2: Quadratic Wage Regression

the individual dummies F EM ALE and N ON W HIT E plus the coefficient

on the interaction dummy F EM ALE*N ON W HIT E.
In Figure 6.4 we show results for

LW AGE → EDU C, EXP ER,

F EM ALE, U N ION, N ON W HIT E,

F EM ALE∗U N ION, F EM ALE∗N ON W HIT E, U N ION ∗N ON W HIT E.

The dummy interactions are insignificant.

6.6.1 Non-Linearity in Continuous and Discrete Variables Simul-

taneously

Now let’s incorporate powers and interactions in EDU C and EXP ER, and
interactions in F EM ALE, U N ION and N ON W HIT E.
114 CHAPTER 6. NONLINEARITY

Figure 6.3: Wage Regression on Education, Experience, Group Dummies, and Interactions

In Figure 6.4 we show results for

LW AGE → EDU C, EXP ER,

EDU C 2 , EXP ER2 , EDU C ∗ EXP ER,

F EM ALE, U N ION, N ON W HIT E,

F EM ALE∗U N ION, F EM ALE∗N ON W HIT E, U N ION ∗N ON W HIT E.

The dummy interactions remain insignificant.

Note that we could explore additional interactions among EDU C, EXP ER
and the various dummies. We leave that to the reader.
Assembling all the results, our tentative “best” model thus far is that of
section 6.6,
LW AGE → EDU C, EXP ER,
6.7. EXERCISES, PROBLEMS AND COMPLEMENTS 115

Figure 6.4: Wage Regression with Continuous Non-Linearities and Interactions, and Discrete
Interactions

EDU C 2 , EXP ER2 , EDU C ∗ EXP ER,

F EM ALE, U N ION, N ON W HIT E.

The RESET statistic has a p-value of .19, so we would not reject adequacy
of functional form at conventional levels.

6.7 Exercises, Problems and Complements

1. (Tax revenue and the tax rate)

The U.S. Congressional Budget Office (CBO) is helping the president
to set tax policy. In particular, the president has asked for advice on
where to set the average tax rate to maximize the tax revenue collected
per taxpayer. For each of 65 countries the CBO has obtained data on
the tax revenue collected per taxpayer and the average tax rate.
116 CHAPTER 6. NONLINEARITY

(a) Is tax revenue likely related to the tax rate? (That is, do you think
that the mean of tax revenue conditional on the tax rate actually is
a function of the tax rate?)
(b) Is the relationship likely linear? (Hint: how much revenue would be
collected at tax rates of zero or one hundred percent?)
(c) If not, is a linear regression nevertheless likely to produce a good
approximation to the true relationship?

2. (Graphical regression diagnostic: scatterplot of ei vs. xi )

This plot helps us assess whether the relationship between y and x is
truly linear, as assumed in linear regression analysis. If not, the linear
regression residuals will depend on x. In the case where there is only
one right-hand side variable, as above, we can simply make a scatterplot
of ei vs. xi . When there is more than one right-hand side variable, we
can make separate plots for each, although the procedure loses some of
its simplicity and transparency.

3. (Difficulties with non-linear optimization)

Non-linear optimization can be a tricky business, fraught with problems.
Some problems are generic. It’s relatively easy to find a local optimum,
for example, but much harder to be confident that the local optimum
is global. Simple checks such as trying a variety of startup values and
checking the optimum to which convergence occurs are used routinely,
but the problem nevertheless remains. Other problems may be software
specific. For example, some software may use highly accurate analytic
derivatives whereas other software uses approximate numerical deriva-
tives. Even the same software package may change algorithms or details
of implementation across versions, leading to different results.

4. (Conditional mean functions)

6.7. EXERCISES, PROBLEMS AND COMPLEMENTS 117

Consider the regression model,

yi = β1 + β2 xi + β3 x2i + β4 zi + εi

under the full ideal conditions. Find the mean of yi conditional upon
xi = x∗i and zi = zi∗ . Is the conditional mean linear in (x∗i ? zi∗ )?

5. (OLS vs. NLS)

Consider the following three regression models:

yi = β1 + β2 xi + εi
yi = β1 eβ2 xi εi
yi = β1 + eβ2 xi + εi .

a. For each model, determine whether OLS may be used for estimation
(perhaps after transforming the data), or whether NLS is required.
b. For those models for which OLS is feasible, do you expect NLS and
OLS estimation results to agree precisely? Why or why not?
c. For those models for which NLS is “required,” show how to avoid it
using series expansions.

6. (What is linear regression really estimating?)

It is important to note the distinction between a conditional mean and
a linear projection. The conditional mean is not necessarily a lin-
ear function of the conditioning variable(s). The linear projection is of
course a linear function of the conditioning variable(s), by construction.
Linear projections are best viewed as approximations to generally non-
linear conditional mean functions. That is, we can view an empirical
linear regression as estimating the population linear projection, which
in turn is an approximation to the population conditional expectation.
118 CHAPTER 6. NONLINEARITY

Sometimes the linear projection may be an adequate approximation, and

sometimes not.

7. Putting lots of things together.

Consider the cross-sectional (log) wage equation that we studied exten-
sively, which appears again in Figure 6.5 for your reference.

(a) The model was estimated using ordinary least squares (OLS). What
loss function is optimized in calculating the OLS estimate? (Give
a formula and a graph.) What is the formula (if any) for the OLS
estimator?
(b) Consider instead estimating the same model numerically (i.e., by
NLS) rather than analytically (i.e., by OLS). What loss function is
optimized in calculating the NLS estimate? (Give a formula and a
graph.) What is the formula (if any) for the NLS estimator?
(c) Does the estimated equation indicate a statistically significant effect
of union status on log wage? An economically important effect?
What is the precise interpretation of the estimated coefficient on
UNION? How would the interpretation change if the wage were not
logged?
(d) Precisely what hypothesis does the F-statistic test? What are the
restricted and unrestricted sums of squared residuals to which it
is related, and what are the two OLS regressions to which they
correspond?
(e) Consider an additional regressor, AGE, where AGE = 6 + EDUC
+ EXPER. (The idea is that 6 years of early childhood, followed by
EDUC years of education, followed by EXPER years of work expe-
rience should, under certain assumptions, sum to a person’s age.)
Discuss the likely effects, if any, of adding AGE to the regression.
6.7. EXERCISES, PROBLEMS AND COMPLEMENTS 119

(f) The log wage may of course not be linear in EDUC and EXPER.
How would you assess the possibility of quadratic nonlinear effects
using t-tests? An F-test? The Schwarz criterion (SIC)? R2 ?
(g) Suppose you find that the log wage relationship is indeed non-linear
but still very simple, with only EXPER2 entering in addition to the
∂ E(LWAGE | X)
variables in Figure 6.5. What is ∂ EXPER in the expanded model?
∂ E(LWAGE | X)
How does it compare to ∂ EXPER in the original model of Figure
6.5? What are the economic interpretations of the two derivatives?
(X refers to the full set of included right-hand-side variables in a
regression.)
(h) Return to the original model of Figure 6.5. How would you assess
the overall adequacy of the fitted model using the standard error of
the regression? The model residuals? Which is likely to be more
useful/informative?
(i) Consider estimating the model not by OLS or NLS, but rather by
quantile regression (QR). What loss function is optimized in calcu-
lating the QR estimate? (Give a formula and a graph.) What is the
formula (if any) for the QR estimator? How is the least absolute
deviations (LAD) estimator related to the QR estimator? Under
the IC, are the OLS and LAD estimates likely very close? Why or
why not?
(j) Discuss whether and how you would incorporate trend and seasonal-
ity by using a linear time trend variable and a set of seasonal dummy
variables.
120 CHAPTER 6. NONLINEARITY

Figure 6.5: Regression Output

Chapter 7

Heteroskedasticity

We continue exploring issues associated with possible failure of the ideal con-
ditions. This chapter’s issue is “Do we really believe that disturbance vari-
ances are constant?” As always, consider: ε ∼ N (0, Ω). Heteroskedasticity
corresponds to Ω diagonal but Ω ̸= σ 2 I
 
σ12 0 . . . 0
 0 σ22 . . .
 
0 
Ω=
 ... ... . . . ..

 . 

2
0 0 . . . σN
Heteroskedasticity can arise for many reasons. A leading cause is that σi2
may depend on one or more of the xi ’s. A classic example is an “Engel curve”,
a regression relating food expenditure to income. Wealthy people have much
more discretion in deciding how much of their income to spend on food, so
their disturbances should be more variable, as routinely found.

7.1 Consequences of Heteroskedasticity for Estimation,

Inference, and Prediction

As regards point estimation, OLS remains largely OK, insofar as parameter

estimates remain consistent and asymptotically normal. They are, however,

121
122 CHAPTER 7. HETEROSKEDASTICITY

rendered inefficient. But consistency is key. Inefficiency is typically inconse-

quential in large samples, as long as we have consistency.
As regards inference, however, heteroskedasticity wreaks significant havoc.
Standard errors are biased and inconsistent. Hence t statistics do not have the
t distribution in finite samples and do not even have the N (0, 1) distribution
asymptotically.
Finally, as regards prediction, results vary depending on whether we’re
talking about point or density prediction. Our earlier feasible point forecasts
constructed under homoskedasticity remain useful under heteroskedasticity.
Because parameter estimates remain consistent, we still have

| xi =x∗i ) →p E(yi | xi =x∗i ).

E(yi\

In contrast, our earlier feasible density forecasts do not remain useful,

because under heteroskedasticity it is no longer appropriate to base them on
“identical σ’s for different people”. Now we need to base them on “different
σ’s for different people”.

7.2 Detecting Heteroskedasticity

We will consider both graphical heteroskedasticity diagnostics and formal

heteroskedasticity tests. The two approaches are complements, not substi-
tutes.

7.2.1 Graphical Diagnostics

The first thing we can do is graph e2i against xi , for various regressors, looking
for relationships. This makes sense because e2i is effectively a proxy for σi2 .
Recall, for example, our “Final” wage regression, shown in Figure 7.1.
In Figure 7.2 we graph the squared residuals agains EDUC. There is appar-
ently a positive relationship, although it is noisy. This makes sense, because
7.2. DETECTING HETEROSKEDASTICITY 123

Figure 7.1: Final Wage Regression

very low education almost always leads to very low wage, whereas high ed-
ucation can produce a larger variety of wages (e.g., both neurosurgeons and
college professors are highly educated, but neurosurgeons typically earn much
more).

7.2.2 Formal Tests

The Breusch-Pagan-Godfrey Test (BPG)

An important limitation of the graphical method for heteroskedasticity de-

tection is that it is purely pairwise (we can only examine one x at a time),
whereas the disturbance variance might actually depend on more than one
x. Formal tests let us blend the information from multiple x’s, and they also
let us assess statistical significance.
The BPG test proceeds as follows:

1. Estimate the OLS regression, and obtain the squared residuals

124 CHAPTER 7. HETEROSKEDASTICITY

Figure 7.2: Squared Residuals vs. Years of Education

2. Regress the squared residuals on all regressors

3. To test the null hypothesis of no relationship, examine N · R2 from this

regression. It can be shown that in large samples N · R2 ∼ χ2K−1 under
the null of homoskedasticity, where K is the number of regressors in the
test regression.

We show the BPG test results in Figure 7.3.

White’s Test

White’s test is a simple extension of BPG, replacing the linear BPG test
regression with a more flexible (quadratic) regression:

1. Estimate the OLS regression, and obtain the squared residuals

2. Regress the squared residuals on all regressors, squared regressors, and

pairwise regressor cross products
7.2. DETECTING HETEROSKEDASTICITY 125

Figure 7.3: BPG Test Regression and Results

3. To test the null hypothesis of no relationship, examine N · R2 from this

regression. It can be shown that in large samples N · R2 ∼ χ2K−1 under
the null of homoskedasticity, where K is the number of regressors in the
test regression.

We show the White test results in Figure 7.4.

Figure 7.4: White Test Regression and Results

126 CHAPTER 7. HETEROSKEDASTICITY

Figure 7.5: Wage Regression with Heteroskedasticity-Robust Standard Errors

7.3 Dealing with Heteroskedasticity

We will consider both adjusting standard errors and adjusting density fore-
casts.

7.3.1 Adjusting Standard Errors

Using advanced methods, one can obtain consistent standard errors, even
when heteroskedasticity is present. Mechanically, it’s just a simple regression
option. e.g., in EViews, instead of “ls y,c,x”, use “ls(cov=white) y,c,x”.
Even if you’re only interested in prediction, you still might want to use
robust standard errors, in order to do credible inference regarding the con-
tributions of the various x variables to the point prediction.
In Figure 7.5 we show the final wage regression with robust standard errors.
Although the exact values of the standard errors change, it happens in this
case that significance of all coefficients is preserved.
7.4. EXERCISES, PROBLEMS AND COMPLEMENTS 127

7.3.2 Adjusting Density Forecasts

Recall operational density forecast under the ideal conditions (which include,
among other things, Gaussian homoskedastic disturbances):

yi | xi =x∗ ∼ N (x∗ ′ β̂LS , s2 ).

Now, under heterskedasticity (but maintaining normality), we have the nat-

ural extension,
yi | xi =x∗ ∼ N (x∗ ′ β̂LS , σ̂∗2 ),

where σ̂∗2 is the fitted value from the BPG or White test regression evaluated
at x∗ .

7.4 Exercises, Problems and Complements

1. (Vocabulary)
All these have the same meaning:

(a) “Heteroskedasticity-robust standard errors”

(b) “Heteroskedasticity-consistent standard errors”
(c) “Robust standard errors”
(d) “White standard errors”
(e) “White-washed” standard errors”

2. (Generalized Least Squares (GLS))

For arbitrary Ω matrix, it can be shown that full estimation efficiency re-
quires “generalized least squares” (GLS) estimation. The GLS estimator
is:
β̂GLS = (X ′ Ω−1 X)−1 X ′ Ω−1 y.
128 CHAPTER 7. HETEROSKEDASTICITY

Under the ideal conditions (but allowing for Ω ̸= σ 2 I) it is consistent,

MVUE, and normally distributed with covariance matrix (X ′ Ω−1 X)−1 :

β̂GLS ∼ N β, (X ′ Ω−1 X)−1 .

(a) Show that when Ω = σ 2 I the GLS estimator is just the standard
OLS estimator:

β̂GLS = β̂OLS = (X ′ X)−1 X ′ y.

(b) Show that when Ω = σ 2 I the covariance matrix of the GLS estimator
is just that of the standard OLS estimator:

cov(β̂GLS ) = cov(β̂OLS ) = σ 2 (X ′ X)−1 .

3. (GLS for Heteroskedasticity)

(a) Show that GLS for heteroskedasticity amounts to OLS on data

weighted by the inverse disturbance standard deviation (1/σi ), of-
ten called “weighted least squares” (WLS). This is “infeasible” WLS
since in general we don’t know the σi ’s.
(b) To see why WLS works, consider the heteroskedastic DGP:

yi = x′i β + εi
εi ∼ iidN (0, σi2 ).

Now weight the data (yi , xi ) by 1/σi :

yi x′i β εi
= + .
σi σi σi

The transformed (but equivalent) DGP is then:

yi∗ = x∗i ′ β + ε∗i

7.4. EXERCISES, PROBLEMS AND COMPLEMENTS 129

ε∗i ∼ iidN (0, 1).

The weighted data satisfies the IC and so OLS is MVUE! So GLS is

just OLS on appropriately transformed data. In the heteroskedastic-
ity case the appropriate transfomation is weighting. We downweight
high-variance observations, as is totally natural.

4. (Details of Weighted Least Squares)

Note that weighting the data by 1/σi is the same as
weighting the residuals by 1/σi2 :

N  2 N
X yi − x ′ β X 1 2
min i
= min 2 (yi − x′i β) .
β
i=1
σi β σ
i=1 i

5. (Feasible Weighted Least Squares)

To make WLS feasible, we need to replace the unknown σi2 ’s with esti-
mates.

ˆ Good idea: Use weights wi = 1/eb2i , where eb2i are from the BGP test
regression
ˆ Good idea: Use wi = 1/eb2i , where eb2i are from the White test regres-
sion.
ˆ Bad idea: Use wi = 1/e2i is not a good idea. e2i is too noisy; we’d like
to use not e2i but rather E(e2i |xi ). So we use an estimate of E(e2i |xi ),
namely eb2 from e2 → X
i

In Figure 7.6 we show regression results with weighting based on the

results from the White test regression.

6. (Robustness iteration)
Sometimes, after an OLS regression, people do a second-stage WLS with
weights 1/|ei |, or something similar. This is not a heteroskedasticity
130 CHAPTER 7. HETEROSKEDASTICITY

Figure 7.6: Regression Weighted by Fit From White Test Regression

correction, but rather a strategy to downweight outliers. But notice

that the two are closely related.

7. (Spatial Correlation)
So far we have studied a heteroskedastic situation (εi independent across
i but not identically distributed across t). But do we really believe that
the disturbances are uncorrelated over space (i)? Spatial correlation in
cross sections is another type of violation of the IC. (This time it’s “non-
zero disturbance covariances” as opposed to “non-constant disturbance
variances”.) As always, consider ε ∼ N (0, Ω). Spatial correlation (with
possible heteroskedasticity as well) corresponds to:
 
σ12 σ12 . . . σ1N
 σ21 σ22 . . . σ2N 
 
Ω=  ... .. . . . ..  .

 . . 
2
σN 1 σN 2 . . . σ N
7.4. EXERCISES, PROBLEMS AND COMPLEMENTS 131

8. (“Clustering” in spatial correlation)

Ω could be non-diagonal in cross sections but still sparse in certain ways.
A key case is block-diagonal Ω, in which there is nonzero covariance
within certain sets of disturbances, but not across sets (“clustering”).
132 CHAPTER 7. HETEROSKEDASTICITY
Chapter 8

Limited Dependent Variables

In this chapter we study so-called “discrete-response models”, or “qualitative-

response models”, or “limited dependent variable models”, or “classification
models”. Terms like “limited dependent variables,” refer to variables that
can take only a limited number of values. The classic case is a 0-1 “dummy
variable.” The twist is that the dummy variable appears on the left side of a
regression, as opposed to the already-discussed use of dummies on the right.
Dummy right-hand side variables (RHS) variables create no problem, and
you already understand them. The new issue is Dummy left-hand-side vari-
ables (LHS), which do raise special issues.

8.1 Binary Response

Note that the basic regression model,

yi = x′i β + εi

immediately implies that

E(yi |xi ) = x′i β.

Here we consider left-hand-side variables yi = Ii (z), where the dummy vari-

able (“indicator variable”) Ii (z) indicates whether event z occurs; that

133
134 CHAPTER 8. LIMITED DEPENDENT VARIABLES

is,
(
1 if event z occurs
Ii (z) =
0 otherwise.
In that case we have
E(Ii (z)|xi ) = x′i β.

A key insight, however, is that

E(Ii (z)|xi ) = P (Ii (z) = 1|xi ),

so the model is effectively

P (Ii (z) = 1|xi ) = x′i β. (8.1)

That is, when the LHS variable is a 0-1 indicator variable, the model is effec-
tively a model relating a conditional probability to the conditioning variables.
There are numerous “events” that fit the 0-1 paradigm. Examples pur-
chasing behavior does a certain consumer buy or not buy a certain product?,
hiring behavior (does a certain firm hire or not hire a certain worker?), and
loan defaults (does a certain borrower default or not default on a loan?), and
recessions (will a certain country have or not have a recession begin during
the next year?).
But how should we “fit a line” when the LHS variable is binary? The
linear probability model does it by brute-force OLS regression Ii (z) → xi .
There are several econometric problems associated with such regressions, but
the one of particular relevance is simply that the linear probability model
fails to constrain the fitted values of E(Ii (z)|xi ) = P (Ii (z) = 1|xi ) to lie in
the unit interval, in which probabilities must of course lie. We now consider
models that impose that constraint by running x′i β through a “squashing
function,” F (·), that keeps P (Ii (z) = 1|xi ) in the unit interval. That is, we
8.2. THE LOGIT MODEL 135

move to models with

P (Ii (z) = 1|xi ) = F (x′i β),

where F (·) is monotone increasing, with limw→−∞ F (w) = 0 and limw→∞ F (w) =
1. Many squashing functions can be entertained, and many have been enter-
tained.

8.2 The Logit Model

The most popular and useful squashing function for our purposes is the logis-
tic function, which takes us to the so-called “logit” model. There are several
varieties and issues, to which we now turn.

8.2.1 Logit

In the logit model, the squashing function F (·) is the logistic function,

ew 1
F (w) = = ,
1 + ew 1 + e−w
so ′
exi β
P (Ii (z) = 1|xi ) = ′ .
1 + e xi β
At one level, there’s little more to say; it really is that simple. The likelihood
function can be derived, and the model can be immediately estimated by
numerical maximization of the likelihood function.
But an alternative latent variable formulation yields useful insights. In
particular, consider a latent variable, yt∗ , where

yi∗ = x′i β + εi
εi ∼ logistic(0, 1),

and let Ii (z) be Ii (yi∗ > 0), or equivalently, Ii (ε > −x′i β). Interestingly, this
136 CHAPTER 8. LIMITED DEPENDENT VARIABLES

is the logit model. To see this, note that

E(Ii (yi∗ > 0)|xi ) = P ((yi∗ > 0)|xi ) = P (εi > −x′i β)
= P (εi < x′i β) (by symmetry of the logistic density of ε)
′
exi β
= ′ ,
1 + e xi β
where the last equality holds because the logistic cdf is ew /(1 + ew ).
This way of thinking about the logit DGP – a continuously-evolving latent
variable yi∗ with an observed indicator that turns “on” when yi∗ > 0 – is
very useful. For example, it helps us to think about consumer choice as a
function of continuous underlying utility, business cycle regime as a function
of continuous underlying macroeconomic conditions, etc.
The latent-variable approach also leads to natural generalizations like or-
dered logit, to which we now turn.

8.2.2 Ordered Logit

Here we still imagine a continuously-evolving underlying latent variable, but

we have a more-refined indicator, taking not just two values, but several
(ordered) values. Examples include financial analyst stocks ratings of “buy,”
“hold” and “sell”, and surveys that ask about degree of belief in three or
more categories ranging from “strongly disagree” through “strongly agree.”
Suppose that there are N ordered outcomes. As before, we have a
continuously-evolving latent variable,

yi∗ = x′i β + εi

εi ∼ logistic(0, 1).
8.2. THE LOGIT MODEL 137

But now we have an indicator with a finer gradation:




 0 if yi∗ < c1
 ∗
 1 if c1 < yi < c2



Ii (yi∗ ) = 2 if c2 < yi∗ < c3

 ..


 .

 N if c < y ∗ .

N i

We can estimate this ordered logit model by maximum likelihood, just as

with the standard logit model. Under some assumptions, all interpretation
remains the same.

8.2.3 Complications

In logit regression, both the marginal effects and the R2 are hard to determine
and/or interpret directly.

Marginal Effects

Logit marginal effects ∂E(y|x)/∂xi are hard to determine directly; in partic-

ular, they are not simply given by the βi ’s. Instead we have

∂E(y|x)
= f (x′ β)βi ,
∂xi

where f (x) = dF (x)/dx is the density corresponding the cdf f .1 So the

marginal effect is not simply βi ; instead it is βi weighted by f (x′ β), which
depends on all β’s and x’s. However, signs of β’s are the signs of the effects,
because f must be positive. In addition, ratios of β’s do give ratios of effects,
because the f ’s cancel.
1
In the leading logit case, f (x) would be the logistic density.
138 CHAPTER 8. LIMITED DEPENDENT VARIABLES

R2

Recall that traditional R2 for continuous LHS variables is

(yi − ŷi )2
P
2
R =1− P .
(yi − ȳi )2

It’s not clear how to define or interpret R2 when the LHS variable is 0-1, but
several variants have been proposed. The two most important are Effron’s
and McFadden’s.
Effron’s R2 is

(yi − P̂ (Ii (z) = 1|xi ))2

P
R2 = 1 − P .
(yi − ȳi )2

Effron’s R2 attempts to maintain the R2 interpretation as variation explained

and as correlation between actual and fitted values.
McFadden’s R2 is
lnL̂1
R2 = 1 − ,
lnL̂0
where lnL̂0 is the maximized restricted log likelihood (only an intercept in-
cluded) and lnL̂1 is the maximized unrestricted log likelihood. McFadden’s
R2 attempts to maintain the R2 interpretation as improvement from restricted
to unrestricted model.

8.3 Classification and “0-1 Forecasting”

Classification maps probabilities into 0-1 forecasts. The so-called “Bayes

classifier” uses a cutoff (“decision boundary”) of .5, which is hardly surprising.
That is, we predict 1 when logit(x′ β) > 1/2. Note, however, that that’s the
same as predicting 1 when x′ β > 0. If there are 2 RHS variables (potentially
plus an intercept), then the condition x′ β > 0 defines a line in R2 . Points on
one side will be classified as 0, and points on the other side will be classified
8.4. EXERCISES, PROBLEMS AND COMPLEMENTS 139

as 1. That line is the decision boundary.

We can also have non-linear decision boundaries. Suppose for example
that that x vector contains not only x1 and x2 , but also x21 and x22 . Now the
condition x′ β > 0 defines a circle in R2 . Points inside will be classified as 0,
and points outside will be classified as 1. The circle is the decision boundary.

8.4 Exercises, Problems and Complements

1. (Logit and Ordered-Logit Situations)

In the chapter we gave several examples where logit or ordered-logit
modeling would be appropriate.

a. Give three additional examples where logit modeling would be appro-

priate. Why?
b. Give three additional examples where ordered-logit modeling would
be appropriate. Why?

2. (The Logistic Squashing Function)

We used the logistic function throughout this chapter. In particular, it
is the foundation on which the logit model is built.

a. What is the logistic function? Write it down precisely.

b. From where does the logistic function come?
c. Verify that the logistic function is a legitimate squashing function.
That is, verify that it is monotone increasing, with limw→∞ F (w) = 1
and and limw→−∞ F (w) = 0.

3. (The Logit Likelihood Function)

Consider the logit model (8.1). It is more formally called a binomial
logit model, in reference to its two outcome categories.
140 CHAPTER 8. LIMITED DEPENDENT VARIABLES

a. Derive the likelihood function. (Hint: Consider the binomial struc-

ture.)
b. Must the likelihood be maximized numerically, or is an analytic for-
mula available?

4. (Logit as a Linear Model for Log Odds)

The odds O(Ii (z) = 1|xi ) of an event z are just a simple transformation
of its probability

P (Ii (z) = 1|xi )

O(Ii (z) = 1|xi ) = .
1 − P (Ii (z) = 1|xi )

Consider a linear model for log odds

 
P (Ii (z) = 1|xi )
ln = x′i β.
1 − P (Ii (z) = 1|xi )

Solving the log odds for P (Ii (z) = 1|xi ) yields the logit model,
′
1 e xi β
P (Ii (z) = 1|xi ) = ′ = ′ .
1 + e−xi β 1 + e xi β
Hence the logit model is simply a linear regression model for log odds.
A full statement of the model is

yi ∼ Bern(pi )
 
pi
ln = x′i β.
1 − pi
5. (Probit and GLM Squashing Functions)
Other squashing functions are sometimes used for binary-response re-
gression.

a. In the probit model, we simply use a different squashing function

to keep probabilities in the unit interval. F (·) is the standard normal
8.4. EXERCISES, PROBLEMS AND COMPLEMENTS 141

cumulative density function (cdf), so the model is

P (Ii (z) = 1|xi ) = Φ(x′i β),

where Φ(x) = P (z ≤ x) for N (0, 1) random variable z.

b. More exotic, but equally simple, squashing functions have also been
used. Almost all (including those used with logit and probit) are
special cases of those allowed in the generalized linear model
(GLM), a flexible regression framework with uses far beyond just
binary-response regression. In the GLM,

E(yi |xi ) = G−1 (x′i β),

and very wide classes of “link functions” G can be entertained.

6. (Multinomial Models)
In contrast to the binomial logit model, we can also have more than
two categories (e.g., what transportation method will I choose to get to
work: Private transportation, public transportation, or walking?), and
use multinomial logit.

7. (Other Situations/Mechanisms Producing Limited Dependent Variables)

Situations involving censoring or counts also produce limited dependent
variables.

a. Data can be censored by definition (e.g. purchases can’t be negative).

For example, we might see only yi , where yi = yi∗ if yi∗ ≥ 0, and 0
otherwise, and where

yi∗ = β0 + β1 xi + εi .

This is the framework in which the Tobit model works.

142 CHAPTER 8. LIMITED DEPENDENT VARIABLES

b. Data can be censored due to sample selection, for example if income

is forecast using a model fit only to high-income people.
c. “Counts” (e.g., points scored in hockey games) are automatically cen-
sored, as they must be in the natural numbers, 1, 2, 3...
Chapter 9

Causal Estimation

In this chapter we distinguish between the predictive modeling perspective

(which we have adopted so far) and what we will call the causal estimation
perspective. Both are tremendously important in econometrics.
We will investigate the properties of OLS from each perspective. It turns
out that much hinges on the validity of IC2, which, as you recall from Chap-
ter 3, says that X and ε are independent, and which we have so far not
discussed.1 Roughly, it turns out that from the predictive modeling perspec-
tive everything remains fine asymptotically even if IC2 fails (which is why we
have not yet had reason to discuss it): β̂OLS is still consistent in an appro-
priate sense and asymptotically normally distributed. But from the causal
estimation perspective disasters can occur if IC2 fails, depending on the na-
ture of the failure. In general β̂OLS is not even consistent, so that inference
is potentially severely distorted even in arbitrarily large samples.
Of course none of this makes sense yet, as we have yet introduced the
causal estimation perspective. We now do so.
1
The IC2 assumption of full independence is generally stronger than necessary for reliable causal esti-
mation and can be weakened, as per EPC 13 of chapter 3, but for ease of exposition we will continue to
maintain full independence here.

143
144 CHAPTER 9. CAUSAL ESTIMATION

9.1 Predictive Modeling vs. Causal Estimation

Here we distinguish “predictive modeling” from “causal estimation”, or “non-

causal prediction” from “causal prediction”, or “conditional expectation es-
timation” from “partial derivative estimation”.
A major goal in econometrics, as we have emphasized thus far, is predicting
y. In the language of estimation, the question is “If a new person i arrives
with covariates xi , what is my minimum-MSE estimate of her yi ?” So we
are estimating a conditional mean E(y|x). That is the domain of predictive
modeling.
Sometimes another goal in econometrics is predicting the effects of exoge-
nous “treatments” or “interventions” or “policies”. Phrased in the language
of estimation, the question is “If I intervene and give someone a certain
treatment ∂x, what is my minimum-MSE estimate of her ∂y?” So we are es-
timating the partial derivative ∂y/∂x, which in general is very different from
estimating a conditional expectation E(y|x). That is the domain of causal
estimation.
Related, it is important to note the distinction between what we will call
“consistency for a predictive effect” and “consistency for a treatment effect.”
In large samples, under very general conditions, the relationship estimated
by running y → c, x is useful for predicting y given an observation on x
(“consistency for a predictive effect”). But it may or may not be useful for
determining the effect on y of an exogenous shift in x (“consistency for a
treatment effect”). The two types of consistency coincide under the IC, but
they diverge when IC2 fails.
9.1. PREDICTIVE MODELING VS. CAUSAL ESTIMATION 145

9.1.1 Predictive Modeling and P-Consistency

Consider again a standard linear regression setting with K regressors. Under

quadratic loss, the predictive risk of a parameter configuration β is

R(β) = E(y − x′ β)2 .

Let B be a set of β’s and let β ∗ ∈ B minimize R(β). We will say that β̂
is consistent for a predictive effect (“P-consistent”) if plim R(β̂) = R(β ∗ ).
Hence in large samples β̂ provides a good way to predict y for any hypothet-
ical x: simply use x′ β̂. OLS is effectively always P-consistent; we require
almost no conditions of any kind! P-consistency is effectively induced by the
minimization problem that defines OLS, as the minimum-MSE predictor is
the conditional mean.

9.1.2 Causal Estimation and T-Consistency

Consider a standard linear regression setting with K regressors. We will say

that an estimator β̂ is consistent for a treatment effect (“T-consistent”) if
plim β̂k = ∂E(y|x)/∂xk , ∀k = 1, ..., K. Hence in large samples β̂k provides a
good estimate of the effect on y of a one-unit “treatment” or “intervention”
performed on xk . OLS is T-consistent under the IC including IC2. OLS is
generally not T-consistent without IC2.

9.1.3 Correlation vs. Causality, and P-Consistency vs. T-Consistency

The distinction between P-consistency and T-consistency is intimately re-

lated to the distinction between correlation and causality. As is well known,
correlation does not imply causality! As long as x and y are correlated, we
can exploit the correlation (as captured in β̂LS from the regression y → x)
very generally to predict y given knowledge of x. That is, there will be a
nonzero “predictive effect” of x knowledge on y. But nonzero correlation
146 CHAPTER 9. CAUSAL ESTIMATION

doesn’t necessarily tell us anything about the causal “treatment effect” of x

treatments on y. That requires the ideal conditions, and in particular IC2.
Even if there is a non-zero predictive effect of x on y (as captured by β̂LS ),
there may or may not be a nonzero treatment effect of x on y, and even if
there is a nonzero treatment effect it will generally not equal the predictive
effect.
So, assembling things, we have that:

1. P-consistency is consistency for a non-causal predictive effect. It is al-

most trivially easy to obtain, by virtue of the objective function that
OLS optimizes.

2. T-consistency is consistency for a causal predictive effect. It is quite

difficult to obtain reliably, because it requires IC2, which may fail for a
variety of reasons.

Thus far we have sketched why P-consistency holds very generally, whereas
we have simply asserted that T-consistency is much more difficult to obtain
and relies critically on IC2. We now sketch why T-consistency is much more
difficult to obtain and relies critically on IC2.
Consider the following example. Suppose that y and z are in fact causally
unrelated, so that the true treatment effect of z on y is 0 by construction.
But suppose that z is correlated with an unobserved variable x that does
cause y. Then y and z will be correlated due to their joint dependence on x,
and that correlation can be used to predict y given z, despite the fact that, by
construction, z treatments (interventions) will have no effect on y. Clearly
this sort of situation – omission of a relevant variable correlated with an
included variable – may happen commonly, and it violates IC2. In the next
section we sketch several situations that produce violations of IC2, beginning
with an elaboration on the above-sketched omitted-variables problem.
9.2. REASONS FOR FAILURE OF IC2 147

9.2 Reasons for Failure of IC2

IC2 can fail for several reasons, and we now sketch some of the most impor-
tant.

9.2.1 Omitted Variables (“Confounders”)

Omission of relevant variables is a clear violation of the ideal conditions,

insofar as the IC explicitly state that the fitted model matches the DGP. But
there is a deeper way to see why and when omitted variables cause trouble,
and when they don’t, and it involves IC2.
Suppose that the DGP is

y = βx + ε,

with all IC satisfied, but that we incorrectly regress y → z, where corr(x, z) >
0. If we erroneously interpret the regression y → z causally, then clearly
we’ll estimate a positive causal effect of z on y, in large as well as small
samples, even though it’s completely spurious and would vanish if x had been
included in the regression. The positive bias arises because in our example
corr(x, z) > 0; in general the sign of the bias could go either way, depending
on the sign of the correlation. We speak of “omitted variable bias”.
In this example the problem is that condition IC2 is violated in the regres-
sion y → z, because the disturbance is correlated with the regressor. β̂OLS
is P-consistent, as always. But it’s not T-consistent, because the omitted
variable x is lurking in the disturbance of the fitted regression y → z, which
makes the disturbance correlated with the regressor (i.e., IC2 fails in the fit-
ted regression). The fitted OLS regression coefficient on z will be non-zero
and may be very large, even asymptotically, despite that fact that the true
causal impact of z on y is zero by construction. The OLS estimated coeffi-
cient is reliable for predicting y given z, but not for assessing the effects on
148 CHAPTER 9. CAUSAL ESTIMATION

y of treatments in z.

9.2.2 Misspecified Functional Form

Note that, from a series expansion perspective, misspecified functional form

corresponds to a form of omitted variables. For example, suppose that the
true y = f (x) + ε relationship is quadratic, but you use a linear specification.
Then x2 is an omitted variable, correlated with x, resulting in an error term
in the fitted regression that is correlated with the included regressor.

9.2.3 Measurement Error

Suppose as above that the DGP is

y = βx + ε,

with all IC satisfied, but that we can’t measure x accurately. Instead we

measure
xm = x + v.

Think of v as an iid measurement error with variance σv2 . (Assume that it is

also independent of ε.) Sometimes xm is called a “proxy variable”.
Clearly, as σv2 gets larger relative to σx2 , the fitted regression y → xm
is progressively less able to identify the true relationship, as the measured
regressor is polluted by progressively more noise. In the limit as σv2 / σx2 → ∞,
it is impossible, and β̂OLS = 0. In any event, β̂OLS is biased toward zero,
in small as well as large samples. We speak of the “errors in variables
problem”, or measurement-error bias.
So far we have motivated measurement-error bias intuitively. Formally, it
9.2. REASONS FOR FAILURE OF IC2 149

arises from violation of IC2. To see this, note that we have

y = βx + ε
= β(xm − v) + ε
= βxm + (ε − v)
= βxm + ν.

In the fitted regression y → xm , the disturbance ν is clearly negatively

correlated with the included regressor xm .
In more complicated cases involving multiple regression with some vari-
ables measured with error, some measured without error, possibly correlated
measurement errors, etc., things quickly get very complicated. Bias still ex-
ists, but it is difficult to ascertain its direction.

9.2.4 Simultaneity

Suppose that y and x are jointly determined, as for example in simultane-

ous determination of quantity (Q) and price (P ) in market supply-demand
equilibrium. Then we may write

Q = βP + ε,

but note that the ε shocks affect not only Q but also P . (Any shock to Q
is also a shock to P , since Q and P are determined jointly by supply and
demand!) That is, ε is correlated with P , violating IC2. So if you want
to estimate a demand curve (or a supply curve), simply running a mongrel
regression of Q on P will produce erroneous results. To estimate a demand
curve we need exogenous supply shifters, and to estimate a supply curve we
need exogenous demand shifters.
150 CHAPTER 9. CAUSAL ESTIMATION

9.2.5 Sample Selection

Let us proceed with by example, using a development economics context.

You might be interested in whether adoption of a new fertilizer enhances crop
yield. You can’t just make it available, see who adopts and who doesn’t, and
then regress yield on an adoption dummy,

yield → c, adopt, (9.1)

because those farms that adopted the new fertilizer may have done so because
their characteristics made them particularly likely to benefit from it. That
is, the sample of adopters may be systematic rather than random. This is
known as “sample selection”. Effectively it produces simultaneity (again, a
violation of IC2) since it produces clear reason for yield to cause adopt in
addition to adopt causing yield.

9.3 Confronting Failures of IC2

9.3.1 Controling for Omitted Variables

The remedy for omitted relevant variables is simple in principle: start includ-
ing them!2 Let’s continue with our earlier example. The DGP is

y = βx + ε,

with all IC satisfied, but we incorrectly regress y → z, where corr(x, z) > 0.

We saw that we estimate a positive causal effect of z on y, in large as well as
small samples, even though it’s completely spurious, due to the failure of IC2
in the fitted model. Now consider instead controlling for x as well. In the
OLS regression y → x, z, all IC are satisfied, so z will get a zero coefficient
asymptotically, but x will get a β coefficient, which will be accurate for the
2
In the lingo, the problem is that we failed to “control” for, or include, the omitted variable.
9.3. CONFRONTING FAILURES OF IC2 151

predictive effect of x on y (of course – OLS is always consistent for predictive

effects) and the treatment effect of x on y. (Remember, the predictive and
treatment effects coincide under the IC.)
Of course the recipe “start including omitted variables” is easier said than
done. We simply may never know about various omitted variables, or we
may suspect them but be unable to measure them. In a wage equation, for
example, in addition to the usual regressors like education and experience,
we might want to include “ability” – but how? In any event it’s important
to use all devices available, from simple introspection to formal theory, in an
attempt to assemble an adequate set of controls.

9.3.2 Instrumental Variables

Consider the simple regression

y = βx + ε.

Following standard usage, let us call a regressor x that satisfies IC2 “ex-
ogenous” (satisfying IC2 at least insofar as x is uncorrelated with ε if not
conditionally independent or fully independent), and a regressor x that fails
IC2 “endogenous”.
If x is endogenous it means that IC2 fails, so we need to do something
about it. One solution is to find an acceptable “instrument” for x. An in-
strument inst is a new regressor that is both exogenous (uncorrelated with ε)
and “strong” or “relevant” (highly-correlated with x). IV estimation proceeds
as follows: (0) Find an exogenous and relevant inst, (1) In a first-stage regres-
sion run x → inst and get the fitted values x̂(inst), and (2) in a second-stage
regression run y → x̂(inst). So in the second-stage regression we replace the
endogenous x with our best linear approximation to x based on inst, namely
x̂(inst). This second-stage regression does not violate IC2 – inst is exogenous
so x̂(inst) must be as well.
152 CHAPTER 9. CAUSAL ESTIMATION

In closing this section, we note that just as the prescription “start including
omitted variables” for a specific violation of IC2 (omitted variables) may at
first appear vapid, so too might the general prescription for violations of
IC2, “find a good instrument”. But plausible, if not unambiguously “good”,
instruments can often be found. Economists generally rely on a blend of
introspection and formal economic theory. Economic theory requires many
assumptions, but if the assumptions are plausible, then theory can be used
to suggest plausible instruments.

9.3.3 Randomized Controlled Trials (RCT’s)

Randomized controlled trials (RCT’s), or randomized experiments, are the

gold standard in terms of assessing causal effects. The basic idea is to ran-
domly select some people into a treatment group, and some into a non-
treatment group, and to estimate the mean difference (the “average treatment
effect.”)
Recall our development economics example. You might be interested in
whether adoption of a new fertilizer enhances crop yield. You can’t just
introduce it, see who adopts and who doesn’t, and then regress yield on an
adoption dummy,
yield → c, adopt, (9.2)

because those farms that adopted the new fertilizer may have done so because
their characteristics made them particularly likely to benefit from it. Instead
you’d need an instrument for adoption.
An RCT effectively creates an instrument for adopt in regression (9.2), by
randomizing. You randomly select some farms for adoption (treatment) and
some not (control), and you inspect the difference between yields for the two
groups. More formally, for the firms in the experiment you could run

yield → c, treatment. (9.3)

9.3. CONFRONTING FAILURES OF IC2 153

The OLS estimate of c is the mean yield for the non-adoption (control) group,
and the OLS estimate of the coefficient on the treatment dummy is the mean
enhancement from adoption (treatment). You can test its significance with
the usual t test. The randomization guarantees that the regressor in (9.3) is
exogenous, so that IC2 is satisfied.
The key insight bears repeating: RCT’s, if successfully implemented, guar-
antee that IC2 is satisfied. But of course there’s no free lunch, and there
are various issues and potential problems with “successful implementation of
an RCT” in all but the simplest cases (like the example above), just as there
are many issues and potential problems with “finding a strong and exogenous
instrument”.

9.3.4 Regression Discontinuity Designs (RDD’s)

RCT’s can be expensive and wasteful when estimating the efficacy of a treat-
ment, as many people who don’t need treatment will be randomly assigned
treatment anyway. Hence alternative experimental designs are often enter-
tained. A leading example is the “regression discontinuity design” (RDD).
To understand the RDD, consider a famous scholarship example. You want
to know whether receipt of an academic scholarship causes enhanced aca-
demic performance among top academic performers. You can’t just regress
academic performance on a scholarship receipt dummy, because recipients
are likely to be stong academic performers even without the scholarship.
The question is whether scholarship receipt causes enhanced performance for
already-strong performers.
You could do an RCT, but in an RCT approach you’re going to give lots
of academic scholarships to lots of randomly-selected people, many of whom
are not strong performers. That’s wasteful.
An RDD design is an attractive alternative. You give scholarships only to
those who score above some threshold in the scholarship exam, and compare
154 CHAPTER 9. CAUSAL ESTIMATION

the performances of students who scored just above and below the threshold.
In the RDD you don’t give any scholarships to weak performers, so it’s not
wasteful.
Notice how the RDD effectively attempts to approximate an RCT. People
just above and below the scholarship threshold are basically the same in terms
of academic talent – the only difference is that one group gets the scholarship
and one doesn’t. The RCT does the controlled experiment directly; it’s
statistically efficient but can be wasteful. The RDD does the controlled
experiment indirectly; it’s statistically less efficient but also less wasteful.

9.3.5 Propensity-Score Matching

In the scholarship exam example, you could do an RCT, but, as we discussed,

it’s not attractive for certain reasons. Sometimes it’s even worse – an RCT
is simply infeasible. Consider, for example, estimating the causal effect of
college education on subsequent earnings. As usual, we don’t just want to
compare earnings of college grads and non college grads (regress earnings on a
college dummy), because college grads may earn more for many reasons other
than college – perhaps higher intelligence, more supportive family, etc. We
need to control for such things. An RCT works in principle but is infeasible in
practice – you’d have to randomly select a large group of high school students,
randomly send part to college, and prohibit the rest from attending college,
and follow their outcomes for decades.
The propensity score approach attempts to approximate an RCT by esti-
mating a logit or similar model for the probability of attending college. The
idea is to compare people with similar propensity scores, some of whom went
to college and some of whom didn’t, to control for everything other than
college vs. non-college. Causal effect estimation based on propensity scores
is one example of the general idea of matching estimation.
9.3. CONFRONTING FAILURES OF IC2 155

9.3.6 Differences in Differences (“Diff in Diff”) and Panel Data

The “differences in differences” estimation design, also known as diff-in-diff,

or simply DD, has become a standard tool in the applied microeconometric
estimation of causal effects. A typical application of DD designs in economics
is to identify the effects of policies on outcomes using a “natural experiment,”
which refers to a variation in some treatment variable that affects only some
units over time and that occurs naturally, as opposed to being made explicitly
to measure the effects of the policies.
To understand the DD design, consider a famous minimum wage example.
In competitive labor markets, an increase in the minimum wage would reduce
employment because that would move the equilibrium point up a downward-
sloping demand curve. However, in monopsony labor markets an increase in
the minimum wage can lead to an increase in employment. Therefore, the
effects of minimum wages on employment are an empirical question depending
on local characteristics of the industry under study. David Card and Alan
Krueger, two labor economists, studied this question in 1994 in the context
of fast-food restaurant employment. As a natural experiment they used the
minimum wage increase that happened in New Jersey (NJ) in November 1992
but not in neighboring Pennsylvania (PA).
Their idea was to isolate the effect of the increase in minimum wage by
comparing the two states’ employments in a sample of fast-food restaurants
before and after the policy was enacted. One may be tempted to simply
compare average employment in New Jersey before and after the policy, but
such a difference will not measure the effect of the policy. This is because we
cannot distinguish between the policy effect and other simultaneous changes;
at least not without the strong assumption that in absence of the policy there
would be no change in average employment. In the DD design the estimated
change in average PA employment plays the role of control group estimate
for the change in average NJ employment.
156 CHAPTER 9. CAUSAL ESTIMATION

To fix ideas let’s introduce some notation. Let average employment be

represented by Y and let D be a binary variable that is equal to one if a
higher minimum wage is imposed. Our goal is to estimate D’s causal effect
on employment, Y . Moreover, let T be a dummy variable representing trend
effects before and after the policy enactment. Hence the employment level in
NJ before the policy is Y = αN J and after the policy is Y = αN J + T + D.
Similarly, the employment level in PA before the policy is Y = αP A and after
the policy is Y = αP A + T .3
To see why DD deserves its name, note that the difference in PA em-
ployment before and after the policy is T , whereas the difference in NJ em-
ployment before and after the policy is T + D. Hence the difference in the
differences is just D, the effect we want to isolate. Notice the parallel trend
assumption, namely that, absent the policy, the average difference on out-
comes is the same for treatment and controls, as illustrated in Figure 9.1 for
the minimum wage example.

9.4 Internal and External Validity

9.4.1 Internal Validity and its Problems

Successfully-implemented RCT’s are generally “internally valid” for some-

thing; that is, they produce credible estimates of the treatment effect for the
precise experiment performed and situation studied. But lots of things can
go wrong, casting doubt on internal validity. A short list, with many items
inter-related, includes:

1. Is the randomization really credible?

2. Are the treatment and control sample sizes large enough?

3. Are there placebo effects?

3
αN J and αP A denote NJ and PA state employment fixed effects.
9.4. INTERNAL AND EXTERNAL VALIDITY 157

Figure 9.1: Example of Difference in Difference Parallel Assumption

4. Who among the participants knows what about who is treated and who
is not, and related, are the behaviors of the groups evolving over time,
perhaps due to interaction with each other (“spillovers”)?

5. Are people entering and/or leaving the study at different times? If so,
when and why?

6. Might experimenters coddle the treatment group in various ways in at-

tempts to raise the likelihood of “significant” effects of their pet treat-
ments?

7. Might “negative” or “insignificant” results be discarded, and hence only

positive and significant results published? (This is the “file-drawer prob-
lem,” referring to insignificant results that are left unpublished in re-
searchers’ file drawers, or on their hard drives.)
158 CHAPTER 9. CAUSAL ESTIMATION

9.4.2 External Validity

Even if an RCT study is internally valid, its results may not generalize to
other populations or situations. That is, even if internally valid, it may not
be externally valid.
Consider, for example, a study of the effects of fertilizer on crop yield done
for region X in India during a heat wave. Even if successfully randomized,
and hence internally valid, the estimated treatment effect is for the effects
of fertilizer on crop yield in region X during a heat wave. The results do
not necessarily generalize – and in this example surely do not generalize –
to times of “normal” weather, even in region X. And of course, for a variety
of reasons, they may not generalize to regions other than X, even in a heat
wave.
Hence, even if an RCT is internally valid, there is no guarantee that it
is externally valid, or “extensible”. That is, there is no guarantee that its
results will hold in other cross sections and/or time periods.

9.4.3 A Final Thought

A good thing about RCTs compared to other causal estimation methods is

that they can be repeated at the same location to assess internal validity, and
repeated elsewhere to assess external validity.

9.5 Exercises, Problems and Complements

1. Omitted variable bias.

How might you assess whether a regression suffers from omitted variable
bias?

2. Included irrelevant variables.

9.5. EXERCISES, PROBLEMS AND COMPLEMENTS 159

Another violation of the full ideal conditions is inclusion of irrelevant

variables. Fortunately the effects are minor; some degrees of freedom
are wasted, but otherwise there’s no problem. How would you assess
whether an included variable in a regression is irrelevant? Whether a
set of included variables is irrelevant?

3. Panels vs. time series of cross sections.

The two are not the same. Why?

4. Panel models.
Our discussion of “differences-in-differences” designs, which involve com-
parisons over both time and space, raises the general issue of panel data
models.
Consider, for example, the bivariate panel regression case. (Extension
to multiple panel regression is immediate but notationally tedious.) The
data are (yit , xit ) (for “person” i at time t), i = 1, ..., N (cross section
dimension), t = 1, ..., T (time series dimension).
In panels we have N × T “observations,” or data points, which is a lot of
data. In a pure cross section we have just i = 1, ..., N observations, so we
could never allow for different intercepts across all people (“individual
effects”), because there are N people, and we have only N observations,
so we’d run out of degrees of freedom. Similarly, in a pure time series we
have just t = 1, ..., T observations, so we could never allow for different
intercepts (say) across all time periods (“time effects”), because there
are T time periods, and we have only T observations, so we’d run out
of degrees of freedom. But with panel data, allowing for such individual
and time effects is possible. There are N + T coefficients to be estimated
(N individual effects and T time effects), but we have N ×T observations!

5. Estimating panel models with fixed individual effects.

160 CHAPTER 9. CAUSAL ESTIMATION

Let us consider a situation of unobserved heterogeneity corresponding

to fixed individual effects. We write

yit = αi + βxit + εit .

Fixed individual effects correspond to a different intercept (αi ) for each

person. The estimation strategy, in principle, is to run

yit → I(i = 1), I(i = 2), I(i = 3), ..., I(i = N ), xit .

Note that this regression is impossible in pure cross sections of size N ,

in which case one could at best consider a “middle ground” in which
individuals are grouped into ui < N units. In the panel case the regres-
sion can be run in principle, and it has the benefit of allowing one to
examine the individual effects (αi ’s) and the common effect (β), but the
cost of potentially significant numerical/computational difficulty unless
N is small. In panels of typical size, the regression is infeasible and
rarely attempted.

6. “Differencing out” fixed effects.

How then do we really estimate the panel regression allowing for fixed
individual effects? We eliminate the αi , which are often viewed as un-
interesting “nuisance parameters”, by writing the model in deviations
from means,

(yit − yi ) = (αi − αi ) + β (xit − xi ) + (εit − εi ) ,

T T
1 1
P P
where xi = T xit and εi = T εit . Because αi is constant, αi = αi ,
t=1 t=1
so (αi − αi ) = 0. Hence we simply run

(yit − yi ) → (xit − xi ) ,
9.5. EXERCISES, PROBLEMS AND COMPLEMENTS 161

which delivers an estimate of the key common effect β, while allowing

for individual fixed effects, even if it does not deliver estimates of the
individual effects.

7. Tradeoffs between instrument exogeneity and relevance.

We want instruments that are both exogenous (uncorrelated with ε)
and “strong” or “relevant” (highly-correlated with x). There is a trade-
off. For example, an exogenous but weakly-relevant instrument might
nevertheless be valuable, as might a relevant but “slightly-endogenous”
instrument. The former instrument may produce an IV estimator that
is consistent but high-variance, whereas the latter may produce an esti-
mator that is (slightly) inconsistent but low-variance.
162 CHAPTER 9. CAUSAL ESTIMATION
 Part III

Time Series

163
Chapter 10

Trend and Seasonality

The time series that we want to model vary over time, and we often men-
tally attribute that variation to unobserved underlying components related
to trend and seasonality.

10.1 Linear Trend

Trend involves slow, long-run, evolution in the variables that we want to

model and forecast. In business, finance, and economics, for example, trend
is produced by slowly evolving preferences, technologies, institutions, and
demographics. We’ll focus here on models of deterministic trend, in which
the trend evolves in a perfectly predictable way. Deterministic trend models
are tremendously useful in practice.
Linear trend is a simple linear function of time,

T rendt = β1 + β2 T IM Et .

The indicator variable T IM E is constructed artificially and is called a “time

trend”, or “time indicator”, or “time dummy.” T IM E equals 1 in the first
period of the sample, 2 in the second period, and so on. Thus, for a sample
of size T , T IM E = (1, 2, 3, ..., T − 1, T ). Put differently, T IM Et = t, so that
the TIME variable simply indicates the time. β1 is the intercept; it’s the

165
166 CHAPTER 10. TREND AND SEASONALITY

Figure 10.1: Various Linear Trends

value of the trend at time t=0. β2 is the slope; it’s positive if the trend is
increasing and negative if the trend is decreasing. The larger the absolute
value of β1 , the steeper the trend’s slope. In Figure 10.1, for example, we show
two linear trends, one increasing and one decreasing. The increasing trend
has an intercept of β1 = −50 and a slope of β2 = .8, whereas the decreasing
trend has an intercept of β1 = 10 and a gentler absolute slope of β2 = −.25.
In business, finance, and economics, linear trends are typically increasing,
corresponding to growth, but they don’t have to increase. In recent decades,
for example, male labor force participation rates have been falling, as have
the times between trades on stock exchanges. Morover, in some cases, such
as records (e.g., world records in the marathon), trends are decreasing by
definition.
Estimation of a linear trend model (for a series y, say) is easy. First we
need to create and store on the computer the variable T IM E. Fortunately we
don’t have to type the T IM E values (1, 2, 3, 4, ...) in by hand; in most good
software environments, a command exists to create the trend automatically.
Then we simply run the least squares regression y → c, T IM E.
10.2. NON-LINEAR TREND 167

10.2 Non-Linear Trend

Nonlinearity can be important in time series just as in cross sections, but

there is a special case of key importance in time series: nonlinear trend. Here
we introduce it.

10.2.1 Quadratic Trend

Sometimes trend appears non-linear, or curved, as for example when a vari-

able increases at an increasing or decreasing rate. Ultimately, we don’t require
that trends be linear, only that they be smooth.
We can allow for gentle curvature by including not only T IM E, but also
T IM E 2 ,
T rendt = β1 + β2 T IM Et + β3 T IM Et2 .

This is called quadratic trend, because the trend is a quadratic function of

T IM E.1 Linear trend emerges as a special (and potentially restrictive) case
when β3 = 0.
A variety of different non-linear quadratic trend shapes are possible, de-
pending on the signs and sizes of the coefficients; we show several in Figure
10.2. In particular, if β2 > 0 and β3 > 0 as in the upper-left panel, the trend is
monotonically, but non-linearly, increasing, Conversely, if β2 < 0 and β3 < 0,
the trend is monotonically decreasing. If β2 < 0 and β3 > 0 the trend has a
U shape, and if β2 > 0 and β3 < 0 the trend has an inverted U shape. Keep
in mind that quadratic trends are used to provide local approximations; one
rarely has a “U-shaped” trend, for example. Instead, all of the data may lie
on one or the other side of the “U”.
Estimating quadratic trend models is no harder than estimating linear
trend models. We first create T IM E and its square; call it T IM E2, where
1
Higher-order polynomial trends are sometimes entertained, but it’s important to use low-order poly-
nomials to maintain smoothness.
168 CHAPTER 10. TREND AND SEASONALITY

Figure 10.2: Various Quadratic Trends

T IM E2t = T IM Et2 . Because T IM E = (1, 2, ..., T ), T IM E2 = (1, 4, ..., T 2 ).

Then we simply run the least squares regression y → c, T IM E, T IM E2.
Note in particular that although the quadratic is a non-linear function, it is
linear in the variables T IM E and T IM E2.

10.2.2 Exponential Trend

The insight that exponential growth is non-linear in levels but linear in log-
arithms takes us to the idea of exponential trend, or log-linear trend,
which is very common in business, finance and economics.2
Exponential trend is common because economic variables often display
roughly constant real growth rates (e.g., two percent per year). If trend is
2
Throughout this book, logarithms are natural (base e) logarithms.
10.2. NON-LINEAR TREND 169

characterized by constant growth at rate β2 , then we can write

T rendt = β1 eβ2 T IM Et .

The trend is a non-linear (exponential) function of time in levels, but in

logarithms we have

ln(T rendt ) = ln(β1 ) + β2 T IM Et . (10.1)

Thus, ln(T rendt ) is a linear function of time.

In Figure 10.3 we show the variety of exponential trend shapes that can
be obtained depending on the parameters. Depending on the signs and sizes
of the parameter values, exponential trend can achieve a variety of patterns,
increasing or decreasing at increasing or decreasing rates.
Although the exponential trend model is non-linear, we can estimate it by
simple least squares regression, because it is linear in logs. We simply run the
least squares regression, ln y → c, T IM E. Note that because the intercept
in equation (10.1) is not β1 , but rather ln(β1 ), we need to exponentiate the
estimated intercept to get an estimate of β1 . Similarly, the fitted values from
this regression are the fitted values of ln y, so they must be exponentiated to
get the fitted values of y. This is necessary, for example, for appropriately
comparing fitted values or residuals (or statistics based on residuals, like AIC
and SIC) from estimated exponential trend models to those from other trend
models.
It’s important to note that, although the same sorts of qualitative trend
shapes can sometimes be achieved with quadratic and exponential trend,
there are subtle differences between them. The non-linear trends in some se-
ries are well approximated by quadratic trend, while the trends in other series
are better approximated by exponential trend. Ultimately it’s an empirical
matter as to which is best in any particular application.
170 CHAPTER 10. TREND AND SEASONALITY

Figure 10.3: Various Exponential Trends

10.2.3 Non-Linearity in Liquor Sales Trend

We already fit a non-linear (exponential) trend to liquor sales, when we fit a

linear trend to log liquor sales. But it still didn’t fit so well.
We now examine quadratic trend model (again in logs). The log-quadratic
trend estimation results appear in Figure 10.4. Both T IM E and T IM E2
are highly significant. The adjusted R2 for the log-quadratic trend model
is 90%, higher than for the the log-linear trend model. As with the log-
linear trend model, the Durbin-Watson statistic provides no evidence against
the hypothesis that the regression disturbance is white noise. The residual
plot (Figure 10.5) shows that the fitted quadratic trend appears adequate,
and that it increases at a decreasing rate. The residual plot also continues
10.2. NON-LINEAR TREND 171

Figure 10.4: Log-Quadratic Trend Estimation

to indicate obvious residual seasonality. (Why does the Durbin-Watson not

detect it?)
In Figure 10.6 we show the results of regression on quadratic trend and
a full set of seasonal dummies. The trend remains highly significant, and
the coefficients on the seasonal dummies vary significantly. The adjusted R2
rises to 99%. The Durbin-Watson statistic, moreover, has greater ability to
detect residual serial correlation now that we have accounted for seasonal-
ity, and it sounds a loud alarm. The residual plot of Figure 10.7 shows no
seasonality, as the model now accounts for seasonality, but it confirms the
Durbin-Watson statistic’s warning of serial correlation. The residuals appear
highly persistent.
There remains one model as yet unexplored, exponential trend fit to
LSALES. We do it by N LS (why?) and present the results in Figure ***.
Among the linear, quadratic and exponential trend models for LSALES,
both SIC and AIC clearly favor the quadratic.
172 CHAPTER 10. TREND AND SEASONALITY

Figure 10.5: Residual Plot, Log-Quadratic Trend Estimation

10.3 Seasonality

In the last section we focused on the trends; now we’ll focus on seasonal-
ity. A seasonal pattern is one that repeats itself every year.3 The annual
seasonal repetition can be exact, in which case we speak of deterministic
seasonality Here we focus exclusively on deterministic seasonality models.
Seasonality arises from links of technologies, preferences and institutions
to the calendar. The weather (e.g., daily high temperature) is a trivial but
very important seasonal series, as it’s always hotter in the summer than in
the winter. Any technology that involves the weather, such as production of
agricultural commodities, is likely to be seasonal as well.
Preferences may also be linked to the calendar. Consider, for example,
gasoline sales. People want to do more vacation travel in the summer, which
tends to increase both the price and quantity of summertime gasoline sales,
both of which feed into higher current-dollar sales.
3
Note therefore that seasonality is impossible, and therefore not an issue, in data recorded once per year,
or less often than once per year.
10.3. SEASONALITY 173

Figure 10.6: Liquor Sales Log-Quadratic Trend Estimation with Seasonal Dummies

Finally, social institutions that are linked to the calendar, such as holidays,
are responsible for seasonal variation in a variety of series. In Western coun-
tries, for example, sales of retail goods skyrocket every December, Christmas
season. In contrast, sales of durable goods fall in December, as Christmas
purchases tend to be nondurables. (You don’t buy someone a refrigerator for
Christmas.)
You might imagine that, although certain series are seasonal for the rea-
sons described above, seasonality is nevertheless uncommon. On the con-
trary, and perhaps surprisingly, seasonality is pervasive in business and eco-
nomics. Many industrialized economies, for example, expand briskly every
fourth quarter and contract every first quarter.
174 CHAPTER 10. TREND AND SEASONALITY

Figure 10.7: Residual Plot, Liquor Sales Log-Quadratic Trend Estimation With Seasonal
Dummies

10.3.1 Seasonal Dummies

A key technique for modeling seasonality is regression on seasonal dum-

mies. Let S be the number of seasons in a year. Normally we’d think of four
seasons in a year, but that notion is too restrictive for our purposes. Instead,
think of S as the number of observations on a series in each year. Thus S = 4
if we have quarterly data, S = 12 if we have monthly data, and so on.
The pure seasonal dummy model is
S
X
Seasonalt = γs SEASts
s=1

1 if observation t falls into s
where SEASts =
0 otherwise

The SEASts variables are called seasonal dummy variables. They

simply indicate which season we’re in.
Operationalizing the model is simple. Suppose, for example, that we have
10.3. SEASONALITY 175

quarterly data, so that S = 4. Then we create four variables4 :

SEAS1 = (1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, ..., 0)′

SEAS2 = (0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, ..., 0)′
SEAS3 = (0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, ..., 0)′
SEAS4 = (0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, ..., 1)′ .

SEAS1 indicates whether we’re in the first quarter (it’s 1 in the first quarter
and zero otherwise), SEAS2 indicates whether we’re in the second quarter
(it’s 1 in the second quarter and zero otherwise), and so on. At any given
time, we can be in only one of the four quarters, so one seasonal dummy is
1, and all others are zero.
To estimate the model for a series y, we simply run the least squares
regression,
y → SEAS1 , ..., SEASS .

Effectively, we’re just regressing on an intercept, but we allow for a different

intercept in each season. Those different intercepts (that is γs ’s) are called the
seasonal factors; they summarize the seasonal pattern over the year, and we
often may want to examine them and plot them. In the absence of seasonality,
those intercepts are all the same, so we can drop all the seasonal dummies
and instead simply include an intercept in the usual way.
In time-series contexts it’s often most natural to include a full set of sea-
sonal dummies, without an intercept. But of course we could instead include
any S − 1 seasonal dummies and an intercept. Then the constant term is the
intercept for the omitted season, and the coefficients on the seasonal dum-
mies give the seasonal increase or decrease relative to the omitted season. In
no case, however, should we include S seasonal dummies and an intercept.
Including an intercept is equivalent to including a variable in the regression
4
For illustrative purposes, assume that the data sample begins in Q1 and ends in Q4.
176 CHAPTER 10. TREND AND SEASONALITY

whose value is always one, but note that the full set of S seasonal dummies
sums to a variable whose value is always one, so it is completely redundant.
Trend may be included as well. For example, we can account for season-
ality and linear trend by running5

y → T IM E, SEAS1 , ..., SEASS .

In fact, you can think of what we’re doing in this section as a generalization
of what we did in the last, in which we focused exclusively on trend. We still
want to account for trend, if it’s present, but we want to expand the model
so that we can account for seasonality as well.

10.3.2 More General Calendar Effects

The idea of seasonality may be extended to allow for more general calendar
effects. “Standard” seasonality is just one type of calendar effect. Two
additional important calendar effects are holiday variation and trading-
day variation.
Holiday variation refers to the fact that some holidays’ dates change over
time. That is, although they arrive at approximately the same time each year,
the exact dates differ. Easter is a common example. Because the behavior
of many series, such as sales, shipments, inventories, hours worked, and so
on, depends in part on the timing of such holidays, we may want to keep
track of them in our forecasting models. As with seasonality, holiday effects
may be handled with dummy variables. In a monthly model, for example,
in addition to a full set of seasonal dummies, we might include an “Easter
dummy,” which is 1 if the month contains Easter and 0 otherwise.
Trading-day variation refers to the fact that different months contain dif-
ferent numbers of trading days or business days, which is an important con-
sideration when modeling and forecasting certain series. For example, in a
5
Note well that we drop the intercept!
10.4. TREND AND SEASONALITY IN LIQUOR SALES 177

Figure 10.8: Liquor Sales

monthly forecasting model of volume traded on the London Stock Exchange,

in addition to a full set of seasonal dummies, we might include a trading day
variable, whose value each month is the number of trading days that month.
More generally, you can model any type of calendar effect that may arise,
by constructing and including one or more appropriate dummy variables.

10.4 Trend and Seasonality in Liquor Sales

We’ll illustrate trend and seasonal modeling with an application to liquor

sales. The data are measured monthly.
We show the time series of liquor sales in Figure 10.8, which displays
clear trend (sales are increasing) and seasonality (sales skyrocket during the
Christmas season, among other things).
We show log liquor sales in Figure 10.9 ; we take logs to stabilize the
variance, which grows over time.6 Log liquor sales has a more stable variance,
and it’s the series for which we’ll build models.7
6
The nature of the logarithmic transformation is such that it “compresses” an increasing variance. Make
a graph of log(x) as a function of x, and you’ll see why.
7
From this point onward, for brevity we’ll simply refer to “liquor sales,” but remember that we’ve taken
logs.
178 CHAPTER 10. TREND AND SEASONALITY

Figure 10.9: Log Liquor Sales

Linear trend estimation results appear in Table 10.10. The trend is in-
creasing and highly significant. The adjusted R2 is 84%, reflecting the fact
that trend is responsible for a large part of the variation in liquor sales.
The residual plot (Figure 10.11) suggests, however, that linear trend is
inadequate. Instead, the trend in log liquor sales appears nonlinear, and the
neglected nonlinearity gets dumped in the residual. (We’ll introduce nonlin-
ear trend later.) The residual plot also reveals obvious residual seasonality.
In Figure 10.12 we show estimation results for a model with linear trend
and seasonal dummies. All seasonal dummies are of course highly significant
(no month has average sales of 0), and importantly the various seasonal
coefficients in many cases are significantly different from each other (that’s
the seasonality). R2 is higher.
In Figure 10.13 we show the corresponding residual plot. The model now
picks up much of the seasonality, as reflected in the seasonal fitted series
and the non-seasonal residuals. However, it clearly misses nonlinearity in the
trend, which therefore appears in the residuals.
In Figure 10.14 we plot the estimated seasonal pattern (the set of 12
estimated seasonal coefficients), which peaks during the winter holidays.
10.5. EXERCISES, PROBLEMS AND COMPLEMENTS 179

Figure 10.10: Linear Trend Estimation

All of these results are crude approximations, because the linear trend
is clearly inadequate. We will subsequently allow for more sophisticated
(nonlinear) trends.

10.5 Exercises, Problems and Complements

1. (Mechanics of trend estimation and detrending)

Obtain from the web a quarterly time series of U.S. real GDP in levels,
spanning the last forty years, and ending in Q4.

a. Produce a time series plot and discuss.

b. Fit a linear trend. Discuss both the estimation results and the residual
plot.
c. Is there any evidence of seasonality in the residuals? Why or why
not?
180 CHAPTER 10. TREND AND SEASONALITY

Figure 10.11: Residual Plot, Linear Trend Estimation

d. The residuals from your fitted model are effectively a linearly de-
trended version of your original series. Why? Discuss.

2. (Using model selection criteria to select a trend model)

You are tracking and forecasting the earnings of a new company develop-
ing and applying proprietary nano-technology. The earnings are trend-
ing upward. You fit linear, quadratic, and exponential trend models,
yielding sums of squared residuals of 4352, 2791, and 2749, respectively.
Which trend model would you select, and why?

3. (Seasonal adjustment)
Just as we sometimes want to remove the trend from a series, sometimes
we want to seasonally adjust a series before modeling it. Seasonal
adjustment may be done using a variety of methods.

a. Discuss in detail how you’d use a linear trend plus seasonal dummies
model to seasonally adjust a series.
10.5. EXERCISES, PROBLEMS AND COMPLEMENTS 181

Figure 10.12: Estimation Results, Linear Trend with Seasonal Dummies

b. Seasonally adjust the log liquor sales data using a linear trend plus
seasonal dummy model. Discuss the patterns present and absent from
the seasonally adjusted series.
c. Search the Web (or the library) for information on the latest U.S.
Census Bureau seasonal adjustment procedure, and report what you
learned.

4. (Handling sophisticated calendar effects)

Describe how you would construct a purely seasonal model for the fol-
lowing monthly series. In particular, what dummy variable(s) would you
use to capture the relevant effects?

a. A sporting goods store suspects that detrended monthly sales are

182 CHAPTER 10. TREND AND SEASONALITY

Figure 10.13: Residual Plot, Linear Trend with Seasonal Dummies

roughly the same for each month in a given three-month season. For
example, sales are similar in the winter months of January, February
and March, in the spring months of April, May and June, and so on.
b. A campus bookstore suspects that detrended sales are roughly the
same for all first, all second, all third, and all fourth months of
each trimester. For example, sales are similar in January, May, and
September, the first months of the first, second, and third trimesters,
respectively.
c. (Trading-day effects) A financial-markets trader suspects that de-
trended trading volume depends on the number of trading days in
the month, which differs across months.
d. (Time-varying holiday effects) A candy manufacturer suspects that
detrended candy sales tend to rise at Easter.

5. (Testing for seasonality)

10.5. EXERCISES, PROBLEMS AND COMPLEMENTS 183

Figure 10.14: Seasonal Pattern

Using the log liquor sales data:

a. As in the chapter, construct and estimate a model with a full set of

seasonal dummies.
b. Test the hypothesis of no seasonal variation. Discuss.
c. Test for the equality of the January through April seasonal factors.
Discuss.
d. Test for equality of the May through November seasonal factors. Dis-
cuss.
e. Estimate a suitable “pruned” model with fewer than twelve seasonal
dummies that nevertheless adequately captures the seasonal pattern.

6. Specifying and testing nonlinear trend models.

In 1965, Intel co-founder Gordon Moore predicted that the number of
transistors that one could place on a square-inch integrated circuit would
double every twelve months.

a. What sort of trend is this?

184 CHAPTER 10. TREND AND SEASONALITY

b. Given a monthly series containing the number of transistors per square

inch for the latest integrated circuit, how would you test Moore’s pre-
diction? How would you test the currently accepted form of “Moore’s
Law,” namely that the number of transistors actually doubles every
eighteen months?

7. (Properties of polynomial trends)

Consider a sixth-order deterministic polynomial trend:

Tt = β1 + β2 T IM Et + β3 T IM Et2 + ... + β7 T IM Et6 .

a. How many local maxima or minima may such a trend display?

b. Plot the trend for various values of the parameters to reveal some of
the different possible trend shapes.
c. Is this an attractive trend model in general? Why or why not?
d. Fit the sixth-order polynomial trend model to a trending series that
interests you, and discuss your results.

8. (Selecting non-linear trend models)

Using AIC and SIC, perform a detailed comparison of polynomial vs.
exponential trend in LSALES. Do you agree with our use of quadratic
trend in the text?

9. (Difficulties with non-linear optimization)

Non-linear optimization can be a tricky business, fraught with problems.
Some problems are generic. It’s relatively easy to find a local optimum,
for example, but much harder to be confident that the local optimum
is global. Simple checks such as trying a variety of startup values and
checking the optimum to which convergence occurs are used routinely,
but the problem nevertheless remains. Other problems may be software
10.5. EXERCISES, PROBLEMS AND COMPLEMENTS 185

specific. For example, some software may use highly accurate analytic
derivatives whereas other software uses approximate numerical deriva-
tives. Even the same software package may change algorithms or details
of implementation across versions, leading to different results.

10. (Direct estimation of exponential trend in levels)

We can estimate an exponential trend in two ways. First, as we have
emphasized, we can take logs and then use OLS to fit a linear trend.
Alternatively we can use NLS, proceeding directly from the exponential
representation and letting the computer find
T
X 2
yt − β1 eβ2 T IM Et

(β̂1 , β̂2 ) = argminβ1 ,β2 .
t=1

a. The NLS approach is more tedious? Why?

b. The NLS approach is less thoroughly numerically trustworthy? Why?
c. Nevertheless the NLS approach can be very useful? Why? (Hint:
Consider comparing SIC values for quadratic vs. exponential trend.)

11. (Logistic trend)

In the main text we introduced the logistic functional form. A key
example is logistic trend, which is
1
T rendt = ,
a + brT IM Et
with 0<r<1.

a. Graph the trend shape for various combinations of a and b values.

When might such a trend shape be useful?
b. Can you think of other specialized situations in which other special-
ized trend shapes might be useful? Produce mathematical formulas
for the additional specialized trend shapes you suggest.
186 CHAPTER 10. TREND AND SEASONALITY

12. (Modeling Liquor Sales Trend and Seasonality)

Consider the liquor sales data. Never include an intercept. Discuss all
results in detail.

(a) Fit a linear trend plus seasonal dummy model to log liquor sales
(LSALES), using a full set of seasonal dummies.
(b) Find a “best” linear trend plus seasonal dummy LSALES model.
That is, consider tightening the seasonal specification to include
fewer than 12 seasonal dummies, and decide what’s best.
(c) Keeping the same seasonality specification as in (12b), re-estimate
the model in levels (that is, the LHS variable is now SALES rather
than LSALES) using exponential trend and nonlinear least squares.
Do your coefficient estimates match those from (12b)? Does the SIC
match that from (12b)?
(d) Repeat (12c), again using SALES and again leaving intact your
seasonal specification from (12b), but try linear and quadratic trend
instead of the exponential trend in (12c). What is your “final”
SALES model?
(e) Critique your final SALES model from (12d). In what ways is it
likely still deficient? You will of course want to discuss its residual
plot (actual values, fitted values, residuals), as well as any other
diagnostic plots or statistics that you deem relevant.
(f) Take your final estimated SALES model from (12d), and include
as regressors three lags of SALES (i.e., SALESt−1 , SALESt−2 and
SALESt−3 ). What role do the lags of SALES play? Consider this
new model to be your “final, final” SALES model, and repeat (12e).

13. Moving-Average Trend and De-Trending

10.5. EXERCISES, PROBLEMS AND COMPLEMENTS 187

The trend regression technique is one way to estimate trend. An addi-

tional way involves model-free smoothing techniques. A leading case
is “moving-average smoothing”. We’ll focus on three moving-average
smoothers: two-sided moving averages, one-sided moving averages, and
one-sided weighted moving averages. Denote the original data by {yt }Tt=1
and the smoothed data by {zt }Tt=1 . Then the two-sided moving aver-
age is
m
−1
X
zt = (2m + 1) yt−i ,
i=−m

the one-sided moving average is

m
−1
X
zt = (m + 1) yt−i ,
i=0

and the one-sided weighted moving average is

m
X
zt = wi yt−i ,
i=0

where the wi are weights and m is an integer chosen by the user. The
“standard” one-sided moving average corresponds to a one-sided weighted
moving average with all weights equal to (m + 1)−1 .

a. For each of the smoothing techniques, discuss the role played by m.

What happens as m gets very large? Very small? In what sense does
m play a role similar to p, the order of a polynomial trend?
b. If the original data runs from time 1 to time T , over what range
can smoothed values be produced using each of the three smoothing
methods? What are the implications for “real-time” smoothing or
“on-line” smoothing versus “ex post” smoothing or “off-line”
smoothing?

14. Hodrick-Prescott Trend and De-Trending

188 CHAPTER 10. TREND AND SEASONALITY

Another model-free approach to trend fitting and de-trending is known

as Hodrick-Prescott filtering. The “HP trend” solves:
T
X T −1
X
min
T
2
(yt − st ) + λ ((st+1 − st ) − (st − st−1 ))2
{st }t=1
t=1 t=2

a. λ is often called the “penalty parameter.” What does λ govern?

b. What happens as λ → 0?
c. What happens as λ → ∞?
d. People routinely use bigger λ for higher-frequency data. Why? (Com-
mon values are λ = 100, 1600 and 14,400 for annual, quarterly, and
monthly data, respectively.)

15. Regime Switching I: Observed-Regime Threshold Model


(u)


 c(u) + ϕ(u) yt−1 + εt , θ(u) < yt−d





(m)
yt = c(m) + ϕ(m) yt−1 + εt , θ(l) < yt−d < θ(u)





 (l)
c(l) + ϕ(l) yt−1 + εt , θ(l) > yt−d



16. Regime Switching II: Markov-Switching Model

Regime governed by latent 2-state Markov process:
!
p00 1 − p00
M=
1 − p11 p11

Switching mean:
!
1 −(yt − µst )2
f (yt |st ) = √ exp .
2πσ 2σ 2
10.5. EXERCISES, PROBLEMS AND COMPLEMENTS 189

Switching regression:
!
1 −(yt − x′t βst )2
f (yt |st ) = √ exp .
2πσ 2σ 2
190 CHAPTER 10. TREND AND SEASONALITY
Chapter 11

Serial Correlation

In this chapter we consider serially correlated regression disturbances, a new

type of violation of the IC of crucial importance in time series. Disturbance
serial correlation, or autocorrelation, means correlation over time. That is,
the current disturbance is correlated with one or more past disturbances.
Under the IC we have:
ε ∼ N (0, σ 2 I).

Now, with serial correlation, we have:

ε ∼ N (0, Ω),

where Ω is not diagonal. A key cause is omission of serially-correlated x’s

in the regression specification, which results in serially-correlated ε. Hence
the “ommitted variables problem” and the “serial correlation problem” are
closely related.
A leading example is “first-order autoregressive” or “AR(1)” disturbance
serial correlation, where the disturbance at each period depends on the dis-

191
192 CHAPTER 11. SERIAL CORRELATION

turbance from the previous period:

yt = x′t β + εt
εt = ϕεt−1 + vt , |ϕ| < 1
vt ∼ iid N (0, σ 2 )

Extension to “AR(p)” disturbance serial correlation that includes p lagged

values is immediate.
Serial correlation has important consequences for β estimation and infer-
ence. As with heteroskedasticity, point estimates remain acceptable, at least
asymptotically (OLS parameter estimates remain consistent and asymptoti-
cally normal), but inference is damaged (OLS standard errors are biased and
inconsistent).
Serial correlation also has important consequences for y prediction. Unlike
with heteroskedasticity, even point predictions need re-thinking. Hence serial
correlation is a bigger problem for prediction than heteroskedasticity. Here’s
the intuition. Serial correlation in disturbances / residuals implies that the
included “x variables” have missed something that could be exploited for
improved point forecasting of y (and hence also improved interval and den-
sity forecasting). That is, all types of forecasts are sub-optimal when serial
correlation is neglected. Put differently, serial correlation in forecast errors
means that you can forecast your forecast errors (!), in which case something
is wrong and can hopefully be improved.

11.1 Characterizing Serial Correlation (in Population,

Mostly)

We’ve already considered models with trend and seasonal components. In this
chapter we consider a crucial third component, cycles. When you think of a
“cycle,” you might think of a rigid up-and-down pattern, as for example with
11.1. CHARACTERIZING SERIAL CORRELATION (IN POPULATION, MOSTLY)193

a cos or sin function, but cyclical fluctuations in business, finance, economics

and government are typically much less rigid. In fact, when we speak of
cycles, we have in mind a much more general, all-encompassing, notion of
cyclicality: any sort of dynamics not captured by trends or seasonals.
Cycles, according to our broad interpretation, simply have to have some
dynamics, some persistence, some way in which the present is linked to the
past, and the future to the present. Cycles are present in most of the series
that concern us, and it’s crucial that we know how to model and forecast
them, because their history conveys information regarding their future.
Trend and seasonal dynamics are simple, so we can capture them with
simple deterministic models. Cyclical dynamics, however, are more compli-
cated. Because of the wide variety of cyclical patterns, the sorts of models we
need are substantially more involved. The material is also a bit difficult the
first time around because it’s unavoidably rather mathematical, so careful,
systematic study is required.

11.1.1 Covariance Stationary Time Series

We begin by introducing the idea of a covariance stationary time series. We

will generally use yt to denote a time series. That series could be unobserved
(like the disturbance in a regression, which we called εt in the motivational
discussion above), or it could be observed (like U.S. GDP). Everything we
say below is valid either way.
Formally, a time series is an ordered infinite-dimensional random variable.
A realization of a time series is an ordered set,

{..., y−2 , y−1 , y0 , y1 , y2 , ...}.

Typically the observations are ordered in time – hence the name time series
– but they don’t have to be. We could, for example, examine a spatial series,
such as office space rental rates as we move along a line from a point in
194 CHAPTER 11. SERIAL CORRELATION

midtown Manhattan to a point in the New York suburbs thirty miles away.
But the most important case, by far, involves observations ordered in time,
so that’s what we’ll stress.
In theory, a time series realization begins in the infinite past and continues
into the infinite future. This perspective may seem abstract and of limited
practical applicability, but it will be useful in deriving certain very important
properties of the models we’ll be using shortly. In practice, of course, the data
we observe is just a finite subset of a realization, {y1 , ..., yT }, called a sample
path.
Shortly we’ll be building models for cyclical time series. If the underlying
probabilistic structure of the series were changing over time, we’d be doomed
– there would be no way to relate the future to the past, because the laws gov-
erning the future would differ from those governing the past. At a minimum
we’d like a series’ mean and covariance structure (that is, the covariances
between current and past values) to be stable over time, in which case we say
that the series is covariance stationary.
Let’s discuss covariance stationarity in greater depth. The first require-
ment for a series to be covariance stationary is that its mean be stable over
time. The mean of the series at time t is Eyt = µt . If the mean is stable over
time, as required by covariance stationarity, then we can write Eyt = µ, for
all t. Because the mean is constant over time, there’s no need to put a time
subscript on it.
The second requirement for a series to be covariance stationary is that
its covariance structure be stable over time. Quantifying stability of the
covariance structure is a bit tricky, but tremendously important, and we do
it using the autocovariance function. The autocovariance at displacement
τ is just the covariance between yt and yt−τ . It will of course depend on τ ,
11.1. CHARACTERIZING SERIAL CORRELATION (IN POPULATION, MOSTLY)195

and it may also depend on t, so in general we write

γ(t, τ ) = cov(yt , yt−τ ) = E(yt − µ)(yt−τ − µ).

If the covariance structure is stable over time, as required by covariance

stationarity, then the autocovariances depend only on displacement, τ , not
on time, t, and we write γ(t, τ ) = γ(τ ), for all t.
The autocovariance function is important because it provides a basic sum-
mary of cyclical dynamics in a covariance stationary series. By examining
the autocovariance structure of a series, we learn about its dynamic behav-
ior. We graph and examine the autocovariances as a function of τ . Note that
the autocovariance function is symmetric; that is, γ(τ ) = γ(−τ ), for all τ .
Typically, we’ll consider only non-negative values of τ . Symmetry reflects the
fact that the autocovariance of a covariance stationary series depends only
on displacement; it doesn’t matter whether we go forward or backward. Note
also that γ(0) = cov(yt , yt ) = var(yt ).
There is one more technical requirement of covariance stationarity: we
require that the variance of the series – the autocovariance at displacement
0, γ(0), be finite. It can be shown that no autocovariance can be larger
in absolute value than γ(0), so if γ(0) < ∞, then so too are all the other
autocovariances.
It may seem that the requirements for covariance stationarity are quite
stringent, which would bode poorly for our models, almost all of which in-
voke covariance stationarity in one way or another. It is certainly true that
many economic, business, financial and government series are not covariance
stationary. An upward trend, for example, corresponds to a steadily increas-
ing mean, and seasonality corresponds to means that vary with the season,
both of which are violations of covariance stationarity.
But appearances can be deceptive. Although many series are not covari-
ance stationary, it is frequently possible to work with models that give special
196 CHAPTER 11. SERIAL CORRELATION

treatment to nonstationary components such as trend and seasonality, so that

the cyclical component that’s left over is likely to be covariance stationary.
We’ll often adopt that strategy. Alternatively, simple transformations often
appear to transform nonstationary series to covariance stationarity. For ex-
ample, many series that are clearly nonstationary in levels appear covariance
stationary in growth rates.
In addition, note that although covariance stationarity requires means and
covariances to be stable and finite, it places no restrictions on other aspects
of the distribution of the series, such as skewness and kurtosis.1 The upshot
is simple: whether we work directly in levels and include special components
for the nonstationary elements of our models, or we work on transformed
data such as growth rates, the covariance stationarity assumption is not as
restrictive as one might naively fear.
At the beginning of this chapter we noted that autocorrelation corresponds
to non-diagonal Ω. Now, having introduced the autocovariance function, we
can display the precise form of Ω under serial correlation:
 
γ(0) γ(1) . . . γ(T − 1)
 
 γ(1) γ(0) . . . γ(T − 2)
Ω= .. .. ... ..


 . . . 

γ(T − 1) γ(T − 2) ... γ(0)
Note the “band symmetric” structure, illustrated here for T = 4:
 
a b c d
 
b a b c
Ω= c b a b

 
d c b a

Now we introduce the closely-related autocorrelation function. Recall that

1
For that reason, covariance stationarity is sometimes called second-order stationarity or weak sta-
tionarity.
11.1. CHARACTERIZING SERIAL CORRELATION (IN POPULATION, MOSTLY)197

the correlation between two random variables x and y is defined by

cov(x, y)
corr(x, y) = .
σx σy

That is, the correlation is simply the covariance, “normalized,” or “stan-

dardized,” by the product of the standard deviations of x and y. Both the
correlation and the covariance are measures of linear association between two
random variables. The correlation is often more informative and easily inter-
preted, because the construction of the correlation coefficient guarantees that
corr(x, y) ∈ [−1, 1], whereas the covariance between the same two random
variables may take any value. The correlation, moreover, does not depend
on the units in which x and y are measured, whereas the covariance does.
Thus, for example, if x and y have a covariance of ten million, they’re not
necessarily very strongly associated, whereas if they have a correlation of .95,
it is unambiguously clear that they are very strongly associated.
In light of the superior interpretability of correlations as compared to
covariances, we often work with the correlation, rather than the covariance,
between yt and yt−τ . That is, we work with the autocorrelation function,
ρ(τ ), rather than the autocovariance function, γ(τ ). The autocorrelation
function is obtained by dividing the autocovariance function by the variance,

γ(τ )
ρ(τ ) = , τ = 0, 1, 2, ....
γ(0)

The formula for the autocorrelation is just the usual correlation formula,
specialized to the correlation between yt and yt−τ . To see why, note that the
variance of yt is γ(0), and by covariance stationarity, the variance of y at any
other time yt−τ is also γ(0). Thus,

cov(yt , yt−τ ) γ(τ ) γ(τ )

ρ(τ ) = p p =p p = ,
var(yt ) var(yt−τ ) γ(0) γ(0) γ(0)
198 CHAPTER 11. SERIAL CORRELATION

γ(0)
as claimed. Note that we always have ρ(0) = γ(0) = 1, because any series is
perfectly contemporaneously correlated with itself. Thus the autocorrelation
at displacement 0 isn’t of interest; rather, only the autocorrelations beyond
displacement 0 inform us about a series’ dynamic structure.
Finally, the partial autocorrelation function, p(τ ), is sometimes use-
ful. p(τ ) is just the coefficient of yt−τ in a population linear regression of
yt on yt−1 , ..., yt−τ .2 We call such regressions autoregressions, because the
variable is regressed on lagged values of itself. It’s easy to see that the
autocorrelations and partial autocorrelations, although related, differ in an
important way. The autocorrelations are just the “simple” or “regular” corre-
lations between yt and yt−τ . The partial autocorrelations, on the other hand,
measure the association between yt and yt−τ after controlling for the effects
of yt−1 , ..., yt−τ +1 ; that is, they measure the partial correlation between yt
and yt−τ .
As with the autocorrelations, we often graph the partial autocorrelations
as a function of τ and examine their qualitative shape, which we’ll do soon.
Like the autocorrelation function, the partial autocorrelation function pro-
vides a summary of a series’ dynamics, but as we’ll see, it does so in a different
way.3
All of the covariance stationary processes that we will study subsequently
have autocorrelation and partial autocorrelation functions that approach
zero, one way or another, as the displacement gets large. In Figure 11.1 we
show an autocorrelation function that displays gradual one-sided damping.
The precise decay patterns of autocorrelations and partial autocorrelations of
a covariance stationary series, however, depend on the specifics of the series.
2
To get a feel for what we mean by “population regression,” imagine that we have an infinite sample
of data at our disposal, so that the parameter estimates in the regression are not contaminated by sampling
variation – that is, they’re the true population values. The thought experiment just described is a population
regression.
3
Also in parallel to the autocorrelation function, the partial autocorrelation at displacement 0 is always
one and is therefore uninformative and uninteresting. Thus, when we graph the autocorrelation and partial
autocorrelation functions, we’ll begin at displacement 1 rather than displacement 0.
11.1. CHARACTERIZING SERIAL CORRELATION (IN POPULATION, MOSTLY)199

Figure 11.1

11.1.2 Estimation and Inference for the Mean, Autocorrelation

and Partial Autocorrelation Functions

Now suppose we have a sample of data on a time series, and we don’t know
the true model that generated the data, or the mean, autocorrelation function
or partial autocorrelation function associated with that true model. Instead,
we want to use the data to estimate the mean, autocorrelation function, and
partial autocorrelation function, which we might then use to help us learn
about the underlying dynamics, and to decide upon a suitable model or set
of models to fit to the data.

Sample Mean

The mean of a covariance stationary series is

µ = Eyt .
200 CHAPTER 11. SERIAL CORRELATION

A fundamental principle of estimation, called the analog principle, suggests

that we develop estimators by replacing expectations with sample averages.
Thus our estimator for the population mean, given a sample of size T , is the
sample mean,
T
1X
ȳ = yt .
T t=1
Typically we’re not directly interested in the estimate of the mean, but it’s
needed for estimation of the autocorrelation function.

Sample Autocorrelations

The autocorrelation at displacement τ for the covariance stationary series y

is
E [(yt − µ)(yt−τ − µ)]
ρ(τ ) = .
E[(yt − µ)2 ]
Application of the analog principle yields a natural estimator,
1
PT PT
T t=τ +1 [(yt − ȳ)(yt−τ − ȳ)] t=τ +1 [(yt − ȳ)(yt−τ − ȳ)]
ρ̂(τ ) = 1
PT = PT .
2 2
T t=1 (yt − ȳ) t=1 (yt − ȳ)

This estimator, viewed as a function of τ , is called the sample autocorre-

lation function, or correlogram. Note that some of the summations begin
at t = τ + 1, not at t = 1; this is necessary because of the appearance of yt−τ
in the sum. Note that we divide those same sums by T , even though only
T − τ terms appear in the sum. When T is large relative to τ (which is the
relevant case), division by T or by T − τ will yield approximately the same
result, so it won’t make much difference for practical purposes, and moreover
there are good mathematical reasons for preferring division by T .
It’s often of interest to assess whether a series is reasonably approximated
as white noise, which is to say whether all its autocorrelations are zero in
population. A key result, which we simply assert, is that if a series is white
noise, then the distribution of the sample autocorrelations in large samples
11.1. CHARACTERIZING SERIAL CORRELATION (IN POPULATION, MOSTLY)201

is  
1
ρ̂(τ ) ∼ N 0, .
T
Note how simple the result is. The sample autocorrelations of a white noise
series are approximately normally distributed, and the normal is always a
convenient distribution to work with. Their mean is zero, which is to say the
sample autocorrelations are unbiased estimators of the true autocorrelations,
which are in fact zero. Finally, the variance of the sample autocorrelations
√
is approximately 1/T (equivalently, the standard deviation is 1/ T ), which
is easy to construct and remember. Under normality, taking plus or minus
two standard errors yields an approximate 95% confidence interval. Thus, if
the series is white noise, approximately 95% of the sample autocorrelations
√
should fall in the interval 0 ± 2/ T . In practice, when we plot the sample
autocorrelations for a sample of data, we typically include the “two standard
error bands,” which are useful for making informal graphical assessments of
whether and how the series deviates from white noise.
The two-standard-error bands, although very useful, only provide 95%
bounds for the sample autocorrelations taken one at a time. Ultimately,
we’re often interested in whether a series is white noise, that is, whether all
its autocorrelations are jointly zero. A simple extension lets us test that
hypothesis. Rewrite the expression
 
1
ρ̂(τ ) ∼ N 0,
T
as
√
T ρ̂(τ ) ∼ N (0, 1).

Squaring both sides yields4

T ρ̂2 (τ ) ∼ χ21 .
4
Recall that the square of a standard normal random variable is a χ2 random variable with one degree
of freedom. We square the sample autocorrelations ρ̂(τ ) so that positive and negative values don’t cancel
when we sum across various values of τ , as we will soon do.
202 CHAPTER 11. SERIAL CORRELATION

It can be shown that, in addition to being approximately normally dis-

tributed, the sample autocorrelations at various displacements are approxi-
mately independent of one another. Recalling that the sum of independent χ2
variables is also χ2 with degrees of freedom equal to the sum of the degrees
of freedom of the variables summed, we have shown that the Box-Pierce
Q-statistic,
m
X
QBP = T ρ̂2 (τ ),
τ =1

is approximately distributed as a χ2m random variable under the null hy-

pothesis that y is white noise.5 A slight modification is the Ljung-Box Q-
statistic, designed to follow more closely the χ2 distribution in small samples,
it is m  
X 1
QLB = T (T + 2) ρ̂2 (τ ).
τ =1
T −τ
Under the null hypothesis that y is white noise, QLB is approximately dis-
tributed as a χ2m random variable. Note that the Ljung-Box Q-statistic is the
same as the Box-Pierce Q statistic, except that the sum of squared autocor-
relations is replaced by a weighted sum of squared autocorrelations, where
the weights are (T + 2)/(T − τ ). For moderate and large T , the weights
are approximately 1, so that the Ljung-Box statistic differs little from the
Box-Pierce statistic.
Selection of m is done to balance competing criteria. On one hand, we
don’t want m too small, because after all, we’re trying to do a joint test on
a large part of the autocorrelation function. On the other hand, as m grows
relative to T , the quality of the distributional approximations we’ve invoked
√
deteriorates. In practice, focusing on m in the neighborhood of T is often
reasonable.
5
m is a maximum displacement selected by the user. Shortly we’ll discuss how to choose it.
11.1. CHARACTERIZING SERIAL CORRELATION (IN POPULATION, MOSTLY)203

Sample Partial Autocorrelations

Recall that the partial autocorrelations are obtained from population linear
regressions, which correspond to a thought experiment involving linear re-
gression using an infinite sample of data. The sample partial autocorrelations
correspond to the same thought experiment, except that the linear regression
is now done on the (feasible) sample of size T . If the fitted regression is

ŷt = ĉ + β̂1 yt−1 + ... + β̂τ yt−τ ,

then the sample partial autocorrelation at displacement τ is

p̂(τ ) ≡ β̂τ .

Distributional results identical to those we discussed for the sample auto-

correlations hold as well for the sample partial autocorrelations. That is, if
the series is white noise, approximately 95% of the sample partial autocorre-
√
lations should fall in the interval ±2/ T . As with the sample autocorrela-
tions, we typically plot the sample partial autocorrelations along with their
two-standard-error bands.
A “correlogram analysis” simply means examination of the sample au-
tocorrelation and partial autocorrelation functions (with two standard error
bands), together with related diagnostics, such as Q statistics.
We don’t show the sample autocorrelation or partial autocorrelation at
displacement 0, because as we mentioned earlier, they equal 1.0, by construc-
tion, and therefore convey no useful information. We’ll adopt this convention
throughout.
Note that the sample autocorrelation and partial autocorrelation are iden-
tical at displacement 1. That’s because at displacement 1, there are no earlier
lags to control for when computing the sample partial autocorrelation, so it
equals the sample autocorrelation. At higher displacements, of course, the
204 CHAPTER 11. SERIAL CORRELATION

two diverge.

11.2 Modeling Serial Correlation (in Population)

11.2.1 White Noise

In this section we’ll study the population properties of certain important time
series models, or time series processes. Before we estimate time series
models, we need to understand their population properties, assuming that
the postulated model is true. The simplest of all such time series processes
is the fundamental building block from which all others are constructed. In
fact, it’s so important that we introduce it now. We use y to denote the
observed series of interest. Suppose that

yt = εt

εt ∼ (0, σ 2 ),

where the “shock,” εt , is uncorrelated over time. We say that εt , and hence
yt , is serially uncorrelated. Throughout, unless explicitly stated otherwise,
we assume that σ 2 < ∞. Such a process, with zero mean, constant variance,
and no serial correlation, is called zero-mean white noise, or simply white
noise.6 Sometimes for short we write

εt ∼ W N (0, σ 2 )

and hence
yt ∼ W N (0, σ 2 ).

Note that, although εt and hence yt are serially uncorrelated, they are
not necessarily serially independent, because they are not necessarily nor-
6
It’s called white noise by analogy with white light, which is composed of all colors of the spectrum,
in equal amounts. We can think of white noise as being composed of a wide variety of cycles of differing
periodicities, in equal amounts.
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 205

mally distributed.7 If in addition to being serially uncorrelated, y is serially

independent, then we say that y is independent white noise.8 We write

yt ∼ iid(0, σ 2 ),

and we say that “y is independently and identically distributed with zero

mean and constant variance.” If y is serially uncorrelated and normally
distributed, then it follows that y is also serially independent, and we say
that y is normal white noise, or Gaussian white noise.9 We write

yt ∼ iidN (0, σ 2 ).

We read “y is independently and identically distributed as normal, with zero

mean and constant variance,” or simply “y is Gaussian white noise.” In
Figure 11.2 we show a sample path of Gaussian white noise, of length T = 150,
simulated on a computer. There are no patterns of any kind in the series due
to the independence over time.
You’re already familiar with white noise, although you may not realize it.
Recall that the disturbance in a regression model is typically assumed to be
white noise of one sort or another. There’s a subtle difference here, however.
Regression disturbances are not observable, whereas we’re working with an
observed series. Later, however, we’ll see how all of our models for observed
series can be used to model unobserved variables such as regression distur-
bances. Let’s characterize the dynamic stochastic structure of white noise,
yt ∼ W N (0, σ 2 ). By construction the unconditional mean of y is E(yt ) = 0,
and the unconditional variance of y is var(yt ) = σ 2 .
Note that the unconditional mean and variance are constant. In fact, the
7
Recall that zero correlation implies independence only in the normal case.
8
Another name for independent white noise is strong white noise, in contrast to standard serially
uncorrelated weak white noise.
9
Carl Friedrich Gauss, one of the greatest mathematicians of all time, discovered the normal distribution
some 200 years ago; hence the adjective “Gaussian.”
206 CHAPTER 11. SERIAL CORRELATION

Figure 11.2

unconditional mean and variance must be constant for any covariance sta-
tionary process. The reason is that constancy of the unconditional mean was
our first explicit requirement of covariance stationarity, and that constancy
of the unconditional variance follows implicitly from the second requirement
of covariance stationarity, that the autocovariances depend only on displace-
ment, not on time.10
To understand fully the linear dynamic structure of a covariance station-
ary time series process, we need to compute and examine its mean and its
autocovariance function. For white noise, we’ve already computed the mean
and the variance, which is the autocovariance at displacement 0. We have
yet to compute the rest of the autocovariance function; fortunately, however,
it’s very simple. Because white noise is, by definition, uncorrelated over time,
all the autocovariances, and hence all the autocorrelations, are zero beyond
displacement 0.11 Formally, then, the autocovariance function for a white
10
Recall that σ 2 = γ(0).
11
If the autocovariances are all zero, so are the autocorrelations, because the autocorrelations are propor-
tional to the autocovariances.
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 207

Figure 11.3

noise process is 
σ 2 if τ = 0
γ(τ ) =
0 if τ ≥ 1
and the autocorrelation function for a white noise process is

1 if τ = 0
ρ(τ ) =
0 if τ ≥ 1.

In Figure 11.3 we plot the white noise autocorrelation function.

Finally, consider the partial autocorrelation function for a white noise
series. For the same reason that the autocorrelation at displacement 0 is
always one, so too is the partial autocorrelation at displacement 0. For a
white noise process, all partial autocorrelations beyond displacement 0 are
zero, which again follows from the fact that white noise, by construction, is
serially uncorrelated. Population regressions of yt on yt−1 , or on yt−1 and
yt−2 , or on any other lags, produce nothing but zero coefficients, because the
process is serially uncorrelated. Formally, the partial autocorrelation function
208 CHAPTER 11. SERIAL CORRELATION

Figure 11.4

of a white noise process is


1 if τ = 0
p(τ ) =
0 if τ ≥ 1.

We show the partial autocorrelation function of a white noise process in

Figure 11.4 . Again, it’s degenerate, and exactly the same as the autocorre-
lation function!
White noise is very special, indeed degenerate in a sense, as what happens
to a white noise series at any time is uncorrelated with anything in the past,
and similarly, what happens in the future is uncorrelated with anything in the
present or past. But understanding white noise is tremendously important
for at least two reasons. First, as already mentioned, processes with much
richer dynamics are built up by taking simple transformations of white noise.
Second, the goal of all time series modeling (and 1-step-ahead forecasting)
is to reduce the data (or 1-step-ahead forecast errors) to white noise. After
all, if such forecast errors aren’t white noise, then they’re serially correlated,
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 209

which means that they’re forecastable, and if forecast errors are forecastable
then the forecast can’t be very good. Thus it’s important that we understand
and be able to recognize white noise.
Thus far we’ve characterized white noise in terms of its mean, variance,
autocorrelation function and partial autocorrelation function. Another char-
acterization of dynamics involves the mean and variance of a process, condi-
tional upon its past. In particular, we often gain insight into the dynamics in
a process by examining its conditional mean.12 In fact, throughout our study
of time series, we’ll be interested in computing and contrasting the uncondi-
tional mean and variance and the conditional mean and variance of
various processes of interest. Means and variances, which convey information
about location and scale of random variables, are examples of what statisti-
cians call moments. For the most part, our comparisons of the conditional
and unconditional moment structure of time series processes will focus on
means and variances (they’re the most important moments), but sometimes
we’ll be interested in higher-order moments, which are related to properties
such as skewness and kurtosis.
For comparing conditional and unconditional means and variances, it will
simplify our story to consider independent white noise, yt ∼ iid(0, σ 2 ). By
the same arguments as before, the unconditional mean of y is 0 and the un-
conditional variance is σ 2 . Now consider the conditional mean and variance,
where the information set Ωt−1 upon which we condition contains either the
past history of the observed series, Ωt−1 = yt−1 , yt−2 , ..., or the past history of
the shocks, Ωt−1 = εt−1 , εt−2 .... (They’re the same in the white noise case.)
In contrast to the unconditional mean and variance, which must be constant
by covariance stationarity, the conditional mean and variance need not be
constant, and in general we’d expect them not to be constant. The uncondi-
tionally expected growth of laptop computer sales next quarter may be ten
12
If you need to refresh your memory on conditional means, consult any good introductory statistics book,
such as Wonnacott and Wonnacott (1990).
210 CHAPTER 11. SERIAL CORRELATION

percent, but expected sales growth may be much higher, conditional upon
knowledge that sales grew this quarter by twenty percent. For the indepen-
dent white noise process, the conditional mean is

E(yt |Ωt−1 ) = 0,

and the conditional variance is

var(yt |Ωt−1 ) = E[(yt − E(yt |Ωt−1 ))2 |Ωt−1 ] = σ 2 .

Conditional and unconditional means and variances are identical for an inde-
pendent white noise series; there are no dynamics in the process, and hence
no dynamics in the conditional moments.

11.2.2 The Lag Operator

The lag operator and related constructs are the natural language in which
time series models are expressed. If you want to understand and manipulate
time series models – indeed, even if you simply want to be able to read the
software manuals – you have to be comfortable with the lag operator. The
lag operator, L, is very simple: it “operates” on a series by lagging it. Hence
Lyt = yt−1 . Similarly, L2 yt = L(L(yt )) = L(yt−1 ) = yt−2 , and so on. Typically
we’ll operate on a series not with the lag operator but with a polynomial
in the lag operator. A lag operator polynomial of degree m is just a linear
function of powers of L, up through the m-th power,

B(L) = b0 + b1 L + b2 L2 + ...bm Lm .

To take a very simple example of a lag operator polynomial operating on

a series, consider the m-th order lag operator polynomial Lm , for which

Lm yt = yt−m .
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 211

A well-known operator, the first-difference operator ∆, is actually a first-order

polynomial in the lag operator; you can readily verify that

∆yt = (1 − L)yt = yt − yt−1 .

As a final example, consider the second-order lag operator polynomial 1 +

.9L + .6L2 operating on yt . We have

(1 + .9L + .6L2 )yt = yt + .9yt−1 + .6yt−2 ,

which is a weighted sum, or distributed lag, of current and past values.

All time-series models, one way or another, must contain such distributed
lags, because they’ve got to quantify how the past evolves into the present
and future; hence lag operator notation is a useful shorthand for stating and
manipulating time-series models.

11.2.3 Autoregression

When building models, we don’t want to pretend that the model we fit is
true. Instead, we want to be aware that we’re approximating a more complex
reality. That’s the modern view, and it has important implications for time-
series modeling. In particular, the key to successful time series modeling
is parsimonious, yet accurate, approximations. Here we emphasize a very
important class of approximations, the autoregressive (AR) model.
We begin by characterizing the autocorrelation function and related quan-
tities under the assumption that the AR model is “true.”13 These charac-
terizations have nothing to do with data or estimation, but they’re crucial
for developing a basic understanding of the properties of the models, which
is necessary to perform intelligent modeling. They enable us to make state-
ments such as “If the data were really generated by an autoregressive process,
13
Sometimes, especially when characterizing population properties under the assumption that the models
are correct, we refer to them as processes, which is short for stochastic processes.
212 CHAPTER 11. SERIAL CORRELATION

then we’d expect its autocorrelation function to have property x.” Armed
with that knowledge, we use the sample autocorrelations and partial auto-
correlations, in conjunction with the AIC and the SIC, to suggest candidate
models, which we then estimate.
The autoregressive process is a natural approximation to time-series dy-
namics. It’s simply a stochastic difference equation, a simple mathematical
model in which the current value of a series is linearly related to its past
values, plus an additive stochastic shock. Stochastic difference equations are
a natural vehicle for discrete-time stochastic dynamic modeling.

The AR(1) Process

The first-order autoregressive process, AR(1) for short, is

yt = ϕyt−1 + εt
εt ∼ W N (0, σ 2 ).

In lag operator form, we write

(1 − ϕL)yt = εt.

In Figure 11.5 we show simulated realizations of length 150 of two AR(1)

processes; the first is
yt = .4yt−1 + εt ,

and the second is

yt = .95yt−1 + εt ,

where in each case

εt ∼ iidN (0, 1),

and the same innovation sequence underlies each realization. The fluctuations
in the AR(1) with parameter ϕ = .95 appear much more persistent that those
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 213

Figure 11.5

of the AR(1) with parameter ϕ = .4. Thus the AR(1) model is capable of
capturing highly persistent dynamics.
Certain conditions must be satisfied for an autoregressive process to be
covariance stationary. If we begin with the AR(1) process,

yt = ϕyt−1 + εt ,

and substitute backward for lagged y’s on the right side, we obtain

yt = εt + ϕεt−1 + ϕ2 εt−2 + ...

This representation of y in terms of current and past shocks is called a

moving-average representation. In lag operator form we write
1
yt = εt = B(L)εt .
1 − ϕL

This moving average representation for y is convergent if and only if |ϕ| < 1;
thus, |ϕ| < 1 is the condition for covariance stationarity in the AR(1) case.
214 CHAPTER 11. SERIAL CORRELATION

Figure 11.6

Equivalently, the condition for covariance stationarity is that the inverse of

the root of the autoregressive lag operator polynomial be less than one in
absolute value.
From the moving average representation of the covariance stationary AR(1)
process, we can compute the unconditional mean and variance,

E(yt ) = E(εt + ϕεt−1 + ϕ2 εt−2 + ...)

= E(εt ) + ϕE(εt−1 ) + ϕ2 E(εt−2 ) + ...
=0
and var(yt ) = var(εt + ϕεt−1 + ϕ2 εt−2 + ...)
= σ 2 + ϕ2 σ 2 + ϕ4 σ 2 + ...
∞
X
2
=σ ϕ2i
i=0
2
σ
= .
1 − ϕ2
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 215

Figure 11.7

The conditional moments, in contrast, are

E(yt |yt−1 ) = E(ϕyt−1 + εt |yt−1 )

= ϕE(yt−1 |yt−1 ) + E(εt |yt−1 )
= ϕyt−1 + 0
= ϕyt−1
and var(yt |yt−1 ) = var((ϕyt−1 + εt )|yt−1 )
= ϕ2 var(yt−1 |yt−1 ) + var(εt |yt−1 )
= 0 + σ2
= σ2.

Note in particular that the simple way that the conditional mean adapts
to the changing information set as the process evolves.
To find the autocovariances, we proceed as follows. The process is

yt = ϕyt−1 + εt ,
216 CHAPTER 11. SERIAL CORRELATION

Figure 11.8

so that multiplying both sides of the equation by yt−τ we obtain

yt yt−τ = ϕyt−1 yt−τ + εt yt−τ .

For τ ≥ 1, taking expectations of both sides gives

γ(τ ) = ϕγ(τ − 1).

This is called the Yule-Walker equation. It is a recursive equation; that is,

given γ(τ ), for any τ , the Yule-Walker equation immediately tells us how to
get γ(τ + 1). If we knew γ(0) to start things off (an “initial condition”), we
could use the Yule-Walker equation to determine the entire autocovariance
sequence. And we do know γ(0); it’s just the variance of the process, which
we already showed to be
σ2
γ(0) = .
1 − ϕ2
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 217

Figure 11.9

Thus we have
2
γ(0) = σ 1−ϕ2
2
γ(1) = ϕσ 1−ϕ2
2
γ(2) = ϕ2 σ 1−ϕ2 ,

and so on. In general, then,

2
γ(τ ) = ϕτ σ 1−ϕ2 , τ = 0, 1, 2, ....

Dividing through by γ(0) gives the autocorrelations,

ρ(τ ) = ϕτ , τ = 0, 1, 2, ....

Note the gradual autocorrelation decay, which is typical of autoregressive

processes. The autocorrelations approach zero, but only in the limit as the
displacement approaches infinity. In particular, they don’t cut off to zero, as
is the case for moving average processes. If ϕ is positive, the autocorrelation
218 CHAPTER 11. SERIAL CORRELATION

decay is one-sided. If ϕ is negative, the decay involves back-and-forth oscilla-

tions. The relevant case in business and economics is ϕ > 0, but either way,
the autocorrelations damp gradually, not abruptly. In Figures 11.6 and 11.7
we show the autocorrelation functions for AR(1) processes with parameters
ϕ = .4 and ϕ = .95. The persistence is much stronger when ϕ = .95.
Finally, the partial autocorrelation function for the AR(1) process cuts off
abruptly; specifically, 
ϕ if τ = 1
p(τ ) =
0 if τ > 1

It’s easy to see why. The partial autocorrelations are just the last coeffi-
cients in a sequence of successively longer population autoregressions. If the
true process is in fact an AR(1), the first partial autocorrelation is just the
autoregressive coefficient, and coefficients on all longer lags are zero.
In Figures 11.8 and 11.9 we show the partial autocorrelation functions for
our two AR(1) processes. At displacement 1, the partial autocorrelations are
simply the parameters of the process (.4 and .95, respectively), and at longer
displacements, the partial autocorrelations are zero.

The AR(p) Process

The general p-th order autoregressive process, or AR(p) for short, is

yt = ϕ1 yt−1 + ϕ2 yt−2 + ... + ϕp yt−p + εt

εt ∼ W N (0, σ 2 ).

In lag operator form we write

Φ(L)yt = (1 − ϕ1 L − ϕ2 L2 − ... − ϕp Lp )yt = εt.

In our discussion of the AR(p) process we dispense with mathematical

derivations and instead rely on parallels with the AR(1) case to establish
11.2. MODELING SERIAL CORRELATION (IN POPULATION) 219

intuition for its key properties.

An AR(p) process is covariance stationary if and only if the inverses of all
roots of the autoregressive lag operator polynomial Φ(L) are inside the unit
circle.14 In the covariance stationary case we can write the process in the
convergent infinite moving average form
1
yt = εt.
Φ(L)

The autocorrelation function for the general AR(p) process, as with that of
the AR(1) process, decays gradually with displacement. Finally, the AR(p)
partial autocorrelation function has a sharp cutoff at displacement p, for
the same reason that the AR(1) partial autocorrelation function has a sharp
cutoff at displacement 1.
Let’s discuss the AR(p) autocorrelation function in a bit greater depth.
The key insight is that, in spite of the fact that its qualitative behavior
(gradual damping) matches that of the AR(1) autocorrelation function, it
can nevertheless display a richer variety of patterns, depending on the order
and parameters of the process. It can, for example, have damped monotonic
decay, as in the AR(1) case with a positive coefficient, but it can also have
damped oscillation in ways that AR(1) can’t have. In the AR(1) case, the
only possible oscillation occurs when the coefficient is negative, in which case
the autocorrelations switch signs at each successively longer displacement. In
higher-order autoregressive models, however, the autocorrelations can oscil-
late with much richer patterns reminiscent of cycles in the more traditional
sense. This occurs when some roots of the autoregressive lag operator poly-
14
Pp
A necessary condition for covariance stationarity, which is often useful as a quick check, is i=1 ϕi < 1.
If the condition is satisfied, the process may or may not be stationary, but if the condition is violated, the
process can’t be stationary.
220 CHAPTER 11. SERIAL CORRELATION

nomial are complex.15 Consider, for example, the AR(2) process,

yt = 1.5yt−1 − .9yt−2 + εt.

The corresponding lag operator polynomial is 1 − 1.5L + .9L2 , with two com-
plex conjugate roots, .83±.65i. The inverse roots are .75±.58i, both of which
are close to, but inside, the unit circle; thus the process is covariance station-
ary. It can be shown that the autocorrelation function for an AR(2) process
is

ρ(0) = 1
ϕ1
ρ(1) =
1 − ϕ2
ρ(τ ) = ϕ1 ρ(τ − 1) + ϕ2 ρ(τ − 2), τ = 2, 3, ...

Using this formula, we can evaluate the autocorrelation function for the
process at hand; we plot it in Figure 11.10. Because the roots are complex,
the autocorrelation function oscillates, and because the roots are close to the
unit circle, the oscillation damps slowly.

11.3 Modeling Serial Correlation (in Sample)

Here we return to regression with AR(p) disturbances, as always using the

liquor sales data for illustration.

11.3.1 Detecting Serial Correlation

If a model has extracted all the systematic information from the data, then
what’s left – the residual – should be iid random noise. Hence the usefulness
of various residual-based tests of the hypothesis that regression disturbances
are white noise (i.e., not serially correlated).
15
Note that complex roots can’t occur in the AR(1) case.
11.3. MODELING SERIAL CORRELATION (IN SAMPLE) 221

Figure 11.10

Of course the most obvious thing is simply to inspect the residual plot.
For convenience we reproduce our liquor sales residual plot in Figure 11.11.
There is clear visual evidence of serial correlation in our liquor sales residu-
als. Sometimes, however, things are not so visually obvious. Hence we now
introduce some additional tools.

Residual Scatterplots

The relevant scatterplot for detecting serial correlation involves plotting et

against et−τ . A leading case, corresponding to potential relevance of first-
order serial correlation, involves plotting et against et−1 . We show this scat-
terplot for our liquor sales residuals in Figure 11.12. There is an obvious
relationship.
222 CHAPTER 11. SERIAL CORRELATION

Figure 11.11

Figure 11.12
11.3. MODELING SERIAL CORRELATION (IN SAMPLE) 223

Durbin-Watson

The Durbin-Watson is a more formal test. We work in the simple paradigm

(AR(1)):

yt = x′t β + εt
εt = ϕεt−1 + vt
vt ∼ iid N (0, σ 2 )

The regression disturbance is serially correlated when ϕ ̸= 0 . The hy-

pothesis of interest is that ϕ = 0 . When ϕ = 0, the ideal conditions hold,
but when ϕ ̸= 0, the disturbance is serially correlated. More specifically,
when ϕ ̸= 0, εt follows an AR(1) processs. If ϕ > 0 the disturbance is posi-
tively serially correlated, and if ϕ < 0 the disturbance is negatively serially
correlated. Positive serial correlation is typically the relevant alternative in
economic applications.
Proceeding, we want to test H0 : ϕ = 0 against H1 : ϕ ̸= 0. We regress
y → X and obtain the residuals et
PT 2
t=2 (et − et−1 )
DW = PT 2
t=1 et

DW takes values in the interval [0, 4], and if all is well, DW should be
around 2. If DW is substantially less than 2, there is evidence of positive
serial correlation. As a rough rule of thumb, if DW is less than 1.5, there
may be cause for alarm, and we should consult the tables of the DW statistic,
available in many statistics and econometrics texts.
224 CHAPTER 11. SERIAL CORRELATION

Let us attempt to understand DW more thoroughly. We have:

PT 2 1
PT 2
t=2 (e t − e t−1 ) T t=2 (et − et−1 )
DW = PT 2 = 1
PT 2
e
t=1 t T t=1 et
1
PT 2 1 PT 2 1
PT
T t=2 t e + T e
t=2 t−1 − 2 T t=2 et et−1
= 1 T 2
P
T t=1 et

Hence as T → ∞:
σ 2 + σ 2 − 2cov(et , et−1 )
DW ≈ = 2(1 − corr(et , et−1 ))
σ2 | {z }
ρe (1)

Hence DW ∈ [0, 4], DW → 2 as ϕ → 0, and DW → 0 as ϕ → 1.

Also note that the Durbin-Watson test is effectively based only on the first
sample autocorrelation and really only tests whether the first autocorrelation
is zero. We say therefore that the Durbin-Watson is a test for first-order
serial correlation.
In addition, the Durbin-Watson test is not valid if the regressors include
lagged dependent variables.16 (See EPC 6.) On both counts, we’d like more
general and flexible approaches for diagnosing serial correlation.
For liquor sales, DW = .59 – clear evidence of residual serial correlation!

The Breusch-Godfrey Test

The Breusch-Godfrey test is an alternative to the Durbin-Watson test.

It’s designed to detect pth -order serial correlation, where p is selected by the
user, and is also valid in the presence of lagged dependent variables.
16
Following standard, if not strictly appropriate, practice, in this book we often report and examine the
Durbin-Watson statistic even when lagged dependent variables are included. We always supplement the
Durbin-Watson statistic, however, with other diagnostics such as the residual correlogram, which remain
valid in the presence of lagged dependent variables, and which almost always produce the same inference as
the Durbin-Watson statistic.
11.3. MODELING SERIAL CORRELATION (IN SAMPLE) 225

Figure 11.13: BG Test Equation, 4 Lags

We work in a general AR(p) environment:

yt = x′t β + εt
εt = ϕ1 εt−1 + ... + ϕp εt−p + vt
vt ∼ iidN (0, σ 2 )

We want to test H0 : (ϕ1 , ..., ϕp ) = 0 against H1 : (ϕ1 , ..., ϕp ) ̸= 0. We

proceed as follows:

1. Regress yt → xt and obtain the residuals et

2. Regress et → xt , et−1 , ..., et−p

3. The test statistic is BG = T R2 where T is sample size R2 is the R2 from

the regression. In large samples BG ∼ χ2p under the null.

This should sound familiar, as it precisely parallels the BGP heteroskedas-

ticity test that we studied earlier in Chapter 7.
226 CHAPTER 11. SERIAL CORRELATION

Figure 11.14: BG Test Equation, 8 Lags

Some test regression results appear in Figures 11.13-11.14. In particular

we have a BG for AR(4) disturbances of T R2 = 216.7, (p = 0.0000), and
a BG for AR(8) Disturbances of T R2 = 219.0 (p = 0.0000). There is is
strong evidence of autoregressve dynamics through lag 4, but not after lag 4,
suggesting AR(4).

11.3.2 The Residual Correlogram

The residual sample autocorrelations are:

1
P
cov(e
c t , et−τ ) T t et et−τ
ρ̂e (τ ) = = 1
P 2
.
vd
ar(et ) T t et

The residual sample partial autocorrelation at displacement τ s, p̂e (τ ), is

the coefficient on et−τ in the regression

et → c, et−1 , ..., et−(τ −1) , et−τ .

The approximate 95% “Bartlett bands” remain 0 ± √2 . The Q statistics

T
11.3. MODELING SERIAL CORRELATION (IN SAMPLE) 227

Figure 11.15: Residual Correlogram From Trend + Seasonal Model

also remain unchanged:

m
X
QBP = T ρ̂2e (τ ) ∼ χ2m−K
τ =1
m  
X 1
QLB = T (T + 2) ρ̂2e (τ ) ∼ χ2m−K .
τ =1
T −τ

The only wrinkle is that, when we earlier introduced the correlogram, we

focused on the case of an observed time series, in which case we showed
that the Q statistics are distributed as χ2m . Now, however, we want to assess
whether unobserved model disturbances are white noise. To do so, we use the
model residuals, which are estimates of the unobserved disturbances. Because
we fit a model to get the residuals, we need to account for the degrees of
freedom used. The upshot is that the distribution of the Q statistics under the
white noise hypothesis is better approximated by a χ2m−K random variable,
where k is the number of parameters estimated.
We show the residual correlogram for the trend + seasonal model in Figure
11.15. It strongly supports the AR(4) specification. The sample autocorrela-
tions decay gradually, and the sample partial autocorrelations cut off sharply
at displacement 4.
228 CHAPTER 11. SERIAL CORRELATION

11.3.3 Estimation with Serial Correlation

The remaining issue is how to estimate a regression model with serially cor-
related disturbances. Let us illustrate with the AR(1) case. The model is:

yt = x′t β + εt
εt = ϕεt−1 + vt
vt ∼ iid N (0, σ 2 ).

It is possible to work out the likelihood function and maximize it, or to

implement various approximations to the maximum-likelihood estimator, but
the exact likelihood function is complicated. Hence it is fortunate that there
is a simpler route. Manipulating the above equations we have:

ϕyt−1 = ϕx′t−1 β + ϕεt−1

=⇒ (yt − ϕyt−1 ) = (x′t − ϕx′t−1 )β + (εt − ϕεt−1 )
=⇒ yt = ϕyt−1 + x′t β − x′t−1 (ϕβ) + vt .

This “new” model satisfies the IC (recall that v is iid), so dealing with
autocorrelated disturbances amounts to little more than including an autore-
gressive lag in the regression.17 The IC are satisfied so OLS is fine. AR(1)
disturbances require 1 lag, as we just showed. General AR(p) disturbances
require p lags.
For liquor sales, everything points to AR(4) disturbance dynamics – from
the residual correlogram of the original trend + seasonal model, to the DW
test results (DW is designed to detect AR(1) but of course it can also re-
ject against higher-order autoregressive alternatives), to the BG test results,
to the SIC pattern (AR(1) = −3.797, AR(2) = −3.941, AR(3) = −4.080,
AR(4) = −4.086, AR(5) = −4.071, AR(6) = −4.058, AR(7) = −4.057,
17
Note the constraint that the coefficient on xt−1 is the product of the coefficients on yt−1 and xt . One
may or may not want to rigidly impose that contraint; in any event the main thing is to “clean out” the
dynamics in εt by including lags of yt .
11.3. MODELING SERIAL CORRELATION (IN SAMPLE) 229

Figure 11.16: Trend + Seasonal Model with Four Autoregressive Lags

AR(8) = −4.040). In Figures 11.16-11.19 we show the “final” model estima-

tion results, the corresponding residual plot, and the residual histogram and
normality test.
230 CHAPTER 11. SERIAL CORRELATION

Figure 11.17: Trend + Seasonal Model with Four Lags of y, Residual Plot
11.3. MODELING SERIAL CORRELATION (IN SAMPLE) 231

Figure 11.18: Trend + Seasonal Model with Four Autoregressive Lags, Residual Scatterplot

Figure 11.19: Trend + Seasonal Model with Four Autoregressive Lags, Residual Autocorre-
lations
232 CHAPTER 11. SERIAL CORRELATION

11.4 Exercises, Problems and Complements

1. (Autocorrelation functions of covariance stationary series)

While interviewing at a top investment bank, your interviewer is im-
pressed by the fact that you have taken a course on time series. She
decides to test your knowledge of the autocovariance structure of covari-
ance stationary series and lists five autocovariance functions:

a. γ(t, τ ) = α
b. γ(t, τ ) = e−ατ
c. γ(t, τ ) = ατ
d. γ(t, τ ) = ατ , where α is a positive constant.

Which autocovariance function(s) are consistent with covariance station-

arity, and which are not? Why?

2. (Autocorrelation vs. partial autocorrelation)

Describe the difference between autocorrelations and partial autocorre-
lations. How can autocorrelations at certain displacements be positive
while the partial autocorrelations at those same displacements are neg-
ative?

3. (Simulating time series processes)

Many cutting-edge estimation techniques involve simulation. Moreover,
simulation is often a good way to get a feel for a model and its behavior.
White noise can be simulated on a computer using random number
generators, which are available in most statistics, econometrics and
forecasting packages.

a. Simulate a Gaussian white noise realization of length 200. Call the

white noise εt . Compute the correlogram. Discuss.
11.4. EXERCISES, PROBLEMS AND COMPLEMENTS 233

b. Form the distributed lag yt = εt + .9εt−1 , t = 2, 3, ..., 200. Compute

the sample autocorrelations and partial autocorrelations. Discuss.
c. Let y1 = 1 and yt = .9yt−1 + εt , t = 2, 3, ..., 200. Compute the sample
autocorrelations and partial autocorrelations. Discuss.

4. (Outliers in Time Series)

Outliers can arise for a number of reasons. Perhaps the outlier is simply
a mistake due to a clerical recording error, in which case you’d want to
replace the incorrect data with the correct data. We’ll call such outliers
measurement outliers, because they simply reflect measurement er-
rors. In a time-series context, if a particular value of a recorded series is
plagued by a measurement outlier, there’s no reason why observations
at other times should necessarily be affected.
Alternatively, outliers in time series may be associated with large unan-
ticipated shocks, the effects of which may certainly linger. If, for exam-
ple, an adverse shock hits the U.S. economy this quarter (e.g., the price
of oil on the world market triples) and the U.S. plunges into a severe
depression, then it’s likely that the depression will persist for some time.
Such outliers are called innovation outliers, because they’re driven by
shocks, or “innovations,” whose effects naturally last more than one pe-
riod due to the dynamics operative in business, economic, and financial
series.

5. (DW from a pure trend model)

Fit a quadratic trend to the liquor sales data, and check the DW statistic.
It looks fine. Why? Are things really fine?

6. (Diagnostic checking of model residuals)

The Durbin-Watson test is invalid in the presence of lagged dependent
variables. Breusch-Godfrey remains valid.
234 CHAPTER 11. SERIAL CORRELATION

a. Durbin’s h test is an alternative to the Durbin-Watson test. As

with the Durbin-Watson test, it’s designed to detect first-order serial
correlation, but it’s valid in the presence of lagged dependent vari-
ables. Do some background reading as well on Durbin’s h test and
report what you learned.
b. Which do you think is likely to be most useful to you in assessing the
properties of residuals from time-series models: the residual correlo-
gram, Durbin’s h test, or the Breusch-Godfrey test? Why?

7. Dynamic logit.
Note that, in a logit regression, one or more of the RHS variables could
be lagged dependent variables, It−i (z), i = 1, 2, ...

8. IC2.1 in time-series.
In cross sections we wrote IC2 as “εi independent of xi ”. We did not
yet have occasion to state IC2 in time series, since we will not introduce
time series until now. In time series IC2 becomes “εt independent of
xt , xt−1 , ...”.

9. Instruments in time-series environments.

In time-series contexts with xt serially correlated, an obvious instrument
for xt is its lag, xt−1 . Due to the serial correlation in x, xt−1 is correlated
with xt , yet xt−1 can’t be correlated with its innovation εt , because the
innovation is independent over time and hence uncorrelated with xt−1 .

10. Event studies for causal estimation.

In Chapter 9 we introduced “diff-in-diff” estimation, which made use
of panel data. Time-series “event studies”, or “synthetic controls”, are
closely related.
In many time series situations we never intervene and do an experiment;
instead we are in the world of “observational studies”, trying to make
11.4. EXERCISES, PROBLEMS AND COMPLEMENTS 235

causal inferences from the historical record. One would of course like
to watch the realizations of two universes, the one that actually oc-
curred with some treatment applied, and a parallel counterfactual uni-
verse without the treatment applied. That’s not possible in general,
but we can approximate the comparison by estimating a model on pre-
treatment data and using it to predict what would have happened in
the absence of the treatment, and comparing it to what happened in the
real data with the treatment.
“Treatment” sounds like active intervention, but again, the treatment is
usually passive in event study contexts. Consider the following example.
We want to know the effect of a new gold discovery on stock returns of a
certain gold mining firm. We can’t just look at the firm’s returns on the
announcement day, because daily stock returns vary greatly for lots of
reasons. Event studies proceed by (1) specifying and estimating a model
for the object of interest (in this case a firm’s daily stock returns) over
the pre-event period, using only pre-event data, 1, ..., T (in this case pre-
announcement data), (2) using the model to predict into the post-event
period T + 1, T + 2, ..., and (3) comparing the post-event forecast to the
post-event realization.
236 CHAPTER 11. SERIAL CORRELATION
Chapter 12

Forecasting

12.1 ***

The “Forecasting the Right-Hand-Side Variables Problem”

For now assume known parameters.

yt = x′t β + εt

=⇒ yt+h = x′t+h β + εt+h

Projecting on current information,

yt+h,t = x′t+h,t β

“Forecasting the right-hand-side variables problem” (FRVP):

We don’t have xt+h,t

But FRVP is not a Problem for Us!

FRVP no problem for trends. Why?

FRVP no problem for seasonals. Why?

FRVP also no problem for autoregressive effects

237
238 CHAPTER 12. FORECASTING

(lagged dependent variables)

e.g., consider a pure AR(1)

yt = ϕyt−1 + εt

yt+h = ϕyt+h−1 + εt+h

yt+h,t = ϕyt+h−1,t

No FRVP for h = 1. There seems to be an FRVP for h > 1.

But there’s not...

We build the multi-step forecast recursively.

First 1-step, then 2-step, etc.
“Wold’s chain rule of forecasting”

Interval and Density Forecasting

Assuming Gaussian shocks, we immediately have

2
yt+h | yt , yt−1 , ... ∼ N (yt+h,t , σt+h,t ).

We know how to get yt+h,t (Wold’s chain rule).

The question is how to get

2
σt+h,t = var(et+h,t ) = var(yt+h − yt+h,t ).

[Of course to make things operational we eventually replace parameters

2
with estimates and use N (ŷt+h,t , σ̂t+h,t ).]
Interval and Density Forecasting
(1-Step-Ahead, AR(1))

yt = ϕyt−1 + εt εt ∼ W N (0, σ 2 )
12.1. *** 239

Back substitution yields

yt = εt + ϕεt−1 + ϕ2 εt−2 + ϕ3 εt−3 + ...

=⇒ yt+1 = εt+1 + ϕεt + ϕ2 εt−1 + ϕ3 εt−2 + ...

Projecting yt+1 on time-t information (εt , εt−1 , ...) gives:

yt+1,t = ϕεt + ϕ2 εt−1 + ϕ3 εt−2 + ...

Corresponding 1-step-ahead error (zero-mean, unforecastable):

et+1,t = yt+1 − yt+1,t = εt+1

with variance
2
σt+1,t = var(et+1,t ) = σ 2

Interval and Density Forecasting

(h-Step-Ahead, AR(p))

yt = ϕ1 yt−1 + ϕ2 yt−2 + ... + ϕp yt−p + εt εt ∼ W N (0, σ 2 )

Back substitution yields

yt = εt + b1 εt−1 + b2 εt−2 + b3 εt−3 + ...

=⇒ yt+h = εt+h + b1 εt+h−1 + ... + bh−1 εt+1 + bh εt + bh+1 εt−1 + ...

(Note that the b′ s are functions of the ϕ′ s.)

Projecting yt+h on time-t information (εt , εt−1 , ...) gives:

yt+h,t = bh εt + bh+1 εt−1 + bh+2 εt−2 + ...

240 CHAPTER 12. FORECASTING

Corresponding h-step-ahead error (zero-mean, unforecastable):

et+h,t = yt+h − yt+h,t = εt+h + b1 εt+h−1 + ... + bh−1 εt+1

with variance (non-decreasing in h):

2
σt+h,t = var(et+h,t ) = σ 2 (1 + b21 + ... + b2h−1 )

Liquor Sales History and

1- Through 12-Month-Ahead Point and Interval Forecasts

From Trend + Seasonal Model with Four Lags of y

Now With Realization Superimposed...

SIC Estimates of Out-of-Sample Forecast Error Variance)

LSALESt → c, T IM Et

SIC = 0.45

LSALESt → c, T IM Et , T IM Et2
12.2. EXERCISES, PROBLEMS AND COMPLEMENTS 241

SIC = 0, 28

LSALESt → T IM Et , T IM Et2 , Dt,1 , ..., Dt,12

SIC = 0.04

LSALESt → T IM Et , T IM Et2 , Dt,1 , ..., Dt,12 , LSALESt−1 , ..., LSALESt−4

SIC = 0.02

(We report exponentiated SIC’s because the software actually reports ln(SIC))

12.2 Exercises, Problems and Complements

1. ****
242 CHAPTER 12. FORECASTING
Chapter 13

Structural Change

Recall the full ideal conditions, one of which was that the model coefficients are fixed. Violations of that
condition are of great concern in time series. The cross-section dummy variables that we already studied
effectively allow for structural change in the cross section (heterogeneity across groups). But structural
change is of special relevance in time series. It can be gradual (Lucas critique, learning, evolution of tastes,
...) or abrupt (e.g., new legislation).
Structural change is related to nonlinearity, because structural change is actually a type of nonlinearity.
Structural change is also related to outliers, because outliers can sometimes be viewed as a kind of structural
change – a quick intercept break and return.
For notational simplicity we consider the case of simple regression throughout, but the ideas extend
immediately to multiple regression.

13.1 Gradual Parameter Evolution

In many cases, parameters may evolve gradually rather than breaking abruptly. Suppose, for example, that

yt = β1t + β2t xt + εt

where
β1t = γ1 + γ2 T IM Et

β2t = δ1 + δ2 T IM Et .

Then we have:
yt = (γ1 + γ2 T IM Et ) + (δ1 + δ2 T IM Et )xt + εt .

We simply run:
yt → c, , T IM Et , xt , T IM Et · xt .

This is yet another important use of dummies. The regression can be used both to test for structural
change (F test of γ2 = δ2 = 0), and to accommodate it if present.

243
244 CHAPTER 13. STRUCTURAL CHANGE

13.2 Abrupt Parameter Breaks

13.2.1 Exogenously-Specified Breaks
Suppose that we don’t know whether a break occurred, but we know that if it did occur, it occurred at time
T ∗.
A Dummy-Variable Approach That is, we entertain the possibility that
(
β11 + β21 xt + εt , t = 1, ..., T ∗
yt =
β12 + β22 xt + εt , t = T ∗ + 1, ..., T

Let (
0, t = 1, ..., T ∗
Dt =
1, t = T ∗ + 1, ...T
Then we can write the model as:

yt = (β11 + (β12 − β11 )Dt ) + (β21 + (β22 − β21 )Dt )xt + εt

We simply run:
yt → c, Dt , xt , Dt · xt

The regression can be used both to test for structural change, and to accommodate it if present. It represents
yet another use of dummies. The no-break null corresponds to the joint hypothesis of zero coefficients on Dt
and Dt · xt , for which an F test is appropriate.
The Chow Test The dummy-variable setup and associated F test above is actually just a laborious way
of calculating the so-called Chow breakpoint test statistic,

(SSRres − SSR)/K
Chow = ,
SSR/(T − 2K)

where SSRres is from the regression using sample t = 1, ..., T and SSR = SSR1 + SSR2 , where SSR1 is
from the regression using sample t = 1, ..., T ∗ and SSR2 is from the regression using sample t = T ∗ + 1, ...T .
Under the IC, Chow is distributed F , with K and T − 2K degrees of freedom.

13.2.2 The Chow test with Endogenous Break Selection

Thus far we have (unrealistically) assumed that the potential break date is known. In practice, potential
break dates are often unknown and are identified by “peeking” at the data. We can capture this phenomenon
in stylized fashion by imagining splitting the sample sequentially at each possible break date, and picking
the split at which the Chow breakpoint test statistic is maximized. Implicitly, that’s what people often do
in practice, even if they don’t always realize or admit it.
The distribution of such a test statistic is not F , as for the traditional Chow breakpoint test statistic.
Rather, the distribution is that of the maximum of many draws from an F , which will be pushed far to the
right of the distribution of a single F draw.
The test statistic is
M axChow = max Chow(τ ),
τ1 ≤τ ≤τ2
13.3. DUMMY VARIABLES AND OMITTED VARIABLES, AGAIN AND AGAIN 245

where τ denotes sample fraction (typically we take τ1 = .15 and τ2 = .85). The distribution of M axChow
has been tabulated.
************************
– Exogenously-specified break in log-linear trend model
– Endogenously-selected break in log-linear trend model
– SIC for best broken log-linear trend model vs. log-quadratic trend model
*********************************

13.3 Dummy Variables and Omitted Variables, Again

and Again
13.3.1 Dummy Variables
Notice that dummy (indicator) variables have arisen repeatedly in our discussions. We used 0-1 dummies to
handle group heterogeneity in cross-sections. We used time dummies to indicate the date in time series. We
used 0-1 seasonal dummies to indicate the season in time series.
Now, in this chapter, we used both (1) time dummies to allow for gradual parameter evolution, and (2)
0-1 dummies to indicate a sharp break date, in time series.

13.3.2 Omitted Variables

Notice that omitted variables have also arisen repeatedly in our discussions.

1. If there are neglected group effects in cross-section regression, we fix the problem (of omitted group
dummies) by including the requisite group dummies.

2. If there is neglected trend or seasonality in time-series regression, we fix the problem (of omitted trend
or seasonal dummies) by including the requisite trend or seasonal dummies.

3. If there is neglected non-linearity, we fix the problem (effectively one of omitted Taylor series terms)
by including the requisite Taylor series terms.

4. If there is neglected structural change in time-series regression, we fix the problem (effectively one of
omitted parameter trend dummies or break dummies) by including the requisite trend dummies or
break dummies.

You can think of the basic “uber-strategy” as ”If some systematic feature of the DGP is missing from the
model, then include it.” That is, if something is missing, then model what’s missing, and then the new uber-
model won’t have anything missing, and all will be well (i.e., the IC will be satisfied). This is an important
recognition. In a subsequent chapter, for example, we’ll study another violation of the IC known as serial
correlation (Chapter ??). The problem amounts to a feature of the DGP neglected by the initially-fitted
model, and we address the problem by incorporating the neglected feature into the model.
246 CHAPTER 13. STRUCTURAL CHANGE

13.4 Recursive Analysis and CUSUM

13.5 Structural Change in Liquor Sales Trend

13.6 Exercises, Problems and Complements

1. Rolling Regression for Generic Structural Change ***
Chapter 14

Vector Autoregression

**************
After introducing Granger causality, introduce time series causal est via event studies.

14.0.1 Event Studies and Synthetic Controls

Event studies have a long tradition in financial economics and have received renewed attention in the past
20 years in applied microeconomics in the form of synthetic controls. Event studies aim to measure the
impact of an event on an outcome variable. For example, this could be a policy change, new information
being disclosed, or an environmental change that is thought to affect the outcome variable.
The basic intuition of an event study is to compare the outcome variable before and after the event in
order to measure its effect. We can think of measuring the effect of an event as measuring treatment effects
in an RCT. If we think of the event as a treatment affecting just some units in a sample but not others,
then after the event has occurred, we can compare the outcome variable between the treated and the control
groups to measure the effect that can be attributed to the event. However, the main challenge in this design
is that we only observe the outcome variable after the event but not the counterfactual of what would have
been the outcome variable in the treated group without the treatment of the event.
To understand how event studies deal with this challenge consider a classic example of earnings an-
nouncements. Imagine you are asked to measure the effect of an earnings announcement on the stock price
of a publicly traded company. In this example, the event is the earnings announcement, and the outcome
variable is the stock price of the company. As we mentioned above, the idea is to compare the evolution
of the stock price before and after the earning announcement. We can easily imagine that if the earning
announcement contains “good news” about the company, then the price will increase due to new investors,
which will increase the demand for the stock and thus the price. For this example, assume the price is higher
after the announcement. However, after the event, it could be that the price increase is due to a demand
increase but not because of the earning announcement; it could be the case that a rival company announced
“bad news” and investors substituted away from the rival and to the company you are studying. Even if we
can say that “good news” are contained in the earnings announcement, how you can be sure that such good
news caused the price increase? Synthetic controls provide a strategy to estimate this causal effect.
The goal is to create a synthetic control, that is, an artificial variable, that reproduces the behavior of
the outcome variable for the treated unit in the absence of the treatment. In our example what we want

247
248 CHAPTER 14. VECTOR AUTOREGRESSION

is to create an artificial time series for the stock price that matches the behavior of the price before the
earnings announcement. If we construct such artificial time series with data from companies unaffected by
the earning announcement, and we construct it in such a way that it matches the behavior of the treated
company prior to treatment, then we can use this constructed time series as a control group post treatment.
The synthetic control will capture how the price would have evolved after the event maintaining most of the
features of the environment except the earnings announcement. Then using the synthetic control, we can
measure the effect of the earning announcement by comparing the observed time series for the stock price
and the artificial price time series from the synthetic control.
This strategy requires that we can find price data for companies that are unaffected by the earnings
announcement and then that their data can be used to construct the synthetic control. In other words,
we need to find good controls, and intuitively, this implies finding companies with similar features to the
company we are studying but for which we can claim the earnings announcement has not effect, that is,
these companies are not treated by the event.
In our example the synthetic control is constructed by combining the price time series of the control
companies. Typically, linear combination weights are carefully selected so that the resulting time series
matches the behavior of the price of the treated company prior to the earnings announcement.
——————————————————————
A univariate autoregression involves one variable. In a univariate autoregression of order p, we regress
a variable on p lags of itself. In contrast, a multivariate autoregression – that is, a vector autoregression, or
V AR – involves N variables. In an N -variable vector autoregression of order p, or V AR(p), we estimate N
different equations. In each equation, we regress the relevant left-hand-side variable on p lags of itself, and
p lags of every other variable.1 Thus the right-hand-side variables are the same in every equation – p lags
of every variable.
The key point is that, in contrast to the univariate case, vector autoregressions allow for cross-variable
dynamics. Each variable is related not only to its own past, but also to the past of all the other variables
in the system. In a two-variable V AR(1), for example, we have two equations, one for each variable (y1 and
y2 ) . We write
y1,t = ϕ11 y1,t−1 + ϕ12 y2,t−1 + ε1,t

y2,t = ϕ21 y1,t−1 + ϕ22 y2,t−1 + ε2,t .

Each variable depends on one lag of the other variable in addition to one lag of itself; that’s one obvious
source of multivariate interaction captured by the V AR that may be useful for forecasting. In addition, the
disturbances may be correlated, so that when one equation is shocked, the other will typically be shocked
as well, which is another type of multivariate interaction that univariate models miss. We summarize the
disturbance variance-covariance structure as

ε1,t ∼ W N (0, σ12 )

ε2,t ∼ W N (0, σ22 )

cov(ε1,t , ε2,t ) = σ12 .

The innovations could be uncorrelated, which occurs when σ12 = 0, but they needn’t be.
1
Trends, seasonals, and other exogenous variables may also be included, as long as they’re all included in
every equation.
14.1. PREDICTIVE CAUSALITY 249

You might guess that V ARs would be hard to estimate. After all, they’re fairly complicated models,
with potentially many equations and many right-hand-side variables in each equation. In fact, precisely the
opposite is true. V ARs are very easy to estimate, because we need only run N linear regressions. That’s one
reason why V ARs are so popular – OLS estimation of autoregressive models is simple and stable. Equation-
by-equation OLS estimation also turns out to have very good statistical properties when each equation has
the same regressors, as is the case in standard V ARs. Otherwise, a more complicated estimation procedure
called seemingly unrelated regression, which explicitly accounts for correlation across equation disturbances,
would be required to obtain estimates with good statistical properties.
When fitting V AR’s to data, we ca use the Schwarz criterion, just as in the univariate case. The formula
differs, however, because we’re now working with a multivariate system of equations rather than a single
equation. To get an SIC value for a V AR system, we could add up the equation-by-equation SIC’s, but
unfortunately, doing so is appropriate only if the innovations are uncorrelated across equations, which is
a very special and unusual situation. Instead, explicitly multivariate versions of information criteria are
required, which account for cross-equation innovation correlation. We interpret the SIC values computed
for V ARs of various orders in exactly the same way as in the univariate case: we select that order p such
that SIC is minimized.
We construct V AR forecasts in a way that precisely parallels the univariate case. We can construct
1-step-ahead point forecasts immediately, because all variables on the right-hand side are lagged by one
period. Armed with the 1-step-ahead forecasts, we can construct the 2-step-ahead forecasts, from which we
can construct the 3-step-ahead forecasts, and so on in the usual way, following the chain rule of forecasting.
We construct interval and density forecasts in ways that also parallel the univariate case. The multivariate
nature of V AR’s makes the derivations more tedious, however, so we bypass them. As always, to construct
practical forecasts we replace unknown parameters by estimates.

14.1 Predictive Causality

There’s an important statistical notion of causality that’s intimately related to forecasting and naturally
introduced in the context of V AR’s. It is based on two key principles: first, cause should occur before
effect, and second, a causal series should contain information useful for forecasting that is not available in
the other series (including the past history of the variable being forecast). In the unrestricted V AR’s that
we’ve studied thus far, everything causes everything else, because lags of every variable appear on the right
of every equation. Cause precedes effect because the right-hand-side variables are lagged, and each variable
is useful in forecasting every other variable.
We stress from the outset that the notion of predictive causality contains little if any information about
causality in the philosophical sense. Rather, the statement “yi causes yj ” is just shorthand for the more
precise, but long-winded, statement, “ yi contains useful information for predicting yj (in the linear least
squares sense), over and above the past histories of the other variables in the system.” To save space, we
simply say that yi causes yj .
To understand what predictive causality means in the context of a V AR(p), consider the j-th equation
of the N -equation system, which has yj on the left and p lags of each of the N variables on the right. If yi
causes yj , then at least one of the lags of yi that appear on the right side of the yj equation must have a
nonzero coefficient.
It’s also useful to consider the opposite situation, in which yi does not cause yj . In that case, all of the lags
250 CHAPTER 14. VECTOR AUTOREGRESSION

of that yi that appear on the right side of the yj equation must have zero coefficients.2 Statistical causality
tests are based on this formulation of non-causality. We use an F -test to assess whether all coefficients on
lags of yi are jointly zero.
Note that we’ve defined non-causality in terms of 1-step-ahead prediction errors. In the bivariate V AR,
this implies non-causality in terms of h-step-ahead prediction errors, for all h. (Why?) In higher dimensional
cases, things are trickier; 1-step-ahead noncausality does not necessarily imply noncausality at other horizons.
For example, variable i may 1-step cause variable j, and variable j may 1-step cause variable k. Thus, variable
i 2-step causes variable k, but does not 1-step cause variable k.
Causality tests are often used when building and assessing forecasting models, because they can inform
us about those parts of the workings of complicated multivariate models that are particularly relevant for
forecasting. Just staring at the coefficients of an estimated V AR (and in complicated systems there are many
coefficients) rarely yields insights into its workings. Thus we need tools that help us to see through to the
practical forecasting properties of the model that concern us. And we often have keen interest in the answers
to questions such as “Does yi contribute toward improving forecasts of yj ?,” and “Does yj contribute toward
improving forecasts of yi ?” If the results violate intuition or theory, then we might scrutinize the model
more closely. In a situation in which we can’t reject a certain noncausality hypothesis, and neither intuition
nor theory makes us uncomfortable with it, we might want to impose it, by omitting certain lags of certain
variables from certain equations.
Various types of causality hypotheses are sometimes entertained. In any equation (the j-th, say), we’ve
already discussed testing the simple noncausality hypothesis that:

(a) No lags of variable i aid in one-step-ahead prediction of variable j.

We can broaden the idea, however. Sometimes we test stronger noncausality hypotheses such as:

1. No lags of a set of other variables aid in one-step-ahead prediction of variable j.

2. No lags of any other variables aid in one-step-ahead prediction of variable j.

3. No variable in a set A causes any variable in a set B, in which case we say that the variables in A are
block non-causal for those in B.

14.2 Application: Housing Starts and Completions

We estimate a bivariate V AR for U.S. seasonally-adjusted housing starts and completions, two widely-
watched business cycle indicators, 1968.01-1996.06. We use the V AR to produce point extrapolation fore-
casts. We show housing starts and completions in Figure 14.1. Both are highly cyclical, increasing during
business-cycle expansions and decreasing during contractions. Moreover, completions tend to lag behind
starts, which makes sense because a house takes time to complete.
We split the data into an estimation sample, 1968.01-1991.12, and a hold-out sample, 1992.01-1996.06
for forecasting. We therefore perform all model specification analysis and estimation, to which we now turn,
on the 1968.01-1991.12 data. We show the starts correlogram in Table 14.2 and Figure 14.3. The sample
2
Note that in such a situation the error variance in forecasting yj using lags of all variables in the system
will be the same as the error variance in forecasting yj using lags of all variables in the system except yi .
14.2. APPLICATION: HOUSING STARTS AND COMPLETIONS 251

Figure 14.1: Housing Starts and Completions, 1968 - 1996

Figure 14.2: Housing Starts Correlogram

autocorrelation function decays slowly, whereas the sample partial autocorrelation function appears to cut
off at displacement 2. The patterns in the sample autocorrelations and partial autocorrelations are highly
statistically significant, as evidenced by both the Bartlett standard errors and the Ljung-Box Q-statistics.
The completions correlogram, in Table 14.4 and Figure 14.5, behaves similarly.
We’ve not yet introduced the cross correlation function. There’s been no need, because it’s not relevant
for univariate modeling. It provides important information, however, in the multivariate environments that
now concern us. Recall that the autocorrelation function is the correlation between a variable and lags of
itself. The cross-correlation function is a natural multivariate analog; it’s simply the correlation between a
252 CHAPTER 14. VECTOR AUTOREGRESSION

Figure 14.3: Housing Starts Autocorrelations and Partial Autocorrelations

variable and lags of another variable. We estimate those correlations using the usual estimator and graph
them as a function of displacement along with the Bartlett two- standard-error bands, which apply just as
in the univariate case.
The cross-correlation function (Figure 14.6) for housing starts and completions is very revealing. Starts
and completions are highly correlated at all displacements, and a clear pattern emerges as well: although
the contemporaneous correlation is high (.78), completions are maximally correlated with starts lagged by
roughly 6-12 months (around .90). Again, this makes good sense in light of the time it takes to build a
house.
Now we proceed to model starts and completions. We need to select the order, p, of our V AR(p). Based
on exploration using SIC, we adopt a V AR(4).
First consider the starts equation (Table 14.7a), residual plot (Figure 14.7b), and residual correlogram
(Table 14.8, Figure 14.9). The explanatory power of the model is good, as judged by the R2 as well as
the plots of actual and fitted values, and the residuals appear white, as judged by the residual sample
autocorrelations, partial autocorrelations, and Ljung-Box statistics. Note as well that no lag of completions
has a significant effect on starts, which makes sense – we obviously expect starts to cause completions,
but not conversely. The completions equation (Table 14.10a), residual plot (Figure 14.10b), and residual
correlogram (Table 14.11, Figure 14.12) appear similarly good. Lagged starts, moreover, most definitely
14.2. APPLICATION: HOUSING STARTS AND COMPLETIONS 253

Figure 14.4: Housing Completions Correlogram

have a significant effect on completions.

Table 14.13 shows the results of formal causality tests. The hypothesis that starts don’t cause completions
is simply that the coefficients on the four lags of starts in the completions equation are all zero. The F -statistic
is overwhelmingly significant, which is not surprising in light of the previously-noticed highly-significant t-
statistics. Thus we reject noncausality from starts to completions at any reasonable level. Perhaps more
surprising, we also reject noncausality from completions to starts at roughly the 5% level. Thus the causality
appears bi-directional, in which case we say there is feedback.
254 CHAPTER 14. VECTOR AUTOREGRESSION

Figure 14.5: Housing Completions Autocorrelations and Partial Autocorrelations

Figure 14.6: Housing Starts and Completions Sample Cross Correlations

14.2. APPLICATION: HOUSING STARTS AND COMPLETIONS 255

(a) VAR Starts Equation

(b) VAR Starts Equation - Residual Plot

Figure 14.7: VAR Starts Model

256 CHAPTER 14. VECTOR AUTOREGRESSION

Figure 14.8: VAR Starts Residual Correlogram

14.2. APPLICATION: HOUSING STARTS AND COMPLETIONS 257

Figure 14.9: VAR Starts Equation - Sample Autocorrelation and Partial Autocorrelation
258 CHAPTER 14. VECTOR AUTOREGRESSION

(a) VAR Completions Equation

(b) VAR Completions Equation - Residual Plot

Figure 14.10: VAR Completions Model

14.2. APPLICATION: HOUSING STARTS AND COMPLETIONS 259

Figure 14.11: VAR Completions Residual Correlogram

260 CHAPTER 14. VECTOR AUTOREGRESSION

Figure 14.12: VAR Completions Equation - Sample Autocorrelation and Partial Autocorre-
lation
14.2. APPLICATION: HOUSING STARTS AND COMPLETIONS 261

Figure 14.13: Housing Starts and Completions - Causality Tests

Finally, we construct forecasts for the out-of-sample period, 1992.01-1996.06. The starts forecast appears
in Figure 14.14. Starts begin their recovery before 1992.01, and the V AR projects continuation of the
recovery. The V AR forecasts captures the general pattern quite well, but it forecasts quicker mean reversion
than actually occurs, as is clear when comparing the forecast and realization in Figure 14.15. The figure
also makes clear that the recovery of housing starts from the recession of 1990 was slower than the previous
recoveries in the sample, which naturally makes for difficult forecasting. The completions forecast suffers the
same fate, as shown in Figures 14.16 and 14.17. Interestingly, however, completions had not yet turned by
1991.12, but the forecast nevertheless correctly predicts the turning point. (Why?)
262 CHAPTER 14. VECTOR AUTOREGRESSION

Figure 14.14: Housing Starts Forecast

Figure 14.15: Housing Starts Forecast and Realization

14.2. APPLICATION: HOUSING STARTS AND COMPLETIONS 263

Figure 14.16: Housing Completions Forecast

Figure 14.17: Housing Completions Forecast and Realization

264 CHAPTER 14. VECTOR AUTOREGRESSION

14.3 Exercises, Problems and Complements

1. Housing starts and completions, continued.
Our VAR analysis of housing starts and completions, as always, involved many judgment calls. Using
the starts and completions data, assess the adequacy of our models and forecasts. Among other things,
you may want to consider the following questions:

a. Should we allow for a trend in the forecasting model?

b. How do the results change if, in light of the results of the causality tests, we exclude lags of
completions from the starts equation, re-estimate by seemingly-unrelated regression, and forecast?
c. Are the VAR forecasts of starts and completions more accurate than univariate forecasts?

2. Comparative forecasting performance of V ARs and univariate models.

Using the housing starts and completions data on the book’s website, compare the forecasting perfor-
mance of the VAR used in this chapter to that of the obvious competitor: univariate autoregressions.
Use the same in-sample and out-of-sample periods as in the chapter. Why might the forecasting per-
formance of the V AR and univariate methods differ? Why might you expect the V AR completions
forecast to outperform the univariate autoregression, but the V AR starts forecast to be no better
than the univariate autoregression? Do your results support your conjectures?
Chapter 15

Dynamic Heteroskedasticity

Recall the full ideal conditions.

The celebrated Wold decomposition makes clear that every covariance stationary series may be viewed
as ultimately driven by underlying weak white noise innovations. Hence it is no surprise that every model
discussed in this book is driven by underlying white noise. To take a simple example, if the series yt follows an
AR(1) process, then yt = ϕyt−1 + εt , where εt is white noise. In some situations it is inconsequential whether
εt is weak or strong white noise, that is, whether εt is independent, as opposed to merely serially uncorrelated.
Hence, to simplify matters we sometimes assume strong white noise, εt ∼ iid(0, σ 2 ). Throughout this book,
we have thus far taken that approach, sometimes explicitly and sometimes implicitly.
When εt is independent, there is no distinction between the unconditional distribution of εt and the
distribution of εt conditional upon its past, by definition of independence. Hence σ 2 is both the unconditional
and conditional variance of εt . The Wold decomposition, however, does not require that εt be serially
independent; rather it requires only that εt be serially uncorrelated.
If εt is dependent, then its unconditional and conditional distributions will differ. We denote the un-
conditional innovation distribution by εt ∼ (0, σ 2 ). We are particularly interested in conditional dynamics
characterized by heteroskedasticity, or time-varying volatility. Hence we denote the conditional distribu-
tion by εt |Ωt−1 ∼ (0, σt2 ), where Ωt−1 = εt−1 , εt−2 , .... The conditional variance σt2 will in general evolve as
Ωt−1 evolves, which focuses attention on the possibility of time-varying innovation volatility.1
Allowing for time-varying volatility is crucially important in certain economic and financial contexts.
The volatility of financial asset returns, for example, is often time-varying. That is, markets are sometimes
tranquil and sometimes turbulent, as can readily be seen by examining the time series of stock market
returns in Figure 1, to which we shall return in detail. Time-varying volatility has important implications
for financial risk management, asset allocation and asset pricing, and it has therefore become a central
part of the emerging field of financial econometrics. Quite apart from financial applications, however,
time-varying volatility also has direct implications for interval and density forecasting in a wide variety
of applications: correct confidence intervals and density forecasts in the presence of volatility fluctuations
require time-varying confidence interval widths and time-varying density forecast spreads. The models that
we have considered thus far, however, do not allow for that possibility. In this chapter we do so.
1
In principle, aspects of the conditional distribution other than the variance, such as conditional skewness,
could also fluctuate. Conditional variance fluctuations are by far the most important in practice, however,
so we assume that fluctuations in the conditional distribution of ε are due exclusively to fluctuations in σt2 .

265
266 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY

15.1 The Basic ARCH Process

Consider the general linear process,

yt = B(L)εt
∞
X
B(L) = bi Li
i=0
∞
X
b2i < ∞
i=0

b0 = 1

εt ∼ W N (0, σ 2 ).
We will work with various cases of this process.
Suppose first that εt is strong white noise, εt ∼ iid(0, σ 2 ). Let us review some results already discussed
for the general linear process, which will prove useful in what follows. The unconditional mean and variance
of y are

E(yt ) = 0

and
∞
X
E(yt2 ) = σ 2 b2i ,
i=0

which are both time-invariant, as must be the case under covariance stationarity. However, the condi-
tional mean of y is time-varying:
∞
X
E(yt |Ωt−1 ) = bi εt−i ,
i=1

where the information set is

Ωt−1 = εt−1 , εt−2 , ....

The ability of the general linear process to capture covariance stationary conditional mean dynamics is
the source of its power.
Because the volatility of many economic time series varies, one would hope that the general linear
process could capture conditional variance dynamics as well, but such is not the case for the model as
presently specified: the conditional variance of y is constant at

E (yt − E(yt |Ωt−1 ))2 |Ωt−1 = σ 2 .

This potentially unfortunate restriction manifests itself in the properties of the h-step-ahead conditional
prediction error variance. The minimum mean squared error forecast is the conditional mean,
∞
X
E(yt+h |Ωt ) = bh+i εt−i ,
i=0
15.1. THE BASIC ARCH PROCESS 267

and so the associated prediction error is

h−1
X
yt+h − E(yt+h |Ωt ) = bi εt+h−i ,
i=0

which has a conditional prediction error variance of

  h−1
X
2 2
E (yt+h − E(yt+h |Ωt )) |Ωt = σ b2i .
i=0

The conditional prediction error variance is different from the unconditional variance, but it is not time-
varying: it depends only on h, not on the conditioning information Ωt . In the process as presently specified,
the conditional variance is not allowed to adapt to readily available and potentially useful conditioning
information.
So much for the general linear process with iid innovations. Now we extend it by allowing εt to be weak
rather than strong white noise, with a particular nonlinear dependence structure. In particular, suppose that,
as before,

yt = B(L)εt
∞
X
B(L) = bi Li
i=0
∞
X
b2i < ∞
i=0

b0 = 1,

but now suppose as well that

εt |Ωt−1 ∼ N (0, σt2 )

σt2 = ω + γ(L)ε2t
p
X X
ω > 0γ(L) = γi Li γi ≥ 0f oralli γi < 1.
i=1

Note that we parameterize the innovation process in terms of its conditional density,

εt |Ωt−1 ,

which we assume to be normal with a zero conditional mean and a conditional variance that depends
linearly on p past squared innovations. εt is serially uncorrelated but not serially independent, because the
current conditional variance σt2 depends on the history of εt .2 The stated regularity conditions are sufficient
to ensure that the conditional and unconditional variances are positive and finite, and that yt is covariance
stationary.
2
In particular, σt2 depends on the previous p values of εt via the distributed lag

γ(L)ε2t .
268 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY

The unconditional moments of εt are constant and are given by

E(εt ) = 0

and

2 ω
E(εt − E(εt )) = P .
1− γi
The important result is not the particular formulae for the unconditional mean and variance, but the
fact that they are fixed, as required for covariance stationarity. As for the conditional moments of εt , its
conditional variance is time-varying,

E (εt − E(εt |Ωt−1 ))2 |Ωt−1 = ω + γ(L)ε2t ,

and of course its conditional mean is zero by construction.

Assembling the results to move to the unconditional and conditional moments of y as opposed to εt ,
it is easy to see that both the unconditional mean and variance of y are constant (again, as required by
covariance stationarity), but that both the conditional mean and variance are time-varying:
∞
X
E(yt |Ωt−1 ) = bi εt−i
i=1

E (yt − E(yt |Ωt−1 ))2 |Ωt−1 = ω + γ(L)ε2t .

Thus, we now treat conditional mean and variance dynamics in a symmetric fashion by allowing for
movement in each, as determined by the evolving information set Ωt−1 . In the above development, εt
is called an ARCH(p) process, and the full model sketched is an infinite-ordered moving average with
ARCH(p) innovations, where ARCH stands for autoregressive conditional heteroskedasticity. Clearly εt is
conditionally heteroskedastic, because its conditional variance fluctuates. There are many models of condi-
tional heteroskedasticity, but most are designed for cross-sectional contexts, such as when the variance of a
cross-sectional regression disturbance depends on one or more of the regressors.3 However, heteroskedasticity
is often present as well in the time-series contexts relevant for forecasting, particularly in financial markets.
The particular conditional variance function associated with the ARCH process,

σt2 = ω + γ(L)ε2t ,

is tailor-made for time-series environments, in which one often sees volatility clustering, such that
large changes tend to be followed by large changes, and small by small, of either sign. That is, one may
see persistence, or serial correlation, in volatility dynamics (conditional variance dynamics), quite apart
from persistence (or lack thereof) in conditional mean dynamics. The ARCH process approximates volatility
dynamics in an autoregressive fashion; hence the name autoregressiveconditional heteroskedasticity. To un-
derstand why, note that the ARCH conditional variance function links today’s conditional variance positively
to earlier lagged ε2t ’s, so that large ε2t ’s in the recent past produce a large conditional variance today, thereby
increasing the likelihood of a large ε2t today. Hence ARCH processes are to conditional variance dynamics
3
The variance of the disturbance in a model of household expenditure, for example, may depend on
income.
15.2. THE GARCH PROCESS 269

precisely as standard autoregressive processes are to conditional mean dynamics. The ARCH process may be
viewed as a model for the disturbance in a broader model, as was the case when we introduced it above as a
model for the innovation in a general linear process. Alternatively, if there are no conditional mean dynamics
of interest, the ARCH process may be used for an observed series. It turns out that financial asset returns
often have negligible conditional mean dynamics but strong conditional variance dynamics; hence in much
of what follows we will view the ARCH process as a model for an observed series, which for convenience we
will sometimes call a “return.”

15.2 The GARCH Process

Thus far we have used an ARCH(p) process to model conditional variance dynamics. We now introduce the
GARCH(p,q) process (GARCH stands for generalized ARCH), which we shall subsequently use almost
exclusively. As we shall see, GARCH is to ARCH (for conditional variance dynamics) as ARMA is to AR
(for conditional mean dynamics).
The pure GARCH(p,q) process is given by4
yt = εt

εt |Ωt−1 ∼ N (0, σt2 )

σt2 = ω + α(L)ε2t + β(L)σt2

p
X q
X
α(L) = αi Li , β(L) = βi Li
i=1 i=1
X X
ω > 0, αi ≥ 0, βi ≥ 0, αi + βi < 1.

The stated conditions ensure that the conditional variance is positive and that yt is covariance stationary.
Back substitution on σt2 reveals that the GARCH(p,q) process can be represented as a restricted infinite-
ordered ARCH process,
∞
ω α(L) 2 ω X
σt2 = + εt = P + δi ε2 ,
1 − βi i=1 t−i
P
1− βi 1 − β(L)

which precisely parallels writing an ARMA process as a restricted infinite-ordered AR. Hence the
GARCH(p,q) process is a parsimonious approximation to what may truly be infinite-ordered ARCH volatility
dynamics.
It is important to note a number of special cases of the GARCH(p,q) process. First, of course, the
ARCH(p) process emerges when

β(L) = 0.

Second, if both α(L) and β(L) are zero, then the process is simply iid Gaussian noise with variance ω.
Hence, although ARCH and GARCH processes may at first appear unfamiliar and potentially ad hoc, they
4
By “pure” we mean that we have allowed only for conditional variance dynamics, by setting yt = εt . We
could of course also introduce conditional mean dynamics, but doing so would only clutter the discussion
while adding nothing new.
270 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY

are in fact much more general than standard iid white noise, which emerges as a potentially highly-restrictive
special case.
Here we highlight some important properties of GARCH processes. All of the discussion of course applies
as well to ARCH processes, which are special cases of GARCH processes. First, consider the second-order
moment structure of GARCH processes. The first two unconditional moments of the pure GARCH process
are constant and given by

E(εt ) = 0

and

ω
E(εt − E(εt ))2 = P P ,
1− αi − βi
while the conditional moments are

E(εt |Ωt−1 ) = 0

and of course

E (εt − E(εt |Ωt−1 ))2 |Ωt−1 = ω + α(L)ε2t + β(L)σt2 .

In particular, the unconditional variance is fixed, as must be the case under covariance stationarity, while
the conditional variance is time-varying. It is no surprise that the conditional variance is time-varying – the
GARCH process was of course designed to allow for a time-varying conditional variance – but it is certainly
worth emphasizing: the conditional variance is itself a serially correlated time series process.
Second, consider the unconditional higher-order (third and fourth) moment structure of GARCH pro-
cesses. Real-world financial asset returns, which are often modeled as GARCH processes, are typically
unconditionally symmetric but leptokurtic (that is, more peaked in the center and with fatter tails than a
normal distribution). It turns out that the implied unconditional distribution of the conditionally Gaus-
sian GARCH process introduced above is also symmetric and leptokurtic. The unconditional leptokurtosis
of GARCH processes follows from the persistence in conditional variance, which produces clusters of “low
volatility” and “high volatility” episodes associated with observations in the center and in the tails of the
unconditional distribution, respectively. Both the unconditional symmetry and unconditional leptokurtosis
agree nicely with a variety of financial market data.
Third, consider the conditional prediction error variance of a GARCH process, and its dependence on the
conditioning information set. Because the conditional variance of a GARCH process is a serially correlated
random variable, it is of interest to examine the optimal h-step-ahead prediction, prediction error, and
conditional prediction error variance. Immediately, the h-step-ahead prediction is

E(εt+h |Ωt ) = 0,

and the corresponding prediction error is

εt+h − E(εt+h |Ωt ) = εt+h .

This implies that the conditional variance of the prediction error,

15.2. THE GARCH PROCESS 271

E (εt+h − E(εt+h |Ωt ))2 |Ωt = E(ε2t+h |Ωt ),

depends on both h and

Ωt ,

because of the dynamics in the conditional variance. Simple calculations reveal that the expression for the
GARCH(p, q) process is given by

h−2
!
X
E(ε2t+h |Ωt ) =ω (α(1) + β(1)) i
+ (α(1) + β(1))h−1 σt+1
2
.
i=0

In the limit, this conditional variance reduces to the unconditional variance of the process,

ω
lim E(ε2t+h |Ωt ) = .
h→∞ 1 − α(1) − β(1)

For finite h, the dependence of the prediction error variance on the current information set Ωt can be
exploited to improve interval and density forecasts.
Fourth, consider the relationship between ε2t and σt2 . The relationship is important: GARCH dynamics
in σt2 turn out to introduce ARMA dynamics in ε2t .5 More precisely, if εt is a GARCH(p,q) process, then

ε2t

has the ARMA representation

ε2t = ω + (α(L) + β(L))ε2t − β(L)νt + νt ,

where
νt = ε2t − σt2

is the difference between the squared innovation and the conditional variance at time t. To see this, note
that if εt is GARCH(p,q), then
σt2 = ω + α(L)ε2t + β(L)σt2 .

Adding and subtracting

β(L)ε2t

from the right side gives

σt2 = ω + α(L)ε2t + β(L)ε2t − β(L)ε2t + β(L)σt2

= ω + (α(L) + β(L))ε2t − β(L)(ε2t − σt2 ).

5
Put differently, the GARCH process approximates conditional variance dynamics in the same way that
an ARMA process approximates conditional mean dynamics.
272 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY

Adding
ε2t

to each side then gives

σt2 + ε2t = ω + (α(L) + β(L))ε2t − β(L)(ε2t − σt2 ) + ε2t ,

so that
ε2t = ω + (α(L) + β(L))ε2t − β(L)(ε2t − σt2 ) + (ε2t − σt2 ),

= ω + (α(L) + β(L))ε2t − β(L)νt + νt.

Thus,
ε2t

is an ARMA((max(p,q)), p) process with innovation νt , where

νt ∈ [−σt2 , ∞).

ε2t is covariance stationary if the roots of α(L)+β(L)=1 are outside the unit circle.
Fifth, consider in greater depth the similarities and differences between σt2 and

ε2t .

It is worth studying closely the key expression,

νt = ε2t − σt2 ,

which makes clear that

ε2t

is effectively a “proxy” for σt2 , behaving similarly but not identically, with νt being the difference, or error.
In particular, ε2t is a noisy proxy: ε2t is an unbiased estimator of σt2 , but it is more volatile. It seems
reasonable, then, that reconciling the noisy proxy ε2t and the true underlying σt2 should involve some sort
of smoothing of ε2t . Indeed, in the GARCH(1,1) case σt2 is precisely obtained by exponentially smoothing
ε2t . To see why, consider the exponential smoothing recursion, which gives the current smoothed value as a
convex combination of the current unsmoothed value and the lagged smoothed value,

ε̄2t = γε2t + (1 − γ)ε̄2t−1 .

Back substitution yields an expression for the current smoothed value as an exponentially weighted
moving average of past actual values:
X
ε̄2t = wj ε2t−j ,

where
wj = γ(1 − γ)j .

Now compare this result to the GARCH(1,1) model, which gives the current volatility as a linear com-
bination of lagged volatility and the lagged squared return, σt2 = ω + αε2t−1 + βσt−1
2
.
2 ω j−1 2
P
Back substitution yields σt = 1−β + α β εt−j , so that the GARCH(1,1) process gives current volatil-
15.3. EXTENSIONS OF ARCH AND GARCH MODELS 273

ity as an exponentially weighted moving average of past squared returns.

Sixth, consider the temporal aggregation of GARCH processes. By temporal aggregation we mean
aggregation over time, as for example when we convert a series of daily returns to weekly returns, and
then to monthly returns, then quarterly, and so on. It turns out that convergence toward normality under
temporal aggregation is a feature of real-world financial asset returns. That is, although high-frequency (e.g.,
daily) returns tend to be fat-tailed relative to the normal, the fat tails tend to get thinner under temporal
aggregation, and normality is approached. Convergence to normality under temporal aggregation is also
a property of covariance stationary GARCH processes. The key insight is that a low-frequency change is
simply the sum of the corresponding high-frequency changes; for example, an annual change is the sum of
the internal quarterly changes, each of which is the sum of its internal monthly changes, and so on. Thus,
if a Gaussian central limit theorem can be invoked for sums of GARCH processes, convergence to normality
under temporal aggregation is assured. Such theorems can be invoked if the process is covariance stationary.
In closing this section, it is worth noting that the symmetry and leptokurtosis of the unconditional distri-
bution of the GARCH process, as well as the disappearance of the leptokurtosis under temporal aggregation,
provide nice independent confirmation of the accuracy of GARCH approximations to asset return volatility
dynamics, insofar as GARCH was certainly not invented with the intent of explaining those features of
financial asset return data. On the contrary, the unconditional distributional results emerged as unantici-
pated byproducts of allowing for conditional variance dynamics, thereby providing a unified explanation of
phenomena that were previously believed unrelated.

15.3 Extensions of ARCH and GARCH Models

There are numerous extensions of the basic GARCH model. In this section, we highlight several of the most
important. One important class of extensions allows for asymmetric response; that is, it allows for last
period’s squared return to have different effects on today’s volatility, depending on its sign.6 Asymmetric
response is often present, for example, in stock returns.

15.3.1 Asymmetric Response

The simplest GARCH model allowing for asymmetric response is the threshold GARCH, or TGARCH,
model.7 We replace the standard GARCH conditional variance function, σt2 = ω + αε2t−1 + βσt−1 2
, with
1, if εt < 0
σt2 = ω + αε2t−1 + γε2t−1 Dt−1 + βσt−1
2
, where Dt = .
0otherwise.
The dummy variable D keeps track of whether the lagged return is positive or negative. When the lagged
return is positive (good news yesterday), D=0, so the effect of the lagged squared return on the current
conditional variance is simply α. In contrast, when the lagged return is negative (bad news yesterday),
D=1, so the effect of the lagged squared return on the current conditional variance is α + γ. If γ = 0, the
response is symmetric and we have a standard GARCH model, but if γ ̸=0 we have asymmetric response of
volatility to news. Allowance for asymmetric response has proved useful for modeling “leverage effects” in
6
In the GARCH model studied thus far, only the square of last period’s return affects the current condi-
tional variance; hence its sign is irrelevant.
7
For expositional convenience, we will introduce all GARCH extensions in the context of GARCH(1,1),
which is by far the most important case for practical applications. Extensions to the GARCH(p,q) case are
immediate but notationally cumbersome.
274 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY

stock returns, which occur when γ <0.8 Asymmetric response may also be introduced via the exponential
GARCH (EGARCH) model,

ln(σt2 ) = ω + α ε t−1 + γε t−1

2
+ β ln(σt−1 ).
σt−1 σt−1

Note that volatility is driven by both size and sign of shocks; hence the model allows for an asymmetric
response depending on the sign of news.9 The log specification also ensures that the conditional variance
is automatically positive, because σt2 is obtained by exponentiating ln(σt2 ) ; hence the name “exponential
GARCH.”

15.3.2 Exogenous Variables in the Volatility Function

Just as ARMA models of conditional mean dynamics can be augmented to include the effects of exogenous
variables, so too can GARCH models of conditional variance dynamics.
We simply modify the standard GARCH volatility function in the obvious way, writing

σt2 = ω + αε2t−1 + βσt−1

2
+ γxt ,

where γ is a parameter and x is a positive exogenous variable.10 Allowance for exogenous variables in the
conditional variance function is sometimes useful. Financial market volume, for example, often helps to
explain market volatility.

15.3.3 Regression with GARCH disturbances and GARCH-M

Just as ARMA models may be viewed as models for disturbances in regressions, so too may GARCH models.
We write
yt = β0 + β1 xt + εt

εt |Ωt−1 ∼ N (0, σt2 )

σt2 = ω + αε2t−1 + βσt−1

2
. Consider now a regression model with GARCH disturbances of the usual sort,
with one additional twist: the conditional variance enters as a regressor, thereby affecting the conditional
mean. We write
yt = β0 + β1 xt + γσt2 + εt

εt |Ωt−1 ∼ N (0, σt2 )

σt2 = ω + αε2t−1 + βσt−1

2
. This model, which is a special case of the general regression model with GARCH
disturbances, is called GARCH-in-Mean (GARCH-M). It is sometimes useful in modeling the relationship
between risks and returns on financial assets when risk, as measured by the conditional variance, varies.11
8
Negative shocks appear to contribute more to stock market volatility than do positive shocks. This is
called the leverage effect, because a negative shock to the market value of equity increases the aggregate
debt/equity ratio (other things the same), thereby increasing leverage.
9
The absolute “size” of news is captured by |rt−1 /σt−1 | , and the sign is captured by rt−1 /σt−1 .
10
Extension to allow multiple exogenous variables is straightforward.
11
One may also allow the conditional standard deviation, rather than the conditional variance, to enter
the regression.
15.4. ESTIMATING, FORECASTING AND DIAGNOSING GARCH MODELS 275

15.3.4 Component GARCH

Note that the standard GARCH(1,1) process may be written as (σt2 − ω̄) = α(ε2t−1 − ω̄) + β(σt−1 2
− ω̄), where
ω 12
ω̄ = 1−α−β is the unconditional variance. This is precisely the GARCH(1,1) model introduced earlier,
rewritten in a slightly different but equivalent form. In this model, short-run volatility dynamics are governed
by the parameters α and β, and there are no long-run volatility dynamics, because ω̄ is constant. Sometimes
we might want to allow for both long-run and short-run, or persistent and transient, volatility dynamics
in addition to the short-run volatility dynamics already incorporated. To do this, we replace ω̄ with a
time-varying process, yielding (σt2 − qt ) = α(ε2t−1 − qt−1 ) + β(σt−1
2
− qt−1 ), where the time-varying long-run
2 2
volatility, qt , is given by qt = ω + ρ(qt−1 − ω) + ϕ(εt−1 − σt−1 ). This “component GARCH” model effectively
lets us decompose volatility dynamics into long-run (persistent) and short-run (transitory) components, which
sometimes yields useful insights. The persistent dynamics are governed by ρ , and the transitory dynamics
are governed by α and β.13

15.3.5 Mixing and Matching

In closing this section, we note that the different variations and extensions of the GARCH process may of
course be mixed. As an example, consider the following conditional variance function: (σt2 − qt ) = α(ε2t−1 − qt−1 ) + γ(ε2t−1 − qt
This is a component GARCH specification, generalized to allow for asymmetric response of volatility to news
via the sign dummy D, as well as effects from the exogenous variable x.

15.4 Estimating, Forecasting and Diagnosing GARCH

Models
Recall that the likelihood function is the joint density function of the data, viewed as a function of the
model parameters, and that maximum likelihood estimation finds the parameter values that maximize the
likelihood function. This makes good sense: we choose those parameter values that maximize the likelihood of
obtaining the data that were actually obtained. It turns out that construction and evaluation of the likelihood
function is easily done for GARCH models, and maximum likelihood has emerged as the estimation method
of choice.14 No closed-form expression exists for the GARCH maximum likelihood estimator, so we must
maximize the likelihood numerically.15 Construction of optimal forecasts of GARCH processes is simple. In
fact, we derived the key formula earlier but did not comment extensively on it. Recall, in particular, that

h−1
!
X i h−1
2
= E ε2t+h |Ωt = ω 2
 
σt+h,t [α(1) + β(1)] + [α(1) + β(1)] σt+1 .
i=1

12
ω̄ is sometimes called the “long-run” variance, referring to the fact that the unconditional variance is
the long-run average of the conditional variance.
13
It turns out, moreover, that under suitable conditions the component GARCH model introduced here is
covariance stationary, and equivalent to a GARCH(2,2) process subject to certain nonlinear restrictions on
its parameters.
14
The precise form of the likelihood is complicated, and we will not give an explicit expression here, but
it may be found in various of the surveys mentioned in the Notes at the end of the chapter.
15
Routines for maximizing the GARCH likelihood are available in a number of modern software packages
such as Eviews. As with any numerical optimization, care must be taken with startup values and convergence
criteria to help insure convergence to a global, as opposed to merely local, maximum.
276 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY

In words, the optimal h-step-ahead forecast is proportional to the optimal 1-step-ahead forecast. The
2
optimal 1-step-ahead forecast, moreover, is easily calculated: all of the determinants of σt+1 are lagged by
at least one period, so that there is no problem of forecasting the right-hand side variables. In practice, of
course, the underlying GARCH parameters α and β are unknown and so must be estimated, resulting in the
2
feasible forecast σ̂t+h,t formed in the obvious way. In financial applications, volatility forecasts are often of
2
direct interest, and the GARCH model delivers the optimal h-step-ahead point forecast, σt+h,t . Alternatively,
and more generally, we might not be intrinsically interested in volatility; rather, we may simply want to use
GARCH volatility forecasts to improve h-step-ahead interval or density forecasts of εt , which are crucially
2
dependent on the h-step-ahead prediction error variance, σt+h,t . Consider, for example, the case of interval
forecasting. In the case of constant volatility, we earlier worked with Gaussian ninety-five percent interval
forecasts of the form
yt+h,t ± 1.96σh ,

where σh denotes the unconditional h-step-ahead standard deviation (which also equals the conditional h-
step-ahead standard deviation in the absence of volatility dynamics). Now, however, in the presence of
volatility dynamics we use
yt+h,t ± 1.96σt+h,t .

The ability of the conditional prediction interval to adapt to changes in volatility is natural and desirable:
when volatility is low, the intervals are naturally tighter, and conversely. In the presence of volatility
dynamics, the unconditional interval forecast is correct on average but likely incorrect at any given time,
whereas the conditional interval forecast is correct at all times. The issue arises as to how to detect GARCH
effects in observed returns, and related, how to assess the adequacy of a fitted GARCH model. A key and
simple device is the correlogram of squared returns, ε2t . As discussed earlier, ε2t is a proxy for the latent
conditional variance; if the conditional variance displays persistence, so too will ε2t .16 Once can of course
also fit a GARCH model, and assess significance of the GARCH coefficients in the usual way.
Note that we can write the GARCH process for returns as εt = σt vt , where vt ∼ iidN (0, 1), σt2 = ω + αε2t−1 + βσt−1
2
.
Equivalently, the standardized return, v, is iid, ε σt = vt ∼ iidN (0, 1).
t
This observation suggests a way to evaluate the adequacy of a fitted GARCH model: standardize returns
by the conditional standard deviation from the fitted GARCH model, σ̂ , and then check for volatility
dynamics missed by the fitted model by examining the correlogram of the squared standardized return,
2
(εt /σ̂t ) . This is routinely done in practice.

15.5 Exercises, Problems and Complements

1. (Graphical regression diagnostic: time series plot of e2t or |et |)
Plots of e2t or |et | reveal patterns (most notably serial correlation) in the squared or absolute residuals,
which correspond to non-constant volatility, or heteroskedasticity, in the levels of the residuals. As
16
Note well, however, that the converse is not true. That is, if ε2t displays persistence, it does not necessarily
follow that the conditional variance displays persistence. In particular, neglected serial correlation associated
with conditional mean dynamics may cause serial correlation in εt and hence also in ε2t . Thus, before
proceeding to examine and interpret the correlogram of ε2t as a check for volatility dynamics, it is important
that any conditional mean effects be appropriately modeled, in which case εt should be interpreted as the
disturbance in an appropriate conditional mean model.
15.5. EXERCISES, PROBLEMS AND COMPLEMENTS 277

with the standard residual plot, the squared or absolute residual plot is always a simple univariate
plot, even when there are many right-hand side variables. Such plots feature prominently, for example,
in tracking and forecasting time-varying volatility.

2. (Removing conditional mean dynamics before modeling volatility dynamics)

In the application in the text we noted that NYSE stock returns appeared to have some weak condi-
tional mean dynamics, yet we ignored them and proceeded directly to model volatility.

a. Instead, first fit autoregressive models using the SIC to guide order selection, and then fit GARCH
models to the residuals. Redo the entire empirical analysis reported in the text in this way, and
discuss any important differences in the results.
b. Consider instead the simultaneous estimation of all parameters of AR(p)-GARCH models. That
is, estimate regression models where the regressors are lagged dependent variables and the distur-
bances display GARCH. Redo the entire empirical analysis reported in the text in this way, and
discuss any important differences in the results relative to those in the text and those obtained in
part a above.

3. (Variations on the basic ARCH and GARCH models) Using the stock return data, consider richer
models than the pure ARCH and GARCH models discussed in the text.

a. Estimate, diagnose and discuss a threshold GARCH(1,1) model.

b. Estimate, diagnose and discuss an EGARCH(1,1) model.
c. Estimate, diagnose and discuss a component GARCH(1,1) model.
d. Estimate, diagnose and discuss a GARCH-M model.

4. (Empirical performance of pure ARCH models as approximations to volatility dynamics)

Here we will fit pure ARCH(p) models to the stock return data, including values of p larger than p=5
as done in the text, and contrast the results with those from fitting GARCH(p,q) models.

a. When fitting pure ARCH(p) models, what value of p seems adequate?

b. When fitting GARCH(p,q) models, what values of p and q seem adequate?
c. Which approach appears more parsimonious?

5. (Direct modeling of volatility proxies)

In the text we fit an AR(5) directly to a subset of the squared NYSE stock returns. In this exercise,
use the entire NYSE dataset.

a. Construct, display and discuss the fitted volatility series from the AR(5) model.
b. Construct, display and discuss an alternative fitted volatility series obtained by exponential smooth-
ing, using a smoothing parameter of .10, corresponding to a large amount of smoothing, but less
than done in the text.
c. Construct, display and discuss the volatility series obtained by fitting an appropriate GARCH
model.
278 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY

d. Contrast the results of parts a, b and c above.

e. Why is fitting of a GARCH model preferable in principle to the AR(5) or exponential smoothing
approaches?

6. (Assessing volatility dynamics in observed returns and in standardized returns)

In the text we sketched the use of correlograms of squared observed returns for the detection of
GARCH, and squared standardized returns for diagnosing the adequacy of a fitted GARCH model.
Examination of Ljung-Box statistics is an important part of a correlogram analysis. It can be shown
that the Ljung-Box statistics may be legitimately used on squared observed returns, in which case it
will have the usual χ2m distribution under the null hypothesis of independence. One may also use the
Ljung-Box statistic on the squared standardized returns, but a better distributional approximation
is obtained in that case by using a χ2m−k distribution, where k is the number of estimated GARCH
parameters, to account for degrees of freedom used in model fitting.

7. (Allowing for leptokurtic conditional densities)

Thus far we have worked exclusively with conditionally Gaussian GARCH models, which correspond
to εt = σt vt vt ∼ iidN (0, 1), or equivalently, to normality of the standardized return, εt /σt .

a. The conditional normality assumption may sometimes be violated. However, GARCH parameters
are consistently estimated by Gaussian maximum likelihood even when the normality assumption
is incorrect. Sketch some intuition for this result.
b. Fit an appropriate conditionally Gaussian GARCH model to the stock return data. How might you
use the histogram of the standardized returns to assess the validity of the conditional normality
assumption? Do so and discuss your results.
c. Sometimes the conditionally Gaussian GARCH model does indeed fail to explain all of the lep-
tokurtosis in returns; that is, especially with very high-frequency data, we sometimes find that
the conditional density is leptokurtic. Fortunately, leptokurtic conditional densities are easily in-
corporated into the GARCH model. For example, in the conditionally Student’s-t GARCH
model, the conditional density is assumed to be Student’s t, with the degrees-of-freedom d treated
as another parameter to be estimated. More precisely, we write

td
vt ∼ iid .
std(td )

ε t = σ t vt

What is the reason for dividing the Student’s t variable, td , by its standard deviation, std(td ) ?
How might such a model be estimated?

8. (Multivariate GARCH models)

In the multivariate case, such as when modeling a set of returns rather than a single return, we need
to model not only conditional variances, but also conditional covariances.

a. Is the GARCH conditional variance specification introduced earlier, say for the i − th return,
2
σit = ω + αε2i,t−1 + βσi,t−1
2
, still appealing in the multivariate case? Why or why not?
15.6. NOTES 279

b. Consider the following specification for the conditional covariance between i − th and j-th returns:
σij,t = ω + αεi,t−1 εj,t−1 + βσij,t−1 . Is it appealing? Why or why not?
c. Consider a fully general multivariate volatility model, in which every conditional variance and
covariance may depend on lags of every conditional variance and covariance, as well as lags of
every squared return and cross product of returns. What are the strengths and weaknesses of such
a model? Would it be useful for modeling, say, a set of five hundred returns? If not, how might
you proceed?

15.6 Notes
280 CHAPTER 15. DYNAMIC HETEROSKEDASTICITY
 Part IV

Econometrics and Machine Learning

281
Chapter 16

Misspecification and Model Selection

The IC’s of Chapter 3 are surely heroic in economic contexts, so let us begin to relax them. One aspect of
IC 1 is that the fitted model matches the true DGP. In reality we can never know the DGP, and surely any
model that we might fit fails to match it, so there is an issue of how to select and fit a “good” model.
Recall that the Akaike information criterion, or AIC, is effectively an estimate of the out-of-sample
forecast error variance, as is s2 , but it penalizes degrees of freedom more harshly. It is used to select among
competing forecasting models. The formula is:
PN 2
i=1 ei
AIC = e( N )
2K
.
N

Also recall that the Schwarz information criterion, or SIC, is an alternative to the AIC with the
same interpretation, but a still harsher degrees-of-freedom penalty. The formula is:
PN 2
i=1 ei
SIC = N ( N )
K
.
N

Here we elaborate. We start with more on selection (“hard threshold” – variables are either kept
or discarded), and then we introduce shrinkage (“soft threshold” – all variables are kept, but parameter
estimates are coaxed in a certain direction), and then lasso, which blends selection and shrinkage.

16.0.1 Information Criteria (Hard Thresholding)

All-subsets model selection means that we examine every possible combination of K regressors and select
the best. Examples include SIC and AIC.
Let us now discuss SIC and AIC in greater depth, as they are tremendously important tools for building
forecasting models. We often could fit a wide variety of forecasting models, but how do we select among
them? What are the consequences, for example, of fitting a number of models and selecting the model with
highest R2 ? Is there a better way? This issue of model selection is of tremendous importance in all of
forecasting.
It turns out that model-selection strategies such as selecting the model with highest R2 do not produce
good out-of-sample forecasting models. Fortunately, however, a number of powerful modern tools exist to
assist with model selection. Most model selection criteria attempt to find the model with the smallest out-

283
284 CHAPTER 16. MISSPECIFICATION AND MODEL SELECTION

of-sample 1-step-ahead mean squared prediction error. The criteria we examine fit this general approach;
the differences among criteria amount to different penalties for the number of degrees of freedom used in
estimating the model (that is, the number of parameters estimated). Because all of the criteria are effectively
estimates of out-of-sample mean square prediction error, they have a negative orientation – the smaller the
better.
First consider the mean squared error,
PN 2
i=1 ei
M SE = ,
N
where N is the sample size and ei = yi − ŷi . M SE is intimately related to two other diagnostic statistics
routinely computed by regression software, the sum of squared residuals and R2 . Looking at the M SE
formula reveals that the model with the smallest M SE is also the model with smallest sum of squared
residuals, because scaling the sum of squared residuals by 1/N doesn’t change the ranking. So selecting
the model with the smallest M SE is equivalent to selecting the model with the smallest sum of squared
residuals. Similarly, recall the formula for R2 ,
PN 2
2 i=1 ei M SE
R = 1 − PN =1− 1
PN .
i=1 (yi − ȳ)2 N i=1 (yi − ȳ)
2

The denominator of the ratio that appears in the formula is just the sum of squared deviations of y from its
sample mean (the so-called “total sum of squares”), which depends only on the data, not on the particular
model fit. Thus, selecting the model that minimizes the sum of squared residuals – which as we saw is
equivalent to selecting the model that minimizes MSE – is also equivalent to selecting the model that
maximizes R2 .
Selecting forecasting models on the basis of MSE or any of the equivalent forms discussed above – that
is, using in-sample MSE to estimate the out-of-sample 1-step-ahead MSE – turns out to be a bad idea.
 285

In-sample MSE can’t rise when more variables are added to a model, and typically it will fall continuously
as more variables are added, because the estimated parameters are explicitly chosen to minimize the sum of
squared residuals. Newly-included variables could get estimated coefficients of zero, but that’s a probability-
zero event, and to the extent that the estimate is anything else, the sum of squared residuals must fall. Thus,
the more variables we include in a forecasting model, the lower the sum of squared residuals will be, and
therefore the lower M SE will be, and the higher R2 will be. Again, the sum of squared residuals can’t rise,
and due to sampling error it’s very unlikely that we’d get a coefficient of exactly zero on a newly-included
variable even if the coefficient is zero in population.
The effects described above go under various names, including in-sample overfitting, reflecting the idea
that including more variables in a forecasting model won’t necessarily improve its out-of-sample forecasting
performance, although it will improve the model’s “fit” on historical data. The upshot is that in-sample
M SE is a downward biased estimator of out-of-sample M SE, and the size of the bias increases with the
number of variables included in the model. In-sample M SE provides an overly-optimistic (that is, too small)
assessment of out-of-sample M SE.
To reduce the bias associated with M SE and its relatives, we need to penalize for degrees of freedom
used. Thus let’s consider the mean squared error corrected for degrees of freedom,
PN 2
i=1 ei
s2 = ,
N −K
where K is the number of degrees of freedom used in model fitting.1 s2 is just the usual unbiased estimate of
the regression disturbance variance. That is, it is the square of the usual standard error of the regression. So
selecting the model that minimizes s2 is equivalent to selecting the model that minimizes the standard error
of the regression. s2 is also intimately connected to the R2 adjusted for degrees of freedom (the “adjusted
R2 ,” or R̄2 ). Recall that
PN 2
e /(N − K) s2
R̄2 = 1 − PN i=1 i = 1 − PN .
2 2
i=1 (yi − ȳ) /(N − 1) i=1 (yi − ȳ) /(N − 1)

The denominator of the R̄2 expression depends only on the data, not the particular model fit, so the
model that minimizes s2 is also the model that maximizes R̄2 . In short, the strategies of selecting the model
that minimizes s2 , or the model that minimizes the standard error of the regression, or the model that
maximizes R̄2 , are equivalent, and they do penalize for degrees of freedom used.
To highlight the degree-of-freedom penalty, let’s rewrite s2 as a penalty factor times the M SE,
  PN 2
N i=1 ei
s2 = .
N −K N
Note in particular that including more variables in a regression will not necessarily lower s2 or raise R̄2 –
the M SE will fall, but the degrees-of-freedom penalty will rise, so the product could go either way.
As with s2 , many of the most important forecast model selection criteria are of the form “penalty factor
times M SE.” The idea is simply that if we want to get an accurate estimate of the 1-step-ahead out-of-
sample forecast M SE, we need to penalize the in-sample residual M SE to reflect the degrees of freedom
used. Two very important such criteria are the Akaike Information Criterion (AIC) and the Schwarz
Information Criterion (SIC). Their formulas are:
1
The degrees of freedom used in model fitting is simply the number of parameters estimated.
286 CHAPTER 16. MISSPECIFICATION AND MODEL SELECTION

PN 2
i=1 ei
AIC = e( N )
2K

N
and
PN 2
i=1 ei
SIC = N ( N )
K
.
N
How do the penalty factors associated with M SE, s2 , AIC and SIC compare in terms of severity? All
of the penalty factors are functions of K/N , the number of parameters estimated per sample observation,
and we can compare the penalty factors graphically as K/N varies. In Figure *** we show the penalties as
K/N moves from 0 to .25, for a sample size of N = 100. The s2 penalty is small and rises slowly with K/N ;
the AIC penalty is a bit larger and still rises only slowly with K/N . The SIC penalty, on the other hand,
is substantially larger and rises much more quickly with K/N .
It’s clear that the different criteria penalize degrees of freedom differently. In addition, we could propose
many other criteria by altering the penalty. How, then, do we select among the criteria? More generally,
what properties might we expect a “good” model selection criterion to have? Are s2 , AIC and SIC “good”
model selection criteria?
We evaluate model selection criteria in terms of a key property called consistency, also known as the
oracle property. A model selection criterion is consistent if:

a. when the true model (that is, the data-generating process, or DGP) is among a fixed set models
considered, the probability of selecting the true DGP approaches one as the sample size gets large, and

b. when the true model is not among a fixed set of models considered, so that it’s impossible to select the
true DGP, the probability of selecting the best approximation to the true DGP approaches one as the
sample size gets large.

We must of course define what we mean by “best approximation” above. Most model selection criteria
– including all of those discussed here – assess goodness of approximation in terms of out-of-sample mean
squared forecast error.
Consistency is of course desirable. If the DGP is among those considered, then we’d hope that as the
sample size gets large we’d eventually select it. Of course, all of our models are false – they’re intentional
simplifications of a much more complex reality. Thus the second notion of consistency is the more compelling.
M SE is inconsistent, because it doesn’t penalize for degrees of freedom; that’s why it’s unattractive. s2
does penalize for degrees of freedom, but as it turns out, not enough to render it a consistent model selection
procedure. The AIC penalizes degrees of freedom more heavily than s2 , but it too remains inconsistent;
even as the sample size gets large, the AIC selects models that are too large (“overparameterized”). The
SIC, which penalizes degrees of freedom most heavily, is consistent.
The discussion thus far conveys the impression that SIC is unambiguously superior to AIC for selecting
forecasting models, but such is not the case. Until now, we’ve implicitly assumed a fixed set of models.
In that case, SIC is a superior model selection criterion. However, a potentially more compelling thought
experiment for forecasting may be that we may want to expand the set of models we entertain as the sample
size grows, to get progressively better approximations to the elusive DGP. We’re then led to a different
optimality property, called asymptotic efficiency. An asymptotically efficient model selection criterion
chooses a sequence of models, as the sample size get large, whose out-of-sample forecast MSE approaches
16.1. CROSS VALIDATION (HARD THRESHOLDING) 287

the one that would be obtained using the DGP at a rate at least as fast as that of any other model selection
criterion. The AIC, although inconsistent, is asymptotically efficient, whereas the SIC is not.
In practical forecasting we usually report and examine both AIC and SIC. Most often they select the
same model. When they don’t, and despite the theoretical asymptotic efficiency property of AIC, this author
recommends use of the more parsimonious model selected by the SIC, other things equal. This accords with
the parsimony principle of Chapter ?? and with the results of studies comparing out-of-sample forecasting
performance of models selected by various criteria.
The AIC and SIC have enjoyed widespread popularity, but they are not universally applicable, and
we’re still learning about their performance in specific situations. However, the general principle that we
need somehow to inflate in-sample loss estimates to get good out-of-sample loss estimates is universally
applicable.
The versions of AIC and SIC introduced above – and the claimed optimality properties in terms of
out-of-sample forecast MSE – are actually specialized to the Gaussian case, which is why they are written
in terms of minimized SSR’s rather than maximized lnL’s.2 More generally, AIC and SIC are written not
in terms of minimized SSR’s, but rather in terms of maximized lnL’s. We have:

AIC = −2lnL + 2K

and
SIC = −2lnL + KlnN.

These are useful for any model estimated by maximum likelihood, Gaussian or non-Gaussian.

16.1 Cross Validation (Hard Thresholding)

Cross validation (CV) proceeds as follows. Consider selecting among J models. Start with model 1, estimate
it using all data observations except the first, use it to predict the first observation, and compute the
associated squared prediction error. Then estimate it using all observations except the second, use it to
predict the second observation, and compute the associated squared error. Keep doing this – estimating the
model with one observation deleted and then using the estimated model to predict the deleted observation
– until each observation has been sequentially deleted, and average the squared errors in predicting each of
the N sequentially deleted observations. Repeat the procedure for the other models, j = 2, ..., J, and select
the model with the smallest average squared prediction error.
Actually this is “N − f old” CV, because we split the data into N parts (the N individual observations)
and predict each of them. More generally we can split the data into M parts (M < N ) and cross validate
on them (“M − f old” CV). As M falls, M -fold CV eventually becomes consistent. M = 10 often works well
in practice.
It is instructive to compare SIC and CV, both of which have the oracle property. SIC achieves it by
penalizing in-sample residual MSE to obtain an approximately-unbiased estimate of out-of-sample MSE. CV,
in contrast, achieves it by directly obtaining an unbiased estimated out-of-sample MSE.
CV is more general than information criteria insofar as it can be used even when the model degrees of
freedom is unclear. In addition, non-quadratic loss can be introduced easily.
2
Recall that in the Gaussian case SSR minimization and lnL maximization are equivalent.
288 CHAPTER 16. MISSPECIFICATION AND MODEL SELECTION

16.2 Stepwise Selection (Hard Thresholding)

All-subsets selection, whether by AIC, SIC or CV, quickly gets hard as there are 2K subsets of K regressors.
Other procedures, like the stepwise selection procedures that we now introduce, don’t explore every possible
subset. They are more ad hoc but very useful.

16.2.1 Forward
Algorithm:
– Begin regressing only on an intercept
– Move to a one-regressor model by including that variable with the smallest t-stat p-value
– Move to a two-regressor model by including that variable with the smallest p-value
– Move to a three-regressor model by including that variable with the smallest p-value
Often people use information criteria or CV to select from the stepwise sequence of models. This is a
“greedy algorithm,” producing an increasing sequence of candidate models. Often people use information
criteria or CV to select from the stepwise sequence of models. No guaranteed optimality properties of the
selected model.
“forward stepwise regression”
– Often people use information criteria or cross validation to select from the stepwise sequence of models.

16.2.2 Backward
Algorithm:
– Start with a regression that includes all K variables
– Move to a K − 1 variable model by dropping the variable with the largest t-stat p-value
– Move to a K − 2 variable model by dropping the variable with the largest p-value
Often people use information criteria or CV to select from the stepwise sequence of models. This is a
“greedy algorithm,” producing a decreasing sequence of candidate models. Often people use information
criteria or CV to select from the stepwise sequence of models. No guaranteed optimality properties of the
selected model.

16.3 Bayesian Shrinkage (Soft Thresholding)

Shrinkage is a generic feature of Bayesian estimation. The Bayes rule under quadratic loss is the posterior
mean, which is a weighted average of the MLE and the prior mean,

β̂bayes = ω1 β̂M LE + ω2 β0 ,

where the weights depend on prior precision. Hence the the Bayes rule pulls, or “shrinks,” the MLE toward
the prior mean.
A classic shrinkage estimator is ridge regression,,3

β̂ridge = (X ′ X + λI)−1 X ′ y.
3
The ridge regression estimator can be shown to be the posterior mean for a certain prior and likelihood.
16.4. SELECTION AND SHRINKAGE (MIXED HARD AND SOFT THRESHOLDING)289

λ → 0 produces OLS, whereas λ → ∞ shrinks completely to 0. λ can be chosen by CV. (Notice that λ
can not be chosen by information criteria, as K regressors are included regardless of λ. Hence CV is a
more general selection procedure, useful for selecting various “tuning parameters” (like λ) as opposed to just
numbers of variables in hard-threshold procedures.

16.4 Selection and Shrinkage (Mixed Hard and Soft

Thresholding)
16.4.1 Penalized Estimation
Consider the penalized estimator,
 !2 
N
X X K
X
β̂P EN = argminβ  yi − βi xit +λ |βi |q  ,
i=1 i i=1

or equivalently
N
!2
X X
β̂P EN = argminβ yi − βi xit
i=1 i

s.t.
K
X
|βi |q ≤ c.
i=1

Concave penalty functions non-differentiable at the origin produce selection. Smooth convex penalties pro-
duce shrinkage. Indeed one can show that taking q → 0 produces subset selection, and taking q = 2 produces
ridge regression. Hence penalized estimation nests those situations and includes an intermediate case (q = 1)
that produces the lasso, to which we now turn.

16.4.2 The Lasso

The lasso solves the L1-penalized regression problem of finding
 !2 
XN X K
X
β̂LASSO = argminβ  yi − βi xit +λ |βi |
i=1 i i=1

or equivalently
N
!2
X X
β̂LASSO = argminβ yi − βi xit
i=1 i

s.t.
K
X
|βi | ≤ c.
i=1

Ridge shrinks, but the lasso shrinks and selects. Figure ?? says it all. Notice that, like ridge and other
Bayesian procedures, lasso requires only one estimation. And moreover, the lasso uses minimization problem
290 CHAPTER 16. MISSPECIFICATION AND MODEL SELECTION

Figure 16.1: Lasso and Ridge Comparison

16.5. DISTILLATION: PRINCIPAL COMPONENTS 291

is convex (lasso uses the smallest q for which it is convex), which renders the single estimation highly tractable
computationally.
Lasso also has a very convenient d.f. result. The effective number of parameters is precisely the number
of variables selected (number of non-zero β’s). This means that we can use info criteria to select among “lasso
models” for various λ. That is, the lasso is another device for producing an “increasing” sequence of candidate
models (as λ increases). The “best” λ can then be chosen by information criteria (or cross-validation, of
course).

16.5 Distillation: Principal Components

16.5.1 Distilling “X Variables” into Principal Components
Data Summarization. Think of a giant (wide) X matrix and how to “distill” it.
X ′ X eigen-decomposition:
X ′ X = V D2 V ′

The j th column of V , vj , is the j th eigenvector of X ′ X

Diagonal matrix D2 contains the descending eigenvalues of X ′ X

First principal component (PC):

z1 = Xv1
var(z1 ) = d21 /N
(maximal sample variance among all possible l.c.’s of columns of X)

In general:
zj = Xvj ⊥ zj ′ , j ′ ̸= j
var(zj ) ≤ d2j /N

16.5.2 Principal Components Regression

The idea is to enforce parsimony with little information loss by regressing not on the full X, but rather
on the first few PC’s of X. We speak of “Principal components regression” (PCR), or “Factor-Augmented
Regression”.
Ridge regression and PCR are both shrinkage procedures involving PC’s. Ridge effectively includes all
PC’s and shrinks according to sizes of eigenvalues associated with the PC’s. PCR effectively shrinks some
PCs completely to zero (those not included) and doesn’t shrink others at all (those included).
*********** More:
model selection (zero restrictions)
other regularizations (e.g., reduced-rank)
hard vs. soft constraints and shrinkage
nonparametric nonlinearity
high dimensionality
messy data (mixed-freq, unequally spaced, text)
expected utility maximization vs. regret minimization
292 CHAPTER 16. MISSPECIFICATION AND MODEL SELECTION

16.6 Exercises, Problems and Complements

1. (The sum of squared residuals, SSR)

(a) What is SSR and why is it reported?

(b) Do you agree with “bigger is better,” “smaller is better,” or neither? Be careful.
(c) Describe in detail and discuss the use of regression statistics R2 , R̄2 , F , SER, and SIC. What
role does SSR play in each of the test statistics?
(d) Under the IC, is the maximized log likelihood related to the SSR? If so, how? Would your
answer change if we dropped normality?

2. (The variety of “information criteria” reported across software packages)

Some authors, and software packages, examine and report the logarithms of the AIC and SIC,

PN !
2
 
i=1 ei 2K
ln(AIC) = ln +
N N
PN !
2
i=1 ei K ln(N )
ln(SIC) = ln + .
N N

The practice is so common that log(AIC) and log(SIC) are often simply called the “AIC” and “SIC.”
AIC and SIC must be greater than zero, so log(AIC) and log(SIC) are always well-defined and can
take on any real value. The important insight, however, is that although these variations will of course
change the numerical values of AIC and SIC produced by your computer, they will not change the
rankings of models under the various criteria. Consider, for example, selecting among three models.
If AIC1 < AIC2 < AIC3 , then it must be true as well that ln(AIC1 ) < ln(AIC2 ) < ln(AIC3 ) , so
we would select model 1 regardless of the “definition” of the information criterion used.

3. (“All-subset”, “partial-subset”, and “one-shot” model selection)

Note that model selection by information criteria or cross validation are all-subset strategies, insofar as
we examine all possible models and pick the one that looks best according to the criterion. Stepwise
procedures are partial-subset strategies, insofar as we examine many models, but not all possible
models, and pick the one that looks best according to the criterion. Ridge and LASSO, in contrast,
are one-shot strategies, insofar as we need to perform only a single estimation. All-subset strategies
become unappealing as the number of regressors, K, grows, because there are 2K subsets of K
regressors, requiring running and comparing 2K regressions. One-shot strategies, in contrast, remain
appealing in situations with many regressors, because just one estimation is required.
 Part V

Appendices

293
Appendix A

Probability and Statistics Review

Here we review a few aspects of probability and statistics that we will rely upon at various times.

A.1 Populations: Random Variables, Distributions and

Moments
A.1.1 Univariate
Consider an experiment with a set O of possible outcomes. A random variable Y is simply a mapping
from O to the real numbers. For example, the experiment might be flipping a coin twice, in which case
O = {(Heads, Heads), (T ails, T ails), (Heads, T ails), (T ails, Heads)}. We might define a random variable
Y to be the number of heads observed in the two flips, in which case Y could assume three values, y = 0,
y = 1 or y = 2.1
Discrete random variables, that is, random variables with discrete probability distributions,
can assume only a countable number of values yi , i = 1, 2, ..., each with positive probability pi such that
P
i pi = 1. The probability distribution f (y) assigns a probability pi to each such value yi . In the example
at hand, Y is a discrete random variable, and f (y) = 0.25 for y = 0, f (y) = 0.50 for y = 1, f (y) = 0.25 for
y = 2, and f (y) = 0 otherwise.
In contrast, continuous random variables can assume a continuous range of values, and the proba-
bility density function f (y) is a non-negative continuous function such that the area under f (y) between
any points a and b is the probability that Y assumes a value between a and b.2
In what follows we will simply speak of a “distribution,” f (y). It will be clear from context whether we
are in fact speaking of a discrete random variable with probability distribution f (y) or a continuous random
variable with probability density f (y).
Moments provide important summaries of various aspects of distributions. Roughly speaking, moments
are simply expectations of powers of random variables, and expectations of different powers convey different
1
Note that, in principle, we use capitals for random variables (Y ) and small letters for their realizations
(y). We will often neglect this formalism, however, as the meaning will be clear from context.
2
In addition, the total area under f (y) must be 1.

295
296 APPENDIX A. PROBABILITY AND STATISTICS REVIEW

sorts of information. You are already familiar with two crucially important moments, the mean and variance.
In what follows we’ll consider the first four moments: mean, variance, skewness and kurtosis.3
The mean, or expected value, of a discrete random variable is a probability-weighted average of the
values it can assume,4
X
E(y) = pi yi .
i

Often we use the Greek letter µ to denote the mean, which measures the location, or central tendency,
of y.
The variance of y is its expected squared deviation from its mean,

var(y) = E(y − µ)2 .

We use σ 2 to denote the variance, which measures the dispersion, or scale, of y around its mean.
Often we assess dispersion using the square root of the variance, which is called the standard deviation,
p
σ = std(y) = E(y − µ)2 .

The standard deviation is more easily interpreted than the variance, because it has the same units of
measurement as y. That is, if y is measured in dollars (say), then so too is std(y). V ar(y), in contrast,
would be measured in rather hard-to-grasp units of “dollars squared”.
The skewness of y is its expected cubed deviation from its mean (scaled by σ 3 for technical reasons),

E(y − µ)3
S= .
σ3

Skewness measures the amount of asymmetry in a distribution. The larger the absolute size of the skewness,
the more asymmetric is the distribution. A large positive value indicates a long right tail, and a large negative
value indicates a long left tail. A zero value indicates symmetry around the mean.
The kurtosis of y is the expected fourth power of the deviation of y from its mean (scaled by σ 4 , again
for technical reasons),
E(y − µ)4
K= .
σ4
Kurtosis measures the thickness of the tails of a distribution. A kurtosis above three indicates “fat tails”
or leptokurtosis, relative to the normal, or Gaussian distribution that you studied earlier. Hence a
kurtosis above three indicates that extreme events (“tail events”) are more likely to occur than would be the
case under normality.

A.1.2 Multivariate
Suppose now that instead of a single random variable Y , we have two random variables Y and X.5 We
can examine the distributions of Y or X in isolation, which are called marginal distributions. This is
3
In principle, we could of course consider moments beyond the fourth, but in practice only the first four
are typically examined.
4
R
A similar formula holds in the continuous case, E(y) = y f (y) dy.
5
We could of course consider more than two variables, but for pedagogical reasons we presently limit
ourselves to two.
A.2. SAMPLES: SAMPLE MOMENTS 297

effectively what we’ve already studied. But now there’s more: Y and X may be related and therefore move
together in various ways, characterization of which requires a joint distribution. In the discrete case the
joint distribution f (y, x) gives the probability associated with each possible pair of y and x values, and in
the continuous case the joint density f (y, x) is such that the area in any region under it gives the probability
of (y, x) falling in that region.
We can examine the moments of y or x in isolation, such as mean, variance, skewness and kurtosis. But
again, now there’s more: to help assess the dependence between y and x, we often examine a key moment
of relevance in multivariate environments, the covariance. The covariance between y and x is simply the
expected product of the deviations of y and x from their respective means,

cov(y, x) = E[(y − µy )(x − µx )].

A positive covariance means that y and x are positively related; that is, when y is above its mean x tends
to be above its mean, and when y is below its mean x tends to be below its mean. Conversely, a negative
covariance means that y and x are inversely related; that is, when y is below its mean x tends to be above
its mean, and vice versa. The covariance can take any value in the real numbers.
Frequently we convert the covariance to a correlation by standardizing by the product of σy and σx ,

cov(y, x)
corr(y, x) = .
σy σx

The correlation takes values in [-1, 1]. Note that covariance depends on units of measurement (e.g., dollars,
cents, billions of dollars), but correlation does not. Hence correlation is more immediately interpretable,
which is the reason for its popularity.
Note also that covariance and correlation measure only linear dependence; in particular, a zero covariance
or correlation between y and x does not necessarily imply that y and x are independent. That is, they may be
non-linearly related. If, however, two random variables are jointly normally distributed with zero covariance,
then they are independent.
Our multivariate discussion has focused on the joint distribution f (y, x). In various chapters we will also
make heavy use of the conditional distribution f (y|x), that is, the distribution of the random variable Y
conditional upon X = x. Conditional moments are similarly important. In particular, the conditional
mean and conditional variance play key roles in econometrics, in which attention often centers on the
mean or variance of a series conditional upon the past.

A.2 Samples: Sample Moments

A.2.1 Univariate
Thus far we’ve reviewed aspects of known distributions of random variables, in population. Often, however,
we have a sample of data drawn from an unknown population distribution f ,

{yi }N
i=1 ∼ f (y),
298 APPENDIX A. PROBABILITY AND STATISTICS REVIEW

and we want to learn from the sample about various aspects of f , such as its moments. To do so we use
various estimators.6 We can obtain estimators by replacing population expectations with sample averages,
because the arithmetic average is the sample analog of the population expectation. Such “analog estimators”
turn out to have good properties quite generally. The sample mean is simply the arithmetic average,

N
1 X
ȳ = yi .
N i=1

It provides an empirical measure of the location of y.

The sample variance is the average squared deviation from the sample mean,
PN
2 i=1 (yi − ȳ)2
σ̂ = .
N

It provides an empirical measure of the dispersion of y around its mean.

We commonly use a slightly different version of σ̂ 2 , which corrects for the one degree of freedom used in
the estimation of ȳ, thereby producing an unbiased estimator of σ 2 ,
PN
i=1 (yi− ȳ)2
s2 = .
N −1

(Note that the difference between σ̂ 2 and s2 vanishes as N → ∞.)

Similarly, the sample standard deviation is defined either as
s
PN
√ − ȳ)2
i=1 (yi
σ̂ = σ̂ 2 =
N

or s
PN
√ − ȳ)2
i=1 (yi
s= s2 = .
N −1
It provides an empirical measure of dispersion in the same units as y.
The sample skewness is
1
PN
(yi − ȳ)3
Ŝ = N i=1 3 .
σ̂
It provides an empirical measure of the amount of asymmetry in the distribution of y.
The sample kurtosis is
1
PN 4
N i=1 (yi − ȳ)
K̂ = .
σ̂ 4
It provides an empirical measure of the fatness of the tails of the distribution of y relative to a normal
distribution.
Many of the most famous and important statistical sampling distributions arise in the context of sample
moments, and the normal distribution is the father of them all. In particular, the celebrated central limit
theorem establishes that under quite general conditions the sample mean ȳ will have a normal distribution
as the sample size gets large. The χ2 distribution arises from squared normal random variables, the t
distribution arises from ratios of normal and χ2 variables, and the F distribution arises from ratios of χ2
6
An estimator is an example of a statistic, or sample statistic, which is simply a function of the sample
observations.
A.3. FINITE-SAMPLE AND ASYMPTOTIC SAMPLING DISTRIBUTIONS OF THE SAMPLE MEAN2

variables. Because of the fundamental nature of the normal distribution as established by the central limit
theorem, it has been studied intensively, a great deal is known about it, and a variety of powerful tools have
been developed for use in conjunction with it.

A.2.2 Multivariate
We also have sample versions of moments of multivariate distributions. In particular, the sample covariance
is
N
1 X
cov(y,
c x) = [(yi − ȳ)(xi − x̄)],
N i=1

and the sample correlation is

cov(y,
c x)
corr(y,
d x) = .
σ̂y σ̂x

A.3 Finite-Sample and Asymptotic Sampling Distri-

butions of the Sample Mean
Here we refresh your memory on the sampling distribution of the most important sample moment, the sample
mean.

A.3.1 Exact Finite-Sample Results

In your earlier studies you learned about statistical inference, such as how to form confidence intervals for
the population mean based on the sample mean, how to test hypotheses about the population mean, and so
on. Here we partially refresh your memory.
Consider the benchmark case of Gaussian simple random sampling,

yi ∼ iid N (µ, σ 2 ), i = 1, ..., N,

which corresponds to a special case of what we will later call the “full ideal conditions” for regression
modeling. The sample mean ȳ is the natural estimator of the population mean µ. In this case, as you
learned earlier, ȳ is unbiased, consistent, normally distributed with variance σ 2 /N , and efficient (minimum
variance unbiased, MVUE). We write
σ2
 
ȳ ∼ N µ, ,
N
or equivalently
√
N (ȳ − µ) ∼ N (0, σ 2 ).

We estimate σ 2 consistently using s2 .

We construct exact finite-sample confidence intervals for µ as
 
s
µ ∈ ȳ ± t1− 2 (N − 1) √
α w.p. 1 − α,
N
α
where t1− α2 (N − 1) is the 1 − 2 percentile of a t distribution with N − 1 degrees of freedom. Similarly,
300 APPENDIX A. PROBABILITY AND STATISTICS REVIEW

we construct exact finite-sample (likelihood ratio) hypothesis tests of H0 : µ = µ0 against the two-sided
alternative H0 : µ ̸= µ0 using
ȳ − µ0
s ∼ t1− α2 (N − 1).
√
N

A.3.2 Approximate Asymptotic Results (Under Weaker Assump-

tions)
Much of statistical inference is linked to large-sample considerations, such as the law of large numbers and
the central limit theorem, which you also studied earlier. Here we again refresh your memory.
Consider again a simple random sample, but without the normality assumption,

yi ∼ iid(µ, σ 2 ), i = 1, ..., N.

Despite our dropping the normality assumption we still have that ȳ is consistent, asymptotically normally
distributed with variance σ 2 /N , and asymptotically efficient. We write,

a
σ2
 
ȳ ∼ N µ, .
N

More precisely, as T → ∞,
√
N (ȳ − µ) →d N (0, σ 2 ).

We estimate σ 2 consistently using s2 .

This result forms the basis for asymptotic inference. We construct asymptotically-valid confidence in-
tervals for µ as  
s
µ ∈ ȳ ± z1− α2 √ w.p. 1 − α,
N
where z1− α2 is the 1 − α2 percentile of a N (0, 1) distribution. Similarly, we construct asymptotically-valid
hypothesis tests of H0 : µ = µ0 against the two-sided alternative H0 : µ ̸= µ0 using

ȳ − µ0
∼ N (0, 1).
√s
N

A.4 Exercises, Problems and Complements

1. (Interpreting distributions and densities)
The Sharpe Pencil Company has a strict quality control monitoring program. As part of that program,
it has determined that the distribution of the amount of graphite in each batch of one hundred pencil
leads produced is continuous and uniform between one and two grams. That is, f (y) = 1 for y in [1,
2], and zero otherwise, where y is the graphite content per batch of one hundred leads.

a. Is y a discrete or continuous random variable?

b. Is f (y) a probability distribution or a density?
c. What is the probability that y is between 1 and 2? Between 1 and 1.3? Exactly equal to 1.67?
A.4. EXERCISES, PROBLEMS AND COMPLEMENTS 301

d. For high-quality pencils, the desired graphite content per batch is 1.8 grams, with low variation
across batches. With that in mind, discuss the nature of the density f (y).

2. (Covariance and correlation)

Suppose that the annual revenues of world’s two top oil producers have a covariance of 1,735,492.

a. Based on the covariance, the claim is made that the revenues are “very strongly positively related.”
Evaluate the claim.
b. Suppose instead that, again based on the covariance, the claim is made that the revenues are
“positively related.” Evaluate the claim.
c. Suppose you learn that the revenues have a correlation of 0.93. In light of that new information,
re-evaluate the claims in parts a and b above.

3. (Simulation)
You will often need to simulate data of various types, such as iidN (µ, σ 2 ) (Gaussian simple random
sampling).

a. Using a random number generator, simulate a sample of size 30 for y, where y ∼ iidN (0, 1).
b. What is the sample mean? Sample standard deviation? Sample skewness? Sample kurtosis?
Discuss.
c. Form an appropriate 95 percent confidence interval for E(y).
d. Perform a t test of the hypothesis that E(y) = 0.
e. Perform a t test of the hypothesis that E(y) = 1.

4. (Sample moments of the CPS wage data)

Use the 1995 CPS wage dataset.

a. Calculate the sample mean wage and test the hypothesis that it equals $9/hour.
b. Calculate sample skewness.
c. Calculate and discuss the sample correlation between wage and years of education.
302 APPENDIX A. PROBABILITY AND STATISTICS REVIEW
Appendix B

Construction of the Wage Datasets

We construct our datasets by adjusting and sampling from the much-larger Current Population Survey (CPS)
datasets.
We extract the data from the March CPS for three years: I, II, and III, each approximately a decade
apart, using the National Bureau of Economic Research (NBER) front end (http://www.nber.org/data/
cps.html) and NBER SAS, SPSS, and Stata data definition file statements (http://www.nber.org/data/
cps_progs.html). Here we focus our discussion on the CPS-I dataset.
There are many CPS observations for which earnings data are missing. We drop those observations,
leaving 14363 observations. From those, we draw a random subsample with 1323 observations.
We use seven variables. From the CPS we obtain AGE (age), FEMALE (1 if female, 0 otherwise),
NONWHITE (1 if nonwhite, 0 otherwise), and UNION (1 if union member, 0 otherwise). We also create
EDUC (years of schooling) based on CPS variable PEEDUCA (educational attainment). Because the CPS
does not ask about years of experience, we create EXPER (potential working experience) as AGE minus
EDUC minus 6. Finally, we create WAGE as PRERNHLY (earnings per hour) in dollars for those paid
hourly, and PRERNWA (gross earnings last week) divided by PEHRUSL1 (usual working hours per week)
for those not paid hourly (PRERNHLY=0).

303
304 APPENDIX B. CONSTRUCTION OF THE WAGE DATASETS
Appendix C

Some Popular Books Worth

Encountering

I have cited many of these books elsewhere, typically in various end-of-chapter complements. Here I list
them collectively.
Lewis (2003) [Michael Lewis, Moneyball ]. “Appearances may lie, but the numbers don’t, so pay attention
to the numbers.”
Gladwell (2000) [Malcolm Gladwell, The Tipping Point]. “Nonlinear phenomena are everywhere.”
Gladwell pieces together an answer to the puzzling question of why certain things “take off” whereas
others languish (products, fashions, epidemics, etc.) More generally, he provides deep insights into nonlinear
environments, in which small changes in inputs can lead to small changes in outputs under some conditions,
and to huge changes in outputs under other conditions.
Taleb (2007) [Nassim Nicholas Taleb, The Black Swan] “Warnings, and more warnings, and still more
warnings, about non-normality and much else.” See Chapter 4 EPC 1.
Angrist and Pischke (2009) [Joshua Angrist and Jorn-Steffen Pischke, Mostly Harmless Econometrics].
“Natural and quasi-natural experiments suggesting instruments.”
This is a fun and insightful treatment of instrumental-variables and related methods. Just don’t be
fooled by the book’s attempted landgrab, as discussed in a 2015 No Hesitations post.
Silver (2012) [Nate Silver, The Signal and the Noise]. “Pitfalls and opportunities in predictive modeling.”

305
306 APPENDIX C. SOME POPULAR BOOKS WORTH ENCOUNTERING
Bibliography

Angrist, J.D. and J.-S. Pischke (2009), Mostly Harmless Econometrics, Princeton University Press.

Gladwell, M. (2000), The Tipping Point, Little, Brown and Company.

Lewis, M. (2003), Moneyball, Norton.

Silver, N.. (2012), The Signal and the Noise, Penguin Press.

Taleb, N.N. (2007), The Black Swan, Random House.

Tufte, E.R. (1983), The Visual Display of Quantatative Information, Chesire: Graphics Press.

307
Index

F distribution, 298 Calendar effects, 176

F statistic, 48 Central tendency, 296
R-squared, 50 Chartjunk, 20
R2 , 284 Collinearlty, 60
s-squared, 49 Common scales, 20
s2 , 285 Conditional distribution, 297
t distribution, 298 Conditional mean, 297
t statistic, 43 Conditional mean and variance, 209
Conditional mean function, 103
χ2 distribution, 298
Conditional moment, 297
Fitted values, 33
Conditional variance, 297
Holiday variation, 176
Consistency, 286
Seasonal dummy variables, 174
Constant term, 43
Adjusted R-squared, 51 Continuous data, 5
Adjusted R2 , 285 Continuous random variable, 295
Akaike information criterion, 51, 283 Correlation, 297
Akaike Information Criterion (AIC), 285 Correlogram, 200
Analog principle, 200 Correlogram analysis, 203
Analysis of variance, 100 Covariance, 297
AR(p) process, 218 Covariance stationary, 194
ARCH(p) process, 268 Cross sectional data, 6
Aspect ratio, 20 Cross sections, 7
Asymmetric response, 273 Cycles, 192
Asymmetry, 296
Data mining, 52
Asymptotic efficiency, 286
Data-generating process (DGP), 38
Asymptototic, 300
Data-generating process, or DGP, 286
Autocorrelation function, 197
De-trending, 180
Autocovariance function, 194
Deterministic seasonality, 172
Autoregressions, 198
Deterministic trend, 165
Autoregressive (AR) model, 211
Discrete probability distribution, 295
Banking to 45 degrees, 20 Discrete random variable, 295
Binary data, 5 Dispersion, 296
binomial logit, 139 Distributed lag, 211
Box-Cox transformation, 106 Disturbance, 38
Box-Pierce Q-statistic, 202 Dummy left-hand-side variable, 133
Breusch-Godfrey test, 224 Dummy right-hand-side variable, 133

308
INDEX 309

Dummy variable, 95 Leptokurtosis, 296

Durbin’s h test, 234 Leverage, 77
Likelihood function, 46
Errors in variables, 148
Likelihood-ratio tests, 48
Estimator, 298
Limited dependent variable, 133
Ex post smoothing, 187
Linear probability model, 134
Expected value, 296
Linear projection, 103, 117
Exponential GARCH, 274
Linear trend, 165
Exponential trend, 168
Link function, 107, 141
Financial econometrics, 265 Ljung-Box Q-statistic, 202
First-order serial correlation, 224 Location, 296
Functional form, 103 Log-lin regression, 105
Log-linear trend, 168
GARCH(p,q) process, 269 Log-log regression, 104
Gaussian distribution, 296 Logistic function, 135
Gaussian white noise, 205 Logistic model, 107
Generalized linear model, 106, 141 Logistic trend, 185
GLM, 106, 141 Logit model, 135
Golden ratio, 26 Longitudinal data, 6
Goodness of fit, 50
Marginal distribution, 296
Heteroskedasticity, 265 Maximum likelihood estimation, 46
Histogram, 14 Mean, 296
Hodrick-Prescott filtering, 188 Mean squared error, 284
Measurement error, 148
Imperfect multicollinearity, 61
Measurement outliers, 233
In-sample overfitting, 52, 285
Measurement-error bias, 148
Inclusion of irrelevant variables, 159
Model selection, 283
Independent white noise, 205
Moments, 209, 295
Indicator variable, 95, 133
Moving-average representation, 213
Innovation outliers, 233
Multicollinearity, 60
Instrumental Variables Estimation, 151
Multinomial logit, 141
Interaction effects, 109
Multiple comparisons, 19
Intercept, 165
Multiple linear regression, 36
Intercept dummies, 97
Multivariate, 13
Interval data, 10
Multivariate GARCH, 278
Intrinsically non-linear models, 107
Multiway scatterplot, 16
Jarque-Bera test, 75
Nominal data, 10
Joint distribution, 297
Non-data ink, 20
Kurtosis, 296 non-linear least squares (NLS), 107
Non-linearity, 103
Lag operator, 210
Non-normality, 73
Least absolute deviations, 79
Normal distribution, 296
Least squares, 33
310 INDEX

Normal white noise, 205 Residuals, 33

Ridge regression, 288
Odds, 140 Robustness iteration, 129
Off-line smoothing, 187
Omitted variable bias, 147 Sample, 297
Omitted variables, 147 Sample autocorrelation function, 200
On-line smoothing, 187 Sample correlation, 299
One-sided moving average, 187 Sample covariance, 299
One-sided weighted moving average, 187 Sample kurtosis, 298
Oracle property, 286 Sample mean, 200, 298
Ordered logit, 137 Sample mean of the dependent variable, 45
Ordered outcomes, 136 Sample partial autocorrelation, 203
Ordinal data, 10 Sample path, 194
Outliers, 73 Sample selection, 150
Sample skewness, 298
Panel data, 6 Sample standard deviation, 298
Panels, 7 Sample standard deviation of the dependent variable,
Parameters, 38 45
Partial autocorrelation function, 198 Sample statistic, 298
Partial correlation, 23 Sample variance, 298
Perfect multicollinearity, 61 Scale, 296
Polynomial in the lag operator, 210 Scatterplot matrix, 16
Polynomial trend, 167 Schwarz information criterion, 52, 283
Population, 297 Schwarz Information Criterion (SIC), 285
Population regression, 198 Seasonal adjustment, 180
Positive serial correlation, 223 Seasonality, 165, 172
Prob(F statistic), 49 Second-order stationarity, 196
Probability density function, 295 Serially uncorrelated, 204
Probability value, 44 Simple correlation, 23
Probit model, 140 Simple random sampling, 299
Simulating time series processes, 232
QQ plots, 74
Simultaneity, 149
Quadratic trend, 167
Skewness, 296
Random number generator, 232 Slope, 166
Ratio data, 10 Slope dummies, 99
Real-time smoothing, 187 Smoothing, 187
Realization, 193 Sparklines, 21
Regression intercept, 38 Standard deviation, 296
Regression on seasonal dummies, 174 Standard error of the regression, 50
Regression slope coefficients, 38 Standard errors, 43
Relational graphics, 14 Statistic, 298
RESET test, 111 Stochastic processes, 211
Residual plot, 53 Strong white noise, 205
Residual scatter, 53 Student’s-t GARCH, 278
INDEX 311

Sum of squared residuals, 46, 284

Threshold GARCH, 273

Time dummy, 165
Time series, 7, 193
Time series data, 6
Time series of cross sections, 6
Time series plot, 14
Time series process, 204
Time-varying volatility, 265
Tobit model, 141
Trading-day variation, 176
Trend, 165
Two-sided moving average, 187

Unconditional mean and variance, 209

Univariate, 13

Variance, 296
Variance inflation factor, 62
Volatility clustering, 268
Volatility dynamics , 268

Weak stationarity, 196

Weak white noise, 205
White noise, 204

Yule-Walker equation, 216

Zero-mean white noise, 204