Introductory Econometrics: A Modern Approach (7e)

Chapter 3
The Simple Regression Model

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 1
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (7 of 39)

• Deriving the ordinary least squares estimates
• In order to estimate the regression model one needs data
• A random sample of n observations

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 2
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (9 of 39)

• OLS fits as good as possible a regression line through the data points

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 3
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (10 of 39)

• Example of a simple regression
• CEO salary and return on equity

• Fitted regression

• Causal interpretation?
© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 4
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (11 of 39)

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 5
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (12 of 39)

• Example of a simple regression
• Wage and education

• Fitted regression

• Causal interpretation?
© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 6
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (13 of 39)

• Example of a simple regression
• Voting outcomes and campaign expenditures (two parties)

• Fitted regression

• Causal interpretation?
© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 7
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (14 of 39)

• Properties of OLS on any sample of data
• Fitted values and residuals

• Algebraic properties of OLS regression

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 8
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (15 of 39)

obsno roe salary salaryhat uhat
1 14.1 1095 1224.058 -129.058 • This table presents fitted
2 10.9 1001 1164.854 -163.854 values and residuals for 15
3 23.5 1122 1397.960 -275.969
CEOs.
4 5.9 578 1072.348 -494.348
5 13.8 1368 1218.508 149.493
6 20.0 1145 1333.215 -188.215 • For example, the 12th CEO’s
7 16.4 1078 1266.611 188.611 predicted salary is $526,023
8 16.3 1094 1264.761 -170.761 higher than their actual
9 10.5 1237 1157.454 79.546
10 26.3 833 1449.773 -616.773
salary.
11 25.9 567 1442.372 -875.372
12 26.8 933 1459.023 -526.023 • By contrast the 5th CEO’s
13 14.8 1339 1237.009 101.991 predicted salary is $149,493
14 22.3 937 1375.768 -438.768
lower than their actual
15 56.3 2011 2004.808 6.192
salary.

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 9
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (16 of 39)

• Goodness of fit
• How well does an explanatory variable explain the dependent variable?

• Measures of variation:

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 10
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (17 of 39)

• Decomposition of total variation

• Goodness-of-fit measure (R-squared)

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 11
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (18 of 39)

• CEO Salary and return on equity

• Voting outcomes and campaign expenditures

• Caution: A high R-squared does not necessarily mean that the regression
has a causal interpretation!

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 12
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (19 of 39)

• Incorporating nonlinearities: Semi-logarithmic form
• Regression of log wages on years of education

• This changes the interpretation of the regression coefficient:

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 13
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (20 of 39)

• Fitted regression

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 14
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (21 of 39)

• Incorporating nonlinearities: Log-logarithmic form
• CEO salary and firm sales

• This changes the interpretation of the regression coefficient:

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 15
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (22 of 39)

• CEO salary and firm sales: fitted regression

• The log-log form postulates a constant elasticity model, whereas the

semi-log form assumes a semi-elasticity model.

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 16
 Introductory Econometrics: A Modern Approach (7e)

The Simple Regression Model (23 of 39)

• Expected values and variances of the OLS estimators
• The estimated regression coefficients are random variables because
they are calculated from a random sample

• The question is what the estimators will estimate on average and how
large will their variability be in repeated samples

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 17
 Introductory Econometrics: A Modern Approach (7e)

A Simple Assumption
• The average value of u, the error term, in the population is 0. That is,

• E(u) = 0

• This is not a restrictive assumption, since we can always use b0 to

normalize E(u) to 0

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
18
 Introductory Econometrics: A Modern Approach (7e)

Zero Conditional Mean

• We need to make a crucial assumption about how u and x are related
• We want it to be the case that knowing something about x does not
give us any information about u, so that they are completely
unrelated. That is, that
• E(u|x) = E(u) = 0, which implies
• E(y|x) = b0 + b1x

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
19
 Introductory Econometrics: A Modern Approach (7e)

E(y|x) as a linear function of x, where for any x

the distribution of y is centered about E(y|x)
y
f(y)

. E(y|x) = b + b x
.
0 1

x1 x2
© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 20
 Introductory Econometrics: A Modern Approach (7e)

Ordinary Least Squares

• Basic idea of regression is to estimate the population parameters
from a sample
• Let {(xi,yi): i=1, …,n} denote a random sample of size n from the
population
• For each observation in this sample, it will be the case that
• yi = b0 + b1xi + ui

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
21
 Introductory Econometrics: A Modern Approach (7e)

Population regression line, sample data points

and the associated error terms
y E(y|x) = b0 + b1x
y4 .{
u4

y3 .} u3
y2 u2 {.

y1 . } u1

x1 x2 x3 x4 x
© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 22
 Introductory Econometrics: A Modern Approach (7e)

Deriving OLS Estimates

• To derive the OLS estimates we need to realize that our main
assumption of E(u|x) = E(u) = 0 also implies that

• Cov(x,u) = E(xu) = 0

• Why? Remember from basic probability that Cov(X,Y) = E(XY) –

E(X)E(Y)

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
23
 Introductory Econometrics: A Modern Approach (7e)

Deriving OLS continued

• We can write our 2 restrictions just in terms of x, y, b0 and b1 , since u
= y – b0 – b1x

• E(y – b0 – b1x) = 0
• E[x(y – b0 – b1x)] = 0

• These are called moment restrictions

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
24
 Introductory Econometrics: A Modern Approach (7e)

Deriving OLS using M.O.M.

• The method of moments approach to estimation implies imposing
the population moment restrictions on the sample moments

• What does this mean? Recall that for E(X), the mean of a population
distribution, a sample estimator of E(X) is simply the arithmetic mean
of the sample

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
25
 Introductory Econometrics: A Modern Approach (7e)

More Derivation of OLS

• We want to choose values of the parameters that will ensure that
the sample versions of our moment restrictions are true
• The sample versions are as follows:

 y 
n
n 1
i  ˆ 0  ˆ1 xi  0
i 1

 
n
n 1
 i i 0 1 i
x y
i 1
 ˆ  ˆ x  0


© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 26
 Introductory Econometrics: A Modern Approach (7e)

More Derivation of OLS

• Given the definition of a sample mean, and
properties of summation, we can rewrite the first
condition as follows

y  ˆ0  ˆ1 x ,
or
ˆ0  y  ˆ1 x

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use. 27
 Introductory Econometrics: A Modern Approach (7e)

More Derivation of OLS

   
n

 i i
x y
i 1
 y  ˆ x  ˆ x  0
 1 1 i

n n
x 
 i iy  y   ˆ
 1  xi xi  x 
i 1 i 1
n n

 xi  x  yi  y   1  xi  x 
ˆ 2

i 1 i 1

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
28
 Introductory Econometrics: A Modern Approach (7e)

So the OLS estimated slope is

n

 x  x  y i i  y
ˆ1  i 1
n

 x  x 
2
i
i 1
n
provided that  xi  x   0
2

i 1

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
29
 Introductory Econometrics: A Modern Approach (7e)

Summary of OLS slope estimate

• The slope estimate is the sample covariance between x and y divided
by the sample variance of x
• If x and y are positively correlated, the slope will be positive
• If x and y are negatively correlated, the slope will be negative
• Only need x to vary in our sample

© 2020 Cengage. May not be scanned, copied or duplicated, or posted to a publicly accessible website, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a
password-protected website or school-approved learning management system for classroom use.
30