Chapter 2

The Simple Linear Regression Model

Principles of
Econometrics
Fifth Edition
R. Carter Hill William E. Griffiths Guay C. Lim
Chapter Outline: 1 of 1
1. An Economic Model
2. An Econometric Model
3. Estimating the Regression Parameters
4. Assessing the Least Squares Estimators
5. The Gauss–Markov Theorem
6. The Probability Distributions of the Least Squares Estimators
7. Estimating the Variance of the Error Term
8. Estimating Nonlinear Relationships
9. Regression with Indicator Variables
10. The Independent Variable

Copyright ©2018 John Wiley & Son, Inc. 2

2.1 An Economic Model (1 of 3)
 Economic theory suggests many relationships between economic
variables.
 A regression model is helpful in questions such as the following: if one
variable changes in a certain way, by how much will another variable
change?
 The regression model is based on assumptions.

 The continuous random variable y has a probability density function (pdf).

 The pdf is a conditional probability density function because it is

‘‘conditional’’ upon an x.
Copyright ©2018 John Wiley & Son, Inc. 3
2.1 An Economic Model (1 of 2)
 Households with an income of $1000 per week would have various food
expenditure per person for a variety of reasons.
 The pdf f(y) describes how expenditures are distributed over the
population.
 This is a conditional pdf because it is “conditional” upon household
income.
E ( y|x $1000)  y|x
 The conditional mean, or expected value, of y is and is
our population’s mean weekly food expenditure per person.
2
var(y|x $1000) 
 The conditional variance of y is which measures
the dispersion of household expenditures y about their mean.

Copyright ©2018 John Wiley & Son, Inc. 4

 2.1 An Economic Model (1 of 3)
 The parameters  y|x and  , if they were known, would give us some
2

valuable information about the population we are considering.

 In order to investigate the relationship between expenditure and income
we must build an economic model and then a corresponding
econometric model that forms the basis for a quantitative or empirical
economic analysis.
 This econometric model is also called a regression model.

Copyright ©2018 John Wiley & Son, Inc. 5

Copyright ©2018 John Wiley & Son, Inc. 6
2.2 An Econometric Model
 A household spends $80 plus 10 cents of each dollar of income received on
food.
 Algebraically their rule is y = 80 + 0.10x where y = weekly household food
expenditure ($) and x = weekly household income ($).
 In reality, many factors may affect household expenditure on food.

 Let e (error term) = everything else affecting food other than income.

 x + e This equation is the simple regression model.

 A simple linear regression analysis examines the relationship between a y-

variable and one x-variable.
Copyright ©2018 John Wiley & Son, Inc. 7
 2.2.1 Data Generating Process
 For the household food expenditure example, let us assume that we
can obtain a sample at a point in time (cross-sectional data).
 The sample consisting of N data pairs are randomly selected from the
population. Let (, ) denote the ith pair.
 The variables are random variables because their values are not
known until they are observed. Each observation pair is statistically
different from other pairs.
 All pairs drawn from the same population are assumed to follow the
same joint pdf and are identically distributed i.i.d.
Copyright ©2018 John Wiley & Son, Inc. 8
2.2.2 The Random Error and Strict
Exogeneity
 The second assumption of the simple regression model concerns
the “everything else” term e.
 Unlike food expenditure and income, the random error term is not
observable; it is unobservable.
 The x-variable, income, cannot be used to predict the value of .

 = 0 has two implications.

 cov 0

Copyright ©2018 John Wiley & Son, Inc. 9

 2.2.3 The Regression Function
 The conditional expectation + is called the regression function.

 This says the population the average value of the dependent variable for the ith

observation, conditional on xi is given by +

 This also says given a change in x, Δx, the resulting change in is holding all else
constant.
 We can say that a change in x leads to, or causes, a change in the expected
(population average) value of y given , .

Copyright ©2018 John Wiley & Son, Inc. 10

Copyright ©2018 John Wiley & Son, Inc. 11
2.2.4 Random Error Variation
 Ideally, the conditional variance of the random error is constant.

 This is the homoskedasticity assumption.

 Assuming the population relationship + the conditional variance

of the dependent variable is

=
 If this assumption is violated, and then the random errors are
said to be heteroskedastic.

Copyright ©2018 John Wiley & Son, Inc. 12

Copyright ©2018 John Wiley & Son, Inc. 13
 2.2.5 Variation in X
 In a regression analysis, one of the objectives is to estimate
.
 If we are to hope that a sample of data can be used to estimate the
effects of changes in x, then we must observe some different
values of the explanatory variable x in the sample.
 The minimum number of x-values in a sample of data that will
allow us to proceed is two.

Copyright ©2018 John Wiley & Son, Inc. 14

2.2.6 Error Normality
 It is not at all necessary for the random errors to be conditionally
normal in order for regression analysis to “work.”
 When samples are small, it is advantageous for statistical inferences
that the random errors, and dependent variable y, given each x-
value, are normally distributed.
 Central Limit Theorem says roughly that collections of many
random factors tend toward having a normal distribution.
 It is entirely plausible that the random are normally distributed.
Copyright ©2018 John Wiley & Son, Inc. 15
2.2.7 Generalizing the Exogeneity
Assumption
 A lack of independence occurs naturally when using financial or
macroeconomic time-series data.
 The data series is likely to be correlated across time.

 The assumption that the pairs (, ) represent random iid draws from
a probability distribution is not realistic.
 We cannot predict the random error at time , using any of the
values of the explanatory variable.

Copyright ©2018 John Wiley & Son, Inc. 16

 2.2.8 Error Correlation
 It is possible that there are correlations between the random error terms.

 With cross-sectional data, data collected at one point in time, there may be a lack
of statistical independence between random errors for individuals who are
spatially connected.
 Within a larger sample of data, there may be clusters of observations with
correlated errors because of the spatial component.

Copyright ©2018 John Wiley & Son, Inc. 17

2.2.9 Summarizing the Assumptions

Copyright ©2017 John Wiley & Son, Inc. 18

2.3 Estimating the Regression
Parameters (1 of 2)
 We can use the sample information in
Table 2.1, specific values of and , to
estimate the unknown regression
parameters and
 These parameters represent the
unknown intercept and slope
coefficients for the food expenditure–
income relationship.

Copyright ©2018 John Wiley & Son, Inc. 19

 2.3 Estimating the Regression
Parameters (2 of 2)
 If we represent the 40 data points as ,
and plot them, we obtain the scatter
diagram in Figure 2.6.
 Our problem is to estimate the location
of the mean expenditure line.
 We would expect this line to be
somewhere in the middle of all the data
points.

Copyright ©2018 John Wiley & Son, Inc. 20

2.3.1 The Least Squares Principle
 The fitted regression line is

(2.5) yˆ i b1  b2 xi

 The least squares residual is

(2.6) eˆi  yi  yˆ i  yi  b1  b2 xi
Copyright ©2018 John Wiley & Son, Inc. 21
Graphs of the fitted regression line and least
squares residual equations

Copyright ©2018 John Wiley & Son, Inc. 22

 The Ordinary Least Squares (OLS)
Estimators
 We will call the estimators and , given in equations (2.7) and (2.8), the ordinary

least squares estimators. “Ordinary least squares” is abbreviated as OLS.

(2.7)
b2 
 ( x  x )( y 
i i y)
 (x  x) i
2

(2.8)
b1  y  b2 x

Copyright ©2018 John Wiley & Son, Inc. 23

2.3.2 Other Economic Models
 The simple regression model can be applied to estimate the parameters of

many relationships in economics, business, and the social sciences.

 Any time you ask how much a change in one variable will affect another

variable, regression analysis is a potential tool.

 Similarly, any time you wish to predict the value of one variable given the

value of another, then least squares regression is a tool to consider.

Copyright ©2018 John Wiley & Son, Inc. 24

 2.4 Assessing the Least Squares
Estimators
We call and the least squares estimators. We can investigate the properties of the
estimators b1 and b2, which are called their sampling properties, and deal with the
following important questions:

1. If the least squares estimators are random variables, then what are
their expected values, variances, covariances, and probability
distributions?

2. How do the least squares estimators compare with other procedures

that might be used, and how can we compare alternative estimators?

Copyright ©2018 John Wiley & Son, Inc. 25

2.4.1 The Estimator b2
 The estimator b2 can be rewritten as:
(2.10) N
b2  wi yi
i 1

 Where xi  x
wi 
(2.11)  ( xi  x) 2

 It could also be written as:

(2.12) b2 β 2   wi ei

Copyright ©2018 John Wiley & Son, Inc. 26

2.4.2 The Expected Values of b1 and b2
(1 of 2)
 We will show that if our model assumptions hold, then E(b2) = β2, which means

that the estimator is unbiased. We can find the expected value of b2 using the
fact that the expected value of a sum is the sum of the expected values:

(2.13)

 Using E(ei)=0 and E(wiei)=wiE(ei)

Copyright ©2018 John Wiley & Son, Inc. 27

2.4.2 The Expected Values of b1 and b2
(2 of 2)
The property of unbiasedness is about the average values of and if many
samples of the same size are drawn from the same population.
 If we took the averages of estimates from many samples, these averages
would approach the true parameter values and
 Unbiasedness does not say that an estimate from any one sample is close
to the true parameter value, and thus we cannot say that an estimate is
unbiased.
 We can say that the least squares estimation procedure (or the least
squares estimator) is unbiased.

Copyright ©2018 John Wiley & Son, Inc. 28

2.4.3 Sampling Variation
 To illustrate how the concept
of unbiased estimation
relates to sampling variation,
we present in Table 2.2 least
squares estimates of the food
expenditure model from 10
hypothetical random
samples.

Copyright ©2018 John Wiley & Son, Inc. 29

 2.4.4 The Variances and Covariance of
b1 and b2
If the regression model assumptions SR1-SR5 are correct, then the
variances and covariance of b1 and b2 are:

 (2.14) 2
var(b1 ) σ 
  x 2
i


 N   xi  x  
2

σ2
var(b2 ) 
 (2.15)  x  x i
2

 x 
2
cov(b1 , b2 ) σ  
   xi  x  
2
 (2.16)
Copyright ©2018 John Wiley & Son, Inc. 30
 Major Points About the Variances and
Covariances of b1 and b2
1. The larger the variance term, the greater the uncertainty there is in the statistical model,
and the larger the variances and covariance of the least squares estimators.

The larger the sum of squares, x  x  , the smaller the variances of the least squares
2
2. i

estimators and the more precisely we can estimate the unknown parameters.

3. The larger the sample size N, the smaller the variances and covariance of the least
squares estimators.

The larger the term  x

2
4. i
, the larger the variance of the least squares estimator b1.

5. The absolute magnitude of the covariance increases the larger in magnitude is the
sample mean x , and the covariance has a sign opposite to that of x .
Copyright ©2018 John Wiley & Son, Inc. 31
2.5 The Gauss—Markov Theorem

Given x and under the assumptions SR1–SR5 of the linear regression

model, the estimators b1 and b2 have the smallest variance of all

linear and unbiased estimators of b1 and b2. They are the best linear

unbiased estimators (BLUE) of b1 and b2.

Copyright ©2018 John Wiley & Son, Inc. 32

 Major Points About the Gauss-Markov
Theorem (1 of 2)
1. The estimators b1 and b2 are “best” when compared to similar estimators, those which
are linear and unbiased. The Theorem does not say that b1 and b2 are the best of all
possible estimators.

2. The estimators b1 and b2 are best within their class because they have the minimum
variance. When comparing two linear and unbiased estimators, we always want to use
the one with the smaller variance, because that estimation rule gives us the higher
probability of obtaining an estimate that is close to the true parameter value.

3. In order for the Gauss-Markov Theorem to hold, assumptions SR1-SR5 must be true. If
any of these assumptions are not true, then b1 and b2 are not the best linear unbiased

estimators of β1 and β2.

Copyright ©2018 John Wiley & Son, Inc. 33
 Major Points About the Gauss-Markov
Theorem (2 of 2)
4. The Gauss-Markov Theorem does not depend on the assumption of normality (assumption

SR6).

5. In the simple linear regression model, if we want to use a linear and unbiased estimator, then

we have to do no more searching. The estimators b1 and b2 are the ones to use. This explains

why we are studying these estimators and why they are so widely used in research, not only

in economics but in all social and physical sciences as well.

6. The Gauss-Markov theorem applies to the least squares estimators. It does not apply to the

least squares estimates from a single sample.

Copyright ©2018 John Wiley & Son, Inc. 34
2.6 The Probability Distributions of the
Least Squares Estimators
 If we make the normality assumption (assumption SR6 about
the error term), and treat x as given, then the least squares
estimators are normally distributed:
 (2.17)

 (2.18)

Copyright ©2018 John Wiley & Son, Inc. 35

A Central Limit Theorem

A Central Limit Theorem: If assumptions SR1–SR5 hold, and if the

sample size N is sufficiently large, then the least squares estimators

have a distribution that approximates the normal distributions shown

in (2.17) and (2.18).

Copyright ©2018 John Wiley & Son, Inc. 36

2.7 Estimating the Variance of the Error
Term
 The variance of the random error ei is:
var(ei | x) 2 E{[ei  E(ei | x)]2 | x} E(e i2 | x)

 If the assumption E(ei) = 0 is correct

 Because the “expectation” is an average value we might consider

estimating σ2 as the average of the squared errors:

σ̂ 2 
i
e 2

N
 Where the error terms are ei  yi  β1  β 2 xi

Copyright ©2018 John Wiley & Son, Inc. 37

 2.7.1 Estimating the Variances and
Covariance of the Least Squares
Estimators (1 of 2)
2
 Replace the unknown error variance σ2 in (2.14) – (2.16) by ˆ to obtain:

 (2.20)

 (2.21)

 (2.22)

Copyright ©2018 John Wiley & Son, Inc. 38

 2.7.1 Estimating the Variances and
Covariance of the Least Squares
Estimators (2 of 2)
 Replace the unknown error variance σ2 in (2.14) – (2.16) by ˆ 2 to
obtain:
 (2.23) 𝑠𝑒 (𝑏1)= √ ¿ ¿
 (2.24) 𝑠𝑒 (𝑏2 )= √ ¿ ¿

Copyright ©2018 John Wiley & Son, Inc. 39

 2.7.2 Interpreting the Standard Errors (1
of 3)
 The standard errors of b1 and b2 are measures of the sampling variability of the

least squares estimates b1 and b2 in repeated samples.

 The estimators are random variables. As such, they have probability

distributions, means, and variances.

 In particular, if assumption SR6 holds, and the random error terms ei are

normally distributed, then:

Copyright ©2018 John Wiley & Son, Inc. 40

 2.7.2 Interpreting the Standard Errors (2
of 3)
 The estimator variance, var(b2), or its square root,  b  var(b 2 | x ) which
2

we might call the true standard deviation of b2, measures the sampling

variation of the estimates b2.

 b2
 The bigger is the more variation
 in the least squares estimates b2 we
b2
see from sample to sample. If is large then the estimates might
change a great deal from sample to sample.
 b2
 If is small relative to the parameter b2, we know that the least

squares estimate will fall near b2 with high probability.

Copyright ©2018 John Wiley & Son, Inc. 41
2.7.2 Interpreting the Standard Errors (3
of 3)

The question we address with the standard error is

‘‘How much variation about their means do the

estimates exhibit from sample to sample?’’

Copyright ©2018 John Wiley & Son, Inc. 42

2.8 Estimating Nonlinear Relationships
 Economic variables are not always related by straight-line
relationships; in fact, many economic relationships are represented
by curved lines, and are said to display curvilinear forms.

 Fortunately, the simple linear regression model y = β1 + β2 + e is much

more flexible than it looks at first glance.

 The variables y and x can be transformations, involving logarithms,
squares, cubes, or reciprocals, of the basic economic variables, or
they can be indicator variables that take only the values zero and one.

Copyright ©2018 John Wiley & Son, Inc. 43

 Nonlinear Relationships House Price
Example
 Consider the linear model of house prices:

 Where SQFT is the square footage

 It may be reasonable to assume that larger and more expensive homes

have a higher value for an additional square foot of living area than
smaller, less expensive, homes

 We can build this into our model in two ways:

 A quadratic equation in which the explanatory variable is SQFT2

 A loglinear equation in which the dependent variable is ln(PRICE)

Copyright ©2018 John Wiley & Son, Inc. 44
2.8.1 Quadratic Functions

The quadratic function y = β1 + β2x2 is a parabola.

The elasticity, or the percentage change in y given a 1% change in

2
x, is:  slope x y 2bx y

Copyright ©2018 John Wiley & Son, Inc. 45

 2.8.2 Using a Quadratic Model
 A quadratic model for house prices includes the squared value of
SQFT, giving:

 (2.26) PRICE α1  α 2 SQFT 2  e

 The slope is:

 (2.27)

d PRICE  2αˆ SQFT
2
dSQFT

 If α̂ 2  0 , then larger houses will have larger slope, and a larger

estimated price per additional square foot.
Copyright ©2018 John Wiley & Son, Inc. 46
Copyright ©2018 John Wiley & Son, Inc. 47
 2.8.3 A Log-Linear Function
 The log-linear equation ln(y) = a + bx has a logarithmic term on the left-hand side

of the equation and an untransformed (linear) variable on the right-hand side

 Both its slope and elasticity change at each point and are the same sign as b

 The slope is: dy dx by

 The elasticity, the percentage change in y given a 1% increase in x, at a point

on this curve is:  slope x y bx

Copyright ©2018 John Wiley & Son, Inc. 48

2.8.4 Using a Log-Linear Model

 Consider again the model for the price of a house as a function of

the square footage, but now written in semi-log form:

 (2.29) ln  PRICE  γ1  γ 2 SQFT  e

 This logarithmic transformation can regularize data that is

skewed with a long tail to the right.

Copyright ©2018 John Wiley & Son, Inc. 49

Figure 2.16 (a) Histogram of PRICE, (b) Histogram of ln(PRICE)

Copyright ©2018 John Wiley & Son, Inc. 50

Copyright ©2018 John Wiley & Son, Inc. 51
2.8.5 Choosing a Functional Form
 We should do our best to choose a functional form that:

 Consistent with economic theory

 Fits the data well

 Such that the assumptions of the regression model are satisfied

 In real-world problems it is sometimes difficult to achieve all these

goals.
 In applications of econometrics we must simply do the best we can
to choose a satisfactory functional form.

Copyright ©2018 John Wiley & Son, Inc. 52

 2.9 Regression with Indicator Variables
(1 of 2)
 An indicator variable is a binary variable that takes the values zero or

one; it is used to represent a nonquantitative characteristic, such as

gender, race, or location.

1 house is in University Town
UTOWN 
0 house is in Golden Oaks

PRICE β1  β 2UTOWN  e

 How do we model this?

Copyright ©2018 John Wiley & Son, Inc. 53
 2.9 Regression with Indicator Variables
(2 of 2)
 When an indicator variable is used in a regression, it is important to write the

regression function for the different values of the indicator variable.

β1  β 2 if UTOWN  1
E  PRICE  
β1 if UTOWN  0
 In the simple regression model, an indicator variable on the right-hand side

gives us a way to estimate the differences between population means.

Copyright ©2018 John Wiley & Son, Inc. 54

 2.10 The Independent Variable
This section contains a more advanced discussion of the assumptions of the
simple regression model.

 We say more about different possible DGPs.

 Explore their implications for the assumptions of the simple regression

model.

 Investigate how the properties of the least squares estimator change, if at

all, when we no longer condition on x.
Copyright ©2018 John Wiley & Son, Inc. 55
2.10.1 Random and Independent x

Copyright ©2018 John Wiley & Son, Inc. 56

 2.10.2 Random and Strictly Exogenous x
 Statistical independence between xi and xj for all values of i and j is a very strong
assumption and most likely only suitable in experimental situations.

 A weaker assumption is that the explanatory variable x is strictly exogenous.

 The implications of strict exogeneity:

 Implication 1: E(ei)=0. The “average” of all factors omitted from the regression
model is zero.

 Implication 2: cov(xi, ej)=0 . There is no correlation between the omitted factors

associated with observation j and the value of the explanatory variable for
observation.

Copyright ©2018 John Wiley & Son, Inc. 57

2.10.3 Random Sampling
 Survey methodology is an important area of statistics.

 The idea is to collect data pairs (yi , xi ) in such a way that the ith pair is

statistically independent of the pair.

 This ensures that xj is statistically independent of ej .

Copyright ©2018 John Wiley & Son, Inc. 58

 Key Words
 assumptions  heteroskedastic  random error term
 asymptotic  homoskedastic  random-x
 biased estimator  independent variable  regression model
 BLUE  indicator variable  regression parameters
 degrees of freedom  
least squares estimates repeated sampling
 dependent variable  
least squares estimators sampling precision
 deviation from the mean  
least squares principle sampling properties
form
 linear estimator  simple linear regression analysis
 econometric model
 log-linear model  simple linear regression
 economic model

 nonlinear relationship  specification error
elasticity
  prediction  strictly exogenous
exogenous variable
  quadratic model  unbiased estimator
Gauss–Markov theorem
Copyright ©2018 John Wiley & Son, Inc. 59
Copyright
Copyright © 2018 John Wiley & Sons, Inc.
All rights reserved. Reproduction or translation of this work beyond that permitted in
Section 117 of the 1976 United States Act without the express written permission of the
copyright owner is unlawful. Request for further information should be addressed to the
Permissions Department, John Wiley & Sons, Inc. The purchaser may make back-up copies
for his/her own use only and not for distribution or resale. The Publisher assumes no
responsibility for errors, omissions, or damages, caused by the use of these programs or
from the use of the information contained herein.

Copyright ©2017 John Wiley & Son, Inc. 60