CHAPTER 2

SIMPLE LINEAR
REGRESSION MODEL

Le Hang My Hanh, FTU CS2 1

Outline
1. Definition
1.1 The Linear Regression model
1.2. Population Regression Function (PRF)
1.3. Sample regression function (SRF)
2. Estimating the coefficients of the regression
function
2.1 The Method of Ordinary Least Squares (OLS)
2.2 Unbiased and efficient properties
2.3 The assumptions underlying the method of
least squares
3. A measure of “Goodness of fit”
Le Hang My Hanh, FTU CS2 2
1. The Linear Regression Model
• Galton’s law of universal regression: how
the average height of sons changes, given
the fathers’ height
• The average height of children born of
parents of given height tended to move or
regress toward the average height in the
population as a whole.

Le Hang My Hanh, FTU CS2 3

Regression analysis

Le Hang My Hanh, FTU CS2 4

 Regression analysis?
Is the study of the dependence of one variable,
the dependent variable (y), on one or more
other variables, the explanatory variables (x).

• With a view to estimating and/or predicting the

(population) mean or average value of the
dependent variable on the basis of the known
or fixed values of the explanatory variable(s).

Le Hang My Hanh, FTU CS2 5

• A hypothetical example
• Table 2.1: a total population of 60 families
and their weekly income (X) and weekly
consumption expenditure (Y). The 60 families
are divided into 10 income groups.

Le Hang My Hanh, FTU CS2 6

Le Hang My Hanh, FTU CS2 7
Some findings
• There is considerable variation in weekly
consumption expenditure in each income group.
• On the average, weekly consumption
expenditure increases as income increases. →
The dependence of consumption expenditure
on income.

Le Hang My Hanh, FTU CS2 8

The population regression line (PRL)

9
1.2. Population Regression Function (PRF)
• E(Y/Xi)= f(Xi) → conditional expectation function (CEF) or
population regression function (PRF) → How the mean or
average response of Y varies with X.
• The functional form of the PRF is an empirical question. For
example, assume : E(Y | Xi) = β1 + β2Xi.

Our interest is in estimating the unknown beta 1, beta 2 on the

basis of observations on Y and X.

• One independent variable = simple regression

• More than one independent variable = multiple regression

Le Hang My Hanh, FTU CS2 10

1.2. Population Regression Function (PRF)

Le Hang My Hanh, FTU CS2 11

 Stochastic specification of PRF
• The deviation of an individual Yi around its
expected value: ui = Yi − E (Y | X i )
Yi = E (Y | X i ) + ui = 1 +  2 X i + ui

• E(Y|Xi) = the mean consumption of all families

with the same level of income → the
systematic, or deterministic component
• ui = stochastic disturbance or stochastic error
term, the random, or nonsystematic
component.
Le Hang My Hanh, FTU CS2 12
The significance of the stochastic error term
The error term is a representative for all those
variables that are omitted from the model but
that collectively affect Y.
• Why don’t we introduce them into the model
explicitly?
→ Vagueness of theory
→ Unavailability of data
→ Core variables versus peripheral variables
→ Limited resource

Le Hang My Hanh, FTU CS2 13

The meaning of term linear
❑ Linearity in the Variables
• The conditional expectation of Y is a linear function of Xi,
the regression curve in this case is a straight line. But
E(Y | Xi) = β1 + β2X2i is not a linear function
❑ Linearity in the Parameters
• The conditional expectation of Y, E(Y | Xi), is a linear
function of the parameters, the β’s; it may or may not be
linear in the variable X.
E(Y | Xi) = β1 + β2X2i is a linear (in the parameter)
regression model.
The term “linear” regression will always mean a
regression that is linear in the parameters.
Le Hang My Hanh, FTU CS2 14
Examples of linear regression models

1
Yi = 1 +  2 ( ) + ui
Xi

ln Yi =  +  ln X i + ui
Yi = 1 +  2 X i + 3 X + ui i
2

Le Hang My Hanh, FTU CS2 15

 1.3. The Sample Regression Function (SRF)

A sample of Y values corresponding to some fixed

X’s. Can we estimate the PRF from the sample data?

• We may not be able to estimate the PRF

“accurately” because of sampling fluctuations.
• Suppose we draw another random sample from
the population of Table 2.1, as presented in Table
2.4 and 2.5, we obtain the scatter-gram given in
Figure 2.4.

Le Hang My Hanh, FTU CS2 16

1.3 The Sample Regression Function (SRF)

Le Hang My Hanh, FTU CS2 17

1.3 The Sample Regression Function (SRF)

Le Hang My Hanh, FTU CS2 18

 1.3 The Sample Regression Function (SRF)
• Population regression function
E(Y / X i ) = f ( X i ) = 1 +  2 X i
• Sample regression function Yˆi = ˆ1 + ˆ2 X i

• 𝑌෠ i = estimator of E(Y | Xi)

• 𝛽መ 1 = estimator of β1
• 𝛽መ 2 = estimator of β2

• An estimator, also known as a (sample) statistic, is simply a

rule or formula or method that tells how to estimate the
population parameter from the information provided by
the sample.
• A particular numerical value obtained by the estimator in
an application is known as an estimate.
Le Hang My Hanh, FTU CS2 19
1.3 The Sample Regression Function (SRF)
• The stochastic form of SRF:
Yi = ˆ1 + ˆ2 X i +uˆ i

𝑢ො i = the (sample) residual term.

• Conceptually 𝑢ො i is analogous to u and can be i

regarded as an estimate of ui. It is introduced

in the SRF for the same reasons as ui was
introduced in the PRF.

Le Hang My Hanh, FTU CS2 20

1.3 The Sample Regression Function (SRF)
• Our primary objective in regression analysis is
to estimate the PRF
Yi = E(Y | X i ) + ui = 1 +  2 X i + ui

= ˆ + ˆ X +uˆ

• On the basis of the SRF i
Y 1 2 i i

How should the SRF be constructed so that βˆ1

is as “close” as possible to the true β1 and βˆ2
is as “close” as possible to the true β2 even
though we will never know the true β1 and β2?

Le Hang My Hanh, FTU CS2 21

The SRF is an approximation of the PRF

Le Hang My Hanh, FTU CS2 22

 2. Estimating the coefficients of the
linear regression model
2.1 Ordinary least square estimators (OLS)
2.2 Unbiased and efficient properties
2.3 Assumptions of OLS

23
2.1 The method of Ordinary Least Squares (OLS)
• “The method of least squares is the
automobile of modern statistical analysis;
despite its limitations, occasional accidents,
and incidental pollution, it and its numerous
variations, extensions and related
conveyances carry the bulk of statistical
analysis, and are known and valued by all”.
Stephen M. Stigler

Le Hang My Hanh, FTU CS2 24

2.2.1 The method of Ordinary Least Squares (OLS)

• Two-variable PRF: Yi = 1 +  2 X i + ui
• The PRF is not directly observable. We estimate
it from the SRF: Yi = ˆ1 + ˆ2 X i +uˆ i

• Or uˆ i = Yi − Yˆi = Yi − ˆ1 − ˆ 2 X i
The residuals are the differences between the
actual and estimated Y values.

Le Hang My Hanh, FTU CS2 25

2.1 The method of Ordinary Least Squares (OLS)
• Given n pairs of observations on Y and X, we
would like to determine the SRF in such a
manner that it is as close as possible to the
actual Y. To this end, we may adopt the
following criterion:
Choose the SRF in such a way that the sum of
the residuals uˆi = Yi − Yˆi or Q is as small as
possible.
n
Q( 1 ,  2 ) =  (Yi − 1 −  2 X i )
ˆ ˆ ˆ ˆ 2

i =1
26
2.1 The method of Ordinary Least Squares (OLS)

27
 2.1 The Method of Ordinary Least Squares (OLS)
uˆi 2 = (Yi − ˆ1 − ˆ2 X i ) 2 (3.1.2 )
( )
n n 2

i  i 1 2 i
ˆ
u 2

i =1
= Y − ˆ − ˆ X → min

i =1

Derivation of least squares estimates: Derivative partially

with respect to βˆ1 and βˆ2 we obtain:
n
 ( uˆi2 ) n n  
i =1
 = −2 ui = −2 (Yi − 1 −  2 X i ) = 0
1 i =1 i =1
n
 ( uˆi2 ) n n  
i =1
 = −2 ui X i = −2 X i (Yi − 1 −  2 X i ) = 0
 2 i =1 i =1

Setting these equations to zero, we obtain 28

 2.1 The Method of Ordinary Least Squares

 i 1 2 Xi
Y = nˆ + ˆ (3.1.4)
 i i 1 i 2  i
X Y = ˆ
 X + ˆ
 X 2 (3.1.5)
▪ Solving the normal equations simultaneously, we obtain:

(3.1.6)

(3.1.7)
29
 2.1. The Method of Ordinary Least Squares

Review:
▪X,Y independent: var(X + Y) = var(X) + var(Y)
▪X,Y dependent: var(X + Y) = var(X) + var(Y) + 2Cov(X,Y)
▪Covariance

▪Correlation (to normalize covariance)

n n

(X i − X )(Yi − Y ) Y X i i − n. X .Y
cov( X , Y )
ˆ2 = i =1
n
= i =1
n
=
 i  i
Var ( X )
( X − X ) 2
X 2
− n.( X ) 2
30
i =1 i =1
2.1 The method of Ordinary Least Squares (OLS)

ˆ2 =
S xy ˆ1 = Y − ˆ2 X
S xx
• Where
S xy =  ( X i − X )(Yi − Y ) =  X iY i−nXY

S xx =  ( X i − X ) =  X − nX
2
i
2 2

S yy =  (Yi − Y ) 2 =  Yi 2 − nY 2

31
Example 1
• The effect of working time on income. Data on income in
dollars and working time of ten workers, X= working time,
Y= income
Obs X Y X2 Y2 XY

1 10 11 100 121 110

2 7 10 49 100 70
3 10 12 100 144 120
4 5 6 25 36 30
5 8 10 64 100 80
6 8 7 64 49 56
7 6 9 36 81 54
8 7 10 49 100 70
9 9 11 81 121 99
10 10 10 100 100 100
Sum 80 96 668 952 789
32
Example 1

S xx = 668 − 10  8 = 28 2

S xy = 789 − 10  8  9,6 = 21
S yy = 952 − 10  9,6 2 = 30,4
S xy 21
ˆ2 = = = 0,75
S xx 28
ˆ1 = Y − ˆX = 9,6 − 0,75  8 = 3,6
• SRF: Yˆ = 3,6 + 0,75 X
33
Example 1
• Eviews with Quick/Estimate Equation or Object /New
object/ Equation
• Stata: reg Y X
Variable Coefficient Std. Error t-Statistic Prob.

C 3.600000 2.090177 1.722342 0.1233

X 0.750000 0.255738 2.932692 0.0189

R-squared 0.518092 Mean dependent var 9.600000

Adjusted R-squared 0.457854 S.D. dependent var 1.837873
S.E. of regression 1.353237 Akaike info criterion 3.619732
Sum squared resid 14.65000 Schwarz criterion 3.680249
Log likelihood -16.09866 F-statistic 8.600683
Durbin-Watson stat 2.346416 Prob(F-statistic) 0.018920

34
Example 2
• The effect of rice price on rice demand. Y= demand (ton
per month), X= price (thousand dong per kg)

Stt Xi Yi XiYi X^2

1 1 10 10 1
2 4 6 24 16
3 2 9 18 4
4 5 5 25 25
5 5 4 20 25
6 7 2 14 49
sum 24 36 111 120
• SRF
Yˆi = 11.5 − 1.375 X i
35
• http://wps.pearsoned.co.uk/ema_ge_stock_ie
_3/193/49605/12699041.cw/content/index.ht
ml

36
37
Interpretation of coefficient estimates
• Relation between working time and income:
Yˆ = 3.6 + 0.75 X
→ If working time increases by 1 hour, the estimated
increase in wages is about 75 cents.

• Relation between rice price and demand price:

Yˆi = 11.5 − 1.375 X i

→ If the rice price increases by 1 dong per kg, the

estimated decrease in rice demand is about 1.375 ton/m

38
2.2. Properties of OLS statistics
• The sum and the sample average of the OLS
n
residuals is zero.  uˆ
i =1
i =0

• The sample covariance between the regressors

n
and the OLS residuals is zero.  X uˆ
i =1
i i =0

• The point ( X ,Y ) is always on the OLS

regression line.
(See Appendix 3A.1, p100, Gujarati)

39
2.2. Properties of least-squares estimators

• Gauss-Markov Theorem: Given the

assumptions of the classical linear regression
model, the least-squares estimators in the class
of unbiased linear estimators, have minimum
variance, that is, they are BLUE. (BLUE- Best
Linear Unbiased Estimator).

40
 Properties of OLS Estimators under the
Normality Assumption
• With the assumption that ui follow the normal
distribution, the OLS estimators have the following
properties:

• Yi is itself normally distributed with

• Mean: E(Yi ) = 1 +  2 X i
Var(Yi ) =  2

41
Precision or standard errors of Least-Squares estimates
• The least-squares estimates are a function of the sample data. But since
the data change from sample to sample, the estimates will change.
Therefore, what is needed is some measure of “reliability” or precision of
the estimators βˆ1 and βˆ2. In statistics the precision of an estimate is
measured by its standard error (se), which can be obtained as follows:

See Appendix 3A.2,

p100-102

42
Precision or standard errors of Least-Squares estimates

σ2 is the constant or homoscedastic variance of ui

See Ap 3.A5, 102

Where:
• ˆσ2 is the OLS estimator of the true but unknown
σ2 and where the expression n−2 is known as the
number of degrees of freedom (df).
• is the residual sum of squares (RSS).

See 3.5.2, 83

43
 Example
• Wage and education (WAGE1.dta)

44
 2.3. The assumptions underlying the OLS
Assumptions 1: Linear in Parameters. The
regression model is linear in the parameters.

Yi = 1 +  2 X i + ui
• Keep in mind that the regressand Y and the
regressor X themselves may be nonlinear.

45
 2.3. The assumptions underlying the OLS
Assumption 2: X values are fixed in repeated
samplings. X is assumed to be non-stochastic.

• Keeping the value of income X fixed, say, at $80,

we draw at random a family and observe its
weekly family consumption expenditure Y as, say,
$60. Still keeping X at $80, we draw at random
another family and observe its Y value as $75. In
each of these drawings (i.e., repeated sampling),
the value of X is fixed at $80.
• Our regression analysis is conditional regression
analysis, that is, conditional on the given values
of the regressor(s) X.
46
2.3. The assumptions underlying the OLS
Assumptions 3: Zero mean value of disturbance

E(ui / X i ) = 0
• Each Y population corresponding to a given X is
distributed around its mean value with some Y
values above the mean and some below it. The
mean value of these deviations corresponding to
any given X should be zero.
• Note that the assumption E(ui | Xi) = 0 implies
that E(Yi | Xi) = β1 + β2Xi.

47
Assumption 3
E(ui | Xi) = 0

48
 2.3. The assumptions underlying the OLS
Assumption 3 is satisfy : Zero covariance between ui and
Xi
Cov(ui X i ) = 0

• The disturbance u and explanatory variable X are

uncorrelated. The PRF assumes that X and u (which
may represent the influence of all the omitted variables)
have separate (and additive) influence on Y. But if X and
u are correlated, it is not possible to assess their
individual effects on Y.

49
 2.3. The assumptions underlying the OLS

Assumption 4: Homoscedasticity or equal

variance of disturbance
Var (ui | X i ) =  2

• The variation around the regression line

(which is the line of average relationship
between Y and X) is the same across the X
values; it neither increases or decreases as X
varies
50
51
52
 2.3. The assumptions underlying the OLS
Assumption 5: No autocorrelation between the
disturbance: Cov(ui .u j ) = 0 , i≠j.

• The disturbances ui and uj are uncorrelated, i.e., no serial

correlation. This means that, given Xi , the deviations of any
two Y values from their mean value do not exhibit patterns.
Ex: (Yt = β1 + β2Xt + ut) that ut and ut−1 are positively
correlated. Then Yt depends not only on Xt but also on ut−1
for ut−1 to some extent determines ut.
• We will see how intercorrelations among the disturbances
can brought into the analysis and with what consequences.

53
54
 2.3. The assumptions underlying the OLS

• Assumption 6: The number of observations n must be

greater than the number of parameters to be estimated.
• Assumption 7: Variability in X values. The X values in a
given sample must not all be the same.

55
 2.3. The assumptions underlying the OLS
Assumption 8: The regression model is correctly
specified. There is no specification bias or error in the
model used in empirical analysis.

• Some important questions that arise in the

specification of the model include the following:

(1) What variables should be included in the model?

(2) What is the functional form of the model? Is it
linear in the parameters, the variables, or both?
(3) What are the probabilistic assumptions made
about the Yi , the Xi, and the ui entering the model?
56
An example of assumption 9
• Two models depict the underlying
relationship between the rate of
change of money wages and the
unemployment rate:
Yi = α1 + α2Xi + ui

Yi = β1 + β2 (1/Xi ) + ui

• If the second model is the “correct” or

the “true” model, fitting the first one to
the scatterpoints shown in Figure 3.7
will give us wrong predictions.
57
58
 2.3. The assumptions underlying the OLS

Assumption 9: There is no perfect

multicollinearity.

• That is, there are no perfect linear

relationships among the independent
variables.
• → We will discuss in the multiple regression
models.

59
3. A measure of “Goodness of fit”
The goodness of fit: how “well” the sample
regression line fits the data.

• The coefficient of determination r2 (two-

variable case) or R2 (multiple regression) is a
summary measure that tells how well the
sample regression line fits the data.

60
3. A measure of “Goodness of fit”

61
3. A measure of “Goodness of fit”
• To compute this r2, we proceed as follows: Recall that
Yi = Yˆi +uˆ i
or in the deviation form yi = yˆi +uˆ i
Squaring on both sides and summing over the sample, we obtain

• Since = 0 and yˆi = βˆ2xi . (p83)

62
 3. A measure of “Goodness of fit”
• Total sum of squares (TSS): Total variation of
the actual Y values about their sample mean.

• Explained sum of squares (ESS): The variation

of the estimated Y values about their mean.

• Residual sum of squares (RSS): The residual of

the Y values about the regression line
• TSS = ESS +RSS

63
 3. A measure of “Goodness of fit”

The coefficient of
determination

64
3. A measure of “Goodness of fit”

65
3. A measure of “Goodness of fit”
• The coefficient of determination r2 is a measure of the
goodness of fit of a regression line.

r2 measures the proportion or percentage of the total

variation in Y explained by the regression model.

→ r2 is a nonnegative quantity.
→ 0 ≤ r2 ≤ 1

• An r2 of 1 means a perfect fit, that is, Yˆi = Yi for each i.

On the other hand, an r2 of zero means that there is no
relationship between the regressand and the regressor
whatsoever (i.e., βˆ2 = 0).

66
 Example
• CEO salary and ROE (CEOSAL1.DTA)
. reg salary roe

Source SS df MS Number of obs = 209

F( 1, 207) = 2.77
Model 5166419.04 1 5166419.04 Prob > F = 0.0978
Residual 386566563 207 1867471.32 R-squared = 0.0132
Adj R-squared = 0.0084
Total 391732982 208 1883331.64 Root MSE = 1366.6

salary Coef. Std. Err. t P>|t| [95% Conf. Interval]

roe 18.50119 11.12325 1.66 0.098 -3.428196 40.43057

_cons 963.1913 213.2403 4.52 0.000 542.7902 1383.592

67
Example: CEO salary and ROE

68