Lecture 2

Ordinary Least Squares

Doruk Gökalp
Good Evening
• Last time:
• What is econometrics?
• What are the sources of economic data?
• What types of economic data are there?

• Today:
• Study correlation between two variables 𝑌 and 𝑋
• The concept of “The OLS Coefficient”
You will hear the term “OLS” millions of times in the future.
The Simple Linear Regression Model

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖

• We assume, the relation between 𝑌 and 𝑋 is as above.

• This is a “model” in the sense that we assume reality is described this way.

• As such, we treat 𝛾0 and 𝛾1 as “the truth we are after”.

• Every 𝑖 is a different individual, every 𝜀 is a different “residual error”.
The Simple Linear Regression Model

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖

• Y here is called the “dependent” variable.

• X is called the “independent variable.
• 𝛾0 , 𝛾1 are called the “regression coefficients”.
• We call 𝛾0 the intercept, and 𝛾1 the slope.
• 𝜀 is called the “residual” or “error”.

• This equation regresses Y on X. (Back out Y given X).

Important: How to Interpret the model?

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖

• This equation comes from idea that, “in reality”, 𝐸 𝑌 𝑋 = 𝛾0 + 𝛾1 𝑋

• This is what we refer to as “the model”.
• Given X, the average value of Y is given by intercept + slope times X.

• The error term, 𝜀𝑖 , is the error you make on individual observations.

• This is a predictive equation – you are predicting the value of Y.
• This is NOT a causal equation in itself. Just a correlation.
Important: How to Interpret the model?

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖
• Our goal is to find 𝛾0 , 𝛾1 , and in some cases, 𝜀𝑖 .
• They are not observable. We need to estimate them.

• How do we do that? How do we know how good is the estimate?

Throughout this lecture, with hats we will denote “estimates”.

If someone gives you an estimate for 𝛾0 and 𝛾1 , then:

𝑌𝑖 = 𝛾ො0 + 𝛾ො1 𝑋𝑖 + 𝜀𝑖Ƹ

Important: How to Interpret the model?

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖
Important: How to Interpret the model?

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖
Important: How to Interpret the model?

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖

𝑌෠𝑖 = 𝛾ො0 + 𝛾ො1 𝑋𝑖

We call this “the fitted model”.

Important: How to Interpret the model?

𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖

𝜀𝑖Ƹ = 𝑌𝑖 − 𝑌෠𝑖

𝑌෠𝑖 = 𝛾ො0 + 𝛾ො1 𝑋𝑖

 𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖
• How to find 𝛾0 and 𝛾1 ?
• Idea: Minimize the squared errors! Denote your “estimate” of them with hats, 𝛾
ෞ0 , 𝛾ෝ1

SSR = σ𝑛 𝜀
𝑖=1 𝑖Ƹ 2
= σ 𝑛
𝑌
𝑖=1 𝑖 − 𝛾
ෞ0 − 𝛾
ෝ 𝑋
1 𝑖
2

𝑛
𝜕𝑆𝑆𝑅
= −2 ෍ 𝑌𝑖 − 𝛾 ෞ0 = 𝑌ത − 𝛾ෝ1 𝑋ത
ෞ0 − 𝛾ෝ1 𝑋𝑖 = 0 ⇒ 𝛾
𝜕ෞ
𝛾0
𝑖=1
 SSR = σ𝑛 𝜀
𝑖=1 𝑖Ƹ 2
= σ 𝑛
𝑌
𝑖=1 𝑖 − 𝛾
ෞ0 − 𝛾
ෝ 𝑋
1 𝑖
2

ෞ0 = 𝑌ത − 𝛾ෝ1 𝑋ത
𝛾

𝑛 1 𝑛
𝜕𝑆𝑆𝑅 σ𝑖 𝑌𝑖 𝑋𝑖 − 𝑋ത 𝑌ത
= −2 ෍ 𝑌𝑖 − 𝛾
ෞ0 − 𝛾ෝ1 𝑋𝑖 𝑋𝑖 = 0 ⇒ 𝛾ෝ1 = 𝑛
𝜕𝛾ෝ1 1 𝑛 2
𝑖=1 (σ𝑖 𝑋𝑖 ) − 𝑋ത 2
𝑛

1 𝑛 ത 𝑖 − 𝑋)ത
σ𝑖 (𝑌𝑖 − 𝑌)(𝑋 ෣
𝐶 𝑂𝑉 𝑋, 𝑌
𝛾ෝ1 = 𝑛 =
1 𝑛 ෣
𝑉 𝐴𝑅(𝑋)
σ𝑖 𝑋𝑖 − 𝑋ത 2
𝑛
International convergence
• Solow model predicts that poor
countries will “catch up” with
developed countries (at least
when it comes to GDP per
capita)
• In the data: poorer countries in
the past must grow quicker than
richer countries
• Negative relationship between
both
International convergence
Let’s study the relationship in this
data by estimating the following
statistical model:

𝐺𝑅𝑖 = 𝛾0 + 𝛾1 𝐺𝐷𝑃𝑖 + 𝜀𝑖

• This is a linear regression

• No causal interpretation
whatsoever
• Only association
International convergence
Goal: find the coefficients 𝛾ො0 and 𝛾ො1
that finds the closest linear equation to
the data.

Define the error term:

𝜀𝑖Ƹ = 𝐺𝑅𝑖 − 𝛾ො0 − 𝛾ො1 𝐺𝐷𝑃𝑖

We will then minimize the total

distance:
𝑁
2
𝑆 = ෍ 𝐺𝑅𝑖 − 𝛾ො0 − 𝛾ො1 𝐺𝐷𝑃𝑖
𝑖=1
Derivation of OLS estimators
𝑌𝑖 = 𝛾0 + 𝛾1 𝑋𝑖 + 𝜀𝑖
𝑁
2
𝑆𝑆𝑅 = ෍ 𝑌𝑖 − 𝛾ො0 − 𝛾ො1 𝑋𝑖
𝑖=1
Summary
First-order conditions:
𝜕𝛾0 : −2σ 𝑌𝑖 − 𝛾ො0 − 𝛾ො1 𝑋𝑖 = 0
𝜕𝛾1 : −2σ 𝑌𝑖 − 𝛾ො0 − 𝛾ො1 𝑋𝑖 𝑋𝑖 = 0

Mechanically:
• Sum of the residuals is zero
• Residuals and regressors are
orthogonal
• No more correlation to be extracted
The solution
𝜕𝛾0 : −2σ 𝑌𝑖 − 𝛾ො0 − 𝛾ො1 𝑋𝑖 = 0
𝜕𝛾1 : −2σ 𝑌𝑖 − 𝛾ො0 − 𝛾ො1 𝑋𝑖 𝑋𝑖 = 0
International convergence
International convergence
Regression equation:

𝐺𝐷𝑃
෢ = 0.025 − 0.0034
𝐺𝑅
1000

An increase of 1,000 dollars in GDP/capita in

1870 IS ASSOCIATED with a decrease of 0.0034
percent point in the average growth rate.
• Important!

• The previous analysis says NOTHING causal. This is just a correlation!

Ordinary Least Squares (OLS) is NOT a model. It is an estimator!

Benefits of OLS
• OLS estimators have some desirable properties…
• They are called “BLUE”
• Best Linear Unbiased Estimators.
• Unbiased means 𝐸 𝛾ො = 𝛾, no structural biases.
• Linear means the estimator arises from a linear relationship.
• Best means it has the most information content among this class.
(More on this later!)
Done for today!
• Today:
• How to derive OLS estimators.
• An example with them.
• Their basic properties.
• Unbiased
• Linear
• Next Week:
• Learning from OLS Coefficients (Hypothesis Tests).
• Weaknesses of the simple linear regression model.
Lead into Tuesday’s lecture…
• From the discussion on Tuesday:
our sample is random, so our
estimators 𝛾ො0 and 𝛾ො1 are random
variables.

• 2 samples from the same

population yield different
numbers for 𝛾ො0 and 𝛾ො1
Next Week
• Central Limit Theorem:

𝛾ො1 − 𝛾1
𝑡= ~𝑁𝑜𝑟𝑚𝑎𝑙(0,1)
𝑠ෞ𝑒 𝛾ො1

• How to use all this for statistical inference on the coefficients.