ECONOMETRICS

Simple Linear Regression

Professor: Antonio Di Paolo

Degree in International Business

Academic Year 2017/2018
Universitat de Barcelona

1
Introduction (1)
 What is Econometrics?
- Econometrics concerns with the application of mathematical statistics and the tools of statistical inference to the empirical
measurement of relationships postulated by underlying (economic) theory (Greene, 2012).
- Econometrics is based upon the development of statistical methods for estimating economic relationships, testing economic
theories and evaluating and implementing government and business policies (Wooldridge, 2013).

 Two main branches:

1) Microeconometrics
- What is the effect of participating in a training program on workers’ productivity?
- Do smaller classes improves students’ achievements?
- Is the share of tertiary educated workers beneficial for firm’s productivity?
- What is the effect of R&D expenditure on innovation (from cross-section data)?

2) Macroeconometrics
- What is the effect of human capital on economic growth?
- What is the relationship between unemployment rate and inflation rate?
- To what extent the abolition of custom duties stimulates international trade?
- What is the effect of R&D expenditure on innovation (from time series data?)

 An important distinction: Econometric Theory vs Applied Econometrics.

2
Introduction (2)
 Deterministic vs empirical (economic) relationships.

- Let’s consider a simple Keynes’s model of consumption (C = consumption, I = income):

𝐶 = 𝑓 𝐼 ֜ 𝑙𝑖𝑛𝑒𝑎𝑟 𝑚𝑜𝑑𝑒𝑙: 𝐶 = 𝑎 + 𝑏𝐼
The parameter “b” (marginal propensity to consume) should be comprised between 0 and 1 and the
parameter “a” should be greater than zero.

- Is the theory consistent with the real world (i.e. with the data)?

The empirical counterpart of the Keynes’s model must include the inherent “randomness” of real world (ε).

𝐶 = 𝑓 𝐼, 𝜀 ֜ 𝑙𝑖𝑛𝑒𝑎𝑟 𝑒𝑚𝑝𝑖𝑟𝑖𝑐𝑎𝑙 𝑚𝑜𝑑𝑒𝑙: 𝐶 = 𝛼 + 𝛽𝐼 + 𝜀

 The term ε (error term) captures all the unobserved factors that may affect consumption on top of income,
as well as errors in measuring the econòmic variables of interest.
3
Introduction (3)
 Types of data in Econometrics?
- Cross-sectional data: sample of individuals, households, firms, cities, regions, countries that is taken in a
given (i.e. fixed) period of time.

4
Introduction (4)
 Types of data in Econometrics?
- Time series data: observations of one or more variables over time, referring to a given subject (i.e. one firm,
one region, one country, etc.).

5
Introduction (5)
 Types of data in Econometrics?
- Panel or longitudinal data: time series for each cross-sectional member in the dataset (i.e. the same
individual/firm is followed over time).

6
Introduction (6)
 Association, causality and the notion of “ceteris paribus”
- Our final aim as economists (and applied econometricians) should be to inform policy-makers about the “causal” effect
of (e.g.) implementing a given policy on one or more relevant outcomes.
- Unfortunately, inferring about causal relationship from econometric models applied to non-experimental data is quite
difficult (and sometimes impossible).
- In the real world, there are always a variety of factors that could potentially affect the outcome of interest, but are not
directly observed (i.e. not available in the survey) or are implicitly unobservable (i.e. cannot be measured with data).
- With the tool of multiple regression (as you will see) it is possible to isolate the effect of one possible observable
determinant of the outcome from all the other observed factors that affect the outcome  this is the so-called notion
of “ceteris paribus” (i.e. other relevant factors being equal).
- However, there are always a large variety of “unobservables” that may co-vary with the outcome and its determinants,
so it is generally not possible to “control for everything”  in these cases, we would not be able to talk about causal
effects, but just about “conditional correlations” (some example?).

Possible solutions:
- Use more advanced techniques that could deal with the issue of causality ( Econometrics II in Eco)
- Use experimental data, either from the laboratory or from randomized experiments (more advanced)

7
Introduction (7)

 Outline of the course:

1) The simple and multiple linear regression analysis: Specification, estimation and
inference

2) Data problems in regression analysis: Outliers and Multicollinearity

3) Multiple Regression Analysis with Qualitative Information: Dummy Variables

4) Heteroscedasticity

5) Introduction to binary choice models (LPM, Probit and Logit)

6) Introduction to time series analysis

8
Linear Regression Model and Ordinary Least Squares (1)
 Let’s start with some definition,
- Y: dependent variable, or endogenous variable, or left-hand-side variable, or regressand…
- X: independent variable, or exogenous variable, or right-hand-side variable, or control variable, or predictor, or regressor….
 X will be one single variable in the simple regression framework and a vector of “k” variables in the multiple regression
framework (i.e. X = (X1, X2, X2,…..Xk)).

- ε (or u): error term, stochastic disturbance, unobservable(s)…..

 Dependent and independent variables are usually subscripted with either “i” (for cross-sections) or with “t” (for time series) or
both (“it”, for panel data).

- α (or β0) denotes the intercept/constant term of the regression.

- β (or βk) denotes the slope parameter that relates the dependent and independent variables.
A linear regression model with “k” variables will take the form: 𝑌𝑖 = 𝛼 + 𝛽1 𝑋1𝑖 + 𝛽2 𝑋2𝑖 + ⋯ + 𝛽𝑘 𝑋𝑘𝑖 + 𝜀𝑖

- The Ordinary Least Squares (OLS) is an estimation method that seeks to find the best linear approximation that fits the
relationship of interest.
We will start by considering OLS as an algebraic tool (rather than a statistical tool) and then we will describe and analyze the
conditions under which OLS is the “best” representation of the unknown parameters of interest (α and β).
9
Simple Linear Regression (1)
 The Regression with just a single independent variable (and a constant) is usually called “Simple Linear
Regression” (or Bivariate Regression).
 𝑌𝑖 = 𝛼 + 𝛽𝑋𝑖 + 𝜀𝑖 (notice that we suppressed some redundant subscript for simplicity).

- We have a sample of “n” observations, for which we observe the values of the two variables X and Y.
The next figure represents a Scatter Plot of the pairs (yi, xi).

 How can we find the line that better fits the points?
10
Simple Linear Regression (2)
 Intuitively, we would like to find the values of α and β that better approximates the points given by the observed
values of X and Y ( remember that there is just one line that crosses two given points in space).

- For each observation “i”, the difference between the line and the points of the scatter plot are the “errors” that
are committed by considering that the dependent variable Y is explained by X only.
Therefore, we would like to choose the values of α and β that minimizes the errors that are committed, in order to
find the best empirical approximation to the observed data.
However, positive and negative deviations would compensate each other…..

- The Ordinary Least Squares (OLS) method: minimizing the Sum of Squared Residuals (SSR), where the
residuals are the difference between the observed value of our dependent variable Y and the value of Y that
is derived (predicted) from the model.
2
𝑌෠ 𝑖 𝑌෠ 𝑖
2
መ 𝑖 ֜ 𝑆𝑆𝑅 = σ𝑛𝑖=1 𝜀𝑖Ƹ 2 = σ𝑛𝑖=1 𝑌𝑖 − 𝑌෠𝑖
 𝑌𝑖 = 𝛼 + 𝛽𝑋𝑖 + 𝜀𝑖 ֜ 𝜀ෝ𝑖 = 𝑌𝑖 − 𝑌෠𝑖 = 𝑌𝑖 − 𝛼ො + 𝛽𝑋 መ 𝑖
= σ𝑛𝑖=1 𝑌𝑖 − 𝛼ො + 𝛽𝑋

11
Simple Linear Regression (3)
- The Ordinary Least Squares (OLS) method: minimizing the Sum of Squared Residuals (SSR), where the
residuals are the difference between the observed value of our dependent variable Y and the value of Y that
is derived (predicted) from the model.
2
𝑌෠ 𝑖 𝑌෠ 𝑖
2
መ 𝑖 ֜ 𝑆𝑆𝐸 = σ𝑛𝑖=1 𝜀𝑖Ƹ 2 = σ𝑛𝑖=1 𝑌𝑖 − 𝑌෠𝑖
𝑌𝑖 = 𝛼 + 𝛽𝑋𝑖 + 𝜀𝑖 ֜ 𝜀ෝ𝑖 = 𝑌𝑖 − 𝑌෠𝑖 = 𝑌𝑖 − 𝛼ො + 𝛽𝑋 መ 𝑖
= σ𝑛𝑖=1 𝑌𝑖 − 𝛼ො + 𝛽𝑋

- OLS method consists in an optimization method that enables finding the values of α and β that minimize the Sum
of Squared Residuals, that is:
2
𝑛 𝑌෠ 𝑖
𝑚𝑖𝑛 𝑚𝑖𝑛
֜ መ 𝑆𝑆𝑅 = መ 𝑖
෍ 𝑌𝑖 − 𝛼ො + 𝛽𝑋 :
𝛼,
ො 𝛽 𝛼,
ො 𝛽መ
𝑖=1

- Since there are two parameters to be found, there will be two first-order conditions for this optimization process
(i.e. the derivative must be taken with respect to both α and β).
 For simplicity, we start with the intercept and then we proceed with the slope…..

12
Simple Linear Regression (4)
- OLS method consists in an optimization method that enables finding the values of α and β that minimize the
Sum of Squared Residuals, that is:
2
𝑛 𝑌෠ 𝑖
𝑚𝑖𝑛 𝑚𝑖𝑛
֜ መ 𝑆𝑆𝑅 = መ 𝑖
෍ 𝑌𝑖 − 𝛼ො + 𝛽𝑋 :
𝛼,
ො 𝛽 𝛼,
ො 𝛽መ
𝑖=1

Equating to 0 the derivative with respect to α gives the following result:

𝑛 𝑛 𝑛 𝑛 𝑛 𝑛
𝜕𝑆𝑆𝑅
֜ መ 𝑖 = ෍ 𝑌𝑖 − ෍ 𝛼ො − 𝛽መ ෍ 𝑋𝑖 = ෍ 𝑌𝑖 − 𝑛𝛼ො − 𝛽መ ෍ 𝑋𝑖 = 0
= −2 ෍ 𝑌𝑖 − 𝛼ො − 𝛽𝑋
𝜕𝛼ො
𝑖=1 𝑖=1 𝑖=1 𝑖=1 𝑖=1 𝑖=1

𝜕𝑆𝑆𝑅 σ𝑛𝑖=1 𝑌𝑖 𝑛𝛼ො σ𝑛𝑖=1 𝑋𝑖

֜ = − − 𝛽መ = 0 ֜ 𝛼ො = 𝑌ത − 𝛽መ 𝑋ത
𝜕𝛼ො 𝑛 𝑛 𝑛

13
Simple Linear Regression (5)
- OLS method consists in an optimization method that enables finding the values of α and β that minimize the Sum
of Squared Residuals, that is:
2
𝑛 𝑌෠ 𝑖
𝑚𝑖𝑛 𝑚𝑖𝑛
֜ 𝑆𝑆𝑅 = መ 𝑖
෍ 𝑌𝑖 − 𝛼ො + 𝛽𝑋 :
𝛼,
ො 𝛽መ 𝛼,
ො 𝛽መ
𝑖=1

Equating to 0 the derivative with respect to β gives the following result:

2
𝑌෠ 𝑖

𝜕 σ𝑛𝑖=1 𝑌𝑖 − 𝑌ത − 𝛽መ 𝑋ത + 𝛽𝑋
መ 𝑖
2
𝜕𝑆𝑆𝑅 ෝ
𝛼 𝜕 σ𝑛𝑖=1 𝑌𝑖 − 𝑌ത − 𝛽መ 𝑋𝑖 − 𝑋ത
֜ = = =0
𝜕𝛽መ 𝜕 𝛽መ 𝜕 𝛽መ
𝑛 𝑛 𝑛
𝜕𝑆𝑆𝑅
֜ = −2 ෍ 𝑋𝑖 − 𝑋ത 𝑌𝑖 − 𝑌ത − 𝛽መ 𝑋𝑖 − 𝑋ത = ෍ 𝑋𝑖 − 𝑋ത 𝑌𝑖 − 𝑌ത − 𝛽መ ෍ 𝑋𝑖 − 𝑋ത 2
=0
𝜕 𝛽መ
𝑖=1 𝑖=1 𝑖=1

σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 𝑌𝑖 − 𝑌ത 𝐶𝑜𝑣(𝑋, 𝑌)
መ
֜𝛽 = =
σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2 𝑉𝑎𝑟(𝑋)
14
 Simple Linear Regression (6)
- OLS method consists in an optimization method that enables finding the values of α and β that minimize the
Sum of Squared Residuals, that is:
2
𝑛 𝑌෠ 𝑖
𝑚𝑖𝑛 𝑚𝑖𝑛
֜ 𝑆𝑆𝑅 = መ 𝑖
෍ 𝑌𝑖 − 𝛼ො + 𝛽𝑋 :
𝛼,
ො 𝛽መ 𝛼,
ො 𝛽መ
𝑖=1

Summarizing, the OLS methods gives the following result:

σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 𝑌𝑖 − 𝑌ത 𝐶𝑜𝑣(𝑋, 𝑌)
መ
֜𝛽 = =
σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2 𝑉𝑎𝑟(𝑋)

𝐶𝑜𝑣 𝑋, 𝑌
֜ 𝛼ො = 𝑌ത − 𝛽መ 𝑋ത = 𝑌ത − 𝑋ത
𝑉𝑎𝑟 𝑋

𝛽መ represents the (estimated) slope coefficient and reflects the marginal change in Y after a unitary change in X.
𝛼ො represents the (estimated) intercept coefficient and reflects the value of Y when X = 0.
15
Simple Linear Regression (7)
- Let’s see a very simple example, considering the following 10 observations of the variables X and Y:
Xi Yi
4 9
5 24
2 7
3 13
7 15.5
9 23
1 11 𝛽መ = 1.90909
8 21
0 1.5
6 11

𝛼ො = 5.01

16
Simple Linear Regression (8)
 The previous results come from the fact that:
σ 10
𝑌2𝑖
𝑌ത = 𝑖=1
10
= 13.6,

σ10
𝑖=1 𝑋𝑖
𝑋ത = 10
= 4.5,

σ10 ത 2
𝑖=1 𝑋𝑖 −𝑋 82.5
𝑉𝑎𝑟 𝑋 = 10−1
= 9
= 9.1666666667

σ10 ത
𝑖=1 𝑋𝑖 − 𝑋 𝑌2𝑖 − 𝑌2 157.5
𝐶𝑜𝑣 𝑋, 𝑌 = = = 17.5
10 − 1 9

- Therefore,

σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 𝑌𝑖 − 𝑌ത 𝐶𝑜𝑣 𝑋, 𝑌 17.5

መ
֜𝛽 = = = = 1.90909
σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2 𝑉𝑎𝑟 𝑋 9.166ത

𝐶𝑜𝑣 𝑋,𝑌
֜ 𝛼ො = 𝑌ത − 𝑉𝑎𝑟 𝑋 𝑋ത = 13.6 − 1.90909 · 4.5 =5.01

17
Simple Linear Regression (9)
 How can we obtain the fitted line from the estimated coefficients?
መ 𝑖
֜ 𝑌෠𝑖 = 𝛼ො + 𝛽𝑋

- In practice: substituting the estimated values of the slope and intercept parameters, for each observed value of X we will
෠ which represents the regression line.
get the “predicted” or fitted value of Y (i.e. 𝑌),

 How can we obtain the regression’s residuals (i.e. the estimated error term)?

መ 𝑖
֜ 𝑌𝑖 = 𝛼 + 𝛽𝑋𝑖 + 𝜀𝑖 ֜ 𝜀𝑖Ƹ = 𝑌𝑖 − 𝑌෠𝑖 = 𝑌𝑖 − 𝛼ො + 𝛽𝑋 Little exercise:
i Xi Yi 𝑌෠𝑖 = 𝛼ො + 𝛽መ 𝑋𝑖 𝜀Ƹ𝑖 = 𝑌𝑖 − 𝑌෠𝑖 = 𝑌𝑖 − 𝛼ො + 𝛽መ 𝑋𝑖 εi
2
- Try to compute the residuals using “different”
1 4 9 12.64545455 -3.645454545 13.28934 values of α and β (e.g. 5.5 and 2).
2 5 24 14.55454545 9.445454545 89.21661
3 2 7 8.827272727 -1.827272727 3.338926 - Then obtain the new residuals and their squared
4 3 13 10.73636364 2.263636364 5.12405 values.
5 7 15.5 18.37272727 -2.872727273 8.252562 - Sum up all the squared residuals and compare
6 9 23 22.19090909 0.809090909 0.654628
7 6.918181818 4.081818182
with the results obtained using the coeficients
1 11 16.66124
8 8 21 20.28181818 0.718181818 0.515785 estimated using OLS.
9 0 1.5 5.009090909 -3.509090909 12.31372
10 6 11 16.46363636 -5.463636364 29.85132 18
Simple Linear Regression (10)
 How good is our simple regression model?
- From the estimated values of the intercept and slope parameters, the corresponding prediction of the
dependent variable and the observed values of Y, it is possible to construct a measure of the “goodness of
fit” of our regression (i.e. the degree of “adjustment” of the fitted line to the data).

Let’s define:
- 𝑆𝑢𝑚 𝑜𝑓 𝑇𝑜𝑡𝑎𝑙 𝑆𝑞𝑢𝑎𝑟𝑒𝑠 = 𝑆𝑆𝑇 = σ𝑛𝑖=1 𝑌𝑖 − 𝑌ത 2

2 2
- 𝑆𝑢𝑚 𝑜𝑓 𝑆𝑞𝑢𝑎𝑟𝑒𝑑 𝑅𝑒𝑠𝑖𝑑𝑢𝑎𝑙𝑠 = 𝑆𝑆𝑅 = σ𝑛𝑖=1 𝑌𝑖 − 𝑌෠𝑖 = σ𝑛𝑖=1 መ 𝑖
𝑌𝑖 − 𝛼ො + 𝛽𝑋 = σ𝑛𝑖=1 𝜀𝑖Ƹ2
2
- 𝑆𝑢𝑚 𝑜𝑓𝑅𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑆𝑞𝑢𝑎𝑟𝑒𝑠 = 𝑆𝑅𝑆 = σ𝑛𝑖=1 𝑌෠𝑖 − 𝑌ത = 𝛽መ 2 σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2

 The so-called R2 (or Coefficient of Determination) is the proportion of the total sample variability of the
dependent variable Y that we can explain with the regression (using X as explanatory variable), that is,
𝑛 2
𝑆𝑅𝑆 𝑆𝑆𝐸 σ𝑖=1 𝜀𝑖Ƹ
𝑅2 = =1− =1− 𝑛
𝑆𝑆𝑇 𝑆𝑆𝑇 σ𝑖=1 𝑌𝑖 − 𝑌ത 2

19
Simple Linear Regression (11)
 The so-called R2 (or Coefficient of Determination) is the proportion of the total sample variability of the dependent variable Y that we can
explain with the regression (using X as explanatory variable), that is,
𝑛 2
𝑆𝑅𝑆 𝑆𝑆𝑅 σ𝑖=1 𝜀𝑖Ƹ
𝑅2 = =1− =1− 𝑛
𝑆𝑆𝑇 𝑆𝑆𝑇 σ𝑖=1 𝑌𝑖 − 𝑌ത 2

- The R2 will be always comprised between 0 and 1, where 1 indicates the case of perfect adjustment in which the residuals are zero (i.e. Y is
a linear transformation of X) and 0 indicates the case of no statistical relationship between the two variables.

 In the previous example, the R2 would be equal to: We are explaining around 63%
of the variation of Y using X as
𝑆𝑅𝑆 𝑆𝑆𝑅 σ𝑛𝑖=1 𝜀𝑖Ƹ 2 179.22
֜𝑅 =2
=1− =1− 𝑛 =1− ≈ 0.627 explanatory variable
𝑆𝑆𝑇 𝑆𝑆𝑇 σ𝑖=1 𝑌𝑖 − 𝑌ത 2 479.9

- Notice also that the R2 is strictly related to the correlation coefficient, since:

2
2
𝑆𝑆𝑅 𝛽෠ 2 σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2 σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 𝑌𝑖 − 𝑌ത σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2 σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 𝑌𝑖 − 𝑌ത 2
1
֜𝑅 = 1− = = ∙ 𝑛 = ∙
𝑆𝑆𝑇 σ𝑛𝑖=1 𝑌𝑖 − 𝑌ത 2 σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2 σ𝑖=1 𝑌𝑖 − 𝑌ത 2 σ𝑛𝑖=1 𝑋𝑖 − 𝑋ത 2 σ𝑛𝑖=1 𝑌𝑖 − 𝑌ത 2

෡
𝛽
2
= 𝐶𝑜𝑟𝑟 𝑋, 𝑌

20
Simple Linear Regression (12)
Interpretation of the OLS coefficients from a linear regression.
- Simple linear model (both variables in levels): 𝑌𝑖 = 𝛼 + 𝛽𝑋𝑖 + 𝜀𝑖
መ change in Y after one unit of increase in X.
Interpretation: 𝛽=
- Log-linear models:
a) Y expressed in natural logs, X expressed in levels: ln(𝑌𝑖 ) = 𝛼 + 𝛽𝑋𝑖 + 𝜀𝑖
෡
𝛽
መ % change in Y after one unit of increase in X (semi-elasticity: %∆𝑌𝑖 =
 Interpretation: 𝛽= ∆𝑋𝑖 ).
100
b) Y expressed in levels, X expressed in natural logs: 𝑌𝑖 = 𝛼 + 𝛽ln(𝑋𝑖 ) + 𝜀𝑖
෡
𝛽
Interpretation: 𝛽=መ change in Y after a percentage increase in X (semi-elasticity: ∆𝑌𝑖 = %∆𝑋𝑖 ).
100

- Log-log model (both variables in natural logarithms): ln(𝑌𝑖 ) = 𝛼 + 𝛽 ln 𝑋𝑖 + 𝜀𝑖

 Interpretation: ෡𝛽= % change in Y after a percentage increase in X (elasticity: %∆𝑌𝑖 = 𝛽%∆𝑋
መ 𝑖 ).

 When both the dependent and the explanatory variables are expressed in logs, the beta coefficient represents
the (approximated value of the) elasticity:
መ 𝑖 − 1 ; 𝑖𝑓 ∆𝑋𝑖 ֜ %∆𝑌𝑖 = 100 · [exp 𝛽መ − 1]
- Exact computation:֜ %∆𝑌𝑖 = 100 · exp 𝛽∆𝑋
21
Simple Linear Regression (13)
 Advantages of log (or log-log) specification:
- Expressing the dependent variable in logarithms usually provides a better adjustment, since it reduced the
variability of the residuals (i.e. the log-transformation of the dependent variable reduces its variance).
- Related to the previous point, taking logs reduces the possible existence of heteroscedasticity (i.e. non-
constant variance of the error term), although it could not eliminate the problem (see later…), and problems
due to the asymmetry of variables (log-normality).
- Considering logs of the variables can mitigate the potential problems due to the existence of outliers
(extreme values of either Y or X or both).
- Taking logs facilitate the interpretation of the model, since the coefficients express percentage changes,
which do not depend on the units of measurement of the variables and are invariant to rescaling (i.e.
dividing earnings by 1000).
 Limitations of using logarithm specifications:
- It may create extreme values when the logged variable takes values that are close to zero.
- The logarithm is not defined when the value is equal to zero or for negative values ( usual solution,
transform the variable to x + 1 or x + 0.1 if x = 0).
- Notice that the R-squared for models with and without logs are not strictly comparable, since the
dependent variable (which appears in the denominator of the R-squared) is defined in a different way.
22