Chapter 5 Maximum Likelihood Estimator

The OLS estimator does not need the assumption about the exact distribution of the error-term
e.g. the GAUSS-MARKOV Theorem does not require the normality assumption. If we assume the
normality assumption of the error terms. We put stricter condition for the behavior of the random
variables.
The assumption, regarding the exact distribution function of the error terms, open up a new way
of estimation called “Maximum Likelihood Estimator”, MLE. MLE can be applied not only to the
Linear Regression Model but also Nonlinear models and many other applications.
Since MLE requires assumption of the exact distribution of the error terms, MLE falls into a
class of estimator called “Full Parametric Estimator”. Theoretically, MLE is considered to be the best
estimation method if the assumption of the distribution is true. The distribution does not need to be
normal distribution but normal distribution is the most used assumption as it is the mostly to happen in
real world if we have large enough sample size according to the Centrel Limit Theorem, CLT.

Normal Distribution
Recalled the Probability Density Function of normal distribution, pdf of x:
2

pdf ( x)= f ( x )=
1
√2 π σ 2
exp
2 σ[
−1 x−μ
]
Distribution of Y i
Recalled Population Regression Function:
Y 1=β0 +β1 X 11 +β2 X 21+β3 X 31 +...+β K X K1+ϵ 1
Y 2 =β0 +β1 X 12 +β 2 X 22 +β3 X 32+...+βK X K2 +ϵ 2
Y 3=β0 +β1 X 13 +β 2 X 23 +β3 X 33+...+β K X K3 +ϵ3
⋮
Y N =β0+β1 X 1N +β 2 X 2N +β3 X 3N +...+β K X KN +ϵN
or Y i =β0 +β1 X 1i +β2 X 2i+β3 X 3i +...+β K X Ki +ϵ i
also, E (Y i∣X i )= X i β
and, Y i = E(Y i∣X i )+ϵi
Assuming the NORMAL DISTRIBUTION OF THE ERROR TERM, ϵi , it is followed that
Yi is also normally distributed.
Y i ∼ N ( X i β , σ 2)
 2

Conditional pdf of Y i = f (Y i∣X β ,σ )=

1
√2 π σ 2
exp−
1 Y i −E (Y i∣X i )
2

2 σ [ ]
2
2
f (Y i∣X β , σ )=
1
√2π σ 2
exp−
1
2
Y i− X i β
σ [ ]
2

or f (Y i∣X β ,σ 2 )=
1
√2 π σ 2
exp−
2
[
1 Y i −β0 −β1 X 1i −β2 X 2i −β3 X 3i−...−β K X Ki
σ ]
Joint PDF of all observations ( Y 1, Y 2, … ,Y N )
Recalled that each error term is independent and identically distributed, i.i.d., as such, the joint PDF
can be written as a product of n individual density functions.
f (Y 1, Y 2, …, Y N∣X β , σ 2 )= f (Y ∣X β , σ 2 )= f (Y 1∣X β , σ 2) f (Y 2∣X β , σ 2)… f (Y N∣X β , σ 2 )
2

as 2
f (Y i∣X β , σ )=
1
√2 π σ 2
exp−
1 Y i− X i β
2 σ [ ]
2 2 2

f (Y ∣X β ,σ 2 )=
1
√2 π σ 2
exp−
2
[
1 Y 1− X 1 β
σ ]
1
√2 π σ 2
exp−
1 Y 2− X 2 β
2 σ …
1
[
√2 π σ 2
exp−
1 Y N− X N β
2 σ ] [ ]
2
f (Y X β ,σ )=
∣ 1
( √2 π σ 2)
N
1
exp− ∑
2
Y i− X i β 2
σ [ ( )]

Likelihood Function

[∑ (Y −σX β ) ] , if Y are known but the

2
1 1
From f (Y ∣X β ,σ 2 )= exp− i i

( √2 π σ 2 )
N
2
parameters, β , σ 2 , are unknown. The joint PDF will be called “Likelihood Function”, LF.

[∑ (Y −σX β ) ]
2
1 1
LF (β ,σ 2 )= exp− i i

( √2 π σ 2 )
N
2

Take natural log on both sides, we will get “Log Likelihood Function”, LLF or ln LF.
Y −X β 2
N
2
N 1
ln σ 2 − ln 2 π− ∑ i σ i
LLF=−
2 2
( )
2
Y −β −β X −β X −β X −...−β K X Ki
N
2
N
2
1
LLF=− ln σ 2 − ln 2 π− ∑ i 0 1 1i 2 σ2i 3 3i
2
( )
Maximum Likelihood Estimators
Conceptually, MLE are the estimators that acquired by selecting the values of the unknown
parameters, β , σ 2 , that maximize the likelihood function.
max LLF
β̂ , σ̂ 2
2
N N 1
(
Y −β̂ −β̂ X −β̂ X −β̂ X −...−β̂K X Ki
max − ln σ̂ 2− ln 2 π− ∑ i 0 1 1i 2 2i 3 3i
β̂ , σ̂
2 2 2 2 σ̂ )
By performing an uncontrainted optimization with regarded to β̂ , σ̂ 2 , we will get the MLE of
the Linear Regression Model.

If have time, in class, show the MLE of 2 variables model.

Homework:
- Consider the 3 variables model, Y i =β0 +β1 X 1i +β2 X 2i+ϵ i , using Method of Maximum Likelihood
to find the ML Estimators of β0, β1, β2 , and, σ 2 .

HINT: Except the variance, all other estimators will be the same as OLS estimator since OLS is BLUE.
Also, don't forget to check the second order conditions.