Outline

Basic concepts
Estimators and their properties
Estimation methods

Estimation methods and their properties

Statistical Methods

Magdalena Smyk/Lucas van der Velde

Warsaw School of Economics

October 20, 2017

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts

Estimators and their properties

Estimation methods

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Population - any well-defined group of subjects, which could be
individuls, firms, cities, or many other possibilities.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Population - any well-defined group of subjects, which could be
individuls, firms, cities, or many other possibilities.

Statistical sample - subset of the population chosen to represent

the population in a statistical analysis; denoted as (X1 ,X2 , ... Xn ).

Random sample - randomly chosen from the population sample

of individuals.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Population - any well-defined group of subjects, which could be
individuls, firms, cities, or many other possibilities.

Statistical sample - subset of the population chosen to represent

the population in a statistical analysis; denoted as (X1 ,X2 , ... Xn ).

Random sample - randomly chosen from the population sample

of individuals.
In the case of random sampling, the following techniques can be used:
I independent sampling (draw with replacement) - after each draw
the unit returns to the population.
I dependent sampling (draw without replacement) - after each draw
the unit does not return to the population (no loger participate in
the drawing).
Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Statistical Inference theory is based on the assumption of having a
simple random sample.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Statistical Inference theory is based on the assumption of having a
simple random sample.
Simple random sample
I sample collected from random independent individual
sampling, so sampling from the population individual units
indirectly, with the same probablity that each unit will be
drawn;
I group of independent random variables with an identical
distribution of probablity of a characteristic as in the
population.
Random sample space - set of all possible random samples
denoted as (X1 ,X2 , ... Xn ).

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Statistical Inference theory is based on the assumption of having a
simple random sample.
Simple random sample
I sample collected from random independent individual
sampling, so sampling from the population individual units
indirectly, with the same probablity that each unit will be
drawn;
I group of independent random variables with an identical
distribution of probablity of a characteristic as in the
population.
Random sample space - set of all possible random samples
denoted as (X1 ,X2 , ... Xn ).
Sample credibility - probability of receiving a specific sample,
P(X1 = x1 , X2 = x2 , ..., Xn = xn ).
Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Mathematical statistics deals with partial research.
Statistical inference - generalization of the results of partial
research into the population, i.e. the conclusion of the population
on the basis of information from the sample.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Mathematical statistics deals with partial research.
Statistical inference - generalization of the results of partial
research into the population, i.e. the conclusion of the population
on the basis of information from the sample.
Parameters - characteristics that define population.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Basic concepts
Mathematical statistics deals with partial research.
Statistical inference - generalization of the results of partial
research into the population, i.e. the conclusion of the population
on the basis of information from the sample.
Parameters - characteristics that define population.
Statistical inference:
I Estimation of unknown parameters or functions, that
characterize distribution of the selected feature in the
population
I Testing statistical hypotheses, e.g. hypotheses considering
value of the parameter of the distribution of the feature in the
population
Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimators

Let the distribution of a random variable X from the population be

described by F (θ), where θ is the distribution parameter.
The unknown value of the parameter θ is estimated from an
n-element simple random sample (X1 , X2 , ..., Xn ).
A sample statistic - a random variable Zn that is a function of
random variables X1 , X2 , ..., Xn (random sample).
Estimator Tn of the population parameter θ is a sample statistic
Tn = t(X1 , X2 , ...Xn ) that estimates the value of the unknown
parameter θ.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimators

Estimate of the parameter θ is a single value tn = t(x1 , x2 , ..., xn )

calculated based on the specific sample (x1 , x2 , ..., xn ).
Estimate is a random variable.

I In the point estimation, this estimate is assumed to be an

estimator of the unknown parameter.
I Since estimates are recovered from random samples, estimates
might be burdened with error.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimators

Estimation error of the parameter θ is the difference between the

estimate value and the parameter value: Tn − θ = d.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimators

Estimation error of the parameter θ is the difference between the

estimate value and the parameter value: Tn − θ = d.
A measure of the estimation error is the mean square error:
MSE = E [(Tn − θ)]2

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimators

Estimation error of the parameter θ is the difference between the

estimate value and the parameter value: Tn − θ = d.
A measure of the estimation error is the mean square error:
MSE = E [(Tn − θ)]2

I In the formula, substituting θ with an expected value E [(Tn )]

results with a variance var (Tn ).

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimators

Estimation error of the parameter θ is the difference between the

estimate value and the parameter value: Tn − θ = d.
A measure of the estimation error is the mean square error:
MSE = E [(Tn − θ)]2

I In the formula, substituting θ with an expected value E [(Tn )]

results with a variance var (Tn ).
I The standard (average) estimation error of the parameter θ is
equal to standard deviation S(Tn ).

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimators

Estimation error of the parameter θ is the difference between the

estimate value and the parameter value: Tn − θ = d.
A measure of the estimation error is the mean square error:
MSE = E [(Tn − θ)]2

I In the formula, substituting θ with an expected value E [(Tn )]

results with a variance var (Tn ).
I The standard (average) estimation error of the parameter θ is
equal to standard deviation S(Tn ).
I Relative estimation error is S(Tn )/θ .

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Unbiasedness of an estimator

An estimator, Tn of a parameter θ, is an unbiased estimator if

E (Tn ) = θ
for all possible values of θ.
If Tn is a biased estimator of θ, its bias is defined as

Bias(Tn ) = E (Tn ) − θ

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Unbiasedness of an estimator

Unbiasedness of an estimator means that, with multiple sampling,

the average value of the estimates is equal to the estimated value
of the parameter.
If the estimator is unbiased then the evaluation of the parameter
are not burdened with the systematic error.
Estimator Tn of the parameter θ is asymptotically unbiased if
limn→∞ b(Tn ) = 0

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Unbiasedness of an estimator and MSE

In general, the MSE has two elements: bias and the variance
of the estimator

MSE (Tn ) = varθ (Tn ) + bias 2 (Tn )

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Unbiasedness of an estimator

Example 1
Prove that arithmetic mean X̄ = n1 ni=1 Xi is an unbiased
P
estimator of expected value of X in the population.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Unbiasedness of an estimator

Example 1
Prove that arithmetic mean X̄ = n1 ni=1 Xi is an unbiased
P
estimator of expected value of X in the population.

n n
1X 1X
Bias(Tn ) = E (X̄ )−E (X ) = E ( Xi )−E (X ) = E (Xi )−E (X ) =
n n
i=1 i=1

1
= nE (X ) − E (X ) = E (X ) − E (X ) = 0
n

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Unbiasedness of an estimator

Example 2
Show that sample variance:
Pn
I S2 = 1 2
n−1 i=1 (Xi − X̄ ) is an unbiased estimator of Var (X )
Pn
I S̃ 2 = 1 2
n i=1 (Xi − X̄ ) is a biased estimator of Var (X )

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Unbiasedness of an estimator

Example 2
Show that sample variance:
Pn
I S2 = 1 2
n−1 i=1 (Xi − X̄ ) is an unbiased estimator of Var (X )
Pn
I S̃ 2 = 1 2
n i=1 (Xi − X̄ ) is a biased estimator of Var (X )
Hint:
Var (X )
E (X̄ 2 ) = + [E (X )2 ]
n
The bias is equal to
b(S̃ 2 ) = n−1 1
n Var (X ) − Var (X ) = − n Var (X )

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Consistency

Let Tn be an estimator of θ based on a sample Y1 , Y2 , ..., Yn of

size n. Then Tn is a consistent estimator of θ if for every ε > 0,

lim P(|Tn − θ| > ε) = 0.

n→∞

It the estimator is consistent, as the sample size n gets large, the

probability that the value of the estimator will be close to the value
of the true parameter increases.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Consistency and unbiasedness

PROPERTIES
I If the estimator Tn of the parameter θ is consistent, it is also
asymptotically unbiased.
I If the estimator Tn of the parameter θ is unbiased (or
asymptotically unbiased) and limn→∞ Var (Tn ) = 0, then Tn is
an consistent estimator.
Example 3
Show that the mean of X is a consistent esitmator of E (X ) in
population.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Efficiency

For parameter θ there may exist more than one unbiased estimator.
If Tn1 and Tn2 are two unbiased estimators of parameter θ and

Var (Tn1 ) < Var (Tn2 )

then estimator Tn1 is more efficient than estimator Tn2 .

From the total collection of possible unbiased estimators, the one
with the lowest variance (Tn∗ ) will be called the most efficient
estimator.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Efficiency

Efficiency of the estimator Tni of the parameter θ is a ratio of the

variance of the estimator Tn∗ in the population (most efficient) and
the variance of a given estimator Tni :

Var (Tn∗ )
e(Tni ) =
Var (Tni )

I Efficiency of the most efficient estimator is equal to 1.

I Cramer’s-Rao lower bound is used to determine the variance
of the most efficient estimator.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Efficiency - Cramer-Rao bound

Theorem
If Tn is an unbiased estimator of the parameter θ, then
1
Var (Tn ) ≥ ∂
nE [( ∂θ ln f (x; θ))2 |θ]

where f (x; θ) is a density or probability mass function of the

population.
Estimator Tn is asymptotically the most efficient if

lim e(Tn ) = 1
n→∞

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Examples of the estimators

Unknown Estimator Properties Family of

parameter distributions
consistent, any
1 Pn
E(X) X̄ = n i=1 Xi unbiased distribution

consistent, any
E(X) Median asymptotically distribution
unbiased
Variance, consistent, any
1 Pn
when E(X) s̃ 2 = n i=1 (Xi − X̄ )2 asymptotically distribution
unknown unbiased
Variance, consistent, any
1 Pn
when E(X) s2 = n−1 i=1 (Xi − X̄ )2 unbiased distribution
unknown
consistent, Bernoulli
k
Fraction θ= n
unbiased, distribution
efficient

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Estimation methods

I Maximum Likelihood Method - defined by R. A. Fisher in

1920’s; it is one of the most popular method of estimation.
I Method of Moments - the simplest and the oldest method;
estimators based on the method of moments are not very
efficient.
I Least Squares Method - defined by A.M. Lagendre, K. F.
Gauss and A. A. Markow; it’s most commonly used to
estimate parameters in the linear model regressions. It is
sensitive to outliers.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Maximum Likelihood Estimation (MLE)

Let (X1 , X2 , ...Xn ) be a random sample from a population

distribution f (x; θ). Because of random sampling, the joint
distribution of (X1 , X2 , ...Xn ) is simply the product of the densities
f (x1 ; θ), f (xn ; θ), ..., f (xn ; θ).
The likelihood function is:

n
Y
L(θ; X1 , ..., Xn ) = f (X1 ; θ)f (X2 ; θ)...f (Xn ; θ) = f (Xi ; θ)
i=1

which is a random variable because it is the outcome of a random

sample (X1 , X2 , ...Xn ).

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Maximum Likelihood Estimation (MLE)

In the discrete case, instead of density function, probability mass

function is used: P(X1 = x1 , X2 = x2 , ..., Xn = xn ).
The likelihood function is then:
n
Y
L(θ; X1 , ..., Xn ) = P(Xi ; θ),
i=1

The maximum likelihood principle says that, out of all the possible
values for θ, the value that maximizes the likelihood of the
observed data should be chosen.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Properties of the MLE estimators

Maximum Likelihood estimators:

I are consistent
I have asymptotical normal distribution with the expected value
1
θ and variance ∂lnf (x;θ) 2
nE ( ∂θ
)
I are at least asymptotically unbiased and asymptotically the
most efficient
If the most efficient estimator of the parameter θ exists, it is the
MLE estimator.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Method of Moments (MM)

Let (X1 , X2 , ..., Xn ) be a random sample.

The k-th sample moments:
I Raw
n
1X k
Mk = Xi
n
i=1
I Central
n
1X
Mk = (Xi − M1 )k
n
i=1

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Method of Moments (MM)

Let (X1 , X2 , ..., Xn ) be a random sample.
The k-th distribution moments:
I Raw: mk = E (X k )

(P
x k pi if random variable is discrete
mk = R ∞i i k
−∞ x f (x)dx if random variable is continuous

I Central: µk = E [(X − E (X ))k ]

(P
(xi − E (X ))k pi if random variable is discrete
µk = R ∞i k
−∞ (x − E (X )) f (x)dx if random variable is continuous

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Method of Moments (MM)

In the Method of Moments we assume that a sample moment is

equal to the moment from a distribution.
We use as many moments as unknown parameters we have.

By solving the equation systems we obtain estimates of these

parameters.

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Task

There is a simple random sample with n − elements drawn from

the population with the exponential distribution with the
parameter λ > 0.
(
1
exp(− λx ) if x ≥ 0
f (x) = λ
0 if x < 0
Expected value: E (X ) = λ
Variance: Var (X ) = λ2

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Obtain an estimator of parameter λ using:

I a. Method of Moments
I b. Maximum Likelihood Estimation
Is the estimator ubiased?
Is the estimator the most efficient?

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Task - method of moments

m1 = E (X ) = λ

n
1X
M1 = Xi = X̄
n
i=1

λ̂ = X̄

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Task - MLE

1 x1 1 xn 1 x1 xn
L(xi , λ) = exp(− )... exp(− ) = ( )n exp(− − ... − )
λ λ λ λ λ λ λ
n
1 1 1X
L(xi , λ) = ( )n exp(− (x1 + ... + xn )) = λ−n exp(− xi )
λ λ λ
i=1
n n
1 X 1 X
ln(L) = ln[λ−n exp(− xi )] = −nlnλ − xi
λ λ
i=1 i=1

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Task - MLE

n
∂ln(L) n 1 X
=− + 2 xi = 0
∂λ λ λ
i=1
n
n 1 X
= xi
λ̂ λ̂2 i=1
Pn
i=1 xi
λ̂ = = X̄
n

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Task

Is the estimator ubiased?

n n n
1X 1 X 1X
E (λ̂) = E (X̄ ) = E ( Xi ) = E ( Xi ) = E (Xi )
n n n
i=1 i=1 i=1

1
E (λ̂) = nE (X ) = λ
n
YES!

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Task

Is the estimator consistent?

n n
2 2 1X 1 2 X
2
lim S (λ̂) = lim S (X̄ ) = lim S ( Xi ) = lim S ( Xi )
n→∞ n→∞ n→∞ n n→∞ n
i=1 i=1

n
1 X 2 1 λ2
lim S 2 (λ̂) = lim S (Xi ) = lim nS 2
(X ) = lim =0
n→∞ n→∞ n2 n→∞ n2 n→∞ n
i=1

YES!

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties
 Outline
Basic concepts
Estimators and their properties
Estimation methods

Task

Is the estimator the most efficient?

1 x x x
ln(f (x; λ)) = ln( exp(− )) = lnλ−1 − = −lnλ −
λ λ λ λ
∂ln(f (x; λ)) 1 x
=− + 2
∂λ λ λ
x 1 1 1 1 1
E ( 2 − )2 = 4 E (x − λ)2 = 4 S 2 (x) = 4 λ2 = 2
λ λ λ λ λ λ
1 λ2
S 2 (Tn∗ ) = 1 =
n λ2 n
λ2
S 2 (X̄ ) =
n
YES!

Magdalena Smyk/Lucas van der Velde Estimation methods and their properties