Probability & Statistics II

Chapter4: Estimation
Presented by: Dr. BOUREDJA Sara

[email protected]
1- Introduction

The concept of estimation was introduced in inferential statistics. Its

essential mission is to obtain an approximate value for one or more
statistical parameters, for a 𝑅𝑉 in a population, from data observed in a
sample.

[email protected]
2-Estimator

Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 a 𝑛 −sample from 𝑅𝑉. 𝑋 distributed according to the same

probability distribution 𝑓(𝑥, 𝜃), where 𝜃 is a parameter associated with this
probability distribution and its value is unknown.
An estimator of 𝜃 is a random variable 𝑇 as a function of 𝑋𝑖 .
𝑇 = 𝑓(𝑋1 , 𝑋2 , … , 𝑋𝑛 ).
When the sample has actually been achieved, the estimator takes on a
numerical value, called the estimator of the parameter 𝜃:
𝜃෠ = 𝑓(𝑥1 , 𝑥2 , … , 𝑥𝑛 )

Statistical model= Probability distribution

[email protected]
3-Estimator calculation methods

3-1-The method of moments

Let 𝜇𝑟 be the theoretical moment of order 𝑟 of the distribution, i.e.:
Continuous case 𝜇𝑟 = 𝐸 𝑋 𝑟 = ‫𝑥 𝑓 𝑟 𝑥 ׬‬, 𝜃 𝑑𝑥,
Discrete case 𝜇𝑟 = 𝐸(𝑋 𝑟 ) = σ𝑛𝑖=1 𝑥𝑖𝑟 𝑃 𝑥, 𝜃
1
Let 𝑚𝑟 the empirical moment of order 𝑟 of the sample, i.e.: 𝑚𝑟 = σ𝑛𝑖=1 𝑋𝑖𝑟
𝑛

෡ is obtained by solving the following equation:

The estimator 𝜽
1
Continuous case ‫𝑥 𝑓 𝑟 𝑥 ׬‬, 𝜃 𝑑𝑥 = 𝑛 σ𝑛𝑖=1 𝑋𝑖𝑟
1 𝑛
Discrete case σ𝑛𝑖=1 𝑥𝑖𝑟 𝑃 𝑥, 𝜃 = σ𝑖=1 𝑋𝑖𝑟
𝑛

[email protected]
Example 1
1 − 𝑥−𝜃
ቐ𝜃 𝑒 , 𝑥≥0
𝜃
Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 a sample from the RV 𝑋, with density: 𝑓 𝑥, 𝜃 =
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
-Determine an estimator of 𝜃 by the method of moments.
Solution
1
The empirical moment: 𝑚1 = σ𝑛𝑖=1 𝑋𝑖 = 𝑋ത
𝑛
+∞
The theoretical moment: 𝜇1 = 𝐸 𝑋 = ‫׬‬−∞ 𝑥𝑓 𝑥, 𝜃 𝑑𝑥
+∞ 1 − 𝑥−𝜃
= ‫׬‬0 𝑥 𝜃 𝑒 𝜃 𝑑𝑥
𝑥−𝜃 +∞ 𝑥−𝜃
− 𝜃 +∞ − 𝜃
= − 𝑥𝑒 0 − ‫׬‬0 −𝑒 𝑑𝑥
𝑥−𝜃 +∞
−
= 𝜃 + −𝜃𝑒 𝜃 0
=𝜃+𝜃
= 2𝜃
So, the estimator 𝜃መ is obtained by solving the following equation: 2𝜃 = 𝑋ത
𝑋ഥ
𝜃መ =
2
[email protected]
 3-2-Maximum Likelihood Method (MLE)

We call the likelihood function of 𝜃 a realization 𝑥1 , 𝑥2 , … , 𝑥𝑛 of a sample 𝑋1 , 𝑋2 , … , 𝑋𝑛 the

function of 𝜃 :
Continuous case 𝐿 𝑥1 , 𝑥2 , … , 𝑥𝑛 , 𝜃 = ς𝑛𝑖=1 𝑓(𝑥𝑖 , 𝜃)
Discrete case 𝐿 𝑥1 , 𝑥2 , … , 𝑥𝑛 , 𝜃 = ς𝑛𝑖=1 𝑃(𝑋 = 𝑥𝑖 )

The method of estimating 𝜃 by the value that maximises the likelihood 𝐿, is called the
maximum likelihood method (𝑀𝐿𝐸).
𝜕𝑙𝑛 𝐿(𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜃)
=0
𝜕𝜃
𝜕 2 ln𝐿(𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜃)
෠ <0
(𝜃)
2
𝜕 𝜃

[email protected]
Example2
We want to estimate the parameter 𝜆 of a Poisson distribution from a sample.
𝑥
−𝜆 𝜆
We have: 𝑃 𝑋 = 𝑥 = 𝑒 𝑥!
The likelihood function
𝑛
: 𝑛 𝑛 𝑛
−𝜆
𝜆 𝑥𝑖 −𝜆𝑛
𝜆 𝑥𝑖 −𝜆𝑛
𝜆σ𝑖=1 𝑥𝑖
𝐿(𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜆) = ෑ 𝑃 𝑋 = 𝑥𝑖 = ෑ 𝑒 =𝑒 ෑ =𝑒
𝑥𝑖 ! 𝑥𝑖 ! ς𝑛𝑖=1 𝑥𝑖 !
𝑖=1 𝑖=1 𝑖=1

𝑛
σ𝑖=1 𝑥𝑖 𝑛 𝑛 𝑛
𝜆 𝑛
𝑙𝑛 𝐿(𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜆) = 𝑙𝑛 𝑒 −𝜆𝑛 𝑛 = 𝑙𝑛𝑒 −𝜆𝑛 + 𝑙𝑛𝜆σ𝑖=1 𝑥𝑖 + 𝑙𝑛 ෑ 𝑥𝑖 ! = −𝜆𝑛 + 𝑙𝑛 𝜆 ෍ 𝑥𝑖 − ෍ 𝑙𝑛 𝑥𝑖 !
ς𝑖=1 𝑥𝑖 !
𝑖=1 𝑖=1 𝑖=1
𝜕𝑙𝑛 𝐿(𝑥1 ,𝑥2 ,…,𝑥𝑛 ;𝜆) σ𝑛
𝑖=1 𝑥𝑖
The first derivative : 𝜕𝜆
= −𝑛 +
𝜆
σ𝑛
𝑖=1 𝑥𝑖
Annuls for 𝜆መ = 𝑛
𝜕2 𝑙𝑛 𝐿(𝑥1 ,𝑥2 ,…,𝑥𝑛 ;𝜆) σ𝑛
𝑖=1 𝑥𝑖
The second derivative : 𝜕2 𝜆
=−
𝜆2
Is negative.
σ𝑛 𝑖=1 𝑥𝑖
Thus, the estimator given 𝜆መ = 𝑛
= 𝑋ത leads to a maximum likelihood estimator equal to
𝜆መ 𝑀𝐿𝐸 = 𝑥ҧ

[email protected]
 4- Main qualities of an estimator
❖Convergent estimator: 𝜃መ is said to be convergent if: lim 𝑅 𝜃;
መ 𝜃 =0
𝑛→∞
መ 𝜃 = 𝐸 𝑙 𝜃,
Risk of an estimator: 𝑅 𝜃; መ 𝜃
መ 𝜃) = |𝜃መ − 𝜃|2
Quadratic loss: 𝑙(𝜃,
Note1 in the case of quadratic loss, the risk will then be:
𝑅 𝜃;መ 𝜃 = 𝐸 |𝜃,መ 𝜃|2

❖Unbiased estimator: 𝜃መ is unbiased if 𝐸 𝜃መ = 𝜃.

መ 𝜃 = 𝑉 𝜃መ
Note2 If the estimator is unbiased, then: 𝑅 𝜃;

❖ Asymptotically unbiased estimator: 𝜃መ is an asymptotically unbiased estimator of 𝜃 if: lim 𝐸( 𝜃)

መ =𝜃
𝑛→∞
❖Best estimator:
Let 𝜃መ1 and 𝜃መ2 be two estimators of 𝜃, we say that 𝜃መ1 is a better estimator than 𝜃መ2 if: 𝑅 𝜃መ1 ; 𝜃 < 𝑅 𝜃መ2 ; 𝜃 , ∀ 𝜃.
❖Admissible estimator: An estimator is said to be admissible if there is no better estimator than it.

[email protected]
Example 3: Let 𝑋 be a random variable with uniform distribution on [0, 𝜃] et 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample from 𝑋.
- Find the 𝑀𝐿𝐸 de 𝜃.
Solution:
1
10≤𝑥≤𝜃 ,
The density of 𝑋 is written: 𝑓 𝑥 = ቐ𝜃
0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
1 1 1
The likelihood is therefore: 𝐿 𝑥 ; 𝜃 = 10≤𝑥1 ≤𝜃 × 10≤𝑥2 ≤𝜃 × ⋯× 10≤𝑥𝑛 ≤𝜃
𝜃 𝜃 𝜃
1
= 1 1 .
𝜃𝑛 {0≤min 𝑥𝑖 } {max 𝑥𝑖 ≤𝜃}
Since the domain of 𝑓 𝑥 ; 𝜃 depends on 𝜃, the maximum likelihood of 𝜃 cannot be obtained by deriving with
respect to 𝜃, but by choosing 𝜃 as large as possible. Since 𝜃 ≥ 𝑥, in particular 𝜃 ≥ max 𝑥𝑖 , the likelihood
𝐿 𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜃 is then written:
1
, 𝑓𝑜𝑟 𝜃 ≥ max 𝑥𝑖 ;
𝐿 𝑥; 𝜃 = 𝐿 𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜃 = ቐ𝜃 𝑛
0, 𝑓𝑜𝑟 0 ≤ 𝜃 < max 𝑥𝑖 .

1
𝐿 𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜃 cannot be larger than 𝜃𝑛 . Therefore, the MLE of 𝜃
is 𝜃෠ = max 𝑥𝑖 .
[email protected] Figure1.Graph of 𝐿 𝑥; 𝜃 as a function of 𝜃
5-Sufficient statistics (exhaustive)

Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be an n-sample, the likelihood of the sample is given by the function

𝐿 𝑥; 𝜃 .
Let 𝑇 be a function statistic of 𝑋1 , 𝑋2 , … , 𝑋𝑛 with distribution 𝑔(𝑡; 𝜃).
𝑇 Is said to be exhaustive if the likelihood 𝐿 𝑥; 𝜃 is written in the following form:
𝐿 𝑥; 𝜃 = 𝑔 𝑡; 𝜃 ℎ 𝑥
This is the Neyman-Fisher factorisation principle.
Example 4
Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be an n-sample from a RV 𝑋 with the Poisson distribution 𝑃 𝜃 defined
𝜃𝑥
by: 𝑃 𝑋 = 𝑥 = 𝑒 −𝜃
𝑥!
- Is the statistic 𝑇 = σ𝑛𝑖=1 𝑋𝑖 sufficient for the parameter 𝜃?

[email protected]
Solution
First we determine the distribution of 𝑇:
𝑥
−𝜃 𝜃
𝑋 ↝ 𝑃(𝜃) ⟶ 𝑃 𝑋 = 𝑥 = 𝑒
𝑥!
Since the Poisson distribution is stable, then 𝑇 = 𝑡σ𝑛𝑖=1 𝑋𝑖 ↝ 𝑃(𝑛𝜃). Then the distribution of the
(𝑛𝜃)
statistic 𝑇 will be: 𝑔 𝑡, 𝜃 = 𝑃 𝑇 = 𝑡 = 𝑒 −𝑛𝜃 , avec 𝑡 = σ𝑛𝑖=1 𝑥𝑖 .
𝑛 𝑡!

𝐿(𝑥1 , 𝑥2 , … , 𝑥𝑛 ; 𝜃) = ෑ 𝑃 𝑥𝑖 , 𝜃
𝑖=1
𝑛 −𝜃 𝜃 𝑥𝑖
= ς𝑖=1 𝑒
𝑥𝑖 !
σ𝑛
−𝜃𝑛 𝜃 𝑖=1 𝑥𝑖
= 𝑒 ς𝑛
𝑖=1 𝑥𝑖!
−𝜃𝑛 𝜃𝑡
= 𝑒 ς𝑛
𝑖=1 𝑥𝑖!
𝑒 −𝜃𝑛 (𝑛𝜃)𝑡 𝑡!
= ×
𝑡! 𝑛𝑡 (ς𝑛
𝑖=1 𝑥𝑖 !)
= 𝑔 𝑡; 𝜃 ℎ 𝑥
According to the factorisation principle, 𝑇 is exhaustive for 𝜃.
[email protected]
 6- Fisher information

The amount of Fisher information 𝐼𝑛 𝜃 , provided by a sample on the parameter 𝜃 is

expressed as follows:
2
𝜕𝑙𝑛 𝐿 𝑥; 𝜃
𝐼𝑛 𝜃 = 𝐼𝑥1 ,…,𝑥𝑛 𝜃 = 𝐸
𝜕𝜃

Theorem 3
If the domain of definition of the random variable 𝑋 does not depend on 𝜃 then:
𝜕 2 𝑙𝑛 𝐿 𝑥; 𝜃
𝐼𝑛 𝜃 = −𝐸
𝜕𝜃 2

[email protected]
Example 5
let 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample from a RV 𝑋 with the normal distribution 𝑁(𝜇, 𝜎 2 ).
-Find the Fisher information provided by the sample on the parameter 𝜇?
Solution :
𝜕2 ln 𝐿 𝑥;𝜇,𝜎 2
The domain of 𝑋 does not depend on 𝜇, so: 𝐼𝑛 𝜇 = −𝐸
𝜕𝜇 2
1 1
2 − 2 σ𝑛𝑖=1(𝑥𝑖 −𝜇)2
𝐿 𝑥; 𝜇, 𝜎 = ( )𝑒 2𝜎
𝜎 2𝜋 𝑛
1
𝑙𝑛 𝐿 𝑥; 𝜇, 𝜎 = −𝑛 𝑙𝑛 𝜎 2𝜋 − 2 ෍(𝑥𝑖2 − 2𝑥𝑖 𝜇 + 𝜇2 )
2
2𝜎
𝑛 𝑖=1
𝜕𝑙𝑛 𝐿 𝑥; 𝜇, 𝜎 2 −1
= 2 ෍(−2𝑥𝑖 + 2𝜇)
𝜕𝜇 2𝜎
2 2𝑖=1
𝜕 𝑙𝑛 𝐿 𝑥; 𝜇, 𝜎 −𝑛
2
= 2
𝜕𝜇 𝜎
−𝑛 𝑛
𝐼𝑛 (𝜇) = −𝐸 2 = 2
𝜎 𝜎
[email protected]
Property 1

If the domain of defining the random variable 𝑋 does not depend on 𝜃 then:
𝐼𝑛 𝜃 = 𝑛 𝐼𝑋 (𝜃)
Example 6
let 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample from a RV 𝑋 with the normal distribution 𝑁(𝜇, 𝜎 2 ).
-Find the Fisher information provided by random variable 𝑋 on the parameter 𝜇?
-Find the Fisher information provided by the sample on the parameter 𝜇?
Solution
𝜕2 𝑙𝑛 𝑓 𝑥;𝜇,𝜎2
𝐼𝑋 𝜇 = −𝐸
𝜕𝜇2
(𝑥−𝜇)2
𝑙𝑛 𝑓 𝑥; 𝜇, 𝜎 2 = − 𝑙𝑛 𝜎 2𝜋 −
2𝜎2
𝜕𝑙𝑛 𝑓 𝑥;𝜇,𝜎2 −1 𝑥−𝜇
= 2 −2𝑥𝑖 + 2𝜇 = 2
𝜕𝜇 2𝜎 𝜎
2
𝜕 𝑙𝑛 𝑓 𝑥;𝜇,𝜎 2 −1
=
𝜕𝜇2 𝜎2
−1 1
𝐼𝑋 (𝜇) = −𝐸 2 = 2
𝜎 𝜎
1 𝑛
The Fisher information provided by the sample is then: 𝐼𝑛 𝜇 = 𝑛 𝐼𝑋 (𝜇) = 𝑛 × =
𝜎2 𝜎2
[email protected]
Note 4
Let 𝐼𝑛 𝜃 be the Fisher information contributed by the sample and 𝐼𝑇 𝜃 by the
statistic.
If 𝐼𝑛 𝜃 = 𝐼𝑇 𝜃 ⟹ 𝑇 is sufficient for 𝜃.

Example 7
let 𝑋 a RV with uniform distribution on 0, 𝜃 , and 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample from 𝑋
1 𝑥
with density 𝑓 𝑥 = , 𝑥 ∈ [0, 𝜃], and distribution function 𝐹 𝑥 = , 𝑥 ∈ [0, 𝜃].
𝜃 𝜃
1-Find the Fisher information provided by an n-sample on the parameter 𝜃?
2-Let the order statistic 𝑇 = 𝑠𝑢𝑝 𝑋𝑖 , calculate the Fisher information carried by 𝑇 .
3-Is the statistic 𝑇 sufficient for 𝜃.

[email protected]
Solution:
1-The Fisher information provided by an n-sample on the parameter 𝜃:
𝜕ln 𝐿 𝑥;𝜃 2
The domain of 𝑋 depends on the parameter 𝜃. Therefore: 𝐼𝑛 𝜃 = 𝐸
𝜕𝜃
𝑋 ↷⊔[0,𝜃]
1
𝑓 𝑥; 𝜃 = , 𝑥 ∈ [0, 𝜃].
𝜃

Therefore, the likelihood is:

1
𝐿 𝑥 ; 𝜃 = 𝜃𝑛, 𝑥 ∈ [0, 𝜃],
ln 𝐿 𝑥 ; 𝜃 = −𝑛 ln 𝜃 , 𝑥 ∈ [0, 𝜃],
𝜕 ln 𝐿 𝑥 ;𝜃 −𝑛
= , 𝑥 ∈ [0, 𝜃],
𝜕𝜃 𝜃

Therefore,
−𝑛 2 −𝑛 2
𝐼𝑛 𝜃 = 𝐸 =𝐸
𝜃 𝜃
𝜃 𝜃 𝜃 𝜃 𝜃
2 2
𝑛 2 𝑛 1 𝑛 1 𝑛2 1 𝑛 𝑛2
𝐼𝑛 𝜃 = න … න 𝐿(𝑥, 𝜃) 𝑑𝑥1 … 𝑑𝑥𝑛 = න … න 2 𝑛 𝑑𝑥1 … 𝑑𝑥𝑛 = 2 𝑛 න 1 𝑑𝑥1 … 𝑑𝑥𝑛 = 2 𝑛 𝜃 = 2
𝜃 𝜃 𝜃 𝜃 𝜃 𝜃 𝜃 𝜃
0 0 0 0 0
[email protected]
2-The Fisher information carried by 𝑇 = 𝑠𝑢𝑝 𝑋𝑖 :
Calculating the distribution 𝑔 𝑡; 𝜃 of the statistic 𝑇 = 𝑠𝑢𝑝 𝑋𝑖 :
𝐺 𝑡; 𝜃 = 𝑃(𝑇 ≤ 𝑡) = 𝑃(𝑠𝑢𝑝 𝑋𝑖 ≤ 𝑡) = (𝐹𝑋 (𝑡))𝑛
𝑡 𝑛−1 1 𝑛 𝑛−1
𝑔 𝑡; 𝜃 = 𝑛(𝐹𝑋 (𝑡))𝑛−1 × 𝑓𝑋 (𝑡) =𝑛 = 𝑡
𝜃 𝑛−1 𝜃 𝜃𝑛
𝑛 𝑛−1
𝑙𝑛 𝑔 𝑡; 𝜃 = ln 𝑛 𝑡 = 𝑙𝑛 𝑛 − 𝑛 𝑙𝑛 𝜃 + 𝑛 − 1 𝑙𝑛 𝑡
𝜃
𝜕𝑙𝑛𝑔 𝑡 ; 𝜃 −𝑛
=
𝜕𝜃 𝜃

𝑛2
3- 𝐼𝑛 𝜃 = 𝐼𝑇 𝜃 = ⟹ the statistic is sufficient
[email protected] for 𝜃.
𝜃2
Fréchet-Darmois-Cramér-Rao (FDCR) bound

Theorem4
If the domain of 𝑋 does not depend on the parameter to be estimated for any
unbiased estimator 𝑇 of 𝜃 we have:
1
𝑉𝑎𝑟 𝜃 𝑇 ≥ = 𝐵𝐹 (𝜃)
𝐼𝑛 𝜃
The quantity 𝐵𝐹 𝜃 is the Fréchet-Darmois-Cramer-Rao (FDCR) lower bound.

Efficient estimator
An unbiased estimator 𝑇 is said to be efficient if its variance is equal to the lower
bound of 𝐹𝐷𝐶𝑅 :
1
𝑉𝑎𝑟 𝜃 𝑇 = 𝐵𝐹 𝜃 =
𝐼𝑛 𝜃

[email protected]
Example 8
Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample from a RV 𝑋 with the normal distribution 𝑁(𝜇, 𝜎 2 ).
1
- The statistic 𝑇 = σ𝑛𝑖=1 𝑋𝑖 an estimator of 𝜇. Is 𝑇 efficient ?
𝑛
Solution 𝑛
1 𝜎2
𝑉𝑎𝑟 𝑇 = 𝑉𝑎𝑟 ෍ 𝑋𝑖 =
𝑛 𝑛
𝑖=1
𝑛
𝐼𝑛 (𝜇) = 2
𝜎
1 𝜎2
𝐵𝐹 𝜇 = = = 𝑉𝑎𝑟 𝑇 ⟹ 𝑇 is efficient for 𝜇.
𝐼𝑛 𝜇 𝑛

[email protected]
 7- Complete Statistics
Exponential family
The distribution of 𝑋 is said to be in the exponential family if its density 𝑓(𝑥, 𝜃) is
written in the form :
𝑓 𝑥, 𝜃 = 𝐶(𝜃)𝑒 𝑞 𝜃 𝑇(𝑥) ℎ(𝑥)
Example 9
Let 𝑋 ↷ 𝑃(𝜃)
−𝜃 𝜃𝑥
𝑃 𝑋=𝑥 =𝑒
𝑥!
𝑥 1
= 𝑒 −𝜃 𝑒 𝑙𝑛(𝜃 )
𝑥!
1
= 𝑒 −𝜃 𝑒 𝑥𝑙𝑛(𝜃)
𝑥!
1
𝐶 𝜃 =𝑒 −𝜃
, 𝑞 𝜃 = 𝑙𝑛 𝜃 , 𝑇 𝑥 = 𝑥 and ℎ 𝑥 = .
𝑥!
The Poisson distribution belongs to the exponential family.

[email protected]
Theorem5 (DARMOIS)
If the domain of 𝑋 does not depend on 𝜃 and the distribution of 𝑋 belongs to the
exponential family, then the statistic 𝑇 = σ𝑛𝑖=1 T(𝑥𝑖 ) is sufficient and complete statistic
for 𝜃.

Example 10
Let 𝑋 ↷ 𝑃 𝜃 .
- Is 𝑇 = σ𝑛𝑖=1 𝑥𝑖 a complete statistic for 𝜃?
Solution
The Poisson distribution belongs to the exponential family (see Example 9)
The domain of 𝑋 does not depend on 𝜃 and its distribution belongs to the exponontiel
family,
So according to DARMOIS' theorem 𝑇 = σ𝑛𝑖=1 T(𝑥𝑖 ) = σ𝑛𝑖=1 𝑥𝑖 is sufficient and complete
statistic for 𝜃.

[email protected]
 8-MINIMUM VARIANCE UNBIASED estimator (MVUE)

Theorem6 (Rao-Blackwell-Lehman-Scheffe)
Let 𝑋 be a random variable of probability density 𝑓(𝑥, 𝜃). Let 𝑇 be a sufficient and complete
statistic for 𝜃, then for every estimable 𝛾 𝜃 there exists a unique MVUE of 𝛾 𝜃 based on 𝑇.
Example 11
Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample from 𝑋 ↷ 𝑃 𝜃 .
1-Find a sufficient and complete statistic for 𝜃.
2-Find the MVUE of 𝜃.
Solution
1- Sufficient and complete statistic for 𝜃 :
From previous example 𝑇 = σ𝑛𝑖=1 𝑥𝑖 is a complete and sufficient statistic.
2-MVUE of 𝜃:
𝑇 is a complete and sufficient statistic for 𝜃, it still remains to check that 𝑇 is unbiased.
We have 𝑇 = σ𝑛𝑖=1 𝑥𝑖 ,
𝐸 𝑇 = 𝐸 σ𝑛𝑖=1 𝑥𝑖 = 𝑛 𝐸 𝑥 = 𝑛𝜃 ⟹ 𝑇 is with bias.
1 σ𝑛
𝑖=1 𝑥𝑖
𝑇′ = 𝑇 = is the 𝑀𝑉𝑈𝐸 of 𝜃.
𝑛 𝑛
[email protected]
9-Estimation by confidence interval
Confidence interval estimation of a parameter 𝜃 consists of associating with a sample a random
interval 𝐼 chosen in such a way that the probability of it containing the unknown value of the
parameter is equal to a number fixed in advance, as large as desired. We write: 𝑃 𝜃 ∈ 𝐼 = 1 − 𝛼
1 − 𝛼: is the probability associated with the interval of containing the true value of the
parameter; this is the confidence level or quasi-certainty.
9-1-Construction principal
Let 𝑇 be an estimator of 𝜃 (best possible estimator) whose distribution as a function of 𝜃 is
known. We then determine a probability interval of level 1 − 𝛼 for 𝑇 i.e.:
𝑃(𝑡1 𝜃 ≤ 𝑇 ≤ 𝑡2 𝜃 ) = 1 − 𝛼
We then need to invert this interval for 𝑇, whose bounds depend on 𝜃 to obtain an interval for 𝜃,
i.e.:
𝑃(𝑇1 ≤ 𝜃 ≤ 𝑇2 ) = 1 − 𝛼
𝛼
𝑃(𝜃 < 𝑇1 ) =
2
𝛼
𝑃(𝜃 > 𝑇2 ) =
2

[email protected]
9-2-Principle of construction Confidence interval for the mean of a normal
distribution

❑𝝈𝟐 is known
Example 12. 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample drawn from a normal distibution 𝑋 ↷ 𝑁 𝜇, 𝜎 2 ,
with the variance 𝜎 2 is known.
- Find a confidence interval for the mean 𝜇 .
• We know that 𝑇 = 𝑋ത is the best estimator of 𝜇.
1 𝑛 1
𝐸 𝑇 =𝐸 σ 𝑋 = 𝑛 𝐸 𝑋𝑖 =𝜇
𝑛 𝑖=1 𝑖 𝑛
1 𝑛 1 𝜎2
𝑉 𝑇 =𝑉 σ 𝑋 = 2 𝑛 𝑉 𝑋𝑖 =
𝑛 𝑖=1 𝑖 𝑛 𝑛
𝛼
𝑃 𝑇 < 𝑇1 = 𝑃 𝑋ത < 𝑇1 =
2

[email protected]
As the distribution function of the normal distribution is only tabulated for the standardized
𝜎2 𝑇−𝜇
variable, we will therefore standardize the variable 𝑇. We obtain: 𝑇 ↷ 𝑁(𝜇, ) ⟹ 𝜎 ↷ 𝑁(0,1)
𝑛
𝑛
𝑇−𝜇 𝑇1 −𝜇 𝛼 𝑇1 −𝜇 𝛼 𝑇1 −𝜇 𝛼 𝑇1 −𝑋ത 𝛼
𝑃 𝜎 < 𝜎 = ⟹𝑃 𝑍 < 𝜎 = ⟹∅ 𝜎 = ⟹ 𝜎 = ∅−1
2 2 2 2
𝑛 𝑛 𝑛 𝑛 𝑛
𝑇1 −𝑋ത 𝛼
⟹ 𝜎 = −∅−1 1 −
2
𝑛
𝜎 𝛼
⟹ 𝑇1 = 𝑋ത − ∅−1 1 −
𝑛 2
𝛼 𝛼 𝑇−𝜇 𝑇2 −𝜇 𝛼 𝑇2 −𝑋ത 𝛼
𝑃 𝑇 > 𝑇2 = ⟹ 𝑃 𝑇 < 𝑇2 = 1 − ⟹𝑃 𝜎 < 𝜎 =1 − ⟹ 𝑃 𝑍< 𝜎 =1−
2 2 2 2
𝑛 𝑛 𝑛

𝑇2 −𝑋ത 𝛼
⟹∅ 𝜎 =1−
2
𝑛
𝑇2 −𝑋ത 𝛼
⟹ 𝜎 = ∅−1 1 −
2
𝑛
𝜎 𝛼
⟹ 𝑇2 = 𝑋ത + ∅−1 1 −
𝑛 2
𝜎 −1 𝛼 𝜎 −1 𝛼
ത
𝐶𝐼1−𝛼 (𝜇) = 𝑋 − ∅ 1− ത
;𝑋 + ∅ 1−
2
𝑛 [email protected] 𝑛 2
❑𝝈𝟐 is Unknown
Example 13 𝑋1 , 𝑋2 , … , 𝑋𝑛 an n-sample drawn from a normal distibution 𝑋 ↷ 𝑁 𝜇, 𝜎 2 , with the variance
𝜎 2 is unknown.
- Find a confidence interval for the mean 𝜇 .

1
X = σni=1 Xi
• The mean of a sample: ഥ
n
1
• The unbiased variance of the sample: 𝑆 ′2 = σ𝑛𝑖=1(𝑋𝑖 ത 2
− 𝑋)
𝑛−1

1
Note 5 we use the biased empirical variance S = 𝑛 σ𝑛𝑖=1(𝑋𝑖 − 𝑋)
ത 2 in theory and mathematical models,
1
and the unbiased empirical variance 𝑆 ′2 = 𝑛−1 σ𝑛𝑖=1(𝑋𝑖 − 𝑋)
ത 2 in estimation from sample data.

ത
𝑋−𝜇
• Change of variable: 𝑆′
↷ 𝑡𝑛−1
𝑛
• Because:
𝜎2
The mean of a sample: 𝑋ത ↷ 𝑁 𝜇,
n
The unbiased variance of the sample: 𝑆 ′2 ↷ 𝜒 2
⟹The standardized ratio of these two independent quantities gives a student distribution.
[email protected]
 ത
𝑋−𝜇
𝑃 −𝑡 ≤ 𝑆′
≤𝑡 =1−𝛼
𝑛
𝑆′ 𝑆′
𝑃 𝑋ത − 𝑡 ≤ 𝜇 ≤ 𝑋ത + 𝑡 =1−𝛼
𝑛 𝑛

Where: 𝑡 = 𝑡 𝛼
𝑛−1,1−
⟹𝑡=𝑡 𝑛−1,
𝛼
2 2
𝑡=𝑡 𝑛−1,1− 2
𝛼 ⟹𝑡=𝑡 𝑛−1, 2
𝛼 𝑤𝑖𝑙𝑙 𝑏𝑒 𝑡𝑎𝑘𝑒𝑛 𝑓𝑟𝑜𝑚 𝑡ℎ𝑒 𝑆𝑡𝑢𝑑𝑒𝑛𝑡′𝑠 𝑑𝑖𝑠𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛 𝑡𝑎𝑏𝑙𝑒.

Note 6 The symmetry of STUDENT's distribution gives the following property: 𝑡(𝑛,1−𝛼) = −𝑡 𝑛,𝛼 ).

𝑆′
𝑇1 = 𝑋ത − 𝑡 𝛼
𝑛 𝑛−1, 2
𝑆′
𝑇2 = 𝑋ത + 𝑡 𝛼
𝑛 𝑛−1, 2
𝑆′ 𝑆′
𝐶𝐼1−𝛼 (𝜇) = 𝑋ത − 𝑡 𝛼
𝑛−1, 2
; 𝑋ത + 𝑡 𝛼
𝑛−1, 2
𝑛 𝑛
[email protected]
[email protected]