Statistical Inference

Dr. Venkataramana B

Vellore Institute of Technology, Vellore

August 12, 2021

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 1/1

Overview

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 2/1

The Theory of Estimation was founded by Prof. R A Fisher in 1930.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 3/1

Keywords

Population
Sample
Parameter
Statistic

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 4/1

Notations

Statistical Measure Parameter Statistic

Mean µ x̄
Median M m
Variance σ2 s2
Standard deviation σ s
Proportion P p
Correlation Coefficient ρ r
Regression Coefficient β b

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 5/1

Let us consider a random variable X with probability density function (p.d.f )
f (x, θ). The functional form of the population distribution is assumed to be
known except for the value of some unknown parameter(s) θ which may take
any value on a set Θ.
Parameter Space
The set Θ, which is the set of all possible values of θ is called the Parameter
space i.e. the p.d.f can be written as f (x, θ), θ ∈ Θ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 6/1

Estimate & Estimator

Definition 1.1
Any function of the random sample x1 , x2 , . . . ,xn that are being observed, say
Tn (x1 , x2 , . . . ,xn ) is called a statistic. Clearly, a statistic is a random variable.
If it is used used to estimate an unknown parameter θ of the distribution, it is
called an estimator.

Definition 1.2
A particular value of the estimator, say Tn (x1 , x2 , . . . ,xn ) is called an estimate
of θ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 7/1

Example

How to assess the performance of estimators and to choose the best one?

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 8/1

Example

How to assess the performance of estimators and to choose the best one?
Are there methods for obtaining estimators other than "ad-hoc" methods.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 8/1

Characteristics of Estimators

Unbiasedness
Consistency
Efficiency
Sufficiency

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 9/1

Unbiasedness
Definition 2.1
An estimator Tn = T (x1 , x2 , . . . ,xn ) is said to be an unbiased estimator of
γ(θ) if E(Tn ) = γ(θ), for all θ ∈ Θ.

E(Statistic) = P arameter

Remark. If E(Tn ) > θ, Tn is said to be positively biased and

if E(Tn ) < θ, it is said to be negatively biased,
the amount of bias b(θ) is given by b(θ) = E(Tn ) − γ(θ), θ ∈ Θ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 10 / 1

Example
Suppose that X is a random variable with mean µ and variance σ 2 . Let X1 , X2 , . . . ,Xn be a
random sample of size n from the population represented by X. Show that the sample mean
X̄ and sample variance S 2 are unbiased estimators of µ and σ 2 , respectively.

Solution: Since E(Xi ) = µ and V (Xi ) = σ 2 ; i = 1, 2, . . . , n

Pn
Xi
Sample Mean, X̄ = i=1
n
 Pn
i=1 Xi

E(X̄) = E
n
E(X1 ) + E(X2 ) + . . . + E(Xn )
=
n
µ + µ + ... + µ
=
n
nµ
=
n
E(X̄) = µ

Therefore, the sample mean X̄ is unbiased estimator of the population mean µ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 11 / 1

 Pn
− X̄)2
i=1 (Xi
SampleV ariance, s2 =
n
Pn
(Xi − X̄)2
S 2 = i=1
n−1

n
"P # " #
n
− X̄)2
i=1 (Xi 1 X
E(S 2 ) = E = E (Xi − X̄)2
n−1 n−1 i=1
n
" #
1 X
= E (Xi2 + X̄ 2 − 2X̄Xi )
n−1 i=1
n n n
" #
1 X X X
= E Xi2 + X̄ 2 − 2X̄ Xi
n−1 i=1 i=1 i=1
n
" #
1 X
= E Xi2 + nX̄ 2 − 2nX̄ 2
n−1 i=1
n n
" # " #
1 X 1 X
= E Xi2 − nX̄ 2 = E(Xi2 ) − nE(X̄ 2 )
n−1 i=1
n − 1 i=1

However, since E(Xi2 ) = µ2 + σ 2 and E(X̄ 2 ) = µ2 + σ 2 /n

" n #
1 X
E(S 2 ) = (µ2 + σ 2 ) − n(µ2 + σ 2 /n)
n−1 i=1
1  2 
= nµ + nσ 2 − nµ2 − σ 2
n−1
E(S 2 ) = σ 2

Therefore, the sample variance S 2 is an unbiased estimator of the population variance σ 2 .

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 12 / 1

Example
Let X1 , X2 , . . . , Xn is a random sample from a Normal population N (µ, 1).
Show that t = n1 ni=1 Xi2 , is an unbiased estimator of 1 + µ2 .
P

Solution:
Given, E(Xi ) = µ, V (Xi ) = 1, ∀ i = 1, 2, . . . , n
=⇒ E(Xi2 ) = V (Xi ) + {E(Xi )}2 = 1 + µ2 {∵ V (x) = E(X 2 ) − (E(X))2 }
n n n
" #
1X 1X 1X
E(t) = E Xi2 = E(Xi2 ) = (1 + µ2 )
n i=1 n i=1 n i=1
n(1 + µ2 )
=
n
E(t) = 1 + µ2

Hence, t is an unbiased estimator of 1 + µ2 .

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 13 / 1

Example
If T is an unbiased estimator for θ, show that T 2 is a biased estimator for θ2 .

Solution: Since, T is an unbiased estimator for θ,

we have, E(T ) = θ

Also, V ar(T ) = E(T 2 ) − [E(T )]2 = E(T 2 ) − θ2

=⇒ E(T 2 ) = θ2 + V ar(T )
Since, E(T 2 ) 6= θ2 , T 2 is a biased estimator for θ2 .

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 14 / 1

Example
hP P i
xi ( xi −1)
Show that n(n−1) is an unbiased estimator of θ2 , for the sample x1 , x2 , . . . ,xn drawn on X which takes the values 1 or 0
with respective probabilities θ and (θ − 1).

Solution: Since x1 , x2 , . . . ,xn is a random sample from Bernoulli population with parameter θ,
n
X
Let T = xi ∼ B(n,θ)
i=1
=⇒ E(T ) = nθ and V ar(T ) = nθ(1 − θ)
n o
X ∼ B(1, p) =⇒ E(X) = p & V ar(X) = pq

h P i
xi − 1)
P
xi ( 
T (T − 1)

E =E
 
n(n − 1) n(n − 1)
1 h i
= E(T 2 ) − E(T )
n(n − 1)
1 h i
= V ar(T ) + {E(T )}2 ) − E(T )
n(n − 1)
1 h i
= nθ(1 − θ) + n2 θ2 − nθ
n(n − 1)
nθ2 (n − 1)
= = θ2
n(n − 1)

hP P i
xi ( xi −1)
=⇒ n(n−1) is an unbiased estimator of θ2 .

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 15 / 1

Example
Let X be distributed in the Poisson form with parameter θ. Show that only
unbiased estimator of exp − (k + 1)θ ,k > 0, is T (X) = (−k)X so that

T (X) > 0 if x is even and T (X) < 0 if x is odd.

Solution:
e−θ θx
Since X ∼ P oisson(θ) =⇒ P (X = x) = x! ; X = 0, 1, . . . , ∞

E{T (X)} = E[(−k)X ], k > 0

∞
X n e−θ θ x o
= (−k)X
x=0
x!
∞ h
(−kθ)X i
= e−θ
X

x=0
x!
−θ −kθ
=e ·e
−(k+1)θ
E{T (X)} =e

=⇒ T (X) = (−k)X is an unbiased estimator for exp − (k + 1)θ , k > 0.



Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 16 / 1

Consistency

Definition 2.2
An estimator Tn = T (x1 , x2 , . . . ,xn ) based on a random sample of size n, is
said to be consistent estimator of γ(θ), θ ∈ Θ, the parameter space,
p
if Tn converges to γ(θ) in probability, i.e., if Tn −→ γ(θ) as n −→ ∞.

In other words, Tn is a consistent estimator of γ(θ) if for every ε > 0, η > 0,

there exists a positive integer n ≥ m(ε, η) such that

P {|Tn − γ(θ)| < ε} → 1 as n −→ ∞

⇒P {|Tn − γ(θ)| < ε} > 1 − η; ∀ n ≥ m(ε, η)

where ε, η are arbitrarily small positive numbers and m is some large value of
n.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 17 / 1

Remarks
1 If X1 , X2 , . . . , Xn is a random sample from population with finite mean,
E(Xi ) = µ, then by Khinchine’s weak law of large numbers (W.L.L.N),
we have
1 Pn p
X̄n = n i=1 Xi −→ E(Xi ) = µ, as n → ∞.
Hence the sample mean (X̄n ) is always a consistent estimator of the
population mean (µ).

2 Consistency is a property concerning the behavior of an estimator for

indefinitely large values of the sample size n, i.e., as n → ∞.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 18 / 1

In-variance Property

Theorem 2.3
If Tn is a consistent estimator of γ(θ) and ψ{γ(θ)} is a continuous function
of γ(θ), then ψ(Tn ) is a consistent estimator of ψ{γ(θ)}.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 19 / 1

Sufficient conditions for Consistency

Theorem 2.4
Let {Tn } be a sequence of estimators such that for all θ ∈ Θ,

(i)Eθ (Tn ) → γ(θ), n −→ ∞ and (ii)V arθ (Tn ) → 0, as n −→ ∞.

Then Tn is a consistent estimator of γ(θ).

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 20 / 1

Example
Prove that in sampling from a N (µ, σ 2 ) population, the sample mean is
consistent estimator of µ.

Solution: Given X ∼ N (µ, σ 2 ) =⇒ E(X) = µ and V (X) = σ 2

X + . . . + X 
1 n
V ar(X̄) = V ar
X + . . . + X  n
1 n 1
E(X̄) = E =V (X1 + . . . + Xn )
n n2
1
={E(X1 ) + . . . + E(Xn )} { Since Xi ’s are independent RV’s}
n
1 1
= (µ + . . . + µ) = 2 {V (X1 ) + . . . + V (Xn )}
n n
1 1
= nµ = 2 (σ 2 + . . . + σ 2 )
n n
1
E(X̄) = µ = 2 nσ 2
n
σ2 n σ o
V (X̄) = ∵ sd(X̄) = √
n n

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 21 / 1

 σ2
=⇒ E(X̄) = µ and V (X̄) =
n
Thus as n → ∞,
=⇒ E(X̄) = µ and V (X̄) = 0

Hence by theorem 2.4, X̄ is a consistent estimator of µ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 22 / 1

Example
If X1 , X2 , . . . , Xn is a random observations on a Bernoulli variate X taking
the value 1 with probability p and the value 0 with probability (1 − p), show
that
P  P 
xi xi
n 1− n is a consistent estimator of p(1 − p).

Solution:
Ṡince X1 , . . . , Xn are i.i.d Bernoulli random variables with the parameter ‘p0 ,
n
X
T = Xi ∼ B(n, p)
i=1
E(T ) = np and V ar(T ) = npq
E(X̄) =
V ar(X̄) =

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 23 / 1

Efficiency

Definition 2.5
If Tn is the most efficient estimator with variance V1 and T2 is any other
estimator with variance V2 , then the efficiency E of T2 is defined as :
V1
E= V2

E cannot exceed unity.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 24 / 1

Example: In sampling from a Normal population N (µ, σ 2 ), when σ 2 is known, sample mean x̄ is an unbiased
and consistent estimator of population mean µ.
From symmetry it follows immediately that sample median (Md) is an unbiased estimate of µ, which is the same
as the population median. Also for large n,
1
V (M d) =
4nf12
Here f1 = Median ordinate of the parent distribution
= Modal ordinate of the parent distribution
h 1 2 2
i
= √ e−(x−µ) /2σ
σ 2π x=µ
1
= √
σ 2π
1 πσ 2
∴ V (M d) = · 2πσ 2 =
4n 2n

Since E(M d) = µ and V (M d) = 0, as n→∞

σ2
For all n, V (x̄) =
n
πσ 2 σ2
For large n, V (M d) = = 1.57
2n n
Since V (x̄) < V (M d), we conclude that for normal distribution, sample mean is more efficient estimator for µ
than the sample median, for large samples at least.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 25 / 1

Example
A random sample(X1 , X2 , X3 , X4 , X5 ) of size 5 is drawn from a normal
population with unknown mean µ. Consider the following estimators to
estimate µ :
(i)t1 = X1 +X2 +X53 +X4 +X5 , (ii)t2 = X1 +X
2
2
+ X3 , (iii)t3 = 2X1 +X32 +λX3 ,
where λ is such that t3 is an unbiased estimator of µ.
(a) Find λ. Are t1 and t2 are unbiased?
(b) Also find the estimator which is best among t1 , t2 and t3 .

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 26 / 1

Example
Let the random sample (X1 , X2 ) and X3 of size 3 is drawn from a population
with mean µ and variance σ 2 . T1 , T2 and T3 are the estimators used to
estimate mean value µ, where
T1 = X1 + X2 − X3 , T2 = 2X1 + 3X3 − 4X2 and T3 = 13 (λX1 + X2 + X3 ).
1 Are T1 and T2 unbiased estimators?
2 FInd the value of λ such that T3 is unbiased estimator for µ.
3 With this value of λ is T3 a consistent estimator?
4 Which is the best estimator?

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 27 / 1

MVUE

Definition 2.6 (MVUE)

If a statistic T = T (x1 , x2 , . . . ,xn ), based on sample of size n is such that:
(i) T is unbiased for γ(θ), for all θ ∈ Θ and
(ii) It has the smallest variance among the class of all unbiased estimators of
γ(θ), then T is called minimum variance unbiased estimator (MVUE) of γ(θ).

T is MVUE of γ(θ) if,

Eθ (T ) = γ(θ) for all θ ∈ Θ

and V arθ (T ) ≤ V arθ (T 0 ) for all θ ∈ Θ

where T 0 is any other unbiased estimator of γ(θ).

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 28 / 1

MVUE

Theorem 2.7
An M.V.U.E is unique in the sense that if T1 and T2 are M.V.U. estimators for
γ(θ), then T1 = T2 almost surely.

Theorem 2.8
Let T1 and T2 be unbiased estimators of γ(θ) with efficiencies e1 and e2
√ p and ρ = ρθ be the correlation
respectively
√
coefficient
p between them. Then
e1 e2 − (1 − e1 )(1 − e2 ) ≤ ρ ≤ e1 e2 + (1 − e1 )(1 − e2 )

Corollary 2.9
√ √ √
If we take e1 = 1 and e2 = e, we get e≤ρ≤ e =⇒ ρ = e

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 29 / 1

Example
Data on pull-off force (pounds) for connectors used in an automobile engine
application are as follows: 79.3, 75.1, 78.2, 74.1, 73.9, 75.0, 77.6, 77.3,
74.6, 75.5, 74.0, 74.7, 75.9, 72.9, 73.8, 74.2, 78.1, 75.4, 76.3, 75.3,
76.2, 74.9, 78.0, 75.1, 76.8.
1 Calculate the point estimate of the mean pull-off force of all
connectors in the population. State which estimator you used and
why.
2 Calculate the point estimates of the population variance and
population standard deviation.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 30 / 1

Sufficiency

An estimator is said to be sufficient for a parameter, if it contains all the infor-

mation in the sample regarding the parameter.
Definition 2.10 (Sufficiency)
If a statistic T = t(x1 , x2 , . . . ,xn ), is an estimator of a parameter θ, based
on a sample x1 , x2 , . . . ,xn of size n from the population with the probability
density function f (x,θ) such that the conditional distribution of x1 , x2 , . . . ,xn
given T , is independent of θ then T is sufficient estimator for θ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 31 / 1

Example
Let x1 , x2 , . . . , xn be random sample from a Bernoulli population with the parameter ‘p0 , 0 < p < 1. Find the
sufficient estimator of p.

Solution:
Let x1 , x2 , . . . , xn be random sample from a Bernoulli
 population with the parameter
 ‘p‘, 0 < p < 1, i.e.,
 1, with probability p
 

xi =
 0, with probability q=(1-p) 
 

Then T = t(x1 , x2 , . . . , xn ) = x1 + x2 + . . . + xn ∼ B(n,p)

 
 n  k n−k
∴ P (T = k) =   p (1 − p)
k
The conditional distribution of (x1 , x2 , . . . , xn ) given T is
P [x1 ∩ x2 ∩ . . . ∩ xn ∈ T = k]
P [x1 ∩ x2 ∩ . . . ∩ xn |T = k] =
P (T = k)
 k (1−p)n−k



  p  = 1   

 
  n  k  n  

 
n−k

p (1−p)
=    


 k k 



 

Pn
x 6= k

0, if
 

i=1 i
n
Since this does not depend on 0 p0 , T = xi , is sufficient for 0 p0 .
X

i=1

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 32 / 1

Factorization Theorem
T = t(x) is sufficient for θ if and only if the joint density function L, of the
sample values can be expressed in the form:

L = gθ [t(x)].h(x)

where gθ [t(x)] is depends on θ and x only through the value of t(x) and
h(x) is independent of θ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 33 / 1

Fisher-Neyman Criterion
A statistic t1 = t(x1 , x2 , . . . , xn ) is sufficient estimator of parameter θ if and
only if the likelihood function (joint p.d.f (p.m.f)) can be expressed as:
n
Y
L= f (xi , θ) = g1 (t1 , θ) · k(x1 , x2 , . . . , xn )
i=1

where g1 (t1 , θ) is the p.d.f(p.m.f) of the statistic t1 and k(x1 , x2 , . . . , xn ) is a

function of sample observations only, independent of θ.

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 34 / 1

Example
Let x1 , x2 , . . . , xn be random sample from N (µ, σ 2 ) population. Find the sufficient
estimator for µ and σ 2 .

Solution:
Let us write θ = (µ, σ 2 ); −∞ < µ < ∞, 0 < σ 2 < ∞
Then  
n 1
Pn
Y 1 n
 − (x −µ)2
i=1 i
2σ 2
L= fθ (xi ) = √ ·e
i=1 σ 2π
h P i
n
− 12
P
 1 n i=1
xi 2 −2µ xi +nµ2
= √ · e 2σ
σ 2π
=gθ [t(x)] · h(x)
h i
1
 1 n − {t2 (x)−2µt1 (x)+nµ2 }
2σ 2
gθ [t(x)] = √ e
σ 2π
Where, X X 
t(x) =[t1 (x), t2 (x)] = xi , xi 2 and h(x) = 1
X
Thus t(x) = xi is sufficient for µ and

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 35 / 1

Example
Let x1 , x2 , . . . , xn be random sample from a distribution with the p.d.f :

f (x, θ) = θxθ−1 ; 0 < X < 1; θ > 0

n
Y
Show that t1 = Xi is sufficient estimator for θ.
i=1

Solution:
n n  
θ xiθ−1
Y Y
L(x, θ) = f (xi , θ) =
i=1 i=1
n 
Y 
n θ−1
=θ xi
i=1
Yn θ 1
=θn xi · Q 
n
i=1 i=1 xi

=g(t1 , θ) · h(x1 , x2 , . . . , xn ), (say)

Hence by Factorization theorem,
n
Y
t1 = xi , is sufficient for θ.
i=1

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 36 / 1

Cramer-Rao Inequality
If t is an unbiased estimator for γ(θ), a function of parameter θ, then

h i2 h i2
d
dθ · γ(θ) γ 0 (θ)
V ar(t) ≥ i2 =
I(θ)
h
∂
E ∂θ logL

where I(θ) is the information on θ, supplied by the sample.

Corollary
If t is an unbiased estimator for θ, i.e.

E(t) = θ =⇒ γ(θ) = θ =⇒ γ 0 (θ) = 1,

then by the cramer-rao in-equality, we get

1 1
V ar(t) ≥ i2 =
I(θ)
h
∂
E ∂θ logL

h 2 i
∂
Where I(θ) = E ∂θ logL is called by R. A. Fisher as the amount of information on θ supplied by the
sample (x1 , x2 , . . . , xn ) and its reciprocal 1/I(θ), as the information limit to the variance of estimator
t = t(x1 , x2 , . . . , xn ).

Dr. Venkataramana B (VIT-Vellore) Statistical Inference August 12, 2021 37 / 1