Large Sample Inference

Dr. Xinyue Li

Department of Data Science, City University of Hong Kong

Lecture 6

1 / 20
Central Limit Theorem

Theorem 1
Let X1 , X2 , . . . , Xn denote the observations of a random sample from a
distribution that has mean µ and variance σ 2 . Then we have
√ D
Yn = n(X n − µ)/σ −
→ N(0, 1)

2 / 20
Proof of the Central Limit Theorem

Let M(t) be the mgf of X. Consider the mgf of X − µ

m(t) = E (e t(X −µ) ) = e −µt M(t)

As m(t) is the mgf for X − µ, it must follow that

m(0) = 1, m′ (0) = E (X − µ) = 0

and

m′′ (0) = E (X − µ)2 = σ 2 .

By Taylor’s formula, there exists a number ξ ∈ (0, t) such that

m′′ (ξ)t 2
m(t) = m(0) + m′ (0)t +
2
m′′ (ξ)t 2 σ2 t 2 [m′′ (ξ) − σ 2 ]t 2
= 1+ =1+ +
2 2 2

3 / 20
Proof of the Central Limit Theorem
Now consider the mgf of Yn and written as M(t; n)
  P 
Xi − nµ
M(t; n) = E exp t √
  σ n    
X1 − µ X2 − µ Xn − µ
= E exp t √ exp t √ · · · exp t √
  σ n   σ n  σ n
X1 − µ Xn − µ
= E exp t √ · · · E exp t √
   σ n n σ n
X1 − µ
= E exp t √
 n σ n
t t
= m( √ ) , −h < √ < h
σ n σ n
√
Replace t by t/σ n in previous equation we have
t t2 [m′′ (ξ) − σ 2 ]t 2
m( √ ) = 1 + +
σ n 2n 2nσ 2
√
where ξ is now between 0 and t/σ n. Accordingly
n n
t2 [m′′ (ξ) − σ 2 ]t 2 t2
 
2
M(t; n) = 1 + + → 1+ → e t /2
2n 2nσ 2 2n

4 / 20
Large sample inference for µ

Example 2
Let X1 , X2 . . . , Xn be a random sample from a distribution with mean µ and
variance σ 2 , where µ and σ 2 are unknown. Let X and S be the sample mean
and sample standard deviation, respectively. Then we have

X −µ D
√ −→ N(0, 1)
S/ n

Why does this hold?

5 / 20
Recall the Example on Sn

Example 3
Let X1 , . . . , Xn be a sample from the distribution. Show that the variance
n
1X
Sn2 = (Xi − X )2
n i=1

is an consistent estimator of σ 2 .

n n
1X 1X 2 2
Sn2 = (Xi − X )2 = Xi − X n
n i=1 n i=1
P
−
→ E (X12 ) − µ2 = σ 2

6 / 20
Normal Approximation to the Binomial Distribution

Example 4
Let X1 , X2 . . . , Xn be a random sample from a binomial distribution b(1, p). If
Yn = X1 + X2 + . . . + Xn , then Yn had binomial distribution b(n, p). The mean
and variance of Yn are np and np(1 − p) respectively. Show that Yn can be
approximated with a normal distribution.

7 / 20
Normal Approximation to the Binomial Distribution
In practice, when n is as small as 10 and p = 0.5 (so np = 5), the
approximation can still be rather good.
A comparison between binomial b(n, p) = b(10, 0.5) and normal
N(np, np(1 − p)) = N(5, 2.5).

8 / 20
Poisson Approximation to the Binomial Distribution

Example 5
Let Yn have a distribution that is b(n, p). Suppose that the mean λ = np is
the same for every n, that is p = λ/n, where λ is a constant. The limiting
distribution of the binomial distribution is Poisson.

9 / 20
Normal Approximation to the Binomial Distribution

Example 6
Let Y follow a binomial distribution with n = 100, p = 0.5. Find
P(Y = 48, 49, 50, 51, 52)

10 / 20
Large sample inference for proportions

Example 7
Let X1 , X2 . . . , Xn be a random sample from a binomial distribution b(1, p) with
p as the probability of success. Let p̂ = X be the sample proportion of success.
Then
pb − p D
p −
→ N(0, 1)
pb(1 − pb)/n

Why does this hold?

11 / 20
Large sample inference for χ2 -Test

Example 8
Let X1 , X2 . . . , Xn be a random sample from a binomial distribution b(1, p) with
p as the probability of success. Let p̂ = X be the sample proportion of success.
Then
pb − p D
p −
→ N(0, 1)
pb(1 − pb)/n

We actually further have

!2
pb − p D
p −
→
pb(1 − pb)/n

12 / 20
χ2 , Exponential and Gamma distribution

χ2 and exponential distribution are special cases of Gamma distribution

The Gamma distribution Γ(α, β) has the pdf
(
1 α−1 −x/β
αx e 0<x <∞
f (x) = Γ(α)β
0 elsewhere

where
Z ∞
Γ(α) = y α−1 e −y dy
0

For integer α, we have Γ(α) = (α − 1)!

The mean and variance of Gamma distribution

µ = αβ, σ 2 = αβ 2

Show by yourself using MGF.

Gamma distribution is frequently used to model the waiting time.

13 / 20
Gamma Distribution

Theorem 9
Let X1 , ..., Xn be idependent random
P variables. Suppose forPi = 1, ..., n, Xi has
a Γ(αi , β) distribution. Let Y = ni=1 Xi , then Y has a Γ( ni=1 αi , β)
distribution.

14 / 20
Poisson Process: A Recall

Consider a process that occurs repeatedly through time. Let Xt denote

the number of occurrences of the process over the interval (0, t]
Define the probability mass function of Xt as P(Xt = k) = g (k, t).
Assume three conditions.
(a) P
g (1, h) = λh + o(h)
∞
(b) k=2 g (k, h) = o(h)
(c) The number of occurrences in non-overlapping intervals are
independent
By above three conditions, we will have

(λt)x e −λt
g (x, t) =
x!

15 / 20
Gamma distribution
Let W be the time that is needed to obtain exact k occurrences. Then
the CDF of W is

G (w ) = P(W ≤ w ) = 1 − P(W > w )

The event W > w is equivalent to the event that there are fewer than k
occurrences in a interval of length w .
Let X be the number of occurrences in an interval of length w , then
k−1 k−1
X X (λw )x e −λw
P(W > w ) = P(X = x) =
x=0 x=0
x!

It then follows that

k−1
(λw )x e −λw
Z w k k−1 −λw
X λ w e
G (w ) = 1 − = dw
x=0
x! 0 Γ(k)

This leads to the Gamma distribution with α = k, β = 1/λ.

16 / 20
Exponential distribution

If W is the waiting time until the first occurrence, i.e., k = 1, then the
pdf of W is
(
λe −λw 0<w <∞
g (w ) =
0 elsewhere

Then W is said to have the exponential distribution.

Let the waiting time W have a Gamma pdf with α = k, β = 1/λ. Then

E (W ) = k/λ

17 / 20
Gamma Distribution

18 / 20
χ2 distribution

A χ2 distribution is a special case of Gamma distribution with α = r /2,

β = 2, where r is a positive integer.
The pdf of χ2 distribution
(
1
Γ(r /2)2r /2
x r /2−1 e −x/2 0<x <∞
f (x) =
0 elsewhere

For χ2 distribution, r is the degree of freedom. The mean of χ2 (r ) is r ,

and the variance is 2r .

19 / 20
χ2 distribution
If X ∼ N(0, 1), then

X 2 ∼ χ2 (1)

If X1 , X2 , . . . , Xr are i.i.d. N(0, 1), then

r
X
Xi2 ∼ χ2 (r )
i=1

If W ∼ N(0, 1), V ∼ χ2 (r ), then

W
p ∼ t(r )
V /r

If U ∼ χ2 (r1 ), V ∼ χ2 (r2 ), then

U/r1
∼ F (r1 , r2 )
V /r2

20 / 20