Inferential Statistic

Pr. Patrice Takam Soh,

Dept. of Math., University of Yaound I

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 1 / 16
 Statistical Inference: Estimation

Outline

1. Introduction
2. Statistical Inference: Estimation
3. Method of Maximum Likelihood
4. Principles of estimation
5. The sampling distribution of a statistic
6. Confidence intervals
7. The (Cramer-Rao) Information Inequality
8. Large Sample Properties of the MLE
9. Hypothesis Testing

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 2 / 16
 Statistical Inference: Estimation

The objectives

At the end of this course, each learner should be able to

I identify a statistical model in a given situation
I estimate the parameters of a statistical model using the method of
moments or the maximum likelihood method
I study the statistical properties of estimators
I construct confidence intervals for the parameters of a statistical model
I construct statistical tests associated to the parameters of a statistical
model
I estimate parameters in a linear regression model
I distinguish between parametric and non-parametric statistics
I distinguish between frequentist and Bayesian statistics

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 3 / 16
 Statistical Inference: Estimation

Definition and Uses for Inferential Statistics

1. Definition
1.1 Techniques that allow us to study samples and then make
generalizations about the population.
1.2 Inferential statistics are a very crucial part of scientific research in that
these techniques are used to test hypotheses
2. Uses for Inferential Statistics
2.1 Statistics for determining differences between experimental and control
groups in experimental research
2.2 Statistics used in descriptive research when comparisons are made
between different groups
2.3 These statistics enable the researcher to evaluate the effects of an
independent variable on a dependent variable

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 4 / 16
 Statistical Inference: Estimation

Basic terminology

1. Population: any collection of entities that have at least one

characteristic in common
2. Parameter: the numbers that describe characteristics of scores in the
population (mean, variance, s.d., etc.)
3. Sample: a part of the population
4. Statistic: the numbers that describe characteristics of scores in the
sample (mean, variance, s.d., correlation coefficient, reliability
coefficient, etc.)
5. Estimate: a number computed by using the data collected from a
sample
6. Estimator: formula used to compute an estimate

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 5 / 16
 Statistical Inference: Estimation

Statistical Inference: Motivation (1)

I Example 1.1. A company sells a certain kind of electronic component.

The company is interested in knowing about how long a component is
likely to last on average.
I They can collect data on many such components that have been used
under typical conditions.
I They choose to use the family of exponential distributions to model
the length of time (in years) from when a component is put into
service until it fails.
I The company believes that, if they knew the failure rate θ, then
X n = (X1 , X2 , ..., Xn ) would be n i.i.d random variables having the
exponential distribution with parameter θ.

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 6 / 16
 Statistical Inference: Estimation

Statistical Inference: Motivation (2)

We may ask the following questions:

I Can we estimate θ from this data? If so, what is a reasonable
estimator?
=⇒ find an estimator θ̂
I Can we quantify the uncertainty in the estimation procedure, i.e., can
we construct confidence interval for θ?
=⇒ find an interval I such that θ belongs to I with high probability
I What are the statistical properties of θ̂?
I Bias of θ̂
I Is θ̂ converges? (in probability? almost surely? in distribution?)
I How to make inference around θ̂?
I Hypotheses testing

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 7 / 16
 Statistical Inference: Estimation

Some results from probability (2)

Convergences
I We say that the sequence of r.v’s {Zn }n≥1 converges to Z a.s, and write
a.s
Zn → Z , if
Pr (lim Zn = Z ) = 1.
I We say that the sequence of r.v’s {Zn }n≥1 converges to Z in probability, and
P
write Zn → Z , if for every ε > 0,

Pr (|Zn − Z | > ε) → 0.

I We say that the sequence of r.v’s {Zn }n≥1 with c.d.f ’s Fn (.) converges in
distribution to F if
lim Fn (u) = F (u)
n→∞

for all u such that F is continuous at u (here F is itself a c.d.f).

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 8 / 16
 Statistical Inference: Estimation

Some results from probability (3)

Central limit theorem

If X1 , X2 , ..., are i.i.d with mean zero and variance 1, then
n
1 X d
√ Xi → N (0, 1)
n
i=1

where N (0, 1) is the standard normal distribution. More generally, the

usual rescaling tell us that, for X1 , X2 , ..., are i.i.d with mean µ and
variance σ 2 < ∞
n
√ 1 X d
n(X̄n − µ) ≡ √ (Xi − µ) → N (0, σ 2 )
n
i=1

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 9 / 16
 Statistical Inference: Estimation

Some results from probability (4)

P
I If Zn → b and if g (.) is a function that is continuous at b, then
P
g (Zn ) → g (b)
d
I If Zn → b and if g (.) is a function that is continuous at b, then
d
g (Zn ) → g (b)
a.s
I If Zn → b and if g (.) is a function that is continuous at b, then
a.s
g (Zn ) → g (b)

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 10 / 16
 Statistical Inference: Estimation

Applications: Back to Example 1.1

I by the LLN, the sample mean X̄n converges in probability to the

expectation 1/θ (failure rate), i.e.,

P 1
X̄n →
θ
I by the continuous mapping theorem X̄n−1 converges in probability to
θ, i.e.,
X̄n−1 → θ
P

I by the CLT, we know that

√ d
n(X̄n − θ−1 ) → N (0, θ−2 )

where Var (X1 ) = θ−2 ;

I But how does one find an approximation to the distribution of X̄n−1
Pr. Patrice Takam Soh, Dept. of Math., University of Yaound
Inferential
I Statistic 11 / 16
 Statistical Inference: Estimation

Delta method
I The first thing to note is that if {Zn }n≥1 converges in distribution (or
d
probability) to a constant θ, then g (Zn ) → g (θ), for any continuous
function g (.).
I We can also ’zoom in’ to look at the asymptotic distribution (not just the
limit point) of the sequence of r.v’s {Zn }n≥1 , whenever g (.) is sufficiently
smooth.

Theorem
Let Z1 , Z2 , ..., Zn be a sequence of r.v’s and let Z be a r.v with a
continuous c.d.f F ? . Let θ ∈ R, and let a1 , a2, .... be a sequence such that
an → ∞. Suppose that
d
an (Zn − θ) → F ?
Let g (.) be a function with a continuous derivative such that g 0 (θ) 6= 0.
Then
g (Zn ) − g (θ) d ?
an →F
g 0 (θ)
Pr. Patrice Takam Soh, Dept. of Math., University of Yaound
Inferential
I Statistic 12 / 16
 Statistical Inference: Estimation

Application

I The main application is when we’ve already proven a CLT for Zn ,

√
n(Zn − µ) d
→ N (0, 1),
σ
in which case
√ d
n(g (Zn ) − g (µ)) → N (0, (g 0 (µ)2 )σ 2 )
d
I Exercise 1: Assume n1/2 Zn → N (0, 1).
I What is the asymptotic distribution of g (Zn ) = (Zn − 1)2
I What about g (Zn ) = Zn2 ? Does anything go wrong when applying the
delta-method in this case? Can you fix this problem?
√ d
I Exercise 2: Back to Example 1.1; n(X̄n−1 − θ) →??????

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 13 / 16
 Statistical Inference: Estimation

III-1 Empirical Estimation: Mean

We consider a sample X1 , ..., Xn which is i.i.d with the common mean µ

and common variance σ 2 .
1. Empirical mean
n
1X
X̄n = Xi (1)
n
i=1

2. Expectation of X̄n

E(X̄n ) = µ (2)
3. Variance of X̄n

σ2
V(X̄n ) = (3)
n

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 14 / 16
 Statistical Inference: Estimation

III-2 Empirical Estimation: Variance

We consider a sample X1 , ..., Xn which is i.i.d with the common mean µ
and common variance σ 2 .
1. Empirical variance: µ is known
n
1X
Tn2 = (Xi − µ)2 (4)
n
i=1

with E(Tn2 ) = σ 2
2. Empirical variance: µ is unknown
n
1X
Sn2 = (Xi − X̄n )2 (5)
n
i=1

3. Expectation of Sn2
n−1 2
E(Sn2 ) = σ (6)
n
Pr. Patrice Takam Soh, Dept. of Math., University of Yaound
Inferential
I Statistic 15 / 16
 Statistical Inference: Estimation

III-3 Empirical Estimation: Proportion

We consider a sample X1 , ..., Xn which is i.i.d with Xi ∈ {0, 1} with

p = P(Xi = 1).
1. Empirical proportion
n
1X
pn = Xi (7)
n
i=1

2. Expectation of pn

E(pn ) = p (8)
3. Variance of pn

p(1 − p)
V(pn ) = (9)
n

Pr. Patrice Takam Soh, Dept. of Math., University of Yaound

Inferential
I Statistic 16 / 16