Point estimation

(Module 2)
Statistics (MAST20005) & Elements of Statistics (MAST90058)
Semester 2, 2018

Contents
1 Estimation & sampling distributions 1

2 Estimators 3

3 Method of moments 6

4 Maximum likelihood estimation 8

Aims of this module

• Introduce the main elements of statistical inference and estimation, especially the idea of a sampling distribution
• Show the simplest type of estimation: that of a single number
• Show some general approaches to estimation, especially the method of maximum likelihood

1 Estimation & sampling distributions

Motivating example

On a particular street, we measure the time interval (in minutes) between each car that passes:
2.55 2.13 3.18 5.94 2.29 2.41 8.72 3.71
We believe these follow an exponential distribution:
Xi ∼ Exp(λ)
What can we say about λ?
Can we approximate it from the data?
Yes! We can do it using a statistic. This is called estimation.

Statistics: the big picture

We want to start learning how to do inference. First, we need a good

understanding of the ‘sampling’ part.

1
Distributions of statistics

Consider sampling from X ∼ Exp(λ = 1/5).

Convenient simplification: set θ = 1/λ. This makes E(X) = θ and var(X) = θ2 .
Note: There are two common parameterisations,

fX (x) = λe−λx , x ∈ [0, ∞)

1 1
fX (x) = e− θ x , x ∈ [0, ∞)
θ
λ is called the rate parameter (relates to a Poisson process)
Be clear about which is being used!
Take a large number of samples, each of size n = 100:
1. 1.84 1.19 11.73 5.64 17.98 0.26 ...
2. 2.67 7.15 5.99 1.03 0.65 3.18 ...
3. 16.99 2.15 2.60 5.40 3.64 2.01 ...
4. 2.21 1.54 4.27 5.29 3.65 0.83 ...
5. 12.24 1.59 2.56 1.38 5.72 0.69 ...
...
Then calculate some statistics (x̄, x(1) , x(n) , etc.) for each one:
Min. Median Mean Max.
1. 0.02 4.10 5.17 23.96
2. 0.16 4.48 5.84 39.90
3. 0.17 3.39 4.38 15.61
4. 0.03 3.73 5.43 34.02
5. 0.01 3.12 4.71 19.94
...
As we continue this process, we get some information on the distributions of these statistics.

Sampling distribution (definition)

Recall that any statistic T = φ(X1 , . . . , Xn ) is a random variable.

The sampling distribution of a statistic is its probability distribution, given an assumed population distribution and a
sampling scheme (e.g. random sampling).
Sometimes we can determine it exactly, but often we might resort to simulation.
In the current example, we know that:

X(1) ∼ Exp(100λ)
X
Xi ∼ Gamma(100, λ)

How to estimate?

Suppose we want to estimate θ from the data. What should we do?

Reminder:
• Population mean, E(X) = θ = 5
• Population variance, var(X) = θ2 = 52
• Population standard deviation, sd(X) = θ = 5

2
Can we use the sample mean, X̄, as an estimate of θ? Yes!
Can we use the sample standard deviation, S, as an estimate of θ? Yes!
Will these statistics be good estimates? Which one is better? Let’s see. . .
We need to know properties of their sampling distributions, such as their mean and variance.
Note: we are referring to the distribution of the statistic, T , rather than the population distribution from which we
draw samples, X.
For example, it is natural to expect that:
• E(X̄) ≈ µ (sample mean ≈ population mean)
• E(S 2 ) ≈ σ 2 (sample variance ≈ population variance)
Let’s see for our example:
0.8

0.06
0.6

0.04
)
f ( x)

0.4

2
f (s

0.02
0.2

0.00
0.0

2 3 4 5 6 7 8 10 20 30 40 50 60
2
x s

Left: distribution of X̄. Right: distribution of S 2 . Vertical dashed lines: true values, E(X) = 5 and var(X) = 52 .
• Should we use X̄ or S to estimate θ? Which one is the better estimator?
• We would like the sample distribution of the estimator to be as close as possible to the true value θ = 5.
• In practice, for any given dataset, we don’t know which estimate is the closest, since we don’t know the true
value.
• We should use the one that is more likely to be the closest.
• Simulation: consider 250 samples of size n = 100 and compute:

x̄1 , . . . , x̄250 ,

s1 , . . . , s250
> summary(x.bar)
Min. 1st Qu. Median Mean 3rd Qu. Max.
3.789 4.663 4.972 5.015 5.365 6.424
> sd(x.bar)
[1] 0.4888185
> summary(s)
Min. 1st Qu. Median Mean 3rd Qu. Max.
3.502 4.473 4.916 5.002 5.512 7.456
> sd(s)
[1] 0.7046119
From our simulation, sd(X̄) ≈ 0.49 and sd(S) ≈ 0.70. So, in this case it looks like X̄ is superior to S.

2 Estimators
Definitions
• A parameter is a quantity that describes the population distribution, e.g. µ and σ 2 for N(µ, σ 2 )

3
 • The parameter space is the set of all possible values that a parameter might take, e.g. −∞ < µ < ∞ and
0 ≤ σ < ∞.
• An estimator (or point estimator) is a statistic that is used to estimate a parameter. It refers specifically to the
random variable version of the statistic, e.g. T = u(X1 , . . . , Xn ).
• An estimate (or point estimate) is the observed value of the estimator for a given dataset. In other words, it is
a realisation of the estimator, e.g. t = u(x1 , . . . , xn ), where x1 , . . . , xn is the observed sample (data).
• ‘Hat’ notation: If T is an estimator for θ, then we usually refer to it by θ̂ for convenience.

Examples

We will now go through a few important examples:

• Sample mean
• Sample variance
• Sample proportion
In each case, we assume a sample of iid rvs, X1 , . . . , Xn , with mean µ and variance σ 2 .

Sample mean

n
1 1X
X̄ = (X1 + X2 + . . . Xn ) = Xi
n n i=1
Properties:
• E(X̄) = µ
σ2
• var(X̄) = n

Also, the Central Limit Theorem implies that usually:

σ2
 
X̄ ≈ N µ,
n

Often used to estimate the population mean, µ̂ = X̄.

Sample variance

n
1 X
2
S2 = Xi − X̄
n − 1 i=1
Properties:
• E(S 2 ) = σ 2
• var(S 2 ) = (a messy formula)
Often used to estimate the population variance, σ̂ 2 = S 2 .

Sample proportion

For a discrete random variable, we might be interested in how often a particular value appears. Counting this gives
the sample frequency:
Xn
freq(a) = I(Xi = a)
i=1

Let the population proportion be p = Pr(X = a). Then we have:

freq(a) ∼ Bi(n, p)

4
Divide by the sample size to get the sample proportion. This is often used as an estimator for the population proportion:
n
freq(a) 1X
p̂ = = I(Xi = a)
n n i=1

For large n, we can approximate this with a normal distribution:

 
p(1 − p)
p̂ ≈ N p,
n

Note:
• The sample pmf and the sample proportion are the same, both of them estimate the probability of a given event
or set of events.
• The pmf is usually used when the interest is in many different events/values, and is written as a function, e.g.
p̂(a).
• The proportion is usually used when only a single event is of interest (getting heads for a coin flip, a certain
candidate winning an election, etc.).

Examples for a normal distribution

If the sample is drawn from a normal distribution, Xi ∼ N(µ, σ 2 ), we can derive exact distributions for these statistics.
Sample mean:
σ2
 
X̄ ∼ N µ,
n

Sample variance:
σ2 2
S2 ∼ χ
n − 1 n−1
2σ 4
E(S 2 ) = σ 2 , var(S 2 ) =
n−1

χ2k is the chi-squared distribution with k degrees of freedom. (more details in Module 3)

Bias

Consider an estimator θ̂ of θ.
• If E(θ̂) = θ, the estimator is said to be unbiased
• The bias of the estimator is, E(θ̂) − θ
Examples:
• The sample variance is unbiased for the population variance, E(S 2 ) = σ 2 . (problem 5 in week 3 tutorial)
• What if we divide by n instead of n − 1 in the denominator?

Transformations and biasedness

E( n−1 2
n S )=
n−1 2
n σ < σ2
⇒ biased!
In general, if θ̂ is unbiased for θ, then it will usually be the case that g(θ̂) is biased for g(θ).
Unbiasedness is not preserved under transformations.

Challenge problem
√
Is the sample standard deviation, S = S 2 , biased for the population standard deviation, σ?

5
Choosing between estimators
• Evaluate and compare the sampling distributions of the estimators.
• Generally, prefer estimators that have smaller bias and smaller variance (and it can vary depending on the
aim of your problem).
• Sometimes, we only know asymptotic properties of estimators (will see examples later).
Note: this approach to estimation is referred to as frequentist or classical inference. The same is true for most of the
techniques we will cover. We will also learn about an alternative approach, called Bayesian inference, later in the
semester.

Challenge problem (uniform distribution)

Take a random sample of size n from the uniform distribution with pdf:
1 1
f (x) = 1 (θ − <x<θ+ )
2 2
Can you think of some estimators for θ? What is their bias and variance?

Challenge problem (boundary problem)

Take a random sample of size n from the shifted exponential distribution, with pdf:

f (x) = e−(x−θ) (x > θ)

Equivalently:
Xi ∼ θ + Exp(1)
Can you think of some estimators for θ? What is their bias and variance?

Coming up with (good) estimators?

How can we do this for any given problem?

We will cover two general methods:
• Method of moments
• Maximum likelihood

3 Method of moments
Method of moments (MM)
• Idea:
– Make the population distribution resemble the empirical (data) distribution. . .
– . . . by equating theoretical moments with sample moments
– Do this until you have enough equations, and then solve them
• Example: if E(X̄) = θ, then the method of moments estimator of θ is X̄.
• General procedure (for r parameters):
1. X1 , . . . , Xn i.i.d. f (x | θ1 , . . . , θr ).
2. kth moment is E(X k )
1
Xik
P
3. kth sample moment is Mk = n

4. Set E(X k ) = Mk , for k = 1, . . . r and solve for (θ1 , . . . , θr ).

• Alternative: Can use the variance instead of the second moment (sometimes more convenient).

6
Remarks
• An intuitive approach to estimation
• Can work in situations where other approaches are too difficult
• Usually biased
• Usually not optimal (but may suffice)
• Note: some authors use a ‘bar’ (θ̄) or a ‘tilde’ (θ)
e to denote MM estimators rather than a ‘hat’ (θ̂). This helps
to distinguish different estimators when comparing them to each other.

Example: Geometric distribution

• Sampling from: X ∼ Geom(p)
• The first moment:
∞
X 1
E(X) = xp(1 − p)x−1 =
x=1
p

• The MM estimator is obtained by solving

1
X̄ =
p
which gives
1
pe =
X̄

Example: Normal distribution

• Sampling from: X ∼ N(µ, σ 2 )
• Population moments: E(X) = µ and E(X 2 ) = σ 2 + µ2
• Sample moments: M1 = X̄ and M2 = n1
P 2
Xi
• Equating them:
1X 2
X̄ = µ and Xi = σ 2 + µ2
n
Solving these gives:
n
1X
µ
e = X̄ e2 =
and σ (Xi − X̄)2
n i=1

Note:
• This not the usual sample variance!
n−1 2
e2 =
• σ n S
n−1 2
σ2 ) =
• This one is biased, E(e n σ 6= σ 2 .

Example: Gamma distribution

• Sampling from: X ∼ Gamma(α, θ)
• The pdf is:  
1 −x
f (x | α, θ) = xα−1 exp
Γ(α)θα θ

• Population moments: E(X) = αθ and var(X) = αθ2

1
• Sample moments: M = X̄ and S 2 = n−1 (Xi − X̄)2
P

7
Equating them:
X̄ = αθ and S 2 = αθ2
Solving these gives:
S2 X̄ 2
θe = and α
e=
X̄ S2
Note:
• This is an example of using S 2 instead of M2

4 Maximum likelihood estimation

Method of maximum likelihood (ML)
• Idea: find the ‘most likely’ explanation for the data
• More concretely: find parameter values that maximise the probability of the data

Example: Bernoulli distribution

• Sampling from: X ∼ Be(p)
• Data are 0’s and 1’s
• Then pmf is
f (x | p) = px (1 − p)1−x , x = 0, 1, 0≤p≤1

• Observe values x1 , . . . , xn of X1 , . . . , Xn (iid)

• The probability of the data (the random sample) is
n
Y n
Y
Pr(X1 = x1 , . . . , Xn = xn | p) = f (xi | p) = pxi (1 − p)1−xi
i=1 i=1
P P
xi
=p (1 − p)n− xi

• Regard the sample x1 , . . . , xn as known (since we have observed it) and regard the probability of the data as a
function of p.
• When written this way, this is called the likelihood of p:
L(p) = L(p | x1 , . . . , xn )
= Pr(X1 = x1 , . . . , Xn = xn | p)
P P
xi
=p (1 − p)n− xi

• Want to find the value of p that maximizes this likelihood.

• It often helps to find the value of θ that maximizes the log of the likelihood rather than the likelihood
• This is called the log-likelihood P P
xi
ln L(p) = ln p + ln(1 − p)n− xi

• The final answer (the maximising value of p) is the same, since the log of non-negative numbers is a one-to-one
function whose inverse is the exponential, so any value θ that maximises the log-likelihood also maximises the
likelihood.
Pn
• Putting x = i=1 xi so that x is the number of 1’s in the sample,
ln L(p) = x ln p + (n − x) ln(1 − p)

• Find the maximum of this log-likelihood with respect to p by differentiating and equating to zero,
∂ ln L(p) 1 −1
= x + (n − x) =0
∂p p 1−p

• This gives p = x/n

• Therefore, the maximum likelihood estimator is p̂ = X/n = X̄

8
 x = 50 x = 40 x = 80

−100

n = 100
log likelihood(p)

−200

−300

0.00 0.25 0.50 0.75 1.00

p
Figure 1: Log-likelihoods for Bernoulli trials with parameter p

Maximum likelihood: general procedure

• Random sample (iid): X1 , . . . , Xn
• Likelihood function with m parameters θ1 , . . . , θm and data x1 , . . . , xn is:
n
Y
L(θ1 , . . . , θm ) = f (xi | θ1 , . . . , θm )
i=1

• If X is discrete, for f use the pmf

• If X is continuous, for f use the pdf
• The maximum likelihood estimates (MLEs) or the maximum likelihood estimators (MLEs) θ̂1 , . . . , θ̂m are values
that maximize L(θ1 , . . . , θm ).
• Note: same abbreviation and notation for both the estimators (random variable) and the estimates (realised
values).
• Often (but not always) useful to take logs and then differentiate and equate derivatives to zero to find MLE’s.
• Sometimes this is too hard, but we can maximise numerically. No closed-form expression in this case.

Example: Exponential distribution

Sampling (iid) from: X ∼ Exp(λ)

1 −x/λ
f (x | λ) = e , x > 0, 0<λ<∞
λ
 Pn 
1 − i=1 xi
L(λ) = n exp
λ λ
n
1X
ln L(λ) = −n ln(λ) − xi
λ i=1

9
 P
∂ ln L(λ) n xi
=− + 2 =0
∂λ λ λ
This gives: λ̂ = X̄

Example: Exponential distribution (simulated)

> x <- rexp(25) # simulate 25 observations from Exp(1)
> x
[1] 0.009669867 3.842141708 0.394267770 0.098725403
[5] 0.386704987 0.024086824 0.274132718 0.872771164
[9] 0.950139285 0.022927997 1.538592014 0.837613769
[13] 0.634363088 0.494441270 1.789416017 0.503498224
[17] 0.000482703 1.617899321 0.336797648 0.312564298
[21] 0.702562098 0.265119483 3.825238461 0.238687987
[25] 1.752657238
> mean(x) # maximum likelihood estimate
[1] 0.8690201

Log−likelihood curve
−10
−20

●
−30
log(L)

−40
−50
−60

0.5 1.0 1.5 2.0 2.5

What if we repeat the sampling process several times?

Log−likelihood curves
−10

●
●
−20

●●
●●●
●
●●
●●
−30

●●
●
log(L)

−40
−50
−60

0.5 1.0 1.5 2.0 2.5

10
Example: Geometric distribution

Sampling (iid) from: X ∼ Geom(p)

n
Y P
L(p) = p(1 − p)xi −1 = pn (1 − p) xi −n
, 0≤p≤1
i=1
Pn
∂ ln L(p) n xi − n
i=1
= − =0
∂p p 1−p
This gives: p̂ = 1/X̄

Example: Normal distribution

Sampling (iid) from: X ∼ N(θ1 , θ2 )

n
(xi − θ1 )2
 
Y 1
L(θ1 , θ2 ) = √ exp −
i=1
2πθ2 2θ2

n
n 1 X
ln L(θ1 , θ2 ) = − ln(2πθ2 ) − (xi − θ1 )2
2 2θ2 i=1
Take partial derivatives with respect to θ1 and θ2 .

n
∂ ln L(θ1 , θ2 ) 1 X
= (xi − θ1 )
∂θ1 θ2 i=1
n
∂ ln L(θ1 , θ2 ) n 1 X
=− + 2 (xi − θ1 )2
∂θ2 2θ2 2θ2 i=1
Pn
Set both of these to zero and solve. This gives: θ1 = x̄ and θ̂2 = n−1 i=1 (xi − x̄)2 . The maximum likelihood
estimators are therefore:
n
1X n−1 2
θb1 = X̄, θb2 = (Xi − X̄)2 = S
n i=1 n

Note: θb2 is biased.

Stress and cancer: VEGFC

> x <- c(0.97, 0.52, 0.73, 0.96, 1.26)
> n <- length(x)
> mean(x) # MLE for population mean
[1] 0.888

> sd(x) * sqrt((n - 1) / n) # MLE for the pop. st. dev.

[1] 0.2492709

> qqnorm(x) # Draw a QQ plot

> qqline(x) # Fit line to QQ plot

11
 Normal Q−Q Plot

0.6
●
0.1
1.2

0.5
0.2

0.3
1.0
Sample Quantiles

0.4
●
● 0.4

0.5

σ
0.6

0.7

0.3
0.8

0.8

0.2
0.6

0.1
−1.0 −0.5 0.0 0.5 1.0 0.6 0.7 0.8 0.9 1.0 1.1 1.2

Theoretical Quantiles µ

Challenge problem (boundary problem)

Take a random sample of size n from the shifted exponential distribution, with pdf:

f (x | θ) = e−(x−θ) (x > θ)

Equivalently:
Xi ∼ θ + Exp(1)
Derive the MLE for θ. Is it biased? Can you create an unbiased estimator from it?

Invariance property

Suppose we know θ̂ but are actually interestd in φ = g(θ) rather than θ itself. Can we estimate φ?
Yes! It is simply φ̂ = g(θ̂).
This is known as the invariance property of the MLE. In other words, transformations don’t affect the value of the
MLE.
Consequence: MLEs are usually biased since expectations are not invariant under transformations.

Is the MLE a good estimator?

Some useful results:

• Asymptotically unbiased
• Asymptotically optimal variance (‘efficient’)
• Asymptotically normally distributed
The proofs of these rely on the CLT. More details of the mathematical theory will be covered towards the end of the
semester.

12