CHAPTER 2

Parameter Estimation

2.1 Point estimation

When sampling from a population described by a probability distribution function,

f (x; θ), then knowing the parameter θ yields all the necessary knowledge of the entire
population. Therefore, it is natural to find a good estimator of the point θ in case is
unknown. That is, a good point estimator.

Definition 2.1.1. A point estimator is any function T (X1 , X2 , . . . , Xn ) of a sample.

That is, any statistic is a point estimator.

In many cases, there are natural candidate for particular parameters. For example,
X̄ for µ and p̂ = X/n for the true proportion of a binomial experiment. However,
there are complex cases where we need some specialised techniques to give us reason-
able candidate estimators for consideration. There are four main methods of finding
estimators, though these won’t be elaborated in this course.

i. Methods of moments

2-1
2.1. Point estimation 2. Parameter Estimation

ii. Maximum likelihood estimation

iii. Bayes estimators

iv. The EM algorithm

All these techniques provide reasonable estimators which can be different. Therefore,
the next question would then be how do we choose one estimator as the best for a
given parameter? Well, there is no obvious answer to this question. However, there is
list of desirable attributes that determines the quality of a good estimator.

Properties of a good estimator

1. Unbiasedness: A statistic, θ̂, is said to be an unbiased estimator of the param-

eter, θ, if

E(θ̂) = θ (2.1)

Example 2.1.1. Show that S 2 is an unbiased estimator of the parameter σ 2 .

Pn
Solution 2.1.1. Recall from Section 1.2.4 that we can write i=1 (Xi − X̄)2 =
Pn 2 2
i=1 (Xi − µ) − n(X̄ − µ) . Therefore,

n
!
1 X
E(S 2 ) = E (Xi − µ)2 − nE (X̄ − µ)2
   
n−1 i=1

2-2
2. Parameter Estimation 2.1. Point estimation

By the definition of variance, i.e V(X) = E[(X − u)2 ] and the sampling distribu-
tion of X̄ being N (µ, σ 2 /n), we get

n
!
1 X σ2
= V (Xi ) − n
n−1 i=1
n
n
!
1 X
= σ2 − σ2
n−1 i=1
1
= (n − 1)σ 2 = σ 2 .
n−1

2. Efficiency: Suppose θ̂1 and θ̂2 are two unbiased estimators of the parameter, θ.
Then we would want to choose an estimator which does not vary a lot; that is,
an estimator with smaller variance. We say θ̂1 is efficient than θ̂2 if

V(θ̂1 ) < V(θ̂2 ) (2.2)

Example 2.1.2. Suppose a sample of size 3 is taken from a Normal distribution

with mean µ and variance σ 2 . Consider the following estimators
X1 + X2 + 3X3 X1 + 2X2 + X3
θ̂1 = and θ̂2 = for µ.
5 4

(a) Discuss whether the two estimators are biased or not.

(b) Find the relative efficiency of θ̂1 compared to θ̂2 .

3. Sufficiency: An estimator is said to be sufficient if it uses all the information

about the population parameter that the sample provides.

4. Consistency: An estimator is said to be consistent if when the sample size

approach the population size, its estimates should approach the population pa-
rameter.

2-3
2.2. Interval estimation 2. Parameter Estimation

Please note the distinction between an estimator and an estimate. An estimate

is the realised value of the estimator. For example, X̄ is an estimator while its
realised value from a given random sample, x̄ is the estimate.

2.2 Interval estimation

It is very important to remember that point estimates are almost never equal to the
true value of the parameter. In order for the point estimate to be useful, it should
reported with their standard deviation (uncertainty) which measures how precise the
point estimator is. In this section, we show how to obtain more information about its
precision by computing its confidence intervals. The confidence intervals provide an
interval within which we would expect to find the value of the parameter. Such an
interval is called an interval estimate.

An interval estimate of parameter θ is an interval of the form θ̂L < θ < θ̂U , where θ̂L
and θ̂U depend on the value of the statistic θ̂ from a particular sample and its sampling
distribution. From the sampling distribution of θ̂ we shall be able to determine θ̂L and
θ̂U such that

P(θ̂L < θ < θ̂U ) = 1 − α (2.3)

for 0 < α < 1, so that we have a probability of 1 − α of selecting a random sample that
will produce an interval containing θ. The interval θ̂L < θ < θ̂U is called a 100(1 − α)%
confidence interval, α is called the significance level and the endpoints, θ̂L and
θ̂U , are called the lower and upper confidence limits.

2-4
2. Parameter Estimation 2.2. Interval estimation

2.2.1 Confidence intervals for a population mean

Let us now consider the interval estimate of µ. If our sample is selected from a normal
population or, failing this, if n is sufficiently large, we can establish a confidence interval
for µ by considering the sampling distribution of X̄. According to the CLT, X̄ ∼
N (µ, σ 2 /n) and for standard normal variates, we have that

P(−zα/2 < Z < zα/2 ) = 1 − α

x̄ − µ
Since √ is standard normal then we have
σ/ n

 
X̄ − µ
P −zα/2 < √ < zα/2 = 1 − α
σ/ n

After simple algebraic simplifications we get

√ √

P X̄ − zα/2 σ/ n < µ < X̄ + zα/2 σ/ n = 1 − α

Hence we have the following result.

Let x̄ be a mean of a random sample of size n from a population with known
variance σ 2 . The a 100(1 − α)% confidence interval for µ is given by

√ √
x̄ − zα/2 σ/ n < µ < x̄ + zα/2 σ/ n (2.4)

where zα/2 is the z-value leaving an area of α/2 to the right.

For small samples selected from non-normal populations, we cannot expect our
degree of confidence to be accurate. However, for samples of size n ≥ 30, with the

2-5
2.2. Interval estimation 2. Parameter Estimation

shape of the distributions not too skewed, sampling theory guarantees good results.

Example 2.2.1. The average zinc concentration recovered from a sample of measure-
ments taken in 36 different locations in a river is found to be 2.6 grams per milliliter.
Find the 95% and 99% confidence intervals for the mean zinc concentration in the river.
Assume that the population standard deviation is 0.3 gram per milliliter.

Variance unknown

It is often recommended that even when normality cannot be assumed, variance is

unknown, and n ≥ 30, then the sample variance can replace the population variance
and the confidence interval remains as in (2.4) but with s replacing σ. This is referred
to as a large-sample confidence interval. The justification lies on the CLT. However, if
the sample size is small (n < 30), then we have to use the sampling distributions of X̄
and S 2 .

Theorem 2.2.1. Let X1 , X2 , . . . , Xn be independent random variables that

are all normal with mean µ and standard deviation σ. Let

n n
1X 2 1 X
2
X̄ = Xi and S = Xi − X̄ .
n i=1 n − 1 i=1

Then the random variable

X̄ − µ
T = √ (2.5)
S/ n

has a t-distribution with n − 1 degrees of freedom.

The random variable T can be used to construct a confidence interval on µ. The

2-6
2. Parameter Estimation 2.2. Interval estimation

procedure is the same as that with σ known except that here is replaced by S and the
standard normal distribution is replaced by the t-distribution.
Let x̄ and s be the mean and standard deviation of a random sample from a
normal population with unknown variance σ 2 , respectively. Then a
100(1 − α)% confidence interval for µ is

s
x̄ ± tα/2 √ (2.6)
n

where tα/2 is the t-value with n − 1 degrees of freedom leaving an area of α/2
to the right.

Example 2.2.2. The contents of seven similar containers of sulfuric acid are 9.8, 10.2,
10.4, 9.8, 10.0, 10.2, and 9.6 liters. Find a 95% confidence interval for the mean contents
of all such containers, assuming an approximately normal distribution

Determining the sample size needed for a confidence interval

of specified width

It follows from (2.4) that the width of the confidence interval for a population mean
based on a sample of size n drawn from a population with standard deviation, σ, is
√
±zα/2 σ/ n. If the confidence level is specified, we can look up the value zα/2 and the
standard deviation is specified, then we can compute the value of n needed to produce
a specified width.

If x̄ is used as an estimate of µ, we can be 100(1 − α)% confident that the error will

2-7
2.2. Interval estimation 2. Parameter Estimation

not exceed a specified amount, e when the sample size is

z
α/2 σ
2
n= (2.7)
e

Example 2.2.3. The sugar content in a one-cup serving of a certain breakfast cereal
was measured for a sample of 140 servings. The average was 11.9g and the standard
deviation was 1.1g

(a) Find a 99% confidence interval for the mean sugar content.

(b) What is the confidence interval of the interval (11.81, 11.99)?

(c) How large a sample is needed so that a 95% confidence interval specifies the mean
to within ±0.1?

2.2.2 Confidence intervals for a proportions

Let X be the number of successes in n independent Bernoulli trials with

success probability p, so that X ∼ Bin(n, p). Then a level 100(1 − α)%
confidence interval for p id given

r
p̂(1 − p̂)
p̂ ± zα/2 (2.8)
n

where zα/2 is the z-value leaving an area of α/2 to the right.

Example 2.2.4. A soft-drink manufacturer purchases aluminum cans from an outside

vendor. A random sample of 70 cans is selected from a large shipment, and each is
tested for strength by applying an increasing load to the side of the can until punctures.
Of the 70 cans, 52 meet the specification for puncture resistance.

2-8
2. Parameter Estimation 2.2. Interval estimation

(a) Find a 95% confidence interval for the proportion of cans in the shipment that
meet the specification.

(b) Find the sample size needed for a 90% confidence interval to specify the propor-
tion to within ±0.05.

2.2.3 Confidence intervals for the difference between two means

Let X1 , X2 , . . . , Xn be a large random sample of size n from a population

with mean µX and standard deviation σX , and let Y1 , Y2 , . . . , Ym be a large
random sample of size m from a population with mean µY and standard
deviation σY . If the two samples are independent, then a 100(1 − α)%
confidence level for µX − µY is

r
2
σX σ2
X̄ − Ȳ ± zα/2 + Y (2.9)
n m

when the values of σX and σY are unknown, they can be replaced with the
sample standard deviations, sX and sY .

Example 2.2.5. A random sample of size n1 = 25, taken from a normal population
with a standard deviation σ1 = 5, has a mean x̄ = 80. A second random sample of size
n2 = 36, taken from a different normal population with a standard deviation σ2 = 3,
has a mean x̄2 = 75. Find a 95% confidence interval for µ1 − µ2 .

Example 2.2.6. A study was conducted to determine if a certain treatment has any
effect on the amount of metal removed in a pickling operation. A random sample of
100 pieces was immersed in a bath for 24 hours without the treatment, yielding an
average of 12.2 millimeters of metal removed and a sample standard deviation of 1.1

2-9
2.2. Interval estimation 2. Parameter Estimation

millimeters. A second sample of 200 pieces was exposed to the treatment, followed by
the 24-hour immersion in the bath, resulting in an average removal of 9.1 millimeters of
metal with a sample standard deviation of 0.9 millimeter. Compute a 99% confidence
interval estimate for the difference between the population means. Does the treatment
appear to reduce the mean amount of metal removed?

Variance unknown but equal

Consider the case where σ12 and σ22 are unknown. If σ12 and σ22 are equal, we obtain a
standard normal variable of the form

(X̄1 − X̄2 ) − (µ1 − µ2 )

Z= r   (2.10)
2 1 1
σ n1 + n2

We have also seen from Chapter 1 that the two random variables

(n1 − 1)S12 (n2 − 1)S22

and
σ2 σ2

have χ2 distributions with n1 − 1 and n2 − 1 degrees of freedom respectively. Since

these two are independent of each, their sum

(n1 − 1)S12 + (n2 − 1)S22

V = (2.11)
σ2

has a χ2 distribution with n1 + n2 − 2 degrees of freedom.

In addition, Z is standard normal variable and V is χ2 variable and also independent

2-10
2. Parameter Estimation 2.2. Interval estimation

of each other, it follows that the statistic

p
T = Z/ V /(n1 + n2 − 2) (2.12)

where Z and V are given by (2.10) and (2.11), has t-distribution with n1 + n2 − 2
degrees of freedom.

Definition 2.2.1. A point estimate of the unknown common variance σ 2 can be ob-
tained by pooling the sample variances defined as follows

(n1 − 1)S12 + (n2 − 1)S22

Sp2 = (2.13)
n1 + n2 − 2

We can now substitute for the unknown variance with Sp2 then use the T statistic
in (2.12) to construct the 100(1 − α)% confidence interval for the difference of means
when the variance is unknown.

Let x̄1 and x̄2 be the means of two independent random samples of sizes n1
and n2 respectively, from approximately normal populations with unknown
but equal variances. Then a 100(1 − α)% confidence interval for µ1 − µ2 is
given by

r
1 1
(x̄1 − x̄2 ) ± tα/2 sp + (2.14)
n1 n2

where s2p is the pooled variance estimate and tα/2 is the t-value with
n1 + n2 − 2 degrees of freedom leaving an area of α/2 to the right.

Example 2.2.7. An experiment reported in Popular Science compared fuel economies

for two types of similarly equipped diesel mini-trucks. Let us suppose that 12 Volkswa-

2-11
2.2. Interval estimation 2. Parameter Estimation

gen and 10 Toyota trucks were tested in 90-kilometer-per-hour steady-paced trials. If

the 12 Volkswagen trucks averaged 16 kilometers per liter with a standard deviation of
1.0 kilometer per liter and the 10 Toyota trucks averaged 11 kilometers per liter with
a standard deviation of 0.8 kilometer per liter, construct a 90% confidence interval
for the difference between the average kilometers per liter for these two mini-trucks.
Assume that the distances per liter for the truck models are approximately normally
distributed with equal variances.

Paired observations

We shall consider estimation procedures for the difference of two means when the sam-
ples are not independent and the variances of the two populations are not necessarily
equal. The situation considered here deals with a very special experimental condi-
tion, namely that of paired observations. Unlike in the two independent samples, each
experimental unit has a pair of observations, one for each population.

For example, if we run a test on a new diet using 15 individuals, the weights be-
fore and after going on the diet form the information for our two samples. The two
populations are ‘before’ and ‘after’, and the experimental unit is the individual. To
determine if the diet is effective, we consider the differences d1 , d2 , . . . , dn in the paired
observations. These differences are the values of a random sample D1 , D2 , . . . , Dn from
a population of differences that we shall assume to be normally distributed with mean
2
µD = µ1 − µ2 and variance σD .

2-12
2. Parameter Estimation 2.2. Interval estimation

If d¯ and sd are the sample mean and sample standard deviation, respectively,
of the normally distributed differences of n random pairs of measurements, a
100(1 − α)% confidence interval for µD = µ1 − µ2 is

sd
d¯ ± tα/2 √ (2.15)
n

where tα/2 is the t-value with n − 1 degrees of freedom, leaving an area of

α/2 to the right.

Example 2.2.8. Fortune magazine (March 1997) reported the total returns to in-
vestors for the 10 years prior to 1996 and also for 1996 for 431 companies. The total
returns for 10 of the companies are listed below. Find a 95% confidence interval for
the mean change in percent return to investors.

Company 1986–96 1996

Coca-Cola 29.8% 43.3%

Mirage Resorts 27.9% 25.4%
Merck 22.1% 24.0%
Microsoft 44.5% 88.3%
Johnson & Johnson 22.2% 18.1%
Intel 43.8% 131.2%
Pfizer 21.7% 34.0%
Procter & Gamble 21.9% 32.1%
Berkshire Hathaway 28.3% 6.2%
S&P 500 11.8% 20.3%

2-13
2.2. Interval estimation 2. Parameter Estimation

2.2.4 Confidence intervals for the difference between two pro-

portions

Let p̂1 and p̂2 be the proportions of successes in random samples of sizes n1
and n2 respectively. Then the 100(1 − α)% confidence interval for the
difference of two binomial parameters, p1 − p2 , is given by

s
p̂1 (1 − p̂1 ) p̂2 (1 − p̂2 )
(p̂1 − p̂2 ) ± zα/2 + (2.16)
n1 n2

where zα/2 is the z-value leaving an area of α/2 to the right.

Example 2.2.9. A certain geneticist is interested in the proportion of males and

females in the population who have a minor blood disorder. In a random sample of
1000 males, 250 are found to be afflicted, whereas 275 of 1000 females tested appear to
have the disorder. Compute a 95% confidence interval for the difference between the
proportions of males and females who have the blood disorder.

Example 2.2.10. A clinical trial was conducted to determine if a certain type of

inoculation has an effect on the incidence of a certain disease. A sample of 1000 rats
was kept in a controlled environment for a period of 1 year, and 500 of the rats were
given the inoculation. In the group not inoculated, there were 120 incidences of the
disease, while 98 of the rats in the inoculated group contracted it. If p1 is the probability
of incidence of the disease in uninoculated rats and p2 the probability of incidence in
inoculated rats, compute a 90% confidence interval for p1 − p2 .

2-14