Outline of the course

• Review for Stat515

• Chapter 8 : Estimation

• Chapter 9 : Properties of Point Estimators

and Methods of Estimation

• Chapter 10 : Hypothesis Testing

• Types of Studies and data

• Chapter 11 : Linear Models and Estimation

by Least Squares

• Chapter 12 : Considerations in Designing

Experiments

• Chapter 13 : The Analysis of Variance

• Chapter 14 : Analysis of Categorical Data

1
 [Chapter 8. Estimation]

8.1 Introduction

8.2 The Bias and Mean Square Error of Point

Estimators

8.3 Some Common Unbiased Point Estima-

tors

8.4 Evaluating the Goodness of a Point Esti-

mator

8.5 Confidence Intervals

8.6 Large-Sample Confidence Intervals

8.7 Selecting the Sample size

8.8 Small-Sample Confidence Intervals for µ

and µ1 − µ2

8.9 Confidence Intervals for σ 2

2
8.1 Introduction

• Purpose of Statistics and estimation

- make inference about the population by us-

ing the information contained in a sample
taken from the population of our interest

- find a statistic for a unknown target pa-

rameter characterizing the population : es-
timate the value of a unknown target pa-
rameter characterizing the population
- obtain the sampling distribution of a statis-
tic in making statistical inferences

: (point/interval) estimation and testing

(example) estimate the proportion p of washing ma-

chines that might fail prior to the expiration of a
1-year guarantee time
(example) estimate the mean waiting time µ at a
supermarket checkout station.

(example) estimate the standard deviation of the

error of measurement σ of an electronic instrument

3
• Estimator
- Firing a revolver at a red target
: one shot does not tell us if he/she is a expert, but
many shots(say 100) shots might provide sufficient
amount of evidence
- (Point) Estimation
: estimator - revolver, estimate - a single shot, and
parameter of interest - a red target
(Example) suppose we are interested in the mean of
UMass female students’ height. After we measure
heights of 100 female students, we calculate sam-
ple mean, ȳ using a sample mean formula Ȳ . But
we can not evaluate the goodness of point estima-
tion procedure based on one single constant value,
ȳ. We would like to evaluate this point estimation
procedure after this procedure is used many times.
Thus, we do repeated sampling(i.e., obtain 1000
samples of size 100 from the population and calcu-
late sample mean from each sample, ȳ1 , ȳ2 , . . . , ȳ1000
using a sample mean formula Ȳ . Finally we con-
struct a frequency distribution of 1000 sample means
and see how closely the distribution clusters around
the true mean of UMass female students’ height.
A frequency distribution of 1000 sample means is
an approximation to the sampling distribution of Ȳ .

4
• Estimator

- Estimation in two different forms

i) Point estimation & ii) Interval estimation

(example) interested in estimating the unknown mean
waiting time µ at a supermarket checkout station
i) point estimation by a single number : an expert
considers 10 minutes as the estimate of µ.

ii) interval estimation by the two values enclosing

µ : µ will fall in between 6 and 14(i.e., [6, 14])

- (Definition 8.1) An estimator is a formula

that tells how to calculate the value of
an (point/interval) estimate based on the
measurements contained in a sample.

(example) Suppose one is interested in the mean

of the population, µ. Given n random variables,
Y1 , . . . , Yn from the population, the sample mean is
Ȳ = n1 ni=1 Yi (which is also a random variable). Af-
P

ter observing a particular value of a random vari-

able, Y1 = y1 , . . . , Yn = yn , then Ȳ = ȳ = n1 ni=1 yi .
P

5
 (example) Firing a revolver at a red target
: one shot does not tell us if he/she is an ex-
pert, but many shots(say 100) shots might provide
sufficient amount of evidence
: (estimator - revolver), (estimate - a single shot),
and (parameter of interest - a red target)
(example) Suppose we are interested in the mean of
UMass female students’ height, µ. So, we measure
heights of 100 female students and calculate sample
mean, ȳ using a sample mean formula Ȳ .
: but we can not evaluate the goodness of point
estimation procedure based on one single constant
value, ȳ. We would evaluate this point estimation
procedure after this procedure is used many times.
: Thus, we do repeated sampling(i.e., obtain 1000
samples of size 100 from the population) and calcu-
late sample mean from each sample, ȳ1 , ȳ2 , . . . , ȳ1000 .
: Finally we construct a frequency distribution of
1000 sample means and see how closely the dis-
tribution clusters around the true mean of UMass
female students’ height. Note a frequency distribu-
tion of 1000 sample means is an approximation to
the sampling distribution of Ȳ .

- Many different estimators for the same pop-

ulation parameter. Then we need to find
better estimators
1
Pn
(example) Ȳ = n i=1 Yi , sample mean is one pos-
sible point estimator of µ. How about the median
or mode of n random variables?

6
8.2 Bias and Mean Square Error of Point
Estimators
Four students(A,B, C, D) fire a revolver at a
target n times.

7
Suppose Y1, . . . , Yn constitute a random sample
from a population with a parameter θ of our in-
terest. Let θ̂ = θ̂(Y1, . . . , Yn)(random quantity)
be a point estimator for a parameter θ.

• Since θ̂ is a statistic for θ, it has its own sampling

distribution, say f (θ̂).
• Unbiasedness
(Def 8.2 and 8.3) The bias of θ̂ is given by B(θ̂) =
E(θ̂) − θ. If B(θ̂) = 0, θ̂ is an unbiased estimator of
θ. Otherwise, θ̂ is a biased estimator.
• Variance and standard error
(Def) The variance of the sampling distribution of θ̂
is given by σθ̂2 = V (θ̂) = E(θ̂ − E(θ̂))2 . The standard
deviation qof the sampling distribution of θ̂ is given
p
by σθ̂ = 2
σθ̂ = V (θ̂). We call σθ̂ the standard
error of θ̂.
• Mean square error(MSE)

(Def 8.4) The MSE of θ̂
is given by M SE(θ̂) =

E(θ̂ − θ)2 = V (θ̂) + B(θ̂)2 .

8
(Exercise) Suppose that Y1 , Y2 , Y3 denote a random
sample from an exponential distribution with a param-
eter θ. Consider the following four estimators of θ :
θ̂1 = Y1 , θ̂2 = (Y1 + Y2 )/2, θ̂3 = (Y1 + 2Y2 )/3 and θ4 = Ȳ .

a. Which of these estimators are unbiased?

b. Among the unbiased estimators, which has the small-
est variance?

(Exercise) Suppose Y has a binomial distribution with

parameters n and p(i.e., Y ∼ b(n, p)).
a. Show that p̂1 = Y /n is an unbiased estimator of p.
b. Consider another estimator, p̂2 = (Y + 1)/(n + 2).
Then find the bias of p̂2 .
c. Derive M SE(p̂1 ) and M SE(p̂2 ).

9
8.3 Some Common Unbiased Point Es-
timators
• Methods for point estimators : Chapter 9.
• Common unbiased θ̂ (Table 8.1)
θ Sample θ̂ E(θ̂) σθ̂
size √
1) µ n Ȳ µ σ/
q n
Y pq
2) p n n
p n
q
σ12 σ22
3) µ1 − µ2 n1 , n2 Ȳ1 − Ȳ2 µ1 − µ2 +
q n1 n2
Y1 Y2 p1 q1 p2 q2
4) p1 − p2 n1 , n2 n1
− n2
p1 − p2 n1
+ n2

1) For n random samples, Y1 , . . . , Yn with E(Yi ) =

µ(population mean) and V (Yi ) = σ 2 , Ȳ for µ
Y
2) For Y ∼ b(n, p), sample proportion n
for p
3) For n1 random samples Y11 , . . . , Y1n1 with E(Y1i ) =
µ1 and V (Y1i ) = σ12 , and n2 random samples
Y21 , . . . , Y2n2 with E(Y2j ) = µ2 and V (Y2j ) = σ22
where i = 1, . . . , n1 and j = 1, . . . , n2 , Ȳ1 − Ȳ2 for
µ1 − µ2 .
4) For Y1 ∼ b(n, p1 ) and Y2 ∼ b(n, p2 ), difference in
the sample proportions nY11 − nY22 for p1 − p2 .

(note) the two samples from two populations in

3) and 4) are independent.

10
• Comments for estimators in Table 8.1
– 1) and 3) have valid E(θ̂) and σθ̂ re-
gardless of the form of the population
distribution, p(y) or f (y)
– 1) - 4) have sampling distributions, ei-
ther p(θ̂) or f (θ̂) that are approximately
normal for large samples (by C.L.T in
Chapter 7.3)
– 1) - 4) are unbiased with near-normal(or
bell-shaped) sampling distributions for
moderate-sized samples

• For n random samples Y1, . . . , Yn with E(Yi) =

µ and V (Yi) = σ 2, an estimator for the pop-
ulation variance, σ 2 is (see example 8.1)
1 Pn
2
– S = n−1 i=1(Yi − Ȳ )2 : unbiased
02 1 Pn
– S = n i=1(Yi − Ȳ )2 : biased

• Goodness of a point estimator : how much

faith can one place in the validity of statis-
tical inference? (See Chapter 8.4)

11
8.4 Evaluating the Goodness of a Point
Estimator
• Reasonable measure of the goodness of θ̂
for θ :

(Def 8.5) The error of estimation  is  =|

θ̂ − θ |.

- hopes  to be as small as possible

- varies randomly in repeated sampling

-  is a random quantity, as θ̂ = θ̂(Y1, . . . , Yn)

is a also random variable.

• Probability statements about  : P ( < b)

- Suppose θ̂ satisfies the following proper-

ties, i) E(θ̂) = θ and ii) its sampling distri-
bution is symmetric at θ.

Then, P ( =| θ̂ − θ |< b) = P (θ − b < θ̂ <

θ + b) ≈ the fraction of times, in repeated
sampling, that θ̂ falls within b units of θ
for small b(probabilistic bound on ).

12
 • Calculation of b so that P ( < b) = .90

- If we know the probability

R θ+b distribution,
f (θ̂) of θ̂, b satisfies θ−b f (θ̂)dθ̂ = .90.

- If we do not know f (θ̂), can we obtain

b? We can find an approximate bound on
 using Tchebysheff’s theorem.

: suppose θ̂ is unbiased. Then P (| θ̂ − θ |<

kσθ̂ ) ≥ 1 − 1/k2 for b = kσθ̂ and k ≥ 1.

: for k = 2, P ( < b = 2σθ̂ ) ≥ .75, which is

very conservative.

: P ( < b = 2σθ̂ ) is near .95 in many

situations (see Table 8.2).

: b = 2σθ̂ is a good approximate bound on

.

(Example 8.2)

(Example 8.3)

13
8.5 Confidence interval
• Interval estimator, [θ̂L, θ̂U ] : a procedure
calculating an interval of probable values
of an unknown population parameter, θ by
using the sample measurements

- indicate the reliability of an estimate, as

it represents a range of values around an
estimate that include θ(with a certain prob-
ability in repeated sampling)

- its length and location are random quan-

tities, as one or both of endpoints in the
interval vary randomly from sample to sam-
ple (i.e., θ̂L = θ̂L(Y1, . . . , Yn), θ̂U = θ̂U (Y1, . . . , Yn))

- one can not be certain that θ will fall be-

tween the endpoints of any single interval
calculated from a single sample

14
 - [θ̂L, θ̂U ] are written with a percentage;
what does this percentage represent?

(example) Suppose the researcher takes 100

random samples of size 1000(repeated sam-
pling from the population of interest) and
constructs an interval for each sample with
a 95% confidence. Then, she expects that
for 95 of the 100 samples (95%), the range
of values produced by the interval proce-
dure will include θ. Although she in prac-
tice only has a single sample, she is confi-
dent that their interval contains θ due to
the way to calculate the interval.

- interval estimators: confidence intervals

• Goal : construct a confidence interval, [θ̂L, θ̂U ]

that can generate narrow intervals having
a high probability of enclosing θ(under re-
peated sampling).

15
• Two-sided confidence interval, [θ̂L, θ̂U ]
- θ̂L and θ̂U : random lower and upper end-
point (i.e., confidence limit)
- 1 − α = P (θ̂L ≤ θ ≤ θ̂U ), confidence
coefficient : fraction of the time, in re-
peated sampling, that the constructed in-
tervals will contain θ
[Meaning] The probability that an interval
based on one sample of size n would con-
tain θ is zero or one.
If the same procedure were implemented
many times, each individual interval would
either contain or fail to contain θ, but the
percentage of the intervals that capture θ
would be close to 100(1 − α)%.
- high 1−α : highly confident that any con-
fidence interval from a single sample will
contain θ.
- prefer narrower confidence intervals with
the same confidence coefficient, (1 − α).

16
 • Lower one-sided confidence interval, [θ̂L, ∞)
- P (θ̂L ≤ θ) = 1 − α
• Upper one-sided confidence interval, (−∞, θ̂U ]
- P (θ ≤ θ̂U ) = 1 − α
• How to find confidence intervals?
- Use pivotal method : need to find a piv-
otal quantity having two characteristics
i) it is a function of Y1, . . . , Yn and un-
known θ where θ is the only unknown quan-
tity.
ii) its probability distribution does not de-
pend on θ
- Logic of the pivotal method: for a r.v.
Y , suppose that the probability distribution of the
pivotal quantity is known. Then, P (a ≤ Y ≤ b) =
P (c(a + d) ≤ a(Y + d) ≤ c(b + d))

(Example 8.4)

(Example 8.5)
17
8.6 Large-sample confidence intervals
• Pivotal method to develop confidence in-
tervals for θ when sample size is large

: Approximate probability distribution of

θ̂−θ is a standard normal distribution, N (0, 1)
σ
θ̂
as long as sample size is large and E(θ̂) = θ.
Note that this holds for four unbiased esti-
mators in Table 8.1.

: Suppose a statistic, θ̂ ∼ N (θ, σθ̂ ). Then a

100(1 − α)% two-sided confidence interval
for θ is [θ̂L, θ̂U ] = [θ̂ − zα/2σθ̂ , θ̂ + zα/2σθ̂ ].

Why? Since, θ̂−θ

σθ̂ ∼ N (0, 1),
θ̂ − θ
1 − α = P (−zα/2 ≤ ≤ zα/2)
σθ̂
= P (−zα/2σθ̂ ≤ θ̂ − θ ≤ zα/2σθ̂ )
= P (θ̂ − zα/2σθ̂ ≤ θ ≤ θ̂ + zα/2σθ̂ )

How about a 100(1 − α)% one-sided inter-

val for θ?

18
 : Unknown parameters in σ 2 from Table
θ̂
8.1 might be replaced with appropriate statis-
tics as long as n is large. But the calculated
confidence interval will have approximately
the stated confidence coefficient.

For θ = µ, θ̂ = Ȳ and σ 2 = σ 2/n. Use

θ̂
sample variance, s for unknown σ 2.
2
2
p
For θ = p, θ̂ = p̂ and σ = pq/n. Use p̂
θ̂
for unknown p.

(Example 8.7)
(Example 8.8)

(Exercise 8.56) In a Gallup Poll of n = 800 randomly

chosen adults, 45% indicated that movies were getting
better whereas 43% indicated that movies were getting
worse.
(a) Find a 98% confidence interval for p, the overall
proportion of adults who say that movies are getting
better?

19
8.7 Selecting the sample size
• Method of choosing n for all the large-
sample estimation procedure: for θ̂−θ
σ ·∼N (0, 1),
θ̂

1) The following information should be given

by the experimenter : a desired bound, B
on the error of estimation, and an associ-
ated confidence level, 1 − α.
2) 1 − α = P (| θ̂ − θ |≤ zα/2σθ̂ ) means α =
P (| θ̂ − θ |> zα/2σθ̂ )

3) Calculate (approximate) n by equating

zα/2σθ̂ = B where satisfying P (Z > zα/2) =
α/2 where Z ∼ N (0, 1)
[Tip] The range of a set of measurement = the
difference between the largest and smallest values.
If the distribution of measurements is approximately
normal, the standard deviation of measurements,
σ = range/4.

20
(Example) suppose that one wants to estimate the
average daily yield µ of a chemical. If one wishes
the error of estimation to be less than 5 tons with
probability .95, how large n should be? Assume
also that the range of the daily yields is known to
be approximately 84 tons.

(Example 8.9) The reaction of an individual to a

stimulus in a psychological experiment may take one
of two forms, A and B, If an experiment wishes to
estimate the probability p that a person will react
in manner A, how many people must be included in
the experiment? Assume that the experimenter will
be satisfied if the error of estimation is less than .04
with probability equal to .90. Assume also that he
expects p to be close to .6.

(Example 8.10)

21
8.8 Small-sample confidence intervals
for µ, and µ1 − µ2
[Case 1] Given n random samples, Y1, . . . , Yn ∼
N (µ, σ 2), unknown σ 2, and small n, how to
construct a confidence interval for µ?.

[Case 2] Given n1 random samples, Y11, . . . , Y1n1 ∼

N (µ1, σ 2), n2 random samples, Y21, . . . , Y2n2 ∼
N (µ2, σ 2), unknown σ 2, and small n1 and
n2, how to obtain a confidence interval for
µ1 − µ2(assume that they are independent
samples).

The large sample procedure in Chapter 8.8

might not be suitable. How to obtain con-
fidence intervals for µ, and µ1 − µ2?

→ Use a t-distribution with parameter ν(called

degree of freedom), T = √ Z ∼ t(ν − 1)
W/ν
where Z ∼ N (0, 1), W ∼ χ2(ν), and Z and
W are independnt (see Def 7.2).

→ T is a pivotal quantity and P (−tα/2 ≤

T ≤ tα/2) = 1 − α.
22
[Case 1] Given n random samples, Y1, . . . , Yn ∼
N (µ, σ 2), unknown σ 2, and small n,
Ȳ −µ (n−1)S 2
- σ/√n ∼ N (0, 1), 2 ∼ χ2(n − 1), and
σ
Ȳ and S 2 are independent(Thm 7.1, 7.3)
Ȳ −µ
√
σ/ n Ȳ −µ
√ ∼ t(n − 1)
- T = s  = S/ n
(n−1)S 2
(n−1)
σ2
√
where S = S 2, and then the 100(1 − α)%
two-sided confidence interval for µ is
√ √
[Ȳ − tα/2(S/ n), Ȳ + tα/2(S/ n)].

Why? 1 − α = P (−tα/2 ≤ T ≤ tα/2 ) = P (Ȳ −

√ √
tα/2 (S/ n) ≤ µ ≤ Ȳ + tα/2 (S/ n))

(Example 8.11) suppose a manufacturer of gun-

powder has developed a new powder, which was
tested in eight shells, and measured the muzzle ve-
locity, Yi , i = 1, . . . , 8. Assume that Yi were normally
distributed, its variance σ 2 was unknown, and their
sample mean and sample variance were ȳ = 2959
and s2 = 39.12 . Find a 95% confidence interval for
the true average velocity µ for shells of this type.
23
[Case 2] Given two sets of random samples,
Y11, . . . , Y1n1 ∼ N (µ1, σ 2), Y21, . . . , Y2n2 ∼
N (µ2, σ 2), unknown σ 2, and small n1 and
n2,
(Y¯1 −Y¯2 )−(µ1 −µ2 )
√ 2
σ /n1 +σ 2 /n2 ¯ Y¯2 )−(µ1 −µ2 )
- T = r = (Y1−
√ ∼
(n1 +n2 −2)Sp2 Sp 1/n1 +1/n2
σ 2 (n1 +n2 −2)
2 (n1 −1)S12 +(n2 −1)S22
t(n1+n2−2) where Sp = n1 +n2 −2 ,

- The 100(1 − α)% two-sided confidence

interval for µ is

p
(Y¯1 − Y¯2) − tα/2Sp 1/n1 + 1/n2,

p
(Y¯1 − Y¯2) + tα/2Sp 1/n1 + 1/n2

(Example 8.12)

(Exercise 8.83)

24
8.9 Confidence intervals for σ 2
• Population variance σ 2 : amount of vari-
ability in the population

• An unbiased estimator for unknown σ 2 :

1 Pn
2
S = n−1 i=1(Yi − Ȳ )2 (example 8.1)

• Confidence interval for σ 2 using the pivotal

method

: Suppose a random sample, Y1, . . . , Yn ∼

N (µ, σ 2) where µ and σ 2 are unknown. Since
(n−1)S 2
2 ∼ χ2(n−1)(Thm 7.3), one can find
σ
2 , σ2 ]
a 100(1 − α)% two-sided interval, [σL U
such that

(n − 1)S 2 
2
1 − α = P χ1−(α/2) ≤ ≤ χ 2
σ2 (α/2)
 
(n − 1)S 2 (n − 1)S 2
=P ≤ σ 2 ≤ 
χ2
(α/2)
χ 2
1−(α/2)
h i
≡ P σL2 ≤ σ2 ≤ σ2
U

25
• How about a 100(1 − α)% one-sided inter-
val for σ 2?

(Example 8.13) An experimenter wanted

to check the variability of measurements
obtained by using equipment designed to
measure the volume of an audio source.
Three independent measurements recorded
by this equipment for the same sound were
4.1, 5.2 and 10.2. Estimate σ 2 with confi-
dence coefficient .90.

26