Applied Statistics and

Probability for
Engineers

Sixth Edition
Douglas C. Montgomery George C.

Chapter 7 Runger

Point Estimation of Parameters and Sampling

Distributions
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 7
Point Estimation
of Parameters
and Sampling
CHAPTER OUTLINE
Distributions
7-1 Point Estimation a Point Estimate
7-2 Sampling Distributions 7-3.4 Mean Squared Error
and of an
the Central Limit Estimator
Theorem 7-4 Methods of Point
7-3 General Concepts of Estimation
Point 7-4.1 Method of Moments
Estimation 7-4.2 Method of Maximum
7-3.1 Unbiased Estimators Likelihood
7-3.2 Variance of a Point 7-4.3 Bayesian Estimation
Estimator of
7-3.37 Title
Chapter Standard
and Outline Error: Parameters 2
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
Reporting
 Learning Objectives for Chapter 7

After careful study of this chapter, you should

be able to do the following:
1. General concepts of estimating the parameters of a
population or a probability distribution.
2. Important role of the normal distribution as a
sampling distribution.
3. The central limit theorem.
4. Important properties of point estimators, including
bias, variances, and mean square error.
5. Constructing point estimators using the method of
moments, and the method of maximum likelihood.
6. Compute and explain the precision with which a
parameter is estimated.
7. Constructing a point estimator using the Bayesian
approach.
Chapter 7 Learning Objectives 3
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Point Estimation
• A point estimate is a reasonable
value of a population parameter.
• X1, X2,…, Xn are random variables.
• Functions of these random
variables, x-bar and s2, are also
random variables called statistics.
• Statistics have their unique
distributions which are called
sampling distributions.
Sec 7-1 Point Estimation 4
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Point Estimator

As an example,suppose the random variable X is normally distributed with

an unknown mean μ. The sample mean is a point estimator of the unknown
population mean μ. That is, μ  X . After the sample has been selected,
the numerical value x is the point estimate of μ.
Thus if x1  25, x2  30, x3  29, and x4  31, the point estimate of μ is

25  30  29  31
x  28.75
4

Sec 7-1 Point Estimation 5

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Some Parameters & Their
Statistics
Parameter Measure Statistic
μ Mean of a single population x-bar
σ2 Variance of a single population s2
σ Standard deviation of a single population s
p Proportion of a single population p -hat
μ1 - μ2 Difference in means of two populations x bar1 - x bar2
p1 -p2 Difference in proportions of two populations p hat1 - p hat2

• There could be choices for the point estimator of a

parameter.
• To estimate the mean of a population, we could choose
the:
– Sample mean.
– Sample median.
– Average of the largest & smallest observations in
theEstimation
Sec 7-1 Point sample. 6
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Some Definitions
• The random variables X1, X2,…,Xn
are a random sample of size n if:
a) The Xi ‘s are independent random
variables.
b) Every Xi has the same probability
distribution.
• A statistic is any function of the
observations in a random
sample.
• The probability distribution of a
statistic is called a sampling
Sec 7-2 Sampling Distributions and the Central Limit The
orem
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
7
 Central Limit Theorem

Sec 7-2 Sampling Distributions and the Central Limit The 8

orem
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-2: Central Limit Theorem
Suppose that a random variable X has a
continuous 1 2, 4  x  6
f x  
uniform distribution: 0, otherwise

Find the distribution of the sample mean of a

random
sample
By the CLT of
the size n = X40.
distribution is normal .
ba 64
  5
2 2
b  a  6  4 
2 2

2   13
12 12
 2
13 1
 X2   
Figure 7-5 The
n 40 120
distribution of X and X
Sec 7-2 Sampling Distributions and the Central Limit The for Example 7-2. 9
orem
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Sampling Distribution of a Difference in
Sample Means
• If we have two independent populations with means
μ1 and μ2, and variances σ12 and σ22, and
• If X-bar1 and X-bar2 are the sample means of two
independent random samples of sizes n1 and n2
from these populations:
• Then the sampling distribution of:

is approximately standard normal, if the conditions

of the central limit theorem apply.
• If the two populations are normal, then the
sampling distribution of Z is exactly standard
normal.
Sec 7-2 Sampling Distributions and the Central Limit The 10
orem
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-3: Aircraft Engine
Life
The effective life of a component
used
in jet-turbine aircraft engine is a
random variable with mean 5000 and
SD 40 hours and is close to a normal Figure 7-6 The sampling
distribution. The engine distribution of X2 − X1 in
manufacturer Example 7-3.
introduces an improvement into the Process
Manufacturing process for this Old (1) New (2) Diff (2-1)
component that changes the x -bar = 5,000 5,050 50
parameters to 5050 and 30. Random s= 40 30
samples of size 16 and 25 are n= 16 25
selected. Calculations
s / √n = 10 6 11.7
What is the probability that the z= -2.14
difference in the two sample means P(xbar2-xbar1 > 25) = P(Z > z) = 0.9840
is
= 1 - NORMSDIST(z)
at least 25 hours?
Sec 7-2 Sampling Distributions and the Central Limit The 11
orem
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Unbiased Estimators
Defined

Sec 7-3.1 Unbiased Estimators 12

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-4: Sample Mean & Variance Are
Unbiased-1
• X is a random variable with mean μ and variance
σ2. Let X1, X2,…,Xn be a random sample of size n.
• Show that the sample mean (X-bar) is an
unbiased estimator of μ.

Sec 7-3.1 Unbiased Estimators 13

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-4: Sample Mean & Variance Are
Unbiased-2
Show that the sample variance (S2) is a unbiased estimator of σ2.

Sec 7-3.1 Unbiased Estimators 14

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Minimum Variance Unbiased Estimators

• If we consider all unbiased

estimators of θ, the one with the
smallest variance is called the
minimum variance unbiased
estimator (MVUE).
• If X1, X2,…, Xn is a random sample
of size n from a normal
distribution with mean μ and
variance σ2, then the sample X-bar
is the MVUE for μ.
Sec 7-3.2 Variance of a Point Estimate 15
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Standard Error of an
Estimator

Sec 7-3.3 Standard Error Reporting a Point Estimate 16

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-5: Thermal
Conductivity
• These observations are 10 xi
measurements of thermal 41.60
41.48
conductivity of Armco iron. 42.34
• Since σ is not known, we 41.95
41.86
use s to calculate the 42.18
standard error. 41.72
42.26
• Since the standard error is
41.81
0.2% of the mean, the mean 42.04
estimate is fairly precise. 41.924 = Mean
0.284 = Std dev (s )
We can be very confident 0.0898 = Std error
that the true population
mean is 41.924 ± 2(0.0898)
Sec 7-3.3 Standard Error Reporting a Point Estimate 17
or between Copyright ©41.744
2014 John Wileyand
& Sons, Inc. All rights reserved.
 Mean Squared Error

Conclusion: The mean squared error (MSE)

of
the estimator is equal to the variance of the
estimator plus the bias squared.
Sec 7-3.4 Mean Squared Error of an Estimator 18
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Relative Efficiency
• The MSE is an important criterion
for comparing two estimators.

• If the relative efficiency is less

than 1, we conclude that the 1st
estimator is superior than the 2nd
estimator.
Sec 7-3.4 Mean Squared Error of an Estimator 19
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Optimal Estimator
• A biased estimator
can be preferred
than an unbiased
estimator if it has a
smaller MSE.
• Biased estimators
Figure 7-8 A biased estimator
are occasionally that has a smaller
used in linear variance than the unbiased
estimator .
regression.
• An estimator whose
MSE is smaller than
that of any other
Sec estimator isof called
7-3.4 Mean Squared Error an Estimator 20
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Moments Defined
• Let X1, X2,…,Xn be a random sample from the
probability distribution f(x), where f(x) can be
either a:
– Discrete probability mass function, or
– Continuous probability density function
• The kth population moment (or distribution
moment) is E(Xk), k = 1, 2, ….
n
The k sample moment is 1 / n   X , k  1, 2,...
th
i
k

i 1

• If k = 1 (called the first moment), then:

– Population moment is μ.
– Sample moment is x-bar.
• The sample mean is the moment estimator of
the population mean.
Sec 7-4.1 Method of Moments 21
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Moment Estimators

Sec 7-4.1 Method of Moments 22

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-8: Normal Distribution Moment
Estimators
Suppose that X1, X2, …, Xn is a random sample
from a normal distribution with parameter μ
and σ2 where E(X) = μ and E(X2) = μ2 + σ2.
n n
1 1
  X   X i and  2   2   X i2
n i 1 n i 1
2
n
1  n

2 1 n  X i
2
 n n iX
 i 1 
   X i2  X 2  i 1
n i 1 n
  n
 
2
n

 X X
2
 n  Xi   i
1
   X i2   i 1    i 1
(biased)
n  i 1 n  n
 
 
Sec 7-4.1 Method of Moments 23
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-9: Gamma Distribution Moment
Estimators-1
Suppose that X1, X2, …, Xn is a random sample from a gamma
distribution with
parameter r and λ where E(X) = r/ λ and E(X2) = r(r+1)/ λ2 .
r
 E  X   X is the mean

r
2
 E  
X 2
 E  X 
2
is the variance or

r  r  1
 2
 E  X 2
 and now solving for r and  :

r  X2
n
1/ n   X i2  X 2
i 1

X
  n
1/ n   X i2  X 2
i 1

Sec 7-4.1 Method of Moments 24

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-9: Gamma Distribution Moment
Estimators-2
xi 2
xi
Using the time to failure data in 11.96 143.0416
the table. We can estimate the 5.03 25.3009
67.40 4542.7600
parameters of the gamma 16.07 258.2449
distribution. 31.50 992.2500
x-bar = 21.646 7.73 59.7529
11.10 123.2100
ΣX 2 = 6645.4247 22.38 500.8644

X2 21.646 2
r    1.29
n
1/ n   X i2  X 2 1 8 6645.4247  21.646 2

i 1

X 21.646
    0.0598
n
1/ n   X i2  X 2 1 8 6645.4247  21.646 2

i 1
Sec 7-4.1 Method of Moments 25
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Maximum Likelihood
Estimators
• Suppose that X is a random variable with
probability distribution f(x;θ), where θ is a
single unknown parameter. Let x1, x2, …, xn
be the observed values in a random sample
of size n. Then the likelihood function of the
sample is:

L(θ) = f(x1;θ) ∙ f(x2; θ) ∙…∙ f(xn; θ)

• Note that the likelihood function is now a

function of only the unknown parameter θ.
The maximum likelihood estimator (MLE) of
θ is the value of θ that maximizes the
likelihood function L(θ).
Sec 7-4.2 Method of Maximum Likelihood 26
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-10: Bernoulli Distribution MLE
Let X be a Bernoulli random variable. The probability mass
function is
f(x;p) = px(1-p)1-x, x = 0, 1 where P is the parameter to be
estimated.
 p   p x 1 of
p  a random
 p x 1  p  sample
 ...  p x 1 of
 p size n is:
1 x 1 x 1 x
The likelihoodLfunction 1 1 2 2 n n

n
n  xi n

 xi
  p xi 1  p  1  p 
1 xi n
 p i 1 i 1

i 1

 n   n

ln L  p     xi  ln p   n   xi  ln 1  p 
 i 1   i 1 
n
 n

d ln L  p  
xi 

n  i 1
xi 

 i 1

dp p 1  p 

Sec 7-4.2 Method of Maximum Likelihood 27

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
Example 7-11: Normal Distribution MLE for μ
Let X be a normal random variable with unknown mean μ and
known variance σ2. The likelihood function of a random sample of
size n is:
n
1  xi   2 2 2 
L     e
i 1  2
n
1
  xi   
2
1 2
 e 2 i 1

2 
2 n2

n 1 n
ln L     ln  2   2   xi   
2 2

2 2 i 1
d ln L    1 n
 2   xi   
d  i 1

Sec 7-4.2 Method of Maximum Likelihood 28

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
Example 7-12: Exponential Distribution MLE

Let X be a exponential random variable with

parameter λ.
The likelihood function    xi a random sample of
of
n
n
    e  xi   n e i1
sizeL n is:
i 1
n
ln L     n ln       xi
i 1

d ln L    n n
   xi
d  i 1

Equating the above to zero we get

n
  n x
i 1
i  1 X (same as moment estimator)

Sec 7-4.2 Method of Maximum Likelihood 29

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-13: Normal Distribution MLEs
for μ & σ2
Let X be a normal random variable with both unknown mean μ
and variance σ2. The likelihood function of a random sample of
size n is:
 xi     2 2 
n
1
L   ,   
2
2
e
i 1  2
n
1
  xi   
2
1 2
 e 2 i 1

 2  2 n2

n 1 n
ln L   ,    ln  2 2   2   xi   
2 2

2 2 i 1

 ln L   ,  2  1 n



2
x     0
i 1
i

 ln L   ,  2  n 1 n

  xi   
2
  0
  2
 2 2 2 4 i 1

  xi  X 
2

2
  X and   i 1

Sec 7-4.2 Method of Maximum Likelihood n 30

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Properties of an MLE

Notes:
• Mathematical statisticians will often prefer MLEs
because of
these properties. Properties (1) and (2) state that MLEs
are
MVUEs.
• To use MLEs, the distribution of the population must be
known
Sec 7-4.2 Method of Maximum Likelihood 31

or assumed. Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Invariance Property

his property is illustrated in Example 7-13.

Sec 7-4.2 Method of Maximum Likelihood 32

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-14: Invariance
For the normal distribution, the
MLEs were:

Sec 7-4.2 Method of Maximum Likelihood 33

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Complications of the MLE
Method
The method of maximum likelihood is an
excellent technique, however there are two
complications:
1. It may not be easy to maximize the
likelihood function because the derivative
function set to zero may be difficult to solve
algebraically.
2. It may not always be possible to use
calculus methods directly to determine the
maximum of L(ѳ).

The following example illustrate this.

Sec 7-4.2 Method of Maximum Likelihood 34
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-16: Gamma Distribution
MLE-1
Let X1, X2, …, Xn be a random sample from a gamma
distribution. The log of the likelihood function is:
 n  r xir 1e   x 
i

ln L  r ,    ln   
 i 1  r  
n n
 nr ln      r  1  ln  xi   n ln   r     xi
i 1 i 1

 ln L  r ,   n
 ' r 
 n ln      ln  xi   n
r i 1  r 
 ln L  r ,   nr n
   xi
  i 1
Equating the above derivative to zero we get

Sec 7-4.2 Method of Maximum Likelihood 35

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-16: Gamma Distribution
MLE-2

Figure 7-11 Log likelihood for the gamma distribution

using the failure time data. (a) Log likelihood surface.
(b) Contour plot.

Sec 7-4.2 Method of Maximum Likelihood 36

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
Bayesian Estimation of Parameters-1
• The moment and likelihood methods interpret
probabilities as relative frequencies and are
called objective frequencies.
• The random variable X has a probability
distribution of parameter θ called f(x|θ).
• Additional information about θ is that it can be
summarized as f(θ), the prior distribution, with
mean μ0 and variance σ02. Probabilities
associated with f(θ) are subjective probabilities.
• The joint distribution is f(x1, x2, …, xn|θ).
• The posterior distribution is f(θ|x1, x2, …, xn) is
our degree of belief regarding θ after observing
the sample data.

7-4.3 Bayesian Estimation of Parameters 37

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
Bayesian Estimation of Parameters-2
• Now the joint probability distribution of the
sample is
f(x1, x2, …, xn, θ) = f(x1, x2, …, xn |θ) ∙ f(θ)
  f  x1is:
• The marginal distribution , x2 ,..., xn ,θ  , for θ discrete
 θ
f  x1 , x2 ,..., xn    
  f  x1 , x2 ,..., xn ,θ  dθ, for θ continuous
 

• The desired posterior distribution is:

f  x1 , x2 ,..., xn , θ 
f θ | x1 , x2 ,..., xn  
f  x1 , x2 ,..., xn 

The Bayesian estimator of θ is θ, the mean of the posterior distribution.

7-4.3 Bayesian Estimation of Parameters 38

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-16: Bayes Estimator for the mean of a Normal
Distribution -1
Let X1, X2, …, Xn be a random sample from a normal distribution
unknown mean μ and known variance σ2. Assume that the prior
distribution for μ is:
1 1 
  2  2 0  02 2 0 2 
f μ  
   0  2 02
2

e  e
2 0 2 02
The joint probability distribution of the sample is:
n

1 
 1 2 2
 ( x   ) 2

f  x1 , x2 ,..., xn |   
i

e i 1

 2  2 n2

 n 2 n 
1 
 1 2 2

 xi  2 
  xi  n 2 

 e  i 1 i 1 

 2  2 n2

7-4.3 Bayesian Estimation of Parameters 39

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-17: Bayes Estimator for the mean of a Normal
Distribution-2
Now the joint probability distribution of the sample and μ is:
f  x1 , x2 ,..., xn ,    f  x1, x2 ,..., xn |    f μ 
  1 n  
1/2   2  2  2   2   02  2
xi  
 xi2 02 
 
1 
  0     0    2  02 
 e    

 2  2 n2
2 0
    
1
1/2   2  2  2
1   2   0  x   h ( x , x ,..., x , 2 ,  , 2 )
   2 2  1 1 2 n 0 0
  0   0 
e n   n 

Upon completing the square in the exponent,

2
 1 1   2  ( 2 / n )  0 x 02 
1/ 2 2  2
 
    2

   h2 ( x1 , x2 ,..., xn , 2 , 0 , 02 )
 x1 , x2 ,..., xn ,    e  0  / n     0  / n  0  / n  
2 2 2
f
where hi ( x1 , x2 ,..., xn ,  2 , 0 ,  02 )is a function of the observed
values and the parameters  2 , 0 and  02 .
Since f  x1 , x2 ,..., xn ,   does not depend on 
 1 1   2  ( 2 / n ) 0  x 02 
1/ 2 2  2
        h3 ( x1 , x2 ,..., xn , 2 , 0 , 02 )
f   | x1 , x2 ,..., xn   e  0  / n     02  2 / n  

7-4.3 Bayesian Estimation of Parameters 40

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-17: Bayes Estimator for the mean of a Normal
Distribution-3

which is recognized as a normal

probability density
function with posterior mean

and posterior variance

 0  n 
1 2 2
 1 1 
V     2  2   2

 0  n   0   2
n

7-4.3 Bayesian Estimation of Parameters 41

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Example 7-17: Bayes Estimator for the mean of a Normal
Distribution-4

To illustrate:
– The parameters are: μ0 = 0, σ02= 1
– Sample: n = 10, x-bar = 0.75, σ2 = 4

  
 n 
2
0   2
0 x

 02   2 n


 4 10  0  1 0.75 
 0.536
1   4 10 

7-4.3 Bayesian Estimation of Parameters 42

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Important Terms & Concepts of
Chapter 7
Bayes estimator means
Bias in parameter estimation Parameter estimation
Central limit theorem Point estimator
Estimator vs. estimate Population or distribution
Likelihood function moments
Maximum likelihood estimator Posterior distribution
Mean square error of an Prior distribution
estimator Sample moments
Minimum variance unbiased Sampling distribution
estimator An estimator has a:
Moment estimator – Standard error
Normal distribution as the – Estimated standard error
sampling distribution of Statistic
the:
Statistical inference
– sample mean
Unbiased estimator
– difference in two sample
Chapter 7 Summary 43
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.