ESTIMATION

• One of the main objectives of statistics is to draw inferences

about a population from the analysis of a sample drawn
from that population.

• Two important problems are: Estimation and Testing of

Hypothesis.

• The theory of Estimation was founded by Prof R.A Fisher in

1930.
Basic Terminology

• Population: any collection of

entities that have at least one
characteristic in common

• Parameter: the numbers that

describe characteristics of scores in
the population (mean, variance,
s.d., etc.)
Basic Terminology (cont’d)

• Sample: a part of the

population

• Statistic: the numbers that

describe characteristics of
scores in the sample (mean,
variance, s.d., correlation
coefficient, reliability
coefficient, etc.)
Basic Terminology (con’t)

• Estimate: a number computed by using the data collected from a

sample.

• Estimator: formula used to compute an estimate

• An estimator is a statistic that estimates some fact
about the population. You can also think of an
estimator as the rule that creates an estimate. For
example, the sample mean(x̄) is an estimator for the
population mean, μ.

• The quantity that is being estimated (i.e. the one you

want to know) is called the estimand.
• For example, let’s say you wanted to know the average height of
children in a certain school with a population of 1000 students.
• You take a sample of 30 children, measure them and find that the
mean height is 56 inches.
• This is your sample mean, the estimator. You use the sample mean
to estimate that the population mean (your estimand) is about 56
inches.
• It is also common to use estimation in situations where a researcher
simply wants to learn about an unknown population.
• In this case, a sample is selected from the population and the sample
data are then used to estimate the population parameters.
There are two forms of estimation:

• Point estimation (maximally likely value for parameter)

• Interval estimation (also called confidence interval for parameter)

A point estimate is a single number.

Whereas, a confidence interval,
naturally, is an interval.
Point estimators are not very reliable, as you can guess. Imagine
visiting 5% of the restaurants in London and saying that the
average meal is worth 22.50 pounds
You may be close, but chances are that the true value isn’t really 22.50 but
somewhere around it. When you think about it, it’s much safer to say that the
average meal in London is somewhere between 20 and 25 pounds. In this way, you
have created a confidence interval around your point estimate of 22.50!
Point estimates are single points that
are used to infer parameters directly.
For example,

• Sample proportion pˆ (“p hat”) is the

point estimator of p

• Sample mean x (“x bar”) is the point

estimator of μ

• Sample standard deviation s is the

point estimator of σ
• Imagine that you are given a dataset with a sample mean of 10.

• In this case, is 10 a point estimate or an estimator?

• Of course, it is a point estimate. It is a single number given by an

estimator.

• Here, the estimator is a point estimator and it is the formula for the
mean.
• Notice the use of different symbols to distinguish estimators and
parameters.

• More importantly, point estimates and parameters represent

fundamentally different things.

• Point estimates are calculated from the data; parameters are not.

• Point estimates vary from study to study; parameters do not.

• Point estimates are random variables: parameters are constants.

 POINT ESTIMATION

• Here, we assume that θ is an unknown parameter to be estimated.

• For example, θ might be the expected value of a random variable, θ=EX.

• To estimate θ, we need to collect some data. Specifically, we get a random

sample X1, X2, X3, ..., Xn such that Xi's have the same distribution as X.

• To estimate θ, we define a point estimator Θ to be the sample mean

ˆ X 1  X 2  ...... X n
X
n
 INTERVAL ESTIMATION

The sample mean is called a point estimate because it assigns a single value 
to the estimate. Thus, a point estimate is a single value that best estimates 
the population parameter 
Another type of estimate is the interval estimate.
• In interval estimation, the parameter being estimated lies between a 
certain interval, called the confidence interval, with a certain probability. 
• Thus, an interval estimate consists of two numbers between which the 
population parameter can be expected to lie with a certain level of 
confidence. 
• The end­points of a confidence interval are called the confidence limits
• If a sample of a normally distributed random variable with known standard 
deviation is used to estimate the true mean E[X] Then the 95% confidence 
interval for the sample mean are given by
X −1.96σX ≤ E[X] ≤ X + 1.96σX. 
• There are infinitely many possible estimators for θ, so how can we
make sure that we have chosen a good estimator?

• How do we compare different possible estimators?

• To do this, we provide a list of some desirable properties that we

would like our estimators to have.

• Intuitively, we know that a good estimator should be able to give us

values that are "close" to the real value of θ.
• An unbiased estimator is an accurate statistic that’s used to
approximate a population parameter.

• “Accurate” in this sense means that it’s neither an

overestimate nor an underestimate.

• If an overestimate or underestimate does happen, the mean of

the difference is called a “bias.”

• if the estimator (i.e. the sample mean) equals

the parameter (i.e. the population mean), then it’s an unbiased
estimator.
• Consider a sample of n independent draws from a
normal population having unknown µ and variance σ2
• As an estimator of the mean, we use the sample mean.

X
 X i E( X i )  
n

ˆ 2

 (X i  ) 2
  E( X i   )
2 2

n
 How to find estimators?

Method of Estimation

1.Method of Least square

2.Method of Maximum Likelihood estimation

 Least Square method

• It was defined by A.M. Lagendre, K. F. Gauss and A. A. Markow; it’s

most commonly used to estimate parameters in the linear model
regressions. It is sensitive to outliers.

• This is called least squares estimation because it gives the least value
for the sum of squared errors. Finding the best estimates of the
coefficients is often called “fitting” the model to the data, or sometimes
“learning” or “training” the model.
• Suppose X1,X2,X3…..Xn is a random sample of the population whose
parameter θ is the mean of the population which is unknown.

• Then the reasonable estimate of θ can be found by considering the

sum of squares of the differences.
n
S   ( X i   )2
i 1

• As small as possible. Where (Xi- θ ) is the error term in fitting the

regression line.

• The concept of minimize the sum of squared differences between

observed data and expected data is known as the principle of least
squares.
• Example: Suppose observation 3,4,8 are collected in a random sample of
size 3 from a population with unknown θ . Then
3
S   ( X i   ) 2  (3   ) 2  (4   ) 2  (8   ) 2  89  30  3 2
i 1

• To minimize S consider

θ 0 1 2 3 4 5 6 7 ……
S 89 62 41 26 17 14 17 26 ……

• Cleary for θ=5, S is minimum.

• Therefore θ=5 is mean ie required estimate of the population.

In general n n
S   ( X i   ) 2   ( X i2  2 X i   2 )
i 1 i 1
n n n
=  X  2  X i  
i
2 2
1
i 1 i 1 i 1
n n
=  X i2  2  X i   2 n
i 1 i 1
Which is a quadratic function in θ.
d
To minimize the S S 0
d
d  n 2 n
2 

d  i X  2  X i   n  0
 i 1 i 1 
n
0  2 X i  2 n  0
i 1
n

X i
 i 1
X
n

Ie Sample mean is a least square estimate of population mean θ

An area of soil was divided into 240 regions of equal area called quadrants and in
each quadrant the number of colonies of bacteria found was counted. The data was
given in the table below:

Count 0 1 2 3 4 5 >5
Frequen 11 37 64 55 37 24 12
cy

Here precise record about the sample is not kept. Therefore Method of least squares
is not applicable as mean can not be calculated.

In such cases MLE is adopted.

• Maximum likelihood is a relatively simple method of constructing an estimator for
an unknown parameter θ.

• It was introduced by R. A. Fisher, a great English mathematical statistician, in 1912.

Maximum likelihood estimation (MLE) can be applied in most problems, it has a
strong intuitive appeal, and often yields a reasonable estimator of θ.

• Furthermore, if the sample is large, the method will yield an excellent estimator of
θ. For these reasons, the method of maximum likelihood is probably the most
widely used method of estimation in statistics.
 Method of Maximum Likelihood Estimation
• Suppose we have a random sample X1, X2,..., Xn whose assumed probability
distribution depends on some unknown parameter θ

• If Xi's are discrete, then the likelihood function is defined as the product

P( X 1 , X 2 ,.......... X n ,  )  P( X 1 ,  ).P( X 2 ,  ).............P( X n ,  ).

• If Xi's are jointly continuous, then the likelihood function is defined as

f ( X 1 , X 2 ,.......... X n ,  )  f ( X 1 ,  ). f ( X 2 ,  )............. f ( X n ,  ).

• In some problems, it is easier to work with the log likelihood function given by

l ( )  Log[ f ( X 1 ,  ). f ( X 2 ,  )............. f ( X n ,  ).]   log f(X i ,  )

• The maximum likelihood estimate (mle) of θ is that value of θ
that maximizes lik(θ): it is the value that makes the observed
data the “most probable”.

• Rather than maximizing this product which can be quite

tedious, we often use the fact that the logarithm is an
increasing function so it will be equivalent to maximize the
log likelihood:
 Poisson Distribution

e   . x
P( X  x) 
x!
The likelihood function is
 e   . xi 
L ( )    
 xi ! 
The Log likelihood function is
 e   . xi 
L( )  Log         xi Log   Logxi !
 xi ! 
l ( )    1  Log   xi   Logxi !
l ( )    n  Log   xi   Logxi !
To maximize it

l ( )    n  Log   xi   Logxi !

d 1
l ( )   n   xi  0
d 

 
 xi
x
n
 Exponential Distribution

f ( x,  )   e  x , x  0
The likelihood function is
L(( xi ,  )    e  xi  ( e  x1 ).( e  x2 )..........( e  xn )
n  ( x1  x2 ..... xn )
L(( xi ,  )   e  e 
n xi

Maximize it
d n 1   xi n   xi
L(( xi ,  )  n e  e . xi  0
d
 n 1   xi  0
1n
  
 xi Mean
 Problems

For the following random samples, find the likelihood function:

Xi∼Binomial(3,θ), and we have observed (x1,x2,x3,x4)=(1,3,2,2).

Xi∼Exponential(θ) and we have observed

(x1,x2,x3,x4)=(1.23,3.32,1.98,2.12).
 Problems

For the following random samples, find the maximum

likelihood estimate of θ:

Xi∼Binomial(3,θ), and we have observed (x1,x2,x3,x4)=(1,3,2,2).

Xi∼Exponential(θ) and we have observed

(x1,x2,x3,x4)=(1.23,3.32,1.98,2.12).
 Problems

Suppose that we have observed the random sample X1, X2, X3, ..., Xn,
where Xi∼N(θ1,θ2), so

Find the maximum likelihood estimators for θ1 and θ2.

 Problems

A coin is flipped 100 times. Given that there were 55 heads, find the
maximum likelihood estimate for the probability p of heads on a
single toss.
 Problems

A city dispute center that settles disputes per month follows a Poisson
distribution. The following table gives the number of disputes settled
in a month. Determine the maximum likelihood estimator.

Number of Disputes 1 2 3 4 5 6 7 8 9 10

Frequency 2 11 18 6 24 3 10 9 7 10