Introduction to Sampling 6.2.

1 Sampling Distribution and the

Distribution and Point Estimation of Central Limit of Point Theorem
Parameters 6.2.1. Sampling Distribution
Overview Suppose that we draw all possible
 The field of statistical inference consists of samples of size n from a given
those methods used to make decisions or to population. Suppose further that we
draw conclusions about a population. compute a statistic (e.g., a mean,
proportion, standard deviation) for each
 These methods utilize the information sample. The probability distribution of
contained in a sample from the population in this statistic is called a sampling
drawing conclusions. distribution. And the standard deviation
 Statistical inference may be divided into two of this statistic is called the standard
major areas: error.
o Parameter estimation and; Variability of a Sampling Distribution
o Hypothesis testing
The variability of a sampling distribution is
measured by its variance or its standard
deviation. The variability of a sampling
6.1. Point Estimation distribution depends on three factors:
Point Estimation 
In statistics, point estimation involves o N: The number of observations in the
the use of sample data to calculate a population.
single value (known as a point o n: The number of observations in the sample.
estimate since it identifies a point in
o The way that the random sample is chosen.
some parameter space) which is to
serve as a "best guess" or "best If the population size is much larger than
estimate" of an unknown the sample size, then the sampling
population parameter (for example, distribution has roughly the same
the population mean). More formally, it standard error, whether we
is the application of a point estimator to sample with or without replacement. On
the data to obtain a point estimate. the other hand, if the sample represents
a significant fraction (say, 1/20) of the
Point estimation can be contrasted
population size, the standard error will
with interval estimation: such interval
be meaningfully smaller, when we
estimates are typically either confidence
sample without replacement.
intervals, in the case of frequentist
inference, or credible intervals, in the
case of Bayesian inference.
Sampling Distribution of the Mean
 A point estimate is a reasonable value of a
population parameter. Suppose we draw all possible samples
of size n from a population of size N.
 Data collected, X1, X2,…, Xn are random Suppose further that we compute a
variables. mean score for each sample. In this
 Functions of these random variables, X̄ and s2, way, we create a sampling distribution
are also random variables called statistics. of the mean.
 Statistics have their unique distributions that We know the following about the
are called sampling distributions. sampling distribution of the mean. The
mean of the sampling distribution ( )
is equal to the mean of the population
(μ). And the standard error of the the sampling distribution (σp) is
sampling distribution ( ) is determined determined by the standard deviation of
by the standard deviation of the the population (σ), the population size,
population (σ), the population size (N), and the sample size. These
and the sample size (n). These relationships are shown in the equations
relationships are shown in the equations below:
below:

In the standard error formula, the factor

sqrt[ (N - n ) / (N - 1) ] is called the finite
population correction or fpc. When the
population size is very large relative to
the sample size, the fpc is
approximately equal to one; and the Like the formula for the standard error of
standard error formula can be the mean, the formula for the standard
approximated by: error of the proportion uses the finite
population correction, sqrt[ (N - n ) / (N -
1) ]. When the population size is very
large relative to the sample size, the fpc
is approximately equal to one; and the
You often see this "approximate"
standard error formula can be
formula in introductory statistics texts.
approximated by:
As a general rule, it is safe to use the
approximate formula when the sample
size is no bigger than 1/20 of the
population size.

You often see this "approximate"

Sampling Distribution of the formula in introductory statistics texts.
Proportion As a general rule, it is safe to use the
In a population of size N, suppose that approximate formula when the sample
the probability of the occurrence of an size is no bigger than 1/20 of the
event (dubbed a "success") is P; and the population size.
probability of the event's non-occurrence
(dubbed a "failure") is Q. From this
population, suppose that we draw all 6.2.2 Central Limit Theorem
possible samples of size n. And finally,
within each sample, suppose that we Definition
determine the proportion of
successes p and failures q. In this way, The central limit theorem states that
we create a sampling distribution of the the sampling distribution of the mean of
proportion. any independent,random variable will be
normal or nearly normal, if the sample
We find that the mean of the sampling size is large enough.
distribution of the proportion (μp) is
equal to the probability of success in the How large is "large enough"? The
population (P). And the standard error of answer depends on two factors.
 Example No 1: In practice, researchers
o Requirements for accuracy. The more closely employ a mix of the above guidelines.
the sampling distribution needs to resemble a On this site, we use the normal
normal distribution, the more sample points will distribution when the population
be required. standard deviation is known and the
sample size is large. We might use
o The shape of the underlying population. The
either distribution when standard
more closely the original population resembles
deviation is unknown and the sample
a normal distribution, the fewer sample points
size is very large. We use the t-
will be required.
distribution when the sample size is
In practice, some statisticians say that a small, unless the underlying distribution
sample size of 30 is large enough when is not normal. The t distribution should
the population distribution is roughly not be used with small samples from
bell-shaped. Others recommend a populations that are not approximately
sample size of at least 40. But if the normal.
original population is distinctly not
Assume that a school district has 10,000
normal (e.g., is badly skewed, has
6th graders. In this district, the average
multiple peaks, and/or has outliers),
weight of a 6th grader is 80 pounds, with
researchers like the sample size to be
a standard deviation of 20 pounds.
even larger.
Suppose you draw a random sample of
50 students. What is the probability that
the average weight of a sampled
How to Choose Between T- student will be less than 75 pounds?
Distribution and Normal Distribution
Solution: To solve this problem, we
The t distribution and the normal need to define the sampling distribution
distribution can both be used with of the mean. Because our sample size
statistics that have a bell-shaped is greater than 30, the Central Limit
distribution. This suggests that we might Theorem tells us that the sampling
use either the t-distribution or the normal distribution will approximate a normal
distribution to analyze sampling distribution.
distributions. Which should we choose?
To define our normal distribution, we
Guidelines exist to help you make that need to know both the mean of the
choice. Some focus on the population sampling distribution and the standard
standard deviation. deviation. Finding the mean of the
 sampling distribution is easy, since it is
equal to the mean of the population.
o If the population standard deviation is known,
Thus, the mean of the sampling
use the normal distribution
distribution is equal to 80.
o If the population standard deviation is
unknown, use the t-distribution. The standard deviation of the sampling
distribution can be computed using the
Other guidelines focus on sample size. following formula:

o If the sample size is large, use the normal
distribution. (See the discussion above in the
section on the Central Limit Theorem to
understand what is meant by a "large" sample.)
o If the sample size is small, use the t-
distribution.
Let's review what we know and what we be computed using the following
want to know. We know that the formula:
sampling distribution of the mean is
normally distributed with a mean of 80
and a standard deviation of 2.81. We
want to know the probability that a
sample mean is less than or equal to 75
pounds. Here, the finite population correction is
equal to 1.0, since the population size
Because we know the population (N) was assumed to be infinite.
standard deviation and the sample size Therefore, standard error formula
is large, we'll use the normal distribution reduces to:
to find probability. To solve the problem,
we plug these inputs into the Normal
Probability Calculator: mean = 80,
standard deviation = 2.81, and normal
random variable = 75. The Calculator
tells us that the probability that the
average weight of a sampled student is
less than 75 pounds is equal to 0.038.
Note: Since the population size is more
than 20 times greater than the sample Let's review what we know and what we
size, we could have used the want to know. We know that the
"approximate" formula σx = [ σ / sqrt(n) ] sampling distribution of the proportion is
to compute the standard error. Had we normally distributed with a mean of 0.50
done that, we would have found a and a standard deviation of 0.04564.
standard error equal to [ 20 /] or 2.83. We want to know the probability that no
more than 40% of the sampled births
Example No 2: Find the probability that are boys.
of the next 120 births, no more than
40% will be boys. Assume equal Because we know the population
probabilities for the births of boys and standard deviation and the sample size
girls. Assume also that the number of is large, we'll use the normal distribution
births in the population (N) is very large, to find probability. To solve the problem,
essentially infinite. we plug these inputs into the Normal
Probability Calculator: mean = .5,
Solution: The Central Limit Theorem standard deviation = 0.04564, and the
tells us that the proportion of boys in normal random variable = .4. The
120 births will be approximately Calculator tells us that the probability
normally distributed. that no more than 40% of the sampled
The mean of the sampling distribution births are boys is equal to 0.014.
will be equal to the mean of the Note: This problem can also be treated
population distribution. In the population, as a binomial experiment. Elsewhere,
half of the births result in boys; and half, we showed how to analyze a binomial
in girls. Therefore, the probability of boy experiment. The binomial experiment is
births in the population is 0.50. Thus, actually the more exact analysis. It
the mean proportion in the sampling produces a probability of 0.018 (versus
distribution should also be 0.50. a probability of 0.14 that we found using
The standard deviation of the sampling the normal distribution). Without a
distribution (i.e., the standard error) can computer, the binomial approach is
computationally demanding. Therefore,
 many statistics texts emphasize the (where Xi = 0 or 1) is a point estimator of the
approach presented above, which uses population proportion p. And, the function:
the normal distribution to approximate
the binomial.
is a point
6.3. General Concept of Point estimator of the population variance σ2.
Estimation
Definition. The function u(x1, x2,
We'll start the lesson with some formal ..., xn) computed from a set of data is an
definitions. In doing so, recall that we observed point estimate of θ.
denote the n random variables arising
from a random sample as subscripted For example, if xi are the observed
uppercase letters: grade point averages of a sample of 88
students, then:
X1, X2, ..., Xn
The corresponding observed values of a
specific random sample are then
denoted as subscripted lowercase
letters:

x1, x2, ..., xn

Definition. The range of possible values of is a point estimate of μ, the mean grade
the parameter θ is called the parameter space Ω point average of all the students in the
(the greek letter "omega"). population.
For example, if μ denotes the mean And, if xi = 0 if a student has no tattoo,
grade point average of all college students, and xi = 1 if a student has a tattoo, then:
t hen the parameter space (assuming a 4-
p = 0.11
point grading scale) is:
Ω = {μ: 0 ≤ μ ≤ 4} is a point estimate of p, the proportion of
And, if p denotes the proportion of all students in the population who have
a tattoo.
students who smoke cigarettes, then the
parameter space is:
Ω = {p: 0 ≤ p ≤ 1}
6.3.1. Unbiased Estimator and
Definition. The function of X1, X2, ..., Xn, that
is, the statistic u(X1, X2, ..., Xn), used to Variance of a Point Estimator
estimate θ is called a point estimator of θ.
On the previous page, we showed that if Xi are
For example, the function: Bernoulli random variables with parameter p, then:

Definition. If the following holds:

is a point estimator of the population
mean μ. The function: E [u(X1, X2,…,Xn)] = θE [u (X1, X2,…,Xn)] = θ
then the statistic u (X1,X2,…,Xn) u
(X1,X2,…,Xn) is an unbiased estimator of the
parameter θ. Otherwise, u (X1,X2,…,Xn) u
(X1,X2,…,Xn) is a biased estimator of θ.
is the maximum likelihood estimator of p. And, Also, recall that the expected value of a chi-square
if Xi are normally distributed random variables random variable is its degrees of freedom. That is,
with mean μ and variance σ2, then: if:

are the
maximum likelihood estimators of μ and σ2, Then E(X) = r. Therefore:
respectively. A natural question then is whether
or not these estimators are "good" in any
sense. One measure of "good"
is "unbiasedness." The first equality holds because we effectively
multiplied the sample variance by 1. The second
Example No. 1: If Xi is a Bernoulli random equality holds by the law of expectation that tells us
variable with parameter p, then: we can pull a constant through the expectation. The
third equality holds because of the two facts we
recalled above. That is:

is the maximum likelihood estimator

(MLE) of p. Is the MLE of p an unbiased estimator
of p? And, the last equality is again simple algebra.

In summary, we have shown that, if Xi is a

Solution. Recall that if Xi is a Bernoulli random normally distributed random variable with
variable with parameter p, then E(Xi) = p. mean μ and variance σ2, then S2is an unbiased
Therefore: estimator of σ2. It turns out, however,
that S2 is always an unbiased estimator of σ2,
that is, for anymodel, not just the normal
model. (You'll be asked to show this in the
The first equality holds because we've merely homework.) And, although S2 is always an
replaced p-hat with its definition. The second
unbiased estimator of σ2, S is not an unbiased
equality holds by the rules of expectation for a
linear combination. The third equality holds
estimator of σ.
because E(Xi) = p. The fourth equality holds Example No. 3 :Let T be the time that is
because when you add the value p up n times, you needed for a specific task in a factory to be
get np. And, of course, the last equality is simple completed. In order to estimate the mean and
algebra. In summary, we have shown that:
variance of T, we observe a random
sample T1,T2,⋯⋯,T6. Thus, Ti’s are i.i.d. and
Therefore, the maximum likelihood have the same distribution as T. We obtain the
estimator is an unbiased estimator of p.
following values (in minutes):
Example No. 2 :If Xi are normally distributed
18,21,17,16,24,20.
random variables with mean μ and variance σ2,
what is an unbiased estimator of σ2? Find the values of the sample mean, the
Is S2 unbiased? sample variance, and the sample standard
deviation for the observed sample.
Solution. Recall that if Xi is a normally
distributed random variable with mean μ and The sample mean is:
variance σ2, then:
The sample variance is given by 6.3.3. Mean Squared Error
We seek estimators that are unbiased
and have minimal standard error.
Sometimes these goals are
incompatible. Consider Exhibit 4.2, which
indicates PDFs for two estimators of a
Finally, the sample standard deviation is given by: parameter θ. One is unbiased. The other
is biased but has a lower standard error.
Which estimator should we use?

6.3.2. Standard Error

The standard error of an estimator is its
standard deviation

Let’s calculate the standard error of the sample Exhibit 4.2: PDFs are indicated for two
mean estimator: estimators of a parameter θ. One is unbiased.
The other is biased but has lower standard
error.
Mean squared error (MSE) combines the
notions of bias and standard error. It is defined
as:

Since we have already determined the bias and

standard error of estimator, calculating its mean
squared error is easy:

Faced with alternative estimators for a given

parameter, it is generally reasonable to use the one
with the smallest MSE.
where σ is the standard deviation std(X) being
estimated. We don’t know the standard deviation
σ of X, but we can approximate the standard error
based upon some estimated value s for σ.
Irrespective of the value of σ, the standard error
decreases with the square root of the sample
size m. Quadrupling the sample size halves the
standard error.