See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/12318986
An introduction to statistical inference--3
Article in Journal of Accident & Emergency Medicine · October 2000
Source: PubMed
CITATIONS READS
15 2,226
3 authors, including:
Fiona Lecky
The University of Sheffield
258 PUBLICATIONS 9,807 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
STUMBL Trial. View project
Nutrient management in wheat View project
All content following this page was uploaded by Fiona Lecky on 20 April 2016.
The user has requested enhancement of the downloaded file.
Introduction to Statistical Inference
Dr. Fatima Sanchez-Cabo
[email protected]

http://www.genome.tugraz.at

Institute for Genomics and Bioinformatics,

Graz University of Technology,
Austria

Introduction to Probability Theory. March 9, 2006 – p. 1/29

Summary last session

• Motivation
• Algebra of sets
• Definition of probability space:
• Sample space
• Sigma algebra
• Probability axioms
• Conditional Probability and independence of events
• Random variables
• Continuos vs discrete
• pdf and mass function
• Distribution function

Introduction to Probability Theory. March 9, 2006 – p. 2/29

 Part IV:

From Probability to Statistics

Introduction to Probability Theory. March 9, 2006 – p. 3/29

Statistical Inference

• The target of statistical inference is to provide some information

about the probability distribution P defined over the probability
space (Ω, F).
• Differently from the previous examples where an exhaustive
observation was possible, this is often difficult.
• Hence, statistical inference focusses in the analysis and
interpretation of the realizations of the random variable in order to
draw conclusions about the probability law under study.
• The conclusions can be relative to:
• Estimation of a unique value for a parameter or parameters
essential in the probability distribution (i.e., p for a Binomial r.v)
• Estimation of a confidence interval for this parameter/s.
• Accept or reject a certain hypothesis about the probability
distribution of interest.

Introduction to Probability Theory. March 9, 2006 – p. 4/29

 Statistical Inference

• In general, there is some knowledge about the probability

distribution explaining a certain random process
• The inference process is often involved with deciding (looking at
the available data) which is the distribution that best explains the
available data among a set of them: {Fθ : θǫΘ}. That’s known as
parametric statistics.

Example 4.1: We are betting with a friend to "head" or "tail" when

tossing a coin. We want to know if the coin is unbiased. The most logic
approach will be to toss the coin "enough" times to get an approximate
value for the probability of head.

Introduction to Probability Theory. March 9, 2006 – p. 5/29

Choosing a random sample

• The first step in this process is to choose a sample representative

from the whole population (what is "enough" in the previous
example?)
• We should also take care of the cost/time.
• Random sampling is one way to obtain samples from a
population: all individuals have the same probability to be chosen.
This type of sampling is particularly useful because the
observations areQ then independent from each other; hence
F (x1 , . . . , xn ) = i F (xi )
• Sampling with replacement (some individuals might be
repeated in the sample)
• Sampling without replacement (each individual can be only
once in the sample)

Introduction to Probability Theory. March 9, 2006 – p. 6/29

 Combinatorics

• We can define the probability of an event A as:

number favorables cases

P (A) =
number of possible cases

• The number of possible samples of r elements chosen without

n!
replacement among n possible elements is C(n, r) = (n−r)!r!
• The number of possible ways to create a sample of size n using
r < n elements, if repetition is allowed, is r n

Examples:

1. An urn contains 5 red, 3 green, 2 blue and 4 white balls. A sample of size 8 is
selected at random without replacement. Calculate the probability that the sample
contains 2 red, 2 green, 1 blue, and 3 white balls.
2. Consider a class with 30 students. Calculate the probability that all 30 birthdays
are different. Introduction to Probability Theory. March 9, 2006 – p. 7/29
 Example 4.2

We are betting with a friend to "head" or "tail" when tossing a coin. We

want to know what is the probability of getting "head" so that we decide
what to bet on. Hence, we have the random variable
(
0 Head
X=
1 Tail

with P (X = 0) = θ, P (X = 1) = 1 − θ. Given a random sample

(X1 , X2 , X3 ), determine the mass function (probability for each
possible triplet). Does this sample has a known mass distribution?

Introduction to Probability Theory. March 9, 2006 – p. 8/29

Descriptive statistics

• Given a very large or very complex sample it might not be

possible to determine the distribution easily.
• It is often useful to make use of some tools that help to
understand the target distribution and the main characteristics of
the sample data:

Probability Statistic
Base Population Sample
Mean
Central tendency Expectation
Median
Sample variance
Dispersion Variance IQR
Mode

Introduction to Probability Theory. March 9, 2006 – p. 9/29

 Descriptive statistics

Given a sample (x1 , . . . , xn ) we can calculate the following expressions

to get a rough idea of the properties of the sample.
1. Parametric
• Mean: x̄ = 1 i xi
P
n
• Sample variance: s2 = 1 ni=1 (xi − x̄)2
P
n
Pn
• Sample Quasi-variance S 2 = 1 i=1 (xi − x̄) 2
n−1

2. Non-parametric (order statistics)

• Median:

x n+1 if n is odd
 [ 2 ]




 x[ n2 ] +x[ n2 ]+1

2 if n is even
• Interquantiles range: IQR=q3 -q1
• Mode: most repeated value
Introduction to Probability Theory. March 9, 2006 – p. 10/29
Descriptive statistics

Histogram of x

30

8
25
20

6
Frequency

15

4
10
5

2
0

2 4 6 8

(a) Histogram (b) Boxplot

Introduction to Probability Theory. March 9, 2006 – p. 11/29

 Descriptive statistics

Given x1 , . . . , xn a sample of size n, calculate the pth percentile:

p
1. R = int( 100 · (N + 1));
p p
F R = 100 · (N + 1) − int( 100 · (N + 1))
2. Find x[R] , x[R]+1
3. p = x[R] + (x[R]+1 − x[R] ) ∗ F R

Introduction to Probability Theory. March 9, 2006 – p. 12/29

Estimators and statistics

• Definition: A statistic is a function T : (Ω, F) → (Rk , Bk ), i.e.

T (x1 , . . . , xn ), over which a probability function can be defined.
• k is the dimension of the statistic.
• An estimator is a particular case of a statistic that approximates
one of the unknown parameters of the distribution.
• Examples:
• µ̂ = x̄ = 1 ni=1 xi
P
n
• θ̂ in example 4.2.
• An estimator is a random variable itself: Its moments can be
calculated
• An estimator is said to be unbiased if E[θ̂] = θ

Introduction to Probability Theory. March 9, 2006 – p. 13/29

 Example 4.3

• Given (X1 , . . . , Xn ) a random sample such as E[Xi ] = µ and

2 1
P
V [Xi ] = σ , show that for X̄ = n i Xi then E[X̄] = µ and
V [X̄] = σ 2 /n.
• Find an unbiased estimator for σ 2 ?

Note: The variance of an estimator is the sample error (SE).

Introduction to Probability Theory. March 9, 2006 – p. 14/29

Distribution of the statistics

• Since the statistics are function of random variables they are

theirselves random variables for which a probability distribution
can be defined.
• Particularly important are the distributions of statistics based on
normally distributed random variables.
• We have proved that X̄ ≡ N (µ, σ 2 /n).
• For the Central Limit Theorem X̄−µ
√
σ/ n
≡ N (0, 1).
• If σ is unknown and we substitute it by its natural estimator S 2 ,
then
X̄ − µ
t= √
S/ n
That is the Student-t distribution.

Introduction to Probability Theory. March 9, 2006 – p. 15/29

Student’s t

Introduction to Probability Theory. March 9, 2006 – p. 16/29

 χ2 and F-Snedecor

Other important distributions of statistics from normally distributed

random variables are:
• χ2n = i Xi2 when Xi ≡ N (0, 1)
P

• Fn,m = χ2n /n
χ2m /m

Chi−square (d.f.=10) F(5,10)

0.10

0.7
0.6
0.08

0.5
0.06

0.4
Density

Density

0.3
0.04

0.2
0.02

0.1
0.00

0.0

0 10 20 30 40 0 1 2 3 4 5

x x

(c) χ2 density (d) F-Snedecor

Introduction to Probability Theory. March 9, 2006 – p. 17/29
 Part V:

Hypothesis testing

Introduction to Probability Theory. March 9, 2006 – p. 18/29

Motivation

• There are some situations when we want to test if a certain

hypothesis about the parameter of interest is true
• Is the probability of "head" the same than the probability of "tail"
when flipping the coin of the previous experiment?

Formulate
QUESTION
???? an
hypothesis

EXPERIMENT

Results DON’T
Results support the
support the
hypothesis
hypothesis

Figure 1: The Scientific Method

Introduction to Probability Theory. March 9, 2006 – p. 19/29

Main concepts

• A statistical null hypothesis H0 is the one that want to be tested.

• Statistical tools provide the appropriated framework to guarantee
that its rejection is due to real mechanisms and not to chance. Its
rejection will provide the called "statistical significance".
• We never talk about "accepting" H0 : with the available data we
can say if we reject or not the H0 .

Introduction to Probability Theory. March 9, 2006 – p. 20/29

Error types

H0 is really...
Decision True False
Error! A false hypothesis has
OK!A true hypothesis has
Accept H0 been accepted. This is a Type II error.
been accepted
The probability of this is β
Error! a true hypothesis
has been rejected.
OK!
Reject H0 This is a Type I error.
A false hypothesis has been rejected
The probability of this
occurring is α

Introduction to Probability Theory. March 9, 2006 – p. 21/29

Error types

• The ideal situation would be to minimize the probability of both

errors (α,β), where:

α = P (Reject H0 /H0 is true) = P (ET I)

β = P (Accept H0 /H0 is f alse) = P (ET II)

• However, they are related and cannot be minimized at the same
time.
• Definition: The p-value corresponds to the probability of
rejecting the null hypothesis if it is actually true. If the p-value is
smaller than the allowed α-level H0 is rejected (always with the
available data!)

Introduction to Probability Theory. March 9, 2006 – p. 22/29

Protocol for statistically testing a null hypothesis

1. Question that we want to address (aufpassen: Statistical tests can

only reject the null hypothesis, H0)
2. Experimental design: How many repetitions we need How are we
going to select a sample that represents the population? Which
are the sources of variation? How do we remove the systematic
errors?
3. Data collection
4. Removal systematic errors
5. Which statistic summarizes best my sample. Does it have a
known distribution? (statistic: numerical variable that summarizes
my sample data in a meaningful way)

Introduction to Probability Theory. March 9, 2006 – p. 23/29

Example (5)

Introduction to Probability Theory. March 9, 2006 – p. 24/29

Simple hypothesis testing

1. We want to test if a particular sample comes from a distribution with a particular

mean value, let’s say 3. It is quite logical to try to estimate how close the estimator
of the population mean is from the proposed value. So we will calculate:

(x̄ − 3)

2. To minimize the intrinsic error of the sample mean, we will divide this difference by
the standard error (variance of the sample mean). We have then the statistic:

x̄ − 3
t= √
s/ n

3. Regarding to the definition of t-student, if H0 is true (H0 : µ = 3) then, t ≡ tn−1

where n is the number of samples. This makes sense, because in the t − student
distribution, the probability of getting t = 0 is very high and this is exactly what
would happen under H0 , because:

x̄ ≃ 3 → x̄ − 3 ≃ 0 → t = 0

Introduction to Probability Theory. March 9, 2006 – p. 25/29

Simple hypothesis testing (cont.)

• α was defined as:

1 − α = P (|t| ≤ t∗ ) → α = P (|t| > t∗ )

Where t∗ = tn−1; α and α is as well defined as:

2

α = P (Reject H0 /H0 true) = P (ET I)

• In this particular example, if we are under the H0 the value calculated for the
statistic |t| should be close to 0. Otherwise, we reject H0 (this would mean that
x̄ 6= 3). But, how big must be the difference (x̄-3)? For that, we fixed the value α
such as we just reject H0 being true for 5 out of 100 samples that we take. And for
this particular, this α determines a unique value, called critical value:

t∗ = tn−1; α
2

Introduction to Probability Theory. March 9, 2006 – p. 26/29

• If |t| > t∗ , we reject the H0 , because the probability of t = 0 (µ =3)
is very small (<0.05). Just in 5 out of 100 samples (what we can
consider by chance) the value of the statistic will be very far away
from 0 although the H0 is true. This condition is equivalent to:

p = P (tn−1 > |t|) < α = 0.05

• If |t| < t∗ the result obtained is the one we would expect for a
distribution following a t-distribution with n − 1 degrees of
freedom, as expected if the H0 is true. We don’t have then
enough evidence to reject the H0 . This is equivalent to say that:

p = P (tn−1 > |t|) > α = 0.05

Introduction to Probability Theory. March 9, 2006 – p. 27/29

 t distribution
0.4

0.35

0.3

0.25
Density

0.2

0.15 α=0.05

α/2=0.025
0.1

0.05

0
−6 −4 −t* 0 t*=2.22 4 6

Introduction to Probability Theory. March 9, 2006 – p. 28/29

View publication stats

References
[1] Durbin, R., Eddy, S., Krogh, A. and Mitchison, G. (1996) Biological
sequence analysis, Cambridge University Press.
[2] Durret, R. (1996) Probability: Theory and examples, Duxbury
Press, Second edition.
[3] Rohatgi, V.K. and Ehsanes Saleh, A.K.Md. (1988) An introduction
to probability and statistics, Wiley, Second Edition.
[4] Tuckwell, H.C. (1988) Elementary applications of probability theory,
[5] Gonick, L. and Smith, W. (2000) The cartoon guide to statistics.
Chapman and Hal
[Engineering statistics] http://www.itl.nist.gov/div898/handbook/

Introduction to Probability Theory. March 9, 2006 – p. 29/29