Part IB — Statistics

Based on lectures by D. Spiegelhalter

Notes taken by Dexter Chua

Lent 2015

These notes are not endorsed by the lecturers, and I have modified them (often
significantly) after lectures. They are nowhere near accurate representations of what
was actually lectured, and in particular, all errors are almost surely mine.

Estimation
Review of distribution and density functions, parametric families. Examples: bino-
mial, Poisson, gamma. Sufficiency, minimal sufficiency, the Rao-Blackwell theorem.
Maximum likelihood estimation. Confidence intervals. Use of prior distributions and
Bayesian inference. [5]

Hypothesis testing
Simple examples of hypothesis testing, null and alternative hypothesis, critical region,
size, power, type I and type II errors, Neyman-Pearson lemma. Significance level of
outcome. Uniformly most powerful tests. Likelihood ratio, and use of generalised likeli-
hood ratio to construct test statistics for composite hypotheses. Examples, including
t-tests and F -tests. Relationship with confidence intervals. Goodness-of-fit tests and
contingency tables. [4]

Linear models
Derivation and joint distribution of maximum likelihood estimators, least squares,
Gauss-Markov theorem. Testing hypotheses, geometric interpretation. Examples,
including simple linear regression and one-way analysis of variance. Use of software. [7]

1
Contents IB Statistics

Contents
0 Introduction 3

1 Estimation 4
1.1 Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Mean squared error . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Sufficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.5 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 Bayesian estimation . . . . . . . . . . . . . . . . . . . . . . . . . 15

2 Hypothesis testing 19
2.1 Simple hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 Composite hypotheses . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3 Tests of goodness-of-fit and independence . . . . . . . . . . . . . 25
2.3.1 Goodness-of-fit of a fully-specified null distribution . . . . 25
2.3.2 Pearson’s chi-squared test . . . . . . . . . . . . . . . . . . 26
2.3.3 Testing independence in contingency tables . . . . . . . . 27
2.4 Tests of homogeneity, and connections to confidence intervals . . 29
2.4.1 Tests of homogeneity . . . . . . . . . . . . . . . . . . . . . 29
2.4.2 Confidence intervals and hypothesis tests . . . . . . . . . 31
2.5 Multivariate normal theory . . . . . . . . . . . . . . . . . . . . . 32
2.5.1 Multivariate normal distribution . . . . . . . . . . . . . . 32
2.5.2 Normal random samples . . . . . . . . . . . . . . . . . . . 34
2.6 Student’s t-distribution . . . . . . . . . . . . . . . . . . . . . . . 35

3 Linear models 37
3.1 Linear models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 Simple linear regression . . . . . . . . . . . . . . . . . . . . . . . 39
3.3 Linear models with normal assumptions . . . . . . . . . . . . . . 42
3.4 The F distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.5 Inference for β . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.6 Simple linear regression . . . . . . . . . . . . . . . . . . . . . . . 47
3.7 Expected response at x∗ . . . . . . . . . . . . . . . . . . . . . . . 48
3.8 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.8.1 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . 50
3.8.2 Simple linear regression . . . . . . . . . . . . . . . . . . . 52
3.8.3 One way analysis of variance with equal numbers in each
group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

2
0 Introduction IB Statistics

0 Introduction
Statistics is a set of principles and procedures for gaining and processing quan-
titative evidence in order to help us make judgements and decisions. In this
course, we focus on formal statistical inference. In the process, we assume that
we have some data generated from some unknown probability model, and we aim
to use the data to learn about certain properties of the underlying probability
model.
In particular, we perform parametric inference. We assume that we have
a random variable X that follows a particular known family of distribution
(e.g. Poisson distribution). However, we do not know the parameters of the
distribution. We then attempt to estimate the parameter from the data given.
For example, we might know that X ∼ Poisson(µ) for some µ, and we want
to figure out what µ is.
Usually we repeat the experiment (or observation) many times. Hence we
will have X1 , X2 , · · · , Xn being iid with the same distribution as X. We call the
set X = (X1 , X2 , · · · , Xn ) a simple random sample. This is the data we have.
We will use the observed X = x to make inferences about the parameter θ,
such as
– giving an estimate θ̂(x) of the true value of θ.

– Giving an interval estimate (θ̂1 (x), θ̂2 (x)) for θ

– testing a hypothesis about θ, e.g. whether θ = 0.

3
1 Estimation IB Statistics

1 Estimation
1.1 Estimators
The goal of estimation is as follows: we are given iid X1 , · · · , Xn , and we know
that their probability density/mass function is fX (x; θ) for some unknown θ.
We know fX but not θ. For example, we might know that they follow a Poisson
distribution, but we do not know what the mean is. The objective is to estimate
the value of θ.
Definition (Statistic). A statistic is an estimate of θ. It is a function T of the
data. If we write the data as x = (x1 , · · · , xn ), then our estimate is written as
θ̂ = T (x). T (X) is an estimator of θ.
The distribution of T = T (X) is the sampling distribution of the statistic.
Note that we adopt the convention where capital X denotes a random variable
and x is an observed value. So T (X) is a random variable and T (x) is a particular
value we obtain after experiments.
Example. Let X1 , · · · , Xn be iid N (µ, 1). A possible estimator for µ is
1X
T (X) = Xi .
n
Then for any particular observed sample x, our estimate is
1X
T (x) = xi .
n
What is the sampling distribution P of T ? Recall from IA Probability that in
general, if Xi ∼ N (µi , σi2 ), then Xi ∼ N ( µi , σi2 ), which is something we
P P
can prove by considering moment-generating functions.
So we have T (X) ∼ N (µ, 1/n). Note that by the Central Limit Theorem,
even if Xi were not normal, we still have approximately T (X) ∼ N (µ, 1/n) for
large values of n, but here we get exactly the normal distribution even for small
values of n.
The estimator n1
P
Xi we had above is a rather sensible estimator. Of course,
we can also have silly estimators such as T (X) = X1 , or even T (X) = 0.32
always.
One way to decide if an estimator is silly is to look at its bias.
Definition (Bias). Let θ̂ = T (X) be an estimator of θ. The bias of θ̂ is the
difference between its expected value and true value.
bias(θ̂) = Eθ (θ̂) − θ.
Note that the subscript θ does not represent the random variable, but the thing
we want to estimate. This is inconsistent with the use for, say, the probability
mass function.
An estimator is unbiased if it has no bias, i.e. Eθ (θ̂) = θ.
To find out Eθ (T ), we can either find the distribution of T and find its
expected value, or evaluate T as a function of X directly, and find its expected
value.
Example. In the above example, Eµ (T ) = µ. So T is unbiased for µ.

4
1 Estimation IB Statistics

1.2 Mean squared error

Given an estimator, we want to know how good the estimator is. We have just
come up with the concept of the bias above. However, this is generally not a
good measure of how good the estimator is.
For example, if we do 1000 random trials X1 , · · · , X1000 , we can pick our
estimator as T (X) = X1 . This is an unbiased estimator, but is really bad because
we have just wastedPthe data from the other 999 trials. On the other hand,
T ′ (X) = 0.01 + 1000
1
Xi is biased (with a bias of 0.01), but is in general much
more trustworthy than T . In fact, at the end of the section, we will construct
cases where the only possible unbiased estimator is a completely silly estimator
to use.
Instead, a commonly used measure is the mean squared error.
Definition (Mean squared error). The mean squared error of an estimator θ̂ is
Eθ [(θ̂ − θ)2 ].
Sometimes, we use the root mean squared error, that is the square root of
the above.
We can express the mean squared error in terms of the variance and bias:

Eθ [(θ̂ − θ)2 ] = Eθ [(θ̂ − Eθ (θ̂) + Eθ (θ̂) − θ)2 ]

= Eθ [(θ̂ − Eθ (θ̂))2 ] + [Eθ (θ̂) − θ]2 + 2Eθ [Eθ (θ̂) − θ] Eθ [θ̂ − Eθ (θ̂)]
| {z }
0
2
= var(θ̂) + bias (θ̂).

If we are aiming for a low mean squared error, sometimes it could be preferable to
have a biased estimator with a lower variance. This is known as the “bias-variance
trade-off”.
For example, suppose X ∼ binomial(n, θ), where n is given and θ is to be
determined. The standard estimator is TU = X/n, which is unbiased. TU has
variance
varθ (X) θ(1 − θ)
varθ (TU ) = 2
= .
n n
Hence the mean squared error of the usual estimator is given by

mse(TU ) = varθ (TU ) + bias2 (TU ) = θ(1 − θ)/n.

Consider an alternative estimator

X +1 X 1
TB = = w + (1 − w) ,
n+2 n 2
where w = n/(n + 2). This can be interpreted to be a weighted average (by the
sample size) of the sample mean and 1/2. We have
 
nθ + 1 1
Eθ (TB ) − θ = − θ = (1 − w) −θ ,
n+2 2
and is biased. The variance is given by
varθ (X) θ(1 − θ)
varθ (TB ) = 2
= w2 .
(n + 2) n

5
1 Estimation IB Statistics

Hence the mean squared error is

 2
2 2 θ(1 − θ) 1
mse(TB ) = varθ (TB ) + bias (TB ) = w + (1 − w)2 −θ .
n 2

We can plot the mean squared error of each estimator for possible values of θ.
Here we plot the case where n = 10.
mse

0.03
unbiased estimator

0.02

0.01
biased estimator
0 θ
0 0.2 0.4 0.6 0.8 1.0

This biased estimator has smaller MSE unless θ has extreme values.
We see that sometimes biased estimators could give better mean squared
errors. In some cases, not only could unbiased estimators be worse — they could
be completely nonsense.
Suppose X ∼ Poisson(λ), and for whatever reason, we want to estimate
θ = [P (X = 0)]2 = e−2λ . Then any unbiased estimator T (X) must satisfy
Eθ (T (X)) = θ, or equivalently,
∞
X λx
Eλ (T (X)) = e−λ T (x) = e−2λ .
x=0
x!

The only function T that can satisfy this equation is T (X) = (−1)X .
Thus the unbiased estimator would estimate e−2λ to be 1 if X is even, -1 if
X is odd. This is clearly nonsense.

1.3 Sufficiency
Often, we do experiments just to find out the value of θ. For example, we might
want to estimate what proportion of the population supports some political
candidate. We are seldom interested in the data points themselves, and just
want to learn about the big picture. This leads us to the concept of a sufficient
statistic. This is a statistic T (X) that contains all information we have about θ
in the sample.
Example. Let X1 , · · · Xn be iid Bernoulli(θ), so that P(Xi = 1) = 1 − P(Xi =
0) = θ for some 0 < θ < 1. So
n
Y P P
fX (x | θ) = θxi (1 − θ)1−xi = θ xi
(1 − θ)n− xi
.
i=1
P
This depends on the data only through T (X) = xi , the total number of ones.

6
1 Estimation IB Statistics

Suppose we are now given that T (X) = t. Then what is the distribution of
X? We have
Pθ (X = x, T = t) Pθ (X = x)
fX|T =t (x) = = .
Pθ (T = t) Pθ (T = t)
Where the last equality comes because since if X = x, then T must be equal to
t. This is equal to P P  −1
θ xi (1 − θ)n− xi n
n t


n−t
= .
t θ (1 − θ) t
So the conditional distribution of X given T = t does not depend on θ. So if we
know T , then additional knowledge of x does not give more information about θ.
Definition (Sufficient statistic). A statistic T is sufficient for θ if the conditional
distribution of X given T does not depend on θ.
There is a convenient theorem that allows us to find sufficient statistics.
Theorem (The factorization criterion). T is sufficient for θ if and only if
fX (x | θ) = g(T (x), θ)h(x)
for some functions g and h.
Proof. We first prove the discrete case.
Suppose fX (x | θ) = g(T (x), θ)h(x). If T (x) = t, then
Pθ (X = x, T (X) = t)
fX|T =t (x) =
Pθ (T = t)
g(T (x), θ)h(x)
=P
{y:T (y)=t} g(T (y), θ)h(y)
g(t, θ)h(x)
= P
g(t, θ) h(y)
h(x)
=P
h(y)
which doesn’t depend on θ. So T is sufficient.
The continuous case is similar. If fX (x | θ) = g(T (x), θ)h(x), and T (x) = t,
then
g(T (x), θ)h(x)
fX|T =t (x) = R
y:T (y)=t
g(T (y), θ)h(y) dy
g(t, θ)h(x)
= R
g(t, θ) h(y) dy
h(x)
=R ,
h(y) dy
which does not depend on θ.
Now suppose T is sufficient so that the conditional distribution of X | T = t
does not depend on θ. Then
Pθ (X = x) = Pθ (X = x, T = T (x)) = Pθ (X = x | T = T (x))Pθ (T = T (x)).
The first factor does not depend on θ by assumption; call it h(x). Let the second
factor be g(t, θ), and so we have the required factorisation.

7
1 Estimation IB Statistics

Example. Continuing the above example,

P P
xi
fX (x | θ) = θ (1 − θ)n− xi
.

Take g(t, θ) = θt (1 − θ)n−t and h(x) = 1 to see that T (X) =

P
Xi is sufficient
for θ.
Example. Let X1 , · · · , Xn be iid U [0, θ]. Write 1[A] for the indicator function
of an arbitrary set A. We have
n
Y 1 1
fX (x | θ) = 1[0≤xi ≤θ] = 1[maxi xi ≤θ] 1[mini xi ≥0] .
i=1
θ θn

If we let T = maxi xi , then we have

1
fX (x | θ) = 1
n [t≤θ]
1[mini xi ≥0] .
|θ {z } | {z }
g(t,θ) h(x)

So T = maxi xi is sufficient.
Note that sufficient statistics are not unique. If T is sufficient for θ, then
so is any 1-1 function of T . X is always sufficient for θ as well, but it is not of
much use. How can we decide if a sufficient statistic is “good”?
Given any statistic T , we can partition the sample space X n into sets
{x ∈ X : T (x) = t}. Then after an experiment, instead of recording the actual
value of x, we can simply record the partition x falls into. If there are less
partitions than possible values of x, then effectively there is less information we
have to store.
If T is sufficient, then this data reduction does not lose any information
about θ. The “best” sufficient statistic would be one in which we achieve the
maximum possible reduction. This is known as the minimal sufficient statistic.
The formal definition we take is the following:
Definition (Minimal sufficiency). A sufficient statistic T (X) is minimal if it is
a function of every other sufficient statistic, i.e. if T ′ (X) is also sufficient, then
T ′ (X) = T ′ (Y) ⇒ T (X) = T (Y).
Again, we have a handy theorem to find minimal statistics:
Theorem. Suppose T = T (X) is a statistic that satisfies
fX (x; θ)
does not depend on θ if and only if T (x) = T (y).
fX (y; θ)
Then T is minimal sufficient for θ.
Proof. First we have to show sufficiency. We will use the factorization criterion
to do so.
Firstly, for each possible t, pick a favorite xt such that T (xt ) = t.
Now let x ∈ X N and let T (x) = t. So T (x) = T (xt ). By the hypothesis,
fX (x;θ)
fX (xt :θ) does not depend on θ. Let this be h(x). Let g(t, θ) = fX (xt , θ). Then

fX (x; θ)
fX (x; θ) = fX (xt ; θ) = g(t, θ)h(x).
fX (xt ; θ)

8
1 Estimation IB Statistics

So T is sufficient for θ.
To show that this is minimal, suppose that S(X) is also sufficient. By the
factorization criterion, there exist functions gS and hS such that
fX (x; θ) = gS (S(x), θ)hS (x).
Now suppose that S(x) = S(y). Then
fX (x; θ) gS (S(x), θ)hS (x) hS (x)
= = .
fX (y; θ) gS (S(y), θ)hS (y) hS (y)

This means that the ratio ffX

X (x;θ)
(y;θ) does not depend on θ. By the hypothesis, this
implies that T (x) = T (y). So we know that S(x) = S(y) implies T (x) = T (y).
So T is a function of S. So T is minimal sufficient.
Example. Suppose X1 , · · · , Xn are iid N (µ, σ 2 ). Then
(2πσ 2 )−n/2 exp − 2σ1 2 i (xi − µ)2
 P
fX (x | µ, σ 2 )
=
(2πσ 2 )−n/2 exp − 2σ1 2 i (yi − µ)2

fX (y | µ, σ 2 )
P
( ! !)
1 X
2
X
2 µ X X
= exp − 2 xi − yi + 2 xi − yi .
2σ i i
σ i i

constantPfunction of (µ, σ 2 ) iff i x2i = i yi2 and i xi = i yi . So

P P P P
This is a P
2 2
T (X) = ( i Xi , i Xi ) is minimal sufficient for (µ, σ ).
As mentioned, sufficient statistics allow us to store the results of our exper-
iments in the most efficient way. It turns out if we have a minimal sufficient
statistic, then we can use it to improve any estimator.
Theorem (Rao-Blackwell Theorem). Let T be a sufficient statistic for θ and let
θ̃ be an estimator for θ with E(θ̃2 ) < ∞ for all θ. Let θ̂(x) = E[θ̃(X) | T (X) =
T (x)]. Then for all θ,
E[(θ̂ − θ)2 ] ≤ E[(θ̃ − θ)2 ].
The inequality is strict unless θ̃ is a function of T .
Here we have to be careful with our definition of θ̂. It is defined as the
expected value of θ̃(X), and this could potentially depend on the actual value of
θ. Fortunately, since T is sufficient for θ, the conditional distribution of X given
T = t does not depend on θ. Hence θ̂ = E[θ̃(X) | T ] does not depend on θ, and
so is a genuine estimator. In fact, the proof does not use that T is sufficient; we
only require it to be sufficient so that we can compute θ̂.
Using this theorem, given any estimator, we can find one that is a function
of a sufficient statistic and is at least as good in terms of mean squared error of
estimation. Moreover, if the original estimator θ̃ is unbiased, so is the new θ̂.
Also, if θ̃ is already a function of T , then θ̂ = θ̃.
Proof. By the conditional expectation formula, we have E(θ̂) = E[E(θ̃ | T )] =
E(θ̃). So they have the same bias.
By the conditional variance formula,
var(θ̃) = E[var(θ̃ | T )] + var[E(θ̃ | T )] = E[var(θ̃ | T )] + var(θ̂).
Hence var(θ̃) ≥ var(θ̂). So mse(θ̃) ≥ mse(θ̂), with equality only if var(θ̃ | T ) =
0.

9
1 Estimation IB Statistics

Example. Suppose X1 , · · · , Xn are iid Poisson(λ), and let θ = e−λ , which is

the probability that X1 = 0. Then
P
e−nλ λ xi
pX (x | λ) = Q .
xi !
So P
θn (− log θ) xi
pX (x | θ) = Q .
xi !
P P
We see that T = Xi is sufficient for θ, and Xi ∼ Poisson(nλ).
We start with an easy estimator θ is θ̃ = 1X1 =0 , which is unbiased (i.e.
if we observe nothing in the first observation period, we assume the event is
impossible). Then
n
!
X
E[θ̃ | T = t] = P X1 = 0 | Xi = t
1
Pn
P(X1 = 0)P( 2 Xi = t)
= Pn
P( 1 Xi = t)
 t
n−1
= .
n
P
So θ̂ = (1 − 1/n) xi
. This approximately (1 − 1/n)nX̄ ≈ e−X̄ = e−λ̂ , which
makes sense.
Example. Let X1 , · · · , Xn be iid U [0, θ], and suppose that we want to estimate
θ. We have shown above that T = max Xi is sufficient for θ. Let θ̂ = 2X1 , an
unbiased estimator. Then

E[θ̃ | T = t] = 2E[X1 | max Xi = t]

= 2E[X1 | max Xi = t, X1 = max Xi ]P(X1 = max Xi )
+ 2E[X1 | max Xi = t, X1 ̸= max Xi ]P(X1 ̸= max Xi )
 
1 t n−1
=2 t× +
n 2 n
n+1
= t.
n
n+1
So θ̂ = n max Xi is our new estimator.

1.4 Likelihood
There are many different estimators we can pick, and we have just come up with
some criteria to determine whether an estimator is “good”. However, these do
not give us a systematic way of coming up with an estimator to actually use. In
practice, we often use the maximum likelihood estimator.
Let X1 , · · · , Xn be random variables with joint pdf/pmf fX (x | θ). We
observe X = x.
Definition (Likelihood). For any given x, the likelihood of θ is like(θ) = fX (x |
θ), regarded as a function of θ. The maximum likelihood estimator (mle) of θ is
an estimator that picks the value of θ that maximizes like(θ).

10
1 Estimation IB Statistics

Often there is no closed form for the mle, and we have to find θ̂ numerically.
When we can find the mle explicitly, in practice, we often maximize the
log-likelihood instead of the likelihood. In particular, if X1 , · · · , Xn are iid, each
with pdf/pmf fX (x | θ), then
n
Y
like(θ) = fX (xi | θ),
i=1
Xn
log like(θ) = log fX (xi | θ).
i=1

Example. Let X1 , · · · , Xn be iid Bernoulli(p). Then

X   X 
l(p) = log like(p) = xi log p + n − xi log(1 − p).

Thus P P
dl xi n − xi
= − .
dp p 1−p
P
This is zero when p = xi /n. So this is the maximum likelihood estimator
(and is unbiased).

Example. Let X1 , · · · , Xn be iid N (µ, σ 2 ), and we want to estimate θ = (µ, σ 2 ).

Then
n n 1 X
l(µ, σ 2 ) = log like(µ, σ 2 ) = − log(2π) − log(σ 2 ) − 2 (xi − µ)2 .
2 2 2σ
This is maximized when
∂l ∂l
= = 0.
∂µ ∂σ 2
We have
∂l 1 X ∂l n 1 X
=− 2 (xi − µ), = − + (xi − µ)2 .
∂µ σ ∂σ 2 2σ 2 2σ 4
2
So the solution,
P hence maximum
P likelihood estimator is (µ̂, σ̂ ) = (x̄, Sxx /n),
where x̄ = n1 xi and Sxx = (xi − x̄)2 .
nσ̂ 2 (n−1)σ 2
We shall see later that SXX /σ 2 = σ2 ∼ χ2n−1 , and so E(σ̂ 2 ) = n , i.e.
2
σ̂ is biased.
Example (German tank problem). Suppose the American army discovers some
German tanks that are sequentially numbered, i.e. the first tank is numbered 1,
the second is numbered 2, etc. Then if θ tanks are produced, then the probability
distribution of the tank number is U (0, θ). Suppose we have discovered n tanks
whose numbers are x1 , x2 , · · · , xn , and we want to estimate θ, the total number
of tanks produced. We want to find the maximum likelihood estimator.
Then
1
like(θ) = n 1[max xi ≤θ] 1[min xi ≥0] .
θ
So for θ ≥ max xi , like(θ) = 1/θn and is decreasing as θ increases, while for
θ < max xi , like(θ) = 0. Hence the value θ̂ = max xi maximizes the likelihood.

11
1 Estimation IB Statistics

Is θ̂ unbiased? First we need to find the distribution of θ̂. For 0 ≤ t ≤ θ, the

cumulative distribution function of θ̂ is
 n
n t
Fθ̂ (t) = P (θ̂ ≤ t) = P(Xi ≤ t for all i) = (P(Xi ≤ t)) = ,
θ
ntn−1
Differentiating with respect to T , we find the pdf fθ̂ = θn . Hence
Z θ
ntn−1 nθ
E(θ̂) = t n dt = .
0 θ n +1
So θ̂ is biased, but asymptotically unbiased.
Example. Smarties come in k equally frequent colours, but suppose we do not
know k. Assume that we sample with replacement (although this is unhygienic).
Our first Smarties are Red, Purple, Red, Yellow. Then
like(k) = Pk (1st is a new colour)Pk (2nd is a new colour)
Pk (3rd matches 1st)Pk (4th is a new colour)
k−1 1 k−2
=1× × ×
k k k
(k − 1)(k − 2)
= .
k3
The maximum likelihood is 5 (by trial and error), even though it is not much
likelier than the others.
How does the mle relate to sufficient statistics? Suppose that T is sufficient
for θ. Then the likelihood is g(T (x), θ)h(x), which depends on θ through T (x).
To maximise this as a function of θ, we only need to maximize g. So the mle θ̂
is a function of the sufficient statistic.
Note that if ϕ = h(θ) with h injective, then the mle of ϕ is given by h(θ̂). For
example, if the mle of the standard deviation σ is σ̂, then the mle of the variance
σ 2 is σ̂ 2 . This is rather useful in practice, since we can use this to simplify a lot
of computations.

1.5 Confidence intervals

Definition. A 100γ% (0 < γ < 1) confidence interval for θ is a random interval
(A(X), B(X)) such that P(A(X) < θ < B(X)) = γ, no matter what the true
value of θ may be.
It is also possible to have confidence intervals for vector parameters.
Notice that it is the endpoints of the interval that are random quantities,
while θ is a fixed constant we want to find out.
We can interpret this in terms of repeat sampling. If we calculate (A(x), B(x))
for a large number of samples x, then approximately 100γ% of them will cover
the true value of θ.
It is important to know that having observed some data x and calculated
95% confidence interval, we cannot say that θ has 95% chance of being within
the interval. Apart from the standard objection that θ is a fixed value and
either is or is not in the interval, and hence we cannot assign probabilities to
this event, we will later construct an example where even though we have got a
50% confidence interval, we are 100% sure that θ lies in that interval.

12
1 Estimation IB Statistics

Example. Suppose X1 , · · · , Xn are iid N (θ, 1). Find a 95% confidence interval
for θ. √
We know X̄ ∼ N (θ, n1 ), so that n(X̄ − θ) ∼ N (0, 1).
Let z1 , z2 be such that Φ(z2 ) − Φ(z1 ) = 0.95, where Φ is the standard normal
distribution function.√
We have P[z1 < n(X̄ − θ) < z2 ] = 0.95, which can be rearranged to give
 
z2 z1
P X̄ − √ < θ < X̄ − √ = 0.95.
n n
so we obtain the following 95% confidence interval:
 
z2 z1
X̄ − √ , X̄ − √ .
n n
There are many possible choices for z1 and z2 . Since N (0, 1) density is symmetric,
the shortest such interval is obtained by z2 = z0.025 = −z1 . We can also choose
other values such as z1 = −∞, z2 = 1.64, but we usually choose symmetric end
points.
The above example illustrates a common procedure for finding confidence
intervals:
– Find a quantity R(X, θ) such that the Pθ -distribution of R(X, θ)
√ does not
depend on θ. This is called a pivot. In our example, R(X, θ) = n(X̄ − θ).
– Write down a probability statement of the form Pθ (c1 < R(X, θ) < c2 ) = γ.
– Rearrange the inequalities inside P(. . .) to find the interval.
Usually c1 , c2 are percentage points from a known standardised distribution, often
equitailed. For example, we pick 2.5% and 97.5% points for a 95% confidence
interval. We could also use, say 0% and 95%, but this generally results in a
wider interval.
Note that if (A(X), B(X)) is a 100γ% confidence interval for θ, and T (θ)
is a monotone increasing function of θ, then (T (A(X)), T (B(X))) is a 100γ%
confidence interval for T (θ).
Example. Suppose X1 , · · · , X50 are iid N (0, σ 2 ). Find a 99% confidence interval
for σ 2 .
50
1 X 2
We know that Xi /σ ∼ N (0, 1). So 2 X ∼ χ250 .
σ i=1 i
P50
So R(X, σ 2 ) = i=1 Xi2 /σ 2 is a pivot.
Recall that χ2n (α) is the upper 100α% point of χ2n , i.e.

P(χ2n ≤ χ2n (α)) = 1 − α.

So we have c1 = χ250 (0.995) = 27.99 and c2 = χ250 (0.005) = 79.49.

So  P 2 
Xi
P c1 < < c2 = 0.99,
σ2
and hence
Xi2 Xi2
P P 
P < σ2 < = 0.99.
c2 c1

13
1 Estimation IB Statistics

Using
q P the q remarkabove, we know that a 99% confidence interval for σ is
Xi2
P 2
Xi
c2 , c1 .

Example. Suppose X1 , · · · , Xn are iid Bernoulli(p). Find an approximate

confidence interval for p.P
The mle of p is p̂ = Xi /n.
By the Central Limit theorem, p̂ is approximately N (p, p(1 − p)/n) for large
n. √
n(p̂ − p)
So p is approximately N (0, 1) for large n. So letting z(1−γ)/2 be
p(1 − p)
the solution to Φ(z(1−γ)/2 ) − Φ(−z(1−γ)/2 ) = 1 − γ, we have
r r !
p(1 − p) p(1 − p)
P p̂ − z(1−γ)/2 < p < p̂ + z(1−γ)/2 ≈ γ.
n n

But p is unknown! So we approximate it by p̂ to get a confidence interval for p

when n is large:
r r !
p̂(1 − p̂) p̂(1 − p̂)
P p̂ − z(1−γ)/2 < p < p̂ + z(1−γ)/2 ≈ γ.
n n

Note that we have made a lot of approximations here, but it would be difficult
to do better than this.
Example. Suppose an opinion poll says 20% of the people are going to vote
UKIP, based on a random sample of 1, 000 people. What might the true
proportion be?
We assume we have an observation of x = 200 from a binomial(n, p) distri-
bution with n = 1, 000. Then p̂ = x/n = 0.2 is an unbiased estimate, and also
the mle.
Now var(X/n) = p(1−p) n ≈ p̂(1−
n
p̂)
= 0.00016. So a 95% confidence interval is
r r !
p̂(1 − p̂) p̂(1 − p̂)
p̂ − 1.96 , p̂ + 1.96 = 0.20±1.96×0.013 = (0.175, 0.225),
n n

If we don’t want to make that many approximations, we can note p that p(1 −pp) ≤
1/4 for all 0 ≤ p ≤ 1. So a conservative 95% interval is p̂±1.96 1/4n
√ ≈ p̂± 1/n.
So whatever proportion is reported, it will be ’accurate’ to ±1/ n.
Example. Suppose X1 , X2 are iid from U (θ − 1/2, θ + 1/2). What is a sensible
50% confidence interval for θ?
We know that each Xi is equally likely to be less than θ or greater than θ.
So there is 50% chance that we get one observation on each side, i.e.
1
Pθ (min(X1 , X2 ) ≤ θ ≤ max(X1 , X2 )) = .
2
So (min(X1 , X2 ), max(X1 , X2 )) is a 50% confidence interval for θ.
But suppose after the experiment, we obtain |x1 − x2 | ≥ 12 . For example, we
might get x1 = 0.2, x2 = 0.9, then we know that, in this particular case, θ must
lie in (min(X1 , X2 ), max(X1 , X2 )), and we don’t have just 50% “confidence”!

14
1 Estimation IB Statistics

This is why after we calculate a confidence interval, we should not say “there
is 100(1 − α)% chance that θ lies in here”. The confidence interval just says
that “if we keep making these intervals, 100(1 − α)% of them will contain θ”.
But if have calculated a particular confidence interval, the probability that that
particular interval contains θ is not 100(1 − α)%.

1.6 Bayesian estimation

So far we have seen the frequentist approach to a statistical inference, i.e.
inferential statements about θ are interpreted in terms of repeat sampling. For
example, the percentage confidence in a confidence interval is the probability
that the interval will contain θ, not the probability that θ lies in the interval.
In contrast, the Bayesian approach treats θ as a random variable taking
values in Θ. The investigator’s information and beliefs about the possible values
of θ before any observation of data are summarised by a prior distribution π(θ).
When X = x are observed, the extra information about θ is combined with the
prior to obtain the posterior distribution π(θ | x) for θ given X = x.
There has been a long-running argument between the two approaches. Re-
cently, things have settled down, and Bayesian methods are seen to be appropriate
in huge numbers of application where one seeks to assess a probability about a
“state of the world”. For example, spam filters will assess the probability that a
specific email is a spam, even though from a frequentist’s point of view, this is
nonsense, because the email either is or is not a spam, and it makes no sense to
assign a probability to the email’s being a spam.
In Bayesian inference, we usually have some prior knowledge about the
distribution of θ (e.g. between 0 and 1). After collecting some data, we will find
a posterior distribution of θ given X = x.
Definition (Prior and posterior distribution). The prior distribution of θ is the
probability distribution of the value of θ before conducting the experiment. We
usually write as π(θ).
The posterior distribution of θ is the probability distribution of the value of
θ given an outcome of the experiment x. We write as π(θ | x).
By Bayes’ theorem, the distributions are related by
fX (x | θ)π(θ)
π(θ | x) = .
fX (x)
Thus

π(θ | x) ∝ fX (x | θ)π(θ).
posterior ∝ likelihood × prior.

where the constant of proportionality is chosen to make the total mass of

the posterior distribution equal to one. Usually, we use this form, instead of
attempting to calculate fX (x).
It should be clear that the data enters through the likelihood, so the inference
is automatically based on any sufficient statistic.
Example. Suppose I have 3 coins in my pocket. One is 3 : 1 in favour of tails,
one is a fair coin, and one is 3 : 1 in favour of heads.

15
1 Estimation IB Statistics

I randomly select one coin and flip it once, observing a head. What is the
probability that I have chosen coin 3?
Let X = 1 denote the event that I observe a head, X = 0 if a tail. Let θ
denote the probability of a head. So θ is either 0.25, 0.5 or 0.75.
Our prior distribution is π(θ = 0.25) = π(θ = 0.5) = π(θ = 0.75) = 1/3.
The probability mass function fX (x | θ) = θx (1 − θ)1−x . So we have the
following results:

θ π(θ) fX (x = 1 | θ) fX (x = 1 | θ)π(θ) π(θ | x)

0.25 0.33 0.25 0.0825 0.167
0.50 0.33 0.50 0.1650 0.333
0.75 0.33 0.75 0.2475 0.500
Sum 1.00 1.50 0.4950 1.000

So if we observe a head, then there is now a 50% chance that we have picked
the third coin.
Example. Suppose we are interested in the true mortality risk θ in a hospital
H which is about to try a new operation. On average in the country, around
10% of the people die, but mortality rates in different hospitals vary from around
3% to around 20%. Hospital H has no deaths in their first 10 operations. What
should we believe about θ?
Let Xi = 1 if the ith patient in H dies. The
P P
xi
fx (x | θ) = θ (1 − θ)n− xi
.

Suppose a priori that θ ∼ beta(a, b) for some unknown a > 0, b > 0 so that

π(θ) ∝ θa−1 (1 − θ)b−1 .

Then the posteriori is

P P
π(θ | x) ∝ fx (x | θ)π(θ) ∝ θ xi +a−1 (1 − θ)n− xi +b−1
.
P P
We recognize this as beta( xi + a, n − xi + b). So
P P
θ xi +a−1 (1 − θ)n− xi +b−1
π(θ | x) = P P .
B( xi + a, n − xi + b)
In practice, we need to find a Beta prior distribution that matches our information
from other hospitals. It turns out that beta(a = 3, b = 27) prior distribution has
mean 0.1 and P(0.03 < θ <P .20) = 0.9. P
Then
P we observe data xi = 0, n = 0. So the posterior is beta( xi +
a, n − xi + b) = beta(3, 37). This has a mean of 3/40 = 0.075.
This leads to a different conclusion than a frequentist analysis. Since nobody
has died so far, the mle is 0, which does not seem plausible. Using a Bayesian
approach, we have a higher mean than 0 because we take into account the data
from other hospitals.
For this problem, a beta prior leads to a beta posterior. We say that the
beta family is a conjugate family of prior distributions for Bernoulli samples.
Suppose that a = b = 1 so that π(θ) P = 1 for 0 P < θ < 1 — the uniform
distribution. Then the posterior is beta( xi + 1, n − xi + 1), with properties

16
1 Estimation IB Statistics

mean mode variance

prior P 1/2 non-unique
P P 1/12 P
xi + 1 xi ( xi + 1)(n − xi + 1)
posterior
n+2 n (n + 2)2 (n + 3)
Note that the mode of the posteriorPis the mle.
xi +1
The posterior mean estimator, n+2 is discussed in Lecture 2, where we
showed that this estimator had smaller mse than the mle for non-extreme value
of θ. This is known as the Laplace’s estimator.
The posterior variance is bounded above by 1/(4(n + 3)), and this is smaller
than the prior variance, and is smaller for larger n.
Again, note that the posterior automatically depends on the data through
the sufficient statistic.
After we come up with the posterior distribution, we have to decide what
estimator to use. In the case above, we used the posterior mean, but this might
not be the best estimator.
To determine what is the “best” estimator, we first need a loss function. Let
L(θ, a) be the loss incurred in estimating the value of a parameter to be a when
the true value is θ.
Common loss functions are quadratic loss L(θ, a) = (θ − a)2 , absolute error
loss L(θ, a) = |θ − a|, but we can have others.
When our estimate is a, the expected posterior loss is
Z
h(a) = L(θ, a)π(θ | x) dθ.

Definition (Bayes estimator). The Bayes estimator θ̂ is the estimator that

minimises the expected posterior loss.
For quadratic loss,
Z
h(a) = (a − θ)2 π(θ | x) dθ.

h′ (a) = 0 if Z
(a − θ)π(θ | x) dθ = 0,
or Z Z
a π(θ | x) dθ =
θπ(θ | x) dθ,
R R
Since π(θ | x) dθ = 1, the Bayes estimator is θ̂ = θπ(θ | x) dθ, the posterior
mean.
For absolute error loss,
Z
h(a) = |θ − a|π(θ | x) dθ
Z a Z ∞
= (a − θ)π(θ | x) dθ + (θ − a)π(θ | x) dθ
−∞ a
Z a Z a
=a π(θ | x) dθ − θπ(θ | x) dθ
−∞ −∞
Z ∞ Z ∞
+ θπ(θ | x) dθ − a π(θ | x) dθ.
a a

17
1 Estimation IB Statistics

Now h′ (a) = 0 if Z a Z ∞
π(θ | x) dθ = π(θ | x) dθ.
−∞ a

This occurs when each side is 1/2. So θ̂ is the posterior median.

Example. Suppose that X1 , · · · , Xn are iid N (µ, 1), and that a priori µ ∼
N (0, τ −2 ) for some τ −2 . So τ is the certainty of our prior knowledge.
The posterior is given by

π(µ | x) ∝ fx (x | µ)π(µ)
µ2 τ 2
   
1X
∝ exp − (xi − µ)2 exp −
2 2
"  P 2 #
1 xi
∝ exp − (n + τ 2 ) µ −
2 n + τ2

since we can regard n, τ and all the xi as constants in the normalisation term,
and then complete the square with respect to µ. So P the posterior distribution
of µ given x is a normal distribution with mean xi /(n + τ 2 ) and variance
1/(n + τ 2 ).
The normal density is symmetric, and so the posterior mean and the posterior
xi /(n + τ 2 ).
P
median have the same value
This is the optimal estimator for both quadratic and absolute loss.
Example. Suppose that X1 , · · · , Xn are iid Poisson(λ) random variables, and
λ has an exponential distribution with mean 1. So π(λ) = e−λ .
The posterior distribution is given by
P P
π(λ | x) ∝ enλ λ xi e−λ = λ xi e−(n+1)λ , λ > 0,
P
which is gamma ( xi + 1, n + 1). Hence under quadratic loss, our estimator is
P
xi + 1
λ̂ = ,
n+1
the posterior mean.
Under absolute error loss, λ̂ solves
λ̂
P P
λ xi e−(n+1)λ
xi +1
Z
(n + 1) 1
P dλ = .
0 ( xi )! 2

18
2 Hypothesis testing IB Statistics

2 Hypothesis testing
Often in statistics, we have some hypothesis to test. For example, we want to
test whether a drug can lower the chance of a heart attack. Often, we will have
two hypotheses to compare: the null hypothesis states that the drug is useless,
while the alternative hypothesis states that the drug is useful. Quantitatively,
suppose that the chance of heart attack without the drug is θ0 and the chance
with the drug is θ. Then the null hypothesis is H0 : θ = θ0 , while the alternative
hypothesis is H1 : θ ̸= θ0 .
It is important to note that the null hypothesis and alternative hypothesis
are not on equal footing. By default, we assume the null hypothesis is true.
For us to reject the null hypothesis, we need a lot of evidence to prove that.
This is since we consider incorrectly rejecting the null hypothesis to be a much
more serious problem than accepting it when we should not. For example, it is
relatively okay to reject a drug when it is actually useful, but it is terrible to
distribute drugs to patients when the drugs are actually useless. Alternatively,
it is more serious to deem an innocent person guilty than to say a guilty person
is innocent.
In general, let X1 , · · · , Xn be iid, each taking values in X , each with unknown
pdf/pmf f . We have two hypotheses, H0 and H1 , about f . On the basis of data
X = x, we make a choice between the two hypotheses.
Example.
– A coin has P(Heads) = θ, and is thrown independently n times. We could
have H0 : θ = 12 versus H1 : θ = 34 .
– Suppose X1 , · · · , Xn are iid discrete random variables. We could have H0 :
the distribution is Poisson with unknown mean, and H1 : the distribution
is not Poisson.
– General parametric cases: Let X1 , · · · , Xn be iid with density f (x | θ). f
is known while θ is unknown. Then our hypotheses are H0 : θ ∈ Θ0 and
H1 : θ ∈ Θ1 , with Θ0 ∩ Θ1 = ∅.
– We could have H0 : f = f0 and H1 : f = f1 , where f0 and f1 are densities
that are completely specified but do not come form the same parametric
family.
Definition (Simple and composite hypotheses). A simple hypothesis H specifies
f completely (e.g. H0 : θ = 12 ). Otherwise, H is a composite hypothesis.

2.1 Simple hypotheses

Definition (Critical region). For testing H0 against an alternative hypothesis
H1 , a test procedure has to partition X n into two disjoint exhaustive regions
C and C̄, such that if x ∈ C, then H0 is rejected, and if x ∈ C̄, then H0 is not
rejected. C is the critical region.
When performing a test, we may either arrive at a correct conclusion, or
make one of the two types of error:
Definition (Type I and II error).

19
2 Hypothesis testing IB Statistics

(i) Type I error : reject H0 when H0 is true.

(ii) Type II error : not rejecting H0 when H0 is false.

Definition (Size and power). When H0 and H1 are both simple, let

α = P(Type I error) = P(X ∈ C | H0 is true).

β = P(Type II error) = P(X ̸∈ C | H1 is true).

The size of the test is α, and 1 − β is the power of the test to detect H1 .
If we have two simple hypotheses, a relatively straightforward test is the
likelihood ratio test.
Definition (Likelihood). The likelihood of a simple hypothesis H : θ = θ∗ given
data x is
Lx (H) = fX (x | θ = θ∗ ).
The likelihood ratio of two simple hypotheses H0 , H1 given data x is

Lx (H1 )
Λx (H0 ; H1 ) = .
Lx (H0 )

A likelihood ratio test (LR test) is one where the critical region C is of the form

C = {x : Λx (H0 ; H1 ) > k}

for some k.
It turns out this rather simple test is “the best” in the following sense:
Lemma (Neyman-Pearson lemma). Suppose H0 : f = f0 , H1 : f = f1 , where
f0 and f1 are continuous densities that are nonzero on the same regions. Then
among all tests of size less than or equal to α, the test with the largest power is
the likelihood ratio test of size α.
Proof. Under the likelihood ratio test, our critical region is
 
f1 (x)
C= x: >k ,
f0 (x)

where
R k is chosen such that α = P(reject H0 | H0 ) = P(X ∈ C | H0 ) =
C
f0 (x) dx. The probability of Type II error is given by
Z
β = P(X ̸∈ C | f1 ) = f1 (x) dx.
C̄

∗
Let C be the critical region of any other test with size less than or equal to α.
Let α∗ = P(X ∈ C ∗ | H0 ) and β ∗ = P(X ̸∈ C ∗ | H1 ). We want to show β ≤ β ∗ .
We know α∗ ≤ α, i.e
Z Z
f0 (x) dx ≤ f0 (x) dx.
C∗ C

20
2 Hypothesis testing IB Statistics

Also, on C, we have f1 (x) > kf0 (x), while on C̄ we have f1 (x) ≤ kf0 (x). So
Z Z
f1 (x) dx ≥ k f0 (x) dx
∗ ∗
ZC̄ ∩C ZC̄ ∩C
f1 (x) dx ≤ k f0 (x) dx.
C̄∩C ∗ C̄∩C ∗

Hence
Z Z
∗
β−β = f1 (x) dx − f1 (x) dx
C̄ ∗
ZC̄ Z
= f1 (x) dx + f1 (x) dx
C̄∩C ∗ C̄∩C̄ ∗
Z Z
− f1 (x) dx − f1 (x) dx
∗ ∗
Z C̄ ∩C Z C̄∩C̄
= f1 (x) dx − f1 (x) dx
C̄∩C ∗ C̄ ∗ ∩C
Z Z
≤k f0 (x) dx − k f0 (x) dx
C̄∩C ∗ C̄ ∗ ∩C
Z Z 
=k f0 (x) dx + f0 (x) dx
C̄∩C ∗ C∩C ∗
Z Z 
−k f0 (x) dx + f0 (x) dx
C̄ ∗ ∩C C∩C ∗
∗
= k(α − α)
≤ 0.

C̄ C

C¯∗ ∩ C
C¯∗ β ∗ /H1
(f1 ≥ kf0 )

C ∗ ∩ C̄
C∗ C ∗ ∩ C α∗ /H0
(f1 ≤ kf0 )
β/H1 α/H0

Here we assumed the f0 and f1 are continuous densities. However, this

assumption is only needed to ensure that the likelihood ratio test of exactly size
α exists. Even with non-continuous distributions, the likelihood ratio test is still
a good idea. In fact, you will show in the example sheets that for a discrete
distribution, as long as a likelihood ratio test of exactly size α exists, the same
result holds.
Example. Suppose X1 , · · · , Xn are iid N (µ, σ02 ), where σ02 is known. We want
to find the best size α test of H0 : µ = µ0 against H1 : µ = µ1 , where µ0 and µ1

21
2 Hypothesis testing IB Statistics

are known fixed values with µ1 > µ0 . Then

 
(2πσ02 )−n/2 exp − 2σ1 2 (xi − µ1 )2
P
0
Λx (H0 ; H1 ) =  
(2πσ02 )−n/2 exp − 2σ1 2 (xi − µ0 )2
P
0

n(µ20 − µ21 )
 
µ1 − µ0
= exp nx̄ + .
σ02 2σ02
This is an increasing function of x̄, so for any k, Λx > k ⇔ x̄ > c for some c.
Hence we reject H0 if x̄ > c, where c is chosen
√ such that P(X̄ > c | H0 ) = α.
Under H0 , X̄ ∼ N (µ0 , σ02 /n), so Z = n(X̄ − µ0 )/σ0 ∼ N (0, 1).
Since x̄ > c ⇔ z > c′ for some c′ , the size α test rejects H0 if
√
n(x̄ − µ0 )
z= > zα .
σ0
For example, suppose µ0 = 5, µ1 = 6, σ0 = 1, α = 0.05, n = 4 and x =
(5.1, 5.5, 4.9, 5.3). So x̄ = 5.2.
From tables, z0.05 = 1.645. We have z = 0.4 and this is less than 1.645. So x
is not in the rejection region.
We do not reject H0 at the 5% level and say that the data are consistent
with H0 .
Note that this does not mean that we accept H0 . While we don’t have
sufficient reason to believe it is false, we also don’t have sufficient reason to
believe it is true.
This is called a z-test.
In this example, LR tests reject H0 if z > k for some constant k. The size of
such a test is α = P(Z > k | H0 ) = 1 − Φ(k), and is decreasing as k increasing.
Our observed value z will be in the rejected region iff z > k ⇔ α > p∗ = P(Z >
z | H0 ).
Definition (p-value). The quantity p∗ is called the p-value of our observed data
x. For the example above, z = 0.4 and so p∗ = 1 − Φ(0.4) = 0.3446.
In general, the p-value is sometimes called the “observed significance level” of
x. This is the probability under H0 of seeing data that is “more extreme” than
our observed data x. Extreme observations are viewed as providing evidence
against H0 .

2.2 Composite hypotheses

For composite hypotheses like H : θ ≥ 0, the error probabilities do not have a
single value. We define
Definition (Power function). The power function is

W (θ) = P(X ∈ C | θ) = P(reject H0 | θ).

We want W (θ) to be small on H0 and large on H1 .

Definition (Size). The size of the test is

α = sup W (θ).
θ∈Θ0

22
2 Hypothesis testing IB Statistics

This is the worst possible size we can get.

For θ ∈ Θ1 , 1 − W (θ) = P(Type II error | θ).
Sometimes the Neyman-Pearson theory can be extended to one-sided alter-
natives.
For example, in the previous example, we have shown that the most powerful
size α test of H0 : µ = µ0 versus H1 : µ = µ1 (where µ1 > µ0 ) is given by
 √ 
n(x̄ − µ0 )
C= x: > zα .
σ0
The critical region depends on µ0 , n, σ0 , α, and the fact that µ1 > µ0 . It does
not depend on the particular value of µ1 . This test is then uniformly the most
powerful size α for testing H0 : µ = µ0 against H1 : µ > µ0 .
Definition (Uniformly most powerful test). A test specified by a critical region
C is uniformly most powerful (UMP) size α test for test H0 : θ ∈ Θ0 against
H1 : θ ∈ Θ1 if
(i) supθ∈Θ0 W (θ) = α.
(ii) For any other test C ∗ with size ≤ α and with power function W ∗ , we have
W (θ) ≥ W ∗ (θ) for all θ ∈ Θ1 .
Note that these may not exist. However, the likelihood ratio test often works.
Example. Suppose X1 , · · · , Xn are iid N (µ, σ02 ) where σ0 is known, and we
wish to test H0 : µ ≤ µ0 against H1 : µ > µ0 .
First consider testing H0′ : µ = µ0 against H1′ : µ = µ1 , where µ1 > µ0 . The
Neyman-Pearson test of size α of H0′ against H1′ has
 √ 
n(x̄ − µ0 )
C= x: > zα .
σ0
We show that C is in fact UMP for the composite hypotheses H0 against H1 .
For µ ∈ R, the power function is
W (µ) = Pµ (reject H0 )
√ 
n(X̄ − µ0 )
= Pµ > zα
σ0
√ √ 
n(X̄ − µ) n(µ0 − µ)
= Pµ > zα +
σ0 σ0
 √ 
n(µ0 − µ)
= 1 − Φ zα +
σ0
To show this is UMP, we know that W (µ0 ) = α (by plugging in). W (µ) is an
increasing function of µ. So
sup W (µ) = α.
µ≤µ0

So the first condition is satisfied.

For the second condition, observe that for any µ > µ0 , the Neyman-Pearson
size α test of H0′ vs H1′ has critical region C. Let C ∗ and W ∗ belong to any
other test of H0 vs H1 of size ≤ α. Then C ∗ can be regarded as a test of H0′
vs H1′ of size ≤ α, and the Neyman-Pearson lemma says that W ∗ (µ1 ) ≤ W (µ1 ).
This holds for all µ1 > µ0 . So the condition is satisfied and it is UMP.

23
2 Hypothesis testing IB Statistics

We now consider likelihood ratio tests for more general situations.

Definition (Likelihood of a composite hypothesis). The likelihood of a composite
hypothesis H : θ ∈ Θ given data x to be

Lx (H) = sup f (x | θ).

θ∈Θ

So far we have considered disjoint hypotheses Θ0 , Θ1 , but we are not interested

in any specific alternative. So it is easier to take Θ1 = Θ rather than Θ \ Θ0 .
Then
Lx (H1 ) supθ∈Θ1 f (x | θ)
Λx (H0 ; H1 ) = = ≥ 1,
Lx (H0 ) supθ∈Θ0 f (x | θ)
with large values of Λ indicating departure from H0 .
Example. Suppose that X1 , · · · , Xn are iid N (µ, σ02 ), with σ02 known, and we
wish to test H0 : µ = µ0 against H1 : µ = ̸ µ0 (for given constant µ0 ). Here
Θ0 = {µ0 } and Θ = R.
For the numerator, we have supΘ f (x | µ) = f (x | µ̂), where µ̂ is the mle.
We know that µ̂ = x̄. Hence
 
(2πσ02 )−n/2 exp − 2σ1 2 (xi − x̄)2
P
0
Λx (H0 ; H1 ) =  .
2 −n/2 1
P
(2πσ0 ) exp − 2σ2 (xi − µ0 )2
0

Then H0 is rejected if Λx is large.

To make our lives easier, we can use the logarithm instead:
1 hX X i n
2 log Λ(H0 ; H1 ) = 2 (xi − µ0 )2 − (xi − x̄)2 = 2 (x̄ − µ0 )2 .
σ0 σ0

So we can reject H0 if we have

√
n(x̄ − µ0 )
>c
σ0

for some c. √
n(X̄ − µ0 )
We know that under H0 , Z = ∼ N (0, 1). So the size α
σ0
generalised likelihood test rejects H0 if
√
n(x̄ − µ0 )
> zα/2 .
σ0

n(X̄ − µ0 )2
Alternatively, since ∼ χ21 , we reject H0 if
σ02

n(x̄ − µ0 )2
> χ21 (α),
σ02
2
(check that zα/2 = χ21 (α)).
Note that this is a two-tailed test — i.e. we reject H0 both for high and low
values of x̄.

24
2 Hypothesis testing IB Statistics

The next theorem allows us to use likelihood ratio tests even when we cannot
find the exact relevant null distribution.
First consider the “size” or “dimension” of our hypotheses: suppose that
H0 imposes p independent restrictions on Θ. So for example, if Θ = {θ : θ =
(θ1 , · · · , θk )}, and we have
– H0 : θi1 = a1 , θi2 = a2 , · · · , θip = ap ; or
– H0 : Aθ = b (with A p × k, b p × 1 given); or

– H0 : θi = fi (φ), i = 1, · · · , k for some φ = (φ1 , · · · , φk−p ).

We say Θ has k free parameters and Θ0 has k − p free parameters. We write
|Θ0 | = k − p and |Θ| = k.
Theorem (Generalized likelihood ratio theorem). Suppose Θ0 ⊆ Θ1 and |Θ1 | −
|Θ0 | = p. Let X = (X1 , · · · , Xn ) with all Xi iid. If H0 is true, then as n → ∞,

2 log ΛX (H0 ; H1 ) ∼ χ2p .

If H0 is not true, then 2 log Λ tends to be larger. We reject H0 if 2 log Λ > c,

where c = χ2p (α) for a test of approximately size α.

We will not prove this result here. In our example above, |Θ1 | − |Θ0 | = 1,
and in this case, we saw that under H0 , 2 log Λ ∼ χ21 exactly for all n in that
particular case, rather than just approximately.

2.3 Tests of goodness-of-fit and independence

2.3.1 Goodness-of-fit of a fully-specified null distribution
So far, we have considered relatively simple cases where we are attempting to
figure out, say, the mean. However, in reality, more complicated scenarios arise.
For example, we might want to know if a dice is fair, i.e. if the probability of
getting each number is exactly 16 . Our null hypothesis would be that p1 = p2 =
· · · = p6 = 16 , while the alternative hypothesis allows any possible values of pi .
In general, suppose the observation space X is partitioned into k sets, and
let pi be the probability that an observation is in set i for i = 1, · · · , k. We want
to test “H0 : the pi ’s arise from a fully specifiedP model” against “H1 : the pi ’s
are unrestricted (apart from the obvious pi ≥ 0, pi = 1)”.

Example. The following table lists the birth months of admissions to Oxford
and Cambridge in 2012.

Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug
470 515 470 457 473 381 466 457 437 396 384 394

Is this compatible with a uniform distribution over the year?

Out of n independent observations, let Ni be the number of observations in
ith set. So (N1 , · · · , Nk ) ∼ multinomial(k; p1 , · · · , pk ).
For a generalized likelihood ratio test of H0 , we need to find the maximised
likelihood under H0 and H1 .

25
2 Hypothesis testing IB Statistics

n1 nk
Under H P1 , like(p1 , · · · , pk ) ∝ p1 · · · pk . So the log likelihood P is l =
constant + ni log pi . We want to maximise this subject to pi = 1. Us-
ing the Lagrange multiplier, we will find that the mle is pˆi = ni /n. Also
|Θ1 | = k − 1 (not k, since they must sum up to 1).
Under H0 , the values of pi are specified completely, say pi = p̃i . So |Θ0 | = 0.
Using our formula for pˆi , we find that

p̂1 · · · p̂nk k
 n1   
X ni
2 log Λ = 2 log = 2 ni log (1)
p̃n1 1 · · · p̃nk k np̃i

Here |Θ1 |−|Θ0 | = k −1. So we reject H0 if 2 log Λ > χ2k−1 (α) for an approximate
size α test.
Under H0 (no effect of month of birth), p̃i is the proportion of births in
month i in 1993/1994 in the whole population — this is not simply proportional
1
to the number of days in each month (or even worse, 12 ), as there is for example
an excess of September births (the “Christmas effect”). Then
 
X ni
2 log Λ = 2 ni log = 44.86.
np̃i

P(χ211 > 44.86) = 3 × 10−9 , which is our p-value. Since this is certainly less than
0.001, we can reject H0 at the 0.1% level, or can say the result is “significant at
the 0.1% level”.
The traditional levels for comparison are α = 0.05, 0.01, 0.001, roughly
corresponding to “evidence”, “strong evidence” and “very strong evidence”.
A similar common situation has H0 : pi = pi (θ) for some parameter θ and H1
as before. Now |Θ0 | is the number of independent parameters to be estimated
under H0 .
Under H0 , we find mle θ̂ by maximizing ni log pi (θ), and then
! !
pˆ1 n1 · · · pˆk nk X ni
2 log Λ = 2 log =2 ni log . (2)
p1 (θ̂)n1 · · · pk (θ̂)nk npi (θ̂)

The degrees of freedom are k − 1 − |Θ0 |.

2.3.2 Pearson’s chi-squared test

Notice that the two log likelihoods are of the same form. In general, let oi = ni
(observed number) and let ei = np˜i or npi (θ̂) (expected number). Let δi = oi − ei .
Then
 
X oi
2 log Λ = 2 oi log
ei
 
X δi
=2 (ei + δi ) log 1 +
ei
2
 
X δi δ
=2 (ei + δi ) − i2 + O(δi3 )
ei 2ei
X 2 2

δ δ
=2 δi + i − i + O(δi3 )
ei 2ei

26
2 Hypothesis testing IB Statistics

P P P
We know that δi = 0 since ei = oi . So
X δ2
i
≈
ei
X (oi − ei )2
= .
ei
This is known as the Pearson’s chi-squared test.
Example. Mendel crossed 556 smooth yellow male peas with wrinkled green
peas. From the progeny, let
(i) N1 be the number of smooth yellow peas,
(ii) N2 be the number of smooth green peas,
(iii) N3 be the number of wrinkled yellow peas,
(iv) N4 be the number of wrinkled green peas.
We wish to test the goodness of fit of the model
9 3 3 1


H0 : (p1 , p2 , p3 , p4 ) = 16 , 16 , 16 , 16 .
Suppose we observe (n1 , n2 , n3 , n4 ) = (315, 108, 102, 31).
We find (e1 , e2 , e3 , e4 ) = (312.75, 104.25, 104.25, 34.75). The actual 2 log Λ =
P (oi −ei )2
0.618 and the approximation we had is ei = 0.604.
Here |Θ0 | = 0 and |Θ1 | = 4 − 1 = 3. So we refer to test statistics χ23 (α).
Since χ23 (0.05) = 7.815, we see that neither value is significant at 5%. So there
is no evidence against Mendel’s theory. In fact, the p-value is approximately
P(χ23 > 0.6) ≈ 0.90. This is a really good fit, so good that people suspect the
numbers were not genuine.
Example. In a genetics problem, each individual has one of the three possible
genotypes, with probabilities p1 , p2 , p3 . Suppose we wish to test H0 : pi = pi (θ),
where
p1 (θ) = θ2 , p2 = 2θ(1 − θ), p3 (θ) = (1 − θ)2 .
for some θ ∈ (0, 1).
We observe Ni = ni . Under H0 , the mle θ̂ is found by maximising
X
ni log pi (θ) = 2n1 log θ + n2 log(2θ(1 − θ)) + 2n3 log(1 − θ).

We find that θ̂ = 2n12n

+n2
. Also, |Θ0 | = 1 and |Θ1 | = 2.
After conducting an experiment, we can substitute pi (θ̂) into (2), or find the
corresponding Pearson’s chi-squared statistic, and refer to χ21 .

2.3.3 Testing independence in contingency tables

Definition (Contingency table). A contingency table is a table in which obser-
vations or individuals are classified according to one or more criteria.
Example. 500 people with recent car changes were asked about their previous
and new cars. The results are as follows:

27
2 Hypothesis testing IB Statistics

New car
Large Medium Small

Previous
Large 56 52 42

car
Medium 50 83 67
Small 18 51 81
This is a two-way contingency table: Each person is classified according to the
previous car size and new car size.
Consider a two-way contingency table with r rows and c columns. For
i = 1, · · · , r And j = 1, · · · , c, let pij be the probability that an individual
selected from the population under consideration is classified in row i and
column j. (i.e. in the (i, j) cell of the table).
Let Ppi+P= P(in row i) and p+j = P(in column j). Then we must have
p++ = i j pij = 1.
Suppose a random sample of n individuals is taken, and let nij be the number
of these classified
P in the (i, j) cellP of the table.
Let ni+ = j nij and n+j = i nij . So n++ = n.
We have
(N11 , · · · , N1c , N21 , · · · , Nrc ) ∼ multinomial(rc; p11 , · · · , p1c , p21 , · · · , prc ).
We may be interested in testing the null hypothesis that the two classifications
are independent. So we test
– H0 : pij = pi+ p+j for all i, j, i.e. independence of columns and rows.
– H1 : pij are unrestricted.
Of course we have the usual restrictions like p++ = 1, pij ≥ 0.
n
Under H1 , the mles are pˆij = nij .
n
Under H0 , the mles are p̂i+ = nni+ and p̂+j = n+j .
Write oij = nij and eij = np̂i+ p̂+j = ni+ n+j /n.
Then
r X c r X c
(oij − eij )2
  X
X oij
2 log Λ = 2 oij log ≈ .
i=1 j=1
eij i=1 j=1
eij

using the same approximating steps for Pearson’s Chi-squared test.

We have |Θ1 | = rc − 1, because under H1 the pij ’s sum P to one. Also,
|Θ0 | = (r − 1) + (c − 1) P because p1+ , · · · , pr+ must satisfy i pi+ = 1 and
p+1 , · · · , p+c must satisfy j p+j = 1. So
|Θ1 | − |Θ0 | = rc − 1 − (r − 1) − (c − 1) = (r − 1)(c − 1).
Example. In our previous example, we wish to test H0 : the new and previous
car sizes are independent. The actual data is:
New car
Large Medium Small Total
Large 56 52 42 150
Previous

Medium 50 83 67 120
car

Small 18 51 81 150
Total 124 186 190 500

28
2 Hypothesis testing IB Statistics

while the expected values given by H0 is

New car
Large Medium Small Total
Large 37.2 55.8 57.0 150
Previous
car Medium 49.6 74.4 76.0 120
Small 37.2 55.8 57.0 150
Total 124 186 190 500

Note the margins are the same. It is quite clear that they do not match well,
but we can find the p value to be sure.
X X (oij − eij )2
= 36.20, and the degrees of freedom is (3 − 1)(3 − 1) = 4.
eij
From the tables, χ24 (0.05) = 9.488 and χ24 (0.01) = 13.28.
So our observed value of 36.20 is significant at the 1% level, i.e. there is
strong evidence against H0 . So we conclude that the new and present car sizes
are not independent.

2.4 Tests of homogeneity, and connections to confidence

intervals
2.4.1 Tests of homogeneity
Example. 150 patients were randomly allocated to three groups of 50 patients
each. Two groups were given a new drug at different dosage levels, and the third
group received a placebo. The responses were as shown in the table below.

Improved No difference Worse Total

Placebo 18 17 15 50
Half dose 20 10 20 50
Full dose 25 13 12 50
Total 63 40 47 150

Here the row totals are fixed in advance, in contrast to our last section, where
the row totals are random variables.
For the above, we may be interested in testing H0 : the probability of
“improved” is the same for each of the three treatment groups, and so are
the probabilities of “no difference” and “worse”, i.e. H0 says that we have
homogeneity down the rows.
In general, we have independent observations from r multinomial distributions,
each of which has c categories, i.e. we observe an r × c table (nij ), for i = 1, · · · , r
and j = 1, · · · , c, where

(Ni1 , · · · , Nic ) ∼ multinomial(ni+ , pi1 , · · · , pic )

independently for each i = 1, · · · , r. We want to test

H0 : p1j = p2j = · · · = prj = pj ,

29
2 Hypothesis testing IB Statistics

for j = 1, · · · , c, and
H1 : pij are unrestricted.
Using H1 , for any matrix of probabilities (pij ),
r
Y ni+ !
like((pij )) = pni1i1 · · · pnicic ,
i=1
n i1 ! · · · n ic !

and
r X
X c
log like = constant + nij log pij .
i=1 j=1
ij n
Using Lagrangian methods, we find that p̂ij = ni+ .
Under H0 ,
X c
log like = constant + n+j log pj .
j=1
n+j
By Lagrangian methods, we have p̂j = n++ .
Hence
r X c   r X c  
X p̂ij X nij
2 log Λ = nij log =2 nij log ,
i=1 j=1
p̂j i=1 j=1
ni+ n+j /n++

which is the same as what we had last time, when the row totals are unrestricted!
We have |Θ1 | = r(c − 1) and |Θ0 | = c − 1. So the degrees of freedom is
r(c − 1) − (c − 1) = (r − 1)(c − 1), and under H0 , 2 log Λ is approximately
χ2(r−1)(c−1) . Again, it is exactly the same as what we had last time!
We reject H0 if 2 log Λ > χ2(r−1)(c−1) (α) for an approximate size α test.
n n+j
If we let oij = nij , eij = i+
n++ , and δij = oij − eij , using the same approxi-
mating steps as for Pearson’s chi-squared, we obtain
X (oij − eij )2
2 log Λ ≈ .
eij

Example. Continuing our previous example, our data is

Improved No difference Worse Total

Placebo 18 17 15 50
Half dose 20 10 20 50
Full dose 25 13 12 50
Total 63 40 47 150

The expected under H0 is

Improved No difference Worse Total

Placebo 21 13.3 15.7 50
Half dose 21 13.3 15.7 50
Full dose 21 13.3 15.7 50
Total 63 40 47 150

30
2 Hypothesis testing IB Statistics

We find 2 log Λ = 5.129, and we refer this to χ24 . Clearly this is not significant,
as the mean of χ24 is 4, and is something we would expect to happen solely by
chance.
We can calculate the p-value: from tables, χ24 (0.05) = 9.488, so our observed
value is not significant at 5%, and the data are consistent with H0 .
We conclude that there is no evidence for a difference between the drug at
the given doses and the placebo.
For interest,
X (oij − eij )2
= 5.173,
eij
giving the same conclusion.

2.4.2 Confidence intervals and hypothesis tests

Confidence intervals or sets can be obtained by inverting hypothesis tests, and
vice versa
Definition (Acceptance region). The acceptance region A of a test is the
complement of the critical region C.
Note that when we say “acceptance”, we really mean “non-rejection”! The
name is purely for historical reasons.
Theorem (Duality of hypothesis tests and confidence intervals). Suppose
X1 , · · · , Xn have joint pdf fX (x | θ) for θ ∈ Θ.

(i) Suppose that for every θ0 ∈ Θ there is a size α test of H0 : θ = θ0 . Denote

the acceptance region by A(θ0 ). Then the set I(X) = {θ : X ∈ A(θ)} is a
100(1 − α)% confidence set for θ.
(ii) Suppose I(X) is a 100(1 − α)% confidence set for θ. Then A(θ0 ) = {X :
θ0 ∈ I(X)} is an acceptance region for a size α test of H0 : θ = θ0 .
Intuitively, this says that “confidence intervals” and “hypothesis accep-
tance/rejection” are the same thing. After gathering some data X, we can
produce a, say, 95% confidence interval (a, b). Then if we want to test the
hypothesis H0 : θ = θ0 , we simply have to check whether θ0 ∈ (a, b).
On the other hand, if we have a test for H0 : θ = θ0 , then the confidence
interval is all θ0 in which we would accept H0 : θ = θ0 .
Proof. First note that θ0 ∈ I(X) iff X ∈ A(θ0 ).
For (i), since the test is size α, we have

P(accept H0 | H0 is true) = P(X ∈ A(θ0 ) | θ = θ0 ) = 1 − α.

And so
P(θ0 ∈ I(X) | θ = θ0 ) = P(X ∈ A(θ0 ) | θ = θ0 ) = 1 − α.
For (ii), since I(X) is a 100(1 − α)% confidence set, we have

P (θ0 ∈ I(X) | θ = θ0 ) = 1 − α.

So
P(X ∈ A(θ0 ) | θ = θ0 ) = P(θ ∈ I(X) | θ = θ0 ) = 1 − α.

31
2 Hypothesis testing IB Statistics

Example. Suppose X1 , · · · , Xn are iid N (µ, 1) random variables and we want

a 95% confidence set for µ.
One way is to use the theorem and find the confidence set that belongs to the
hypothesis test that we found in the previous example. We find
√ a test of size 0.05
of H0 : µ = µ0 against H1 : µ ̸= µ0 that rejects H0 when | n(x̄ − µ0 )| > 1.96
(where 1.96 is the upper 2.5% point of N (0, 1)).
√
Then I(X) = {µ : X ∈ A(µ)} √ = {µ : | n( √X̄ − µ)| < 1.96}. So a 95%
confidence set for µ is (X̄ − 1.96/ n, X̄ + 1.96/ n).

2.5 Multivariate normal theory

2.5.1 Multivariate normal distribution
So far, we have only worked with scalar random variables or a vector of iid random
variables. In general, we can have a random (column) vector X = (X1 , · · · , Xn )T ,
where the Xi are correlated.
The mean of this vector is given by

µ = E[X] = (E(X1 ), · · · , E(Xn ))T = (µ1 , · · · , µn )T .

Instead of just the variance, we have the covariance matrix

cov(X) = E[(X − µ)(X − µ)T ] = (cov(Xi , Xj ))ij ,

provided they exist, of course.

We can multiply the vector X by an m × n matrix A. Then we have

E[AX] = Aµ,

and
cov(AX) = A cov(X)AT . (∗)
The last one comes from

cov(AX) = E[(AX − E[AX])(AX − E[AX])T ]

= E[A(X − EX)(X − EX)T AT ]
= AE[(X − EX)(X − EX)T ]AT .

If we have two random vectors V, W, we can define the covariance cov(V, W) to

be a matrix with (i, j)th element cov(Vi , Wj ). Then cov(AX, BX) = A cov(X)B T .
An important distribution is a multivariate normal distribution.
Definition (Multivariate normal distribution). X has a multivariate normal
distribution if, for every t ∈ Rn , the random variable tT X (i.e. t · X) has a
normal distribution. If E[X] = µ and cov(X) = Σ, we write X ∼ Nn (µ, Σ).
Note that Σ is symmetric and is positive semi-definite because by (∗),

tT Σt = var(tT X) ≥ 0.

So what is the pdf of a multivariate normal? And what is the moment generating
function? Recall that a (univariate) normal X ∼ N (µ, σ 2 ) has density
1 (x − µ)2
 
2 1
fX (x; µ, σ ) = √ exp − ,
2πσ 2 σ2

32
2 Hypothesis testing IB Statistics

with moment generating function

 
1
MX (s) = E[esX ] = exp µs + σ 2 s2 .
2
Hence for any t, the moment generating function of tT X is given by
 
T 1
MtT X (s) = E[est X ] = exp tT µs + tT Σts2 .
2
Hence X has mgf
 
T 1
MX (t) = E[et X
] = MtT X (1) = exp tT µ + tT Σt . (†)
2
Proposition.
(i) If X ∼ Nn (µ, Σ), and A is an m × n matrix, then AX ∼ Nm (Aµ, AΣAT ).
(ii) If X ∼ Nn (0, σ 2 I), then
|X|2 XT X X Xi2
2
= = ∼ χ2n .
σ σ2 σ2
Instead of writing |X|2 /σ 2 ∼ χ2n , we often just say |X|2 ∼ σ 2 χ2n .
Proof.
(i) See example sheet 3.
(ii) Immediate from definition of χ2n .
 
X1
Proposition. Let X ∼ Nn (µ, Σ). We split X up into two parts: X = ,
X2
where Xi is a ni × 1 column vector and n1 + n2 = n.
Similarly write    
µ1 Σ11 Σ12
µ= , Σ= ,
µ2 Σ21 Σ22
where Σij is an ni × nj matrix.
Then
(i) Xi ∼ Nni (µi , Σii )
(ii) X1 and X2 are independent iff Σ12 = 0.
Proof.
(i) See example sheet 3.
(ii) Note that by symmetry of Σ, Σ12 = 0 if and only if Σ21 = 0.
 
t
From (†), MX (t) = exp(t T
µ+ 12 tT Σt) for each t ∈ R . We write t = 1 .
n
t2
Then the mgf is equal to

MX (t) =
 
T T T 1 T 1 T 1 T
exp t1 µ1 + t2 µ2 + t1 Σ11 t1 + t2 Σ22 t2 + t1 Σ12 t2 + t2 Σ21 t1 .
2 2 2
From (i), we know that MXi (ti ) = exp(tTi µi + 12 tTi Σii ti ). So MX (t) =
MX1 (t1 )MX2 (t2 ) for all t if and only if Σ12 = 0.

33
2 Hypothesis testing IB Statistics

Proposition. When Σ is a positive definite, then X has pdf

 n  
1 1 1 T −1
fX (x; µ, Σ) = √ exp − (x − µ) Σ (x − µ) .
|Σ|2 2π 2

Note that Σ is always positive semi-definite. The conditions just forbid the
case |Σ| = 0, since this would lead to dividing by zero.

2.5.2 Normal random samples

We wish to use our knowledge about multivariate normals to study univariate
normal data. In particular, we want to prove the following:

Theorem (Joint distribution PSXX ). Suppose X1 , · · · , Xn are iid

of X̄ and
N (µ, σ 2 ) and X̄ = n1 Xi , and SXX = (Xi − X̄)2 . Then
P

(i) X̄ ∼ N (µ, σ 2 /n)

(ii) SXX /σ 2 ∼ χ2n−1 .

(iii) X̄ and SXX are independent.

Proof. We can write the joint density as X ∼ Nn (µ, σ 2 I), where µ =√(µ, µ, · · · , µ).
Let A be an n × n orthogonal matrix with the first row all 1/ n (the other
rows are not important). One possible such matrix is

√1 √1 √1 √1 √1
 
n n n n
··· n
 √1
 2×1 √−1 0 0 ··· 0 
2×1 
√−2
 √1 √1 0 · · · 0

A=
 3×2 3×2 3×2

.. .. .. .. .

 . . . 
 . . . . . . 
−(n−1)
 
√ 1 √ 1 √ 1 √ 1
··· √
n(n−1) n(n−1) n(n−1) n(n−1) n(n−1)

Now define Y = AX. Then

Y ∼ Nn (Aµ, Aσ 2 IAT ) = Nn (Aµ, σ 2 I).

We have √
Aµ = ( nµ, 0, · · · , 0)T .
√
So Y1 ∼ N ( nµ, σ 2 ) and Yi ∼ N (0, σ 2 ) for i = 2, · · · , n. Also, Y1 , · · · , Yn are
independent, since the covariance matrix is every non-diagonal term 0.
But from the definition of A, we have
n
1 X √
Y1 = √ Xi = nX̄.
n i=1

34
2 Hypothesis testing IB Statistics

√ √
So nX̄ ∼ N ( nµ, σ 2 ), or X̄ ∼ N (µ, σ 2 /n). Also

Y22 + · · · + Yn2 = YT Y − Y12

= XT AT AX − Y12
= XT X − nX̄ 2
Xn
= Xi2 − nX̄ 2
i=1
n
X
= (Xi − X̄)2
i=1
= SXX .

So SXX = Y22 + · · · + Yn2 ∼ σ 2 χ2n−1 .

Finally, since Y1 and Y2 , · · · , Yn are independent, so are X̄ and SXX .

2.6 Student’s t-distribution

Definition (t-distribution). Suppose that Z and Y are independent, Z ∼ N (0, 1)
and Y ∼ χ2k . Then
Z
T =p
Y /k
is said to have a t-distribution on k degrees of freedom, and we write T ∼ tk .
The density of tk turns out to be
−(k+1)/2
t2

Γ((k + 1)/2) 1
fT (t) = √ 1+ .
Γ(k/2) πk k

This density is symmetric, bell-shaped, and has a maximum at t = 0, which

is rather like the standard normal density. However, it can be shown that
P(T > t) > P(Z > t), i.e. the T distribution has a “fatter” tail. Also, as k → ∞,
tk approaches a normal distribution.
Proposition. If k > 1, then Ek (T ) = 0.
k
If k > 2, then vark (T ) = k−2 .
If k = 2, then vark (T ) = ∞.
In all other cases, the values are undefined. In particular, the k = 1 case has
undefined mean and variance. This is known as the Cauchy distribution.
Notation. We write tk (α) be the upper 100α% point of the tk distribution, so
that P(T > tk (α)) = α.
Why would we define such a weird distribution? The typical application is
to study random samples with unknown mean and unknown variance. √
Let X1 , · · · , Xn be iid N (µ, σ 2 ). Then X̄ ∼ N (µ, σ 2 /n). So Z = n(X̄−µ)
σ ∼
N (0, 1).
Also, SXX /σ 2 ∼ χ2n−1 and is independent of X̄, and hence Z. So
√
n(X̄ − µ)/σ
p ∼ tn−1 ,
SXX /((n − 1)σ 2 )

35
2 Hypothesis testing IB Statistics

or √
n(X̄ − µ)
p ∼ tn−1 .
SXX /(n − 1)
We write σ̃ 2 = Sn−1
XX
(note that this is the unbiased estimator). Then a 100(1−α)%
confidence interval for µ is found from
  α  √n(X̄ − µ)  α 
1 − α = P −tn−1 ≤ ≤ tn−1 .
2 σ̃ 2

This has endpoints

σ̃ α
X̄ ± √ tn−1 .
n 2

36
3 Linear models IB Statistics

3 Linear models
3.1 Linear models
Linear models can be used to explain or model the relationship between a
response (or dependent) variable, and one or more explanatory variables (or
covariates or predictors). As the name suggests, we assume the relationship is
linear.
Example. How do motor insurance claim rates (response) depend on the age
and sex of the driver, and where they live (explanatory variables)?
It is important to note that (unless otherwise specified), we do not assume
normality in our calculations here.
Suppose we have p covariates xj , and we have n observations Yi . We assume
n > p, or else we can pick the parameters to fix our data exactly. Then each
observation can be written as

Yi = β1 xi1 + · · · + βp xip + εi . ((*))

for i = 1, · · · , n. Here
– β1 , · · · , βp are unknown, fixed parameters we wish to work out (with n > p)
– xi1 , · · · , xip are the values of the p covariates for the ith response (which
are all known).
– ε1 , · · · , εn are independent (or possibly just uncorrelated) random variables
with mean 0 and variance σ 2 .
We think of the βj xij terms to be the causal effects of xij and εi to be a random
fluctuation (error term).
Then we clearly have
– E(Yi ) = β1 xi1 + · · · βp xip .
– var(Yi ) = var(εi ) = σ 2 .
– Y1 , · · · , Yn are independent.
Note that (∗) is linear in the parameters β1 , · · · , βp . Obviously the real world
can be much more complicated. But this is much easier to work with.
Example. For each of 24 males, the maximum volume of oxygen uptake in the
blood and the time taken to run 2 miles (in minutes) were measured. We want
to know how the time taken depends on oxygen uptake.
We might get the results

Oxygen 42.3 53.1 42.1 50.1 42.5 42.5 47.8 49.9

Time 918 805 892 962 968 907 770 743
Oxygen 36.2 49.7 41.5 46.2 48.2 43.2 51.8 53.3
Time 1045 810 927 813 858 860 760 747
Oxygen 53.3 47.2 56.9 47.8 48.7 53.7 60.6 56.7
Time 743 803 683 844 755 700 748 775

37
3 Linear models IB Statistics

For each individual i, we let Yi be the time to run 2 miles, and xi be the maximum
volume of oxygen uptake, i = 1, · · · , 24. We might want to fit a straight line to
it. So a possible model is
Yi = a + bxi + εi ,
where εi are independent random variables with variance σ 2 , and a and b are
constants.
The subscripts in the equation makes it tempting to write them as matrices:
       
Y1 x11 · · · x1p β1 ε1
 ..   .. . .. .
..  , β =  ..  , ε =  ... 
.
Y =  . , X =  .
   

Yn xn1 · · · xnp . βp εn

Then the equation becomes

Y = Xβ + ε. (2)
We also have
– E(ε) = 0.
– cov(Y) = σ 2 I.
We assume throughout that X has full rank p, i.e. the columns are independent,
and that the error variance is the same for each observation. We say this is the
homoscedastic case, as opposed to heteroscedastic.
Example. Continuing our example, we have, in matrix form
     
Y1 1 x1   ε1
Y =  ...  , X =  ...
   ..  , β = a , ε =  .. 
.  b  . 
Y24 1 x24 ε24 .

Then
Y = Xβ + ε.
Definition (Least squares estimator). In a linear model Y = Xβ + ε, the least
squares estimator β̂ of β minimizes

S(β) = ∥Y − Xβ∥2
= (Y − Xβ)T (Y − Xβ)
Xn
= (Yi − xij βj )2
i=1

with implicit summation over j.

If we plot the points on a graph, then the least square estimators minimizes
the (square of the) vertical distance between the points and the line.
To minimize it, we want
∂S
=0
∂βk β=β̂

for all k. So
−2xik (Yi − xij β̂j ) = 0

38
3 Linear models IB Statistics

for each k (with implicit summation over i and j), i.e.

xik xij βˆj = xik Yi

for all k. Putting this back in matrix form, we have

Proposition. The least squares estimator satisfies

X T X β̂ = X T Y. (3)

We could also have derived this by completing the square of (Y − Xβ)T (Y −

Xβ), but that would be more complicated.
In order to find β̂, our life would be much easier if X T X has an inverse.
Fortunately, it always does. We assumed that X is of full rank p. Then

tX T Xt = (Xt)T (Xt) = ∥Xt∥2 > 0

for t ̸= 0 in Rp (the last inequality is since if there were a t such that ∥Xt∥ = 0,
then we would have produced a linear combination of the columns of X that
gives 0). So X T X is positive definite, and hence has an inverse. So

β̂ = (X T X)−1 X T Y, (4)

which is linear in Y.
We have

E(β̂) = (X T X −1 )X T E[Y] = (X T X)−1 X T Xβ = β.

So β̂ is an unbiased estimator for β. Also

cov(β̂) = (X T X)−1 X T cov(Y)X(X T X)−1 = σ 2 (X T X)−1 , (5)

since cov Y = σ 2 I.

3.2 Simple linear regression

What we did above was so complicated. If we have a simple linear regression
model
Yi = a + bxi + εi .
then we can reparameterise it to

Yi = a′ + b(xi − x̄) + εi , (6)

xi /n and a′ = a + bx̄. Since (xi − x̄) = 0, this leads to simplified

P P
where x̄ =
calculations.
In matrix form,  
1 (x1 − x̄)
X =  ... ..
.
 
.
1 (x24 − x̄)
T
P
Since (xi − x̄) = 0, in X X, the off-diagonals are all 0, and we have
 
T n 0
X X= ,
0 Sxx

39
3 Linear models IB Statistics

(xi − x̄)2 .
P
where Sxx =
Hence 1 
0
(X T X)−1 = n
1
0 Sxx
So  
T −1 T Ȳ
β̂ = (X X )X Y = SxY ,
Sxx
P
where SxY = Yi (xi − x̄).
Hence the estimated intercept is â′ = ȳ, and the estimated gradient is
Sxy
b̂ =
Sxx
P
yi (xi − x̄)
= Pi 2
i (xi − x̄)
P r
(yi − ȳ)(xi − x̄) Syy
= pP i × (∗)
S
2
P 2
(x
i i − x̄) (y
i i − ȳ) xx
r
Syy
=r× .
Sxx
P
We have (∗) since ȳ(xi − x̄) = 0, so we can add it to the numerator. Then the
other square root things are just multiplying and dividing by the same things.
So the gradient is the Pearson product-moment correlation coefficient r times
the ratio of the empirical standard deviations of the y’s and x’s (note that the
gradient is the same whether the x’s are standardised to have mean 0 or not).
Hence we get cov(β̂) = (X T X)−1 σ 2 , and so from our expression of (X T X)−1 ,
σ2 σ2
var(â′ ) = var(Ȳ ) = , var(b̂) = .
n Sxx
Note that these estimators are uncorrelated.
Note also that these are obtained without any explicit distributional assump-
tions.
Example. Continuing our previous oxygen/time example, we have ȳ = 826.5,
Sxx = 783.5 = 28.02 , Sxy = −10077, Syy = 4442 , r = −0.81, b̂ = −12.9.
Theorem (Gauss Markov theorem). In a full rank linear model, let β̂ be the
least squares estimator of β and let β ∗ be any other unbiased estimator for β
which is linear in the Yi ’s. Then
var(tT β̂) ≤ var(tT β ∗ ).
for all t ∈ Rp . We say that β̂ is the best linear unbiased estimator of β (BLUE).
Proof. Since β ∗ is linear in the Yi ’s, β ∗ = AY for some p × n matrix A.
Since β ∗ is an unbiased estimator, we must have E[β ∗ ] = β. However, since
β = AY, E[β ∗ ] = AE[Y] = AXβ. So we must have β = AXβ. Since this
∗

holds for any β, we must have AX = Ip . Now

cov(β ∗ ) = E[(β ∗ − β)(β ∗ − β)T ]
= E[(AY − β)(AY − β)T ]
= E[(AXβ + Aε − β)(AXβ + Aε − β)T ]

40
3 Linear models IB Statistics

Since AXβ = β, this is equal to

= E[Aε(Aε)T ]
= A(σ 2 I)AT
= σ 2 AAT .

Now let β ∗ − β̂ = (A − (X T X)−1 X T )Y = BY, for some B. Then

BX = AX − (X T X)−1 X T X = Ip − Ip = 0.

By definition, we have AY = BY + (X T X)−1 X T Y, and this is true for all Y.

So A = B + (X T X)−1 X T . Hence

cov(β ∗ ) = σ 2 AAT
= σ 2 (B + (X T X)−1 X T )(B + (X T X)−1 X T )T
= σ 2 (BB T + (X T X)−1 )
= σ 2 BB T + cov(β̂).

Note that in the second line, the cross-terms disappear since BX = 0.

So for any t ∈ Rp , we have

var(tT β ∗ ) = tT cov(β ∗ )t
= tT cov(β̂)t + tT BB T tσ 2
= var(tT β̂) + σ 2 ∥B T t∥2
≥ var(tT β̂).

Taking t = (0, · · · , 1, 0, · · · , 0)T with a 1 in the ith position, we have

var(β̂i ) ≤ var(βi∗ ).

Definition (Fitted values and residuals). Ŷ = X β̂ is the vector of fitted values.

These are what our model says Y should be.
R = Y − Ŷ is the vector of residuals. These are the deviations of our model
from reality.
The residual sum of squares is

RSS = ∥R∥2 = RT R = (Y − X β̂)T (Y − X β̂).

We can give these a geometric interpretation. Note that X T R = X T (Y −

Ŷ) = X T Y − X T X β̂ = 0 by our formula for β. So R is orthogonal to the
column space of X.
Write Ŷ = X β̂ = X(X T X)−1 X T Y = P Y, where P = X(X T X)−1 X T .
Then P represents an orthogonal projection of Rn onto the space spanned by
the columns of X, i.e. it projects the actual data Y to the fitted values Ŷ. Then
R is the part of Y orthogonal to the column space of X.
The projection matrix P is idempotent and symmetric, i.e. P 2 = P and
T
P = P.

41
3 Linear models IB Statistics

3.3 Linear models with normal assumptions

So far, we have not assumed anything about our variables. In particular, we
have not assumed that they are normal. So what further information can we
obtain by assuming normality?
Example. Suppose we want to measure the resistivity of silicon wafers. We
have five instruments, and five wafers were measured by each instrument (so we
have 25 wafers in total). We assume that the silicon wafers are all the same, and
want to see whether the instruments consistent with each other, i.e. The results
are as follows:

Wafer
1 2 3 4 5
1 130.5 112.4 118.9 125.7 134.0
Instrument

2 130.4 138.2 116.7 132.6 104.2

3 113.0 120.5 128.9 103.4 118.1
4 128.0 117.5 114.9 114.9 98.8
5 121.2 110.5 118.5 100.5 120.9

Let Yi,j be the resistivity of the jth wafer measured by instrument i, where
i, j = 1, · · · , 5. A possible model is

Yi,j = µi + εi,j ,

where εij are independent random variables such that E[εij ] = 0 and var(εij ) =
σ 2 , and the µi ’s are unknown constants.
This can be written in matrix form, with

1 0 ··· 0
     
Y1,1 ε1,1
 ..   .. .. . . .  .. 
 .  . . . ..   . 
     
Y1,5  1 0 · · · 0 ε1,5 
       
Y2,1  0 1 · · · 0 µ1 ε2,1 
 ..   .. .. . . ..   .. 
     
µ2 
 .  . . . .     . 
Y= Y2,5  , X = 0 1 · · · 0 , β = µ3  , ε = ε2,5 
      
    µ4   
 .  . . .
. . ... 
 . 
 ..   .. .. .

    µ5  . 
 
Y5,1  0 0 · · · 1 ε5,1 
     
 .  . . .
. . ... 
 . 
 ..   .. ..  .. 


Y5,5 0 0 ··· 1 ε5,5

Then
Y = Xβ + ε.
We have  
5 0 ··· 0
0 5 ··· 0
XT X =  .
 
.. .. .. 
 .. . . .
0 0 ··· 5

42
3 Linear models IB Statistics

Hence 1 
5 0 ··· 0
1
0
5 ··· 0
(X T X)−1 =  .
 
.. .. .. 
 .. . . .
1
0 0 ··· 5

So we have  
Ȳ1
T −1 T  .. 
µ̂ = (X X) X Y =  . 
Ȳ5
The residual sum of squares is
5 X
X 5 5 X
X 5
RSS = (Yi,j − µ̂i )2 = (Yi,j − Ȳi )2 = 2170.
i=1 j=1 i=1 j=1

Thisphas n − p = 25 − 5 = 20 degrees of freedom. We will later see that

σ̄ = RSS/(n − p) = 10.4.
Note that we still haven’t used normality!
We now make a normal assumption:

Y = Xβ + ε, ε ∼ Nn (0, σ 2 I), rank(X) = p < n.

This is a special case of the linear model we just had, so all previous results hold.
Since Y = Nn (Xβ, σ 2 I), the log-likelihood is
n n 1
l(β, σ 2 ) = − log 2π − log σ 2 − 2 S(β),
2 2 2σ
where
S(β) = (Y − Xβ)T (Y − Xβ).
If we want to maximize l with respect to β, we have to maximize the only term
containing β, i.e. S(β). So
Proposition. Under normal assumptions the maximum likelihood estimator
for a linear model is
β̂ = (X T X)−1 X T Y,
which is the same as the least squares estimator.
This isn’t coincidence! Historically, when Gauss devised the normal distri-
bution, he designed it so that the least squares estimator is the same as the
maximum likelihood estimator.
To obtain the MLE for σ 2 , we require

∂l
= 0,
∂σ 2 β̂,σ̂ 2

i.e.
n S(β̂)
− 2
+ =0
2σ̂ 2σ̂ 4

43
3 Linear models IB Statistics

So
1 1 1
σ̂ 2 = S(β̂) = (Y − X β̂)T (Y − X β̂) = RSS.
n n n
Our ultimate goal now is to show that β̂ and σ̂ 2 are independent. Then we can
apply our other standard results such as the t-distribution.
First recall that the matrix P = X(X T X)−1 X T that projects Y to Ŷ is
idempotent and symmetric. We will prove the following properties of it:

Lemma.
(i) If Z ∼ Nn (0, σ 2 I) and A is n × n, symmetric, idempotent with rank r,
then ZT AZ ∼ σ 2 χ2r .
(ii) For a symmetric idempotent matrix A, rank(A) = tr(A).

Proof.
(i) Since A is idempotent, A2 = A by definition. So eigenvalues of A are either
0 or 1 (since λx = Ax = A2 x = λ2 x).
Since A is also symmetric, it is diagonalizable. So there exists an orthogonal
Q such that

Λ = QT AQ = diag(λ1 , · · · , λn ) = diag(1, · · · , 1, 0, · · · , 0)

with r copies of 1 and n − r copies of 0.

Let W = QT Z. So Z = QW. Then W ∼ Nn (0, σ 2 I), since cov(W) =
QT σ 2 IQ = σ 2 I. Then
r
X
ZT AZ = WT QT AQW = WT ΛW = wi2 ∼ χ2r .
i=1

(ii)

rank(A) = rank(Λ)
= tr(Λ)
= tr(QT AQ)
= tr(AQT Q)
= tr A

Theorem. For the normal linear model Y ∼ Nn (Xβ, σ 2 I),

(i) β̂ ∼ Np (β, σ 2 (X T X)−1 )

σ2 2
(ii) RSS ∼ σ 2 χ2n−p , and so σ̂ 2 ∼ n χn−p .

(iii) β̂ and σ̂ 2 are independent.

The proof is not particularly elegant — it is just a whole lot of linear algebra!
Proof.

44
3 Linear models IB Statistics

– We have β̂ = (X T X)−1 X T Y. Call this CY for later use. Then β̂ has a

normal distribution with mean

(X T X)−1 X T (Xβ) = β

and covariance

(X T X)−1 X T (σ 2 I)[(X T X)−1 X T ]T = σ 2 (X T X)−1 .

So
β̂ ∼ Np (β, σ 2 (X T X)−1 ).

– Our previous lemma says that ZT AZ ∼ σ 2 χ2r . So we pick our Z and A so

that ZT AZ = RSS, and r, the degrees of freedom of A, is n − p.
Let Z = Y − Xβ and A = (In − P ), where P = X(X T X)−1 X T . We first
check that the conditions of the lemma hold:
Since Y ∼ Nn (Xβ, σ 2 I), Z = Y − Xβ ∼ Nn (0, σ 2 I).
Since P is idempotent, In − P also is (check!). We also have

rank(In − P ) = tr(In − P ) = n − p.

Therefore the conditions of the lemma hold.

To get the final useful result, we want to show that the RSS is indeed
ZT AZ. We simplify the expressions of RSS and ZT AZ and show that they
are equal:

ZT AZ = (Y − Xβ)T (In − P )(Y − Xβ) = YT (In − P )Y.

Noting the fact that (In − P )X = 0.

Writing R = Y − Ŷ = (In − P )Y, we have

RSS = RT R = YT (In − P )Y,

using the symmetry and idempotence of In − P .

Hence RSS = ZT AZ ∼ σ 2 χ2n−p . Then

RSS σ2 2
σ̂ 2 =
∼ χ .
n n n−p
   
β̂ C
– Let V = = DY, where D = is a (p + n) × n matrix.
R In − P
Since Y is multivariate, V is multivariate with

cov(V ) = Dσ 2 IDT
CC T C(In − P )T
 
2
=σ
(In − P )C T (In − P )(In − P )T
CC T
 
2 C(In − P )
=σ
(In − P )C T (In − P )
T
 
CC 0
= σ2
0 In − P

45
3 Linear models IB Statistics

Using C(In − P ) = 0 (since (X T X)−1 X T (In − P ) = 0 since (In − P )X = 0

— check!).
Hence β̂ and R are independent since the off-diagonal covariant terms are 0.
So β̂ and RSS = RT R are independent. So β̂ and σ̂ 2 are independent.
RSS
From (ii), E(RSS) = σ 2 (n − p). So σ̃ 2 = n−p is an unbiased estimator of σ 2 .
σ̃ is often known as the residual standard error on n − p degrees of freedom.

3.4 The F distribution

Definition (F distribution). Suppose U and V are independent with U ∼ χ2m
and V ∼ χ2n . The X = U/m
V /n is said to have an F -distribution on m and n degrees
of freedom. We write X ∼ Fm,n
Since U and V have mean m and n respectively, U/m and V /n are approxi-
mately 1. So F is often approximately 1.
It should be very clear from definition that
Proposition. If X ∼ Fm,n , then 1/X ∼ Fn,m .
We write Fm,n (α) be the upper 100α% point for the Fm,n -distribution so
that if X ∼ Fm,n , then P(X > Fm,n (α)) = α.
Suppose that we have the upper 5% point for all Fn,m . Using these in-
formation, it is easy to find the lower 5% point for Fm,n since we know that
P(Fm,n < 1/x) = P(Fn,m > x), which is where the above proposition comes
useful.
Note that it is immediate from definitions of tn and F1,n that if Y ∼ tn , then
Y 2 ∼ F1,n , i.e. it is a ratio of independent χ21 and χ2n variables.

3.5 Inference for β

We know that β̂ ∼ Np (β, σ 2 (X T X)−1 ). So

β̂j ∼ N (βj , σ 2 (X T X)−1

jj ).

The standard error of β̂j is defined to be

q
SE(β̂j ) = σ̃ 2 (X T X)−1
jj ,

where σ̃ 2 = RSS/(n − p). Unlike the actual variance σ 2 (X T X)−1 jj , the standard
error is calculable from our data.
Then
q
β̂j − βj β̂j − βj (β̂ j − β j )/ σ 2 (X T X)−1jj
= q = p
SE(β̂j ) −1 RSS/((n − p)σ ) 2
σ̃ 2 (X T X)jj

By writing it in this somewhat weird form, we now recognize both the numer-
ator and denominator. The numerator is a standard normal N (0, 1), and the

46
3 Linear models IB Statistics

q
denominator is an independent χ2n−p /(n − p), as we have previously shown.
But a standard normal divided by χ2 is, by definition, the t distribution. So

β̂j − βj
∼ tn−p .
SE(β̂j )

So a 100(1 − α)% confidence interval for βj has end points β̂j ± SE(β̂j )tn−p ( α2 ).
In particular, if we want to test H0 : βj = 0, we use the fact that under H0 ,
β̂j
SE(β̂j )
∼ tn−p .

3.6 Simple linear regression

We can apply our results to the case of simple linear regression. We have

Yi = a′ + b(xi − x̄) + εi ,

xi /n and εi are iid N (0, σ 2 ) for i = 1, · · · , n.

P
where x̄ =
Then we have
2
 
′ ′ σ
â = Ȳ ∼ N a ,
n
σ2
 
SxY
b̂ = ∼ N b,
Sxx Sxx
Ŷi = â′ + b̂(xi − x̄)
X
RSS = (Yi − Ŷi )2 ∼ σ 2 χ2n−2 ,
i

and (â′ , b̂) and σ̂ 2 = RSS/n are independent, as we have previously shown.
Note that σ̂ 2 is obtained by dividing RSS by n, and is the maximum likelihood
estimator. On the other hand, σ̃ is obtained by dividing RSS by n − p, and is
an unbiased estimator.
Example. Using the oxygen/time example, we have seen that
RSS 67968
σ̃ 2 = = = 3089 = 55.62 .
n−p 24 − 2

So the standard error of β̂ is

r
3089 55.6
q
2 T −1
SE(b̂) = σ̃ (X X)22 = = = 1.99.
Sxx 28.0
So a 95% interval for b has end points

b̂ ± SE(b̂) × tn−p (0.025) = 12.9 ± 1.99 ∗ t22 (0.025) = (−17.0, −8.8),

using the fact that t22 (0.025) = 2.07.

Note that this interval does not contain 0. So if we want to carry out a size
0.05 test of H0 : b = 0 (they are uncorrelated) vs H1 : b ̸= 0 (they are correlated),
the test statistic would be SE(b̂ b̂) = −12.9
1.99 = −6.48. Then we reject H0 because
this is less than −t22 (0.025) = −2.07.

47
3 Linear models IB Statistics

3.7 Expected response at x∗

After performing the linear regression, we can now make predictions from it.
Suppose that x∗ is a new vector of values for the explanatory variables.
The expected response at x∗ is E[Y | x∗ ] = x∗T β. We estimate this by x∗T β̂.
Then we have
x∗T (β̂ − β) ∼ N (0, x∗T cov(β̂)x∗ ) = N (0, σ 2 x∗T (X T X)−1 x∗ ).
Let τ 2 = x∗T (X T X)−1 x∗ . Then
x∗T (β̂ − β)
∼ tn−p .
σ̃τ
Then a confidence interval for the expected response x∗T β has end points
α
x∗T β̂ ± σ̃τ tn−p .
2
Example. Previous example continued:
Suppose we wish to estimate the time to run 2 miles for a man with an
oxygen take-up measurement of 50. Here x∗T = (1, 50 − x̄), where x̄ = 48.6.
The estimated expected response at x∗T is
x∗T β̂ = â′ + (50 − 48.5) × b̂ = 826.5 − 1.4 × 12.9 = 808.5,
which is obtained by plugging x∗T into our fitted line.
We find
1 x∗2 1 1.42
τ 2 = x∗T (X T X)−1 x∗ = + = + = 0.044 = 0.212 .
n Sxx 24 783.5
So a 95% confidence interval for E[Y | x∗ = 50 − x̄] is
α
x∗T β̂ ± σ̃τ tn−p = 808.5 ± 55.6 × 0.21 × 2.07 = (783.6, 832.2).
2
Note that this is the confidence interval for the predicted expected value,
NOT the confidence interval for the actual obtained value.
The predicted response at x∗ is Y ∗ = x∗ β + ε∗ , where ε∗ ∼ N (0, σ 2 ), and Y ∗
is independent of Y1 , · · · , Yn . Here we have more uncertainties in our prediction:
β and ε∗ .
A 100(1 − α)% prediction interval for Y ∗ is an interval I(Y) such that
P(Y ∗ ∈ I(Y)) = 1 − α, where the probability is over the joint distribution of
Y ∗ , Y1 , · · · , Yn . So I is a random function of the past data Y that outputs an
interval.
First of all, as above, the predicted expected response is Ŷ ∗ = x∗T β. This is
an unbiased estimator since Ŷ ∗ − Y ∗ = x∗T (β̂ − β) − ε∗ , and hence
E[Ŷ ∗ − Y ∗ ] = x∗T (β − β) = 0,
To find the variance, we use that fact that x∗T (β̂ − β) and ε∗ are independent,
and the variance of the sum of independent variables is the sum of the variances.
So
var(Ŷ ∗ − Y ∗ ) = var(x∗T (β̂)) + var(ε∗ )
= σ 2 x∗T (X T X)−1 x∗ + σ 2 .
= σ 2 (τ 2 + 1).

48
3 Linear models IB Statistics

We can see this as the uncertainty in the regression line σ 2 τ 2 , plus the wobble
about the regression line σ 2 . So
Ŷ ∗ − Y ∗ ∼ N (0, σ 2 (τ 2 + 1)).
We therefore find that
Ŷ ∗ − Y ∗
√ ∼ tn−p .
σ̃ τ 2 + 1
So the interval with endpoints
p α
x∗T β̂ ± σ̃ τ 2 + 1tn−p
2
∗
is a 95% prediction interval for Y . We don’t call this a confidence interval —
confidence intervals are about finding parameters of the distribution, while the
prediction interval is about our predictions.
Example. A 95% prediction interval for Y ∗ at x∗T = (1, (50 − x̄)) is
p α
x∗T ± σ̃ τ 2 + 1tn−p = 808.5 ± 55.6 × 1.02 × 2.07 = (691.1, 925.8).
2
Note that this is much wider than our our expected response! This is since there
are three sources of uncertainty: we don’t know what σ is, what b̂ is, and the
random ε fluctuation!
Example. Wafer example continued: Suppose we wish to estimate the expected
resistivity of a new wafer in the first instrument. Here x∗T = (1, 0, · · · , 0) (recall
that x is an indicator vector to indicate which instrument is used).
The estimated response at x∗T is
x∗T µ̂ = µ̂1 = y¯1 = 124.3
We find
1
τ 2 = x∗T (X T X)−1 x∗ = .
5
So a 95% confidence interval for E[Y1∗ ] is
α 10.4
x∗T µ̂ ± σ̃τ tn−p = 124.3 ± √ × 2.09 = (114.6, 134.0).
2 5
Note that we are using an estimate of σ obtained from all five instruments. If
we had only used the data from the first instrument, σ would be estimated as
sP
5
j=1 y1,j − y¯1
σ̃1 = = 8.74.
5−1
The observed 95% confidence interval for µ1 would have been
σ̃1  α 
y¯1 ± √ t4 = 124.3 ± 3.91 × 2.78 = (113.5, 135.1),
5 2
which is slightly wider. Usually it is much wider, but in this special case, we
only get little difference since the data from the first instrument is relatively
tighter than the others.
A 95% prediction interval for Y1∗ at x∗T = (1, 0, · · · , 0) is
p α
x∗T µ̂ ± σ̃ τ 2 + 1tn−p = 124.3 ± 10.42 × 1.1 × 2.07 = (100.5, 148.1).
2

49
3 Linear models IB Statistics

3.8 Hypothesis testing

3.8.1 Hypothesis testing
In hypothesis testing, we want to know whether certain variables influence the
result. If, say, the variable x1 does not influence Y , then we must have β1 = 0.
So the goal is to test the hypothesis H0 : β1 = 0 versus H1 : β1 ̸= 0. We will
tackle a more general case, where β can be split into two vectors β 0 and β 1 ,
and we test if β 1 is zero.
We start with an obscure lemma, which might seem pointless at first, but
will prove itself useful very soon.
Lemma. Suppose Z ∼ Nn (0, σ 2 In ), and A1 and A2 are symmetric, idempotent
n × n matrices with A1 A2 = 0 (i.e. they are orthogonal). Then ZT A1 Z and
ZT A2 Z are independent.

This is geometrically intuitive, because A1 and A2 being orthogonal means

they are concerned about different parts of the vector Z.
Proof. Let Xi = Ai Z, i = 1, 2 and
   
W1 A1
W= = Z.
W2 A2

Then    
0 2 A1 0
W ∼ N2n ,σ
0 0 A2
since the off diagonal matrices are σ 2 AT1 A2 = A1 A2 = 0.
So W1 and W2 are independent, which implies

W1T W1 = ZT AT1 A1 Z = ZT A1 A1 Z = ZT A1 Z

and
W2T W2 = ZT AT2 A2 Z = ZT A2 A2 Z = ZT A2 Z
are independent
Now we go to hypothesis testing
! in general linear models:
 
β0
Suppose X = X0 X1 and B = , where rank(X) = p, rank(X0 ) =
n×p n×p0 n×(p−p0 ) β1
p0 .
̸ 0. Under H0 , X1 β 1 vanishes
We want to test H0 : β 1 = 0 against H1 : β 1 =
and
Y = X0 β 0 + ε.
Under H0 , the mle of β 0 and σ 2 are

ˆ
β̂ 0 = (X0T X0 )−1 X0T Y
ˆ 2 = RSS0 = 1 (Y − X0 β̂ˆ T ˆˆ
σ̂ 0 ) (Y − X0 β 0 )
n n
and we have previously shown these are independent.

50
3 Linear models IB Statistics

Note that our poor estimators wear two hats instead of one. We adopt the
convention that the estimators of the null hypothesis have two hats, while those
of the alternative hypothesis have one.
So the fitted values under H0 are
ˆ
Ŷ = X0 (X0T X0 )−1 X0T Y = P0 Y,

where P0 = X0 (X0T X0 )−1 X0T .

The generalized likelihood ratio test of H0 against H1 is
   
√ 1 exp − 1
2 (Y − X β̂) T
(Y − X β̂)
2π σ̂ 2 2σ̂
ΛY (H0 , H1 ) =  
ˆ ˆ
 
√1 exp − 2σ̂1ˆ 2 (Y − X β̂ 0 )T (Y − X β̂ 0 )
ˆ2
2π σ̂
!n/2
ˆ2
σ̂
=
σ̂ 2
 n/2
RSS0
=
RSS
 n/2
RSS0 − RSS
= 1+ .
RSS
0 −RSS
We reject H0 when 2 log Λ is large, equivalently when RSSRSS is large.
Using the results in Lecture 8, under H0 , we have
 
RSS0 − RSS
2 log Λ = n log 1 + ,
RSS

which is approximately a χ2p1 −p0 random variable.

This is a good approximation. But we can get an exact null distribution, and
get an exact test.
We have previously shown that RSS = YT (In − P )Y, and so

RSS0 − RSS = YT (In − P0 )Y − YT (In − P )Y = YT (P − P0 )Y.

Now both In − P and P − P0 are symmetric and idempotent, and therefore

rank(In − P ) = n − p and

rank(P − P0 ) = tr(P − P0 ) = tr(P ) − tr(P0 ) = rank(P ) − rank(P0 ) = p − p0 .

Also,

(In − P )(P − P0 ) = (In − P )P − (In − P )P0 = (P − P 2 ) − (P0 − P P0 ) = 0.

(we have P 2 = P by idempotence, and P P0 = P0 since after projecting with P0 ,

we are already in the space of P , and applying P has no effect)
Finally,

YT (In − P )Y = (Y − X0 β 0 )T (In − P )(Y − X0 β 0 )

YT (P − P0 )Y = (Y − X0 β 0 )T (P − P0 )(Y − X0 β 0 )

51
3 Linear models IB Statistics

since (In − P )X0 = (P − P0 )X0 = 0.

If we let Z = Y − X0 β 0 , A1 = In − P , A2 = P − P0 , and apply our previous
lemma, and the fact that ZT Ai Z ∼ σ 2 χ2r , then

RSS = YT (In − P )Y ∼ χ2n−p

RSS0 − RSS = YT (P − P0 )Y ∼ χ2p−p0

and these random variables are independent.

So under H0 ,
YT (P − P0 )Y/(p − p0 ) (RSS0 − RSS)/(p − p0 )
F = = ∼ Fp−p0 ,n−p ,
YT (In − P )Y/(n − p) RSS/(n − p)
Hence we reject H0 if F > Fp−p0 ,n−p (α).
RSS0 − RSS is the reduction in the sum of squares due to fitting β 1 in
addition to β 0 .

Source of var. d.f. sum of squares mean squares F statistic

RSS0 −RSS
Fitted model p − p0 RSS0 − RSS p−p0 (RSS0 −RSS)/(p−p0 )
RSS RSS/(n−p)
Residual n−p RSS n−p

Total n − p0 RSS0
0 −RSS
The ratio RSSRSS 0
is sometimes known as the proportion of variance explained
by β 1 , and denoted R2 .

3.8.2 Simple linear regression

We assume that
Yi = a′ + b(xi − x̄) + εi ,
xi /n and εi are N (0, σ 2 ).
P
where x̄ =
Suppose we want to test the hypothesis H0 : b = 0, i.e. no linear relationship.
We have previously seen how to construct a confidence interval, and so we could
simply see if it included 0.
Alternatively, under H0 , the model is Yi ∼ N (a′ , σ 2 ), and so â′ = Ȳ , and the
fitted values are Ŷi = Ȳ .
The observed RSS0 is therefore
X
RSS0 = (yi − ȳ)2 = Syy .
i

The fitted sum of squares is therefore

X X
(yi − ȳ)2 − (yi − ȳ − b̂(xi − x̄))2 = b̂2 (xi − x̄)2 = b̂2 Sxx .


RSS0 − RSS =
i

Source of var. d.f. sum of squares mean squares F statistic

2 2
Fitted model 1 RSS0 − RSS
P = b̂ SXX b̂ Sxx b̂2 Sxx
Residual n−2 RSS = i (yi − ŷ)2 σ̃ 2 . F = σ̃ 2

RSS0 = i (yi − ȳ)2

P
Total n−1

52
3 Linear models IB Statistics

2
Sxy
Note that the proposition of variance explained is b̂2 Sxx /Syy = Sxx Syy = r2 ,
where r is the Pearson’s product-moment correlation coefficient
Sxy
r= p .
Sxx Syy
√
We have previously seen that under H0 , SE(b̂ b̂) ∼ tn−2 , where SE(b̂) = σ̃/ Sxx .
So we let √
b̂ b̂ Sxx
t= = .
SE(b̂) σ̃
Checking whether |t| > tn−2 α2 is precisely the same as checking whether

t2 = F > F1,n−2 (α), since a F1,n−2 variable is t2n−2 .

Hence the same conclusion is reached, regardless of whether we use the
t-distribution or the F statistic derived form an analysis of variance table.

3.8.3 One way analysis of variance with equal numbers in each group
Recall that in our wafer example, we made measurements in groups, and want to
know if there is a difference between groups. In general, suppose J measurements
are taken in each of I groups, and that

Yij = µi + εij ,

where εij are independent N (0, σ 2 ) random variables, and the µi are unknown
constants.
Fitting this model gives
I X
X J I X
X J
RSS = (Yij − µ̂i )2 = (Yij − Ȳi. )2
i=1 j=1 i=1 j=1

on n − I degrees of freedom.
Suppose we want to test the hypothesis H0 : µi = µ, i.e. no difference between
groups.
Under H0 , the model is Yij ∼ N (µ, σ 2 ), and so µ̂ = Ȳ , and the fitted values
are Yˆij = Ȳ .
The observed RSS0 is therefore
X
RSS0 = (yij − ȳ.. )2 .
i,j

The fitted sum of squares is therefore

XX X
(yij − ȳ.. )2 − (yij − ȳi. )2 = J (ȳi. − ȳ.. )2 .


RSS0 − RSS =
i j i

Source of var. d.f. sum of squares mean squares F statistic

−ȳ.. )2
(ȳ − ȳ.. )2 J i (ȳi.I−1
P P
Fitted model I −1 J (ȳi. −ȳ.. )2
P Pi i
P
2 J
Residual n−I i j (yij − ȳi. ) σ̃ 2 i (I−1)σ̃ 2

2
P P
Total n−1 i j (yij − ȳ.. )

53