The Multivariate Normal Distribution

3 The Multivariate Normal Distribution

3.1 Introduction
A generalization of the familiar bell shaped normal density to several
dimensions plays a fundamental role in multivariate analysis

The distribution can be completely described using only means,

variances, and covariances
While real data are never exactly multivariate normal, the normal
density is often a useful approximation to the true population
distribution because of a central limit effect

The multivariate normal distribution is mathematically tractable and

”nice” results can be obtained.

The importance of the normal distribution rests on its dual role as

both population model for certain natural phenomena and
approximate sampling distribution for many statistics.
Abiyot. (JU) Multivariate statistical Methods 46
 The Multivariate Normal Distribution The Multivariate Normal density and Its Properties

3.2 The Multivariate Normal density and Its Properties

Recall that the univariate normal distribution, with mean µ and

variance σ 2 , has the probability density function
1 2 /2
f (x) = √ e −[(x−µ)/σ] −∞<x <∞
2πσ 2
The term  2
x −µ
= (x − µ)(σ 2 )−1 (x − µ)
σ
This can be generalized for p × 1 vector x of observations on serval
variables as
(x − µ)0 Σ−1 (x − µ)
The p × 1 vector µ represents the expected value of the random
vector x, and the p × p matrix Σ is the variance-covariance matrix of
x.
Abiyot. (JU) Multivariate statistical Methods 47
 The Multivariate Normal Distribution The Multivariate Normal density and Its Properties

A p-dimensional normal density for the random vector

X0 = [X1 , X , · · · , Xp ] has the form
1 0 −1
f (x) = e −(x−µ) Σ (x−µ)/2
(2π)p/2 |Σ|1/2
Where −∞ < xi < ∞, i = 1, 2, · · · , p
We should denote this p-dimensional normal density by Np (µ, Σ)

Figure: A normal density with mean µ and variance σ 2 and selected areas under
the curve

Abiyot. (JU) Multivariate statistical Methods 48

 The Multivariate Normal Distribution Bivariate Normal Density

3.3 Bivariate Normal Density

Let us evaluate the p = 2 variate normal density in terms of the

individual parameters
µ1 = E (X1 ), µ2 = E (X2 ), σ12 = var (X1 ), σ22 = var (X2 ), and
ρ= Covσ(X 1 ,X2 )
1 σ2
= Corr (X1 , X2 )

1 − 1
f (x1 , x2 ) = p e 2(1−ρ2 )
2πσ1 σ2 1 − ρ2
(x1 − µ1 )2 (x1 − µ1 )(x2 − µ2 ) (x2 − µ2 )2
 
× − 2ρ +
σ12 σ1 σ2 σ22

Note that |Σ| = (σ1σ2 )2 (1 − ρ2 ) and


σ22 −ρσ1 σ2
Σ−1 = 1
(1−ρ2 )(σ1 σ2 )2
 
−ρσ1 σ2 σ12

Abiyot. (JU) Multivariate statistical Methods 49

 The Multivariate Normal Distribution Bivariate Normal Density

Figure: Two bivariate normal distributions (a) σ11 = σ22 and ρ12 = 0 (b)
σ11 = σ22 and ρ12 = 0.75
Abiyot. (JU) Multivariate statistical Methods 50
 The Multivariate Normal Distribution Bivariate Normal Density

Figure: Contour plots for the bivariate normal distributions

Abiyot. (JU) Multivariate statistical Methods 51

 The Multivariate Normal Distribution Elliptically contoured distribution

Constant probability density contour

An elliptically contoured distribution is one whose density is given by
f (x|µ, Σ, g ) = |Σ|−1/2 g [(x − µ)0 Σ−1 (x − µ)]
where µ ∈ Rp , Σ > 0, g (·) ≥ 0 and
Z
g (z0 z) = 1
Rp
By varying the function g, distributions shorter or longer tails than a
normal distribution can be obtained.
= {all x such that (x − µ)0 Σ−1 (x − µ) = c 2 }
= surface of an ellipsoid centered at µ
Contours of constant density for the p-dimensional normal
distribution are ellipsoids defined by x such that
(x − µ)0 Σ−1 (x − µ) = c 2
Abiyot. (JU) Multivariate statistical Methods 52
 The Multivariate Normal Distribution Elliptically contoured distribution

√
These ellipsoids are centered at µ and have axes ±c λi ei , where
where (λi , ei ) are the eigenvalue-eigenvector pairs of Σ.
This shows why Σ can be a measure of spread
The solid ellipsoid of x values satisfying

(x − µ)0 Σ−1 (x − µ) ≤ χ2p (α)

has probability 1 − α where χ2p (α) is the upper (100α)th percentile of

a chi-square distribution with p degrees of freedom.
p
The half lengths of the axes are c λj in the direction of
ej , j = 1, 2, · · · , p.
Note that most of the properties of multivariate normal distribution
analogously carry over to elliptically-contoured distributions

Abiyot. (JU) Multivariate statistical Methods 53

 The Multivariate Normal Distribution Further Properties of the Multivariate Normal Density

Further Properties of the Multivariate Normal Density

1 μ Σ), then any linear combination of
If x is distributed as Np (µ,
variables a0 x = a1 X1 + a2 X2 + · · · + ap Xp is distributed as
N(a0 µ, a0 Σa).
Also if a0 x is distributed as N(a0 µ, a'Σa) for every a, then x must be
Np (µ, Σ):
2 If x is distributed as Np (µ, Σ), then Z = Ax ∼ Nq (Aµ, AΣA0 ) for any
q × p matrix A.
3 Partition
     
xq×1 µq×1 Σ11 Σ12
x= , then µ= , Σ =  
x(p−q)×1 µ(p−q)×1 Σ21 Σ22

 
Let A0 = Iq 0q×(p−q) .Then xq×1 ∼ Nq (µq×1 , Σ11 )
4 xq×1 ⊥ x(p−q)×1 if and only if Σ12 = 0
Abiyot. (JU) Multivariate statistical Methods 54
 The Multivariate Normal Distribution Further Properties of the Multivariate Normal Density

Further Properties...

The following are true for a normal vector X having a multivariate normal
distribution:
1 Linear combination of the components of X are normally distributed.
2 All subsets of the components of X have a (multivariate) normal
distribution.
3 Zero covariance implies that the corresponding components are
independently distributed.
4 The conditional distributions of the components are normal.

Abiyot. (JU) Multivariate statistical Methods 55

 The Multivariate Normal Distribution Further Properties of the Multivariate Normal Density

Conditional distribution

Given x(2) the conditional distribution of x(1) is

Nq (µ1.2 , Σ11.2 )

where µ1.2 = µ(1) + Σ12 Σ−1

22 (x
(2) − µ(2) ), Σ −1
11.2 = Σ11 − Σ12 Σ22 Σ21 .

Note that the covariance does not depend on the value x(2) of the
conditioning variable.
Example: Bivariate case: x1 is the height of the son. x2 is the height
of the father.
     
1 x 1 µ σ12 ρσ1 σ2
x =  , µ =  , Σ =  
x2 µ2 ρσ1 σ2 σ22

x1 |x2 ∼ N(µ1.2 , Σ11.2 )

Proof?
σ1 ρ
where µ1.2 = µ1 + σ2 (x2 − µ2 ), Σ11.2 = σ12 (1 − ρ2 )
Abiyot. (JU) Multivariate statistical Methods 56
 The Multivariate Normal Distribution Further Properties of the Multivariate Normal Density

Sampling from a Multivariate Normal Distribution and

Maximum Likelihood Estimation
The Multivariate Normal Likelihood
Joint density function of all p × 1 observed random vectors X1 , X2 , . . . , Xn
1
e {− 2 (Xi −µ) Σ }
1 0 −1 (X −µ)
f (X1 , X2 , . . . , Xn ) = Πni=1 p 1
i

(2π) |Σ| 2 2

1 {− 12 ni=1 (Xi −µ)0 Σ−1 (Xi −µ)}

P
= np n e
(2π) 2 |Σ| 2
1 −tr [ 12 Σ−1 { ni=1 (Xi −x)(Xi −x)0 +n(x−µ)(x−µ)0 }]
P
= np n e
(2π) 2 |Σ| 2

np n
ln{f (X1 , X2 , . . . , Xn )} = − ln(2π) + ln(|Σ|−1 |)
2 2
" ( n )#
1 −1 X
− tr Σ (Xi − x)(Xi − x)0 + n(x − µ)(x − µ)0
2 i=1

Abiyot. (JU) Multivariate statistical Methods 57

 The Multivariate Normal Distribution Further Properties of the Multivariate Normal Density

Likelihood

When the numerical values of the observations become available, they may
be substituted for the xj in the equation above. The resulting expression,
now considered as a function of μ and Σ for the fixed set of observations
x1 , x2 , . . . , xn , is called the likelihood.

maximum likelihood estimation

One meaning of best is to select the parameter values that maximize the
joint density evaluated at the observations. This technique is called
maximum likelihood estimation, and the maximizing parameter values are
called maximum likelihood estimates.

Let A be a k × k symmetric matrix and x be a k × 1 vector. Then

x0 Ax = tr (x0 Ax) = tr (Axx0 )
p
X
tr (A) = λi , where the λi are the eigenvalues ofA
i=1
. Abiyot. (JU) Multivariate statistical Methods 58
 The Multivariate Normal Distribution Further Properties of the Multivariate Normal Density

Maximum Likelihood Estimate of µ and Σ

∂ln{f (X1 , X2 , . . . , Xn )} ∂ln{f (X1 , X2 , . . . , Xn )}

= 0, and =0
∂µ ∂Σ
Thus
n
1X
µ̂ = xi = x
n
i=1

n
1X
Σ̂ = (xi − x)(xi − x)0
n
i=1
n−1
= S
n
are the maximum likelihood estimators of µ and Σ, respectively. Their
1 Pn
observed value x and n i=1 (xi − x)(xi − x) are called the maximum
likelihood estimates of µ and Σ

Abiyot. (JU) Multivariate statistical Methods 59

 The Multivariate Normal Distribution Sampling distribution

Sampling distribution

The univariate case (p = 1)

a
1
X ∼ N(µ, σ 2 )
n
b
(n − 1)S 2
∼ χ2 (n − 1) wheren > 1 and σ 2
σ2
are independent where X = n1 ni=1 Xi and
P
c X and S P
1 n 2
S = n−1 i=1 (Xi − X )

Abiyot. (JU) Multivariate statistical Methods 60

 The Multivariate Normal Distribution Sampling distribution

Wishart distribution
Let X1 , · · · , XN be a random sample from Np (µ, Σ). Denote
N
X
A = (N − 1)S = (Xi − x)0 (Xi − x)
i=1

d Pn 0
Let n = N − 1. It can be shown that A = i=1 zi zi where
iid
zi ∼ Np (0, Σ)
Hence A ∼ Wp (n, Σ), a Wishart distribution with n d.f whose density
is given by
n−p−1
|A| 2 exp{− 12 tr (Σ−1 A)}
np n , (n ≥ p)
2 2 |Σ| 2 Γp ( n2 )
where
1
Γp (t) = π p(p−1)/4 Πpi=1 Γ(t − (i − 1))
2
is the multivariate gamma function
Abiyot. (JU) Multivariate statistical Methods 61
 The Multivariate Normal Distribution Sampling distribution

Properties...

If A1 , · · · , Aq are independently distributed with Aj ∼ W (ni , Σ), then

q
X X 
A= Aj ∼ W nj , Σ
j=1

If A ∼ W (n, Σ)), then for any q × p matrix C,

B = CAC0 ∼ W (n, CΣC0 )

Partition    
A11 A12 Σ11 Σ12
A= , Σ =  
A21 A22 Σ21 Σ22

A11 ∼ W (n, Σ11 )

Abiyot. (JU) Multivariate statistical Methods 62

 The Multivariate Normal Distribution Sampling distribution

The Sampling Distribution of X and S

Let X1 , X2 , · · · , Xn be a random sample size n from a p-variate normal

distribution with mean µ and covariance matrix Σ. Then
1 X is distributed as Np (µ, n1 Σ)

2 (n − 1)S is distributed as a Wishart random matrix with n − 1 d.f

3 X and S are independent

Large-Sample Behaviour of X and S
√
n(X − µ) is approximately Np (0, Σ)

and
n(X − µ)0 S(X − µ) is approximately χ2p
for n − p large.

Abiyot. (JU) Multivariate statistical Methods 63