26

Chapter 7. Multivariate Linear Regression Models.

Multivariate linear regression model is essentially several univariate linear regression models putting
together, with the errors being related with each. Here univariate (multivariate) means the response
variables are univariate (multivariate). The notations of this chapter may be nasty.

7.1. Univariate linear regression — a review

(i). Model description:
Yi = β0 + β1 zi1 + · · · βr zir + ²i i = 1, ..., n,
where, for the i-th subject, Yi is the response, zi1 , ..., zik are covariates, β0 , ..., βr are regression
parameters and ²i is the error.
An alternative presentation of the model in matrix form is:

Yn×1 = Zn×(r+1) β(r+1)×1 + ²n×1

where  
  1 z11 ··· z1r    
Y1 β0 ²1
.  1 z21 ··· z2r   .. 
Y =  ..  , Z =  ..  , β = and ² =  ...  .
 ... ..
.
..
. . 
 . 
Yn βr ²n
1 zn1 ··· znr

(ii). Model assumptions:

The errors ²1 , ..., ²n are iid ∼ N (0, σ 2 ) and the covariates are either non-random or are independent
of the errors.
(iii). The least squares estimation (LSE):
The LSE of β is
β̂ = (Z0 Z)−1 Z0 Y = argmin{b ∈ Rr+1 : kY − Zbk2 },
which is unbiased, maximum likelihood estimator of β. A geometric interpretation is that Zβ̂ is
the projection of Y in Rn onto the linear space {Zb ∈ Rn : b ∈ Rr+1 }. The residuals are
 
²̂1
.
²̂ =  ..  = Y − Zβ̂.
²̂n

Then, for i = 1, ..., n,

²̂i = Yi − β̂0 − β̂1 zi1 − · · · − β̂r zir .
The estimator of σ 2 is
X n
1 1
s2 ≡ ²̂0 ²̂ = ²̂2 ,
n−r−1 n − r − 1 i=1 i

which is an unbiased estimator.

(iv). Inferences:
Observe that
β̂ = β + (Z0 Z)−1 Z 0 ².
Inference is based on the facts that
n
1 X 2 s2
β̂ − β ∼ M N (0, (Z0 Z)−1 σ 2 ), ²i = (n − r − 1) ∼ χ2n−r−1
σ 2 i=1 σ2

and that β̂ is independent of s2 .

 27

Confidence interval for βk at confidence level 1 − α:

√
β̂k ± tn−r−1 (α/2)s akk ,

where alm , 0 ≤ l, m ≤ r are the entries of the matrix (Z0 Z)−1 , i.e.,
 
a00 a01 · · · a0r
 a10 a11 · · · a1r 
(Z0 Z)−1 =  ... .. .. ..  .
. . . 
ar0 ar1 ··· arr
This confidence interval is based on
β̂k − βk
√ ∼ tn−r−1 , k = 0, ..., r.
akk s

One might also construct T 2 -type confidence region at confidence level 1 − α for β as a vector of
r + 1 dimension:
© o
β ∈ Rr+1 : (β̂ − β)0 (Z0 Z)(β̂ − β)/s2 ≤ (r + 1)Fr+1,n−r−1 (α)

based on the fact

(β̂ − β)0 (Z0 Z)(β̂ − β)/s2 ∼ (r + 1)Fr+1,n−r−1 .
The T 2 -type simultaneous confidence intervals for βk , k = 0, ..., r at confidence level 1 − α are
√
for βk : β̂k ± Fr+1,n−r−1 (α)s akk , k = 0, 1, ..., r.

Bonferroni’s simultaneous confidence intervals for βk , k = 0, ..., r at confidence level 1 − α are

α √
for βk : β̂k ± tn−r−1 ( )s akk , k = 0, 1, ..., r.
2(r + 1)

(v). Prediction.
Let  
1
 z01 
 
Z0 =  z 
 02 
 ... 
z0r
be any given value of covariates. Let Y0 = Z00 β + ²0
be the response of an experiment not yet carried
out. We wish to predict the response Y0 or estimate its mean Z00 β with accuracy justification. The
predictor or estimator is naturally Z00 β̂, which is unbiased. Moreover,
³ ´ ³ ´
Z00 β̂ − Z00 β ∼ N 0, σ 2 Z00 (Z0 Z)−1 Z0 and Z00 β̂ − Y0 ∼ N 0, σ 2 [1 + Z00 (Z0 Z)−1 Z0 ] ,

by noticing that

Z00 β̂ − Z00 β = Z00 (Z0 Z)−1 Z² and Z00 β̂ − Y0 = Z00 (Z0 Z)−1 Z² + ²0 .

Then, the prediction interval (or just call it confidence interval if you like) for Y0 at confidence level
1 − α is q
Z00 β̂ ± tn−r−1 (α/2)s 1 + Z00 (Z0 Z)−1 Z0 .
And confidence interval for Z00 β, the mean of Y0 , at confidence level 1 − α is
q
Z00 β̂ ± tn−r−1 (α/2)s Z00 (Z0 Z)−1 Z0 .
28

We note that this confidence interval still holds when the first component of Z0 takes any value,
not just 1.

7.2. Multivariate linear regression.

Example 1.1 (continued) Ten American companies Data containing three variables: sales,
profit and asset, and ten companies as observations is given in Appendix A. It is reasonable to
believe that the size of business, measured by sales, and the profitability, measured by profit, both
depends on the size of the company, measured by asset. One might consider two linear regression
models. One is sales regressed on asset and the other is profit regressed on asset. As the errors of the
two models shall inevitably be related, this presents an example of multivariate (actually bivariate
here) linear regression model by putting together the two univariate linear regression models, this
can be jointly presented as

Yi1 = β01 + β11 zi + ²i1 , i = 1, ..., 10

Yi2 = β02 + β12 zi + ²i2 , i = 1, ..., 10

where Yi1 , Yi2 , zi are sales, profit and asset of company i.

Another simple example is a multivariate linear regression model relating students academic perfor-
mance measured by midterm and final exam scores with diligence measured by weekly time devoted
to study and ingenuity measured by IQ score:
½
Yi1 = β01 + β11 zi1 + β21 zi2 + ²i1 ,
i = 1, ..., n
Yi2 = β02 + β12 zi1 + β22 zi2 + ²i2

where Yi1 , Yi2 zi1 and zi2 are, respectively, midterm exam score, final exam score, weekly hours of
time devoted to study and IQ score of the i-th randomly selected student.
(i). Model description:
The k-th univariate linear regression model:

Yik = β0k + zi1 β1k + · · · + zir βrk + ²ik , i = 1, ..., n,

where Yik is the k-th response for subject i. Writing in matrix form:

Y(k) = Zβ(k) + ²(k) ,

where      
Y1k β0k ²1k
.   .
Y(k) =  ..  , β(k) =  ...  and ²(k) =  .. 
Ynk βrk ²nk
The so called multivariate linear regression model is to put the m univariate linear regression model
together as
Yn×m = Zn×(r+1) β(r+1)×m + ²n×m ,
which is the same as
. . . . . .
[Y(1) .. · · · ..Y(m) ] = Z[β(1) .. · · · ..β(m) ] + [²(1) .. · · · ..²(m) ]

One distinct feature here is that the errors, ²i1 , ..., ²im , within one observation/subject may be
related with each other. This suggest that the m univariate linear regression models should be
treated together as one model rather than separately treated as m individual univariate models.
(ii). Model assumptions:
The errors (²i1 , ..., ²im )0 , as random vectors of m dimension, are iid ∼ M N (0, Σ) and are independent
of the covariates. Note that here Σ is an m × m matrix.
 29

(iii). Estimation:
The least squares criterion for the multivariate linear regression model is to minimize over β
n h
m X
X i2 X
m
Yik − (β0k + zi1 β1k + · · · + zir βrk ) = kY(k) − Zβ(k) k2 .
k=1 i=1 k=1

The LSE of β is denoted as

. .
β̂ = [β̂(1) .. · · · ..β̂(m) ].
Notice that
m
X m
X
min kY(k) − Zβ(k) k2 min kY(k) − Zβ(k) k2 ,
β β(k)
k=1 k=1

i.e.,
X n
m X r
X m h
X n
X r
X i
2
min (Yik − [β0k + zil βlk ]) = min (Yik − [β0k + zil βlk ])2 .
β β(k)
k=1 i=1 l=1 k=1 i=1 l=1

Hence β̂(k) is the same as the LSE of β(k) of the univariate linear regression model: Y(k) = Zβ(k) +
²(k) . Therefore,

β(k) = (Z0 Z)−1 Z0 Y(k) , k = 1, ..., m, and

.. .. . .
β̂ = [β̂(1) . · · · .β̂(m) ] = (Z0 Z)−1 Z0 [Y(1) .. · · · ..Y(m) ]Y = (Z0 Z)−1 Z0 Y,

which is formally the same as the expression of the LSE of univariate linear regression model. The
residuals are
²̂n×m = Yn×m − Zn×(r+1) β̂(r+1)×m .
and the estimator of Σ is
 
s11 ··· s1m
 .. .. ..  1
S≡ . . . = ²̂0 ²̂.
n−r−1
sm1 ··· smm

(iv). Inference with β(k) .

It is often of interest to consider inference with regression parameter, say β(k) , for one single uni-
variate regression model. All theory, estimation and inference procedures with univariate regression
model as presented in Section 7.1 apply here. For example,

β̂(k) − β(k) ∼ M N (0, σkk (Z0 Z)−1 )

where σkl is the (k, l)-th entry of Σ.

(β̂(k) − β(k) )(Z0 Z)(β̂(k) − β(k) )

∼ (r + 1)Fr+1,n−r−1
skk

where skl is the (k, l)-th entry of S. Then, a confidence region at confidence level 1 − α for β(k) , as
a vector in Rr+1 , can be constructed as
n (β̂(k) − β(k) )(Z0 Z)(β̂(k) − β(k) ) o
β(k) ∈ Rr+1 : ∼ (r + 1)Fr+1,n−r−1 (α) .
skk
For more detail, such as confidence intervals for βjk , simultaneous confidence intervals for β0k , ..., βrk ,
please refer to Section 7.1 about univariate linear regression models.
(v). Prediction.
30

Let  
1  
²01
 z01  .. 
Y0 = β 0 Z0 + ²0 = β 0   
 ...  + .
²0m
z0r
be a new experiment with covariate Z0 which has not been carried out. We are interested in
predicting its response Y0 , an m-vector, or estimating its mean β 0 Z0 , with accuracy justification.
The predictor of Y0 or estimator of its mean β 0 Z0 is naturally β̂ 0 Z0 . To compute the variance,
observe that, for any non-random n-vector V = (v1 , ..., vn )0 ,

Xn n
X n
X
0 2
var(² V ) = var( vi ² i ) = vi var(²i ) = Σ vi2 = kV k2 Σ == V 0 V Σ,
i=1 i=1 j=1

where ²i , an m-vector, is the i-row of ² representing the m errors for the i-th subject/observation.
Let V = Z(Z0 Z)−1 Z0 . Then,

var(Y0 − β̂ 0 Z0 ) = var(²0 − (β̂ − β)0 Z0 ) = var(²0 ) + var((β̂ − β)0 Z0 )

= Σ + var([(Z0 Z)−1 Z0 ²]0 Z0 ) = Σ + var(²0 Z(Z0 Z)−1 Z0 ) = Σ + [Z00 (Z0 Z)−1 Z0 ]Σ

For simplicity of notation, let

κ = Z00 (Z0 Z)−1 Z0 .
In summary

var(Y0 − β̂00 Z0 ) = Σ(1 + κ), var(β 0 Z0 − β̂00 Z0 ) = κΣ

1
√ Σ−1/2 (Y0 − β̂00 Z0 ) ∼ M N (0, Im )
1+κ
1
and √ Σ−1/2 (β 0 Z0 − β̂00 Z0 ) ∼ M N (0, Im ).
κ

Moreover,

1 0 (n − r − 1)m
T2 ≡ [β̂ Z0 − β 0 Z0 ]0 S−1 [β̂ 0 Z0 − β 0 Z0 ] ∼ Fm,n−r−m
κ n−r−m
1 (n − r − 1)m
and [Y0 − β 0 Z0 ]0 S−1 [Y0 − β 0 Z0 ] ∼ Fm,n−r−m ,
1+κ n−r−m
which provide theoretical foundation for the following inference procedures:
(1). Confidence region for β 0 Z0 = E(Y0 ) at level 1 − α:
n (n − r − 1)m o
β 0 Z0 ∈ Rm : [β̂ 0 Z0 − β 0 Z0 ]0 S−1 [β̂ 0 Z0 − β 0 Z0 ] ≤ κ Fm,n−r−m (α)
n−r−m

(2). Prediction region (or just call it confidence region) for Y0 at level 1 − α:
n (n − r − 1)m o
Y0 ∈ Rm : [Y0 − β 0 Z0 ]0 S−1 [Y0 − β 0 Z0 ] ∼ (1 + κ) Fm,n−r−m (α) .
n−r−m
,
0
(3). Simultaneous confidence intervals for β(k) Z0 = E(Y0k ), k = 1, ..., m at confidence level 1 − α:
r
0 0 (n − r − 1)m
for β(k) Z0 : β̂(k) Z0 ± skk · κ Fm,n−r−m (α) k = 1, ..., m.
n−r−m
 31

(4). Simultaneous prediction/confidence intervals for Y0k , k = 1, ..., m at confidence level 1 − α:

r
0 (n − r − 1)m
for Y0k : β̂(k) Z0 ± skk (1 + κ) Fm,n−r−m (α) k = 1, ..., m.
n−r−m

Example 1.1 (continued) Ten American Companies Sales (Y1 ) and profit (Y2 ) are two responses
regressed on the covariate asset (Z). The multivariate linear regression is the two univariate linear
regression models putting together:

Yi1 = β01 + β11 zi + ²i1 , i = 1, ..., 10

Yi2 = β02 + β12 zi + ²i2 , i = 1, ..., 10

An expression in matrix form is

     
Y11 Y12 1 z1 µ ¶ ²11 ²12
 ... ..   .. ..  β01 β02 . .. 
. = . . +  .. . .
β11 β12
Yn1 Yn2 1 zn ²n1 ²n2
Y = Zβ + ².
In the notations used above, n = 10, m = 2 and r = 1. And
µ ¶ µ ¶
10 812.484 0.34609 −0.00303
Z0 Z = and (Z0 Z)−1 =
812.484 92837.4 −0.00303 0.0000373

(1). The least squares estimator of β:

µ ¶ µ ¶
β̂01 β̂02 0 −1 0 21.09 1.683
β̂ = = (Z Z) Z Y =
β̂11 β̂12 0.507 0.015

(2) Variance estimation:

The estimator of Σ, the 2 × 2 matrix of the variance of the error, is
µ ¶
262.85 2.72
S= .
2.72 0.822

It’s easy to compute µ ¶

0.00394 −0.01304
S−1 = .
−0.01304 1.2597

(3) Inference with regression parameters of one univariate regression model.

Inferences about β01 , β11 , β(1) = (β01 , β11 )0 , predicting sales for a given company with known asset
size or estimating mean sales for companies with a given size of asset are all only related with the first
univariate linear regression model. Likewise, inferences about β02 , β12 , β(2) = (β02 , β12 )0 , predicting
profit for a given company with known asset size or estimating mean profit for companies with a
given size of asset are all only related with the second univariate linear regression model. Then,
inference methods reviewed in Section 7.1 apply. For example, a confidence interval at confidence
level 95% for β11 is
√ √
β̂11 ± t10−2 (0.025) 0.0000373 × s11 = 0.507 ± t8 (.025) .0000373 × 262.85 = .507 ± .228;

and a confidence region at confidence level 95% for β(1) = (β01 , β11 )0 is

{β(1) ∈ R2 : (β̂(1) − β(1) )0 (Z0 Z)(β̂(1) − β(1) ) ≤ 2344}.

(4). Prediction of both sales and profit.

32

Suppose company ABC has 50 billion dollars of asset. We wish to predict both sales and profit
of the company with accuracy justification. Set Z0 = (1 50)0 and denote by Y01 and Y02 the yet
unknown sales and profit of the company. The estimator is
µ ¶ µ ¶
β̂01 + 50β̂11 46.45
β̂ 0 Z0 = =
β̂02 + 50β̂12 2.45

Note that κ = 0.1364. A prediction region at confidence level 95% is

n µ ¶ µ ¶ µ ¶ µ ¶ o
£ S 46.45 ¤0 −1 £ S 46.45 ¤
(S, P) : − S − ≤ 12.31
P 2.45 P 2.45

where S stands for sales and P for profit of the company. And simultaneous prediction intervals at
(nominal) confidence level 95% is

for sales : 46.45 ± 58.87

for profit : 2.45 ± 3.18.

On the other hand, suppose we are interested in providing an estimation and inference for the mean
sales and profit for all companies with asset 50 billion dollars, which is
µ ¶ µ ¶
EY01 β01 + 50β11
=
EY02 β02 + 50β12

The estimator is still same as (46.45, 2.45)0 . A confidence region at confidence level 95% is
n µ ¶ µ ¶ µ ¶ µ ¶ o
£ S 46.45 ¤0 −1 £ S 46.45 ¤
(S, P) : − S − ≤ 1.477
P 2.45 P 2.45

And simultaneous confidence intervals at (nominal) confidence level 95% is

for mean sales : 46.45 ± 19.70

for mean profit : 2.45 ± 1.102.