Quiz 2- Linear Regression Analysis (Based on Lectures 15-31)

Time: 1 Hour
y X β + ε are assumed to be
1. The random errors ε in multiple linear regression model=
identically and independently distributed following the normal distribution with zero
mean and constant variance. Here y is a n ×1 vector of observations on response
variable, X is a n × K matrix of n observations on each of the K explanatory variables,
β is a K ×1 vector of regression coefficients and ε is a n ×1 vector of random errors.
The residuals εˆ= y − yˆ based on the ordinary least squares estimator of β have, in
general,
(A) zero mean, constant variance and are independent
(B) zero mean, constant variance and are not independent
(C) zero mean, non constant variance and are not independent
(D) non zero mean, non constant variance and are not independent

Answer: (C)

Solution:The residual is
εˆ= y − yˆ
= ( X ' X ) −1 X ' y
y − Xb where b =
= X ( X ' X ) −1 X '
( I − H ) y where H =
= ( I − H )ε
E (εˆ ) = 0
(εˆ ) σ 2 ( I − H )
V=

• Since E (εˆ ) = 0 , so εˆi 's have zero mean.

• Since I − H is not generally a diagonal matrix, so εˆi ' s do not have necessarily the
same variances.
• The off-diagonal elements in ( I − H ) are not zero, in general. So εˆi ' s are not
independent.

1
2. Consider the multiple linear regression model X β + ε , E (ε ) =
y= 0,

V (ε ) = diag (σ 12 , σ 22 ,..., σ n2 ) where y is a n ×1 vector of observations on response

variable, X is a n × K matrix of n observations on each of the K explanatory variables,

β is a K ×1 vector of regression coefficients and ε is a n ×1 vector of random errors.
Let hii be the i th diagonal element of matrix H = X ( X ' X ) −1 . The variance of the i th
PRESS residual is
(A) σ i2 (1 − hii )

(B) σ i2 (1 − hii ) 2

σ i2
(C)
1 − hii

σ i2
(D)
(1 − hii ) 2

Answer: (C)

Solution: Let e=i yi − yˆi be the ith ordinary residual based on the ordinary least squares
estimator of β . The ith PRESS residual e(i) is

ei
e(i ) = .
1 − hii

The variance of e(i ) is obtained as follows:

Var (ei )
Var (e(i ) ) =
(1 − hii ) 2
and
) ( I − H )V (ε )
V (e=
σ i2 (1 − hii ).
⇒ Var (ei ) =
Thus
σ i2
Var (e(i ) ) = .
1 − hii

2
3. Consider the linear model y = β1 X 1 + β 2 X 2 + ε , E (ε ) = 0, V (ε ) = I where the study

variable y and the explanatory variables X 1 and X 2 are scaled to length unity and the

correlation coefficient between X 1 and X 2 is 0.5. Let b1 and b2 be the ordinary least

squares estimators of β1 and β 2 respectively. The covariance between b1 and b2 is

(A) 2/3
(B) -2/3
(C) -0.5
(D) 1/3

Answer: (B)

Solution: If r is the correlation coefficient between X 1 and X 2 , then the ordinary least
squares estimators b1 and b2 of β1 and β 2 are the solutions of

 b1  −1 1 r 
=b =  ( X ' X ) X ' y where X=
'X  .
 b2   r 1
Then
1  1 −r 
( X ' X ) −1 =  
1 − r 2  −r 1 
1  1 −r 
=
V (b) (= X ' X ) −1  
1 − r 2  −r 1 
r
Cov(b1 , b2 ) = − .
1− r2

2
When r = 0.5, cov(b1 , b2 ) = − .
3

3
 4. Under the multicolinearity problem in the data in the model y = β1 X 1 + β 2 X 2 + ε , where

the study variable y and the explanatory variables X 1 and X 2 are scaled to length unity

E ( ε ) 0,=
and the random error ε is normally distributed with= V ( ε ) σ 2 I where σ 2 is

=
unknown. What do you conclude about the null hypothesis H 01 : β1 0=
and H 02 : β 2 0
out of the following when sample size is small.
(A) Both H 01 and H 02 are more often accepted.

(B) Both H 01 and H 02 are more often rejected.

(C) H 01 is accepted more often and H 02 is rejected more often.

(D) H 01 is rejected more often and H 02 is accepted more often.

Answer: (A)

Solution: If r is the correlation coefficient between X 1 and X 2 , then the ordinary least
squares estimators b1 and b2 are the solutions of

1 r 
= X ' X ) −1 X ' y where X ' X 
b (= .
 r 1
Then
E (b) = β
σ2  1 −r 
=V (b) σ=
2
( X ' X ) −1  .
1 − r  −r
2
1

As r becomes higher, the variance of b1 and b2 become larger and so the test statistic

bi
=ti = , i 1, 2

var(bi )

for testing H 0i : βi = 0 becomes very small and so H 0i is more often accepted.

4
 y X β + ε where y is a n ×1
5. Consider the setup of multiple linear regression model=
vector of observations on response variable, X is a n × K matrix of n observations on
each of the K explanatory variables, β is a K ×1 vector of regression coefficients and

ε is a n ×1 vector of random errors following N (0, σ 2 I ) with all usual assumptions. Let
H = X ( X ' X ) −1 and hii be the i th diagonal element of H . A data point is a leverage point

if hii is
(A) greater than the value of average size of hat diagonal.
(B) less than the value of average size of hat diagonal.
(C) greater or less than the value of average size of hat diagonal.
(D) none of the above.

Answer: (A)

Solution: The i th diagonal element of H is hii = x1i ( X 1 X )−1 xi where xi is the i th row
of X matrix. The hii is a standardized measure of distance of i th observation from the

center (or centroid) of the x -space.

n

∑h ii
h)
Average size of hat diagonal (=
rank ( H ) trH K
1=i
= = = .
n n n n
If hii > 2h then the point is remote enough to be considered as a leverage point.