0% found this document useful (0 votes)
58 views50 pages

STA302 Week08 Full

The document provides lecture notes on methods of data analysis. Key points include: 1) It discusses midterm and final exams, including remark request policies and dates. 2) It reviews topics from the previous week like variable transformations and simultaneous inferences. 3) It outlines the learning objectives and outcomes for week 8, which include reviewing matrices, the simple linear regression model in matrix form, and inference in regression analysis using matrices.

Uploaded by

tianyuan gu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
58 views50 pages

STA302 Week08 Full

The document provides lecture notes on methods of data analysis. Key points include: 1) It discusses midterm and final exams, including remark request policies and dates. 2) It reviews topics from the previous week like variable transformations and simultaneous inferences. 3) It outlines the learning objectives and outcomes for week 8, which include reviewing matrices, the simple linear regression model in matrix form, and inference in regression analysis using matrices.

Uploaded by

tianyuan gu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 50

STA302/1001 - Methods of Data Analysis I

(Week 08 lecture notes)

Wei (Becky) Lin

Oct. 31, 2016

1/50
Midterm and Final

• If you wrote the midterm, you should receive a mail from Crowdmark.
• For remark request:
• Write me an email within 7 days.
• The request must contain a justification for consideration.
• All remark requests, the whole assignment/ test will be
remarked. There is chance that you might get lower mark points
after remarking.
• Final is on Dec. 12, AM 9-12
• A to J: BN 2N
• K to R: BN 2S
• S to ZHE: BN 3
• Zho to ZZ: ZZ VLAd

2/50
Last Week

• Variable transformations.
• More on logarithmic transformation.
• Box-Cox transformation.
• Interpretation of slope after transformation.
• Chapter 4: Simultaneous Inferences

3/50
Review: from p-value to conclusion
True model Estimated model

Yi = —0 + —1 Xi + ‘i ; Ŷi = b0 + b1 Xi
H0 : —1 = 0; Ha : —1 ”= 0

• test statistic:
b1
tú = |H ≥ tn≠2 b. s×¥× scb.t.tn#xps
s(b1 ) 0
=

• Let the P-value of this test be Pt , assume significance level, – = 5%


• Case 1: Pt < 0.05
• We have weak/moderate/strong evidence to reject the null hypothesis.
• The linear relationship beween Y and X is significant at 5% level.
• We have weak/moderate/strong evidence of a linear relationship
between Y and X.
• Case 2: Pt > 0.05
.
It doesn't ⇒*
empty that Ho is true .*
• We failed to reject the null. (or, there is not enough evidence to reject
the null.)
• We don’t have evidence to indicate a linear relationship between Y and
X.
4/50
Week 08 - Learning objectives & Outcomes

• Review on matrices.
• What is matrix?
• Matrix: transpose, multiplication, addition, inverse
• Special matrices: identity, diagonal, column vector, row vector.
• Variance-covariance matrix of a random vector.
• Matrix Differentiation.
* • Simple Linear Regression Model in Matrix form.
• SLR model in matrix form.
• Least Squares Estimation of Regression Parameters in matrix form.
• Fitted values and residuals in matrix form.
• Inference in Regression Analysis
• ANOVA in matrix form

5/50
Review on Matrices

6/50
Review on Matrices

• Before we take on multiple linear regression, we first look at simple


regression from a matrix perspective.
• It is easy to generalize it to the general analysis of linear models.

• A matrix is a rectangular array of elements arranged in columns and


rows. 5 6 5 6
10 5 8 a1,1 a1,2 a1,3
A= =
4 12 2 a2,1 a2,2 a2,3

• Matrices are often denoted by captial letters.


• Dimension of a matrix is: (# rows )◊(# columns )
• Elements of matrices are referenced by subscripts (i, j), representing
row and column index. e.g. a1,3 = 8, a2,2 = 12

7/50
Review on Matrices (cont.)

• In general, a matrix can be represented in full or abbreviated form


5 6
a1,1 a1,2 a1,3
A= = A2◊3 = [ai,j ], i = 1, 2; j = 1, 2, 3
a2,1 a2,2 a2,3

• Matrix with 1 column called column vector.


5 6
a
A2◊1 = 1,1
a2,1

• Matrix with 1 row called row vector.


# $
A1◊3 = a1,1 a1,2 a1,3

8/50
Review on Matrices (cont.)

• Matrix transpose: flipped version of matrix.


• Denoted by AÕ , or AT .
5 6
a1,1 a1,2 a1,3
A2◊3 =
a2,1 a2,2 a2,3
C D C D
b1,1 b1,2 a1,1 a2,1
B3◊2 = A = b2,1
Õ
b2,2 = a1,2 a2,2 ,
b3,1 b3,2 a1,3 a2,3
• That is, bi,j = aj,i , j = 1, 2; i = 1, 2, 3
• Transpose of a column vector is a row vector.
5 6
a1,1 # $
A2◊1 = ∆ AÕ = a1,1 a2,1
a2,1

• Matrix equality: two matrices are called equal if they have . same
dimensions and the corresponding elements
. are equal.
• (ABC )T = C T B T AT

9/50
Review on Matrices (cont.)

• Matrix addition: the sum of two matrices with equal dimensions is


matrix of the sum of their corresponding elements.

Am◊n + Bm◊n = Cm◊n … [ci,j ] = [ai,j + bi,j ]

• For example, sum of two column vectors is another column vector


5 6 5 6 5 6
4 2 6
A2◊1 + B2◊1 = + =
6 4 10

• Matrix multiplication: product of two matrices is matrix of


cross-product of their corresponding rows and columns.
n
Ëÿ È
Am◊n ·Bn◊p = Cm◊p … [ci,j ] = ai,k bk,j , ’i = 1, . . . , m, j = 1, . . . , p
k=1
5 6
# $ 1 2 # $ # $
A1◊2 B2◊2 = 1 2 = 1·1+2·3 1·2+2·4 = 7 10
3 4
10/50
Review on Matrices (cont.)
Matrix multiplication:
• # columns for A must match # rows for B.
• Matrix multiplication is NOT symmetric.

A1◊2 · B2◊2 ”= B2◊2 · A1◊2


z EE

• Multiplication by a scalar (number)


5 6 5 65 6
6 2 3ú6 3 ú 2 18 6
c · A2◊2 = 3 · =
7 4 3ú7 3 ú 4 21 12

Diagonal matrix:
• It is a square matrix and all off-diagonal elements are 0.
S T S T
d1 0 0 1 0 0
D3◊3 = U 0 d2 0 V , I3◊3 = U0 1 0V
0 0 d3 0 0 1

• Identity matrix: a special case of diagonal matrix, with all diagonal


elements equal to 1. 11/50
Identity Matrix

• Multiplying matrix A by identity matrix I leaves A unchanged.


• IA=AI=A
S TS T S T
1 0 0 2 4 1 2 4 1
I · A = U0 1 0V U5 1 3 V = U5 1 3V=A·I
0 0 1 7 9 12 7 9 12

• Scalar matrix: diagonal matrix with all diagonal elements being equal
S T S T
d 0 0 1 0 0
U0 d 0 V = d · U0 1 0V = d · I
0 0 d 0 0 1

12/50
Inverse of a Matrix

Assume A is invertible, that is the inverse of A exists.


)

A≠1 A = AA≠1 = I

For a 2 by 2 matrix, a simple formula exists to find its inverse


3 4
a b
if A = ,
c d
3 4
1 d ≠b
then A ≠1
=
ad ≠ bc ≠c a

• Note that the quantity ad ≠ bc is the determinant of A.


• 1/(ad ≠ bc) is not defined when ad ≠ bc = 0 since it is never
possible to divide by zero. It is for this reason that the inverse of A
does not exist if the determinant of A is zero.

13/50
Inverse of a Matrix (cont.)

• Not every square matrix A has an inverse.


• Matrix must be non-singular (full rank).
• Matrix rank is maximum number of linearly indepedent columns.
(equivalently rows).
Az=Ar2A2
C D C D
| | | 3 2 ≠1
A3◊3 = a1 a2 a3 = 2 0 2
| | | 1 1 ≠1

This leads to rank(A)=2, since a3 = a1 ≠ 2a2 .


• Square matrix is full-rank if rank(An◊n ) = n.
• (AB)≠1 = B ≠1 A≠1 assume the square matrices A and B, are both
invertible.
• {(A)≠1 }Õ = {AÕ }≠1

14/50
Column Vectors

• A column with all elements 1 and zero are 1 and 0


S T S T
1 0
W1X W0X
W X W X
1r ◊1 = W . X , 0r ◊1 = W . X
U .. V U .. V
1 0

• A square matrix with all elements 1: J


S T
1 S T
W1X # 1 ... 1
W X $ W .. .. .. X
Jr ◊r = 11T = W . X 1 1 . . . 1 = U. . .V
U .. V
1 ... 1
1

• 1Õ 1 = n

15/50
Mean of a random Vector

• Vecotr of random variables (RV’s)


ST
Y1
WY2 X
W X
Yn◊1 =W . X
U .. V
Yn

where Y1 , . . . , Yn are RV’s.


• Mean of random vector: E (Yi ) = µi , i = 1, . . . n
S T S T
E (Y1 ) µ1
WE (Y2 )X Wµ2 X
W X W X
E (Y)n◊1 = W . X = W . X = µn◊1
U .. V U .. V
E (Yn ) µn

16/50
Mean of a linear combination of Random Vector

• Linear combination of random vectors


• Constant matrix, vector: Ar ◊n , br ◊1
• Random vector: Yn◊1 with E (Y) = µy , Var (Y) = Y.

Xr◊1 = br◊1 + Ar◊n Yn◊1


*V Y d
RT
• For example some constant

S T
5 6 5 6 Y
b a a1,2 a1,3 U 1 V
b2◊1 + A2◊3 Y3◊1 = 1 + 1,1 Y2
b2 a2,1 a2,2 a2,3
Y3
5 6
b1 + a1,1 Y1 + a1,2 Y2 + a1,3 Y3
X2◊1 =
b2 + a2,1 Y1 + a2,2 Y2 + a2,3 Y3
• Mean of X: µx = E (X) = E (b+AY) = b + AE (Y) = b + Aµ
µy

17/50
Var-Cov of Random Vectors

• Variance-covariance matrix of random vector


SY T S V (Y ) Cov (Y1 , Y2 ) ... Cov (Y1 , Yn )
T
1 1
Y 2
W X W Cov (Y 2 , Y 1 ) V (Y2 ) ... Cov (Y2 , Yn )X
Var (Y) = Var (U . V) = U .. .. .. .. V= Y
. . . . . .
Yn Cov (Yn , Y1 ) Cov (Yn , Y2 ) ... Var (Yn ) nxn
nxl Hady

µ
mat of F

Eye
var -

an

• Variance-covariance matrix is symmetric


• Cov (Yi , Yj ) = Cov (Yj , Yi ), that is = T 0 on diagonal : var Mil

• variance-covariance for ‘ off diagonal :

ioytljitjl
Var (‘‘) = ‡ 2 I
If
3

L
=
Ty
1
52 = v th )
.

-
.
. .
= VCYN )

{ constant

CMYI ,Yj
variance

errors
uncorrelated
@ )i±j=o are
:

18/50
Var-Cov of Random Vectors
Show Var (Y) = E {[Y ≠ µ] · [Y ≠ µ]Õ }

Var (Y) = Y = E {[Y ≠ µ] · [Y ≠ µ]Õ }


SY ≠ µ T
1 1

WY2 ≠ µ2 X #Y ≠ µ Y 2 ≠ µ2
$
Y n ≠ µn )
= E (U .. V 1 1 ... nxn ⇒ matrix
. Hh

Y n ≠ µn nxl
S (Y1 ≠ µ1 )2 (Y1 ≠ µ1 )(Y2 ≠ µ2 ) ... (Y1 ≠ µ1 )(Yn ≠ µn )
T
W .. .. .. .. X
= E (U . . . . V)
..
(Yn ≠ µn )(Y1 ≠ µ1 ) (Yn ≠ µn )(Y2 ≠ µ2 ) . (Yn ≠ µn )2

S
V (Y1 ) Cov (Y1 , Y2 ) ...
T
Cov (Y1 , Yn )
W Cov (Y2 , Y1 ) V (Y2 ) ... Cov (Y2 , Yn )X
=U .. .. .. .. V
. . . .
Cov (Yn , Y1 ) Cov (Yn , Y2 ) ... Var (Yn )
= n◊n

19/50
Var-Cov Random Vectors

• Variance-covariance matrix for uncorrelated or independent vector of


RV’s
• Y1 , . . . , Yn with Cov (Yi , Yj ) = 0, ’i ”= j.
S T
‡12 0 ... 0
W0 ‡22 ... 0X
Var (Y) = Y =W
U ... .. .. .. XV
. . .
2
0 0 ... ‡n

• It is a diagonal matrix: all off-diagonal elements are 0.


• W, Y: random vectors
• A: a constant matrix
• We then have
• E(A)=A
• W=AY, E(W)=E(AY)=AE(Y)
• W=AY, Var(W)=Var(AY)=AV (Y )AT *

20/50
Example: Var-Cov of linear combination of random vector
• Let X = b+AY.
• Using (A + B)C = AB + AC , (AB)T = B T AT ,
• Show the var-cov matrix of X is
X = E {(X ≠ µx )(X ≠ µx )T } = A yA
T # I

proof :
.
x=bt AT

EHKECBTAYI = bt AEIY )

lbt AELY) Al Ytly )


=(btAY) AER ) ) AY
=

X Ehl
-

° - - =

Now by defh
}
AHMHTLHLMYIT
El
Excel l*µ×)L*µ×T ) =

zi
: CABIEBTA
'

=
} Ef AH .

My )tfµy
a-
)
T
AT

=
A E{ H-µy ) (Y -

µyM } AT

=AZy AT
÷
a. E. D .

21/50
Example 2 on Var-Cov .
Yi~H( 0 ,
Try

{ Yz NLO Tie )
.

6 ⇒ imply
~
,

5 6 5 6 5 2 only .k)=Jnz
Y1 0
.

‡1 ‡1,2
≥ N( , 2 ), Y ≥ N(0, )
Y2 0 ‡1,2 ‡2
Show V (a1 Y1 + a2 Y2 ) = a12 ‡ 2 + a2 ‡22 + 2a1 a2 ‡1,2

Proof : Method 1 :

vcaihtaihl =
vcaihltvcaihltscwlaia ,
ask )
=
airptairitzaiaz Couch ,k )
=
aprptazrit 29929,2
r#
Method 2 :

W=aihtazYz= ( a , as )l¥z ) =
Anika

Varlwl Van AY ) by * ) slide 21


⇒ A Ey AT
-

= =

=
ca , an 1%2
EI
.
's
) ( Aa )
%
=
atop taint 29,925,2
d. E. D.

22/50
Example 3 on Var-Cov
• Example: Y1 , Y2 ≥ N(0, 1) independently.
• Show
5 6 5 65 6 5 6 5 6
W1 1  0 Y1 0 1 fl
W= = ≥ N( , )
W2 fl 1 ≠ fl2 Y2 0 fl 1
= T
proof : W AY and µy= It9)
=
, 181 I =
=L
.
,


Elw ) =
EIAT ) =
A Ely) =
02×1

var ( w
Ee AT
) Var ( AT )
.

=
= A

=
AIAT =AA7

Itif ) tfp
'
1 )
=

.
o .

=
( f ) QE .
D .

23/50
Matrix Differentiation

Y = f (X ), where xn◊1
m◊1

Then the m ◊ n matrix of first-order partial derivatives of the


transformation from x to Y:
S ˆY1 T S ˆy1 ˆy1 ˆy1
T
ˆX ˆx 1 ˆx 2
. . . ˆx n
W ˆy2 ˆy2 ˆy2 X
ˆY W
ˆY2 X
W ˆX X W . . . ˆx X
=W . X=W
ˆx1 ˆx2 n X

.
ˆX U . V U .. W . .
.. .
.. .
.. XV
ˆYn ˆym ˆym ˆym
X . . .
ˆX ˆx1 ˆx2 ˆxn m◊n

24/50
Matrix Differentiation (cont.)

ˆY
MD01 : Y = A X ∆ =A
m◊1 m◊n n◊1 ˆX
Proof:

Hint take

Efta
In

.it?aInMIt*
⇒ Yi =
# AIKXK


Thing and
=
aij Hit 2 m j⇒ 2 n
. -

,
.
.

, ,
,

2 Yi
=)
Tx
=
[ Air Aiz Ain ]
- -
-

, ,


QED .

25/50
Matrix Differentiation (cont.)

MD02 : – = Y T
1◊1
A
1◊m m◊n n◊1
X ∆ OM20L
ˆ–
ˆX
= Y T A, and
ˆ–
ˆY 4
= X T AT

Proof:
@ WEYTA ⇒ 2= WTX


¥× = WT by MDOI


-

=
YTA

× , ⇒ 2=51 =
XTATY

⇒ 22
XTAT
Ty
=

a. ED .

26/50
Matrix Differentiation (cont.)
For the special case in which the scalar – is given by the quadratic form
ˆ–
MD03 : – = X T A X ∆ = X T (A + AT )
1◊1 1◊n n◊n n◊1 ˆX
Proof:
An An
-
-

Ain

by defn 2=1×1×2 ,
-
'

xnl ( ian ain) ( ¥xn )


, an ,
. .

j= EYE
! aij
L - ,
Xi
Xj

2¥ #

.
=
# , akjxj t Aik Xi

=
ix. x. -
-

xn ) Makin ) + ix. .
.

xn ) µa¥m )
,

=
XT Artowk t XTA column k for Kb 2
,
- -

n
,

--
'


2¥ = XTAT + XTA =
XTIATTAI
27/50
Matrix Differentiation (cont.)
For the special case in which the scalar – is given by the quadratic form,
and the A is a symmetric matrix, ⇒ A=AT
ˆ–
MD04 : – = X T A X ∆ = 2X T A
1◊1 1◊n n◊n n◊1 ˆX
Proof:

From MD03 we have

¥ = XTIATAT )

=XT ( ATA ) since A=A7

=2XTA

QE . D .

28/50
Matrix approach to SLR

29/50
SLR in matrix form

• Y: consisting of the n observations on the response variables.


S T
Y1
WY2 X
W X
Yn◊1 = W . X
U .. V
Yn

• X Matrix: S T
1 X1
W1 X2 X
W X
Xn◊2 = W. .. X
U .. . V
1 Xn
often referred to as the design matrix.

30/50
SLR in matrix form

,¥x=
iIn .tn?i&xntItExiFIil
ix. !
'

x.

• Product

/
,
n
ÿ
= Mi .
I -
-

Ynl (
l¥n ) =
Fun Yj2

YT Y = Yi2 : a scalar, a sum of square terms


1◊1
" i
5 q 6
i 2◊1
X T
Y = q Yi
(Xi Yi )
ixixi
: = 'xnt×t¥nkit¥Ixi% )
5 q 6
, T n
X X= q q X2i detcxtx ) NEXT )2
fix
Xi Xi ⇒
-

1◊1
=

S T
Exitnxy
2 =a
1 =n(
+ q(XX̄ ≠X̄ )2 ≠ q(XX̄≠X̄ )2
"

n
∆ (XT X)≠1 = U i i
V =n Ein lxixt
1◊1 ≠ q(XX̄≠X̄ )2 q 1 2
(X ≠X̄ )
,

i i
=nS××

f- Exit
mm

=
#IIENKITY
Ali #
= H÷nm¥xtnItnt¥*
31/50
SLR in matrix form

• Simple linear regression: mean response in matrix form

E (Yi ) = —0 + —1 Xi , i = 1, . . . , n

• Let
S T S T
E (Y1 ) 1 X1
WE (Y2 )X W1 X2 X 5 6
W X W X —0
E (Y)n◊1 = W . X , Xn◊2 = W . .. X , — 2◊1 = —
U .. V U .. . V 1
E (Yn ) 1 Xn

• Columns represent variables.


• Rows represent observations.

32/50
SLR in matrix form (cont.)

• SLR: Mean response in matrix form


S T S T S T
E (Y1 ) 1 X1 —0 + —1 X1
WE (Y2 )X W1 X2 X 5 6 W— + — X X
W X W X —0 W 0 1 2X
E (Y)n◊1 = W . X = Xn◊2 ·— —2◊1 = W . X =W . X
U .. V U .. V —1 U .. V
E (Yn ) 1 Xn —0 + —1 Xn

33/50
SLR model in matrix form (cont.)

• Simple linear regression in matrix form

Yi = —0 + —1 Xi + ‘i , i = 1, . . . , n

Yn◊1 = Xn◊2— 2◊1 + ‘ n◊1


That is S T S T S T
Y1 1 X1 ‘1
WY2 X W1 X2 X 5 6 W‘ X
W X W X —0 W 2X
W .. X = W .. X +W.X
U . V U. V —1 U .. V
Yn 1 Xn ‘n
• For Y, X
• Columns represent variables.
• Rows represent observations.

34/50
SLR model in matrix form (cont.)

• The normal error regression model

Y = X—
— +‘

• ‘: a vector of independent normal r.v.


• E (‘‘) = 0
• V (‘‘) = ‡ 2 I
• Above 3 conditions on the error term can be written as
S T S T
0 1 0 ... 0
W . X 2 W. .. .. .. X
‘ ≥ N(00, ) = N(U .. V , ‡ U .. . . . V)
0 0 0 ... 1

or in a very brief way, ‘ ≥ N(0, ‡ 2 I)

35/50
Least Squares Estimation of Regression Parameters

• Normal Equations

ÿ ÿ
nb0 + b1 Xi = Yi (1)
ÿ ÿ ÿ
b0 Xi + b1 Xi2 = Xi Yi (2)

This is equivalent in matrix form

X ÕX b = X ÕY
2◊2 2◊1 2◊1
5 q 65 6 5 q 6
n X i b0 Y i
X X ·b = q
Õ q 2 =X Y = q
Õ
Xi Xi b1 Xi Yi
• The vector of the least squares regression coefficients
S 2
T
1 q X̄ q X̄ 2 5 q
5 6
b + ≠ 6
V q Yi
n (Xi ≠X̄ ) 2 (Xi ≠X̄ )
b = 0 = (XÕ X)≠1 XÕ Y = U
b1 ≠ q X̄ 2 q 1 2 Xi Yi
(Xi ≠X̄ ) (Xi ≠X̄ )

36/50
Verify the LS matrix form Exercise :

S 2
T
1
5 6
b0 + q(XX̄ ≠X̄ )2 ≠ q(XX̄≠X̄ )2 5 q 6
V q Yi
n
b= = (XÕ X)≠1 XÕ Y = U i i

b1 ≠ q X̄ 2 q 1 Xi Yi
(Xi ≠X̄ ) (Xi ≠X̄ )2

et sxx =
-2¥ Ni # 12

±nt¥× ¥x× Emi


.si/y=IlXi-X1lYiT)-fg
-

Elkin ) hit =
'
-

[email protected]
( )( )
i
-

Fxx s÷× Enxiti

.be#EIIxjIksxIx=nnI+n5gxI.xEhgxiI
5- ¥i¥±# "
5 =b

)= (
.

aximj.IE/=b.=c*bbi
-

*¥× +
age •

's =iIxbY*l 37/50


Least Squares Estimation of Regression Parameters
Minimize the quantity

ya
ÿ

,n×Q
Q= [Yi ≠ (—0 + —1 Xi )]2 B :
2×1
T
= (Y ≠ X —) (Y ≠ X —) ' '' 2 zxnnx
¥ ( BTXTY )
= Y T Y ≠ —T X T Y ≠ Y T X — + —T X T X —
1×1 1×2 lxhnxz =¥p ( YTXP )

"nxiLeIFtexxEf
2×h nxl zx ,

To find the value of — that minimizes Q:


=
YX IMDOZ )
ˆ # $
Q = ˆQ/ˆ—0 ˆQ/ˆ—1 = ≠2Y Õ X + 2— T X T X 2
Fp ( ytxp )
ˆ—
=Y1X
• Setting this equation to zero: ≠2Y Õ X + 2b T X T X = 0 ( MDOZ )
• Transpose both sides of the equation, we have
¥
3

lptxtxp )
X Õ Xb = X Õ Y
ˆ Assume X Õ X is invertible
• Solving it for —. =zpT( XTX )
1 MD 04 )
∆ b = (X Õ X )≠1 X Õ Y
38/50
Fitted Values and Residuals

Fitted Values:

39/50
Fitted Values and Residuals (conts.)


make.tt?IiiIIiwll?d.hiIItIiIIEEI?I
Hat Matrix:
• H = X (X Õ X )≠1 X Õ : Hat matrix or “projection” matrix. I ,
• Important in diagnostics for regression
• Symmetric and idempotent:
.
Eh hi ; = 1

&
H T = H; HH = H -3¥ hiijihii

1
HE [ xixix , ix. 1 '=×[w× .yt×
,
-

,
hij-hjithEiIhiie2n-xtlxTxPItXT-XlXTX5lXT-H@HH-XlXTX5lXTXcXTX51xT-XcxTxjtXT-Hmn_MHH-HH0EiYhIy-hii0Zkhahikh.k

- ;
=
hij
40/50
Residuals

41/50
Residual distribution

Y ≥ N(X —, ‡ 2 I), ⇒ e ≥ N(0, ‡ 2 (I ≠ H))


n◊1 n◊1
Proof:

e=Y 'T y HY
=
=h HIY where

*
KXHX
'
He
-
-

xitxtxxp
,

⇒ EL HM
f- ) =
H -

HIEN ) 2

xp HXB xp =Xp Xp :O
= -
=
- -

w nm

Van e) Var
=
( EHN )
=
(z -

H) Zy H -

HIT
z .

H is idempotent'

'
= ( z -

His 1 a- H )
x

=P ( THHTHI = 02 I Ht )
42/50
Inference in Regression Analysis

symmetric
ttforvaravmatrixif
Regression Coefficients:
• The variance-covariance matrix of b:

5 6
V (b0 ) V (b0 , b1 ) combo b.) 0-25
Var (b) =
=
-

V (b1 , b0 ) V (b1 ) Sxx


11
S 2
T
2 1 X̄ 2 X̄
Var ( b o
) ⇐ ‡ ( n + q(X ≠X̄ )2 ) ≠‡ q(X ≠X̄ )2
=U i
2
i
V
≠‡ 2 q(XX̄≠X̄ )2 q ‡
(X ≠X̄ ) 2
i i

2
= ‡ (X X )Õ ≠1
"

var ( b
,
)=E÷×

43/50
Inference in Regression Analysis (cont.)

• The estmiated variance-covariance of b:

s 2 (b) = MSE (X Õ X )≠1


2◊2
S 2
T
1
n + q(XX̄ ≠X̄ )2 ≠ q(XX̄≠X̄ )2
= MSE U i i
V
≠ q(XX̄≠X̄ )2 q 1
(Xi ≠X̄ )2
i

2
=‡
ˆ (X X )
Õ ≠1

• b = (X Õ X )≠1 X Õ Y = AY
T
• ∆ V (b) = V (AY ) = A YA = ‡ 2 AAT = ‡ 2 (X Õ X )≠1

44/50
Mean response and prediction of new observation

• The mean response at Xh


5 6
1
Xh = ; Ŷh = XhÕ b
2◊1 Xh

V (Ŷh ) = V (XhÕ b) = XhÕ V (b)Xh


= XhÕ ‡ 2 (X Õ X )≠1 Xh
= ‡ 2 XhÕ (X Õ X )≠1 Xh
1 (Xh ≠ X̄ )2
= ‡2 [ + q ]
n (Xi ≠ X̄ )2
∆ s 2 (Ŷh ) = MSE (XhÕ (X Õ X )≠1 Xh )

• Prediction of New observation at Xh


2
spred = MSE (1 + XhÕ (X Õ X )≠1 Xh )

45/50
Analysis of Variance Results (ANOVA)
Total Sum of Squares
ÿ 1 Õ 1
SSTO = (Yi ≠ Ȳ )2 = YÕ Y ≠ Y JY = YÕ [I ≠ J]Y
n n
Proof:
Il Yi -

F) 2= Eyeing
2
= zy ,
?
-

K¥2
'
th 141

Hk
Y Y )
= '
-

th
' ' '
=
Y Y -

IY 12 Y)
'
=
y y -
th LYJY )

his ) y
'
=
Y (z
-

46/50
Analysis of Variance Results (ANOVA) (cont.)
Error Sum of Squares

SSE = e Õ e = (Y ≠ Xb)Õ (Y ≠ Xb) = YÕ Y ≠ bÕ XÕ Y = YÕ (I ≠ H)Y


g- @-
Proof:
1

sseeeie-H-xbjlY-Xb1-Htb1xYH-xb1-Y4-Yxb-b1XYtbX1xb_ib-lXXMXYb1-YlXCX1X5l-Yy-Yxb-b1XYtYIIkXExxbtYY-YXb-b1XYtYXbae-b1XY@sSEee1e-ly-FpCy-P1-ktHNj4tHjY-YCz-HH2-HM-Y1CtH1YQ.E.D

47/50
Analysis of Variance Results (ANOVA) (cont.)
Regression Sum of Squares
ÿ 1 Õ 1
SSR = (Ŷi ≠ Ȳ )2 = bÕ XÕ Y ≠ Y JY = YÕ [H ≠ J]Y
n n
Proof:

EYIETTY
DssRt-HiYitl2-EIcYitzFItF2l-2EFYY-2Y2iFkTth.lxi-xHtny2.Yi-botkXi-FbiItb.Xi-2iFitit25enytb.Elxixntny2-5tbilxiX1-O-EiIFi2-2nT2tny2l-Ei.FYi2-nYT0EFi2tlxblTcxbkbTXTxb-blXx1llxX5lMyI-b1XY@ny2.t

-n( EYIP =
'T ((141417 ) ) =
th YJY

ssR= 1
-
2 =
b 'X' Y -

th YYY
30-20=4
IH HTTY
'

)=Y' AT
-

HYITCHY
'

3
= ( ⇒ ssr=
48/50
Analysis of Variance Results (ANOVA) (Summary)

• Sums of Squares as Quadraatic Froms: Y Õ AY

ÿ ÿ 1
SSTO = Yi2 ≠ ( Yi )/n = YÕ (I ≠ J)Y
n
SSE = e Õ e = (Y ≠ Xb)Õ (Y ≠ Xb) = YÕ (I ≠ H)Y
ÿ 1
SSR = (Ŷi ≠ Ȳ )2 = YÕ [H ≠ J]Y
n

49/50
Practice problems and upcoming topics

• Practice problems after today’s lecture: CH5: 5.1, 5.3, 5.4, 5.8, 5.10,
5.12, 5.17, 5.21, 5.23, 5.29, 5.31
• Extra exercises
• Show I ≠ H is symmetric and idempotent.
• Show H(I ≠ H) = 0
• Upcoming topics
• CH6: Models with more than one predictors.
• Reading for upcoming topics: Chapter 6.

50/50

You might also like