Scribd
Upload
12 views5 pages

Chapter 11

Uploaded by

karthyfsu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views5 pages

Chapter 11

Uploaded by

karthyfsu
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

Chapter 11 : Multiple Linear Regression

We have:
height weight . . . age amount of
lemonade purchased
person 1: x11 x12 ... x1k y1
person 2: x21 x22 ... x2k y2
:
where we assume

Yi = β0 + β1 xi1 + β2 xi2 + · · · + βk xik + ti

for i = 1, . . . , n and ti ∼ N (0, σ 2 ). The xi· ’s are not random.

Is there any way we can fit something that isn’t linear? Like a polynomial?

We can do least squares to find β̂0 , β̂1 , . . . , β̂k : Minimize Q where:

Q= (yi − (β0 + β1 xi1 + β2 xi2 + · · · + βk xik ))2 .


i

Solve it the same way as we did in Chapter 10: set ∂Q/∂βj = 0 for all j. In this
case, we’ll let the computer solve it for us. So now we have all the β̂j ’s.

To assess the goodness of fit, again define:

SSE = (yi − ŷi )2 where ŷi = β̂0 + β̂1 xi1 + β̂2 xi2 + · · · + β̂k xik
i

and compare with:


SST = (yi − ȳ)2 .
i
Again, SSR = SST- SSE.
The coefficient of “multiple” determination is :
SSR SSE
r2 = =1− . (1)
SST SST

1
This time, by convention,
SSE
r =+ 1− .
SST
The square root is only positive, since it is not meaningful to assign an association
between y and multiple x’s.

For hypothesis testing, we’ll need to know:


1. Each of the coefficients obeys:

βˆj ∼ N (βj , σ 2 Vjj )

where Vjj is the j’th diagonal entry of V = (X ' X)−1 , j = 0, 1, · · · , k


2. Because we don’t know σ 2 , we use

SE(βˆj ) = s Vjj
�

SSE
where s2 = n−(k+1)

We could do the hypothesis tests on each βj :

H0j : βj = βj0
H1j : βj = βj0 .

Reject H0j when


|βˆj − βj0 |
|tj | = > tn−(k+1),α/2
SE(βˆj )
and thus if βj0 = 0:

H0j : βj = 0
H1j : βj = 0.

Reject H0j when


|βˆj |
|tj | = > tn−(k+1),α/2 .
SE(βˆj )

2
Or we could test all βj ’s simultaneously:

H0 : β1 = β2 = · · · = βk = 0
H1 : βi = 0 for at least one i.

Reject H0 when F > fk,n−(k+1),α where:


n 2
SSR i=1 (yˆi −ȳ)
M SR k k
F = = SSE
= n 2
.
M SE i=1 (yi −yˆi )
n−(k+1) n−(k+1)

Both the numerator and the denominator look like sample variances so you
M SR
could see the intuition why M SE has an F-distribution.

Equivalently:
SSR r2 SST
M SR k (?) r2 (n − k − 1)
F = = SSE
= (1−r2k)SST =
M SE n−(k+1)
k(1 − r2 )
n−(k+1)

Where did the (?) step come from?

Note: The F-test above does not tell you which βj s are nonzero.
But then how do you do that?

Note: Beware of multicollinearity, meaning that some of the factors in the


model can be determined from the others (i.e. they are linearly dependent).

Example: for savings, income, expenditure where

savings = income - expenditure.

This makes computation numerically unstable and βˆj are not statistically signif­
icant. To avoid this, use only income and expenditure, not savings. (Or savings
and income, not expenditure, etc.)

3
Corresponding ANOVA regression table

Source of variation sum of squares d.f. Mean Square F p

SSR MSR
Regression SSR k MSR = k
F = MSE
p-value

SSE
Error SSE n − (k + 1) MSE = n−(k+1)

Total SST n−1

We can also put the hypothesis tests for the individual βj ’s in a table:

predictor SE t-statistic p-value

βˆ0
βˆ0 SE(βˆ0 ) t= SE(βˆ0 )
p-value

βˆ1
βˆ1 SE(βˆ1 ) t= SE(βˆ1 )
p-value

.. .. .. ..
. . . .

βˆk
βˆk SE(βˆk ) t= SE(βˆk )
p-value

4
MIT OpenCourseWare
http://ocw.mit.edu

15.075J / ESD.07J Statistical Thinking and Data Analysis


Fall 2011

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

You might also like