John Loucks: 1 Slide © 2008 Thomson South-Western. All Rights Reserved
John Loucks: 1 Slide © 2008 Thomson South-Western. All Rights Reserved
John Loucks: 1 Slide © 2008 Thomson South-Western. All Rights Reserved
JOHN
LOUCKS
St. Edward’s
University
where:
0, 1, 2, . . . , p are the parameters, and
is a random variable called the error term
^
y = b0 + b1x1 + b2x2 + . . . + bpxp
Estimated Multiple
b0, b1, b2, . . . , bp Regression Equation
provide estimates of yˆ b0 b1 x1 b2 x2 ... bp x p
0, 1, 2, . . . , p Sample statistics are
b0, b1, b2, . . . , bp
min ( yi yˆ i )2
where
y = annual salary ($1000)
x1 = years of experience
x2 = score on programmer aptitude test
Least Squares
Input Data Output
x1 x2 y Computer b0 =
Package b1 =
4 78 24 for Solving
7 100 43 b2 =
Multiple
. . .
Regression R2 =
. . .
3 89 30 Problems etc.
SALARY
SALARY == 3.174
3.174 ++ 1.404(EXPER)
1.404(EXPER) ++ 0.251(SCORE)
0.251(SCORE)
bb11 == 1.404
1.404
bb22 == 0.251
0.251
i
( y y ) 2
= i
( ˆ
y y ) 2
+ i i
( y ˆ
y ) 2
where:
SST = total sum of squares
SSR = sum of squares due to regression
SSE = sum of squares due to error
R2 = SSR/SST
R2 = 500.3285/599.7855 = .83418
n1
Ra2 2
1 (1 R )
np1
20 1
R 1 (1 .834179)
2
a .814671
20 2 1
The error is
The error is aa random
random variable
variable with
with mean
mean of
of zero.
zero.
The
The variance
variance ofof ,, denoted by
denoted by 22,, is
is the
the same
same for
for all
all
values
values of
of the
the independent
independent variables.
variables.
The
The values of are
values of are independent.
independent.
The error is
The error is aa normally
normally distributed
distributed random random variable variable
reflecting
reflecting the
the deviation
deviation between
between the the yy value
value and and the the
expected
expected value
value of of yy given by 00 ++ 11xx11++ 22xx22++ .. .. ++ ppxxpp..
given by
In
In simple
simple linear
linear regression,
regression, the
the FF and
and tt tests
tests provide
provide
the
the same
same conclusion.
conclusion.
In
In multiple
multiple regression,
regression, the
the FF and
and tt tests
tests have
have different
different
purposes.
purposes.
The
The FF test
test is
is used
used toto determine
determine whether
whether aa significant
significant
relationship
relationship exists
exists between
between the
the dependent
dependent variable
variable
and
and the
the set
set of
of all
all the
the independent
independent variables
variables..
The
The FF test
test is
is referred
referred to
to as
as the
the test
test for
for overall
overall
significance
significance..
If
If the
the FF test
test shows
shows an
an overall
overall significance,
significance, the
the tt test
test is
is
used
used toto determine
determine whether
whether each
each of
of the
the individual
individual
independent
independent variables
variables is
is significant.
significant.
A
A separate
separate tt test
test is
is conducted
conducted for
for each
each of
of the
the
independent
independent variables
variables in
in the
the model.
model.
We
We refer
refer to
to each
each of
of these
these tt tests
tests as
as aa test
test for
for individual
individual
significance
significance..
Hypotheses H 0: 1 = 2 = . . . = p = 0
Ha: One or more of the parameters
is not equal to zero.
Hypotheses H 0: 1 = 2 = 0
Ha: One or both of the parameters
is not equal to zero.
Hypotheses H0 : i 0
H a : i 0
bi
Test Statistics t
sbi
The
The term
term multicollinearity
multicollinearity refers
refers to
to the
the correlation
correlation
among
among the
the independent
independent variables.
variables.
When
When the the independent
independent variables
variables are
are highly
highly correlated
correlated
(say,
(say, ||rr || >> .7),
.7), itit is
is not
not possible
possible to
to determine
determine the
the
separate
separate effect
effect ofof anyany particular
particular independent
independent variable
variable
on
on the
the dependent
dependent variable. variable.
If
If the
the estimated
estimated regression
regression equation
equation is
is to
to be
be used
used only
only
for
for predictive
predictive purposes,
purposes, multicollinearity
multicollinearity is
is usually
usually
not
not aa serious
serious problem.
problem.
Every
Every attempt
attempt should
should be
be made
made to
to avoid
avoid including
including
independent
independent variables
variables that
that are
are highly
highly correlated.
correlated.
The
The procedures
procedures for
for estimating
estimating the
the mean
mean value
value of
of yy
and
and predicting
predicting an
an individual
individual value
value of
of yy in
in multiple
multiple
regression
regression are
are similar
similar to
to those
those in
in simple
simple regression.
regression.
We
We substitute
substitute the
the given
given values
values ofof xx11,, xx22,, .. .. .. ,, xxpp into
into
the
the estimated
estimated regression
regression equation
equation and and use use the the
corresponding
corresponding value
value of
of yy as
as the
the point
point estimate.
estimate.
The
The formulas
formulas required
required to to develop
develop interval
interval estimates
estimates
^
for
for the
the mean
mean value of yy^ and
value of and for
for an
an individual
individual value
value
of
of yy are
are beyond
beyond the
the scope
scope of of the
the textbook.
textbook.
Software
Software packages
packages for
for multiple
multiple regression
regression will
will often
often
provide
provide these
these interval
interval estimates.
estimates.
In
In many
many situations
situations we
we must
must work
work with
with qualitative
qualitative
independent
independent variables
variables such
such as
as gender
gender (male,
(male, female),
female),
method
method of
of payment
payment (cash,
(cash, check,
check, credit
credit card),
card), etc.
etc.
For
For example,
example, xx22 might
might represent
represent gender
gender where
where xx22 == 00
indicates
indicates male
male and
and xx22 == 11 indicates
indicates female.
female.
In
In this
this case,
case, xx22 is
is called
called aa dummy
dummy or
or indicator
indicator variable
variable..
where:
y^ = annual salary ($1000)
x1 = years of experience
x2 = score on programmer aptitude test
x3 = 0 if individual does not have a graduate degree
1 if individual does have a graduate degree
x3 is a dummy variable
Not significant
If
If aa qualitative
qualitative variable
variable has
has kk levels,
levels, kk -- 11 dummy
dummy
variables
variables are
are required,
required, with
with each
each dummy
dummy variable variable
being
being coded
coded asas 00 or
or 1.
1.
For
For example,
example, aa variable
variable with
with levels
levels A,
A, B,
B, and
and C C could
could
be
be represented
represented by by xx11 and
and xx22 values
values of
of (0,
(0, 0)
0) for
for A,
A, (1,
(1, 0)
0)
for
for B,
B, and
and (0,1)
(0,1) for
for C.
C.
Care
Care must
must be
be taken
taken in
in defining
defining and
and interpreting
interpreting the
the
dummy
dummy variables.
variables.
Highest
Degree x1 x2
Bachelor’s 0 0
Master’s 1 0
Ph.D. 0 1
2
Residuals
Standard
0
0 10 20 30 40 50
-1
-2
Predicted Salary
0 1 x1 2 x2 p x p
e
E( y ) 0 1 x1 2 x2 p x p
1 e
Interpretation of E(y) as a
Probability in Logistic Regression
If the two values of y are coded as 0 or 1, the value
of E(y) provides the probability that y = 1 given a
particular set of values for x1, x2, . . . , xp.
E( y ) estimate of P( y 1|x1 , x2 , , x p )
b0 b1 x1 b2 x2 bp x p
e
yˆ b0 b1x1 b2 x2 bp x p
1 e
Odds 95% CI
Predictor Coef SE Coef Z p Ratio Lower Upper
Log-Likelihood = -60.487
Test that all slopes are zero: G = 13.628, DF = 2, P-Value = 0.001
Hypotheses H0: 1 = 2 = 0
Ha: One or both of the parameters
is not equal to zero.
Test Statistics z = bi/sb
i
P( y 1|x1 , x2 , , x p ) P( y 1|x 1 , x 2 , , x p )
odds
P( y 0|x1 , x2 , , x p ) 1 P( y 1|x 1 , x 2 , , x p )
Odds Ratio
odds 1
Odds Ratio
odds 0
Estimated Probabilities
Annual Spending
$1000 $2000 $3000 $4000 $5000 $6000 $7000
Computed
earlier
Comparing Odds
Suppose we want to compare the odds of making a
$200 purchase for customers who spend $2000 annually
and have a Simmons credit card to the odds of making a
$200 purchase for customers who spend $2000 annually
and do not have a Simmons credit card.
.4099
estimate of odds 1 .6946
1 - .4099
.1880
estimate of odds 0 .2315
1 - .1880
.6946
Estimate of odds ratio 3.00
.2315