Slides by

John
Loucks
St. Edward’s
University

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
1
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Chapter 16
Regression Analysis: Model Building
 General Linear Model
 Determining When to Add or Delete Variables
 Variable Selection Procedures
 Multiple Regression Approach to
Experimental Design
 Autocorrelation and the
Durbin-Watson Test

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
2
or duplicated, or posted to a publicly accessible website, in whole or in part.
 General Linear Model

 Models in which the parameters (0, 1, . . . , p ) all

have exponents of one are called linear models.
 A general linear model involving p independent
variables is
y   0   1 z1   2 z2     p zp  

 Each of the independent variables z is a function of

x1, x2,..., xk (the variables for which data have been
collected).

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
3
or duplicated, or posted to a publicly accessible website, in whole or in part.
 General Linear Model

 The simplest case is when we have collected data for

just one variable x1 and want to estimate y by using a
straight-line relationship. In this case z1 = x1.
 This model is called a simple first-order model with
one predictor variable.

y   0   1 x1  

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
4
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Modeling Curvilinear Relationships

 To account for a curvilinear relationship, we might

set z1 = x1 and z2 = x12.
 This model is called a second-order model with one
predictor variable.

y   0   1 x 1   2 x 12  

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
5
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Interaction

 If the original data set consists of observations for y

and two independent variables x1 and x2 we might
develop a second-order model with two predictor
variables.
y   0   1 x 1   2 x 2   3 x 12   4 x 22   5 x 1 x 2  

 In this model, the variable z5 = x1x2 is added to

account for the potential effects of the two variables
acting together.
 This type of effect is called interaction.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
6
or duplicated, or posted to a publicly accessible website, in whole or in part.
Transformations Involving the Dependent Variable

 Often the problem of nonconstant variance can be

corrected by transforming the dependent variable to a
different scale.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
7
or duplicated, or posted to a publicly accessible website, in whole or in part.
© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
8
or duplicated, or posted to a publicly accessible website, in whole or in part.
© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
9
or duplicated, or posted to a publicly accessible website, in whole or in part.
Transformations Involving the Dependent Variable

 Often the problem of nonconstant variance can be

corrected by transforming the dependent variable to a
different scale.
 Most statistical packages provide the ability to apply
logarithmic transformations using either the base-10
(common log) or the base e = 2.71828... (natural log).

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
10
or duplicated, or posted to a publicly accessible website, in whole or in part.
© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
11
or duplicated, or posted to a publicly accessible website, in whole or in part.
Transformations Involving the Dependent Variable

 Often the problem of nonconstant variance can be

corrected by transforming the dependent variable to a
different scale.
 Most statistical packages provide the ability to apply
logarithmic transformations using either the base-10
(common log) or the base e = 2.71828... (natural log).
 Another approach, called a reciprocal transformation,
is to use 1/y as the dependent variable instead of y.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
12
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Nonlinear Models That Are Intrinsically Linear

 Models in which the parameters (0, 1, . . . , p ) have

exponents other than one are called nonlinear models.
 In some cases we can perform a transformation of
variables that will enable us to use regression analysis
with the general linear model.
 The exponential model involves the regression
equation:
E( y )   0  x1

 We can transform this nonlinear model to a linear

model by taking the logarithm of both sides.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
13
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Determining When to Add or Delete Variables

 To test whether the addition of x2 to a model involving

x1 (or the deletion of x2 from a model involving x1 and
x2) is statistically significant we can perform an F Test.
 The F Test is based on a determination of the amount of
reduction in the error sum of squares resulting from
adding one or more independent variables to the model.
(SSE(reduced)-SSE(full))/number of extra terms
F
MSE(full)

(SSE(x1 )-SSE(x1 ,x2 ))/1

F
(SSE(x1 , x2 ))/( n  p  1)

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
14
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Determining When to Add or Delete Variables

 The p–value criterion can also be used to determine

whether it is advantageous to add one or more
dependent variables to a multiple regression model.
 The p–value associated with the computed F statistic
can be compared to the level of significance a .
 It is difficult to determine the p–value directly from
the tables of the F distribution, but computer
software packages, such as Minitab or Excel, provide
the p-value.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
15
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable Selection: Stepwise Regression
Compute F stat. and Any
p-value for each indep. p-value < alpha
variable not in model to enter
? No
No
Indep. variable Yes
Any Yes with largest
p-value > alpha p-value is Stop
to remove removed
? from model

Compute F stat. and next Indep. variable with

p-value for each indep. smallest p-value is
variable in model iteration entered into model

Start with no indep.

variables in model
© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
16
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable-Selection Procedures

 Variable Names and Definitions

Drive: average length of a drive in yards

Fair: percentage of drives that land in the fairway
Green: percentage of greens hit in regulation (a par-3
green is “hit in regulation” if the player’s first
shot lands on the green)
Putt: average number of putts for greens that have
been hit in regulation
Sand: percentage of sand saves (landing in a sand
trap and still scoring par or better)
Score: average score for an 18-hole round
© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
17
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable-Selection Procedures

 Sample Data (Part 1)

Drive Fair Green Putt Sand Score

277.6 .681 .667 1.768 .550 69.10
259.6 .691 .665 1.810 .536 71.09
269.1 .657 .649 1.747 .472 70.12
267.0 .689 .673 1.763 .672 69.88
267.3 .581 .637 1.781 .521 70.71
255.6 .778 .674 1.791 .455 69.76
272.9 .615 .667 1.780 .476 70.19
265.4 .718 .699 1.790 .551 69.73

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
18
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable-Selection Procedures

 Sample Data (Part 2)

Drive Fair Green Putt Sand Score

272.6 .660 .672 1.803 .431 69.97
263.9 .668 .669 1.774 .493 70.33
267.0 .686 .687 1.809 .492 70.32
266.0 .681 .670 1.765 .599 70.09
258.1 .695 .641 1.784 .500 70.46
255.6 .792 .672 1.752 .603 69.49
261.3 .740 .702 1.813 .529 69.88
262.2 .721 .662 1.754 .576 70.27

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
19
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable-Selection Procedures

 Sample Data (Part 3)

Drive Fair Green Putt Sand Score

260.5 .703 .623 1.782 .567 70.72
271.3 .671 .666 1.783 .492 70.30
263.3 .714 .687 1.796 .468 69.91
276.6 .634 .643 1.776 .541 70.69
252.1 .726 .639 1.788 .493 70.59
263.0 .687 .675 1.786 .486 70.20
263.0 .639 .647 1.760 .374 70.81
253.5 .732 .693 1.797 .518 70.26
266.2 .681 .657 1.812 .472 70.96

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
20
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable-Selection Procedures

 Sample Correlation Coefficients

Score Drive Fair Green Putt

Drive -.154
Fair -.427 -.679
Green -.556 -.045 .421
Putt .258 -.139 .101 .354
Sand -.278 -.024 .265 .083 -.296

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
21
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable-Selection Procedures

 Minitab Output
The regression equation
Score = 74.678 - .0398(Drive) - 6.686(Fair)
- 10.342(Green) + 9.858(Putt)
Predictor Coef Stdev t-ratio p
Constant 74.678 6.952 10.74 .000
Drive -.0398 .01235 -3.22 .004
Fair -6.686 1.939 -3.45 .003
Green -10.342 3.561 -2.90 .009
Putt 9.858 3.180 3.10 .006
s = .2691 R-sq = 72.4% R-sq(adj) = 66.8%
© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
22
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Variable-Selection Procedures

 Minitab Output

Analysis of Variance
SOURCE DF SS MS F P
Regression 4 3.79469 .94867
13.10
Error .000 20 1.44865
.07243
Total 24 5.24334

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
23
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 The use of dummy variables in a multiple regression
equation can provide another approach to solving
analysis of variance and experimental design
problems.
 We will use the results of multiple regression to
perform the ANOVA test on the difference in the
means of three populations.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
24
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 Example: Reed Manufacturing
Janet Reed would like to know if there is any
significant difference in the mean number of hours
worked per week for the department managers at
her three manufacturing plants (in Buffalo,
Pittsburgh, and Detroit).
A simple random sample of five managers from
each of the three plants was taken and the number
of hours worked by each manager for the previous
week is shown on the next slide.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
25
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design

Plant 1 Plant 2 Plant 3

Observation Buffalo Pittsburgh Detroit
1 48 73 51
2 54 63 63
3 57 66 61
4 54 64 54
5 62 74 56
Sample Mean 55 68 57
Sample Variance 26.0 26.5 24.5

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
26
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 We begin by defining two dummy variables, A and
B, that will indicate the plant from which each sample
observation was selected.
 In general, if there are k populations, we need to
define k – 1 dummy variables.

A = 0, B = 0 if observation is from Buffalo plant

A = 1, B = 0 if observation is from Pittsburgh plant
A = 0, B = 1 if observation is from Detroit plant

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
27
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 Input Data

Plant 1 Plant 2 Plant 3

Buffalo Pittsburgh Detroit
A B y A B y A B y

0 0 48 1 0 73 0 1 51
0 0 54 1 0 63 0 1 63
0 0 57 1 0 66 0 1 61
0 0 54 1 0 64 0 1 54
0 0 62 1 0 74 0 1 56

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
28
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design

E(y) = expected number of hours worked

= b 0 + b 1A + b 2B

For Buffalo: E(y) = b0 + b1(0) + b2(0) = b0

For Pittsburgh: E(y) = b0 + b1(1) + b2(0) = b0 + b1
For Detroit: E(y) = b0 + b1(0) + b2(1) = b0 + b2

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
29
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design

Excel produced the regression equation:

y = 55 +13A + 2B

Plant Estimate of E(y)

Buffalo b0 = 55
Pittsburgh b0 + b1 = 55 + 13 = 68
Detroit b0 + b2 = 55 + 2 = 57

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
30
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 Next, we observe that if there is no difference in
the means:
E(y) for the Pittsburgh plant – E(y) for the Buffalo plant = 0
E(y) for the Detroit plant – E(y) for the Buffalo plant = 0

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
31
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 Because b0 equals E(y) for the Buffalo plant and
b0 + b1 equals E(y) for the Pittsburgh plant, the first
difference is equal to (b0 + b1) - b0 = b1.
 Because b0 + b2 equals E(y) for the Detroit plant, the
second difference is equal to (b0 + b2) - b0 = b2.
 We would conclude that there is no difference in the
three means if b1 = 0 and b2 = 0.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
32
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 The null hypothesis for a test of the difference of
means is
H 0: b 1 = b 2 = 0

 To test this null hypothesis, we must compare the

value of MSR/MSE to the critical value from an F
distribution with the appropriate numerator and
denominator degrees of freedom.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
33
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 ANOVA Table Produced by Excel

Source of Sum of Degrees of Mean

Variation Squares Freedom Squares F p
Regression 490 2 245 9.55 .003
Error 308 12 25.667
Total 798 14

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
34
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Multiple Regression Approach to
Experimental Design
 At a .05 level of significance, the critical value of
F with k – 1 = 3 – 1 = 2 numerator d.f. and nT – k =
15 – 3 = 12 denominator d.f. is 3.89.
 Because the observed value of F (9.55) is greater than
the critical value of 3.89, we reject the null hypothesis.
 Alternatively, we reject the null hypothesis because
the p-value of .003 < a = .05.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
35
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

 Often, the data used for regression studies in

business and economics are collected over time.
 It is not uncommon for the value of y at one time
period to be related to the value of y at previous time
periods.
 In this case, we say autocorrelation (or serial
correlation) is present in the data.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
36
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

 With positive autocorrelation, we expect a positive

residual in one period to be followed by a positive
residual in the next period.
 With positive autocorrelation, we expect a negative
residual in one period to be followed by a negative
residual in the next period.
 With negative autocorrelation, we expect a positive
residual in one period to be followed by a negative
residual in the next period, then a positive residual,
and so on.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
37
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

 When autocorrelation is present, one of the

regression assumptions is violated: the error terms
are not independent.
 When autocorrelation is present, serious errors can be
made in performing tests of significance based upon
the assumed regression model.
 The Durbin-Watson statistic can be used to detect
first-order autocorrelation.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
38
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

 Durbin-Watson Test Statistic

n
2
 ( et  et  1 )
d  t 2 n
2
 et2
t 1

The ith residual is denoted ei  y i  yˆ i

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
39
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

 Durbin-Watson Test Statistic

• The statistic ranges in value from zero to four.
• If successive values of the residuals are close
together (positive autocorrelation is present),
the statistic will be small.
• If successive values are far apart (negative
autocorrelation is present), the statistic will
be large.
• A value of two indicates no autocorrelation.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
40
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

 Suppose the values of e (residuals) are not

independent but are related in the following manner:
et = r et-1 + zt
where r is a parameter with an absolute value less than
one and zt is a normally and independently distributed
random variable with a mean of zero and variance of s 2.
 We see that if r = 0, the error terms are not related.
 The Durbin-Watson test uses the residuals to
determine whether r = 0.

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
41
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

 The null hypothesis always is:

H0 :   0 there is no autocorrelation
 The alternative hypothesis is:
Ha :   0 to test for positive autocorrelation

Ha :   0 to test for negative autocorrelation

Ha :   0 to test for positive or negative

autocorrelation

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
42
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

A Sample Of Critical Values For The

Durbin-Watson Test For Autocorrelation
Significance Points of dL and dU: a = .05
Number of Independent Variables
1 2 3 4 5
n dL dU dL dU dL dU dU dU dU dU
15 1.08 1.36 0.95 1.54 0.82 1.75 0.69 1.97 0.56 2.21
16 1.10 1.37 0.98 1.54 0.86 1.73 0.74 1.93 0.62 2.15
17 1.13 1.38 1.02 1.54 0.90 1.71 0.78 1.90 0.67 2.10
18 1.16 1.39 1.05 1.53 0.93 1.69 0.82 1.87 0.71 2.06

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
43
or duplicated, or posted to a publicly accessible website, in whole or in part.
 Autocorrelation and the Durbin-Watson Test

Positive Incon- No evidence of

autocor- clusive positive autocorrelation
relation
0 dL dU 2 4-dU 4-dL 4

No evidence of Incon- Negative

negative autocorrelation clusive autocor-
relation
0 dL dU 2 4-dU 4-dL 4

Positive Incon- No evidence of Incon- Negative

autocor- clusive autocorrelation clusive autocor-
relation relation
0 dL dU 2 4-dU 4-dL 4
© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
44
or duplicated, or posted to a publicly accessible website, in whole or in part.
 End of Chapter 16

© 2011 Cengage Learning. All Rights Reserved. May not be scanned, copied
Slide
45
or duplicated, or posted to a publicly accessible website, in whole or in part.