Basic Business

Statistics

Multiple Regression Model Building

Chap 15-1
 Linear vs. Nonlinear Fit

Y Y

X X

residuals
residuals

X X


Linear fit does not give Nonlinear fit gives
random residuals random residuals
 Nonlinear Relationships

The relationship between the dependent

variable and an independent variable may not
be linear
Can review the scatter plot to check for non-
linear relationships
Example: Quadratic model

Yi = β0 + β1X1i + β 2 X + ε i
2
1i

 The second independent variable is the square

of the first variable
 Quadratic Regression Model

Model form:

Yi = β0 + β1X1i + β 2 X + ε i2
1i

 where:
β0 = Y intercept
β1 = regression coefficient for linear effect of X on Y
β2 = regression coefficient for quadratic effect on Y
εi = random error in Y for observation i
 Quadratic Regression Model

Yi = β0 + β1X1i + β 2 X1i2 + ε i
Quadratic models may be considered when the scatter
plot takes on one of the following shapes:
Y Y Y Y

X1 X1 X1 X1
β1 > 0 β1 > 0 β1 < 0 β1 < 0
β2 > 0 β2 < 0 β2 > 0 β2 < 0
β1 = the coefficient of the linear term
β2 = the coefficient of the squared term
 Testing for Significance:
Quadratic Effect

Testing the Quadratic Effect

 Compare quadratic regression equation

Yi = b0 + b1X1i + b 2 X1i2

with the linear regression equation

Yi = b0 + b1X1i
 Testing for Significance:
Quadratic Effect
(continued)
Testing the Quadratic Effect
 Consider the quadratic regression equation

Yi = b0 + b1X1i + b 2 X1i2

Hypotheses

H0: β2 = 0 (The quadratic term does not improve the model)

H1: β2 ≠ 0 (The quadratic term improves the model)
 Testing for Significance:
Quadratic Effect
(continued)
 Testing the Quadratic Effect
Hypotheses
H0: β2 = 0 (The quadratic term does not improve the model)
H1: β2 ≠ 0 (The quadratic term improves the model)

where:
 The test statistic is
b2 = squared term slope
b2 − β2 coefficient
t STAT =
Sb 2 β2 = hypothesized slope (zero)
Sb = standard error of the slope
d.f. = n − 3 2
 Testing for Significance:
Quadratic Effect
(continued)
 Testing the Quadratic Effect

Compare adjusted r2 from simple regression model to

adjusted r2 from the quadratic model

 If adjusted r2 from the quadratic model is larger than

the adjusted r2 from the simple model, then the
quadratic model is likely a better model
 Example: Quadratic Model

Filter Purity increases as filter time increases:

Purity Time
3 1
7 2 Purity vs. Time

8 3
100
15 5
22 7 80
33 8
60
40 10
Purity

54 12
40
67 13
70 14 20

78 15
0
85 15
0 5 10 15 20
87 16 Time
99 17
 Example: Quadratic Model

(continued)
Simple regression results:
^Y = -11.283 + 5.985 Time

Standard
Coefficients Error t Stat P-value
t statistic and r2 are all high,
Intercept -11.28267 3.46805 -3.25332 0.00691 but the residuals are not
Time 5.98520 0.30966 19.32819 2.078E-10 random:

Time Residual Plot

Regression Statistics
10
R Square 0.96888

Residuals
Adjusted R Square 0.96628 5
Standard Error 6.15997 0
-5 0 5 10 15 20
-10
Time
 Example: Quadratic Model in Excel
& Minitab

(continued)
 Quadratic regression results:
^ = 1.539 + 1.565 Time + 0.245 (Time)2
Y
Excel Minitab
Standard The regression equation is
Coefficients Error t Stat P-value Purity = 1.54 + 1.56 Time + 0.245 Time Squared
Intercept 1.53870 2.24465 0.68550 0.50722
Predictor Coef SE Coef T P
Time 1.56496 0.60179 2.60052 0.02467 Constant 1.5390 2.24500 0.69 0.507
Time 1.5650 0.60180 2.60 0.025
Time-squared 0.24516 0.03258 7.52406 1.165E-05
Time Squared 0.24516 0.03258 7.52 0.000

S = 2.59513 R-Sq = 99.5% R-Sq(adj) = 99.4%

The quadratic term is statistically significant (p-value very small)

 Example: Quadratic Model in Excel
& Minitab

(continued)
 Quadratic regression results:
^ = 1.539 + 1.565 Time + 0.245 (Time)2
Y

Regression Statistics The regression equation is

Purity = 1.54 + 1.56 Time + 0.245 Time Squared
R Square 0.99494
Adjusted R Predictor Coef SE Coef T P
Square 0.99402 Constant 1.5390 2.24500 0.69 0.507
Time 1.5650 0.60180 2.60 0.025
Standard Error 2.59513 Time Squared 0.24516 0.03258 7.52 0.000

S = 2.59513 R-Sq = 99.5% R-Sq(adj) = 99.4%

The adjusted r2 of the quadratic model is higher than the adjusted r2 of the
simple regression model. The quadratic model explains 99.4% of the
variation in Y.
 Example: Quadratic Model Residual
Plots

(continued)
Quadratic regression results:
Y = 1.539 + 1.565 Time + 0.245 (Time)2
Time Residual Plot Time-squared Residual Plot
10 10

Residuals
Residuals

5 5

0 0
0 5 10 15 20 0 100 200 300 400
-5 -5
Time Time-squared

The residuals plotted versus both Time and Time-squared show a random
pattern.
 Collinearity
(continued)

Including two highly correlated independent

variables can adversely affect the regression
results
 No new information provided
 Some Indications of Strong
Collinearity

Incorrect signs on the coefficients

Large change in the value of a previous
coefficient when a new variable is added to the
model
A previously significant variable becomes
non-significant when a new independent
variable is added
 Detecting Collinearity
(Variance Inflationary Factor)

VIFj is used to measure collinearity:

1
VIFj =
1− R j
2

where R2j is the coefficient of determination of

variable Xj with all other X variables

If VIFj > 5, Xj is highly correlated with

the other independent variables
 Example: Pie Sales

Pie Price Advertising

Week Sales ($) ($100s)
1 350 5.50 3.3
2 460 7.50 3.3 Recall the multiple regression
3 350 8.00 3.0
equation of chapter 2:
4 430 8.00 4.5
5 350 6.80 3.0
6 380 7.50 4.0
7 430 4.50 3.0 Sales = b0 + b1 (Price)
8 470 6.40 3.7
9 450 7.00 3.5 + b2 (Advertising)
10 490 5.00 4.0
11 340 7.20 3.5
12 300 7.90 3.2
13 440 5.90 4.0
14 450 5.00 3.5
15 300 7.00 2.7
 Detecting Collinearity in Excel
using PHStat

PHStat / regression / multiple regression …

Check the “variance inflationary factor (VIF)” box

Regression Analysis
Output for the pie sales example:
Price and all other X
Regression Statistics  VIF is < 5
Multiple R 0.030438  There is no evidence of
R Square 0.000926
Adjusted R
collinearity between Price
Square -0.075925 and Advertising
Standard Error 1.21527
Observations 15
VIF 1.000927