Regression Analysis

Regression Analysis
 Regression Analysis is used to:
1)understand the relation between two variables
2)predict the value of one variable based on another
variable.
 A regression model is comprised of a dependent
(response) variable and an independent (predictor)
variable.

Independent Variable(s) Dependent Variable

Prediction Relationship
 Regression Analysis

 Linear regression estimates the coefficients of the

linear equation, involving one or more independent
variables that best predict the value of the
dependent variable.

 If you believe that none of your predictor variables

is correlated with the errors in your dependent
variable, you can use the linear regression
procedure
 Simple Linear Regression
The Scatter Diagram – used to graphically
investigate relationship between the dependent
and independent variables

100
Y 50
0
0 20 X 40 60
Plot of all (Xi , Yi) pairs
 Types of Regression Models

Positive Linear Relationship Relationship NOT Linear

Negative Linear Relationship No Relationship

 Simple Linear Regression Model
 Regression models are used to test if a relationship
exist between variables; that is to use one variable to
predict another. However, there is a random error that
cannot be predicted.

Y intercept Random
Error

Yi   0  1 X i   i
Dependent
(Response)
Slope
Variable Independent
(Predictor/Explanatory)
Variable
 Population Linear Regression
Model

Y Yi   0  1X i   i Observed
Value

i = Random Error

   0  1X i
YX

X
Observed Value
Sample Linear Regression Model

yˆ i  b0  b1 xi

yi = Predicted Value of Y for observation i

xi = Value of X for observation i

b0 = Sample Y - intercept used as estimate

of the population 0
b1 = Sample Slope used as estimate of the
population 1
Sample Linear Regression Model

Sample data are used to estimate the true

values for the intercept and slope.

yˆ i  b0  b1 xi
The difference between the actual value of Y
and the predicted value (using sample data) is
known as the error.
Error = actual value – predicted value
 Yi
Sample Linear Regression Model

yˆ i  b0  b1 xi
n
 n  n 
n  xi yi    xi   yi 
b1  i 1  i 1  i 1 
2
n
 n

n  xi    xi 
2

i 1  i 1 
b0  y  b1 x
Table 3.1. Intelligence Test Scores and Freshmen Chemistry Grades
Test Score Chemistry
Student (x) Grade (y)
1 65 85
2 50 74
3 55 76
4 65 90
5 55 85
6 70 87
7 65 94
8 70 98
9 55 81
10 70 91
11 50 76
12 55 74
 Figure 3.1. Scatter Diagram with regression line

100

95
yˆ i  b0  b1 xi
Chemistry Grade

90

85
Determining point
80 estimate of b0 and b1
Using the Method of
75
Least Squares
70
40 45 50 55 60 65 70 75
Intelligence Test Score
 Measures of Variation: The
Sum of Squares

Y 
SSE =(Yi - Yi )2
_ b Xi
 b0 + 1
SST = (Yi - Y) 2
Yi =
 _
SSR = (Yi - Y)2
_
Y

X
Xi
 Method of Least Squares
n n
SSE   e   ( yi  b0  b1 xi )
2
i
2

i 1 i 1
n

The process of differentiating i 1 i

e 2
with respect
to b0 and b1 and equating the derivatives to zero
n
 n  n 
n xi yi    xi   yi 
b1  i 1  i 1  i 1  b0  y  b1 x
2
n
 n

n xi    xi 
2

i 1  i 1 
 Method of Least Squares

n
 n  n 
n xi yi    xi   yi 
b1  i 1  i 1  i 1 
2
n
 n

n xi    xi 
2

i 1  i 1 

 (x i  x )( yi  y )
b1  i 1
n

 i
( x
i 1
 x ) 2
Table 3.1. Intelligence Test Scores and Freshmen Chemistry Grades
Test Score Chemistry
Student (x) Grade (y)
1 65 85
2 50 74
b1  0.897
3 55 76
b0  30.056
4 65 90
5 55 85
yˆ i  b0  b1 xi
6 70 87
yˆ i  30.056  0.897 xi
7 65 94
8 70 98
9 55 81
10 70 91
11 50 76
12 55 74
 Figure 3.1. Scatter Diagram with regression line

100

95 yˆ i  30.056  0.897 xi
Chemistry Grade

90

85

80

75 The slope of 0.897 means for

each increase of one unit in
70
intelligence Test Score (X),
40 45 50 55 60 65 70 75
the Chemistry Grade (Y) is
Intelligence Test Score estimated to increase by
0.897 units.
Using SPSS
Graphs To add regression line Use
Scatter
Simple SPSS Chart Editor

100 Chart
Options
Fit Line

90

Regression
Line
80
Chemistry Grade

Regression
70 Rsq = 0.7438 Prediction
40 50 60 70 80
Line
Test Score
Using SPSS
Analyze
Regression
Linear

Coefficientsa a
Coefficients
Unstandardized Standardized
Unstandardized Standardized
Coefficients Coefficients
Coefficients Coefficients
Model B Std. Error Beta t Sig.
Model B Std. Error Beta t Sig.
1 (Constant) 30.043 10.137 2.964 .014
1 (Constant) 30.043 10.137 2.964 .014
Test Score .897 .167 .862 5.389 .000
Test Score .897 .167 .862 5.389 .000
a. Dependent Variable: Chemistry Grade
a. Dependent Variable: Chemistry Grade

yˆ i  30.043  0.897 xi
Using SPSS Standard Deviation
Analyze Coefficient of
Regression Correlation Determination around the
Linear regression line

Model Summaryb b
Model Summary
Adjusted Std. Error of
Adjusted Std. Error of
Model R R Square R Square the Estimate
Model R a R Square R Square the Estimate
1 .862 .744 .718 4.319
1 .862a .744 .718 4.319 Measures of
a. Predictors: (Constant), Test Score
a. Predictors: (Constant), Test Score
b. Dependent Variable: Chemistry Grade
Variation
b. Dependent Variable: Chemistry Grade

ANOVAb b
ANOVA
Sum of
Sum of
Model Squares df Mean Square F Sig.
Model Squares df Mean Square F Sig. a
1 Regression 541.693 1 541.693 29.036 .000 a
1 Regression 541.693 1 541.693 29.036 .000
Residual 186.557 10 18.656
Residual 186.557 10 18.656
Total 728.250 11
Total 728.250 11
a. Predictors: (Constant), Test Score
a. Predictors: (Constant), Test Score
b. Dependent Variable: Chemistry Grade
b. Dependent Variable: Chemistry Grade
 Testing the Significance of b

 Similar to a test on r in the one-predictor case

t =(0.8972136-0)/0.1665043 = 5.39 H0 is rejected,

i.e. the regression line has a nonzero slope
 2
Variance Explained – r
r2 tells us the proportion of variance in Y which is
explained by X

r 
2
SS regression

SSYˆ

 Yˆ  Y 2

 Y  Y 
2
SS total SSY

• a ratio reflecting the proportion of variance captured

by our model relative to the overall variance in our
data
• highly interpretable: r2 =.50 means 50% of the
variance in Y is explained by X
 Linear Regression Assumptions

For Linear Models

 1. Normality
 Y Values Are Normally Distributed For Each
X
 Probability Distribution of Error is Normal

 2. Homoscedasticity (Constant Variance)

 3. Independence of Errors
 Variation of Errors Around the
Regression Line

y values are normally distributed

f(e) around the regression line.
For each x value, the “spread” or
variance around the regression line
is the same.

Y
X2
X1
X
Regression Line
 Residual Analysis

 Purposes
 Examine Linearity
 Evaluate violations of assumptions

 Graphical Analysis of Residuals

 Plot residuals Vs. Xi values 
 Difference between actual Yi & predicted Yi

 Studentized residuals:
 Allows consideration for the magnitude of the
residuals
 Residual Analysis for Linearity

Not Linear
 Linear
e e

X X
Residual Analysis for Homoscedasticity

Heteroscedasticity 
SR
Homoscedasticity
SR

X X

Using Standardized Residuals

• Predict Chemistry Grade
• Predict residual
• Predict studentized residual
• Predict standardized residual
 Residual Analysis for
Normality

kdensity r, normal swilk r  Normal

kernel = epanechnikov, bandwidth = 2.25
kernel = epanechnikov, bandwidth = 2.25
Kernel density estimate
Kernel density estimate
Normal density
Normal density
.1
.1

.08
.08
Density

.06
Density

.06

.04
.04

Kernel density estimate

Kernel density estimate .02
.02
-10 -5 0 5 10
-10 -5 0
Residuals 5 10
Residuals
Residual Analysis for Linearity

scatter r X, yline(0)

5
5
 Linear

Residuals
Residuals
0
0

-5
-5
50 55 60 65 70
50 55 60
Test Score 65 70
Test Score
 Residual Analysis for
Homoscedasticity

 Homoscedasticity
scatter r1 X, yline(0) scatter sr X, yline(0)
2 2
2 2

1 1
1 Standardized residuals 1

Studentized residuals
Standardized residuals

Studentized residuals
0 0
0 0

-1 -1
-1 -1

-2 -2
50 55 60 65 70 -2 50 55 60 65 70 -2
50 55 60
Test Score 65 70 50 55 60
Test Score 65 70
Test Score Test Score

Using Standardized Residuals Using Studentized Residuals

Residual Analysis for
Homoscedasticity

hettest
 Homoscedasticity
Residual Analysis for
Independence

scatter r obs, yline(0)

 Independent
5
5

Residuals
Residuals
0
0

-5
-5
0 5 10 15
0 5 obs 10 15
obs
Residual Analysis for
Independence

Durbin-Watson Statistic.
The D-W statistic is
defined as:
 Independent