0% found this document useful (0 votes)
125 views

Topic03 Correlation Regression

0 1 1 0 1 1 0 1 1 0 0 0  = -0.49

Uploaded by

pradeep
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
125 views

Topic03 Correlation Regression

0 1 1 0 1 1 0 1 1 0 0 0  = -0.49

Uploaded by

pradeep
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 81

Correlation and

Regression
Cal State Northridge
427
Ainsworth
Major Points - Correlation
 Questions answered by correlation
 Scatterplots

 An example

 The correlation coefficient

 Other kinds of correlations

 Factors affecting correlations

 Testing for significance


The Question

 Are two variables related?


 Does one increase as the other increases?
 e. g. skills and income
 Does one decrease as the other increases?
 e. g. health problems and nutrition
 How can we get a numerical measure of
the degree of relationship?
Scatterplots

 AKA scatter diagram or scattergram.


 Graphically depicts the relationship
between two variables in two dimensional
space.
Direct Relationship
Scatterplot:Video Games and Alcohol Consumption

20
Average Number of Alcoholic Drinks

18
16
14
Per Week

12
10
8
6
4
2
0
0 5 10 15 20 25
Average Hours of Video Games Per Week
Inverse Relationship
Scatterplot: Video Games and Test Score

100
90
80
70
Exam Score

60
50
40
30
20
10
0
0 5 10 15 20
Average Hours of Video Games Per Week
An Example

 Does smoking cigarettes increase systolic


blood pressure?
 Plotting number of cigarettes smoked per
day against systolic blood pressure
 Fairly moderate relationship
 Relationship is positive
Trend?
170

160

150

140

130

120
SYSTOLIC

110

100
0 10 20 30

SMOKING
Smoking and BP

 Note relationship is moderate, but real.


 Why do we care about relationship?

 What would conclude if there were no


relationship?
 What if the relationship were near perfect?
 What if the relationship were negative?
Heart Disease and Cigarettes

 Data on heart disease and cigarette


smoking in 21 developed countries
(Landwehr and Watkins, 1987)
 Data have been rounded for computational
convenience.
 The results were not affected.
Country Cigarettes CHD

The Data
1 11 26
2 9 21
3 9 24
4 9 21
5 8 19
6 8 13
7 8 19
Surprisingly, the 8 6 11
9 6 23
U.S. is the first 10 5 15
country on the list- 11 5 13
12 5 4
-the country 13 5 18
with the highest 14 5 12
15 5 3
consumption and 16 4 11
17 4 15
highest mortality. 18 4 6
19 3 13
20 3 4
21 3 14
Scatterplot of Heart Disease

 CHD Mortality goes on ordinate (Y axis)


 Why?

 Cigarette consumption on abscissa (X


axis)
 Why?

 What does each dot represent?


 Best fitting line included for clarity
30

20

10

{X = 6, Y = 11}

0
2 4 6 8 10 12

Cigarette Consumption per Adult per Day


What Does the Scatterplot Show?

 As smoking increases, so does coronary


heart disease mortality.
 Relationship looks strong

 Not all data points on line.

 This
gives us “residuals” or “errors of
prediction”
 To be discussed later
Correlation

 Co-relation
 The relationship between two variables

 Measured with a correlation coefficient

 Most popularly seen correlation


coefficient: Pearson Product-Moment
Correlation
Types of Correlation
 Positive correlation
 High values of X tend to be associated with
high values of Y.
 As X increases, Y increases
 Negative correlation
 High values of X tend to be associated with
low values of Y.
 As X increases, Y decreases
 No correlation
 No consistent tendency for values on Y to
increase or decrease as X increases
Correlation Coefficient

 A measure of degree of relationship.


 Between 1 and -1

 Sign refers to direction.

 Based on covariance
 Measure of degree to which large scores on
X go with large scores on Y, and small scores
on X go with small scores on Y
 Think of it as variance, but with 2 variables
instead of 1 (What does that mean??)
18
Covariance
 Remember that variance is:
( X  X ) 2
( X  X )( X  X )
VarX  
N 1 N 1
 The formula for co-variance is:

( X  X )(Y  Y )
Cov XY 
N 1
 How this works, and why?
 When would covXY be large and positive?
Large and negative?
Country X (Cig.) Y (CHD) (X  X ) (Y  Y ) ( X  X ) * (Y  Y )
1 11 26 5.05 11.48 57.97
2 9 21 3.05 6.48 19.76
3 9 24 3.05 9.48 28.91
4 9 21 3.05 6.48 19.76
5 8 19 2.05 4.48 9.18
6 8 13 2.05 -1.52 -3.12
7 8 19 2.05 4.48 9.18
8 6 11 0.05 -3.52 -0.18
9 6 23 0.05 8.48 0.42

Example
10 5 15 -0.95 0.48 -0.46
11 5 13 -0.95 -1.52 1.44
12 5 4 -0.95 -10.52 9.99
13 5 18 -0.95 3.48 -3.31
14 5 12 -0.95 -2.52 2.39
15 5 3 -0.95 -11.52 10.94
16 4 11 -1.95 -3.52 6.86
17 4 15 -1.95 0.48 -0.94
18 4 6 -1.95 -8.52 16.61
19 3 13 -2.95 -1.52 4.48
20 3 4 -2.95 -10.52 31.03
21 3 14 -2.95 -0.52 1.53
Mean 5.95 14.52
SD 2.33 6.69
Sum 222.44
Example
21

( X  X )(Y  Y ) 222.44
Covcig .&CHD    11.12
N 1 21  1
 What the heck is a covariance?
 I thought we were talking about
correlation?
Correlation Coefficient

 Pearson’s Product Moment Correlation


 Symbolized by r

 Covariance ÷ (product of the 2 SDs)

Cov XY
r
s X sY
 Correlation is a standardized
covariance
Calculation for Example

 CovXY = 11.12
 sX = 2.33

 sY = 6.69

cov XY 11.12 11.12


r    .713
s X sY (2.33)(6.69) 15.59
Example

 Correlation = .713
 Sign is positive

 Why?

 If sign were negative


 What would it mean?
 Would not alter the degree of relationship.
Other calculations
25

Z-score method

r
 z z x y

N 1

 Computational (Raw Score) Method


N  XY   X  Y
r
 N  X 2  ( X )2   N  Y 2  ( Y )2 
Other Kinds of Correlation
 Spearman Rank-Order Correlation
Coefficient (rsp)
 used with 2 ranked/ordinal variables
 uses the same Pearson formula
Attractiveness Symmetry
3 2
4 6
1 1
2 3
5 4
6 5 26
rsp = 0.77
Other Kinds of Correlation
 Point biserial correlation coefficient (rpb)
 used with one continuous scale and one
nominal or ordinal or dichotomous scale.
 uses the same Pearson formula
Attractiveness Date?
3 0
4 0
1 1
2 1
5 1
6 0
rpb = -0.49 27
Other Kinds of Correlation
 Phi coefficient ()
 used with two dichotomous scales.
 uses the same Pearson formula

Attractiveness Date?
0 0
1 0
1 1
1 1
0 0
1 1
 = 0.71 28
Factors Affecting r
 Range restrictions
 Looking at only a small portion of the total
scatter plot (looking at a smaller portion of
the scores’ variability) decreases r.
 Reducing variability reduces r
 Nonlinearity
 The Pearson r (and its relatives) measure the
degree of linear relationship between two
variables
 If a strong non-linear relationship exists, r will
provide a low, or at least inaccurate measure
of the true relationship.
Factors Affecting r
 Heterogeneous subsamples
 Everyday examples (e.g. height and weight
using both men and women)
 Outliers
 Overestimate Correlation
 Underestimate Correlation
Countries With Low Consumptions
Data With Restricted Range

Truncated at 5 Cigarettes Per Day


20

18

16
CHD Mortality per 10,000

14

12

10

4
2
2.5 3.0 3.5 4.0 4.5 5.0 5.5

Cigarette Consumption per Adult per Day


Truncation
32
Non-linearity
33
Heterogenous samples
34
Outliers
35
Testing Correlations
36

 So you have a correlation. Now what?


 In terms of magnitude, how big is big?
 Smallcorrelations in large samples are “big.”
 Large correlations in small samples aren’t
always “big.”
 Depends upon the magnitude of the
correlation coefficient
AND
 The size of your sample.
Testing r

 Population parameter = 
 Null hypothesis H0:  = 0

 Test
of linear independence
 What would a true null mean here?
 What would a false null mean here?

 Alternative hypothesis (H1)   0


 Two-tailed
Tables of Significance
 We can convert r to t and test for
significance:

N 2
tr
1 r 2

 Where DF = N-2
Tables of Significance
 In our example r was .71
 N-2 = 21 – 2 = 19

N 2 19 19
tr  .71*  .71*  6.90
1 r 2
1  .712
.4959

 T-crit (19) = 2.09


 Since 6.90 is larger than 2.09 reject  = 0.
Computer Printout
 Printout gives test of significance.
Correlations

CIGARET CHD
CIGARET Pears on Correlation 1 .713**
Sig. (2-tailed) . .000
N 21 21
CHD Pears on Correlation .713** 1
Sig. (2-tailed) .000 .
N 21 21
**. Correlation is significant at the 0.01 level (2-tailed).
Regression
What is regression?
42

 How do we predict one variable from


another?
 How does one variable change as the

other changes?
 Influence
Linear Regression
43

 A technique we use to predict the most


likely score on one variable from those
on another variable
 Uses the nature of the relationship (i.e.
correlation) between two variables to
enhance your prediction
Linear Regression: Parts
44

 Y - the variables you are predicting


 i.e. dependent variable
 X - the variables you are using to predict
 i.e. independent variable
 Ŷ - your predictions (also known as Y’)
Why Do We Care?
45

 We may want to make a prediction.


 More likely, we want to understand the
relationship.
 How fast does CHD mortality rise with a
one unit increase in smoking?
 Note: we speak about predicting, but
often don’t actually predict.
An Example
46

 Cigarettes and CHD Mortality again


 Data repeated on next slide

 We want to predict level of CHD


mortality in a country averaging 10
cigarettes per day.
Country Cigarettes CHD
1 11 26

47
The Data 2
3
9
9
21
24
4 9 21
5 8 19
Based on the data we have 6 8 13
what would we predict the 7
8
8
6
19
11
rate of CHD be in a country 9 6 23
10 5 15
that smoked 10 cigarettes on 11 5 13
12 5 4
average? 13 5 18
14 5 12
First, we need to establish a 15 5 3
4 11
prediction of CHD from 16
17 4 15
smoking… 18
19
4
3
6
13
20 3 4
21 3 14
30

We predict a
20
CHD rate of
about 14
Regression
Line

10

For a country that


smokes 6 C/A/D…
0
2 4 6 8 10 12

Cigarette Consumption per Adult per Day

48
Regression Line
49

 Formula

Yˆ  bX  a

 = the predicted value of Y (e.g. CHD
mortality)
 X = the predictor variable (e.g. average
cig./adult/country)
Regression Coefficients
50

 “Coefficients” are a and b


 b = slope

 Change in predicted Y for one unit change


in X
 a = intercept
 value of Yˆ when X = 0
Calculation
51

 Slope cov XY  sy 
b  2 or b  r  
sX  sx 
N  XY   X  Y
or b 
 N  X 2  ( X ) 2 

 Intercept a  Y bX
For Our Data
52

 CovXY = 11.12
 s2X = 2.332 = 5.447

 b = 11.12/5.447 = 2.042

 a = 14.524 - 2.042*5.952 = 2.32

 See SPSS printout on next slide

Answers are not exact due to rounding error and desire to match
SPSS.
SPSS Printout
53
Note:
54

 The values we obtained are shown on


printout.
 The intercept is the value in the B

column labeled “constant”


 The slope is the value in the B column
labeled by name of predictor variable.
Making a Prediction
55

 Second, once we know the relationship


we can predict
Yˆ  bX  a  2.042 X  2.367
Yˆ  2.042*10  2.367  22.787
 We predict 22.77 people/10,000 in a
country with an average of 10 C/A/D
will die of CHD
Accuracy of Prediction
 Finnish smokers smoke 6 C/A/D
 We predict:

Yˆ  bX  a  2.042 X  2.367
Yˆ  2.042*6  2.367  14.619
 They actually have 23 deaths/10,000
 Our error (“residual”) =

23 - 14.619 = 8.38
a large error
56
30

CHD Mortality per 10,000 Residual

20

Prediction

10

0
2 4 6 8 10 12

Cigarette Consumption per Adult per Day

57
Residuals
58

 When we predict Ŷ for a given X, we will


sometimes be in error.
 Y – Ŷ for any X is a an error of estimate

 Also known as: a residual

 We want to Σ(Y- Ŷ) as small as possible.

 BUT, there are infinitely many lines that can do


this.
 Just draw ANY line that goes through the
mean of the X and Y values.
 Minimize Errors of Estimate… How?
Minimizing Residuals
59

 Again, the problem lies with this


definition of the mean:

 ( X  X )  0
 So, how do we get rid of the 0’s?
 Square them.
Regression Line:
A Mathematical Definition
 The regression line is the line which when
drawn through your data set produces the
smallest value of:

 (Y  Y )
ˆ 2

 Called the Sum of Squared Residual or


SSresidual
 Regression line is also called a “least squares
line.” 60
Summarizing Errors of Prediction
61

 Residual variance
 The variability of predicted values
ˆ
(Yi  Yi ) 2
SSresidual
s2
Y Yˆ
 
N 2 N 2
Standard Error of Estimate
62

 Standard error of estimate


 Thestandard deviation of predicted
values
ˆ
(Yi  Yi ) 2
SSresidual
sY Yˆ  
N 2 N 2
 A common measure of the accuracy of
our predictions
 We want it to be as small as possible.
Country X (Cig.) Y (CHD) Y' (Y - Y') (Y - Y')2

Example
1 11 26 24.829 1.171 1.371
2 9 21 20.745 0.255 0.065
3 9 24 20.745 3.255 10.595
63 4 9 21 20.745 0.255 0.065
5 8 19 18.703 0.297 0.088
(Yi  Yˆi ) 2 440.756
6 8 13 18.703 -5.703 32.524 2
s
Y Yˆ
   23.198
7 8 19 18.703 0.297 0.088 N 2 21  2
8 6 11 14.619 -3.619 13.097
9 6 23 14.619 8.381 70.241 (Yi  Yˆi )2 440.756
10 5 15 12.577 2.423 5.871 sY Yˆ   
11 5 13 12.577 0.423 0.179
N 2 21  2
12 5 4 12.577 -8.577 73.565  23.198  4.816
13 5 18 12.577 5.423 29.409
14 5 12 12.577 -0.577 0.333
15 5 3 12.577 -9.577 91.719
16 4 11 10.535 0.465 0.216
17 4 15 10.535 4.465 19.936
18 4 6 10.535 -4.535 20.566
19 3 13 8.493 4.507 20.313
20 3 4 8.493 -4.493 20.187
21 3 14 8.493 5.507 30.327
Mean 5.952 14.524
SD 2.334 6.690
Sum 0.04 440.757
Regression and Z Scores
64

 When your data are standardized (linearly


transformed to z-scores), the slope of the
regression line is called β
 DO NOT confuse this β with the β

associated with type II errors. They’re


different.
 When we have one predictor, r = β

 Zy = βZx, since A now equals 0


Partitioning Variability
65

 Sums of square deviations


 Total
SStotal   (Y  Y )
2

SSregression   (Yˆ  Y )
2
 Regression

 Residual we already covered


SSresidual   (Y  Yˆ )
2

 SStotal = SSregression + SSresidual


Partitioning Variability
66

 Degrees of freedom
 Total
 dftotal =N-1
 Regression
 dfregression = number of predictors
 Residual
 dfresidual = dftotal – dfregression
 dftotal = dfregression + dfresidual
Partitioning Variability
67

 Variance (or Mean Square)


 Total Variance
 s2total = SStotal/ dftotal
 Regression Variance
 s2regression = SSregression/ dfregression
 Residual Variance
 s2residual = SSresidual/ dfresidual
Country X (Cig.) Y (CHD) Y' (Y - Y') (Y - Y')2 (Y' - Ybar) (Y - Ybar)
1 11 26 24.829 1.171 1.371 106.193 131.699
2 9 21 20.745 0.255 0.065 38.701 41.939
3 9 24 20.745 3.255 10.595 38.701 89.795
68
4 9 21 20.745 0.255 0.065 38.701 41.939
5 8 19 18.703 0.297 0.088 17.464 20.035
6 8 13 18.703 -5.703 32.524 17.464 2.323
7 8 19 18.703 0.297 0.088 17.464 20.035
8 6 11 14.619 -3.619 13.097 0.009 12.419
9
10
11
6
5
5
23
15
13
14.619
12.577
12.577
8.381
2.423
0.423
70.241
5.871
0.179
0.009
3.791
3.791
71.843
0.227
2.323
Example
12 5 4 12.577 -8.577 73.565 3.791 110.755
13 5 18 12.577 5.423 29.409 3.791 12.083
14 5 12 12.577 -0.577 0.333 3.791 6.371
15 5 3 12.577 -9.577 91.719 3.791 132.803
16 4 11 10.535 0.465 0.216 15.912 12.419
17 4 15 10.535 4.465 19.936 15.912 0.227
18 4 6 10.535 -4.535 20.566 15.912 72.659
19 3 13 8.493 4.507 20.313 36.373 2.323
20 3 4 8.493 -4.493 20.187 36.373 110.755
21 3 14 8.493 5.507 30.327 36.373 0.275
Mean 5.952 14.524
SD 2.334 6.690
Sum 0.04 440.757 454.307 895.247
Y' = (2.04*X) + 2.37
Example
SSTotal   (Y  Y )  895.247; dftotal  21  1  20
2
69

  (Y  Y )  454.307; df regression  1 (only 1 predictor)


ˆ 2
SSregression

  (Y  Y )  440.757; df residual  20  1  19
ˆ 2
SSresidual


2
(Y  Y ) 895.247
s2
   44.762
N 1
total
20
 (Y  Y )
ˆ 2
454.307
s2
regression    454.307
1 1
 (Y  Y )
ˆ 2
440.757
s2
   23.198
N 2
residual
19
2
Note : sresidual  sY Yˆ
Coefficient of Determination
70

 It is a measure of the percent of


predictable variability
r 2  the correlation squared
or
SS regression
r 
2

SSY
 The percentage of the total variability in
Y explained by X
2
r for our example
71

 r = .713
 r 2 = .7132 =.508

SSregression 454.307
r 
2
  .507
 or SSY 895.247

 Approximately 50% in variability of


incidence of CHD mortality is associated with
variability in smoking.
Coefficient of Alienation
72

 It is defined as 1 - r 2 or
SSresidual
1 r 2

SSY
 Example
1 - .508 = .492
SS residual 440.757
1 r 
2
  .492
SSY 895.247
2
r, SS and sY-Y’
73

 r2 * SStotal = SSregression
 (1 - r2) * SStotal = SSresidual

 We can also use r2 to calculate the


standard error of estimate as:

 N 1   20 
sY Yˆ  s y (1  r ) 
2
  6.690* (.492)    4.816
 N 2  19 
Testing Overall Model
74

 We can test for the overall prediction of


the model by forming the ratio:
2
sregression
2
 F statistic
sresidual
 If the calculated F value is larger than a
tabled value (F-Table) we have a
significant prediction
Testing Overall Model
75

 Example 2
sregression 454.307
2
  19.594
sresidual 23.198

 F-Table – F critical is found using 2 things


dfregression (numerator) and dfresidual.(demoninator)
 F-Table our Fcrit (1,19) = 4.38
 19.594 > 4.38, significant overall
 Should all sound familiar…
SPSS output
76

Model Summary

Adjus ted Std. Error of


Model R R Square R Square the Es timate
1 .713 a .508 .482 4.81640
a. Predictors : (Constant), CIGARETT

ANOVAb

Sum of
Model Squares df Mean Square F Sig.
1 Regress ion 454.482 1 454.482 19.592 .000 a
Res idual 440.757 19 23.198
Total 895.238 20
a. Predictors : (Constant), CIGARETT
b. Dependent Variable: CHD
Testing Slope and Intercept
77

 The regression coefficients can be


tested for significance
 Each coefficient divided by it’s

standard error equals a t value that


can also be looked up in a t-table
 Each coefficient is tested against 0
Testing the Slope
78

 With only 1 predictor, the standard


error for the slope is:
sY Yˆ
seb 
sX N  1
 For our Example:
4.816 4.816
seb    .461
2.334 21  1 10.438
Testing Slope and Intercept
79

 These are given in computer printout as


a t test.
Testing
80

 The t values in the second from right


column are tests on slope and intercept.
 The associated p values are next to
them.
 The slope is significantly different from
zero, but not the intercept.
 Why do we care?
Testing
81

 What does it mean if slope is not


significant?
 How does that relate to test on r?
 What if the intercept is not significant?
 Does significant slope mean we predict

quite well?

You might also like