MULTIPLE LINEAR REGRESSION
JUST A LITTLE POKE
GERNIMO MALDONADO-MARTNEZ, RPT; MPH, PHD(C)
DIANA M. FERNNDEZ-SANTOS, MS; EDD
Sir Francis Galton
Widely promoted regression techniques.
Cousin of Charles Darwin.
Making Sense of Regression
My emphasis here is on
Understanding the key elements of regression
Requirements
Application
Limitations
Regression Is A Powerful Analytical
Technique
Enables researchers to do two things:
1. Determine the strength of the relationship
The r-squared value
Regression Is A Powerful Analytical
Technique
2. Determine the impact of the independent
variable(s) on the dependent variable
The regression coefficient is the predicted
change in the dependent variable for every one
unit of change in the independent variable
Collectively, the regression coefficients enable
the researchers to make estimates of how the
dependent variable will change using different
scenarios for the independent variables
Assumptions
Variables are normally distributed.
Continuous nature
Assumption of a linear relationship
between the independent and
dependent variables.
Assumption of homoscedasticity.
Multiple Regression Equation
Y = a + bX+bX2+bXk + e
Where:
Y = predicted value of the dependent variable
a = the constant or Y intercept (where the
imaginary line crosses the Y access)
b = the regression coefficient
X = the independent variable
e = error
Theoretical Linear Model
Linear Regression
Model Type X=size of house, Y=cost of house
Deterministic Model: an equation or set of equations
that allow us to fully determine the value of the dependent
variable from the values of the independent variables.
Probabilistic Model: a method used to capture the
randomness that is part of a real-life process.
R-square And Its Companions
r = correlation coefficient (overall fit or measure of
association, which is also called r, Pearsons r, Pearson
Product Moment Correlation coefficient, or zero-order
coefficient).
r-square = proportion of the explained variance of the
dependent variable (also called the coefficient of
determination)
1 minus r-square = proportion of unexplained variance
in the dependent variable
Dirty Interpretation
Example: Researchers look at GRE scores and academic
performance in graduate school as measured by grade point
average
The hypothesis is that people who have high GRE scores will
also have high GPAs
From an admissions committee perspective: the belief
that GRE scores are a good predictor of future academic
success and are, therefore, a good criteria for admission
decisions
The researchers report an r-squared of .2
GREs explain 20 percent of the change in GPAs
This means that 80 percent of the changes in GPA are
explained by other factors.
A quick example
Dont get lost
.
.
.
.
. . .. . . .
. . ..
. . .
.
. .
Y Axis: Plane Maintenance Costs
$1,000
$500
Predicted values
if perfect relationship
X Axis: Age of Planes
5 years
10 years
20 years
How It Is Applied
Analysts collect data over the past two years and
crunch it. The computer gives these results:
Y = 100 and .020X
The constant is 100:
If they do not fly at all, the computer estimates
there is still a cost of $100
The .020 is the regression coefficient:
This gets interpreted as: for every mile flown, there
is $.02 change in maintenance costs.
How It Is Applied
Y = 100 and .020X
Interpreting the regression coefficient:
For every mile flown, the maintenance costs
goes up by 2 cents.
For every 100 miles flown, costs are $2
For every 1,000 miles, the costs are $20
For every 100,000 miles, the costs are
$20,000
Making Maintenance Cost Estimates
They can then solve the equation:
Assuming 100,000 miles will be flown, how much
will they need to budget for maintenance?
100,000 multiplied by .020 = $20,000
Y= 100 + $20,000 + error
The estimate maintenance will cost:
$20,100 + error
Practicality
Simple Regression: Another Example
Hypothesis: If schools have a higher
percentage of poor children, then they
will have lower test scores.
A regression analysis shows:
A regression coefficient of -.04
An r-squared value of .25
Even More
Interpretation?
Regression coefficient: For every increase in the
percent of children in poverty within a school, the
average test score goes down by .04
R-squared: 25% of the test scores are explained
by the percent of children in poverty in the school
Researchers will ask: what other factors might
explain differences in test scores in the schools?
Multiple Regression Equation
Y = a + bX1 + bX2 + bX3 + bX4 + e.
Y = dependent variable
X1 = independent variable 1,
for X2, X3, X4
X2 = independent variable 2
controlling for X1, X3, X4
X3 = independent variable 3
controlling for X1, X2, X4
X4= independent variable 4
controlling for X1, X2, X3
controlling
Multiple Regression Equation
It has the same basic structure of simple
regression
Y is still the dependent variable
There is still a constant (a) and some
amount of error (e) that the computer
calculates
But there are more Xs to represent the
multiple independent variables
Multiple Regression:
An Example
Hypothesis: Income is a function of education
and seniority?
We suggest that income (the dependent
variable) will increase as both education and
seniority increases (two independent
variables)
Y (Income) = a + education + seniority +
error
Multiple Regression: Interpretation
Results:
Y= 6000 + 400X1 (education) + 200X2 (seniority)
R square = .67
First look at the R-Square: This shows a strong
relationshipso analysis can continue
Partial regression coefficients:
For every year of education, holding seniority
constant, income increases by $400.
For every year of seniority, holding education
constant, income increases by $200.
Multiple Regression: Application
Estimate the income of someone who has
10 years of education and
5 years of seniority
We solve the regression equation:
Multiply the 10 years of education by the regression
coefficient of 400: equals 4,000
Multiply 5 years of senior by the regression coefficient of
200: equals 1,000
Put it together with the constant and you have
Y=6000 + 400(10) + 200(5) + error
Y= $ 11,000 + error
Demystifying the monster
Statistics
Statistics
Data
Data
x
17
12
Information
Multivariate regression pitfalls
Multi-collinearity
Residual
confounding
Overfitting
Multicollinearity
Multicollinearity
arises when two variables
that measure the same thing or similar things
(e.g., weight and BMI) are both included in a
multiple regression model; they will, in
effect, cancel each other out and generally
destroy your model.
VIF>1
is bad
Tolerance: low is bad
Residuals Diagnostics
You cannot completely wipe out confounding simply
by adjusting for variables in multiple regression
unless variables are measured with zero error (which
is usually impossible).
Example: meat eating and mortality
A clean example in PRISM
A clean example in PRISM
A clean example in PRISM
A clean example in PRISM
A clean example in PRISM
A clean example in PRISM
A real linear regression output
A real linear regression output
A real linear regression output
A real linear regression output
References
Kleinbaum, D. G. Applied
Regression Analysis and
Multivariable Methods.
Third Edition (2011)
