How to do a linear regression?

Tom Broekel
Prerequisites I

1 © TBroekel
 How to do linear regressions: Prerequisites

Sufficient empirical observation

Min. 10 observations per explanatory variable

Crucial for tests of significance, as otherwise regression equation too

flexible (too many degrees of freedom

Too few observations

Use of dimension reduction: e.g. factor analysi

Stepwise regression

2 © TBroekel
:

 How to do linear regressions: Prerequisites

Regression requires continuous dependent variabl

Continuous variable

For each values A and B all values within interval [A,B] possible

Values can be arbitrary divided and remain meaningfu

Not continuous variable

Ordinal - Likert scale, values 1,2,3,4,5; values can be sorted but differences have no
meanin

Binary / dichotomous / nominal variables - values 0 or 1

3 © TBroekel
g

 How to do linear regressions: Prerequisites

Regressions for non-continuous variable

Logistic regressions, … (later in class

Howeve

Sometimes, linear regression models for ordinal dependent variables o.k.

- if diagnostics are not violated (later in class)

4 © TBroekel
r

 How to do linear regressions: Prerequisites

Linear regression only useful when true causal relationship linea

Non-linear relations

Useless model

Failing model diagnostics

5 © TBroekel
s

 How to do linear regressions: Prerequisites

500

400

300
Y

200

100

0
0 25 50 75 100
X

6 © TBroekel
 How to do linear regressions: Prerequisites

500

400

300
Y

200

100

0
0 25 50 75 100
X

Linear regression —
7 © TBroekel
 How to do linear regressions: Prerequisites

5
Y_new

2
0 25 50 75 100
X

8 © TBroekel
 How to do linear regressions: Prerequisites

5
Y_new

2
0 25 50 75 100
X

Linear regression —

9 © TBroekel
 How to do linear regressions: Prerequisites

0.00

−0.05

−0.10
Y2

−0.15

−0.20

−0.25

0 25 50 75 100
X

10 © TBroekel
 How to do linear regressions: Prerequisites

0.1
Linear regressio
b1 = 0.000
p-value = 0.99
0.0
R2 = 0.01
Y2

−0.1

−0.2

0 25 50 75 100
X

Linear regression —

11 © TBroekel
3

 How to do linear regressions: Prerequisites

Solutions for non-linearitie

Use of non-linear regression (not covered

Transformation of dependent or explanatory variables (case-specific

Log-transformatio

Square root-transformatio

Squaring- or u-shape transformatio

S-shap

...

12 © TBroekel
e

 How to do linear regressions: Prerequisites

Transformation:Y=a+b*log(X)

6
6

5
Origina
b1 = 0.03, p-value = 0.00
5

R2 = 0.7
Y_new

4
4
4 Transforme
3 b1 = 1.04, p-value = 0.00
3
3 R2 = 0.97
2

2
00 1 25 2 50 3 75 4 100
X
Log_X

Linear regression —

13 © TBroekel
8

 How to do linear regressions: Prerequisites

Transformation: log(Y)=a+b*X
5
150
150

4
Origina
100
100
b1 = 1,12, p-value = 0.00
3

R2 = 0.7
log(Y_new)
Y_new

250
Transforme
50 b1 = 0.05, p-value = 0.00
1 R2 = 0.99
0

0 0

0 0 2525 50
50 75
75 100
XX

Linear regression —

14 © TBroekel
3

 How to do linear regressions: Prerequisites

Transformation:Y=a+b1*X1+b2*X12
0.1
0.1

Origina
0.0
0.0 b1 = 0,25, p-value = 0.92
R2 = 0.0
Transforme
Y2
Y2

−0.1
−0.1
b1 = -0.001, p-value = 0.00
b2 = 0.00001, p-value = 0.00
−0.2
−0.2 R2 = 0.83

00 25
25 50
50 75
75 100
100
XX

Linear regression —

15 © TBroekel
1

 How to do linear regressions: Prerequisites

Transformation

Explanatory or dependent variable or both can be transforme

StatTrek 2011: http://stattrek.com/ap-statistics-1/transformation.asp

Common transformations

16 © TBroekel
s

 How to do linear regressions: Prerequisites

Watch out

Interpretation of regression equation changes

Slope parameter (b) (and its significance) represents relation between

transformed explanatory (X’) and dependent (Y) variable

It DOES NOT represent relation of original explanatory (X) and

dependent variable (Y)

17 © TBroekel
!

 How to do linear regressions: Prerequisites

Exampl

Original: Y=a+3*X, i.e., Linear regional between X and Y with an increase

of X resulting in 3 times increase in Y

Transformed: Y=a+3*log(X), i.e., Log-linear regional between X and Y

with an increase in log-transformed X resulting in 3 times increase in Y

18 © TBroekel
e

 How to do a linear regression? Tom Broekel

Prerequisites I

19 © TBroekel