Chapter 8

Nonlinear
Regression
Functions

Copyright 2015 Pearson, Inc. All rights reserved.

Outline
1. Nonlinear regression functions general
comments
2. Nonlinear functions of one variable
3. Nonlinear functions of two variables:
interactions
4. Application to the California Test Score
data set

Copyright 2015 Pearson, Inc. All rights reserved.

8-2

Nonlinear regression functions

The regression functions so far have been
linear in the Xs
But the linear approximation is not always
a good one
The multiple regression model can handle
regression functions that are nonlinear in
one or more X.

Copyright 2015 Pearson, Inc. All rights reserved.

8-3

The TestScore STR relation looks linear

(maybe)

Copyright 2015 Pearson, Inc. All rights reserved.

8-4

But the TestScore Income relation looks

nonlinear...

Copyright 2015 Pearson, Inc. All rights reserved.

8-5

Nonlinear Regression Population

Regression Functions General Ideas
(SW Section 8.1)
If a relation between Y and X is nonlinear:
The effect on Y of a change in X depends on the value of X
that is, the marginal effect of X is not constant
A linear regression is mis-specified: the functional form is
wrong
The estimator of the effect on Y of X is biased: in general it
isnt even right on average.
The solution is to estimate a regression function that is
nonlinear in X

Copyright 2015 Pearson, Inc. All rights reserved.

8-6

The general nonlinear population

regression function
Yi = f(X1i, X2i,, Xki) + ui, i = 1,, n
Assumptions
1. E(ui| X1i, X2i,, Xki) = 0 (same); implies that f is the
conditional expectation of Y given the Xs.
2. (X1i,, Xki, Yi) are i.i.d. (same).
3. Big outliers are rare (same idea; the precise mathematical
condition depends on the specific f).
4. No perfect multicollinearity (same idea; the precise
statement depends on the specific f).
The change in Y associated with a change in X1, holding X2,,
Xk constant is:
Y = f(X1 + X1, X2,, Xk) f(X1, X2,, Xk)
Copyright 2015 Pearson, Inc. All rights reserved.

8-7

Nonlinear Functions of a Single

Independent Variable (SW Section 8.2)
Well look at two complementary approaches:
1. Polynomials in X
The population regression function is approximated by a
quadratic, cubic, or higher-degree polynomial

2. Logarithmic transformations
Y and/or X is transformed by taking its logarithm
this gives a percentages interpretation that makes sense
in many applications

Copyright 2015 Pearson, Inc. All rights reserved.

8-8

1. Polynomials in X
Approximate the population regression function by a
polynomial:
Yi = 0 + 1Xi + 2 X i2++ r X r + ui
This is just the linear multiple regression model
except that the regressors are powers of X!
Estimation, hypothesis testing, etc. proceeds as in
the multiple regression model using OLS
The coefficients are difficult to interpret, but the
regression function itself is interpretable
Copyright 2015 Pearson, Inc. All rights reserved.

8-9

Example: the TestScore Income

relation
Incomei = average district income in the ith district
(thousands of dollars per capita)
Quadratic specification:
TestScorei = 0 + 1Incomei + 2(Incomei)2 + ui
Cubic specification:
TestScorei = 0 + 1Incomei + 2(Incomei)2 + 3(Incomei)3 + ui

Copyright 2015 Pearson, Inc. All rights reserved.

8-10

Estimation of the quadratic specification

in STATA
generate avginc2 = avginc*avginc;

Create a new regressor

reg testscr avginc avginc2, r;

Regression with robust standard errors

Number of obs
F( 2,
417)
Prob > F
R-squared
Root MSE

=
=
=
=
=

420
428.52
0.0000
0.5562
12.724

-----------------------------------------------------------------------------|
Robust
testscr |
Coef.
Std. Err.
t
P>|t|
[95% Conf. Interval]
-------------+---------------------------------------------------------------avginc |
3.850995
.2680941
14.36
0.000
3.32401
4.377979
avginc2 | -.0423085
.0047803
-8.85
0.000
-.051705
-.0329119
_cons |
607.3017
2.901754
209.29
0.000
601.5978
613.0056
------------------------------------------------------------------------------

Test the null hypothesis of linearity against the alternative that the regression function is a
quadratic.
Copyright 2015 Pearson, Inc. All rights reserved.

8-11

Interpreting the estimated regression

function:
(a) Plot the predicted values

TestScore

= 607.3 + 3.85Incomei 0.0423(Incomei)2

(2.9)

Copyright 2015 Pearson, Inc. All rights reserved.

(0.27)

(0.0048)

8-12

Interpreting the estimated regression

function, ctd:
(b) Compute effects for different values of X

TestScore
= 607.3 + 3.85Incomei 0.0423(Incomei)2
(2.9) (0.27)

(0.0048)

Predicted change in TestScore for a change in income from

$5,000 per capita to $6,000 per capita:

TestScore = 607.3 + 3.856 0.042362

(607.3 + 3.855 0.042352)
= 3.4

Copyright 2015 Pearson, Inc. All rights reserved.

8-13


TestScore
= 607.3 + 3.85Incomei 0.0423(Incomei)2
Predicted effects for different values of X:
Change in Income ($1000 per capita)

TestScore

from 5 to 6

3.4

from 25 to 26

1.7

from 45 to 46

0.0

The effect of a change in income is greater at low than high income

levels (perhaps, a declining marginal benefit of an increase in school
budgets?)
Caution! What is the effect of a change from 65 to 66?
Dont extrapolate outside the range of the data!

Copyright 2015 Pearson, Inc. All rights reserved.

8-14

Group Activity:
Is quadratic better than linear ?
Based on what we have learned together in
this class, how would we test if

TestScore

= 607.3 + 3.85Incomei 0.0423(Incomei)2

(2.9)

(0.27)

(0.0048)

should be chosen and not the linear form ?

Copyright 2015 Pearson, Inc. All rights reserved.

8-15

Estimation of a cubic specification in

STATA
gen avginc3 = avginc*avginc2;

Create the cubic regressor

reg testscr avginc avginc2 avginc3, r;

Regression with robust standard errors

Number of obs
F( 3,
416)
Prob > F
R-squared
Root MSE

=
=
=
=
=

420
270.18
0.0000
0.5584
12.707

-----------------------------------------------------------------------------|
Robust
testscr |
Coef.
Std. Err.
t
P>|t|
[95% Conf. Interval]
-------------+---------------------------------------------------------------avginc |
5.018677
.7073505
7.10
0.000
3.628251
6.409104
avginc2 | -.0958052
.0289537
-3.31
0.001
-.1527191
-.0388913
avginc3 |
.0006855
.0003471
1.98
0.049
3.27e-06
.0013677
_cons |
600.079
5.102062
117.61
0.000
590.0499
610.108
------------------------------------------------------------------------------

Copyright 2015 Pearson, Inc. All rights reserved.

8-16

Testing the null hypothesis of linearity, against the alternative

that the population regression is quadratic and/or cubic, that is,
it is a polynomial of degree up to 3:
H0: population coefficients on Income2 and Income3 = 0
H1: at least one of these coefficients is nonzero.
testavginc2avginc3;Executethetestcommandafterrunningtheregression

(1)avginc2=0.0
(2)avginc3=0.0

F(2,416)=37.69
Prob>F=0.0000

The hypothesis that the population regression is linear is rejected at

the 1% significance level against the alternative that it is a polynomial
of degree up to 3.

Copyright 2015 Pearson, Inc. All rights reserved.

8-17

Summary: polynomial regression

functions
Estimation: by OLS after defining new regressors
Coefficients have complicated interpretations
To interpret the estimated regression function:
plot predicted values as a function of x
compute predicted Y/X at different values of x

Hypotheses concerning degree r can be tested by t- and Ftests on the appropriate (blocks of) variable(s).
Choice of degree r
plot the data; t- and F-tests, check sensitivity of estimated
effects; judgment.
Or use model selection criteria (later)
Copyright 2015 Pearson, Inc. All rights reserved.

8-18

Reminder Pre-Calc: What Is a Log?

ln(x) = b means that eb = x

For example, since e2 = (2.71828) 2 = 7.389, we can state that:

ln(7.389) = 2
Thus, the natural log of 7.389 is 2! Again, why? Two is the power of e that
produces 7.389

Lets look at some other natural log calculations:

ln(100) = 4.605
ln(1000) = 6.908
ln(10000) = 9.210
ln(1000000) = 13.816
ln(100000) = 11.513

Note that as a number goes from 100 to 1,000,000, its natural log goes from
4.605 to only 13.816! As a result, logs can be used in econometrics if a
researcher wants to reduce the absolute size of the numbers associated with
the same actual meaning
One useful property of natural logs in econometrics is that they make it easier
to figure out impacts in percentage terms (well see this when we get to the
7-19
double-log specification)
Copyright 2015 Pearson, Inc. All rights reserved.

2. Logarithmic functions of Y and/or X

ln(X) = the natural logarithm of X
Logarithmic transforms permit modeling relations in
percentage terms (like elasticities), rather than linearly.
Reminder:

ln(x+x) ln(x) =
(calculus:

Numerically:

x
ln 1
x

x
x

d ln(x) 1 )

dx
x

ln(1.01) = .00995 .01;

ln(1.10) = .0953 .10 (sort of)

Copyright 2015 Pearson, Inc. All rights reserved.

8-20

The three log regression specifications:

Case

Population regression function

I.

linear-log

Yi = 0 + 1ln(Xi) + ui

II.

log-linear

ln(Yi) = 0 + 1Xi + ui

III. log-log

ln(Yi) = 0 + 1ln(Xi) + ui

The interpretation of the slope coefficient differs in each case.

The interpretation is found by applying the general before and
after rule: figure out the change in Y for a given change in X.
Each case has a natural interpretation (for small changes in X)

Copyright 2015 Pearson, Inc. All rights reserved.

8-21

I. Linear-log population regression

function
Compute Y before and after changing X:
Y = 0 + 1ln(X) (before)
Now change X: Y + Y = 0 + 1ln(X + X)(after)
Subtract (after) (before): Y = 1[ln(X + X) ln(X)]
now

ln(X + X) ln(X)

so

or

X
,X

X
X
Y
(small X)
X / X

Copyright 2015 Pearson, Inc. All rights reserved.

8-22

Linear-log case, continued

Yi = 0 + 1ln(Xi) + ui
for small X,
1

Y
X / X

X
Now 100 X = percentage change in X
a 1% increase in X (multiplying X by 1.01) is
associated with a .011 change in Y.
(1% increase in X --> .01 increase in ln(X)
--> .011 increase in Y)
Copyright 2015 Pearson, Inc. All rights reserved.

8-23

Example: TestScore vs. ln(Income)

First defining the new regressor, ln(Income)
The model is now linear in ln(Income), so the linear-log
model can be estimated by OLS:

TestScore

= 557.8 + 36.42ln(Incomei)
(3.8)

(1.40)

so a 1% increase in Income is associated with an increase in

TestScore of 0.36 points on the test.
Standard errors, confidence intervals, R2 all the usual tools
of regression apply here.
How does this compare to the cubic model?

Copyright 2015 Pearson, Inc. All rights reserved.

8-24

The linear-log and cubic regression

functions

Copyright 2015 Pearson, Inc. All rights reserved.

8-25

II. Log-linear population regression

function
ln(Y) = 0 + 1X

(b)

a change in X by one unit (X = 1) is associated with a

1001% change in Y.
1 unit increase in X 1 increase in ln(Y)
1001% increase in Y
Note: What are the units of ui and the SER?
o fractional (proportional) deviations
o for example, SER = .2 means

Copyright 2015 Pearson, Inc. All rights reserved.

8-26

Log-log case
ln(Yi) = 0 + 1ln(Xi) + ui

a 1% change in X is associated with a 1% change in Y.

In the log-log specification, 1 has the interpretation of an

elasticity.

Copyright 2015 Pearson, Inc. All rights reserved.

8-27

Double-Log Form
Here, the natural log of Y is the dependent variable and the
natural log of X is the independent variable:
(7.5)
In a double-log equation, an individual regression coefficient
can be interpreted as an elasticity because:
(7.6)
Note that the elasticities of the model are constant and
the slopes are not
This is in contrast to the linear model, in which the
slopes are constant but the elasticities are not

Copyright 2015 Pearson, Inc. All rights reserved.

7-28

Example: ln(TestScore) vs. ln(Income)

First defining a new dependent variable, ln(TestScore), and
the new regressor, ln(Income)
The model is now a linear regression of ln(TestScore)
against ln(Income), which can be estimated by OLS:

TestScore)
ln(

= 6.336 + 0.0554ln(Incomei)
(0.006) (0.0021)

An 1% increase in Income is associated with an increase of .

0554% in TestScore (Income up by a factor of 1.01, TestScore
up by a factor of 1.000554)

Copyright 2015 Pearson, Inc. All rights reserved.

8-29

Example: ln( TestScore) vs. ln( Income),

ctd.
TestScore)
ln(
= 6.336 + 0.0554ln(Incomei)
(0.006) (0.0021)
For example, suppose income increases from
$10,000 to $11,000, or by 10%.
Then TestScore increases by approximately .
055410% = .554%. If TestScore = 650, this
corresponds to an increase of .00554650 = 3.6
points.
How does this compare to the log-linear model?
Copyright 2015 Pearson, Inc. All rights reserved.

8-30

The log-linear and log-log specifications:

Note vertical axis

Neither seems to fit as well as the cubic or linear-log, at
least based on visual inspection (formal comparison is
difficult because the dependent variables differ)
Copyright 2015 Pearson, Inc. All rights reserved.

8-31

More in class practice and examples: 1

Lets interpret the coefficients in words.

Copyright 2015 Pearson, Inc. All rights reserved.

8-32

More in class practice and examples: 2

Lets interpret
the
coefficients in
words for
estimation (1)
and (2)

Copyright 2015 Pearson, Inc. All rights reserved.

8-33

Interactions Between Independent

Variables (SW Section 8.3)
Perhaps a class size reduction is more effective in some
circumstances than in others
Perhaps smaller classes help more if there are many English
learners, who need individual attention
That is, TestScore

STR

might depend on PctEL

More generally, Y might depend on X2

X 1

How to model such interactions between X1 and X2?

We first consider binary Xs, then continuous Xs

Copyright 2015 Pearson, Inc. All rights reserved.

8-34

(a) Interactions between two binary

variables
Yi = 0 + 1D1i + 2D2i + ui
D1i, D2i are binary
1 is the effect of changing D1=0 to D1=1. In this
specification, this effect doesnt depend on the
value of D2.
To allow the effect of changing D1 to depend on D2,
include the interaction term D1iD2i as a
regressor:
Yi = 0 + 1D1i + 2D2i + 3(D1iD2i) + ui
Copyright 2015 Pearson, Inc. All rights reserved.

8-35

Example: TestScore, STR, English

learners
Let

HiSTR =

1 if STR 20
and HiEL =
0 if STR 20

1 if PctEL l0
0 if PctEL 10

TestScore
= 664.1 18.2HiEL 1.9HiSTR 3.5(HiSTRHiEL)
(1.4)

(2.3)

(1.9)

(3.1)

Effect of HiSTR when HiEL = 0 is 1.9

Effect of HiSTR when HiEL = 1 is 1.9 3.5 = 5.4
Class size reduction is estimated to have a bigger effect
when the percent of English learners is large
This interaction isnt statistically significant: t = 3.5/3.1

Copyright 2015 Pearson, Inc. All rights reserved.

8-36

(b) Interactions between continuous and

binary variables
Yi = 0 + 1Di + 2Xi + ui
Di is binary, X is continuous
As specified above, the effect on Y of X (holding
constant D) = 2, which does not depend on D
To allow the effect of X to depend on D, include
the interaction term DiXi as a regressor:
Yi = 0 + 1Di + 2Xi + 3(DiXi) + ui
Copyright 2015 Pearson, Inc. All rights reserved.

8-37

Binary-continuous interactions: the two

regression lines
Yi = 0 + 1Di + 2Xi + 3(DiXi) + ui
Observations with Di= 0 (the D = 0 group):
Yi = 0 + 2Xi + ui

The D=0 regression line

Observations with Di= 1 (the D = 1 group):

Yi = 0 + 1 + 2Xi + 3Xi + ui
= (0+1) + (2+3)Xi + ui The D=1 regression line

Copyright 2015 Pearson, Inc. All rights reserved.

8-38

Example: TestScore, STR, HiEL (=1 if

PctEL 10)
= 682.2 0.97STR + 5.6HiEL 1.28(STRHiEL)
TestScore
(11.9) (0.59)

(19.5)

(0.97)

When HiEL = 0:
= 682.2 0.97STR
TestScore
When HiEL = 1,
= 682.2 0.97STR + 5.6 1.28STR

TestScore
= 687.8 2.25STR
regression lines: one for each HiSTR group.
Two
TestScore
Class size reduction is estimated to have a larger effect when
the percent of English learners is large.

Copyright 2015 Pearson, Inc. All rights reserved.

8-39

(c) Interactions between two continuous

variables
Yi = 0 + 1X1i + 2X2i + ui
X1, X2 are continuous
As specified, the effect of X1 doesnt depend on X2
As specified, the effect of X2 doesnt depend on X1
To allow the effect of X1 to depend on X2, include
the interaction term X1iX2i as a regressor:
Yi = 0 + 1X1i + 2X2i + 3(X1iX2i) + ui

Copyright 2015 Pearson, Inc. All rights reserved.

8-40

Example: TestScore, STR, PctEL

TestScore
= 686.3 1.12STR 0.67PctEL + .0012(STRPctEL),
(11.8)

(0.59)

(0.37)

(0.019)

The estimated effect of class size reduction is nonlinear because

the size of the effect itself depends on PctEL:

TestScore = 1.12 + .0012PctEL

STR

PctEL

TestScore
STR

0
20%

1.12
1.12+.001220 = 1.10

Copyright 2015 Pearson, Inc. All rights reserved.

8-41

More in class practice and examples: 3

Lets interpret
the
coefficients in
words for
estimation (1)
and (2)

Copyright 2015 Pearson, Inc. All rights reserved.

8-42