Chapter 06.

04
Nonlinear Models for Regression

After reading this chapter, you should be able to

1. derive constants of nonlinear regression models,
2. use in examples, the derived formula for the constants of the nonlinear regression
model, and
3. linearize (transform) data to find constants of some nonlinear regression models.
From fundamental theories, we may know the relationship between two variables.
An example in chemical engineering is the Clausius-Clapeyron equation that relates vapor
pressure P of a vapor to its absolute temperature, T .
B
(1)
log(P ) = A +
T
where A and B are the unknown parameters to be determined. The above equation is not
linear in the unknown parameters. Any model that is not linear in the unknown parameters is
described as a nonlinear regression model.
Nonlinear models using least squares
The development of the least squares estimation for nonlinear models does not
generally yield equations that are linear and hence easy to solve. An example of a nonlinear
regression model is the exponential model.
Exponential model
Given ( x1 ,y1 ) , ( x 2 ,y 2 ) , . . . ( x n ,y n ) , best fit y = ae bx to the data. The variables a and
b are the constants of the exponential model. The residual at each data point xi is

Ei = yi ae bxi
The sum of the square of the residuals is

(2)

S r = Ei2
i =1
n

= y i ae bxi
i =1

06.04.1

(3)

06.04.2

Chapter 06.04

To find the constants a and b of the exponential model, we minimize S r by differentiating

with respect to a and b and equating the resulting equations to zero.

or

n
S r
= 2 y i ae bxi e bxi = 0
a
i =1
n
S r
= 2 y i ae bxi axi e bxi = 0
b
i =1

)(

)(

i =1

i =1
n

(4a,b)

y i e bxi + a e 2bxi = 0
n

y xe
i =1

bxi

a xi e 2bxi = 0

(5a,b)

i =1

Equations (5a) and (5b) are nonlinear in a and b and thus not in a closed form to be
solved as was the case for linear regression. In general, iterative methods (such as
Gauss-Newton iteration method, method of steepest descent, Marquardt's method, direct
search, etc) must be used to find values of a and b .
However, in this case, from Equation (5a), a can be written explicitly in terms of b as
n

a=

ye
i

i =1
n

bxi

(6)

2 bxi

i =1

Substituting Equation (6) in (5b) gives

n

y xe
i =1

bxi

ye
i =1
n

bxi

2 bxi

x e
i =1

2 bxi

=0

(7)

i =1

This equation is still a nonlinear equation in b and can be solved best by numerical methods
such as the bisection method or the secant method.
Example 1
Many patients get concerned when a test involves injection of a radioactive material. For
example for scanning a gallbladder, a few drops of Technetium-99m isotope is used. Half of
the technetium-99m would be gone in about 6 hours. It, however, takes about 24 hours for
the radiation levels to reach what we are exposed to in day-to-day activities. Below is given
the relative intensity of radiation as a function of time.
Table 1 Relative intensity of radiation as a function of time
t (hrs) 0
1
3
5
7
9

1.000 0.891 0.708 0.562 0.447 0.355

Nonlinear Regression

06.04.3

If the level of the relative intensity of radiation is related to time via an exponential formula
= Ae t , find
a) the value of the regression constants A and ,
b) the half-life of Technium-99m, and
c) the radiation intensity after 24 hours.
Solution
a) The value of is given by solving the nonlinear Equation (7),
n

f ( ) = i t i e ti
i =1

e
i =1
n

ti

2 t i

t e
i =1

2 ti

(8)

=0

i =1

and then the value of A from Equation (6),

n

A=

e
i

i =1
n

ti

(9)

2 t i

i =1

Equation (8) can be solved for using bisection method. To estimate the initial
guesses, we assume = 0.120 and = 0.110 . We need to check whether these values
first bracket the root of f ( ) = 0 . At = 0.120 , the table below shows the evaluation of
f ( 0.120) .
Table 2 Summation value for calculation of constants of model
ti i
i
i t i e ti i e ti
t i e 2ti
e 2ti
1
2
3
4
5
6

0
1
3
5
7
9

1
0.891
0.708
0.562
0.447
0.355

i =1

From Table 2
n=6
6

t e
i =1
6

i i

e
i =1
6

i =1

0.120 ti

0.120 ti

= 6.2501

= 2.9062

e 2 (0.120 )ti = 2.8763

0.00000
0.79205
1.4819
1.5422
1.3508
1.0850

1.00000
0.79205
0.49395
0.30843
0.19297
0.12056

1.00000
0.78663
0.48675
0.30119
0.18637
0.11533

0.00000
0.78663
1.4603
1.5060
1.3046
1.0379

6.2501

2.9062

2.8763

6.0954

06.04.4

Chapter 06.04

t e (
i =1

2 0.120 )ti

= 6.0954

f ( 0.120) = (6.2501)
= 0.091357

Similarly
Since

2.9062
(6.0954)
2.8763

f ( 0.110) = 0.10099

f ( 0.120) f ( 0.110) < 0 ,

the value of falls in the bracket of [ 0.120,0.110] . The next guess of the root then is
0.120 + ( 0.110)
=
2
= 0.115
Continuing with the bisection method, the root of f ( ) = 0 is found as = 0.11508 . This
value of the root was obtained after 20 iterations with an absolute relative approximate error
of less than 0.000008%.
From Equation (9), A can be calculated as
6

A=

e
i =1
6

ti

2 ti

i =1

1 e 0.11508( 0 ) + 0.891 e 0.11508(1) + 0.708 e 0.11508( 3) +

=

0.562 e 0.11508( 5) + 0.447 e 0.11508( 7 ) + 0.355 e 0.11508( 9 )

e 2 ( 0.11508)( 0 ) + e 2 ( 0.11508)(1) + e 2 ( 0.11508)( 3) +
e 2 ( 0.11508)( 5) + e 2 ( 0.11508)( 7 ) + e 2 ( 0.11508)( 9 )

2.9373
2.9378
= 0.99983
The regression formula is hence given by
= 0.99983 e 0.11508t
=

b) Half life of Technetium-99m is when =

0.99983 e 0.11508t =
e 0.11508t = 0.5
0.11508t = ln(0.5)
t = 6.0232 hours

1
(0.99983)e 0.11508(0)
2

t =0

Nonlinear Regression

06.04.5

c) The relative intensity of the radiation after 24 hrs is

= 0.99983 e 0.11508(24 )
= 6.3160 10 2
6.3160 10 2
This implies that only
100 = 6.3171% of the initial radioactive intensity is left
0.99983
after 24 hrs.

Figure 1 Relative intensity of radiation as a function of temperature using an

exponential regression model.
Growth model
Growth models common in scientific fields have been developed and used
successfully for specific situations. The growth models are used to describe how something
grows with changes in the regressor variable (often the time). Examples in this category
include growth of thin films or population with time. Growth models include
a
(10)
y=
1 + be cx
a
where a, b and c are the constants of the model. At x = 0 , y =
and as x ,
1+b
ya.
The residuals at each data point xi , are

06.04.6

Chapter 06.04

a
1 + be cxi
The sum of the square of the residuals is
Ei = yi

(11)

S r = Ei2
i =1

(12)
= yi
cxi
1 + be
i =1
To find the constants a , b and c we minimize S r by differentiating with respect to a ,
b and c , and equating the resulting equations to zero.
n

(
)

n
2e cxi ae cxi yi e cxi + b
S r

=
2
a i =1
e cxi + b

(
[by
(e

)] = 0 ,

+ e cxi ( y i a )
= 0,
3
cxi

+b

cxi
cxi
n
2abxi e by i + e ( y i a )
S r
=
= 0.
3

c
i =1
e cxi + b

n
2ae cxi
S r
=

b
i =1

(13a,b,c)

One can use the Newton-Raphson method to solve the above set of simultaneous nonlinear
equations for a , b and c .
Example 2
The height of a child is measured at different ages as follows.
Table 3 Height of the child at different ages.
t ( yrs) 0 5.0 8 12 16
18
H (in ) 20 36.2 52 60 69.2 70
Estimate the height of the child as an adult of 30 years of age using the growth model,
a
H=
1 + be ct
Solution
The saturation growth model of height, H vs. age, t is given as
a
H=
1 + be ct
where the constants a , b and c are the roots of the simultaneous nonlinear equation system

Nonlinear Regression

06.04.7

(
)

)]

2e cti ae cti H i e cti + b

=0

i =1
e cti + b

ct
ct
6
2ae i bH i + e i (H i a )

=0

i =1
e cti + b

cti
cti
6

2abt i e bH i + e (H i a ) = 0

i =1
e cti + b

[
(

(14a,b,c)

We need initial guesses of the roots to get the iterative process started to find the root of
those equations. Suppose we use three of the given data points such as (0, 20), (12, 60) and
(18, 70) to find the initial guesses of roots; we have
a
20 =
1 + be c ( 0 )
a
60 =
1 + be c (12 )
a
70 =
1 + be c (18)
One can solve three unknowns a , b and c for the initial guesses from the three equations as
a = 7.5534 101
b = 2.7767
c = 1.9772 10 1
Applying the Newton-Raphson method for simultaneous nonlinear equations with the above
initial guesses, one can get the roots
a = 7.4321101
b = 2.8233
c = 2.1715 10 1
The saturation growth model of the height of the child then is
7.4321 101
H=
1
1 + 2.8233e 2.171510 t
The height of the child as an adult of 30 years of age is
7.4321 101
H=
1
1 + 2.8233e 2.171510 (30 )
= 74 "
Polynomial Models
Given n data points ( x1 , y1 ), ( x 2 , y 2 ),......, ( x n , y n ) use least squares method to regress the
data to an m th order polynomial.
y = a 0 + a1 x + a 2 x 2 + + a m x m , m < n
The residual at each data point is given by
Ei = y i a 0 a1 xi . . . a m xim

(15)
(16)

06.04.8

Chapter 06.04

The sum of the square of the residuals is given by

n

S r = Ei2
i =1
n

= y i a 0 a1 xi . . . a m x
i =1

(17)

m 2
i

To find the constants of the polynomial regression model, we put the derivatives with respect
to ai to zero, that is,

Figure 2 Height of child as a function of age saturation growth model.

n
S r
= 2 y i a 0 a1 xi . . . a m xim (1) = 0
a 0 i =1

n
S r
= 2 y i a 0 a1 xi . . . a m xim ( xi ) = 0
a1 i =1

.
.

. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

n
S r
= 2 y i a 0 a1 xi . . . a m xim ( xim ) = 0
a m i =1
Setting those equations in matrix form gives

(18)

Nonlinear Regression

06.04.9

n
n

n
. . . xim a
xi

i =1
i =1 0
a
n
n
n
xi xi2 . . . xim+1 1
i =1 i =1
i =1
. .
. . . . . .
. . . . . a

m
n
n
n

2m
m
m +1
xi xi . . . xi

i =1

i =1 i =1
The above are solved for a 0 , a1 ,..., a m

yi
ni =1
=
xi yi
. i =1
. . .

n
xim yi
i =1

(19)

Example 3
To find contraction of a steel cylinder, one needs to regress the thermal expansion coefficient
data to temperature
Table 4 The thermal expansion coefficient at given different temperatures
Temperature, T
Coefficient of thermal

expansion, (in/in/ F)
( F)
80
6.47 10 6
40
6.24 10 6
-40
5.72 10 6
-120
5.09 10 6
-200
4.30 10 6
-280
3.33 10 6
-340
2.45 10 6
Fit the above data to = a0 + a1T + a 2T 2
Solution
Since = a0 + a1T + a 2T 2 is the quadratic relationship between the thermal expansion
coefficient and the temperature, the coefficients a 0 , a1 , a 2 are found as follows

n
Ti
i =n1
T 2
i

i =1

Ti
i =1
n 2
Ti
i =1
n 3
Ti
i =1

n 2
n
Ti
i
i =1 a i =1
0
n 3 n
Ti a1 = Ti i
i =1 i =1
a
n
n 4 2 T 2
Ti
i i
i =1
i =1

06.04.10

Chapter 06.04

Table 5 Summations for calculating constants of model

(in/in/ F) T 2
T ( F)
T3

80

6.4700 10 6

40

6.2400 10 6

-40

-120

5.0900 10 6

-200

4.3000 10 6

-280

3.3300 10 6

-340

2.4500 10 6

8.6000 10 2

3.3600 10 5

i =1

5.7200 10 6

1.6000 10 3

6.4000 10 4

1.4400 10 4

1.7280 10 6

4.0000 10 4

8.0000 10 6

7.8400 10 4

2.1952 10 7

1.1560 10 5

3.9304 10 7

2.5800 10 5

7.0472 10 7

2.0736 10 8

1.6000 10 9

6.1466 10 9

1.3363 1010

8.3300 10 4

2.1363 1010

2.6978 10 3

i =1

n=7
i =1

6.4000 10 4

1.6000 10 3

5.1200 10 5

Table 5 (cont)
T
T
4.0960 10 7 5.1760 10 4
2.5600 10 6 2.4960 10 4
2.2880 10 4
2.5600 10 6

i
1
2

6.4000 10 3

= 8.6000 10 2

6.1080 10 4

8.6000 10 4
9.3240 10 4

T 2
4.1408 10 2
9.9840 10 3
9.1520 10 3
7.3296 10 2
1.7200 10 1
2.6107 10 1
2.8322 10 1
8.5013 10 1

Nonlinear Regression

06.04.11

= 2.5580 10 5

= 7.0472 10 7

= 2.1363 1010

i =1
7

i =1
7

i =1
7

i =1
7

= 3.3600 10 5

T
i =1
7

T
i =1

= 2.6978 10 3

i = 8.5013 10 1

We have
7.0000

2
8.600 10
2.5800 10 5

8.6000 10 2
2.5800 10 5
7.0472 10 7

2.5800 10 5 a 0 3.3600 10 5

7.0472 10 7 a1 = 2.6978 10 3
2.1363 1010 a 2 8.5013 10 1

Solving the above system of simultaneous linear equations, we get

6
a 0 6.0217 10

9
a1 = 6.2782 10
a 2 1.2218 10 11
The polynomial regression model is
= a 0 + a1T + a 2T 2
= 6.0217 10 6 + 6.2782 10 9 T 1.2218 10 11 T 2
Transforming the data to use linear regression formulas

Examination of the nonlinear models above shows that in general iterative methods are
required to estimate the values of the model parameters. It is sometimes useful to use simple
linear regression formulas to estimate the parameters of a nonlinear model. This involves
first transforming the given data such as to regress it to a linear model. Following the
transformation of the data, the evaluation of model parameters lends itself to a direct solution
approach using the least squares method. Data for nonlinear models such as exponential,
power, and growth can be transformed.
Exponential Model
As given in Example 1, many physical and chemical processes are governed by the
exponential function.
(20)
= ae bx
Taking natural log of both sides of Equation (20) gives
(21)
ln = ln a + bx

06.04.12

Let

then

Chapter 06.04

z = ln
a 0 = ln a implying a = e ao

a1 = b
z = a 0 + a1 x

(22)

Figure 3 Second-order polynomial regression model for coefficient of thermal expansion

as a function of temperature.
The data z versus x is now a linear model. The constants a 0 and a1 can be found using the
equation for the linear model as
n

a1 =

n xi z i xi z i
i =1

i =1

i =1

n x xi
i =1
i =1
n

2
i

a 0 = z a1 x

(23a,b)

Nonlinear Regression

06.04.13

Now since a 0 and a1 are found, the original constants with the model are found as
b = a1
a = e a0

(24a,b)

Example 4
Repeat Example 1 using linearization of data.
Solution

= Ae t
ln ( ) = ln ( A) + t

Assuming
y = ln
a 0 = ln( A)
We get

a1 =

y = a 0 + a1t
This is a linear relationship between y and t .
n

n t i y i t i y i

a1 =

i =1

i =1

i =1

n t i2 t i
i =1
i =1
a0 = y a1t
n

(25a,b)

Table 6 Summations of data to calculate constants of model.

ti
i
yi = ln i t i yi
i
t i2
1
0
1
0.00000
0.0000
0.0000
2
1
0.891 -0.11541 -0.11541 1.0000
3
3
0.708 -0.34531 -1.0359 9.0000
4
5
0.562 -0.57625 -2.8813 25.000
5
7
0.447 -0.80520 -5.6364 49.000
6
9
0.355 -1.0356
-9.3207 81.000
6

i =1

n=6
6

t
i =1
6

y
i =1

= 25.000

= 2.8778

25.000

-2.8778

-18.990

165.00

06.04.14

Chapter 06.04

t y
i

2
i

= 165.00

i =1
6

i =1

= 18.990

From Equation (25a,b) we have

6( 18.990) (25)( 2.8778)
a1 =
2
6(165.00) (25)
= 0.11505
25
2.8778
( 0.11505)
a0 =
6
6
4
= 2.6150 10
Since
a0 = ln ( A)
A = e a0

= e 2.615010
= 0.99974
= a1 = 0.11505
The regression formula then is
= 0.99974 e 0.11505t
Compare the formula to the one obtained without data linearization,
= 0.99983 e 0.11508t
b) Half-life is when
1
=
2 t =0
1
0.99974 e 0.11505t = (0.99974 )e 0.11505( 0 )
2
0.11508 t
e
= 0.5
0.11505t = ln(0.5)
t = 6.0248 hours
c) The relative intensity of radiation, after 24 hours is
= 0.99974e 0.11505(24 )
= 0.063200

This implies that only

6.3200 10 2
100 = 6.3216% of the initial radioactivity is left after
0.99974

24 hours.
Logarithmic Functions
The form for the log regression models is
y = 0 + 1 ln ( x )

(26)

Nonlinear Regression

06.04.15

This is a linear function between y and ln (x ) and the usual least squares method applies in
which y is the response variable and ln (x ) is the regressor.

Figure 4 Exponential regression model with transformed data for relative intensity of
radiation as a function of temperature.
Example 5
Sodium borohydride is a potential fuel for fuel cell. The following overpotential ( ) vs.
current (i ) data was obtained in a study conducted to evaluate its electrochemical kinetics.
Table 7 Electrochemical Kinetics of borohydride data.
(V )
-0.29563 -0.24346 -0.19012 -0.18772
i ( A)
0.00226
0.00212
0.00206
0.00202

-0.13407
0.00199

-0.0861
0.00195

At the conditions of the study, it is known that the relationship that exists between the
overpotential ( ) and current (i ) can be expressed as
(27)
= a + b ln i

06.04.16

Chapter 06.04

where a is an electrochemical kinetics parameter of borohydride on the electrode. Use the

data in Table 7 to evaluate the values of a and b .
Solution
Following the least squares method, Table 8 is tabulated where
x = ln i
y =
We obtain
(28)
y = a + bx
This is a linear relationship between y and x , and the coefficients b and a are found as
follow
n

b=

n xi y i xi y i
i =1

i =1

i =1

n x xi
i =1
i =1
a = y bx
n

2
1

(29a,b)

Table 8 Summation values for calculating constants of model

x y
y =
x = ln(i ) x 2
i
#
1
0.00226
-0.29563 -6.0924 37.117
1.8011
2
0.00212
-0.24346 -6.1563 37.901
1.4988
3
0.00206
-0.19012 -6.1850 38.255
1.1759
4
0.00202
-0.18772 -6.2047 38.498
1.1647
5
0.00199
-0.13407 -6.2196 38.684
0.83386
6
0.00195
-0.08610 -6.2399 38.937
0.53726
6

i =1

0.012400

-1.1371

n=6
6

= 37.098

= 1.1371

i =1
6

i =1
6

x y
i =1
6

x
i =1

b=

= 7.0117

2
i

= 229.39

6(7.0117 ) ( 37.098)( 1.1371)

6(229.39 ) ( 37.098)
= 1.3601

-37.098

229.39

7.0117

Nonlinear Regression

06.04.17

1.1371
37.098
( 1.3601)
6
6
= 8.5990

a=

Hence

= 8.5990 1.3601 ln i

Figure 5 Overpotential as a function of current. (V )

Power Functions
The power function equation describes many scientific and engineering phenomena. In
chemical engineering, the rate of chemical reaction is often written in power function form as
(30)
y = ax b
The method of least squares is applied to the power function by first linearizing the data (the
assumption is that b is not known). If the only unknown is a , then a linear relation exists
between x b and y . The linearization of the data is as follows.
(31)
ln( y ) = ln(a ) + b ln( x )
The resulting equation shows a linear relation between ln(y ) and ln (x ) .
Let

06.04.18

Chapter 06.04

z = ln y
w = ln( x)

a 0 = ln a implying a = e a0
we get

a1 = b
z = a 0 + a1 w

(32)
n

a1 =

n wi z i wi z i
i =1

i =1

i =1

n w wi
i =1
i =1
n

zi
i =1

2
i

a0 =

(33a,b)

a1

w
i =1

n
n
Since a 0 and a1 can be found, the original constants of the model are
b = a1
a = e a0

(34a,b)

Example 6
The progress of a homogeneous chemical reaction is followed and it is desired to evaluate the
rate constant and the order of the reaction. The rate law expression for the reaction is known
to follow the power function form
r = kC n
(35)
Use the data provided in the table to obtain n and k .
Table 9 Chemical kinetics.
C A (gmol/l)
4
2.25 1.45 1.0
0.65 0.25 0.006
rA ( gmol/l s) 0.398 0.298 0.238 0.198 0.158 0.098 0.048
Solution
Taking the natural log of both sides of Equation (35), we obtain
ln ( r ) = ln (k ) + n ln (C )
Let
z = ln ( r )
w = ln (C

We get

a 0 = ln(k ) implying that k = e a0

a1 = n

(36)
(37)

Nonlinear Regression

06.04.19

z = a 0 + a1 w
This is a linear relation between z and w , where
n

n wi z i wi z i

a1 =

i =1

i =1

i =1

n w wi
i =1
i =1
n

n
zi
a 0 = i =1
n

2
i

wi
a i =1
1 n

(38a,b)

Table 10 Kinetics rate law using power function

w
i
r
C
1
4
0.398
1.3863
2
2.25
0.298
0.8109
3
1.45
0.238
0.3716
4
1
0.198
0.0000
5
0.65
0.158
-0.4308
6
0.25
0.098
-1.3863
7
0.006 0.048
-5.1160
7

-4.3643

i =1

n=7
7

= 4.3643

= 12.391

i =1
7

i =1
7

w z
i =1
7

i i

w
i =1

2
i

= 16.758

= 30.998

From Equation (38a,b)

7 (16.758) ( 4.3643) ( 12.391)
a1 =
2
7 (30.998) ( 4.3643)
= 0.31943

z
-0.92130
-1.2107
-1.4355
-1.6195
-1.8452
-2.3228
-3.0366

w z
-1.2772
-0.9818
-0.5334
0.0000
0.7949
3.2201
15.535

w2
1.9218
0.65761
0.13806
0.00000
0.18557
1.9218
26.173

-12.391

16.758

30.998

06.04.20

Chapter 06.04

12.391
4.3643
(.31943)
7
7
= 1.5711
From Equation (36) and (37), we obtain
k = e 1.5711
a0 =

= 0.20782
n = a1
= 0.31941
Finally, the model of progress of that chemical reaction is
r = 0.20782 C 0.31941

Figure 6 Kinetic chemical reaction rate as a function of concentration.

Growth Model
Growth models common in scientific fields have been developed and used successfully for
specific situations. The growth models are used to describe how something grows with
changes in a regressor variable (often the time). Examples in this category include growth of
thin films or population with time. In the logistic growth model, an example of a growth
model in which a measurable quantity y varies with some quantity x is
ax
(39)
y=
b+x

Nonlinear Regression

06.04.21

For x = 0 , y = 0 while as x , y a . To linearize the data for this method,

1 b+x
=
y
ax
(40)
b1 1
=
+
ax a
Let
1
z=
y
1
w = ,
x
1
1
a 0 = implying that a =
a
a0
a
b
a1 = implying b = a1 a = 1
a0
a
Then
(41)
z = a 0 + a1 w
The relationship between z and w is linear with the coefficients a 0 and found as follows.
n

a1 =

n wi z i wi z i
i =1

i =1

i =1

n w wi
i =1
i =1
n

2
i

n
wi
zi
a0 = i =1 a1 i =1
n
n

Finding a 0 and a1 , then gives the constants of the original growth model as
1
a=
a0
a
b= 1
a0

(42a,b)

(43a,b)

06.04.22

NONLINEAR REGRESSION
Topic
Nonlinear Regression
Summary Textbook notes of Nonlinear Regression
Major
General Engineering
Authors
Egwu Kalu, Autar Kaw, Cuong Nguyen
Date
June 17, 2015
Web Site http://numericalmethods.eng.usf.edu

Chapter 06.04