Nonlinear Regression

by

Dr. Md. Rajibul Islam

CSE, UAP

10/11/22 1
 Nonlinear Regression
Some popular nonlinear regression models:

1. Exponential model: ( y  aebx )

2. Power model: ( y  ax b )
 ax 
3. Saturation growth model:  y  
 b  x 
4. Polynomial model: ( y  a 0  a1 x  ...  amx m )

2
 Nonlinear Regression
Given n data points ( x1, y1), ( x 2, y 2), ... , ( xn, yn ) best fit y  f (x )
to the data, where f (x) is a nonlinear function of x .

( xn , y n )

( x2 , y 2 )
y  f (x)
( xi , yi )
yi  f ( xi )
(x , y )
1 1

Figure. Nonlinear regression model for discrete y vs. x data

3
 Regression
Exponential Model

4
 Exponential Model
Given ( x1 , y1 ), ( x 2 , y 2 ), ... , ( x n , y n ) best fit y  ae to the data.
bx

( x1 , y1 )
y  aebx

yi  ae bxi
(x , y )
i i

(x , y )
2 2 (x , y )
n n

Figure. Exponential model of nonlinear regression for y vs. x

data

5
 Finding Constants of Exponential Model
The sum of the square of the residuals is defined as

 
n
bxi 2
Sr   yi  ae
i 1
Differentiate with respect to a and b

S r n

a
 
  2 y i  ae bxi  e bxi  0 
i 1

S r
 
n

b
  2 y i  ae bxi bxi
 axi e  0 
i 1

6 http://numericalmethods.eng.usf.edu
 Finding Constants of Exponential Model
Rewriting the equations, we obtain

n n
bxi 2bxi
  yi e  ae 0
i 1 i 1

n n
bxi 2bxi
 y i xi e  a  xi e 0
i 1 i 1

7
 Finding constants of Exponential Model
Solving the first equation for a yields
n
bxi
 yi e
i 1
a n
2bxi
 e
i 1

Substituting a back into the previous equation

n
bxi
n
 yi e n
bxi i 1 2bxi
 y i xi e  n
x
 i e 0
i 1 2bxi i 1
 e
i 1
The constant b can be found through numerical
methods such as bisection method.
8
 Example 1-Exponential Model
Many patients get concerned when a test involves injection of a
radioactive material. For example for scanning a gallbladder, a
few drops of Technetium-99m isotope is used. Half of the
Technetium-99m would be gone in about 6 hours. It, however,
takes about 24 hours for the radiation levels to reach what we
are exposed to in day-to-day activities. Below is given the
relative intensity of radiation as a function of time.

Table. Relative intensity of radiation as a function of time.

t(hrs) 0 1 3 5 7 9
 1.000 0.891 0.708 0.562 0.447 0.355

9
 Example 1-Exponential Model cont.
The relative intensity is related to time by the equation
t
  Ae
Find:
a) The value of the regression constants A and 
b) The half-life of Technetium-99m
c) Radiation intensity after 24 hours

10
 Plot of data

11
 Constants of the Model
  Ae t

The value of λ is found by solving the nonlinear equation

n
ti
n

 i e n
f      i t i e ti
 i 1
n
t
 i e 2ti
0
i 1 2ti i 1
 e
i 1
n

 i
 e t i

A i 1
n

 e 2 t i

i 1
12
 Setting up the Equation in MATLAB
n
ti
n
 ie n
f      i t i e ti
 i 1
n
 ti e
2ti
0
i 1 2ti i 1
e
i 1

t (hrs) 0 1 3 5 7 9
γ 1.000 0.891 0.708 0.562 0.447 0.355
13
 Setting up the Equation in MATLAB
n
ti
n
 ie n
f      i t i e ti
 i 1
n
t
 i e 2ti
0
i 1 2ti i 1
e
i 1

  0.1151
t=[0 1 3 5 7 9]
gamma=[1 0.891 0.708 0.562 0.447 0.355]
syms lamda
sum1=sum(gamma.*t.*exp(lamda*t));
sum2=sum(gamma.*exp(lamda*t));
sum3=sum(exp(2*lamda*t));
sum4=sum(t.*exp(2*lamda*t));
f=sum1-sum2/sum3*sum4;
14
 Calculating the Other Constant
The value of A can now be calculated
6

 e i
ti

A i 1
6  0.9998
 e 2 ti

i 1

The exponential regression model then is

0.1151 t
  0.9998 e

15
 Plot of data and regression curve

  0.9998 e 0.1151t

16
 Relative Intensity After 24 hrs
The relative intensity of radiation after 24 hours
0.1151 24 
  0.9998  e
2
 6.3160  10
This result implies that only

6.316  102
 100  6.317%
0.9998
radioactive intensity is left after 24 hours.
17
 Homework
• What is the half-life of Technetium-99m
isotope?
• Write a program in the language of your
choice to find the constants of the
model.
• Compare the constants of this regression
model with the one where the data is
transformed.
t
• What if the model was   e ?

18
 THE END

19
 Polynomial Model
Given ( x1, y1), ( x 2, y 2), ... , ( xn, yn) best fit y  a  a x  ...  a x m
0 1 m
(m  n  2) to a given data set.

(x , y )
n n

( x2 , y 2 )

(x , y ) y  a0  a1 x    am x m
i i
y  f (x )
i i
( x1 , y1 )

Figure. Polynomial model for nonlinear regression of y vs. x data

20
 Polynomial Model cont.
The residual at each data point is given by
Ei  y i  a 0  a1 xi  . . .  a m xim
The sum of the square of the residuals then is
n
S r   Ei2
i 1
n

  y i  a 0  a1 xi  . . .  a m xim 
2

i 1

21
 Polynomial Model cont.
To find the constants of the polynomial model, we set the derivatives with
respect to ai where i  1, m, equal to zero.
S r n
  2.yi  a0  a1 xi  . . .  am xim (1)  0
a0 i 1
S r n
  2.yi  a0  a1 xi  . . .  am xim ( xi )  0
a1 i 1

   
S r n
  2.yi  a0  a1 xi  . . .  am xim ( xim )  0
am i 1

22
 Polynomial Model cont.
These equations in matrix form are given by
   
   n 
 n  n   n m 
  xi  .  xi  a
   yi
. .  
  i 1   i 1    0 
 n    ni 1
   xi   n 2  n m1  a1 

  xi  . . .  xi   xi y i 
  i 1   i 1   i 1  . . 
.  i 1 
. . . . . . . . . . .  a  . . . 
  m  
n 
 n m   n m1   n
  xim yi 
  xi    xi  . . .  xi2 m  
 i 1 
 i 1   i 1   i 1 

The above equations are then solved for a0 , a1 ,  , am

23
 Example 2-Polynomial Model
Regress the thermal expansion coefficient vs. temperature data to
a second order polynomial.

Table. Data points for 7.00E-06

temperature vs α

Thermal expansion coefficient, α

Temperature, T Coefficient of 6.00E-06
(oF) thermal
expansion, α 5.00E-06
(in/in/oF)
80 6.47×10−6 (in/in/o F) 4.00E-06

40 6.24×10−6
3.00E-06
−40 5.72×10 −6

−120 5.09×10−6 2.00E-06

−200 4.30×10−6 1.00E-06

−280 3.33×10−6 -400 -300 -200 -100 0 100 200

−340 2.45×10−6 Temperature, oF

Figure. Data points for thermal expansion coefficient vs

temperature.
24
 Example 2-Polynomial Model cont.
We are to fit the data to the polynomial regression model
α  a0  a1T  a 2T 2
The coefficients a0 ,a1 , a2 are found by differentiating the sum of the
square of the residuals with respect to each variable and setting the
values equal to zero to obtain

  n   n 2   n 
 n   Ti    Ti     i 
  i 1   i 1   a   i 1 
0
 n   n 2  n 3     n 
  i 
 T   Ti    Ti   a1    Ti  i
 in1   i 1   i 1      i 1 
a  n 
 T 2   n 3  n 4    2   T 2
  i   Ti    Ti  


 i i 

 i1   i 1   i 1   i 1

25
 Example 2-Polynomial Model cont.
The necessary summations are as follows
7
α
Table. Data points for temperature vs.
Temperature, T Coefficient of
T
i 1
i
2
 2.5580 105
(oF) thermal expansion,
7

T
α (in/in/oF) 3
i   7.0472  10 7
80 6.47×10−6
i 1
40 6.24×10 −6
7
−40 5.72×10 −6
T
i 1
i
4
 2.1363 1010
−120 5.09×10−6
7


−200 4.30×10−6
i  3.3600  10 5
−280 3.33×10−6
i 1
−340 2.45×10 −6
7

T 
i 1
i i   2.6978  10 3
7

T
i 1
i
2
 i 8.5013  10 1

26
 Example 2-Polynomial Model cont.
Using these summations, we can now calculate a0 ,a1 , a2
 7.0000  8.6000  10 2 2.5800  10 5  a 0   3.3600  10 5 
   
  8 . 600  10 2
2.5800  10 5  7.0472  10 7   a1    2.6978  10 3 
 2.5800  10 5  7.0472  10 7 2.1363  1010  a 2   8.5013  10 1 

Solving the above system of simultaneous linear equations we have
a 0   6.0217  10 
6

 a    6.2782  10 9 
 1  
a 2   1.2218  10 
11

The polynomial regression model is then

α  a 0  a1T  a 2T 2
 6.0217  10 6  6.2782  10 9 T  1.2218  10 11 T 2

27
 Transformation of Data
To find the constants of many nonlinear models, it results in solving
simultaneous nonlinear equations. For mathematical convenience, some
of the data for such models can be transformed. For example, the data
for an exponential model can be transformed.
As shown in the previous example, many chemical and physical processes
are governed by the equation,
y  ae bx
Taking the natural log of both sides yields,
ln y  ln a  bx
Let z  ln y and a 0  ln a
We now have a linear regression model where z  a 0  a1 x
(implying) a  e ao with a1  b

28
 Transformation of data cont.
Using linear model regression methods,
n n n
n  xi z i   xi  z i
i 1 i 1 i 1
a1  2
n
  n
n xi2    xi 
i 1  i 1 
_ _
a 0  z  a1 x
Once ao , a1 are found, the original constants of the model are found as
b  a1
a  e a0

29
 THE END

30
 Example 3-Transformation of
data
Many patients get concerned when a test involves injection of a radioactive
material. For example for scanning a gallbladder, a few drops of Technetium-
99m isotope is used. Half of the Technetium-99m would be gone in about 6
hours. It, however, takes about 24 hours for the radiation levels to reach what
we are exposed to in day-to-day activities. Below is given the relative intensity
of radiation as a function of time.
1
Table. Relative intensity of radiation as a function

Relative intensity of radiation, γ

of time
t(hrs) 0 1 3 5 7 9
 1.000 0.891 0.708 0.562 0.447 0.355 0.5

0
0 5 10
Time t, (hours)

Figure. Data points of relative radiation intensity

vs. time
31
 Example 3-Transformation of data
cont.
Find:
a) The value of the regression constants A and 
b) The half-life of Technetium-99m
c) Radiation intensity after 24 hours
The relative intensity is related to time by the equation
  Ae t

32
 Example 3-Transformation of data
cont.
Exponential model given as,
  Ae t
ln    ln A  t
Assuming z  ln  , a o  ln  A and a1   we obtain
z a at
0 1
This is a linear relationship between z and t

33
 Example 3-Transformation of data cont.
Using this linear relationship, we can calculate a0 , a1 where
n n n
n t z   t z
i i i i
a1  i 1 i 1 i 1
2

n
 n
n t12    ti 
i 1  i 1 
and
a0  z  a1t
  a1
a0
Ae

34
 Example 3-Transformation of Data
cont.
Summations for data transformation are as follows

Table. Summation data for Transformation of data

With n  6
6

i ti
model
i zi  ln  i tz t 2 t
i 1
i  25.000
i i i
6
1 0 1 0.00000 0.0000 0.0000
2 1 0.891 −0.11541 −0.11541 1.0000 z
i 1
i
 2.8778
3 3 0.708 −0.34531 −1.0359 9.0000
4 5 0.562 −0.57625 −2.8813 25.000 6
5
6
7
9
0.447
0.355
−0.80520
−1.0356
−5.6364
−9.3207
49.000
81.000 t z
i 1
i i
 18.990

6
 25.000 −2.8778 −18.990 165.00
t i
2
 165.00
i 1

35
 Example 3-Transformation of Data
cont.
Calculating a0 , a1
6 18.990  25 2.8778
a1   0.11505
6165.00  25
2

 2.8778 25
a0    0.11505  2.6150  10 4
6 6
Since
a0  ln  A
A  e a0
 2.6150104
e  0.99974
also
  a1  0.11505

36
 Example 3-Transformation of Data
cont.
0.11505 t
Resulting model is   0.99974  e

1
  0.99974  e 0.11505t

Relative
Intensity
0.5
of
Radiation,

0
0 5 10
Time, t (hrs)

Figure. Relative intensity of radiation as a function of

temperature using transformation of data model.

37
 Example 3-Transformation of Data
cont.
The regression formula is then
  0.99974  e 0.11505t
1
b) Half life of Technetium-99m is when   
2 t 0
1
0.99974  e 0 .11505t  0.99974e 0 .115050 
2
e 0 .11508t  0.5
 0.11505t  ln0.5
t  6.0248 hours

38
 Example 3-Transformation of Data
cont.
c) The relative intensity of radiation after 24 hours is then
  0.99974e 0.1150524 
 0.063200
6.3200 10 2
This implies that only 100  6.3216% of the radioactive
0.99983
material is left after 24 hours.

39
 Comparison
Comparison of exponential model with and without data
Transformation:
Table. Comparison for exponential model with and without data
Transformation.
With data Without data
Transformation Transformation
(Example 3) (Example 1)
A 0.99974 0.99983
λ −0.11505 −0.11508
Half-Life (hrs) 6.0248 6.0232
Relative intensity
6.3200×10−2 6.3160×10−2
after 24 hrs.

40
THE END