Parameter Estimation for ODE.

Shyan-Shiou Chen

Ph. D., Applied Mathematics,

Department of Mathematics,
National Taiwan Normal University

August 12, 2024

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 1 / 24
 Outline

Outline

1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm

2 fminsearch in Matlab

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 2 / 24
 lsqcurvefit in Matlab

lsqcurvefit 內含的演算法

1 Gauss-Newton Method (高斯-牛頓法)

2 Levenberg-Marquardt Method (LM 法)

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 3 / 24
 lsqcurvefit in Matlab Gradient Descent Method

Outline

1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm

2 fminsearch in Matlab

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 4 / 24
 lsqcurvefit in Matlab Gradient Descent Method

Gradient Descent Method

xk+1 = xk − α∇f (xk ) ,

Example
Consider f (x, y) = x2 + y 2 /4. Here, x = (x, y).

2 2 2 2
Gradient Descent on f(x,y) = x + y /4 Gradient Descent on f(x,y) = x + y /4
3 3

2 2

1 1

0 0
y

-1 y -1

Contour Contour
-2 Iteration Points -2 Iteration Points
Gradient Directions Gradient Directions

-3 -3
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
x x

(a) (b)

Figure: Matlab Code. (EX0050GradientDescent2dwithGraph.m)

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 5 / 24
 lsqcurvefit in Matlab Newton’s Method for finding zeros

Outline

1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm

2 fminsearch in Matlab

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 6 / 24
 lsqcurvefit in Matlab Newton’s Method for finding zeros

Newton Method for f (x) = 0 (cont.)

1 Also known as the Newton-Raphson method.

2 A fundamental algorithm in numerical analysis for finding the
roots of a real-valued function.
3 This method iteratively generates increasingly accurate
approximations to the roots (or zeros) of a function.

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 7 / 24
 lsqcurvefit in Matlab Newton’s Method for finding zeros

Newton Method for f (x) = 0 (cont.)

f (x0 )
x1 = x0 −
f ′ (x0 )

Geometric Interpretation Geometrically, the point (x1 , 0) represents the

x -intercept of the tangent to the graph of f at (x0 , f (x0 )). In other
words, x1 is the root of the linear approximation of f at the initial guess
x0 . This approximation process is iterated as follows:

f (xn )
xn+1 = xn −
f ′ (xn )

This process continues until the difference between successive

approximations is sufficiently small. The number of correct digits in
the approximation typically doubles with each iteration, showcasing
the quadratic convergence of the method.

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 8 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Outline

1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm

2 fminsearch in Matlab

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 9 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Newton’s method in optimization (cont.)

Given a twice differentiable function f : R → R, we seek to solve the

optimization problem

min f (x)
x∈R

Newton’s method attempts to solve this problem by constructing a se-

quence {xk } from an Newton’s method uses curvature information (i.e.
the second derivative) to take a more direct route. initial guess (starting
point) x0 ∈ R that converges towards a minimizer x∗ of f by using a se-
quence of second-order Taylor approximations of f around the iterates.
The second-order Taylor expansion of f around xk is
1
f (xk + t) ≈ f (xk ) + f ′ (xk ) t + f ′′ (xk ) t2
2

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 10 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Newton’s method in optimization (cont.)

The next iterate xk+1 is defined so as to minimize this quadratic ap-

proximation in t, and setting xk+1 = xk + t. If the second derivative is
positive, the quadratic approximation is a convex function of t, and its
minimum can be found by setting the derivative to zero. Since
( )
d 1 ′′
0= f (xk ) + f (xk ) t + f (xk ) t = f ′ (xk ) + f ′′ (xk ) t
′ 2
dt 2

the minimum is achieved for

f ′ (xk )
t=− .
f ′′ (xk )

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 11 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Newton’s method in optimization (cont.)

Putting everything together, Newton’s method performs the iteration

f ′ (xk )
xk+1 = xk + t = xk −
f ′′ (xk )

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 12 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Parameter Estimation by Newton’s method (cont.)

1 Objective Function: The goal is to minimize the objective

function, which is typically the sum of squared residuals:

∑
n
S(θ) = [yi − f (xi , θ)]2
i=1

where yi are the observed data, f (xi , θ) is the model prediction,

and θ represents the parameters to be estimated.
2 Gradient (First Derivative): The gradient of the objective function
with respect to the parameters θ is given by:

∂S(θ)
∇S(θ) =
∂θ

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 13 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Parameter Estimation by Newton’s method (cont.)

Example
∑
Let θ be a parameter. S(θ) = 2i=1 [yi − f (xi , θ)]2 . Find that ∇S(θ).
Let ri = yi − f (xi , θ). Find J(θ) such that ∇S(θ) = −J(θ) · (r1 , r2 )t .

3 Hessian Matrix (Second Derivative): The Hessian matrix H is the

matrix of second-order partial derivatives of the objective function
with respect to the parameters:

∂ 2 S(θ)
H(θ) =
∂θ2

The Hessian captures the curvature of the objective function and

provides additional information that can be used to update the
parameters more effectively.

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 14 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Parameter Estimation by Newton’s method (cont.)

Example
[( )2 ]
∂ 2 S(θ) ∑2 ∂f (xi ,θ) 2 (x ,θ)
Show that ∂θ 2
=2 i=1 ∂θ − ri ∂ f∂θ 2
i
.

4 Parameter Update Rule: The Newton method updates the

parameters using the following rule:

θk+1 = θk − H (θk )−1 ∇S (θk )

where θk is the current estimate of the parameters, H (θk ) is the

Hessian matrix at θk , and ∇S (θk ) is the gradient at θk .
5 Iteration and Convergence: The method iteratively updates the
parameter estimates until convergence is achieved, which typically
means that the change in parameters or the gradient is below a
specified tolerance.

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 15 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Gauss-Newton Method (cont.)

Hessian Approximation: The Hessian matrix H(θ), which would

normally involve second derivatives, is approximated in the
Gauss-Newton method as:

H(θ) ≈ J(θ)T J(θ)

Parameter Update Rule: The parameters are updated using the

following rule:

θk+1 = θk + ∆θ

where ∆θ is computed as:

[ ]−1
∆θ = J (θk )T J (θk ) J (θk )T r (θk )

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 16 / 24
 lsqcurvefit in Matlab Newton’s method in optimization

Gauss-Newton Method (cont.)

 ∂f (x1 ,θ) ∂f (x1 ,θ) ∂f (x1 ,θ)


∂θ1 ∂θ2 ... ∂θm
 ∂f (x2 ,θ) ∂f (x2 ,θ) ∂f (x2 ,θ) 
 ... 
J(θ) = 

∂θ1
..
∂θ2
.. ..
∂θm
..


 . . . . 
∂f (xn ,θ) ∂f (xn ,θ) ∂f (xn ,θ)
∂θ1 ∂θ2 ... ∂θm

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 17 / 24
 lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm

Outline

1 lsqcurvefit in Matlab
Gradient Descent Method
Newton’s Method for finding zeros
Newton’s method in optimization
The Levenberg-Marquardt Algorithm

2 fminsearch in Matlab

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 18 / 24
 lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm

The Levenberg-Marquardt Algorithm

Parameter Update Rule: The LM algorithm updates the parameters

using:
[ ]−1
θk+1 = θk − J (θk )T J (θk ) + λI J (θk )T r (θk )

Here, λ is the damping factor that controls the step size and direction.
When λ is large, the algorithm behaves more like gradient descent,
ensuring stability. When λ is small, it behaves more like the
Gauss-Newton method, promoting faster convergence.

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 19 / 24
 lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm

Examples for Newton Method (cont.)

Example
Consider a simple curve-fitting problem where we want to fit a model
y = aebx to some data points (xi , yi ) The residuals are given by

ri (a, b) = yi − aebxi

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 20 / 24
 lsqcurvefit in Matlab The Levenberg-Marquardt Algorithm

Examples for Newton Method (cont.)

Newton Method Fitting: y = a * exp(b * x)

6.5

Observed Data
6 Fitted Model

5.5

y 4.5

3.5

2.5

2
1 1.5 2 2.5 3 3.5 4 4.5 5
x

Figure: (Code: EX0052NewtonMethod01.m)

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 21 / 24
 fminsearch in Matlab

fminsearch in Matlab (Animation: Adobe only)

f (x, y) = 100(y − x2 )2 + (1 − x)2

Figure: Nelder-Mead Simplex Method Optimization Process (fminsearch.m)

(Adobe only.)
S-S Chen (Dep. of Math., NTNU)
Parameter Estimation August 12, 2024 22 / 24
 fminsearch in Matlab

Compare three different methods

Comparison of Parameter Estimation Methods

250
Observed Data
lsqcurvefit
fminsearch
200 fminunc

150
y

100

50

0
1 2 3 4 5 6
x

Figure: Compare three methods,

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 23 / 24
 fminsearch in Matlab

Thank You for Your Attention!

S-S Chen (Dep. of Math., NTNU) Parameter Estimation August 12, 2024 24 / 24