Prof.

Dr Sonnet Nguyen HUS, VNU

[email protected] Computational Physics 1

Solving nonlinear equations Table of Contents

 Introduction
 Solving one equation
 Graph method
Course of Computational Physics 1  Bisection method
 Fixed-point Iteration
Lecturer: dr Hưng / Sonnet Nguyen  Newton-Raphson method
 improved Newton-Raphson methods
Hanoi Univ. of Science  Secant method
E-mail: [email protected]  Brent method
 Regula-Falsi method
Copyright 2007-2022
 Solving a system of nonlinear equations
https://classroom.google.com/u/0/c/NTUwNTg1  Minimization and Maximization
MDk1MjI2  Steepest descent method
1 Computational Physics 1, Hanoi Univ of Science 2/73

Introduction Example
Much of the problems of science lead to the problem of solving A geometric representation of a nonlinear system when n  2,
simultaneous equations and inequalities. In this lecture we shall
f1 ( x1 , x2 )  0  f 2 ( x1 , x2 )  0, is given in the Figure below
focus into numerical algorithms for solving one equation or a
system of nonlinear equations. At the beginning we consider the
root-finding problem, i.e. finding a root or solution of an equation
f ( x )  0, which is one of the most basic problems of numerical
approximation. Later, we consider the more difficult problem:
Finding a solution of a system of m equations with n unknowns:
 f1 ( x1 ,....., xn )  0  Given F : R n  R m .

 ⋮ OR    
 f ( x ,....., x )  0 Find x  R such F ( x )  0 .
n
 m 1 n

Computational Physics 1, Hanoi Univ of Science 3/73 Computational Physics 1, Hanoi Univ of Science 4/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Example Graphing method

The 3 × 3 nonlinear system
1  This method is used to determine the interval
f1 ( x1 , x2 , x3 ) = 3x1  cos( x2 x3 )   0, containing a root.
4
f 2 ( x1 , x2 , x3 ) = x12  64( x2  0.1) 2  sin x3  1.06  0,
1
f 3 ( x1 , x2 , x3 ) = e  x1 x2  20 x3  (40  3)  0.
3

Next we consider one of the most basic problems of numerical

approximation, the root-finding problem. Thus, to find a root,
or solution, of an equation of the form f ( x )  0.
A root of this equation is also called a zero of the function f .
This is one of the oldest known approximation problems.
Computational Physics 1, Hanoi Univ of Science 5/73 Computational Physics 1, Hanoi Univ of Science 6/73

Interval contains roots Bisection method

Determine an interval [ a, b] (or open ( a, b)) in which one and Suppose that f is continuous on the interval [a, b] and
only one root exists. The interval can be found by use of the
that f (a) and f (b) are of opposite sign. We want to
graphing method or well-known theorem.
find a solution to f ( x)  0 on an interval (a, b).
Theorem:
Although the method will work for the case when more
If f ( x) is continuous, monotonic and f (a )* f (b)  0, then
the interval ( a, b) contains exactly one root.
than one root is contained in the interval [a, b], we
assume for simplicity of our discussion that the root in
In the theorem above we can replace the "monotonicity" of the this interval is unique.
function f by the condition "unchanging sign of its derivative".

Computational Physics 1, Hanoi Univ of Science 7/73 Computational Physics 1, Hanoi Univ of Science 8/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Bisection method Bisection method

To begin the Bisection method, set a1  a and b1  b and let
ab
p1 = , be the midpoint of the interval [ a, b].
2
If f ( p1 )  0, then the root is given by p  p1.
Otherwise, then f ( p1 ) has the same sign as either f ( a1 ) or f (b1 ).
If f ( p1 ) and f (a1 ) have the same sign, then p is in the interval
(p1 ,b1 ) and we set a2  p1 and b2 =b1.
If f ( p1 ) and f (b1 ) have the same sign, then p is in the interval
(a1 ,p1 ) and we set a2  a1 and b2 =p1.
We reapply the process to the interval [ a2 , b2 ], and continue
forming [ a3 , b3 ], [ a4 , b4 ],...... Each new interval will contain p
and have length one half of the length of the preceding interval.
Computational Physics 1, Hanoi Univ of Science 9/73 Computational Physics 1, Hanoi Univ of Science 10/73

Bisection method Flow chart of Bisection Method

There are three stopping criteria commonly incorporated in Start: Given a,b and ε
the Bisection method.
u = f(a) ; v = f(b)
First, the method stops if one of the midpoints happens to
coincide with the root. c = (a+b) /2 ; w = f(c) no
Secondly, it also stops when the length of the search interval yes
yes is no is
is less than some prescribed tolerance we call TOL. Stop
(b-a)/2 <ε
u w <0
Thirdly, the procedure also stops if the number of iterations
exceeds a preset bound N0 . b=c; v= w a=c; u= w

Computational Physics 1, Hanoi Univ of Science 11/73 Computational Physics 1, Hanoi Univ of Science 12/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Bisection method – C code Bisection method – C code

#include<stdio.h> printf("Please enter endpoints A and B of the interval [A,B]\n");

#include<stdlib.h> scanf("%lf %lf", &A, &B);
#include<math.h> printf("The interval ranges from %lf to %lf\n", A,B);
double ff(double x); YA = ff(A); YB = ff(B); /* compute function values */
double ff(double x) /* EXAMPLE for a function ff */ Max = (int) ( 1 + floor( ( log(B-A) - log(Delta) ) / log(2) ) );
{ printf("Max = %d\n",Max);
return (x*x-2); /* Check whether YA * YB <= 0 */
} if( ( (YA >= 0) && (YB >=0) ) || ( (YA < 0) && (YB < 0) ) ){
void main() /* Main program for bisection method */ printf("The values of function at (A) and (B)
{ do not differ in sign.\n");
double Delta = 1E-5; /* Tolerance for width of interval */ exit(0); /* exit program */
int Satisfied = 0; /* Condition for loop termination */ }
double A, B; /* Endpoints of the interval [A,B] */ for(K = 1; K <= Max ; K++) {
double YA, YB; /* Function values at A and B */ if(Satisfied == 1) break;
int Max; /* The maximum number of iterations */ C = (A + B) / 2; /* Midpoint of interval */
int K; /* Loop Counter */ YC = ff(C); /* Function value at midpoint */
double C, YC; /* Midpoint of interval and f value there */
Computational Physics 1, Hanoi Univ of Science 13/73 Computational Physics 1, Hanoi Univ of Science 14/73

Bisection method – C code Bisection method – exercises

if( YC == 0) {  Replace the “for-loop” by “while-loop” in the

A = C; /* Exact root is found */ above program.
B = C;
}  Modify the program that it prints result of each
else if( ( (YB >= 0) && (YC >=0) ) || ( (YB < 0) && (YC < 0) ) ) { step of calculation into a file. This file will have
B = C; YB = YC; five columns containing the values of A, B, B-A,
} C, f(C).
else {
A = C; YA = YC;
}
if( (B-A) < Delta ) Satisfied = 1;
} /* end of for-loop */
printf("The maximum number of iterations is : %d\n",Max);
printf("The number of performed iterations is : %d\n",K - 1);
printf("The computed root of f(x) = 0 is : %lf \n",C);
printf("The accuracy is +- %lf\n", B-A);
printf("The value of the function f(C) is %lf\n",YC);
}
Computational Physics 1, Hanoi Univ of Science 15/73 Computational Physics 1, Hanoi Univ of Science 16/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Bisection method - Matlab code Bisection method - Matlab code

function root = bisect(func,x1,x2,filter,tol)
% USAGE: root = bisect(func,x1,x2,filter,tol) n = ceil(log(abs(x2 - x1)/tol)/log(2.0)); % n so lan lap: tol >
abs(x2-x2)/2^n
% INPUT: for i = 1:n
% func = ham so; x1,x2 = khoang cach ly nghiem. x3 = 0.5*(x1 + x2); f3 = feval(func,x3);
% filter = singularity filter: 0 = off (default), 1 = on. if (filter == 1) & (abs(f3) > abs(f1)) & (abs(f3) > abs(f2))
% tol = sai so /error tolerance, default is 1.0e4*eps/. root = NaN; return
end
% OUTPUT: if f3 == 0.0
% root = zero of f(x), or NaN if singularity suspected. root = x3; return
end
if nargin < 5; tol = 1.0e4*eps; end; if f2*f3 < 0.0
% Neu dua vao < 5 tham so thi sai so mac dinh la 1.0e4*eps x1 = x3; f1 = f3;
if nargin < 4; filter = 0; end; else
f1 = feval(func,x1); if f1 == 0.0; root = x1; return; end; x2 = x3; f2 = f3;
end
f2 = feval(func,x2); if f2 == 0.0; root = x2; return; end;
end
if f1*f2 > 0; error(‘Nghiem khong nam trong doan (x1,x2)’); end; root=(x1 + x2)/2;

Computational Physics 1, Hanoi Univ of Science 17/73 Computational Physics 1, Hanoi Univ of Science 18/73

Fixed-point problem Fixed-point iteration

Step 1: We transform a root-finding problem to a fixed-point problem:
A point p is called a fixed point for a function g if g ( p )  p. f ( x )  0  x  g ( x).
Problem of finding fixed points for a given function is called This is not unique, i.e. given f , we can choose many possible g.
a fixed-point problem. Step 2: We choose an initial approximation x0 .
Fact: root-finding problem and fixed-point problem is equivalent. Step 3: Generate sequence
Proof. Given a root-finding problem f ( p )  0 then we can x1  g ( x0 ), x2  g ( x1 ), ⋯ , xk  g ( xk 1 ), ⋯
define many functions g having a fixed point at p. If this sequence converges to p and f is continuous, then
For example, one can define: g ( x)  x  k * f ( x). p  lim xn  lim g ( xn1 )  g (lim xn1 )  g ( p ).
n  n  n 
Conversely, if the function g has a fixed point at p, then the Questions: How to to choose the initial value x0 such that the
function f defined by f ( x)  x  g ( x), has a root x  p. sequence {xk } converges?
What is the formal condition for stop the loop in Step 3?
Computational Physics 1, Hanoi Univ of Science 19/73 Computational Physics 1, Hanoi Univ of Science 20/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Flow chart of fixed-point iteration Sufficient condition for fixed-point iteration

Theorem
Suppose that g ( x) is continuous in [ a, b] and for all x in [a, b]
the value g ( x) in [a, b] too.
Suppose, in addition, that g ' exists on (a, b) and exists a
constant q such that 0  q  1, and | g '( x) | q for all x in (a, b).
Then for any number x0 in [a, b] the sequence defined by
x1  g ( x0 ), x2  g ( x1 ), … , xk  g ( xk 1 ), …
converges to the unique fixed-point in [ a, b].

Computational Physics 1, Hanoi Univ of Science 21/73 Computational Physics 1, Hanoi Univ of Science 22/73

Estimation for fixed-point iteration Newton-Raphson method

Newton's method exploits the derivatives f '( x) of the function f ( x)

If g satisfies the hypotheses of the above theorem, then for n  1:
n
to accelerate convergence for solving: f ( x)  0 with the required
q
a) | p  xn | | x1  x0 | . tolerance. An analytic function f ( x) around the point x may be
1 q expanded in a Taylor series:
b) | p  xn | q n max{| x0  a |,| b  x0 |}. f (2) ( x0 )
f ( x )  f ( x0 )  f '( x0 )( x  x0 )  ( x  x0 ) 2  ....
These can be used to estimate the MAX number of iteration to 2!
achieve a given precision. Suppose x is the solution for f ( x)  0. If we keep the first two terms
in our Taylor series, we obtain: f ( x)  0 = f ( x0 )  f '( x0 )* ( x - x0 ).
f ( x0 )
Whence it follows that: x  x0  . We need more than one
f '( x0 )
f ( xk )
iteration to find the root: xk 1  xk  .
f '( xk )
Computational Physics 1, Hanoi Univ of Science 23/73 Computational Physics 1, Hanoi Univ of Science 24/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Geometric meaning of the Newton-Raphson

method Newton-Raphson Algorithm

Choose x0 in the interval (a, b) containing root.

The tangent of f at point A0 ( x0 , f ( x0 )) intersects the x-axis at point x1.
The tangent of f at point A1 ( x1 , f ( x1 )) intersects the x-axis at point x2 .
Continue this iteration. The tangent of f at point Ak ( xk , f ( xk ))
intersects the x -axis at xk .
Since the equation of the tangent of f at point Ak ( xk , f ( xk )) is:
y  f ( xk )  f '( xk )* ( x  xk ),
and point ( xk 1 ,0) belongs to this tangent, thus
0  f ( xk )  f '( xk ) *( xk 1  xk ).
f ( xk )
Therefore we get an explicit formula: xk 1  xk  .
f '( xk )
Computational Physics 1, Hanoi Univ of Science 25/73 Computational Physics 1, Hanoi Univ of Science 26/73

Newton-Raphson theorem Convergence of Newton-Raphson method

Theorem Usually, if the initial x0 chosen to be close to an exact root,

Suppose that f in C 2 ([ a , b ]). If p in [a , b ] such that the Newton-Raphson sequence converges very quickly to
f ( p )  0 and f '( p )  0, the exact root.
then exists a  >0 such that sequence {xn } defined by The Newton-Raphson sequence may not converge. Here
the Newton-Raphson method are some typical cases when the sequence diverges:
f ( xk )  Iteration point is stationary: eg. f ( x)  1  x 3 , x0  0.
xk 1  g ( xk )  xk 
f '( xk )  Starting point enters a cycle: eg. f ( x )  x 3  2 x  2, x0  0.
converges to p for all initial approximation point  Derivative does not exist at the root or is not continuous
x0  [p   , p   ]. at the root.

Computational Physics 1, Hanoi Univ of Science 27/73 Computational Physics 1, Hanoi Univ of Science 28/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Newton-Raphson - C code Newton-Raphson - C code

#include<stdio.h> /* Main program */
#include<stdlib.h> void main()
#include<math.h> {
/* define function f(x) */ double Delta = 1E-6; /* Tolerance */
double f0(double x); double Epsilon = 1E-6; /* Tolerance */
double f1(double x); double Small = 1E-6; /* Tolerance */
/* EXAMPLE for f0 */ int Max = 99; /* Maximum number of iterations */
double ffunction(double x) int Cond = 0; /* Condition fo loop termination */
{ int K; /* Counter for loop */
return ( pow(x,3) - 3 * x + 1 ); double P0; /* INPUT : Must be close to the root */
} double P1; /* New iterate */
/* EXAMPLE for f1, first derivative of f0. */ double Y0; /* Function value */
double f1(double x) double Y1; /* Function value */
{ double Df; /* Derivative */
return ( 3 * pow(x,2) - 3 ); double Dp;
} double RelErr;
/* ------------------------------------------------ */
Computational Physics 1, Hanoi Univ of Science 29/73 Computational Physics 1, Hanoi Univ of Science 30/73

Newton-Raphson - C code Newton-Raphson - C code

printf("----------------------------------------------\n"); P1 = P0 - Dp; /* New iterate */
printf("Please enter initial approximation of root !\n"); Y1 = f0(P1); /* New function value */
scanf("%lf",&P0);
printf("----------------------------------------------\n"); RelErr = 2 * fabs(Dp) / ( fabs(P1) + Small ); /* Relative error */
printf("Initial value for root: %lf\n",P0);
if( (RelErr < Delta) && (fabs(Y1) < Epsilon) ) { /* Check for */
Y0 = f0(P0);
if( Cond != 1) Cond = 2; /* convergence */
for ( K = 1; K <= Max ; K++) {
if(Cond) break; }
Df = f1(P0); /* Compute the derivative */
if( Df == 0) { /* Check division by zero */ P0 = P1;
Cond = 1; Y0 = Y1;
Dp = 0; printf("P0=%4.9lf, Y0=%4.9lf\n",P0,Y0);
} }
else Dp = Y0/Df;

Computational Physics 1, Hanoi Univ of Science 31/73 Computational Physics 1, Hanoi Univ of Science 32/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Newton-Raphson - C code The Fourier theorem

printf("----------------------------------------------\n");
printf("The current %d -th iterate is %lf\n",K-1, P1); Theorem (sufficient condition)
printf("Consecutive iterates differ by %lf\n",Dp);
printf("The value of f(x) is %lf\n",Y1); Suppose that the interval [a, b] contains a root of equation f ( x)  0.
printf("----------------------------------------------\n"); In addition, suppose that the derivatives f '( x), f ''( x) are continuous,
if(Cond == 0) printf("The maximum number of iterations was exceeded do not change a sign and do not vanish in [a, b].
!\n");
Then for any initial value x0 in [a, b] such f ( x0 ) * f ''( x0 )  0,
if(Cond == 1) printf("Division by zero was encountered !\n"); the Newton-Raphson sequence {xk } converges to the exact root.

if(Cond == 2) printf("The root was found with the desired tolerance !\n");

printf("----------------------------------------------\n");

} /* End of main program */

Computational Physics 1, Hanoi Univ of Science 33/73 Computational Physics 1, Hanoi Univ of Science 34/73

Order of convergence Zero of multiplicity m

Suppose that { pn } converges to p , and denote E n  p  pn
for n  0. A solution p of the equation f ( x)  0 is called a zero of
If there exists a constant r and constant A such that multiplicity m if f ( p )  0 and we can write
| p  pn 1 | |E |
lim
n  | p  p |r
 lim n 1r  A
n  | E |
for all x  p : f ( x)  ( x  p )m g ( x), where lim g ( x)  0.
n n x p
then the sequence { pn } is s aid to be convergent to p with the A zero of multiplicity m  1 is called a simple zero.
order of convergence equal r . The constant A is called an
asymptotic error.
If r  1, the convergence is called linear convergence.
If r  2, the convergence is called quadratic convergence.
Order of convergence is a real number.
The higher order of convergence, the faster converg ence.

Computational Physics 1, Hanoi Univ of Science 35/73 Computational Physics 1, Hanoi Univ of Science 36/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Multiple Zeros Order of convergence of Newton-Raphson

f ( x )   x  1  x  2 x  1 has double zeros
2 2

Suppose that the Newton-Raphson sequence {xn } converges to

(zero with muliplicity  2) at x  1.
f ( x)  x  1
2
the root p of the equation f ( x )  0. Then
a) If p is a simple zero, the order of convergence r  2 and
| f ''( p ) |
f ( x)  x3 A | f ''( p ) / (2 f '( p )) | . Thus, En 1  | E n |2 .
2 | f '( p ) |
b) If p is a zero of multiplicity m , the order of convergence r  1
m 1
and E n 1  | En | .
m

f ( x)  x3 has a zero with muliplicity  3 at x  0.

CISE301_Topic2 Computational Physics 1, Hanoi Univ of Science 37/73 Computational Physics 1, Hanoi Univ of Science 38/73

Improved Newton-Raphson method in

Example of the Newton-Raphson finding multiple root

Example. Given a real positive number a. Approximate a. Theorem

Solution. Using the Newton-Raphson method for the function If p is a zero of multiplicity m of the equation f ( x )  0.
f ( x )  x 2  a , we get the iterative sequence Then
f ( xk 1 ) 1  a  f ( x)
xk  g ( xk 1 )  xk 1    xk 1   g ( x)  x  m 
f '( xk 1 ) 2  xk 1  f '( x )
For a  5 and the initiation x0  2: f ( xk )
x1  2.25, x2  2.236111111, xk 1  g ( xk )  xk  m  ,
f '( xk )
x3 =2.236067978, x4 =2.236067978, ...
the sequence {xk } converges to p with the order of
Conclusion: 5  2.236068.
convergence r  2.

Computational Physics 1, Hanoi Univ of Science 39/73 Computational Physics 1, Hanoi Univ of Science 40/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Improved Newton-Raphson method: the Comment on the Newton-Raphson method

Halley method and its variants

Suppose that f in C 3 ([ a , b ]), the interval [ a , b] contains a root p , The Newton-Raphson method requires to compute values
(thus exists p in [ a , b] such f ( p )  0), and suppose f '( p )  0. f ( xk ) and f '( xk ) at each step of the iteration.
Then there exists  >0 such a sequence { xn } defined by For a simple function f , it is not a big problem, but for a
1
f ( x)  f ( x ) f ''( x )  complicated function, the computation of f '( xk ) at each step
g ( x)  x  1 ; xk 1  g ( xk );
f '( x )  2( f '( x )) 2  is too costly.
converges to p for every initial point in the  -neighborhood So it is natural to ask for other fast convergent methods, that
of p , i.e. x0  [ p   , p   ]. require to compute only the value of the function f .
Furthermore if p is a simple root, the order of convergence These methods do exist as we will see below.
of the sequence { xk } is equal 3.

Computational Physics 1, Hanoi Univ of Science 41/73 Computational Physics 1, Hanoi Univ of Science 42/73

The Secant method Secant method

The secant method is a root-finding algorithm that uses a The secant method is defined by the recurrence relation
succession of roots of secant lines to better approximate a
xn  xn1  f  xn1  
 xn 1  xn 2 
root of a function f . f  xn1   f  xn 2 
The secant method can be thought of as a finite difference Note that the secant method requires two initial values,
approximation of Newton-Raphson method.
x0 and x1 , which should ideally be chosen to lie close to
However, it was developed independently of the Newton-Raphson
the root.
method.

Computational Physics 1, Hanoi Univ of Science 43/73 Computational Physics 1, Hanoi Univ of Science 44/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Secant method Secant method

We use two stopping conditions in the Secant method. The iteration equation should not be simplified algebraically to
First, we assume that xn is sufficiently accurate when |xn  xn 1|
xn  xn 1  f  xn 1  
 xn1  xn2  =
f ( xn 1 ) xn 2  f ( xn  2 ) xn 1
is within a given tolerance. f  xn 1   f  xn  2  f ( xn 1 )  f ( xn 2 )
Second, a safeguard exit based upon a maximum number of Although this is algebraically equivalent to the iteration equation,
iterations is given in case the method fails to converge as it could increase the significance of rounding error if the nearly
expected. equal numbers f ( xn 1 ) xn 2 and f ( xn  2 ) xn 1 are subtracted.
The Secant method does not have the root-bracketing property of
the Bisection method. As a consequence, the method does not always
converge, but when it does converge, it generally does so much faster
than the Bisection method.

Computational Physics 1, Hanoi Univ of Science 45/73 Computational Physics 1, Hanoi Univ of Science 46/73

Theorem about the Secant method Brent method

Theorem Suppose that f in C 2 ([a, b]), the interval [a, b] contains a root p,
Suppose that f in C 2 ( a , b ), the interval  a , b  contains a and suppose f '( p)  0. Then there exists  >0 such a sequence {xn }
root p (thus exists p in  a , b  such f  p   0), and suppose defined by
xk 1  g ( xk 2 , xk 1 , xk )  xk 1 
f ’  p   0. Then there exists   0 such a sequence {xn }
f ( xk 2 )  f ( xk 1 )  f ( xk 2 ) ( xk 1  xk )  f ( xk )  f ( xk 1 )  f ( xk ) ( xk 2  xk 1 )
defined by
f ( xk )( xk  xk 1 )  f ( xk )  f ( xk 1) f ( xk )  f ( xk 2 ) f ( xk 1 )  f ( xk 2 )
xk 1  g ( xk 1 , xk )  xk 
f ( xk )  f ( xk 1 )
converges to p for every initial points in the   neighborhood of p,
converges to p for every initial points in the  – neighborhood
i.e. for all x0 , x1 , x2 [p   , p   ].
of p , i.e. for all x0 , x1  [p   , p   ].

Computational Physics 1, Hanoi Univ of Science 47/73 Computational Physics 1, Hanoi Univ of Science 48/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Regula Falsi method or False Position

False Position or Regula Falsi method method
There are other reasonable choices for generating a sequence of The Regula Falsi method is one of the bracketing methods for
approximations based on the intersection of an approximating line finding roots of equations.
and the x-axis. The method of False Position (or Regula Falsi) is Like the bisection method, the Regula Falsi method starts with two
a hybrid bisection-secant method that constructs approximating points a0 and b0 such that f  a0  and f  b0  are of opposite signs.
lines similar to those of the Secant method but always brackets the The method proceeds by producing a sequence of shrinking intervals
root in the manner of the Bisection method.  ak , bk  that all contain a root of f . At iteration number k , the
a f (bk )  bk f (ak )
number ck  k is computed.
f (bk )  f (ak )
If f (ak ) and f (ck ) have the same sign, then we set
ak 1  ck and bk 1  bk , otherwise we set bk 1  ck and ak 1  ak .

Computational Physics 1, Hanoi Univ of Science 49/73 Computational Physics 1, Hanoi Univ of Science 50/73

Regula Falsi method or False Position

method Regula Falsi method – C code

#include <stdio.h>
Theorem. Assume that f  C[a0 , b0 ] and p [a0 , b0 ] #include <math.h>
double f(double x)
is a root. If f (an ) and f (bn ) have opposite signs, and {
return x*x*x-3*cos(x)+2.8;
an f (bn )  bn f (an )
cn  }
f (bn )  f (an ) int main(void)
{
represents the sequence of points generated by the printf(“One root computed by the Regula Falsi method

Regula Falsi method, then the sequence cn  p= %0.10f\n", falsi(-2, -1, 5E-10, 100));
return 0;
converges to the root p. }

Computational Physics 1, Hanoi Univ of Science 51/73 Computational Physics 1, Hanoi Univ of Science 52/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Regula Falsi method – C code Solving a system of non-linear equations

double falsi(double s, double t, double e, int m)
{ Let F : R n  R n be a vector-valued function of a vector
int n;
double r,fr,fs = f(s),ft = f(t);  f1 ( X ) 
 
F ( X )  .......... 
for (n = 1; n <= m; n++){
r = (fs*t - ft*s) / (fs - ft);  f ( X )
 n 
if (fabs(t-s) < e*fabs(t+s)) break;
fr = f(r); We are interested in finding a solution of the system of n
if (fr * ft > 0) { t = r; ft = fr; } non-linear equations:
else if (fs * fr > 0) { s = r; fs = fr; }
else break; Given F , find a vector X such that F ( X )  0, where
}
return r; X  ( x1 ,..., x n )  R n , and F  ( f1 ,...., f n ) : R n  R n .
}

Computational Physics 1, Hanoi Univ of Science 53/73 Computational Physics 1, Hanoi Univ of Science 54/73

The Newton’s method for a system of non- The Newton’s method for a system of non-
linear equations linear equations
Use the Taylor expansion: So, the Newton-Raphson algorithm consists of the following
steps:
0  F ( X )  F ( X 0 )  DF ( X 0 ) * ( X  X 0 ) • Step 0: initialize the iteration counter (k=0) and provide
an initial value for vector x, i.e., x(0).
where X 0  ( x01 ,....x0n ), DF ( X 0 ) is Jacobian matrix at X 0 • Step 1: compute the Jacobian JF(x(0)).
 f1 f1  • Step 2: compute x(k+1).
 x1 ,...., x n  • Step 3: check every element of the absolute value of the
  difference between the values of vector x in two consecutive
J F ( X 0 )  DF ( X 0 )  ...................  , iterations is lower than a tolerance , i.e., check if || x(k+1) -
 f  f  x(k) ||< tol. If so, the algorithm has converged and the
 n ,...., nn  solution is x(k+1) If not, continue at Step 4.
  x1 x  x  X 0 • Step 4: update the iteration counter k=k+1 and continue
at Step 1.
   
1
Thus, X 1  X 0   J X 0  F X0 .
 

Computational Physics 1, Hanoi Univ of Science 55/73 Computational Physics 1, Hanoi Univ of Science 56/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

The Newton’s method for a system of non- The fixed-point iteration for system of non-
linear equations – An example linear equations
Solve approximately the system of non-linear equations:
Fixed-point method
x3  y  1, y 3  x  2,
near (1,1) by the Newton method. 0  F ( X )  X  G ( X ).
Sol. Here Choose: X 0 .
f ( x, y )  x 3  y  1  0,
Recursion: X k 1  G ( X k ).
g ( x, y )  y 3  x  2  0.
The Jacobi matrix: Newton's method
J  [3 x ,1; 1,3 y ].
2 2
0  F ( X ).
We will use the matrix J for each iteration.
Choose: X 0 ,
For the first iteration,we need to solve the following system
J [ x, y ]  [ f (1,1), g (1,1)]; X k 1  g ( X k )  X k  [J ( X k )]1 * F ( X k ).
Computational Physics 1, Hanoi Univ of Science 57/73 Computational Physics 1, Hanoi Univ of Science 58/73

Convergence of the fixed-point iteration Minimization and Maximization

 Minimization and maximization problems
Theo rem. Suppose that g i và g i' are continuous in the All the root-finding methods can be used to find
neighborhood of a fixed point (u1 ,...., u n ), and there a stationary, minimum or maximum of a
function. The derivative, if it exists, must be
exists a constant  that 0< <1 and for all i  1, 2,...., n zero at a minimum or maximum, so the minima
and maxima of a function can be found by
n
g i ( x1 ,..., xn )
 x j
, applying one of the discussed above methods to
j 1
( x1 ,..., xn )  ( u1 ,..., un )
find zero of its derivative.
then the fixed-point iteration converges for every initial
 Next, I’ll introduce you to another (famous)
point in some neighborhood of the fixed point.
algorithm for finding minimum (maximum).

Computational Physics 1, Hanoi Univ of Science 59/73 Computational Physics 1, Hanoi Univ of Science 60/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Steepest Descent method Steepest Descent method

The method of Steepest Descent for finding a Given a function g : Rn to R that is differentiable
local minimum for an arbitrary function g from at x0, the direction of steepest descent is the
Rn into R can be intuitively described as follows: vector – grad g(x0).
– Evaluate g at an initial approximation p(0) = To see this, consider the function
(p1(0) , …., pn(0))t. h(t)=g(x0+tu); where u is a unit vector; that is,
– Determine a direction from p(0) that results in ||u|| = 1. Then, by the Chain Rule, h’(t)=grad
a decrease in the value of g. g(x0+tu)·u; and therefore h’(0)=grad g(x0)·u =
||grad g(x0)|| cos θ;
– Move an appropriate amount in this direction
where θ is the angle between grad g(x0) and u.
and call the new value p(1) .
It follows that h’(0) is minimized when θ = π,
– Repeat the steps with p(0) replaced by p(1). which yields: u = - grad g(x0)/||grad g(x0)||.

Computational Physics 1, Hanoi Univ of Science 61/73 Computational Physics 1, Hanoi Univ of Science 62/73

Steepest Descent method Steepest Descent method

We can therefore reduce the problem of Summation of the Method of Steepest Descent:
minimizing a function of several variables to a given an initial guess x0, the method computes
single variable minimization problem, by finding a sequence of iterates {xk}, where
the minimum of h(t) for this choice of u. That xk+1 = xk - tk grad g(xk); k = 0,1,2,3,…
is, we find the value of t, for t > 0, that where tk > 0 minimizes the function
minimizes h0(t)=g(x0-t grad g(x0)). hk(t) = g(xk – t grad g(xk)).
After finding the minimizer t0, we can set
x1 = x0 - t0 grad g(x0) and continue the
process, by searching from x1 in the direction of
– grad g(x1) to obtain x2 by
minimizing h1(t) = g(x1-t grad g(x1)), and so
on.
Computational Physics 1, Hanoi Univ of Science 63/73 Computational Physics 1, Hanoi Univ of Science 64/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Steepest Descent method Steepest Descent method

Example: We apply the Method of Steepest We therefore set
Descent to the function g(x,y) = 4x2 - 4xy + x1 = x0 – ½ grad g(x0)=(2,3) – ½(4, 4)=(0,1).
2y2 with initial point x0 = (2,3).
Continuing the process we get
We first compute the steepest descent direction
grad g(x1) = grad g(0,1) = (-4, 4).
from grad g(x,y) = (8x - 4y; 4y - 4x)
We then minimize the function
to obtain grad g(x0) = grad g(2,3) = (4, 4).
h(t) = g((0,1)-t(-4, 4)) = g(4t,1-4t)
We then minimize the function
by computing h’(t) = -grad g(4t,1-4t)·(-4, 4) =-
h(t) = g((2,3)-t(4, 4)) = g(2-4t,3-4t)
(8*4t-4(1-4t),4(1-4t)-4*4t).(-4,4)=-[(48t-4)*(-
by computing h’(t) = -grad g(2-4t,3-4t)·(4, 4)
4)+(4-32t)*(4)]=320t-32.
=64t-32.
This function has a strict global minimum when
This function has a strict global minimum when
h’(t)=320t-32=0, or t = 1/10, as can be seen
h’(t)=64t-32=0, or t = 1/2, as can be seen by
by noting that h’’(t) = 320 > 0.
noting that h’’(t) = 64 > 0.
Computational Physics 1, Hanoi Univ of Science 65/73 Computational Physics 1, Hanoi Univ of Science 66/73

Steepest Descent method Steepest Descent method

We therefore set The Method of Steepest Descent is guaranteed to
x2 = x1 – 1/10 grad g(x1)=(0,1) – 1/10(-4, make at least some progress toward a
4)=(2/5,3/5). minimizer x* during each iteration. This
Repeating this process yields x3 = (0, 2/10 ). theorem can be proven by showing that h’(0) <
0, which guarantees the existence of t > 0 such
We can see that the Method of Steepest Descent
that h(t) < h(0).
produces a sequence of iterates xk that is
Theorem Let g: Rn -> R be continuously
converging to the strict global minimizer
differentiable on Rn, and let xk and xk+1, for k ≥
of f(x, y) at x* = (0, 0). 0, be two consecutive iterates produced by the
Method of Steepest Descent. Then the steepest
descent directions from xk and xk+1 are
orthogonal; that is, grad g(xk) · grad g(xk+1) =
0.
Computational Physics 1, Hanoi Univ of Science 67/73 Computational Physics 1, Hanoi Univ of Science 68/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

Steepest Descent method Exercises

This theorem can be proven by noting that xk+1 is
obtained by finding a critical point t* of  Write a simple C/Matlab program implementing
h(t) = g(xk – t grad g(xk)), and therefore the secant method.
h’(t*) = - grad g(xk+1) · g(xk) = 0.  Let f(x) = −x3 − cos x, p0=-1, and p1= 0.
That is, the Method of Steepest Descent Find p3 using
pursues completely independent search
(a) Secant method
directions from one iteration to the next.
However, in some cases this causes the method (b) Regula Falsi method
to “zig-zag” from the initial iterate x0 to the  Write a simple C/Matlab program implementing
minimizer x*. the Newton’s method to solve a system of two
equations.

Computational Physics 1, Hanoi Univ of Science 69/73 Computational Physics 1, Hanoi Univ of Science 70/73

Exercises Exercises

The function f(x)=tanh(x) has a root at x = 0. Using one of the methods to find solutions for
the following system of non-linear equations:
 Write a program to show that the Newton-
Raphson method does not converge for an a) x4+sin(x2+y2)=5,
initial guess of x>1. y4+cos(x2-y2)=1.
 Can you understand what's going wrong by
considering a graph of tanh(x) ? b) 3x1 − cos(x2x3) −1/2=0,
 Derive the critical value of the initial guess x12− 81(x2 + 0.1)2 + sin x3 + 1.06 = 0,
above which convergence will not occur.
exp(-x1x2) + 20 x3 + (10 pi/3 − 1) = 0.
 Try to solve the problem using the secant
method.

Computational Physics 1, Hanoi Univ of Science 71/73 Computational Physics 1, Hanoi Univ of Science 72/73
Prof. Dr Sonnet Nguyen HUS, VNU
[email protected] Computational Physics 1

What will be next week?

 That’s enough for today. If you do not have
any question, then we can finish here.
 See you next week in the lecture of numerical
linear algebra.

Thank you!

Computational Physics 1, Hanoi Univ of Science 73/73