Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

PHY6CBT0119 - Computational Physics

Module I - Solution of Non-Linear Equations

Dr. Lijo T. George

St. Albert’s College (Autonomous)

1
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Table of Contents
1. Types of equations

2. Bisection Method

3. Regula-Falsi Method

4. Iteration Method

5. Newton-Raphsons Method

6. Secant Method

7. Systems of equations

8. Convergence of Methods

2
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Types of equations

• In many scientific and engineering problems, we are required to solve an

equation of the form
f(x) = 0
where f(x) is some function
• Solving an equation means finding the roots of the function f(x) i.e. find
the x where f(x) = 0
• f(x) can take many forms
☞ algebraic e.g. quadratic, cubic, polynomial (f(x) = a0 + a1 x1 + a2 x2 . . . an xn )
☞ transcendental e.g. containing functions like ex , log x, sin x etc.
• Solving algebraic equations for small value of n might be possible (direct
methods) but for larger values of n and for transcendental equations,
analytical solutions are not always possible. In those cases we need to use
numerical approximations to solve the function f(x). (These solutions can
be real or complex)

3
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Indirect/Iterative Methods

Theorem (Intermediate Value Theorem)

If f(x) is continuous in a ⩽ x ⩽ b, and if f(a) and f(b) are of opposite signs, then
there exists a ξ where a < ξ < b, such that f(ξ) = 0

4
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Bisection Method

• This method uses the above theorem to find at least one root between a and
b.
• The algorithm:
Given an f(x) = 0
1. find an a and b, such that f(a)f(b) < 0
2. set xr = a+b2
3. 3.1. if f(a)f(xr ) < 0, the root lies in (a, xr ). Set b = xr and repeat step 2.
3.2. if f(a)f(xr ) > 0, the root lies in (xr , b). Set a = xr and repeat step 2.
3.3. if f(a)f(xr ) = 0, then xr is a root of the equation f(x) = 0. The process
can be terminated.
• In most cases, the exact solution will not be found so we need to put a
condition to stop the algorithm.
• Define ε = xr+1xr−xr as the error (can also be expressed as %) and stop the
algorithm when ε < εp where εp is a predetermined tolerance level.

5
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Bisection Method
• Graphical representation

• This method always converges and will be able to find one of the roots in
the interval.
• This method is very simplistic and takes a long time to converge.
6
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Regula-Falsi (False position) method

• also uses IVT

• instead of the midpoint we look at the equation of the chord joining a and
b given by
y − f(a) f(b) − f(a)
=
x−a b−a
or at y = 0,

f(a)
xr = a − .(b − a)
f(b) − f(a)
af(b) − bf(a)
=
f(b) − f(a)

7
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Regula-Falsi (False position) method

The algorithm:
Given an f(x) = 0
1. find an a and b, such that f(a)f(b) < 0
af(b)−bf(a)
2. set xr = f(b)−f(a)

3. 3.1. if f(a)f(xr ) < 0, the root lies in (a, xr ). Set b = xr and repeat step 2.
3.2. if f(a)f(xr ) > 0, the root lies in (xr , b). Set a = xr and repeat step 2.
3.3. if f(a)f(xr ) = 0, then xr is a root of the equation f(x) = 0. The process can
be terminated.
4. Repeat the process till desired accuracy εp is reached

8
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Regula-Falsi (False position) method

Graphical representation

9
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Problems
1. x4 − x − 10 = 0
2. x = exp x1

3. x3 + x2 + x + 7 = 0

10
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Iteration Method
• To find the root of f(x) = 0, rewrite it into a form x = ϕ(x)
3
e.g. f(x)
q = x +√ x2 − 2 = 0 can be rewritten as
2
x = 1+x , x = 2 − x3 , x = (2 − x2 )1/3 etc.

• Let x0 be the initial guess for the root.

Then,

x1 = ϕ(x0 )
x2 = ϕ(x1 )
..
.
xn+1 = ϕ(xn )

• It is not a guarantee that this sequence will converge, but if it does

converge to some ξ, then ξ is the root of the equation x = ϕ(x) and hence
also of f(x) = 0

11
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Iteration Method

Proof that ξ is a root

Let xn+1 = ϕ(xn )
As n → ∞, xn+1 → ξ (convergence)
and if ϕ(x) is a continuous function, ϕ(xn ) → ϕ(ξ)
i.e. in this limit, ξ = ϕ(ξ)
or, ξ is a root of the equation x = ϕ(x)

12
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Iteration Method
Proof of convergence
ξ = ϕ(ξ)
x1 = ϕ(x0 )
∴ ξ − x1 = ϕ(ξ) − ϕ(x0 )

From the mean value theorem,

ϕ(ξ) − ϕ(x0 )
ϕ ′ (ξ0 ) = , x0 < ξ0 < ξ
(ξ − x0 )
=⇒ ξ − x1 = (ξ − x0 ) ϕ ′ (ξ0 ), x0 < ξ0 < ξ

Similarly,

ξ − x2 = (ξ − x1 ) ϕ ′ (ξ1 ), x1 < ξ1 < ξ

..
.
ξ − xn+1 = (ξ − xn ) ϕ ′ (ξn ), xn < ξn < ξ

13
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Iteration Method
Assume |ϕ ′ (ξi )| ⩽ k, ∀i
Then,

|ξ − x1 | ⩽ k |ξ − x0 |
..
.
|ξ − xn+1 | ⩽ k |ξ − xn |

or,
|ξ − xn+1 | ⩽ kn+1 |ξ − x0 |
If k < 1, then as n → ∞, RHS → 0, =⇒ xn+1 → ξ
=⇒ ϕ(x) must be such that |ϕ′ (x)| < 1

Conditions on ϕ(x)
• ϕ(x) is continuous
• |ϕ ′ (x)| < 1

14
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Iteration Method

This root is unique

Let ξ1 and ξ2 be two roots of x = ϕ(x)
Then, ξ1 = ϕ(ξ1 ) and ξ2 = ϕ(ξ2 )

|ξ1 − ξ2 | = |ϕ(ξ1 ) − ϕ(ξ2 )|

= |ξ1 − ξ2 | ϕ ′ (η), ξ1 < η < ξ2 [mean value theorem]
=⇒ |ξ1 − ξ2 |(1 − ϕ ′ (η)) = 0

But, ϕ ′ (η) < 1

=⇒ |ξ1 − ξ2 | = 0
=⇒ ξ1 = ξ2

15
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Iteration Method
Error in the root
|ξ − xn | ⩽ k |ξ − xn−1 |
= k |ξ − xn + xn − xn−1 |
⩽ k (|ξ − xn | + |xn − xn−1 |)
k
or, |ξ − xn | ⩽ |xn − xn−1 |
1−k
k
⩽ .kn−1 |x1 − x0 |
1−k
kn
= |x1 − x0 |
1−k
=⇒ convergence will be faster for smaller k
k is also the maximum value of ϕ ′ (x) in the interval
Let ε be the accuracy defined as |ξ − xn | ⩽ ε
1−k
∴ |xn − xn−1 | ⩽ ε
k

16
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Iteration Method

Problems
1. x3 + x2 − 1 = 0 in [0, 1] upto ε = 10−4
2. 2x − 3 = cos x in [ 21 , π2 ]
3. xex = 1 in [0, 1]

17
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Newton-Raphson’s Method
Let x0 be an approximate root for the equation f(x) with x1 = x0 + h being
the true root i.e. f(x1 ) = 0.
Expanding f(x0 + h) using Taylor series expansion,
h2 ′′ h3
f(x0 + h) = f(x0 ) + hf ′ (x0 ) + f (x0 ) + f ′′′ (x0 ) + . . . = 0
2! 3!
Neglecting the higher order terms (second and above),
f(x0 )
h=−
f ′ (x0 )
f(x0 )
∴ x0 + h = x1 = x0 −
f ′ (x0 )
Thus,

f(xn )
xn+1 = xn − n = 0, 1, 2 . . .
f ′ (xn )
18
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Newton-Raphson’s Method
Proof of Convergence
Comparing NR method to Iteration method

xn+1 = ϕ(xn )
f(x)
ϕ(x) = x −
f ′ (x)

which gives

f(x)f ′′ (x)
ϕ ′ (x) =
|f ′ (x)|2

Assuming f(x), f′ (x), f′′ (x) are continuous, NR will converge if

|f(x)f ′′ (x)| ⩽ |f ′ (x)|2

If we can find a small region around the root ξ where this holds then the
method will converge.
19
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Newton-Raphson’s Method
Graphical Representation
The derivative of a function f ′ (x) represents the tangent of the function (f(x))
at any point xi . Thus the tangent at any point is used to get the next step in
the iteration

This method, if it converges, does so faster than any of the previous methods
discussed.
20
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Newton-Raphson’s Method

Problems
1. x3 − 2x − 5 = 0
2. 4e−x sin x − 1 = 0
√
3. Find 15 using Newton-Raphson’s Method
4. Find N1/k using NR method

21
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Secant Method
The derivative of a function f ′ (x) cannot always be estimated a-priori so an
approximation can be used instead
Let
f(xi ) − f(xi−1 )
f ′ (xi ) =
xi − xi−1

which can be written as

fi − fi−1
fi′ =
xi − xi−1

Hence the NR formula becomes

xi − xi−1
xi+1 = xi − fi .
fi − fi−1
xi−1 fi − xi fi−1
∴ xi+1 =
fi − fi−1

22
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Secant Method
Note that this method requires two initial approximations

Graphical Representation

Difference between Regula-Falsi and Secant method

Both methods require two initial values but after every step of Regula-Falsi
you need to check the condition again (f(a) × f(b) < 0) whereas in Secant
you take the next value and proceed with the algorithm.
23
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Secant Method

Problems
1. x3 − 2x − 5 = 0
2. 4e−x sin x − 1 = 0
3. xex = 1

24
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

NR Method for two equations

Consider a system of two equations
f(x, y) = 0
g(x, y) = 0

Let (x0 , y0 ) be the initial approximations,

and (x0 + h, y0 + k) are the true solutions, such that
f(x0 + h, y0 + k) = 0
g(x0 + h, y0 + k) = 0

Using two-variable Taylor’s expansion (and ignoring higher order

derivatives),
∂f ∂f
f0 + h +k =0
∂x0 ∂y0
∂g ∂g
g0 + h +k =0
∂x0 ∂y0
∂f ∂f
here, f0 = f(x0 , y0 ), ∂x 0
= [ ∂x ]x=x0 ...
25
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

NR Method for two equations

∂f ∂f
h +k = −f0
∂x0 ∂y0
∂g ∂g
h +k = −g0
∂x0 ∂y0
Let,
∂f ∂f
∂x0 ∂y0
D= ∂g ∂g
∂x0 ∂y0

If the determinant is non-zero, there exists a unique solution to the system of

equations
Then, using Cramer’s Rule
∂f
1 −f0 ∂y0
h= ∂g
D −g0 ∂y0
∂f
1 ∂x0 −f0
k= ∂g
D ∂x0 −g0
26
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

NR Method for two equations

Then the next set of iterations,

x1 = x0 + h
y 1 = y0 + k

Problems
• f(x, y) = 3yx2 − 10x + 7
g(x, y) = y2 − 5y + 4
• sin x − y = −0.9793
cos y − x = −0.6703

27
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Rate of Convergence

Let the error at the kth iterate be defined as εk = xk − ξ, k = 0, 1, 2, . . .

Definition
An iterative method is said to be of order p or has the rate of convergence p, if p is
the largest positive real number for which there exists a finite constant
C ̸= 0, such that
εk+1 ⩽ C|εk |p
The constant C, which is independent of k, is called the asymptotic error
constant and it depends on the derivatives of f(x) at x = ξ

28
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Bisection Method

• Starting from the initial guesses of x1 and x2 , in each step the interval
containing the root is reduced by a factor of 2
x2 −x1 ∆x
• When repeated n times the interval reduces to 2n = 2n

• After n iterations the root lies within ± ∆x

2n of our estimate

2n |
• ∴ Error at the nth step, εn = | ∆x
Similarly, εn+1 = | 2∆x
n+1 | = 2
εn

• That is, error decreases linearly by a factor of 0.5. In other words, bisection
method is linearly convergent.

29
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Newton-Raphson’s Method

The sequence xn converges to ξ as n → ∞

Note that f(ξ) = 0 and use Taylor’s expansion around xn
1
f(xn ) + (ξ − xn )f ′ (xn ) + (ξ − xn )2 f ′′ (xn ) + . . . = 0
2
′′
f(xn ) 1 2 f (xn )
=⇒ − = (ξ − xn ) + (ξ − xn )
f ′ (xn ) 2 f ′ (xn )
1 f ′′ (xn ) f(xn )
=⇒ xn+1 − ξ = (ξ − xn )2 ′ [∵ xn+1 = xn − ]
2 f (xn ) f ′ (xn )
Let εn = xn − ξ,
1 f ′′ (ξ)
=⇒ εn+1 = ε2n ′
2 f (ξ)

∴ Newton-Raphson’s has a quadratic or second-order convergence.

30
Types of equations Bisection Method Regula-Falsi Method Iteration Method Newton-Raphsons Method Secant Method Systems of equations Convergence

Other methods

• The Regula-Falsi Method is also linearly convergent.

• The convergence of Iteration Method ranges from linear to quadratic.
• It can be shown that the order of convergence of the Secant Method is 1.618
(superlinear convergence)

31