MATHEMATICS FOR ENGINEERING 3

IBENG-2

1
Course map

• Introduction & motivation

• Non-linear equations
• Objective and method
• Examples
• Definitions
• Bisection
• General concept
• Algorithm
• Example
• Particularity
• Fixed point method
• Definitions
• General concept
• Algorithm
• Example
• Particularity
• Newton-Raphson
• General concept
• Advantages and drawbacks
• Algorithm
• Example
• Multiple variable case

2
Introduction and motivation

Non-linear problems

• What is a non-linear equation ?

• Real life problems are not always linear. Non-linear problems are much trickier to solve.

Linear equation Non-linear equation Non-linear equation

y y y

x x x
𝑓 𝑥 = 𝑎. 𝑥 + 𝑏
Examples :
𝑏 ; 𝑖𝑓 𝑥 ∈ −∞; 0
- High degree polynom 𝑓 𝑥 =ቊ
- Trigonometric functions 𝑎. 𝑥 + 𝑏 ; 𝑖𝑓 𝑥 ∈ 0; +∞
- Gaussian function
-…
3
Introduction and motivation

Objective and method

Let an interval 𝐼=[𝑎, 𝑏]⊆ ℝ and an application 𝑓 defined on this interval with real values. If it exists, we want to
determine the position of a real number 𝑥ҧ ∈ 𝐼 as:
𝑓 𝑥ҧ = 0
y

𝑦=𝑓 𝑥

𝑥ҧ
a b x

The non-linearity comes with a major difficulty: you do not know in advance if there are zero, one, many or an infinity of
solutions.

4
Introduction and motivation

Examples of problem

Example 1:
• We want to calculate the average annuity rate "𝐼" of an investment fund over several years. We have invested in the
fund 𝑣 = 1000 € each year and we find ourselves after 5 years with an amount 𝑀 = 6000 €.
• We know that the relationship between 𝑀, 𝑣, 𝐼 and the number of years "𝑛" is:
𝑛
1+𝐼
𝑀 = 𝜈෍ 1+𝐼 𝑘 = 𝜈 1+𝐼 𝑛−1
𝐼
𝑘=1
The problem is to find “𝐼” as:
1+𝐼
𝑓 𝐼 =𝑀−𝜈 1+𝐼 𝑛−1 = 0
𝐼
This lead to solve a non-linear equation which we are not capable of finding an exact solution.
Example 2:
• Let’s consider a spring with a non-linear stiffness (duffing equation), the equilibrium of the system is as follow:
𝐹 = 𝛼. 𝑥 + 𝛽𝑥 3
• We want to know the displacement of the spring 𝒙 ഥ according to a certain force 𝐹 F
• Equation to solve : f x = 𝛼. 𝑥 + 𝛽𝑥 3 − 𝐹 = 0
x
𝑘 𝑥

5
Introduction and motivation

Definitions

The method used to solve this kind of problems is an iterative process which is going to calculate an approximation of
the solution at each step in order to « crawl » toward the solution.
The iterative process consists in:
• Building of a sequence 𝑥 (𝑘) , with 𝑘 ∈ ℕ which converges to 𝑥ҧ as lim 𝑥 (𝑘) = 𝑥ҧ
𝑘→∞
• Choosing an initial value 𝑥 (0) that should be close enough to the solution

Convergence
We define the absolute error at step 𝑛 by
𝑘 𝑘
𝑒 = 𝑥 − 𝑥ҧ , 𝑘 ≥ 0
A sequence 𝑥 (𝑘) built with a numerical method converges toward 𝑥ҧ with an order of convergence 𝑝 if
𝑝
∃𝐶 > 0 ∶ 𝑒 (𝑘+1) ≤ 𝐶 𝑒 𝑘 , ∀𝑘 ≥ 0
𝐶 is the convergence factor of the method.
Order of convergence
• If 𝑝 = 1 and 𝐶 < 1, the convergence is linear
• If 𝑝 = 2, the convergence is quadratic

6
Introduction and motivation

Definitions

Types of iterative methods:

• Bracketing methods
• Fixed point methods
Stopping criteria:
• Residue
Optimal for 𝑓′ ≈ 1 close to the solution 𝑥.ҧ Too restrictive for 𝑓′ ≫ 1 and not enough for 𝑓′ ≪ 1
𝑘
𝑓 𝑥 ≤ 𝜀, with 𝜀 the tolerance

• Increment
Optimal criteria most of the time
𝑥 (𝑘+1) − 𝑥 𝑘
≤𝜀

• Number of iterations :
𝑘 ≤ 𝑘𝑚𝑎𝑥

7
 Bisection
• General concept
• Algorithm
• Example
• Particularity

8
8
Bisection

General concept

Let 𝑓 a continuous application of 𝐼 = 𝑎, 𝑏 in ℝ. If 𝑓 𝑎 . 𝑓 𝑏 < 0 then there is a 𝑥ҧ ∈]𝑎, 𝑏[ such as 𝑓 𝑥ҧ = 0

𝑘
The concept is to create an interval containing the solution and divide it in two with the midpoint 𝑥 .
𝒌 𝒌 𝒌 𝒌 𝒌 𝒌
Only three possibilities: 𝒇 𝒂 .𝒇 𝒙 > 𝟎 or 𝒇 𝒂 .𝒇 𝒙 < 𝟎 or 𝒇 𝒂 .𝒇 𝒙 = 𝟎 (very unlikely).
The method selects the subinterval that is guaranteed to bracket the solution as the for the next step.
The process is continued until the interval is sufficiently small and the convergence is linear.
y
𝑦=𝑓 𝑥

0
1
𝑓 𝑎

1
𝑓 𝑥
𝑎 (1) 𝑥ҧ 𝑥 2 𝑏 21 𝑏 0

𝑎(0) 𝑎(1)
𝑥 2 𝑥 (0) x
1
0 0
𝑓 𝑏
𝑥 𝑓 𝑏

9
Bisection

Algorithm

0 0
• Set the initial values of the interval 𝑎 ,𝑏
• Compute the mean value of this interval
𝑎 0 +𝑏 0
𝑥 (0) =
2
• Compute the values of the function at one boundary of the interval and
at the mean value : 𝑓 𝑎 0 , 𝑓 𝑥 0

• If the sign of 𝑓 𝑎 0 and 𝑓 𝑥 0 is different ⇒ the solution is in the

interval 𝑎 0 , 𝑥 0 : 𝑏 (1) = 𝑥 0
Else, the solution is in 𝑥 0 , 𝑏 0 : 𝑎(1) = 𝑥 0

• Check if one of the criteria is matched :

• Increment or residue
• Number of iterations

• If none of the criteria is matched, repeat the process with the new
boundaries

10
Bisection

Example

We want to find the zeros of the non-linear function

𝑓 𝑥 = 𝑥 − 3 ∗ tanh 𝑥 + 1, ∀𝑥 ∈ ℝ+ 0
This function admit 3 zeros (𝑥1 , 𝑥2 , 𝑥3 )
The first step is to define the starting boundaries 𝑎 0 and 𝑏 0
Then apply the algorithm
𝑓 𝑥 Representation of the function

𝑥1 𝑥2 𝑥3
𝑥 11
Bisection

Particularity

Convergence of bissection : If f has a unique zero 𝑥ҧ in the interval 𝑎, 𝑏 then the sequence 𝑥 𝑘 𝑘∈ℕ converge to 𝑥ҧ and
we have
𝑏−𝑎
∀𝑘 ∈ ℕ, 𝑥 𝑘 − 𝑥ҧ ≤ 𝑘+1
2
Globally convergent iterative method
Predictable number of iterations for a given precision on the solution
Few hypotheses are required for this method
Robust method but with slow convergence (linear convergence)
Often used for the initialization of faster (but less robust) methods
If 𝑓 has several zeros in 𝑎, 𝑏 , the algorithm converges toward one of them depending on the conditions

12
 Fixed point method
• Definitions
• General concept
• Algorithm
• Example
• Particularity

13
Fixed point method

Definitions

• Fixed point
A fixed point is a point where we have an equation of the form of 𝑓 𝑥 = 𝑥 which means that it is the intersection of
the linear function 𝑦 = 𝑥 and 𝑦 = 𝑓 𝑥 .
Not every function has a fixed point (ex: 𝑦 = 𝑥 2 + 1)

14
Fixed point method

Definitions

• Contracting function
Let 𝐼 = 𝑎, 𝑏 ∈ ℝ and 𝑔 a function in 𝐼.
𝑔 is said to be (K-) contracting if a constant K exists which verifies

∀ 𝑥, 𝑦 ∈ 𝑎, 𝑏 , 𝑔 𝑥 − 𝑔 𝑦 ≤𝐾 𝑥−𝑦

A sufficient condition for a class 𝐶 1 function to be contracting is

∀𝑥 ∈ 𝑎, 𝑏 , 𝑔′ 𝑥 ≤𝐾<1

• General concept
The fixed point method for solving non linear equations 𝑓 𝑥 = 0 is to transform this function into an equivalent
function 𝑔 𝑥 = 𝑥. Function 𝑔 𝑥 has the following propriety : 𝑔 𝑥ҧ = 𝑥ҧ if and only if 𝑓 𝑥ҧ = 0
𝑥ҧ is said to be a fixed point of 𝑔(𝑥). Approaching the zeros of 𝑓 𝑥 is equivalent to finding the fixed points of 𝑔 𝑥 .

15
Fixed point method

• Convergence process
𝑘+1 𝑘
Build a sequence such as 𝑥 =𝑔 𝑥 , ∀𝑘 ∈ ℕ
𝑘
Because if lim 𝑥 = 𝑥ҧ and 𝑔 𝑥 is continuous on interval 𝑎, 𝑏 , then 𝑔 𝑥ҧ = 𝑥.ҧ
𝑘→∞
A necessary (but not sufficent) condition for convergence is
∀𝑥ҧ ∈ 𝑎, 𝑏 , 𝑔′ 𝑥ҧ < 1
• Examples of convergence

16
Fixed point method

Algorithm

Start
• Preliminary step
Set 𝑥 0
Find a function 𝑔 𝑥 who satisfy 𝑔 𝑥ҧ = 𝑥ҧ with 𝑓 𝑥ҧ = 0
0
Choose a starting point 𝑥 close enough to the solution

• Algorithm
• Compute the value of 𝑥 𝑘+1
with the function 𝑔 𝑥 𝑥 (𝑘+1) = 𝑔 𝑥 𝑘
𝑥 (𝑘) = 𝑥 𝑘+1

• Check if one of the criteria is matched :

• Increment or residue Stopping criteria no
• Number of iterations satisfied ?

• If none of the criteria is matched, repeat the process yes

with the new value of 𝑥 𝑘
End
𝑘+1
Solution is 𝑥

17
Fixed point method

Example 2

We want to find the zeros of the non-linear function

𝑓 𝑥 = ln 𝑥 − 𝑥 2 + 2, ∀𝑥 ∈ ℝ+ 0
This function admit 2 zeros (𝑥1 and 𝑥2 )
The first step is to transform this function as an equivalent function
𝑔 𝑥 =𝑥
2 solutions possible :
2 −2
• 𝑔1 𝑥 = 𝑒 𝑥 Representation of the function
• 𝑔2 𝑥 = ln 𝑥 + 2
𝑓 𝑥

𝑥1 𝑥2

18
 Fixed point method

Example

2 −2 2 −2 1
• 𝑔1 𝑥 = 𝑒 𝑥 , 𝑔1′ 𝑥 = 2𝑥. 𝑒 𝑥 • 𝑔2 𝑥 = ln 𝑥 + 2, 𝑔2′ 𝑥 =
2𝑥 ln 𝑥 +2
𝑘
• 𝑔1 𝑥 converge to 𝑥1 𝑘
• 𝑔2 𝑥 converge to 𝑥2

Contracting interval
Contracting interval
𝑥1 k x^(k) 𝑥2
𝑥1 k x^(k) 𝑥2
0 1
0 1
1 0.36787944
𝑔1 𝑥

𝑔2 𝑥
1 1.41421356
2 0.15494815
2 1.53185299
3 0.13862386
3 1.55771567
… …
… …
11 0.13793483
19 1.56446226
𝑔1′ 𝑥

𝑥 𝑔2′ 𝑥
𝑥 19
Fixed point method

• General method to approach the zeros of a function

• Few conditions needed to define the function 𝑔(𝑥)
• Convergence order at least linear

• Question: Is it possible to have a systematic method to define the function 𝑔 𝑥 such as

𝑔 𝑥ҧ = 𝑥ҧ ⟺ 𝑓 𝑥ҧ = 0 ?

Several methods exist:

• Secant method
• Newton-Raphson method
• Steffensen method
• …

20
 Newton-Raphson method

• General concept
• Advantages and drawbacks
• Algorithm
• Example
• Multiple variable case

21
Newton-Raphson method

General concept

• The Newton-Raphson method is a fixed point method

• Local convergence:
• 𝑓 𝑥 , class 𝐶 2 function close to the solution
0
• 𝑥 , close to one of the zeros
𝑘
If these requirements are matched, the sequence 𝑥 𝑘∈ℕ
converges to 𝑥.ҧ
• Global convergence:
• 𝑓 𝑥 , class 𝐶 2 function on the interval 𝐼 = 𝑎, 𝑏
• 𝑓 𝑎 .𝑓 𝑏 < 0
• ∀𝑥 ∈ 𝐼, 𝑓 ′ 𝑥 ≠ 0
• ∀𝑥 ∈ 𝐼, 𝑓 ′′ 𝑥 ≠ 0
If these requirements are matched, for every 𝑥 (0) ∈ 𝐼 such as 𝑓 𝑥 0 . 𝑓 ′′ 𝑥 𝑘
> 0, the sequence
𝑥 𝑘 𝑘∈ℕ converge to 𝑥,ҧ unique solution of 𝑓 𝑥 = 0 in the interval 𝐼 = 𝑎, 𝑏
• Remarks:
• Most of the time, the convergence is exclusively local.
• The initial value 𝑥 (0) can be obtained by bissection (for example).
Newton –Raphson method

• Formula
𝑘
Linearization of the function at the point 𝑥 by Taylor-Young formula:
𝑘 𝑘 2
𝑓 𝑥 =𝑓 𝑥 + 𝑥−𝑥 𝑓′ 𝑥 𝑘
+𝑂 𝑥−𝑥 𝑘

𝑘+1
By solving this equation for 𝑓 𝑥 = 0 we get:
𝑘+1 𝑘
𝑓 𝑥𝑘
𝑥 =𝑥 − =𝑔 𝑥
𝑓′ 𝑥 𝑘
• Principle
𝑘+1 𝑘 𝑘
The point 𝑥 is calculated by finding the intersection between the tangent of the function 𝑓 at (𝑥 ,𝑓 𝑥 ) and
the 𝑥 axis.
y

3 ഥ
𝒙
𝑥
2 0
𝑥 1 𝑥 𝑥 x
23
Newton –Raphson method

Let 𝑥ҧ a zero of the function 𝑓 𝑥 such as 𝑓 𝑥ҧ = 0. For 𝑓 ′ 𝑥 ≠ 0, we notice that 𝑔′ 𝑥ҧ = 0 because

𝑓 ′ 𝑥 2 − 𝑓 𝑥 . 𝑓 ′′ 𝑥
𝑔′ 𝑥 = 1 −
𝑓′ 𝑥 2
With Newton-Raphson method, we get rid of the condition
𝑔′(𝑥) ≤ 1, ∀𝑥 ∈ 𝐼
The convergence of this method is quadratic
Some cases are problematic for the convergence :
If the derivative of the function at the point 𝑥 𝑘 is null
𝑓′ 𝑥 𝑘 = 0
𝑘+1
If the point 𝑥 is outside of the definition interval of the function
y

1 0 x
𝑥 𝑥 24
Newton –Raphson method

Advantages and drawbacks

• Advantages
• If the sequence converges, the convergence is really quick
• Only one starting point is needed

• Drawbacks
• Being able to get the derivative of the function 𝑓(𝑥)
• 𝑓 𝑥 has to be a 𝐶 2 function
• Less stable than bissection (convergence is not always sure)

25
Newton –Raphson method

Algorithm
Start
Set 𝑥 0
• Preliminary step
Find the expression of the derivative 𝑓 ′ 𝑥
0
Choose a starting point 𝑥 close enough to the solution 𝑥ҧ
Compute
𝑓 𝑥 𝑘 , 𝑓′ 𝑥 𝑘
• Algorithm
𝑘+1
• Compute the value of 𝑥 with
𝑓 𝑥𝑘
𝑥 (𝑘+1) = 𝑥 𝑘
− ′ 𝑘
𝑓 𝑥 𝑓 𝑥𝑘
𝑥 (𝑘+1)
=𝑥 𝑘
− ′ 𝑘 𝑥 (𝑘) = 𝑥 𝑘+1
𝑓 𝑥
• Check if one of the criteria is matched :
• Increment 𝑥 (𝑘+1) − 𝑥 𝑘
≤𝜀
• Number of iterations Stopping criteria no
satisfied ?
• If none of the criteria is matched, repeat the process
with the new value of 𝑥 𝑘 yes

End
𝑘+1
Solution is 𝑥
26
Newton–Raphson method

Multiple variable case

• ഥ ∈ 𝑈 such as
Let a non linear application 𝑓: 𝑈 ⊆ ℝ → ℝ. We search 𝒙

𝑓1 𝑥1 , 𝑥2 , … , 𝑥𝑛 = 0 𝑥1
𝑓 𝑥 , 𝑥 , … , 𝑥𝑛 = 0 𝑥2
𝒇 𝒙 =𝟎⟺ 2 1 2 , with 𝑥 = ⋮
⋮
𝑓𝑛 𝑥1 , 𝑥2 , … , 𝑥𝑛 = 0 𝑥𝑛

• For a N-dimensional problem 𝒇(𝒙) = 𝟎, the idea is the same but the derivative is replaced by the Jacobian
matrix 𝑱(𝒇 𝒙 ):
𝜕𝑓1 𝜕𝑓1
…
𝜕𝑥1 𝜕𝑥𝑛
𝑱 𝒇 𝒙 = ⋮ ⋱ ⋮
𝜕𝑓𝑛 𝜕𝑓𝑛
⋯
𝜕𝑥1 𝜕𝑥𝑛
Similarly as with one variable, the system to solve at each step is
−𝟏
𝒙(𝑘+1) = 𝒙 𝑘
− 𝑱 𝒇 𝒙 𝒇 𝒙𝑘 , with a given starting point 𝒙 0

27