Chapitre 1

Numerical solution of non-linear

equations

First part linear systems

1.1 Theory of linear systems

1.1.1 Denitions
Dénition 1.1.1. We call a linear equation in the variables (or unknowns) x1 , . . . , xp any rela-
tionship of the form
a1 x1 + · · · + ap xp = b, (1.1)

where a1 , . . . , ap and b are given real numbers.

Remarques 1.1.2. ˆ It is important to stress here that these linear equations are implicites,
that is to say, they describe relationships between variables, but do not directly give the
values that the variables can take.
Solving an equation therefore means making it explicit, i.e. making more apparent the values
that the variables can take.

ˆ We can also consider linear equations of rational or complex numbers.

Let n⩾1 be an integer.

Dénition 1.1.3. A system of n linear equations with p unknowns is a list of n linear equations.
is a list of n linear equations.

Such systems are usually written in n lines placed one below the other.

1
1.1. THEORY OF LINEAR SYSTEMS 2

Exemple 1.1.4. The following system has 2 equations and 3 unknowns :


 x1 − 3x2 + x3 = 1
 −2x1 + 4x2 − 3x3 = 9

The general form of a linear system of n equations with p unknowns is as follows :




 a11 x1 +a12 x2 +a13 x3 + · · · +a1p xp = b1 (← équation 1)


a21 x1 +a22 x2 +a23 x3 + · · · +a2p xp = b2 (← équation 2)




. . . . .


 . . . . .
 . . . . = .



 ai1 x1 +ai2 x2 +ai3 x3 + · · · +aip xp = bi (← équation i)
. . . . .


. . . .
= ..

. . . .





 a x +a x +a x + · · · +a x = b

(← équation n)
n1 1 n2 2 n3 3 np p n

The numbers aij , i = 1, . . . , n, j = 1, . . . , p, are the coecients of the system.

The numbers bi , i = 1, . . . , n, form the second member of the system.

Dénition 1.1.5. A solution of the linear system is a list of p real numbers (s1 , s2 , . . . , sp ) (a
p-uplet) such that if we substitute s1 for x1 , s2 for x2 , etc., in the linear system, we obtain
equality. linear system, we obtain an equality. The set of solutions of the system is the set of all
these p-uplets.

Exemple. The system 

 x1 − 3x2 + x3 = 1
 −2x1 + 4x2 − 3x3 = 9

admits as solution (−18, −6, 1), i.e.

x1 = −18 , x2 = −6 , x3 = 1 .

On the other hand, (7, 2, 0) only satises the rst equation. It is therefore not a solution to the

system.

As a general rule, the aim is to determine the set of the solutions of a linear system. This is

called solving the linear system. This leads to the following denition.

Dénition 1.1.6. Two linear systems are said to be equivalent if they have the same set of
solutions.

from there, the game of solving a given linear system is to transform it into an equivalent

system that is simpler to solve than the original system. We'll see later how to do this systema-

tically.
1.1. THEORY OF LINEAR SYSTEMS 3

1.1.2 Dierent types of systems

Here is an important theoretical result for linear systems.

Théorème 1.1.7. A system of linear equations has either no solution, a single solution, or an
innite number of solutions.

In particular, if you nd 2 dierent solutions to a linear system, then you can nd an innite

number of them ! A linear system that has no solution is said to be incompatible.

1.1.3 Homogeneous systems

An important special case is that of homogeneous systems, for which b1 = b2 = · · · = bn = 0,
i.e. where the second member is zero. Such systems are always compatible because they always

admit the solution s1 = s2 = · · · = sp = 0. This solution is called trivial solution.

1.1.4 Solution by the Gauss pivot method

Operations on the equations of a system
We are going to use three elementary on the equations (i.e. on the lines) which are :

1. Li ← λLi with λ ̸= 0 : an equation can be multiplied by a non-zero real.

2. Li ← Li + λLj with λ∈R (and j ̸= i) : we can add to equation Li a multiple of another

equation Lj .

3. Li ↔ Lj : two equations can be exchanged.

These three elementary operations do not change the solutions of a linear system ; in other words,

these operations transform a linear system into an equivalent linear system.

Exemple. Let's use these elementary operations to solve the following system.

 x

 +y +7z = −1 (L1 )

2x −y +5z = −5 (L2 )


−x −3y −9z = −5

(L3 )

Let's start with the operation L2 ← L2 − 2L1 : subtract the rst equation twice from the second

equation. We obtain an equivalent system with a new (simpler) second line :


 x

 +y +7z = −1
−3y −9z = −3 L2 ←L2 −2L1


−x −3y −9z = −5

1.1. THEORY OF LINEAR SYSTEMS 4

Then L3 ← L3 + L1 :

 x

 +y +7z = −1
−3y −9z = −3


−2y −2z = −6

L3 ←L3 +L1

Continue so that a coecient 1 appears at the top of the second line ; to do this, divide the line

L2 by −3 :

 x

 +y +7z = −1
y +3z = 1 L2 ← − 31 L2


−2y −2z = −6


And so we continue
 
 x +y +7z = −1  x +y +7z = −1

 

y +3z = 1 y +3z = 1

 

4z = −4 z = −1 L3 ← 14 L3
 
L3 ←L3 +2L2
 
 x +y +7z = −1  x +y = 6 L1 ←L1 −7L3

 

y = 4 L2 ←L2 −3L3 y = 4

 

z = −1 z = −1
 

We end up with a reduced, echelon system :


 x = 2 L1 ←L1 −L2


y = 4


z = −1


We thus obtain x = 2, y = 4 and z = −1 and the only solution to the system is (2, 4, −1).

The method used for this example is repeated and generalised in the next paragraph.

Echelon and reduce systems

For each row in a matrix, if the row does not consist of only zeros, then the leftmost nonzero

entry is called the [[leading coecient]] (or pivot) of that row. So if two leading coecients are in

the same column, then a row operation of [[Row operations|type 3]] could be used to make one of

those coecients zero. Then by using the row swapping operation, one can always order the rows

so that for every non-zero row, the leading coecient is to the right of the leading coecient of

the row above. If this is the case, then matrix is said to be in row echelon form. So the lower left

part of the matrix contains only zeros, and all of the zero rows are below the non-zero rows. The

word echelon is used here because one can roughly think of the rows being ranked by their size,

with the largest being at the top and the smallest being at the bottom.

So as denition :
1.1. THEORY OF LINEAR SYSTEMS 5

Dénition 1.1.8. A system is echelon if :

ˆ the number of zero coecients starting a line grows strictly line by line.

It is reduced echelon form if in addition :

ˆ the rst non-zero coecient of a line is 1.

ˆ and it is the only non-zero element in its column.

Example 1

For example, the following matrix is in row echelon form, and its leading coecients are
 
0 2 1 −1
 
shown in red : 0 0 3 1  .
 
0 0 0 0
It is in echelon form because the zero row is at the bottom, and the leading coecient of the

second row (in the third column), is to the right of the leading coecient of the rst row (in the

second column).

A matrix is said to be in reduced row echelon form if furthermore all of the leading coecients

are equal to 1 (which can be achieved by using the elementary row operation of type 2), and

in every column containing a leading coecient, all of the other entries in that column are zero

(which can be achieved by using elementary row operations of type 3).

Example 2




 2x1 +3x2 +2x3 −x4 = 5

−x2 −2x3

 = 4
ˆ is echelon (but not reduced).





3x4 = 1



 2x1 +3x2 +2x3 −x4 = 5


ˆ −2x3 = 4 is not echelon (the last line starts with the same va-


x3 +x4 = 1

riable as the line above).

It turns out that linear systems in reduced echelon form are particularly easy to solve.

Exemple. The following linear system with 3 equations and 4 unknowns is echelon and reduced.

 x1 +2x3 = 25


x2 −2x3 = 16


x4 = 1

1.1. THEORY OF LINEAR SYSTEMS 6

This system can be trivially solved by


 x1 = 25 − 2x3


x2 = 16 + 2x3


x4 = 1.


In other words, for any real value of x3 , the values of x1 , x2 and x4 calculated above provide a

solution to the system, and we have thus obtained them all. We can therefore fully describe the

set of solutions :

S = (25 − 2x3 , 16 + 2x3 , x3 , 1) | x3 ∈ R .

Algorithm for Gauss Elimination Method

In linear algebra, Gauss Elimination Method is a procedure for solving systems of linear

equation. It is also known as Row Reduction Technique. In this method, the problem of systems

of linear equation having n unknown variables, matrix having rows n and columns n+1 is formed.

This matrix is also known as Augmented Matrix. After forming n x n+1 matrix, matrix is

transformed to upper trainagular matrix by row operations. Finally result is obtained by Back

Substitution.

Algorithm for Gauss Elimination Method

1. Start

2. Read Number of Unknowns : n

3. Read Augmented Matrix (A) of n by n+1 Size

4. Transform Augmented Matrix (A) to Upper Trainagular Matrix by Row Operations.

5. Obtain Solution by Back Substitution.

6. Display Result.

7. Stop

Examples of the algorithm

Example 1
1.1. THEORY OF LINEAR SYSTEMS 7

Suppose the goal is to nd and describe the set of solutions to the following system of linear

equations :

2x + y − z = 8 (L1 ) (1.2)

−3x − y + 2z = −11 (L2 ) (1.3)

−2x + y + 2z = −3 (L3 ) (1.4)

The table below is the row reduction process applied simultaneously to the system of equations

and its associated augmented matrix. In practice, one does not usually deal with the systems in

terms of equations, but instead makes use of the augmented matrix, which is more suitable for

computer manipulations. The row reduction procedure may be summarized as follows : eliminate

mvar|x from all equations below L1 , and then eliminate y from all equations below L2 . This will

put the system into triangular form. Then, using back-substitution, each unknown can be solved

for.

Example 2 Solve :
1.1. THEORY OF LINEAR SYSTEMS 8

   
2 4 −2 −2 −4
   
1 2 4 −3 5
A= b= 
   

−3 −3 8 −2 7
   
−1 1 6 −3 7
Note that there is not hing wrong with this system. Aisfull rank. The solution exists and is

unique. Form the augmented system.

Subtract 1/2 times the rst row from the second row,

add 3/2 times the rst row to the third row,

add 1/2 times the rst row to the fourth row. The result of these operations is :

The next stage of Gaussian elimination will not work because there is a zero in the pivot location,

~a22 .

Swap second and fourth rows of the augmented matrix.

Continue with elimination : subtract (1times) row 2 from row 3.

Another zero has appear in the pivot position. Swap row 3 and row 4.
1.1. THEORY OF LINEAR SYSTEMS 9

The augmented system is now ready for backward substitution.

Example 3

Consider the system of linear equations :

We wish to know the values of a ,

b and c for which there are there no solutions, one solution or more than one solution. To

solve this problem, we apply Gaussian elimination to the augmented matrix :

An analysis of the last row tells us everything : If b−a ̸= 0 , then there is exactly one solution.
If b−a = 0 , and c − 3 ̸= 0 , then there are no solutions. Otherwise (when b=a and c=3 )

there are an innite number of solutions.

Exercice

Find all solutions to the system of equations

Solution We put the augmented matrix into row echelon form :

1.1. THEORY OF LINEAR SYSTEMS 10

1.1.5 Solving Systems with Cramer's Rule

n unknowns, represented in matrix multiplication form as follows :

Ax = b

where the n×n matrix A has a nonzero determinant, and the vector x = (x1 , . . . , xn )T is

the column vector of the variables. Then the theorem states that in this case the system has a

unique solution, whose individual values for the unknowns are given by :

det(Ai )
xi = i = 1, . . . , n
det(A)
where Ai is the matrix formed by replacing the i-th column of A by the column vector b.

A more general version of Cramer's rule considers the matrix equation

AX = B

where the n×n matrix A has a nonzero determinant, and X, B are n×m matrices. Given

sequences 1 ⩽ i1 < i2 < · · · < ik ⩽ n and 1 ⩽ j1 < j2 < · · · < jk ⩽ m, let XI,J be the k×k
submatrix of X with rows in I := (i1 , . . . , ik ) and columns in J := (j1 , . . . , jk ). Let AB (I, J)
1.1. THEORY OF LINEAR SYSTEMS 11

be the n×n matrix formed by replacing the is column of A by the js column of B, for all

s = 1, . . . , k . Then
det(AB (I, J))
det XI,J = .
det(A)
In the case k = 1, this reduces to the normal Cramer's rule. The rule holds for systems of

equations with coecients and unknowns in any eld , not just in the real numbers.

Explicit formulas for small systems=== Consider the linear system

: 
a x + b y = c
1 1 1
a2 x + b2 y = c2

which in matrix format is

:     
a b x c
 1 1   =  1 .
a2 b2 y c2
Assume a1 b2 − b1 a2 is nonzero. Then, with the help of determinants, x and y can be found

with Cramer's rule as

c1 b1 a1 c1
c2 b2 c1 b2 − b1 c2 a2 c2 a1 c2 − c1 a2
x= = , y= = (1.5)
a1 b1 a1 b2 − b1 a2 a1 b1 a1 b2 − b1 a2

a2 b2 a2 b2
.

The rules for 3 Ö 3 matrices are similar. Given


a1 x + b1 y + c1 z = d1


a2 x + b2 y + c2 z = d2


a3 x + b3 y + c3 z = d3


which in matrix format is

    
a1 b1 c1 x d1
    
a2 b2 c2  y  = d2  .
    
a3 b3 c3 z d3
Then the values of x, y and z can be found as follows :

d1 b1 c1
d2 b2 c2
d3 b3 c3
x= ,
a1 b1 c1
a2 b2 c2
a3 b3 c3
1.1. THEORY OF LINEAR SYSTEMS 12

a1 d1 c1
a2 d2 c2
a3 d3 c3
y= ,
a1 b1 c1
a2 b2 c2
a3 b3 c3
and
a1 b1 d1
a2 b2 d2
a3 b3 d3
z=
a1 b1 c1
a2 b2 c2
a3 b3 c3

Examples
Example1 : Using Cramer's Rule to Solve a 2Ö2 System

solve the following 2Ö2 system using Cramer's Rule.


 12x + 3y = 15
 2x − 3y = 13

Solution

Solve for x

Solve for y
1.2. THEORY OF NON LINEAR EQUATIONS 13

The solution is (2,-3)

Example 2 : Solving a 3Ö3 System Using Cramer's Rule

Find the solution to the given 3Ö3 system using Cramer's Rule.




 x+y−z =6
3x − 2y + z = −5


x + 3y − 2z = 14


Second part non-linear systems

1.2 Theory of non linear equations

1.2.1 The Bisection Method

In this part, we will be interested in solving equations of the form f (x) = 0. Because f (x) is

not assumed to be linear, it could have any number of solutions, from 0 to ∞. In one dimension,
1.2. THEORY OF NON LINEAR EQUATIONS 14

if f (x) is continuous, we can make use of the Intermediate Value Theorem (IVT) to bracket
a root ; i.e., we can nd numbers a and b such that f (a) and f (b) have dierent signs.

Then the IVT tells us that there is at least one magical value x
∗ ∈ (a, b) such that f (x∗ ) = 0. The
number x∗ is called a root or zero of f (x). Solving nonlinear equations is also called root-nding.
To bisect means to divide in half. Once we have an interval (a,b) in which we know x∗ lies,

a systematic way to proceed is to sample f(a+b/ 2 ).

If f(a+b/ 2 ) =0, then x∗ = a + b/2 , and we are done !

Otherwise, the sign of f(a+b /2 ) will either agree with the sign of f(a), or it will agree with

the sign of f(b). Suppose the signs of f(a+b 2 ) and f(a) agree. Then we are no longer guaranteed

that x∗ ∈ (a, a+b/2) but we are still guaranteed that x∗ ∈ (a+b/ 2 ) ,b). So we have narrowed

∗
down the region where x lies. Moreover, we can repeat the process by setting a+b 2 to a (or to

b, as applicable) until the interval containing x∗ is small enough.

Algorithm

Objective

Given a non-linear function f (x), nd the solution x0 to the equation f (x) = 0 in an interval
[a, b] for which f (a)f (b) < 0.
Inputs

A and B (the extremities of the initial interval), ε the desired precision, N0 (the maximum

number of iterations).

Output

A value approximating X0 or a failure message.

1. If f (a)f (b) > 0

Then printed (there is no change of sign between the bounds of this interval).
1.2. THEORY OF NON LINEAR EQUATIONS 15

go to 9)

2. Pose that n=1

3. While n ⩽ N0 carry out steps 4) to 7).

a+b
4. Pose c= 2

b−a
5. if f (c) = 0 or
2 =0 then printed (c) end

6. n=n+1

7. if f (a)f (c) > 0

then a=c
otherwise b=c

8. Printed after N0 iterations, the approximation obtained is X0 = c and the maximum error

b−a
is
2 .

9. end.

Notes

1. Interval bisection is a slow-but-sure algorithm for 

nding a zero of f (x), where f (x) is a real-valued function of a single real variable.

2. We assume that we know an interval [a,b] on which a continuous function f(x) changes

sign.

However, there is likely no oating-point number (or even rational number !) where f(x) is

exactly 0.

So our goal is :

Find a (small) interval (perhaps as small as 2 successive f loating-point numbers) on which

f(x) changes sign.

Sadly, bisection is slow !

3. If f(x) is continuous and we have a starting interval on which f(x) changes sign, then

bisection is guaranteed to reduce that interval to two successive oating-point numbers

that bracket x∗ .
1.2. THEORY OF NON LINEAR EQUATIONS 16

Example
Example

Let the following continuous function

f (x) = 2 − exp(5x)

Find the value x0 that corresponds to f (x) = 0.5 using the bisection method

Solution

f (x) = 0, 5 ⇒ 2 − exp(5x) = 0.5 ⇒ exp(5x) − 1, 5 = 0 ⇒ g(x) = 0 avec g(x) = exp(5x) − 1.5

Let then nd x0 that exp(5x) − 1, 5 = 0
On the one hand, we see that g(0) = −0.5 < 0 and that g(1) = 146.91 > 0. On the other

hand, we see that g ′ = 5 exp(5x) is strictly positive everywhere in the set of real numbers, so the
interval in which we can look for the solution x0 is [0, 0, 1], so a=0 and b = 0, 1.
The results of the bisection algorithm are shown in the table below :

Iteration number ai 2 sig of f (ai ) bi sig of f (bi ) ci sig of f (ci ) | ci − ci−1 |

1 0 - 0.1 + 0.05 - int

2 0.05 - 0.1 + 0.075 - 0.025

3 0.075 - 0.1 + 0.0875 + 0.0125

4 0.075 - 0.0875 + 0.08125 + 0.00625

5 0.075 - 0.08125 + 0.078125 - 0.003125

6 0.078125 - 0.08125 + 0.079875 - 0.0015625

7 0.0796875 - 0.08125 + 0.08046875 - 0.00078125

Results of the bisection method

The solution using the bisection method is x0 = c7 = 0.08046875. This result converges with the

condition | ci − ci−1 |⩽ 10−3

1.2.2 False position method (or descartes method)

This method belongs to the family of interval methods and is achieved by a small, simple

modication to the bisection method. In step 4 of the bisection algorithm, instead of choosing ci
1.2. THEORY OF NON LINEAR EQUATIONS 17

as the midpoint of the interval [ai , bi ], we will take as the value of ci the point where the straight

line joining the points (ai , f (ai )) and (bi , f (bi )) intersects the x-axis (x). The relationship to be

applied from one step to the next is therefore as follows."

f (bi )[bi − ai ]
ci = bi −
f (bi ) − f (ai )

Changing the edge of the A-B interval is done in exactly the same way as with the bisection

algorithm. This gives rise to the following algorithm.

Algorithm

Objective

Given a non-linear function f (x), nd the solution x0 to the equation f (x) = 0 in an interval
[a, b] for which f (a)f (b) < 0.
Inputs

A and B (the extremities of the initial interval), ε the desired precision, N0 (the maximum

number of iterations).

Output

A value approximating X0 or a failure message.

If f (a)f (b) > 0

Then printed (there is no change of sign between the bounds of this interval).

go to 8)

1. Pose that n=1

2. While n ⩽ N0 carry out steps 3) to 6).

f (bi )[bi −ai ]

3. Pose ci = bi − f (bi )−f (ai )
1.2. THEORY OF NON LINEAR EQUATIONS 18

b−a
4. if f (c) = 0 or
2 <ε then printed (c) end

5. n=n+1

6. if f (a)f (c) > 0

then a=c
otherwise b=c

7. Printed after N0 iterations, the approximation obtained is X0 = c and the maximum error

b−a
is
2 .

8. end.

Note

1. If the curve of the function does not change in the interval [a, b], i.e. the curve remains

convex throughout the interval or remains concave throughout this interval, then step 7 of

the false position method algorithm will always lead to a xed bound a or b which does not

change along the iterations. If the function is convex on [a, b], (f ′ (x) < 0), and if f (a) < 0,
then a is the xed bound. And if f (a) > 0, then b is the xed bound. The xed bound will

be the bound with a negative sign.

2. If the function is concave (f

′ (x) > 0) and if f (a) < 0, then the xed bound will be b and

if f (a) > 0, then the xed bound will be a. The xed bound will be the bound with a

positive sign. In conclusion, if the curve of f does not change sign on the interval [a, b], then
the xed bound will be the bound whose image has the same sign as the curve of f on the

interval [a, b].

Exmple
Let's take the same example from the previous section with g(x) = exp(5x − 1.5 and [0,0.1]

as the start interval and applying the false position method. The results converged with the

|ci − ci−1 | ⩽ 10−3 criterion are shown in the following table

Iteration number ai 2 sig of f (ai ) bi sig of f (bi ) ci sig of f (ci ) | ci − ci−1 |

1 0 - 0.1 + 0.0771 - int

2 0.0771 - 0.1 + 0.0809 - 0.0038

3 0.0809 - 0.1 + 0.0811 - 0.0002

So we can see that the false position method needed at least half as many iterations to achieve

the same convergence as the bisection method in the previous example.

1.2. THEORY OF NON LINEAR EQUATIONS 19

1.2.3 Newton's Method

Newton's method for solving f(x) = 0 works in the following fashion. Suppose you have a

guess xn for a root x∗ .

Find the tangent line to y = f(x) at x = xn and follow it down until it crosses the x-axis ; call

the crossing point xn+1 .

This leads to the iteration
f (xn )
xn+1 = xn −
f ′ xn
Often xn+1 will be closer to x∗ than xn was. Repeat the process until we are close enough to x∗ .

Algorithm

Objective

Given a continuous non-linear function f (x), nd the solution x0 to the equation f (x) = 0.
Inputs

An initial approximation x0 of the solution

ε the desired precision, N0 (the maximum number of iterations).

Output

A value approximating X0 or a failure message.

1. Pose that n=1

2. While n ⩽ N0 carry out steps 3) to 6).

f (x0 )
3. Pose x = x0 − f ′ (x0 )

4. if |x − x0 | < ε then printed (x) end

5. Pose n=n+1

6. Pose x0 = x

7. Print Newton's method failed after N0 iterations.

1.2. THEORY OF NON LINEAR EQUATIONS 20

8. end.

Notes

When Newton's method works, it is really great !

In fact, the generalization of the above description of Newton's method is the only viable general-

purpose method to solve systems of nonlinear equations.

But, as a general-purpose algorithm for nding zeros of functions, it has 3 serious drawbacks.

1. The function f(x) must be smooth.

2. The derivative f (x)′ must be computed.

3. The starting guess must be suciently accurate

If f(x) is not smooth, then f (x)′ ) does not exist, and Newton's method is not dened.

Example
Let the following continuous function

f (x) = 2 − exp(5x)

Find the value x0 that corresponds to f (x) = 0.5 using the Newton method.

f (x) = 0, 5 ⇒ 2 − exp(5x) = 0.5 ⇒ exp(5x) − 1, 5 = 0 ⇒ g(x) = 0 avec g(x) = exp(5x) − 1.5

Let then nd x0 that exp(5x) − 1, 5 = 0
Determine the rst derivative of the function g ′ (x) = 5 exp(5x). La relation itérative de

Newton s'écrit alors :

exp(5xi − 1.5)
xi+1 = xi −
5 exp(5xi )
The following table shows the results with the initial solution x0 = 0.
Iteration number Approximation xi
0 0

1 0.1

2 0.0820

3 0.0811

4 0.08111
1.2. THEORY OF NON LINEAR EQUATIONS 21

1.2.4 Secant Method

Newton's method has two obvious disadvantages. The rst is the obligation to calculate the

rst derivative of f (x). The second disadvantage is that it is unlucky to choose an initial solution
whose rst derivative is zero. This second disadvantage can be resolved by choosing another initial

solution. The rst disadvantage suggests replacing the Newton method, of the rst derivative

f (x), f (xi )′ by its rst-order approximation.

f (xi ) − f (xi−1 )
f ′ (xi ) ≈
xi − xi−1

Then the relationship becomes

f (xi )[xi − xi−1 ]

xi+1 = xi −
f (xi ) − f (xi−1 )

Otherwise

Algorithm

Objective

Given a continuous non-linear function f (x), nd the solution x0 to the equation f (x) = 0.
Inputs

Two initial approximation x0 and x1 of the solution

ε the desired precision, N0 (the maximum number of iterations).

Output

A value approximating X0 or a failure message.

1. Pose that n=2

1.3. THEORY OF NON LINEAR SYSTEMS 22

2. While n ⩽ N0 carry out steps 3) to 6).

x1 −x0
3. Pose x = x1 − f (x1 )−f (x0 )

4. if |x − x1 | ⩽ ε then printed (x) end

5. Pose n=n+1

6. Pose x0 = x1 x1 = x, f (x0 ) = f (x1 ) f (x1 ) = f (x)

7. Print Secant method failed after N0 iterations.

8. end.

1.3 Theory of non linear systems

1.3.1 Newton's Method

Many methods for nding roots in one dimension do not generalize directly to m dimensions.

Fortunately, Newton's method can generalize to higher dimensions quite easily.

It is the most popular and powerful method for solving systems of nonlinear equations.

Before discussing it, we will rst need to introduce the concept of the Jacobian matrix (also

known as the matrix of rst partial derivatives).

Let
 
f1 (x1 , x2 , x3 ..., xm )
 
 f2 (x1 , x2 , x3 ..., xm ) 
 
 
f (x) =  f3 (x1 , x2 , x3 ..., xm ) 


 . 
.
.
 
 
fm (x1 , x2 , x3 ..., xm )
∂f (x)
Then the Jacobian matrix Jf (x) = ∂x is an m×m matrix whose (i, j) element is given by

∂fi
[Jf (x)]ij =
∂xj

e.g., when m = 2,  
∂f1 ∂f1
Jf (x) =  ∂x1 ∂x2 
∂f2 ∂f2
∂x1 ∂x2

Let  
x21 − x2 + σ
f (x) =  
−x1 + x22 + σ
1.3. THEORY OF NON LINEAR SYSTEMS 23

Then,  
2x1 −1
Jf (x) =  
−1 2x2
The Newton iteration for systems of nonlinear equations takes the form

xn+1 = xn − Jf−1 (xn )f (xn )

Of course, in practice we never actually form Jf Jf−1 (xn ) !

Instead we solve the linear system

Jf (xn )sn = −f (xn )

then update

xn+1 = xn + sn

Newton's method reduces the solution of a nonlinear system to the solution of a sequence of

linear equations.

Example
Solve the nonlinear system

 
x1 + 2x2 − 2
f (x) =  =0
x21 + 4x22 − 4
First compute the Jacobian :

 
1 2
Jf (x) =  
2x1 8x2

Start with initial guess x0 = [1; 2]T .

Then    
3 1 2
f (x0 ) =   Jf (x0 ) =  
13 2 16
Solve for the correction s0 from
    
1 2 3 −1.83
  s0 = −   ⇒ s0 =  
2 16 13 −058

etc.