1 iterate.

mcd
Introduction to successive iteration and open the door to numerical methods with the square root
example. This worksheet includes several animation clips on successive iteration.
Instructor: Nam Sun Wang

Successive iteration is extremely common in numerical computations whether we are trying to find
a solution to a (set of) linear or nonlinear algebraic equations, matrix inverse, or ordinary and partial
differential equations. You also see it in probability transition matrix, chaos, and the generation of
fractal patterns. The iterative scheme takes the general form of
x g( x )
The above form allows us to start with an initial value of x, plug it into function g, and the function g
will provide us with the next value of x. We insert the new value of x back into f(x) again, which will
crank out yet another x for us. We can repeat the process for as many times as we wish. This
process is called successive iteration or successive approximation (in cases where we resort to
iteration to compensate for approximation). Note that the above successive iteration scheme
contains a purely x term on the LHS. Let us illustrate the successive methods with the old-fashioned
square root problem where the objective is to find a number x such that
2
x x. x a (1)
In the following discussion, let us take as an example:
a 10
Of course, we symbolically denote such a number as:

a or 10
But what is its value? Here, we want to find the square root without resorting to the square root key
on our calculator or without calling the built-in square root operator/function in a numerical package.
We recall that 3*3=9 and 4*4=16, thus the answer should lie between 3 and 4. Let's start with:
x0 3
We have to add a little bit of correction to x0 . Let this correction be ε0 . The updated root after
making the correction is:

x1 x0 ε0 a
To find ε0 , we square both sides of the last equation. Remember, we are after a number whose
square is a.
2 2
x1 x0 ε0 a
Expanding (mark the last equation and choose |Symbolic|Expand Expression| from Mathcad menu)
yields:
2 2 2 2 2
x1 x0 2 . x 0. ε 0 ε0 a⎯⎯→ ε0 2 . x 0. ε0 x0 a 0 (2)

Substituting the values of x 0 =3 and a=10, we get:

2 2 2
ε0 2 . 3 . ε0 3 10 0 ⎯⎯→ ε0 6 . ε0 1 0
 2 iterate.mcd
Recall the well known quadratic formula to solve for ε0 :

2 2
6 6 4 6 6 4
ε0 3 10 and ε0 3 10
2 2
Ooops. We end up with an expression that contains the square root of 10, which we originally set
out to solve in the first place. The problem arises because we are trying to solve the quadratic
equation (2) in ε exactly with the quadratic formula. Since this leads us going in circles, we propose
to make an approximation -- the key word in numerical method! Let us drop the ε0 2 term from
Equation (2).
2
2 . x 0. ε0 x0 a 0 (The notation is a bit sloppy here, because I cannot find the
approximation sign in Mathcad.)
With this approximation, we have a much more manageable equation that does not require the
quadratic formula to find ε0 :
2
a x0
ε0
2. x0
Let us actually plug some numbers into this formula.
2
a x0
ε0 ε0 = 0.167
2. x0

The next value of x is:

2
x 1 x 0 ε0 x 1 = 3.167 Check: x1 = 10.028 ← not quite 10.
Since we have made an approximation by dropping the ε0 2 term, the correction we have just made is
not an exact one, as we can see from the check Nevertheless, we are now a step closer to the
actual root. The price to pay for making an approximation is iteration. Now, x1 is a better
estimate of sqrt(10), let us make another correction ε1 on top of x1 . We hope the resulting estimate
x2 will be an even better one.
x2 x1 ε1
To find ε1 , we follow the same set of steps as before: square both sides of the last equation, equate it
to a, and expand the square term.
2 2
x2 x1 ε1 a
2 2
x1 2 . x 1. ε 1 ε1 a
2 2
ε1 2 . x 1. ε1 x1 a 0
As before, we make an approximating by dropping the ε1 2 term.
2
2 . x 1. ε1 x1 a 0
2
a x1
ε1
2. x1
 3 iterate.mcd
Plug in some numbers.
2
a x1
ε1 ε1 = 0.004
2. x1

The next value of x is:

2
x2 x1 ε1 x 2 = 3.16228 Check: x2 = 10.00002 ← almost 10.

If we are satisfied with the results, we stop here. Otherwise, repeat until we are satisfied with the
accuracy. We shall do it just one more time, this time without derivation.
2
a x2 6
ε2 ε2 = 3.042 10
2. x2

The next value of x is:

2
x 3 x 2 ε2 x 3 = 3.162277660170 Check: x3 = 10.000000000009

Mathcad's build-in function gives: 10 = 3.162277660168 which agrees with our value to more than
10 digits.
The general iteration scheme for square root of a is:
For N 5 i 0 .. N and provide x0 ,
2
a xi
xi 1
xi
2. xi

Although we have started with a reasonably close initial guess of 3, this does not have to be so. We
can see that any nonzero initial guess will eventually lead to a root with this scheme. (We may need
more number of iterations though.) Zero is not a good initial guess because of the problem with
division by 0.
For N 15 i 0 .. N x0 1000 ← A lousy initial guess.
2
a xi
xi xi
1
2. xi x 3 = 125.0262489500585 ← Not quite there yet.
x last( x ) = 3.16227766016838← Good!

Furthermore, a different initial guess may lead to a different root when multiple roots exist. Of
course, we know that x*x=a has two solutions, a positive one and a negative one, although sqrt(a)
refers to the positive root and -sqrt(a) rfers to the negative one.

x0 1000 ← Another lousy initial guess from the negative side.

2
a xi
xi xi
1
2. xi x 3 = 125.0262489500585 ← Not quite there yet.
x last( x ) = 3.16227766016838 ← Good, but a negative one.
 4 iterate.mcd
The same formula is also applicable to different values of a.
a 100
x0 1
2
a xi
xi 1
xi x last( x ) = 10.000000000000000 ← Good!
2. xi

In terms of the general iteration form of x=g(x) mentioned at the beginning of this worksheet, the
iteration fomula is:
2
a x
g( x ) x (3)
2. x
xi 1
g xi

In Mathcad version 6 (not valid in version 5), we can iterate elegantly by defining a function
recursively. The following function says if x=g(x) is satisfied, return x and stop; otherwise call g(x)
to update x and iterate until convergence. The intended usage is to issue the initial guess as the
argument to the "iterate" function.
iterate( x ) ( x g( x ) ) . x ( x g( x ) ) . iterate( g( x ) )
iterate( 3 ) = ... display the value returned with an initial guess of 3

Another way of saying the same thing:

iterate( x ) if( x g( x ) , x , iterate( g( x ) ) )
iterate( 3 ) = ... display the value returned with an initial guess of 3
 5 iterate.mcd
Visualization of Successive Iteration x=g(x). The solution is where the iteration function g(x)
intersects with the diagonal line, which is a straight function of x.
Converging Case -- Equation (3)
2 Animation section: toggle off
x a
a 10 g( x ) N 3 i 0 .. N the next equation and set
2. x FRAME=0..2N
x0 1.5 xi g xi
1 FRAME 2 . N FRAME = 6
Generate the steps for plotting:
j 0 .. FRAME
uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j xx 1 , 1.1 .. 5
1) ) iteration floor( 0.5. FRAME )

Path to Convergence Click on the following icon to

5 play an animation clip.
iteration = 3

iterate1.avi
g(x) & x

1
1 2 3 4 5
x
 6 iterate.mcd
Let us now follow similar steps to find a cubic root of a given number a. The objective is to find x
such that x*x*x=a. Approximate x first, then make a correction.
a 10 ← A given number whose cubic root is to be estimated.
x0 2 ← Initial guess.
x1 x0 ε0
We calculate the correction term ε0 from (|Symbolic|Expand Expression| with Mathcad):
3 3
x1 x0 ε0 a
3 2 2 3
x0 3 . x 0 . ε0 3 . x 0. ε0 ε0 a
Ignore the quadratic term (ε0 2 ) and the cubic term (ε0 3 ):
3 2
x0 3 . x 0 . ε0 a

3
a x0
ε0
2
3. x0
Plug in some numbers.
3
a x0
ε0 ε0 = 0.167
2
3. x0
Update x:
x1 x0 ε0 x 1 = 2.167 3
Check: x1 = 10.171
Repeat the correction a couple more times:
3
a x1 3
ε1 x2 x1 ε1 x 2 = 2.1545 Check: x2 = 10.001
2
3. x1

3
a x2 3
ε2 x3 x2 ε2 x 3 = 2.15443469 Check: x3 = 10.00000003
2
3. x2
1
3
Mathcad's built-in routine gives: a = 2.15443469
It is clear that the scheme is converging rapidly to an actual cubic root. Thus, t he general iteration
scheme for a cubic root of a is:
For N 15 i 0 .. N and provide an initial guess x0 ,
3
a xi
xi 1
xi
2
3. xi
 7 iterate.mcd
We know from mathematical theories that there are three roots to a cubic equation. For x 3 =a, where
a is a real number, there is a real root and two complex roots. To find a complex root, we must start
with a complex guess:
a 1 x0 1 0.7i ← Initial guess. The "i" here denotes the imaginary part, which is
entered by typing "1i", which is not to be confused with the running
3
a xi index "i" below. The resulting answer is very sensitive to the initial
xi 1
xi guess. In fact, we can generate an interesting fractal pattern based
2
3. xi on whether the resulting answer is the real number "1" or a complex
one.
x last( x ) = 1

The general iteration formula for the n-root of a given number a is:
n Example: n 4 a 10 with an initial guess of x 0 2
a x
g( x ) x
n 1 yields the following answer:
n. x
xi 1
g xi x last( x ) = 1.77827941
1
n
Mathcad's built-in routine gives: a = 1.77827941

Below, we present a different way of deriving an iteration formula for finding the square root of a given
number a. We start with what we are trying to achieve.
2
x a
Let us add x2 to both sides of the equation.
2 2 2
x x x a
2 2
2. x x a
Divide the above equation by 2x so that we obtain an equation in the form of x=g(x) suitable for
iteration
2
x a
x
2. x
Thus, the iteration scheme is:
2
x a
g( x ) (4)
2. x
Now, let us apply this iteration scheme to a 10
Start with: x0 3
x1 g x0 x 1 = 3.167
x2 g x1 x 2 = 3.162280701754
:
xi 1
g xi x last( x ) = 3.162277660168
 8 iterate.mcd
A close inspection shows that Equation (4) is identical to Equation (3), which incidentally is identical
to the Newton's method.
2
The algebraic equation f(x) to be solved for is: x a 0
2
f( x ) x a f'( x ) 2. x
f xi
xi 1
xi
f' x i

Many numerical methods exist for solving nonlinear equation of the form f(x)=0. Each one of these
iterate based on the same general form of x=g(x) but a different specific functional form of g(x). Thus,
deriving a good numerical algorithm is to find an expression of x=g(x). Who is to say that we cannot
derive a different formula? There are infinite possibilities. For example, if we add a 2x2 term, instead
of x2 , to both sides of the equation at the beginning of this section, the resulting iteration formula
would be:

2
2. x a
g( x )
3. x

Now, let's apply this new iteration scheme to a 10

Start with: x0 3
x1 g x0 x 1 = 3.111
x2 g x1 x 2 = 3.145502645503
:
xi 1
g xi x last( x ) = 3.162277656689

which is also eventually converging to the value of 10 = 3.162277660168

 9 iterate.mcd
Let us try another scheme by simply adding a term x to the both sides of equation (1). This is a
favorite trick for many because it quickly converts an algebraic equation of the form 0=f(x) into a
successive iteration form x=g(x), which is simply x+f(x).
2
x x x a
2
x x a x
2
g( x ) x a x (5)
Now, let us apply this new iteration scheme to a 10
Start with: x0 3
x1 g x0 x1 = 4
x2 g x1 x2 = 2
x3 g x2 x3 = 4
:
xi 1
g xi x last( x ) = 2

Hmm... The value of x is just switching back and forth, not at all converging.
Plot for Switching Case -- Equation (5)
Animation section: toggle off
N 5 i 0 .. N the next equation and set
x0 3 xi g xi FRAME=0..2N
1
FRAME 2 . N FRAME = 10
Generate the steps for plotting:
j 0 .. FRAME
uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j xx 5 , 4.9 .. 10
1) ) iteration floor( 0.5. FRAME )

Path to Divergence Click on the following icon to

10 play an animation clip.
iteration = 5

5
iterate2.avi
g(x) & x

5
5 0 5 10
x
 10 iterate.mcd
Let us try another scheme. If we add 0.95*x to both sides of equation (1), we get:
2
0.95. x x 0.95. x a
2
a x
x x
0.95
2
a x
g( x ) x (6)
0.95

Start with: x0 3
x1 g x0 x 1 = 4.053
x2 g x1 x 2 = 2.709
x3 g x2 x 3 = 0.090
x4 g x3 x 4 = 10.608 ← Oscillating away from the solution.

Hmm... The value of x is getting larger and larger and not converging into a single number. This is
called divergence.
Plot for Diverging Case -- Equation (6)
Animation section: toggle off
N 5 i 0 .. N the next equation and set
x0 3 xi g xi FRAME=0..2N
1
FRAME 2 . N FRAME = 10
Generate the steps for plotting:
j 0 .. FRAME
uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j xx 5 , 4.9 .. 12
1) ) iteration floor( 0.5. FRAME )

Path to Divergence Click on the following icon to

play an animation clip.
10
iteration = 5

iterate3.avi
5
g(x) & x

5
5 0 5 10
x
 11 iterate.mcd
Now, if we add 1.05*x to both sides of equation (1), we get:
2
a x
g( x ) x (7) ← Try different values for the denominator, in place of 1.05.
1.05
Start with: x0 3
x1 g x0 x 1 = 3.952
x2 g x1 x 2 = 1.401
x3 g x2 x 3 = 6.253
x4 g x3 x 4 = 21.456 ← Getting away from the solution.
:
xi 1
g xi x last( x ) = 2

Plot for Diverging Case -- Equation (7)

uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )
xx 5 , 4.9 .. 12

Path to Divergence

10

5
g(x) & x

5
5 0 5 10
x
 12 iterate.mcd
Another easy way of deriving an iteration expression is to reduce the given algebraic equation first
into the form of f(x)=0. This is also a farorite step for many. Since f(x)=0, we can multiply, divide or
do almost anything to it and the resulting expression is still 0. Then, we can add a term x to both
side of the equation to reach an iteration form of x=g(x). Let us demonstrate this approach where
we divide f(x) by 2x first, then add x.

Step 1. Reduce x 2 =a to the form f(x)=0

2
f( x ) x a 0
Step 2. Multiply by -1/2x (or some other expression of x)
2
x a 0
0
2. x 2.x
Step 3. Add x to both sides (which is an easy way of generating the iteration form x=g(x)).
2
x a
x x
2. x
Step 4. Find g(x), which is the RHS of the last equation
2
x a
g( x ) x
2. x

The resulting formula is the same as that from the Newton's method. Different numerical algorithms
mainly differ in the expression mutiplied to f(x)=0 in Step 2. If we choose to multiply by -0.1/x
instead, we get an expression that has a slower convergence property near the root. A slower
convergence is not necessarily bad. We sometimes prefer a slower convergence when we want to
approach the root gingerly and conservatively.

0.1 . 2
g( x ) x x a
x
Let us generate some numbers based on the above iteration formula and visualize convergence
graphically.
x0 1.5 xi 1
g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )

Path to Convergence
5

iteration = 5
4
g(x) & x

1
1 2 3 4 5
x
 13 iterate.mcd
Of course, not all iteration formula lead to a root. On the other hand, if we choose to multiply by
+0.1/x, we get divergence.
0.1 . 2
g( x ) x x a
x
x0 5 xi 1
g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )

Path to Divergence x0 2 xi g xi
10 1
uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )

Path to Divergence
10
5
g(x) & x

5
0

g(x) & x
iteration = 5

5 0
5 0 5 10
x
iteration = 5

5
5 0 5 10
x
 14 iterate.mcd
When do we achieve convergence and when do we face divergence? In general, we reach
convergence when the |slope of g(x)| at the point of intersection with x is within -1 to 1.
Furthermore, as shown below, we achieve very fast convergence when the slope of g(x) is close to 0.
On the other hand, convergence is slow when the slope of g(x) is close to 1 or -1. Hence, the trick is
to come up with an iteration scheme such that the slope of g(x) lies within -1 and 1, preferably close
to 0. If we did not know anything about the convergence property, we have about 50% chance on
the average of coming up with a converging formula (and 50% chance of a diverging one). So if the
first try does not work, keep trying different manipulations to get x=g(x). The chances are you will hit
one converging formula if you tried often enough. Below, we graphically demonstrate these two
cases.
Case 1a. 0<slope<1 -- Monotonic convergence to the intersection of x and g(x) for all initial guesses
i 0 .. 10
g( x ) 0.9. x x0 5 xi 1
g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )
xx 5 , 4.9 .. 5

Convergence -- 0<slope<1 x0 5 xi g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j

5 1 1) )

Convergence -- 0<slope<1
5
g(x) & x

0
g(x) & x

5
5 0 5
x

5
5 0 5
x

g( x ) 0.5. x x0 5 xi 1
g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )

Convergence -- 0<slope<1
5
g(x) & x

5
5 0 5
x
 15 iterate.mcd
Case 1b. -1<slope<0 -- Oscillatory convergence to the intersection of x and g(x) for all initial guesses
g( x ) 0.5. x x0 5 xi 1
g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )

Convergence -- -1<slope<0 x0 5 xi g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j

5 1 1) )

Convergence -- -1<slope<0
5
g(x) & x

g(x) & x
0

5
5 0 5
x

5
5 0 5
x

Case 2a. 1<slope -- Monotonic divergence from the intersection of x and g(x) for all initial guesses
g( x ) 1.5. x x0 0.5 x i 1
g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )

Divergence -- 1<slope x0 0.5 x i g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j

5 1 1) )

Divergence -- 1<slope
5
g(x) & x

0
g(x) & x

5
5 0 5
x

5
5 0 5
x
 16 iterate.mcd
Case 2b. slope<-1 -- Oscillatory divergence from the intersection of x and g(x) for all initial guesses
g( x ) 1.5. x x0 0.5 x i 1
g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j 1) )

Divergence -- slope<-1 x0 0.5 x i g xi uj x floor( 0.5 . j ) vj x floor( 0.5 . ( j

5 1 1) )

Divergence -- slope<-1
5
g(x) & x

g(x) & x
0

5
5 0 5
x

5
5 0 5
x