Numerical Analysis : [ MA214 ]

Lecture 11

Instructor: Prof. Tony J. Puthenpurakal

Department of Mathematics
Indian Institute of Technology Bombay
[email protected]

September 5, 2017

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Recall

Last time we have discussed order of convergence

xn −→ ξ

en = ξ − xn
If there exists p ≥ 0 and a constant C 6= 0 such that

|en+1 |
lim = C.
n→∞ |en |p

This p is called “order of convergence” and C is called the asymptotic

error constant.
Examples 1. Fixed point iteration

ξ fixed point of g : I −→ I and g 0 (ξ) 6= 0.

Then p = 1 and C = |g 0 (ξ)|.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

2. For Newtons Method
1  f 00 (ξ) 
 
|en+1 |
lim =  0 , provided f 0 (ξ) 6= 0
n→∞ |en |2 2 f (ξ) 
 00 
So order of convergence is 2 and error constant is 12  ff 0 (ξ)
 (ξ) 

(if ξ is a double root then p = 1)

3. For Secant Method

|en+1 | = Cn |en ||en−1 |

√
1+ 5
p= = 1.618 · · ·
2

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

 |en+1 |  1 f 00 (ξ) 1/p
 
lim = 0 , provided f 0 (ξ) 6= 0
n→∞ |en |p 2 f (ξ) 

4. { n1r } −→ 0, here r ≥ 1 p = 1.

In theory if order of convergence p > 1, then it converges “fast” to ξ.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Numerical method to solve Linear system of equations

Suppose we have a system of equations

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
an1 x1 + an2 x2 + · · · + ann xn = bn

We want to find a solution

In applications, n is large (at least 1000). So doing it by hand is out of

equation. We have to use computers.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

It is convenient to use matrices
 
a11 a12 · · · a1n
 a21 a22 · · · a2n 
A= .
 
.. .. ..
 ..

. . . 
an1 an2 · · · ann
   
x1 b1
 x2   b2 
x = , b=
   
.. .. 
 .   . 
xn bn

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Thus we have to solve
Ax = b
This system either has
1 a unique solution
2 no solution
3 infinitely many solutions
Example
1 x1 + x2 = 2, x1 − x2 = 0 has a unique solution x1 = x2 = 1.
2 x1 + x2 = 1, 2x1 + 2x2 = 3 has no solution.
3 2x1 − x2 = 3, 4x1 − 2x2 = 6 has infinitely many solutions
{(x1 , x2 )|2x1 − x2 = 3}.
For many applications the system Ax = b has a unique solution.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Theorem
Ax = b has a unique solution if and only if A is an invertible matrix, i.e.,
there exists
 a matrix B such that BA = AB = In , In = n × n identity
1 0 ··· 0
 0 1 ··· 0 
matrix =  . . .
 
. . . .. 
 . . . . 
0 0 ··· 1

B is usually denoted by A−1

Ax = b
A−1 (Ax) = A−1 b
So x = A−1 b is the unique solution to Ax = b

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

In practice, computation of A−1 takes too many computations. Usually
we don’t need it.

In practice, there is a two step procedure to find solution of Ax = b.

Step 1 (Gaussian Elimination)

The system Ax = b is transformed to an equivalence system Ux = b,

where U = (uij ) is an upper triangular matrix i.e., uij = 0 for i > j.

Ax = b is equivalent to Ux = b means that x0 is a solution of Ax = b if

and only if x0 is a solution of Ux = b.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Step 2 Solving Ux = b.

We do step 2 first

Note that we are assuming Ax = b has a unique solution.

So Ux = b has a unique solution.

=⇒ U is an invertible matrix.

Exercise:
Show that an upper triangular matrix U = (uij ) is invertible iff all
diagonal entries (i.e., uii ) are non-zero.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Step 2 Solution of Ux = b

u11 x1 + u12 x2 + · · · + u1n xn = b1

u22 x2 + · · · + u2n xn = b2
..
.
un−1,n−1 xn−1 + un−1,n xn = bn−1
unn xn = bn

bn
xn =
unn
bn−1 − un−1,n xn
xn−1 =
un−1,n−1
P
bi − uij xj
j>i
xi = for i = n − 1, n − 2, · · · , 2, 1
uii
This process is called back-substitution.
Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11
Example:

3x1 + x2 + 2x3 = 6
4x2 + 2x3 = 7
3x3 = 9

9
x3 = =3
3
1
4x2 + 6 = 7 =⇒ x2 =
4
1 1
3x1 + + 6 = 6 =⇒ x1 = −
4 12
Gaussian Elimination

Recall two linear system Ax = b and Ax = b are equivalent if any solution

of one is a solution of the other.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Theorem
Let Ax = b be a linear system and suppose we subject this system to a
sequence of operation of the following kind
1 Multiplication of one equation by a non-zero constant
2 Addition of a multiple of one equation to another equation
3 Interchange of two equations
If this system produces a new system Ax = b then the system Ax = b
and Ax = b are equivalent. In particular, A is invertible iff A is invertible.

Gaussian Elimination It is possible to convert Ax = b to equivalent

system Ux = b (U upper triangular matrix) by using the above 3
operations.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Example

x1 − x2 + 2x3 = −6
2x1 − 2x2 + 3x3 = −14
x1 + x2 + x3 = −2

  
1 −1 2 : −6 1 −1 2 : −6
 2 −2 3 : −14  R2 − 2R1  0 0 −1 : −2 
−−−−−−→
1 1 1 : −2 1 1 1 : −2
   
1 −1 2 : −6 1 −1 2 : −6
R3 − R1  0 0 −1 : −2  R2 ←→ R3  0 2 −1 : 4 
−−−−−→ −−−−−−−→
0 2 −1 : 4 0 0 −1 : −2

−x3 = −2 =⇒ x3 = 2
2x2 − x3 = 4 =⇒ x2 = 3
x1 − x2 + 2x3 = −6 =⇒ x1 = −7
Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11
 Algorithm for Gaussian Elimination

To solve Ax = b
W = [A : b] “augmented matrix”
Step 1 For i = 1, 2, · · · , n − 1 do steps 2,3,4.
Step 2 Let p be the smallest integer with i ≤ p ≤ n and api 6= 0. If no
integer p can be found then output “no unique solution exists and
stop”.
Step 3 If p 6= i then interchange Row Ri ↔ Row Rj .
Step 4 For j = i + 1, · · · , n do steps 5,6.
aji
Step 5 Set mji = aii .
Step 6 perform Rj − mji Ri .
Step 7 If ann = 0 “no unique solution exists and stop”.
Step 8 U = first n columns of W . b = last column of W .

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Then Ax = b is equivalent to Ux = b where U is an upper triangular
matrix.

Operation Count

We count the number of multiplication/division and addition/subtraction

to do GE.

In general the amount of time required to perform a multiplication or

division on a computer is approximately the same and is considerably
greater than that required to perform addition or subtraction.

No arithmetic operation is performed until step 5 in the algorithm.

Step 5 requires about n − i division be performed.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

In step 6 we replace row Rj by Rj − mji Ri . This require mji to be
multiplied to each terms in Ri .

This requires (n − i)(n − i + 1) multiplication.

Afterwards each term of the resulting equation is subtracted from the

corresponding term in Rj . This requires (n − i)(n − i + 1) subtraction.

Thus for each i = 1, 2, · · · , n − 1 the operations required are

Multiplication/division: n − 1 + (n − i)(n − i + 1) = (n − i)(n − i + 2)

Addition/subtraction: (n − i)(n − i + 1)

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

 n−1
P 2n3 +3n2 −5n
Total multiplication/division: (n − i)(n − i + 2) = 6
i=1

n−1
P n3 −n
Total addition/subtraction: (n − i)(n − i + 1) = 3
i=1

For back substitution (i.e., step 2)

One can show one requires

n2 +n
2 multiplication/division

n2 −n
2 addition/subtraction

Note for large n, n3 is considerably larger than n2 , for example when

n = 100, 1002 is 1% of 1003 .

Thus GE is θ(n3 /3) operation.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Tridiagonal matrix
 
a1 b1 0 0 ··· 0 0
 c1 a2
 b2 0 ··· 0 0 

 0 c3
 a3 b3 ··· 0 0 

 .. .. .. .. .. .. .. 
 .
 . . . . . . 

 0 0 0 ··· ··· · · · bn−1 
0 0 0 0 ··· cn−1 an

A = [aij ] is said to be tridiagonal if aij = 0 for |i − j| > 1

Exercise
Ax = b for tridiagonal systems can be solved in θ(n) steps.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

LU-factorization

Steps used to solve a system Ax = b can be used to factor the matrix A.

The factorization is particularly useful if it has the form A = LU where L

is a lower triangular matrix and U is upper triangular matrix.

Not all matrices have this type of representation. However many matrices
that occur in practice have this property.

Application:- We would want to solve Ax = b for many different values

of b.

If we do GE each time then we would need θ(n3 /3) operation each time
we solve Ax = b.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

On the other hand once A = LU. Then we can solve Ax = b as follows

Set y = Ux. We solve LUx = b, =⇒ Ly = b

L is lower triangular. So determining y requires θ(n2 ) operation.

Then solve Ux = y (U is upper triangular), requires only θ(n2 ) operation.

Thus number of operation to solve the system Ax = b is reduced from

θ(n3 /3) to θ(2n2 ).

When n ≥ 1000 this reduces number of computation by more than 99%.

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Construction of LU factorization

Assumption:- Ax = b can be solved without row interchange.

First step in GE process consists of performing for each j = 2, 3, · · · , n

(1)
aj1
the operations Rj − mj1 R1 where mj1 = (1) .
a11

Equivalently, we can multiply the original matrix A on the left by the

matrix  
1 0 0 ··· 0
 −m21 1 0 ··· 0 
 
 −m31 0 1 ··· 0 
M (1) = 
 
.. .. .. . . .. 

 . . . . . 
 −mn−1,1 0 0 ··· 0 
−mn1 0 0 ··· 1

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Set A = A(1) , b = b (1) , A(1) x = b (1) .
M (1) A(1) x = M (1) b (1) .
Set A(2) = M (1) A(1) , b (2) = M (1) b (1)

So we have system A(2) x = b (2)

(2)
A(2) has ai1 = 0 for i ≥ 2.

In a similar manner we construct M (2) , the identity matrix with entries

below the diagonal in the second column replaced by the negative of the
multiple
(2)
aj2
mj2 = (2)
a22
A(3) = M (2) A(2)
b (3) = M (2) b (2)
A(3) has zeros below the diagonal in the first 2 columns.

So we have A(3) x = b (3)

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11
In general with A(k) x = b (k) already formed, multiply both sides by
 
1 0 ··· 0 ··· 0
 0
 1 ··· 0 ··· 0  
 .. .. .. .. .. .. 
 . . . . . . 
(k)
 
M =  0 0 ··· 1 ··· 0  
 0
 0 · · · −mk+1,k · · · 0  
 .. .. .. .. .. .. 
 . . . . . . 
0 0 ··· −mn,k ··· 1

A(k+1) = M (k) A(k) = M (k) · · · M (1) A

b (k+1) = M (k) b (k) = M (k) · · · M (1) b
So we have A(k+1) x = b (k+1)

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

The process ends with A(n) x = b (n) where
 (1) (1) (1)

a a12 · · · a1n
 11 (2) (2)
a22 · · ·

(n)
 0 a2n 
A = .  .. .. .. 
 .. . . .


(n)
0 0 ··· ann

is upper-triangular matrix.

Set U = A(n)

Therefore = M (n−1) M (n−2) · · · M (1) A and

L = = [M (n−1) M (n−2) · · · M (1) ]−1

= = [M (1) ]−1 [M (2) ]−1 · · · [M (n−1) ]−1

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Note each M (k) is lower triangular. So [M (k) ]−1 is lower triangular.
=⇒ L is lower triangular.

Also A = LU.
 
1 0 0 ··· 0

 −m21 1 0 ··· 0 

(1)
 −m31 0 1 ··· 0 
M =
 
.. .. .. . . .. 

 . . . . . 

 −mn−1,1 0 0 ··· 0 
−mn1 0 0 ··· 1
 
1 0 0 ··· 0
 m21
 1 0 ··· 0  
(1) −1
 m31 0 1 ··· 0 
[M ] = 
 
.. .. .. . . .. 

 . . . . . 

 mn−1,1 0 0 ··· 0 
mn1 0 0 ··· 1
and so on
Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11
One can prove that
 
1 0 0 ··· 0

 m21 1 0 ··· 0 

 m31 m32 1 ··· 0 
L=
 
.. .. .. .. ..

 . . . . .

 mn−1,1 mn−1,2 mn−1,3 · · · 0 
mn1 mn2 mn3 ··· 1

Example:-

x1 + x2 + 0x3 + 3x4 = 4
2x1 + x2 − x3 + x4 = 1
3x1 − x2 + −x3 + 2x4 = −3
−x1 + 2x2 + 3x3 − x4 = 4

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Step 1 :- R2 − 2R1 , R3 − 3R1 , R4 + R1 gives
 
1 1 0 3
0 −1 −1 −5
A(2) = 
 

 0 −4 −1 −7 
0 3 3 2
 
1 0 0 0
 2 1 0 0 
L=  3 0 1 0 


−1 0 0 1

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

Step 2 R3 − 4R2 , R4 + 3R2 gives
 
1 1 0 3
0 −1 −1 −5 
A(3) = 

=U
 0 0 3 13 
0 0 0 −13
 
1 0 0 0
 2 1 0 0 
L=  3

4 1 0 
−1 −3 0 1
 
8
 7 
Solve Ax = b = 
 14 


−7

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

 Set y = Ux
Ly = b

   

1 0 0 0 y1 8
 2 1 0 0   y2   7 
  = 
 3 4 1 0   y3   14 
−1 −3 0 1 y4 −7

y1 = 8
2y1 + y2 = 7 =⇒ y2 = −9
3y1 + 4y2 + y3 = 14 =⇒ y3 = 26
−y1 − 3y2 + y4 = −7 =⇒ y4 = −26

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11

We then solve Ux = y
    
1 1 0 3 x1 8
 0 −1 −1 −5   x2   −9 
  x3  =  26
    
 0 0 3 13 
0 0 0 −13 x4 −26

We use “back substitution”

−13x4 = −26 =⇒ x4 = 2
3x3 + 13x4 = 26 =⇒ x3 = 0
−x2 − x3 − 5x4 = −9 =⇒ x2 = −1
x1 + x2 + 3x4 = 8 =⇒ x1 = 3

Instructor: Prof. Tony J. Puthenpurakal Numerical Analysis : [ MA214 ] Lecture 11