542 CHAPTER 10 NUMERICAL METHODS

10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS

As a numerical technique, Gaussian elimination is rather unusual because it is direct. That
is, a solution is obtained after a single application of Gaussian elimination. Once a “solu-
tion” has been obtained, Gaussian elimination offers no method of refinement. The lack of
refinements can be a problem because, as the previous section shows, Gaussian elimination
is sensitive to rounding error.
Numerical techniques more commonly involve an iterative method. For example, in cal-
culus you probably studied Newton’s iterative method for approximating the zeros of a dif-
ferentiable function. In this section we look at two iterative methods for approximating the
solution of a system of n linear equations in n variables.

The Jacobi Method

The first iterative technique is called the Jacobi method, after Carl Gustav Jacob Jacobi
(1804–1851). This method makes two assumptions: (1) that the system given by

a11 x1 1 a12 x2 1 . . . 1 a1n xn 5 b1

a21 x1 1 a22 x2 1 . . . 1 a2n xn 5 b2
...

...

...
...

an1 x1 1 an2 x2 1 . . . 1 ann xn 5 bn

has a unique solution and (2) that the coefficient matrix A has no zeros on its main diago-
nal. If any of the diagonal entries a11, a22, . . . , ann are zero, then rows or columns must be
interchanged to obtain a coefficient matrix that has nonzero entries on the main diagonal.
To begin the Jacobi method, we solve the first equation for x1, the second equation for
x2, and so on, as follows.

1
x1 5 (b 2 a12 x2 2 a13x3 2 . . . 2 a1n xn)
a11 1

1
x2 5 (b 2 a21 x1 2 a23x3 2 . . . 2 a2n xn)
a22 2
...

1
xn 5 (b 2 an1 x1 2 an2x2 2 . . . 2 an,n21 xn21d
ann n

Then we make an initial approximation of the solution,

(x1, x2, x3, . . . , xn), Initial approximation

and substitute these values of xi into the right-hand side of the rewritten equations to obtain
the first approximation. After this procedure has been completed, we say that one iteration
 SECTION 10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS 543

has been performed. In the same way, the second approximation is formed by substituting
the first approximation’s x-values into the right-hand side of the rewritten equations. By
repeated iterations, we form a sequence of approximations that often converges to the
actual solution. This procedure is illustrated in the following example.

EXAMPLE 1 Applying the Jacobi Method

Use the Jacobi method to approximate the solution of the following system of linear
equations.
−5x1 − 2x2 + 3x3 5 −1
−3x1 + 9x2 + 3x3 5 −2
−2x1 − 2x2 − 7x3 5 −3
Continue the iterations until two successive approximations are identical when rounded to
three significant digits.

Solution To begin, we write the system in the form

x1 5 215 1 25 x2 2 35 x3
x2 5 229 1 39 x1 2 19 x3
x3 5 237 1 27 x1 2 17 x 2 .

Since we have no idea of the actual solution, we choose

x1 5 0, x2 5 0, x3 5 0 Initial approximation

as a convenient initial approximation. This means that the first approximation is

x1 5 215 1 25 (0) 2 35 (0) 5 20.200

x2 5 229 1 39 (0) 2 19 (0) < 20.222
x3 5 237 1 27 (0) 2 17 (0) < 20.429.

Continuing this procedure, we obtain the sequence of approximations shown in Table 10.1.

TABLE 10.1

n 0 1 2 3 4 5 6 7

x1 0.000 20.200 0.146 0.192 0.181 0.185 0.186 0.186

x2 0.000 0.222 0.203 0.328 0.332 0.329 0.331 0.331

x3 0.000 20.429 20.517 20.416 20.421 20.424 20.423 20.423

544 CHAPTER 10 NUMERICAL METHODS

Because the last two columns in Table 10.1 are identical, we conclude that to three
significant digits the solution is
x1 5 0.186, x2 5 0.331, x3 5 20.423.

For the system of linear equations given in Example 1, the Jacobi method is said to
converge. That is, repeated iterations succeed in producing an approximation that is correct
to three significant digits. As is generally true for iterative methods, greater accuracy would
require more iterations.

The Gauss-Seidel Method

We now look at a modification of the Jacobi method called the Gauss-Seidel method,
named after Carl Friedrich Gauss (1777–1855) and Philipp L. Seidel (1821–1896). This
modification is no more difficult to use than the Jacobi method, and it often requires fewer
iterations to produce the same degree of accuracy.
With the Jacobi method, the values of xi obtained in the nth approximation remain
unchanged until the entire (n 1 1) th approximation has been calculated. With the Gauss-
Seidel method, on the other hand, we use the new values of each xi as soon as they are
known. That is, once we have determined x1 from the first equation, its value is then used
in the second equation to obtain the new x2. Similarly, the new x1 and x2 are used in
the third equation to obtain the new x3, and so on. This procedure is demonstrated in
Example 2.

EXAMPLE 2 Applying the Gauss-Seidel Method

Use the Gauss-Seidel iteration method to approximate the solution to the system of
equations given in Example 1.

Solution The first computation is identical to that given in Example 1. That is, using (x1, x2, x3) 5
(0, 0, 0) as the initial approximation, we obtain the following new value for x1.
x1 5 215 1 25 (0) 2 35 (0) 5 20.200
Now that we have a new value for x1, however, we use it to compute a new value for x2.
That is,
2 3 1
x2 5 9 1 9 (20.200) 2 9 (0) < 0.156.
Similarly, we use x1 5 20.200 and x2 5 0.156 to compute a new value for x3.
x3 5 237 1 27 (20.200) 2 17 (0.156) < 20.508
Thus the first approximation is x1 5 20.200, x2 5 0.156, and x3 5 20.508. Continued
iterations produce the sequence of approximations shown in Table 10.2.
 SECTION 10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS 545

TABLE 10.2

n 0 1 2 3 4 5

x1 0.000 20.200 0.167 0.191 0.186 0.186

x2 0.000 0.156 0.334 0.333 0.331 0.331

x3 0.000 20.508 20.429 20.422 20.423 20.423

Note that after only five iterations of the Gauss-Seidel method, we achieved the same
accuracy as was obtained with seven iterations of the Jacobi method in Example 1.

Neither of the iterative methods presented in this section always converges. That is, it is
possible to apply the Jacobi method or the Gauss-Seidel method to a system of linear equa-
tions and obtain a divergent sequence of approximations. In such cases, we say that the
method diverges.

EXAMPLE 3 An Example of Divergence

Apply the Jacobi method to the system

7x1 − 5x2 5 −4
7x1 − 5x2 5 −6,
using the initial approximation (x1, x2) 5 (0, 0), and show that the method diverges.

Solution As usual, we begin by rewriting the given system in the form

x1 5 −4 + 5x2
x2 5 −6 + 7x1.
Then the initial approximation (0, 0) produces
x1 5 −4 + 5(0) 5 −4
x2 5 −6 + 7(0) 5 −6
as the first approximation. Repeated iterations produce the sequence of approximations
shown in Table 10.3.

TABLE 10.3

n 0 1 2 3 4 5 6 7

x1 0 24 234 2174 21,244 26,124 242,874 2214,374

x2 0 26 234 2244 21,244 28,574 242,874 2300,124

546 CHAPTER 10 NUMERICAL METHODS

For this particular system of linear equations we can determine that the actual solution is
x1 5 1 and x2 5 1. Thus we see from Table 10.3 that the approximations given by the
Jacobi method become progressively worse instead of better, and we conclude that the
method diverges.

The problem of divergence in Example 3 is not resolved by using the Gauss-Seidel

method rather than the Jacobi method. In fact, for this particular system the Gauss-Seidel
method diverges more rapidly, as shown in Table 10.4.

TABLE 10.4

n 0 1 2 3 4 5

x1 0 24 2174 26,124 2214,374 27,503,124

x2 0 234 21,224 242,874 21,500,624 252,521,874

With an initial approximation of (x1, x2) 5 (0, 0), neither the Jacobi method nor the
Gauss-Seidel method converges to the solution of the system of linear equations given in
Example 3. We now look at a special type of coefficient matrix A, called a strictly diago-
nally dominant matrix, for which we are guaranteed that both methods will converge.

Definition of Strictly An n 3 n matrix A is strictly diagonally dominant if the absolute value of each entry
Diagonally Dominant on the main diagonal is greater than the sum of the absolute values of the other entries
in the same row. That is,
Matrix
|a | > |a | 1 |a | 1 . . . 1 |a |
11 12 13 1n

|a | > |a | 1 |a | 1 . . . 1 |a |
22 21 23 2n
...

|a | > |a | 1 |a | 1 . . . 1 |a
nn n1 n2 n,n21 |.

EXAMPLE 4 Strictly Diagonally Dominant Matrices

Which of the following systems of linear equations has a strictly diagonally dominant
coefficient matrix?
(a) 3x1 − 5x2 5 −4
2x1 + 5x2 5 −2
(b) 4x1 + 2x2 − 5x3 5 −1
4x1 + 2x2 + 2x3 5 −4
3x1 − 5x2 + 5x3 5 −3
 SECTION 10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS 547

Solution (a) The coefficient matrix

21
32 4
3
A5
5

|| | | || ||
is strictly diagonally dominant because 3 > 21 and 5 > 2 .
(b) The coefficient matrix
21

3 4
4 2
A5 1 0 2
3 25 1
is not strictly diagonally dominant because the entries in the second and third rows do
not conform to the definition. For instance, in the second row a21 5 1, a22 5 0, a23 5 2,
and it is not true that a22 > a21 1 a23 . If we interchange the second and third rows
| | | | | |
in the original system of linear equations, however, then the coefficient matrix becomes
21

3 4
4 2
A9 5 3 25 1 ,
1 0 2
and this matrix is strictly diagonally dominant.

The following theorem, which we list without proof, states that strict diagonal dominance
is sufficient for the convergence of either the Jacobi method or the Gauss-Seidel method.

Theorem 10.1 If A is strictly diagonally dominant, then the system of linear equations given by Ax 5 b
has a unique solution to which the Jacobi method and the Gauss-Seidel method will con-
Convergence of verge for any initial approximation.
the Jacobi and
Gauss-Seidel Methods

In Example 3 we looked at a system of linear equations for which the Jacobi and Gauss-
Seidel methods diverged. In the following example we see that by interchanging the rows
of the system given in Example 3, we can obtain a coefficient matrix that is strictly diago-
nally dominant. After this interchange, we are assured of convergence.

EXAMPLE 5 Interchanging Rows to Obtain Convergence

Interchange the rows of the system

7x1 − 5x2 5 −4
7x1 − 5x2 5 −6
to obtain one with a strictly diagonally dominant coefficient matrix. Then apply the Gauss-
Seidel method to approximate the solution to four significant digits.
548 CHAPTER 10 NUMERICAL METHODS

Solution We begin by interchanging the two rows of the given system to obtain
7x1 − 5x2 5 −6
7x1 − 5x2 5 −4.
Note that the coefficient matrix of this system is strictly diagonally dominant. Then we
solve for x1 and x2 as follows.
x1 5 67 1 17 x2
4 1
x2 5 5 1 5 x1
Using the initial approximation (x1, x2) 5 (0, 0), we obtain the sequence of approximations
shown in Table 10.5.

TABLE 10.5

n 0 1 2 3 4 5

x1 0.0000 0.8571 0.9959 0.9999 1.000 1.000

x2 0.0000 0.9714 0.9992 1.000 1.000 1.000

Thus we conclude that the solution is x1 5 1 and x2 5 1.

Do not conclude from Theorem 10.1 that strict diagonal dominance is a necessary con-
dition for convergence of the Jacobi or Gauss-Seidel methods. For instance, the coefficient
matrix of the system
−4x1 + 5x2 5 1
−4x1 + 2x2 5 3
is not a strictly diagonally dominant matrix, and yet both methods converge to the solution
x1 5 1 and x2 5 1 when we use an initial approximation of (x1, x2) 5 (0, 0). (See
Exercise 21.)
 SECTION 10.2 EXERCISES 549

SECTION 10.2 ❑ EXERCISES

In Exercises 1– 4, apply the Jacobi method to the given system of 19. Interchange the rows of the system of linear equations in
linear equations, using the initial approximation sx1, x2, . . . , xnd 5 Exercise 11 to obtain a system with a strictly diagonally dom-
(0, 0, . . . , 0). Continue performing iterations until two successive inant coefficient matrix. Then apply the Gauss-Seidel method
approximations are identical when rounded to three significant digits. to approximate the solution to two significant digits.
20. Interchange the rows of the system of linear equations in
1. 3x1 2 4x2 5 2 2. 24x1 1 2x2 5 26 Exercise 12 to obtain a system with a strictly diagonally dom-
1. 3x1 1 4x2 5 5 2. 23x1 2 5x2 5 21 inant coefficient matrix. Then apply the Gauss-Seidel method
3. 2x1 2 3x2 1 3x3 5 22 4. 4x1 1 7x2 1 2x3 5 27 to approximate the solution to two significant digits.
3. 2x1 2 3x2 1 3x3 5 22 4. 4x1 2 7x2 1 2x3 5 22 In Exercises 21 and 22, the coefficient matrix of the given system
3. 2x1 1 3x2 2 3x3 5 26 4. 3x1 2 7x2 1 4x3 5 11 of linear equations is not strictly diagonally dominant. Show that
5. Apply the Gauss-Seidel method to Exercise 1. the Jacobi and Gauss-Seidel methods converge using an initial ap-
proximation of sx1, x2, . . . , xnd 5 s0, 0, . . . , 0d.
6. Apply the Gauss-Seidel method to Exercise 2.
7. Apply the Gauss-Seidel method to Exercise 3. 21. 24x1 1 5x2 5 1 22. 4x1 1 2x2 2 2x3 5 0
8. Apply the Gauss-Seidel method to Exercise 4. 21. 24x1 1 2x2 5 3 22. 4x1 2 3x2 2 2x3 5 7
In Exercises 9–12, show that the Gauss-Seidel method diverges for 22. 3x1 2 2x 2 1 4x3 5 5
the given system using the initial approximation sx1, x2, . . . , xnd 5
C In Exercises 23 and 24, write a computer program that applies the
s0, 0, . . . , 0d.
Gauss-Siedel method to solve the given system of linear equations.
9. 2x1 2 2x2 5 21 10. 2x1 1 4x2 5 1
23. 4x1 1 6x2 2 2x3 1 5x4 1 6x5 1 5x6 1 4x7 1 5x8 5 213
9. 2x1 1 2x2 5 23 3x1 2 2x2 5 2
23. 4x1 1 6x2 2 2x3 1 5x4 2 6x5 1 5x6 1 4x7 1 5x8 5 126
11. 2x1 2 3x2 2 10x3 5 27 12. x1 1 3x2 2 2x3 5 5 23. 4x1 1 6x2 1 5x3 1 5x4 2 6x5 1 5x6 1 4x7 1 5x8 5 125
11. 3x1 1 3x2 2 10x3 5 29 12. 3x1 2 3x2 2 2x3 5 5 23. 4x1 1 2x2 1 5x3 1 5x4 2 6x5 1 5x6 2 4x7 2 5x8 5 120
11. 3x1 2 3x2 1 10x3 5 13 12. x2 1 2x3 5 1 23. 4x1 1 6x2 12x3 2 5x4 1 6x5 2 5x6 1 4x7 2 5x8 5 212
In Exercises 13–16, determine whether the given matrix is strictly 23. 4x1 1 6x2 12x3 2 5x4 2 6x5 1 5x6 1 4x7 2 5x8 5 212
diagonally dominant. 23. 4x1 1 6x2 2 2x3 1 2x4 2 6x5 1 5x6 1 4x7 2 5x8 5 122
23. 4x1 1 6x2 2 2x3 1 2x4 2 6x5 1 5x6 2 4x7 1 5x8 5 122
21 22
33 4 3 4
2 1
13. 14.
5 0 1 24. 4x1 2 4x2 2 4x3 2 4x4 2 4x5 2 4x6 2 4x7 2 4x8 5 18
21 24. 2x1 1 4x2 2 4x3 2 4x4 2 4x5 2 4x6 2 4x7 2 4x8 5 18

3 4 3 4
12 6 0 7 5
15. 2 23 2 16. 1 24 1 24. 4x1 2 2x2 1 4x3 2 4x4 2 4x5 2 4x6 2 4x7 2 4x8 5 14
0 6 13 0 2 23 24. 4x1 2 4x2 2 2x3 1 4x4 2 4x5 2 4x6 2 4x7 2 4x8 5 14
24. 4x1 2 4x2 2 4x3 2 2x4 1 4x5 2 4x6 2 4x7 2 4x8 5 26
17. Interchange the rows of the system of linear equations in 24. 4x1 2 4x2 2 4x3 2 4x4 2 2x5 1 4x6 2 4x7 2 4x8 5 16
Exercise 9 to obtain a system with a strictly diagonally domi- 24. 4x1 2 4x2 2 4x3 2 4x4 2 4x5 22x6 1 4x7 2 4x8 5 10
nant coefficient matrix. Then apply the Gauss-Seidel method
24. 4x1 2 4x2 2 4x3 2 4x4 2 4x5 2 4x6 2 2x7 1 4x8 5 32
to approximate the solution to two significant digits.
18. Interchange the rows of the system of linear equations in
Exercise 10 to obtain a system with a strictly diagonally dom-
inant coefficient matrix. Then apply the Gauss-Seidel method
to approximate the solution to two significant digits.