University of Tripoli

Mechanical & Industrial Engineering Department

www.me.uot.edu.ly

ME626 Advance Numerical Analysis

Lecture 6: Solution to a System of

Linear Algebraic Equations
Instructor: Samah Alghoul
Outlines: Solution to a System of Linear Algebraic Equations

➢II. Iterative Solvers (cont’d)

▪ Gauss–Seidel Method

▪ Line-by-line Methods

▪ Alternating Direction Implicit (ADI) Method

▪ Stone’s Strongly Implicit Method

▪ Method of Steepest Descent (MSD)

▪ Conjugate Gradient (CG)

▪ Conjugate Gradient Squared (CGS)

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 2

Gauss–Seidel Method
Gauss–Seidel Method

› Is also a point-wise iteration method.

› Instead of always using previous iteration values, whenever an updated value
becomes available, it is immediately used.

› Where Nnbu,k denotes the number of neighboring nodes to node k that have
already been updated,
› Nnb,k − Nnbu,k is the number of neighboring nodes to node k that have not been
updated and are treated explicitly.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 4

Gauss–Seidel Method

Gauss–Seidel Method Jacobi Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 5

Gauss–Seidel Method

Nodes treated explicitly

Nodes treated implicitly

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 6
Gauss–Seidel Method

Algorithm: Gauss–seidel Method

Step 1: Guess values of φ at all nodes, φ(0)
› If any of the boundaries have Dirichlet boundary conditions, the guessed values
for the boundary nodes corresponding to that boundary must be equal to the
prescribed boundary values.
Step 2: Set φ(n+1) = φ(n) and apply the Gauss–Seidel update formula.
› For the interior nodes, this yields

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 7

Gauss–Seidel Method

Step 3: Compute the residual vector using φ(n+1), and then compute R2(n+1).
Step 4: Monitor convergence, i.e., check if R2(n+1) < εtol? If YES, then go to Step
5. If NO, then go to Step 2.
Step 5: Stop iteration and postprocess the results.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 8

Gauss–Seidel Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 9

Gauss–Seidel Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 10

Gauss–Seidel Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 11

Gauss–Seidel Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 12

Gauss–Seidel Method
41x41 81x81

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 13

Gauss–Seidel Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 14

Gauss–Seidel Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 15

Gauss–Seidel Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 16

Gauss–Seidel Method

› In summary, two popular point-wise iteration schemes have been presented and
demonstrated.
› The Gauss–Seidel method was found to be twice as effective as the Jacobi
method.
› Both schemes have the advantage that they are simple to implement and are
applicable to any mesh topology.
› Although the convergence is slow, the cost per iteration of both methods is also
very low, making them attractive choices.
› However, their major shortcoming is that both schemes scale poorly, and the
number of iterations go up by a factor of four when the number of nodes is
increased by a factor of four.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 17

Relaxation

› Relaxation represents a slight modification of the Gauss-Seidel method.

xnew = λxnew + (1 − λ)xold
▪ where λ is a weighting factor that is assigned a value between 0 and 2.
› If λ = 1, (1 − λ) is equal to 0 and the result is unmodified.
› λ =[0, 1], Underrelaxation.
– Used to make a nonconvergent system converge or to fasten convergence by
dampening out oscillations.
› λ=[1, 2] , Overrelaxation
– Used to accelerate the convergence of an already convergent system.
– The approach is also called successive overrelaxation, or SOR.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 18

Relaxation

› The choice of a proper value for λ is highly problem-specific and is often

determined empirically.
› For a single solution of a set of equations it is often unnecessary.
› However, if the system under study is to be solved repeatedly, the efficiency
introduced by a wise choice of λ can be extremely important.
› Good examples are the very large systems of partial differential equations that
often arise when modeling continuous variations of variables.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 19

Relaxation

› Example 3.1: GS Vs SOR

30000

25000
SOR 1.7 GS 1 No if statement for lamda
20000
Time, sec

15000

10000

5000

0
0 2000 4000 6000 8000 10000 12000
Number of equtions

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 20

Relaxation

› Example 3.1: Use of extra if statement

35000

30000

25000 GS 1 No if statement for lamda

Time, sec

20000 GS 1 with if statement

15000

10000

5000

0
0 2000 4000 6000 8000 10000 12000
Number of equtions

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 21

Line-by-line Method
Line-by-line Method

› For a structured mesh, the 21 22 23 24 25

computational domain could be
thought of as a series of constant 16 17 18 19 20
– i lines
– j lines. 11 12 13 14 15

› Finding the solution at the nodes of

any line represents a 1D 6 7 8 9 10
calculation assuming the other
nodes that do not belong to the line 1 2 3 4 5
are treated explicitly.
Line-by-line Method

› Let us consider the finite difference equation given by

• which may be rewritten as:

• where the equation has been rearranged such that;

▪ Terms belonging to j row on the left-hand side.
▪ Terms belonging to j+1 and j-1 on the right-hand side.
Line-by-line Method

› Similarly, it is also possible to sweep the computational domain column-wise,

in which case, the tridiagonal system of equations would be as follows:
Line-by-line Method

Algorithm: Line-by-line Method with Row-wise Sweep for a 2D

Problem

Step 1: Guess values of φ at all nodes. Set φ(n + 1) = φ(n).

Step 2: Set up a tridiagonal matrix system to solve for the j-th row.
– This entails filling up matrix elements for d, c, a, and Q

Step 3: Solve the tridiagonal system of equations using the algorithm described
before and update the value of φ(n + 1) of the j-th row
Line-by-line Method

Step 4: Repeat Steps 2 and 3 for all rows (values of j).

Step 5: Compute the residual vector using φ(n + 1), and then compute R2(n + 1).

Step 6: Monitor convergence, i.e., check if R2(n + 1) < εtol? If YES, then go to Step
7. If NO, then go to Step 2.

Step 7: Stop iteration and postprocess the results.

Line-by-line Method

› As to which type of sweep (row versus column) is more effective, depends on a

number of factors, include:

▪ the global aspect ratio of the geometry (whether it is long or short or fairly
isotropic),

▪ the aspect ratio of the grid,

▪ The boundary conditions in the problem.

Line-by-line Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 29

Line-by-line Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 30

Line-by-line Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 31

 Alternating Direction
Implicit (ADI) Method
Alternating Direction Implicit (ADI) Method

› The developer has no prior knowledge of what geometry the solver might be
applied to, what mesh might be used, or what boundary conditions might be
applied.
› To avoid drastic changes in convergence behavior with direction of sweep, it is
customary to alter the direction of sweep at each iteration.
› Hence, a row-wise sweep of the whole domain is followed by a column-wise
sweep and vice versa, where each row-wise or column-wise sweep of the entire
computational domain counts as one iteration.
› This method is known as the alternating direction implicit (ADI) method.
› The method is popular because it is easy to implement, while at the same time,
yields reasonably rapid convergence at a low computational cost.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 34

Alternating Direction Implicit (ADI) Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 35

Alternating Direction Implicit (ADI) Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 36

Alternating Direction Implicit (ADI) Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 37

Alternating Direction Implicit (ADI) Method
1.00E+05

1.00E+04

1.00E+03 ADI

1.00E+02 R

1.00E+01 C
Residual, R2

1.00E+00

1.00E-01

1.00E-02

1.00E-03

1.00E-04

1.00E-05

1.00E-06
0 50 100 150 200 250 300 350 400 450
Number of iteration

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 38

Line-by-line Method - Summary

› Thus far, three iterative solvers with increasing degrees of implicitness have
been discussed.
› The convergence was found to improve with an increase in the degree of
implicitness.
› Out of the three solvers, the line-by-line method is applicable only to structured
mesh computations.
› The three solvers, discussed thus far, are often referred to as classical iterative
solvers, in which the explicit versus implicit terms are clearly identifiable.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 39

Stone’s Strongly Implicit
Method
Stone’s Strongly Implicit Method

› Considered one of the most efficient solvers for banded systems arising out of
discretization of a multidimensional PDE.
› Let us consider the system of equations shown

› Using the global nodal index k: Such that the diagonal is positive

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 41

Stone’s Strongly Implicit Method

› we further write this equation in the following five-band form:

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 42

Stone’s Strongly Implicit Method

› The correction form of the equations [A][φ]=[Q] may be written as

› which may be simplified to the form

▪ Where
▪ The correction in the solution from

▪ The residual in the (n)-th iteration

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 43

Stone’s Strongly Implicit Method

› Next, Stone proposed factorizing the matrix [A] into a lower triangular matrix
[L], and an upper triangular matrix [U].

› That is to split the solution procedure into two steps.

Solve for Yn

Solve for n

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 44

Stone’s Strongly Implicit Method

› LU decomposition (or factorization) is based upon that an exact factorization,

[A] = [L][U], exists.
› Stone proposed a factorization in which the factors of [A] are not only
triangular, but also banded, as follows:

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 45

Stone’s Strongly Implicit Method

› The product [L][U] results in a matrix › The relationship between the seven
that has seven bands bands and the coefficients of the [L] and
[U] matrices

[A]
ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 46
Stone’s Strongly Implicit Method

› [A] has five bands, while the product of [L] and [U] results in seven bands.
› This implies that Stone’s proposition of factorizing [A] exactly into [L] and [U]
matrices, with each of these matrices having only three bands, is not realizable.
› Stone realized that even if [A] can be factorized approximately into [L] and
[U], it still serves the needs of the task at hand.
› If the factorization [A] = [L][U] were exact, the correction [(n)] would be such
that, when added to [φ(n)], the result would be the exact solution to [A][φ] =
[Q].
› Therefore, Stone set his goal on approximate factorization of the matrix [A]

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 47

Stone’s Strongly Implicit Method

› [A] only has five terms → The last two terms are missing
› Therefore, in order to establish an equivalence between [A] and [L][U], the last
two terms shown in the expression above must be somehow eliminated.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 48

Stone’s Strongly Implicit Method

› Referring to the nine-point stencil shown, we may write

›

› derivatives may be approximated as

› Therefore,

› A similar procedure is followed to derive

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 49

Stone’s Strongly Implicit Method

› Substitute into the last two terms

›  = Stone’s factor, should have a value close to unity, and could be adjusted
arbitrarily to attain faster convergence.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 50

Stone’s Strongly Implicit Method

› Simplifying the right-hand side and substituting it into

› yields approximately LU-decomposed form of correction equation

› Comparing with the original form of the correction equation

› the following relationships may be established:

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 51

Stone’s Strongly Implicit Method

› Substituting into

› followed by some algebraic manipulations, yield the following relationships:

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 52

Stone’s Strongly Implicit Method

› At first glance, the relationships provided in are implicit.

› For example, computing b using the first relationship requires e, while
computing e using the fourth relationship requires b.
› This ambiguity can be resolved by the fact that all coefficients with a 0 or
negative index must be first assigned a value of zero.
› Thus, computation of b1, for example, does not require the value of e, since
e1− N is zero.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 53

Stone’s Strongly Implicit Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 54

Stone’s Strongly Implicit Method

Algorithm: Stone’s Strongly Implicit Method For A 2d Problem

Step 1: Guess values of φ at all nodes
Step 2: Compute link coefficients (AO, AE, AW, AN, and, AS) and fill up the
elements of vectors E, F, D, B, H, and Q.
Step 3: Compute the elements of the [L] and [U] matrices using [Snippet 3.6].
Step 4: Compute residual vector using
and then compute R2(n).
Step 5: Solve (forward substitution) to obtain [Y(n)]

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 55

Stone’s Strongly Implicit Method

Step 6: Solve (backward substitution) to obtain [(n)].

Step 7: Update solution: [φ(n+1)] = φ(n) + [δ(n) ].
Step 8: Monitor convergence, i.e., check if R2(n) < εtol? If YES, then go to Step 9.
If NO, then go to Step 4 (i.e., proceed to next iteration).
Step 9: Stop iteration and postprocess the results.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 56

Stone’s Strongly Implicit Method
› The two most time-consuming steps in the above algorithm are the forward and backward
substitution steps (Steps 5 and 6 in the above algorithm).
› Earlier, a remark was made that the two-step solution procedure, using forward and backward
substitution steps, is considerably more efficient than the full-blown Gaussian elimination
process because of the absence of the time-consuming forward elimination step.
› According to the discussion in Section 3.1.1, even forward or backward substitutions have long
operation counts that scale as ∼K 2, which would make Stone’s method quite inefficient for
large-scale computations.
› Stone had the foresight to recognize this shortcoming, and hence, proposed that even the [L]
and [U] matrices be banded so that the operation count is much lower.
› Since both the [L] and [U] matrices have only three bands each, the long operation count in the
forward substitution step is only ∼3K, while the long operation count in the backward
substitution step is only ∼2K. Thus, Stone’s method is theoretically linearly scalable as far as
the long operation count is concerned, making it one of the most attractive methods for banded
linear systems.

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 57

Stone’s Strongly Implicit Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 58

Stone’s Strongly Implicit Method

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 59

 600.0

500.0 Gauss–Seidel
SOR
400.0 Gauss Elimination
ADI
Time, min

300.0

200.0

100.0

0.0
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Number of equtions

ME 626 ADVANCED NUMERICAL METHODS 7/13/2019 60

61