2.

Solution of Linear Equations

2.2 Frontal Solution (Banded, Symmetric Banded Matrices)
• It is also known as the wave front solution
• For symmetric and banded coefficient matrix (the stiffness matrix), it is useful.
• Here, assembly of the system stiffness matrix is combined with the solution phase.
• The method results in a considerable reduction in computer memory requirements, especially for
large models.
• The stiffness matrix is banded and sparse (many zero-valued terms)
• In the frontal solution technique, the entire system stiffness matrix is not assembled as such.
Instead, method utilizes the fact that a degree of freedom (an unknown) can be eliminated when
the rows and columns of the stiffness matrix corresponding to that degree of freedom are
complete.
• Eliminating a degree of freedom means that we can write an equation for that degree of freedom
in terms of other degrees of freedom and forcing functions. When such an equation is obtained,
it is written to a file and removed from memory.
• The net result is triangularization of the system stiffness matrix and the solutions are obtained by
simple back substitution.

Figure: A system of bar elements (An assembly of bars)

Frontal Method
• Start by defining a 6 × 6 null matrix [K] and proceed with the assembly step, taking the
elements in numerical order. Adding the element stiffness matrix for element 1 to the
system matrix, we obtain

Since U1 is associated only with element 1, displacement

U1 appears in none of the other equations and can be
eliminated now. The first and second row of Equations are
kU1  kU 2  F1 &  kU1  kU 2  F2
First eqn can be solved for U1 once U2 is known.
Mathematically eliminating U1 from the second row, we
have Displacement U2 does not appear in any remaining
equations and is now eliminated to obtain
 In sequence, processing the remaining elements and following the elimination procedure
results in

Observe that the procedure has triangularized the system stiffness matrix without formally assembling
that matrix. If we take out the constraint equation, the remaining equations are easily solved by back
substitution. Here, forces are known.

2.3 Data Storage and Memory Optimization

Efficient data storage with minimum space in computer memory helps to occupy less space in
computer memory that increases computer’s efficiency.
 2. Iterative Solutions
a) Gauss-Siedel Method
b) Conjugate Gradient Method

a) Gauss-Seidel Method
An iterative method.
Basic Procedure:
-Algebraically solve each linear equation for xi
-Assume an initial guess solution array
-Solve for each xi and repeat
-Use absolute relative approximate error after each iteration
to check if error is within a pre-specified tolerance.
Gauss-Seidel Method
Why?
The Gauss-Seidel Method allows the user to control round-off error.

Elimination methods such as Gaussian Elimination and LU

Decomposition are prone to prone to round-off error.

Also: If the physics of the problem are understood, a close initial

guess can be made, decreasing the number of iterations needed.
Gauss-Seidel Method
Algorithm
A set of n equations and n unknowns:
If: the diagonal elements are
a11x1  a12x2  a13x3  ...  a1n xn  b1 non-zero
a21x1  a22 x2  a23x3  ...  a2n xn  b2 Rewrite each equation solving
. .
. . for the corresponding unknown
. .
ex:
an1x1  an 2 x2  an3 x3  ...  ann xn  bn
First equation, solve for x1
Second equation, solve for x2
Gauss-Seidel Method
Algorithm
Rewriting each equation
c1  a12 x2  a13 x3   a1n xn
x1  From Equation 1
a11

c2  a21x1  a23 x3   a2 n xn
x2  From equation 2
a22
  
cn 1  an 1,1 x1  an 1, 2 x2   an 1,n  2 xn  2  an 1,n xn From equation n-1
xn 1 
an 1,n 1
cn  an1 x1  an 2 x2    an ,n 1 xn 1
xn  From equation n
ann
Gauss-Seidel Method
Algorithm
General Form of each equation
n

a
n
c1   a1 j x j cn 1  n 1, j xj
j 1 j 1
j  n 1
x1 
j 1 xn 1 
a11 an 1,n 1
n
c n   a nj x j
n
c2   a2 j x j
j 1 j 1
j n
x2 
j2
xn 
a 22 a nn
Gauss-Seidel Method
Algorithm
General Form for any row ‘i’
n
ci   aij x j
j 1
j i
xi  , i  1,2,, n.
aii

How or where can this equation be used?

Gauss-Seidel Method
Solve for the unknowns
Assume an initial guess for [X] Use rewritten equations to solve for
each value of xi.
Important: Remember to use the
 x1  most recent value of xi. Which
x  means to apply values calculated to
 2 the calculations remaining in the
current iteration.
  
 
 xn -1 
 xn 
Gauss-Seidel Method

Calculate the Absolute Relative Approximate Error

x new
x old
a i  i
new
i
100
x i

So when has the answer been found?

The iterations are stopped when the absolute relative

approximate error is less than a prespecified tolerance for all
unknowns.
Gauss-Seidel Method: Example 1
The upward velocity of a rocket
is given at three different times
Table 1 Velocity vs. Time data.

Time, t s  Velocity v m/s 

5 106.8
8 177.2
12 279.2

The velocity data is approximated by a polynomial as:

vt   a1t 2  a2t  a3 , 5  t  12.
Gauss-Seidel Method: Example 1

Using a Matrix template of the form t12 t1 1  a1   v1 

2 
t 2 t 2 1 a2   v2 
t32 t3 1  a3  v3 

 25 5 1  a1  106.8 
The system of equations becomes  64 8 1 a   177.2 
   2  
144 12 1  a3  279.2

 a1  1 
 a    2
Initial Guess: Assume an initial guess of
 2  
 a3  5
Gauss-Seidel Method: Example 1

Rewriting each equation

106.8  5a 2  a3
a1 
25
 25 5 1  a1  106.8 
 64 8 1 a   177.2  177.2  64a1  a3
   2   a2 
144 12 1  a3  279.2 8

279.2  144a1  12a 2

a3 
1
Gauss-Seidel Method: Example 1
Applying the initial guess and solving for ai
 a1  1  106.8  5(2)  (5)
a1   3.6720
 a    2 25
 2  
 a3  5 177.2  643.6720  5
a2   7.8510
Initial Guess 8
279.2  1443.6720  12 7.8510
a3   155.36
1

When solving for a2, how many of the initial guess values were used?
Gauss-Seidel Method: Example 1
Finding the absolute relative approximate error
xinew  xiold At the end of the first iteration
a i  new
100
xi
 a1   3.6720 
a    7.8510
3.6720  1.0000  2  
a 1  x100  72.76%
3.6720  a3    155.36 

 7.8510  2.0000 The maximum absolute

a  x100  125.47% relative approximate error is
2
 7.8510
125.47%

 155.36  5.0000
a 3  x100  103.22%
 155.36
Gauss-Seidel Method: Example 1
Iteration #2
Using
 a1   3.6720  the values of ai are found:
a    7.8510 106.8  5 7.8510  155.36
 2   a1   12.056
 a3    155.36  25

from iteration #1
177.2  6412.056  155.36
a2   54.882
8

279.2  14412.056  12 54.882

a3   798.34
1
Gauss-Seidel Method: Example 1

Finding the absolute relative approximate error

12.056  3.6720 At the end of the second iteration
a 1  x100  69.543%
12.056  a   12.056 
1
a    54.882
 2  
 54.882   7.8510  a3   798.54
a 2  x100  85.695%
 54.882
The maximum absolute
 798.34   155.36 relative approximate error is
a 3  x100  80.540% 85.695%
 798.34
 Gauss-Seidel Method: Example 1
Repeating more iterations, the following values are obtained

Iteration a1 a 1 % a2 a 2 % a3 a 3 %
1 3.6720 72.767 −7.8510 125.47 −155.36 103.22
2 12.056 69.543 −54.882 85.695 −798.34 80.540
3 47.182 74.447 −255.51 78.521 −3448.9 76.852
4 193.33 75.595 −1093.4 76.632 −14440 76.116
5 800.53 75.850 −4577.2 76.112 −60072 75.963
6 3322.6 75.906 −19049 75.972 −249580 75.931
Notice – The relative errors are not decreasing at any significant rate
 a1  0.29048
Also, the solution is not converging to the true solution of    
a
 2   19 .690 
a 3   1.0857 
Gauss-Seidel Method: Pitfall
What went wrong?
Even though done correctly, the answer is not converging to the
correct answer
This example illustrates a pitfall of the Gauss-Siedel method: not all
systems of equations will converge.

Is there a fix?
One class of system of equations always converges: One with a diagonally
dominant coefficient matrix.

Diagonally dominant: [A] in [A] [X] = [C] is diagonally dominant if:

n n
aii   aij for all ‘i’ and aii   aij for at least one ‘i’
j 1 j 1
j i j i
 Gauss-Seidel Method: Pitfall
Diagonally dominant: The coefficient on the diagonal must be at least
equal to the sum of the other coefficients in that row and at least one row
with a diagonal coefficient greater than the sum of the other coefficients
in that row.
Which coefficient matrix is diagonally dominant?

 2 5.81 34 124 34 56 

A   45 43 1  [B]   23 53 5 
123 16 1   96 34 129

Most physical systems do result in simultaneous linear equations that

have diagonally dominant coefficient matrices.
Gauss-Seidel Method: Example 2
Given the system of equations The coefficient matrix is:
12 x1  3x2- 5x3  1 12 3  5
x1  5x2  3x3  28 A   1 5 3 
3x1  7 x2  13x3  76  3 7 13 

With an initial guess of Will the solution converge using the

Gauss-Siedel method?
 x1  1
 x   0 
 2  
 x3  1
Gauss-Seidel Method: Example 2

Checking if the coefficient matrix is diagonally dominant

a11  12  12  a12  a13  3   5  8
12 3  5
A   1 5 3  a22  5  5  a21  a23  1  3  4
 3 7 13 
a33  13  13  a31  a32  3  7  10

The inequalities are all true and at least one row is strictly greater than:
Therefore: The solution should converge using the Gauss-Siedel Method
Gauss-Seidel Method: Example 2
Rewriting each equation With an initial guess of
12 3  5  a1   1   x1  1
 1 5 3  a   28  x   0 
   2    2  
 3 7 13  a3  76  x3  1

1  3 x 2  5 x3 1  30  51
x1  x1   0.50000
12 12

28  x1  3x3 28  0.5  31

x2  x2   4.9000
5 5

76  3x1  7 x2 76  30.50000  74.9000

x3  x3   3.0923
13 13
 Gauss-Seidel Method: Example 2
The absolute relative approximate error
0.50000  1.0000
a 1  100  100.00%
0.50000

4.9000  0
a 2
 100  100.00%
4.9000

3.0923  1.0000
a 3
 100  67.662%
3.0923

The maximum absolute relative error after the first iteration is 100%
Gauss-Seidel Method: Example 2
After Iteration #1
 x1  0.5000
 x   4.9000
 2  
 x3  3.0923

Substituting the x values into the After Iteration #2

equations
 x1  0.14679
1  34.9000  53.0923  x    3.7153 
x1   0.14679
12  2  
 x3   3.8118 
28  0.14679  33.0923
x2   3.7153
5

76  30.14679  74.900
x3   3.8118
13
Gauss-Seidel Method: Example 2

Iteration #2 absolute relative approximate error

0.14679  0.50000
a 1  100  240.61%
0.14679
3.7153  4.9000
a 2
 100  31.889%
3.7153
3.8118  3.0923
a 3
 100  18.874%
3.8118
The maximum absolute relative error after the first iteration is 240.61%

This is much larger than the maximum absolute relative error obtained in
iteration #1. Is this a problem?
 Gauss-Seidel Method: Example 2
Repeating more iterations, the following values are obtained
Iteration a1 a 1 % a2 a 2 % a3 a 3 %

1 0.50000 100.00 4.9000 100.00 3.0923 67.662

2 0.14679 240.61 3.7153 31.889 3.8118 18.876
3 0.74275 80.236 3.1644 17.408 3.9708 4.0042
4 0.94675 21.546 3.0281 4.4996 3.9971 0.65772
5 0.99177 4.5391 3.0034 0.82499 4.0001 0.074383
6 0.99919 0.74307 3.0001 0.10856 4.0001 0.00101

 x1  0.99919  x1  1 
The solution obtained  x    3.0001  is close to the exact solution of  x   3 .
 2    2  
 x3   4.0001   x3  4
 Gauss-Seidel Method: Example 3
Given the system of equations

3x1  7 x2  13x3  76 Rewriting the equations

x1  5x2  3x3  28 76  7 x2  13x3

x1 
12 x1  3x2  5x3  1 3
28  x1  3x3
With an initial guess of x2 
5
 x1  1
 x   0  1  12 x1  3x2
 2   x3 
 x3  1 5
Gauss-Seidel Method: Example 3
Conducting six iterations, the following values are obtained

Iteration a1 a 1 % A2 a 2 % a3 a 3 %
1 21.000 95.238 0.80000 100.00 50.680 98.027
2 −196.15 110.71 14.421 94.453 −462.30 110.96
3 −1995.0 109.83 −116.02 112.43 4718.1 109.80
4 −20149 109.90 1204.6 109.63 −47636 109.90
5 2.0364×105 109.89 −12140 109.92 4.8144×105 109.89
6 −2.0579×105 109.89 1.2272×105 109.89 −4.8653×106 109.89

The values are not converging.

Does this mean that the Gauss-Seidel method cannot be used?
Gauss-Seidel Method
The Gauss-Seidel Method can still be used
 3 7 13 
The coefficient matrix is not
diagonally dominant
A   1 5 3 
12 3  5
But this is the same set of 12 3  5
equations used in example #2,
which did converge.
A   1 5 3 
 3 7 13 

If a system of linear equations is not diagonally dominant, check to see if

rearranging the equations can form a diagonally dominant matrix.
Gauss-Seidel Method
Not every system of equations can be rearranged to have a
diagonally dominant coefficient matrix.
Observe the set of equations

x1  x2  x3  3
2 x1  3x2  4 x3  9
x1  7 x2  x3  9

Which equation(s) prevents this set of equation from having a

diagonally dominant coefficient matrix?
Gauss-Seidel Method
Summary

- Advantages of the Gauss-Seidel Method

- Algorithm for the Gauss-Seidel Method
- Pitfalls of the Gauss-Seidel Method
Gauss-Seidel Method

Questions?
2 b) Conjugate Gradient Method
The conjugate gradient method
is an iterative method that is
used to solve large symmetric
linear systems Ax=b.

Method is efficient especially

when the matrix A is sparse.

x0∈RN,r0=b−x0,p0=r0

rk=b−Axkrk=b−Axk which is the residual

at iteration k

Two non-zero vectors u and v are

conjugate (with respect to A) if
u T Av  0
Since A is symmetric and positive-definite,
the left-hand side defines an inner product

u T Av  u , v A :  Au , v  u , AT v  u , Av
 Two vectors are conjugate if and only if they are orthogonal with respect to this inner product. Being
conjugate is a symmetric relation: if u is conjugate to v, then v is conjugate to u. Suppose that
P   p1 ,..., pn 
n
is a set of n mutually conjugate vectors (with respect to A). Then P forms a basis for , and we may
express the solution x* of Ax  b in this basis:

n

x*   i pi
i 1

Based on this expansion we calculate

  
n n n

Ax*   i Api  p Ax* 

T
k  i p Api  p b 
T
k
T
k  i pkT Api
i 1 i 1 i 1

pkT b   k pkT Apk  u T v  0 & i  k : pkT Api  0 

pkT b
k  T
pk Apk
The conjugate gradient method is an iterative method.
It approximately solve systems, where n is so large since the direct method would take too much time
The initial guess for x∗ by x0 (we can assume without loss of generality that x0 = 0,
Otherwise consider the system Ax∗= b − Ax0
The solution x∗ should minimize the quadratic equation
 1 T
f ( x)  x Ax  xT b, x n
2
The existence of a unique minimizer is apparent since its second derivative is given by a symmetric positive-
definite matrix
 2 f ( x)  A
and that the minimizer (use Df(x)=0) solves the initial problem is obvious from its first derivative
f ( x)  Ax  b

Taking the first basis vector p0 (residual) to be the negative of the gradient of f at x = x0.
The gradient of f equals Ax − b. Starting with an initial guess x0, this means we take p0 = b − Ax0.
The other vectors in the basis will be conjugate to the gradient, hence the name conjugate gradient method.
Let rk be the residual at the kth step:
rk  b  Axk
rk is the negative gradient of f at x = xk, so the gradient descent method would require to move in the direction rk.
However, we insist that the directions pk be conjugate to each other. A practical way to enforce this, is by requiring
that the next search direction be built out of the current residual and all previous search directions. This gives the


following expression: piT Ark
pk  rk  T
pi
ik
pi pi
Next optimal location is given by:
xk 1  xk   k pk
 pkT (b  Axk ) pkT rk
and, k  T
 T
pk Apk pk Apk
The expression for 𝜶k is derived by minimizing the expression for f(xk+1) and minimizing it wrt 𝜶k:

The algorithm requires storage of all previous searching directions and residue vectors, as well as many
matrix-vector multiplications, and thus can be computationally expensive. However, a closer analysis of
the algorithm shows that ri is orthogonal to rj , i.e., riT rj  0 (i  j ) . Also, pi is A-orthogonal pj, i.e.,
piT Ap j  0 (i  j ). This can be regarded that as the algorithm progresses, p and r span the same Krylov subspace.
i i
Here, ri form the orthogonal basis with respect to standard inner product, and pi form the orthogonal
basis with respect to inner product induced by A. Therefore, xk can be regarded as the projection of x on
the Krylov subspace.

The algorithm is detailed below for solving Ax = b where A is a real,

symmetric, positive-definite matrix. Vector x0 can be an approximate
initial solution or 0.
Conjugate Gradient Algorithm
 Exact Solution:

Solution:
Problem: