JACOBI and GAUSS-SEIDEL METHOD

The Jacobi method is a method of solving a matrix equation on a matrix that has no zeros along its
main diagonal (Bronshtein and Semendyayev 1997, p. 9!". #a$h diagonal element is solved for, and an
approximate value plugged in. The pro$ess is then iterated until it $onverges. This algorithm is a
stripped%do&n version of the 'a$o(i transformation method of matrix diagonalization.
The 'a$o(i method is easily derived (y examining ea$h of the equations in the linear system of
equations in isolation. )f, in the th equation*
solve for the value of &hile assuming the other entries of remain fixed. This gives
&hi$h is the 'a$o(i method.
)n this method, the order in &hi$h the equations are examined is irrelevant, sin$e the 'a$o(i method
treats them independently. The definition of the 'a$o(i method $an (e expressed &ith matri$es as*
&here the matri$es , , and represent the diagonal, stri$tly lo&er triangular, and stri$tly upper
triangular parts of , respe$tively.
The +auss%Seidel method ($alled Seidel,s method (y 'effreys and 'effreys 19, p. -./" is a te$hnique
for solving the equations of the linear system of equations one at a time in sequen$e, and uses
previously $omputed results as soon as they are availa(le,
There are t&o important $hara$teristi$s of the +auss%Seidel method should (e noted. 0irstly, the
$omputations appear to (e serial. Sin$e ea$h $omponent of the ne& iterate depends upon all previously
$omputed $omponents, the updates $annot (e done simultaneously as in the 'a$o(i method. Se$ondly,
the ne& iterate depends upon the order in &hi$h the equations are examined. )f this ordering is
$hanged, the components of the ne& iterates (and not 1ust their order" &ill also $hange.
)n terms of matri$es, the definition of the +auss%Seidel method $an (e expressed as
&here the matri$es , , and represent the diagonal, stri$tly lo&er triangular, and stri$tly upper
triangular parts of , respe$tively.
The +auss%Seidel method is appli$a(le to stri$tly diagonally dominant or symmetri$ positive definite
matri$es 2.
The Gauss-Seidel method is a te$hnique used to solve a linear system of equations. The method is
named after the +erman mathemati$ians 3arl 0riedri$h +auss and 4hilipp 5ud&ig von Seidel. The
method is similar to the 'a$o(i method (and li6e&ise diagonal dominan$e of the system is suffi$ient to
ensure $onvergen$e, meaning the method &ill &or6".
7e see6 the solution to a set of linear equations, expressed in matrix terms as
The +auss%Seidel iteration is
&here D, 8 L, and 8 U represent the diagonal, lo&er triangular, and upper triangular parts of the
$oeffi$ient matrix A and k is the iteration $ount. This matrix expression is mainly of a$ademi$ interest,
and is not used to program the method. 9ather, an element%(ased approa$h is used*
:ote that the $omputation of uses only those elements of that have already (een
$omputed and only those elements of that have yet to (e advan$ed to iteration k ; 1. This means
that no additional storage is required, and the $omputation $an (e done in pla$e ( repla$es
". 7hile this might seem li6e a rather minor $on$ern, for large systems it is unli6ely that every iteration
$an (e stored. Thus, unli6e the 'a$o(i method, &e do not have to do any ve$tor $opying should &e &ish
to use only one storage ve$tor. The iteration is generally $ontinued until the $hanges made (y an
iteration are (elo& some toleran$e.
:o& let and (i < 1 * n". 7e &ill $onsider the solution of the system of equations

(y an iterative method.
Deinition! The approximation or the approximate value of the solution of system (1" is a ve$tor that
in $ertain sense differs little from the ve$tor 5et us represent system (1" in the form
'a$o(i,s iterative pro$ess is defined (y the algorithm

The +auss%Seidel iterative pro$ess is defined (y the algorithm

)n $ase of (oth 'a$o(i,s and the +auss%Seidel iterative pro$esses the transition from the approximation
of the solution of system (1" to the next approximation $an (e
des$ri(ed using the matri$es
and &hile A<L;D;U. 0or example, 'a$o(i,s algorithm $an (e represented
as

&here MJ<D and NJ<%(L;U". The +auss%Seidel algorithm (-" $an (e represented as

&here MG<D;L and NG<%U.
E"am#le! 5et us solve the system , &here
(y 'a$o(i,s method.
5et us represent the matrix A in the form
7e &ill form the matri$es Mj and Nj*
The algorithm of 'a$o(i,s iterative pro$ess $an (e given in the form
Sin$e
and
then
)f &e ta6e for the initial approximation then &e shall have
*
and so on (the exa$t solution of this equation is ".
GAUSSIA$ ELIMI$ATIO$
)n mathemati$s, Gaussian elimination (not to (e $onfused &ith +auss='ordan elimination", named after
3arl 0riedri$h +auss, is an algorithm in linear alge(ra for determining the solutions of a system of linear
equations, for determining the ran6 of a matrix, and for $al$ulating the inverse of an inverti(le square
matrix. 0ormally, the +aussian elimination algorithm ta6es an ar(itrary system of linear equations as
input and returns its solution ve$tor, if it exists and is unique. #quivalently, the algorithm ta6es an
ar(itrary matrix and redu$es it to redu$ed ro& e$helon form (also 6no&n as row canonical form".
#lementary operations are used throughout the algorithm.
The pro$ess of +aussian elimination has t&o parts. The first part (0or&ard #limination" redu$es a given
system to either triangular or e$helon form, or results in a degenerate equation &ith no solution,
indi$ating the system has no solution. This is a$$omplished through the use of elementary operations.
The se$ond step (Ba$6&ard #limination" uses (a$6%su(stitution to find the solution of the system a(ove.
Stated equivalently for matri$es, the first part redu$es a matrix to ro& e$helon form using elementary
operations &hile the se$ond redu$es it to redu$ed ro& e$helon form, or ro& $anoni$al form.
2nother point of vie&, &hi$h turns out to (e very useful to analyze the algorithm, is that +aussian
elimination $omputes a matrix de$omposition. The three elementary operations used in the +aussian
elimination (multiplying ro&s, s&it$hing ro&s, and adding multiples of ro&s to other ro&s" amount to
multiplying the original matrix &ith inverti(le matri$es from the left. The first part of the algorithm
$omputes an 5> de$omposition, &hile the se$ond part &rites the original matrix as the produ$t of a
uniquely determined inverti(le matrix and a uniquely determined redu$ed ro&%e$helon matrix.
E"am#le* Suppose the goal is to find and des$ri(e the solution(s", if any, of the follo&ing system of
linear equations*
The algorithm is as follo&s* eliminate x from all equations (elo& L1, and then eliminate y from all
equations (elo& L!. This &ill put the system into triangular form. Then, using (a$6%su(stitution, ea$h
un6no&n $an (e solved for.
)n our example, &e eliminate x from L! (y adding to L!, and then &e eliminate x from L- (y adding
L1 to L-. 0ormally*
The result is*
:o& &e eliminate y from L- (y adding 8 ?L! to L-*
The result is*
This result is a system of linear equations in triangular form, and so the first part of the algorithm is
$omplete.
The se$ond part, (a$6%su(stitution, $onsists of solving for the un6no&ns in reverse order. Thus, &e $an
easily see that
Then, z $an (e su(stituted into L!, &hi$h $an then (e solved easily to o(tain
:ext, z and y $an (e su(stituted into L1, &hi$h $an (e solved to o(tain
Thus, the system is solved.
+aussian #limination is $onsidered the &or6horse of $omputational s$ien$e for the solution of a system
of linear equations. @arl 0riedri$h +auss, a great 19
th
$entury mathemati$ian, suggested this elimination
method as a part of his proof of a parti$ular theorem. 3omputational s$ientists use this AproofB as a
dire$t $omputational method.
+aussian #limination is a systemati$ appli$ation of elementary ro& operations to a system of linear
equations in order to $onvert the system to upper triangular form. Cn$e the $oeffi$ient matrix is in upper
triangular form, &e use (a$6 su(stitution to find a solution.
The general pro$edure for +aussian #limination $an (e summarized in the follo&ing steps*
+aussian #limination Steps
1. 7rite the augmented matrix for the system of linear
equations.
!. >se elementary ro& operations on the augmented matrix
DAEbF to transform A into upper triangular form. )f a zero
is lo$ated on the diagonal, s&it$h the ro&s until a
nonzero is in that pla$e. )f you are una(le to do so, stopG
the system has either infinite or no solutions.
-. >se (a$6 su(stitution to find the solution of the
pro(lem.
5etHs illustrate ea$h step of +aussian #limination &ith a system of three linear equations in three
varia(les, using the $ompa$t notation of augmented matri$es. 2s &e go through the steps of +aussian
#limination &ith our - x - example system, 6eep in mind that although the num(ers in the augmented
matrix may $hange signifi$antly after ea$h elementary ro& operation, our solution set has not $hanged.
1. 7rite the augmented matrix for the system of linear equations%
7e use the sym(ol to indi$ate that the matrix pre$eding the arro& &ill (e $hanged using the
operation spe$ifiedG the matrix follo&ing the arro& displays the result of that $hange. 5et r1, r!, and r-
denote ro&s 1, !, and -, respe$tively. 7e &ill also use a shorthand to represent the parti$ular elementary
ro& operations used to $hange an intermediate augmented matrix. 0or example,
&ritten to the right of ro& three of an augmented matrix means* 9epla$e ro& three &ith four times ro&
three plus negative one%third times ro& t&o. :ote that $omputational s$ientists usually denote
multipli$ation &ith an asteris6, so you &ill see uIv rather than uJv or u x v &hen u and v are to (e
multiplied.
!. >se elementary ro& operations on the augmented matrix DAEbF to
transform A into upper triangular form.
Sin$e all of the nonzero elements are no& lo$ated in the Aupper triangleB of the matrix, &e have
$ompleted the first phase of solving a system of linear equations using +aussian #limination.
:oti$e that our original $oeffi$ient matrix had a A.B on the diagonal in ro& 1. Sin$e &e needed to use
multiples of that diagonal element to eliminate the elements (elo& it, &e s&it$hed t&o ro&s in order to
move a nonzero element into that position. 7e $an use the same te$hnique &hen a A.B appears on the
diagonal as a result of $al$ulation. )f it is not possi(le to move a nonzero onto the diagonal (y
inter$hanging ro&s, then the system has either infinitely many solutions or no solution, and the
$oeffi$ient matrix is said to (e sin&ula'.
The se$ond and final phase of +aussian #limination is (a$6 su(stitution. Kuring this phase, &e solve for
the values of the un6no&ns, &or6ing our &ay up from the (ottom ro&.
(% Use bac) substitution to ind the solution o the #'oblem%
The last ro& in the augmented matrix represents the equation
so . The se$ond ro& of the augmented matrix represents the equation
%!y % -z < %.
Su(stituting the value of z into this equation and solving for y gives us*
0inally, the first ro& of the augmented matrix represents the equation
x ; y ; !z < L.
Su(stituting the values of y and z into the first equation yields*
This algorithm &or6s for any system of linear equations. )t is possi(le that the system $annot (e redu$ed
to triangular form, yet still have at least one valid solution* for example, if y had not o$$urred in L! and
L- after our first step a(ove, the algorithm &ould have (een una(le to redu$e the system to triangular
form. Mo&ever, it &ould still have redu$ed the system to e$helon form. )n this $ase, the system does not
have a unique solution, as it $ontains at least one free varia(le. The solution set $an then (e expressed
parametri$ally (that is, in terms of the free varia(les, so that if values for the free varia(les are $hosen, a
solution &ill (e generated".
)n pra$ti$e, one does not usually deal &ith the a$tual systems in terms of equations (ut instead ma6es
use of the augmented matrix (&hi$h is also suita(le for $omputer manipulations". This, then, is the
+aussian #limination algorithm applied to the augmented matrix of the system a(ove, (eginning &ith*
&hi$h, at the end of the first part of the algorithm loo6s li6e this*
That is, it is in ro& e$helon form.
2t the end of the algorithm, &e are left &ith
That is, it is in redu$ed ro& e$helon form, or ro& $anoni$al form.
Othe' a##lications! *indin& the in+e'se o a mat'i"
Suppose A is a matrix and you need to $al$ulate its inverse. The identity matrix is
augmented to the right of A, forming a matrix (the (lo$6 matrix < DA,!F". Through
appli$ation of elementary ro& operations and the +aussian elimination algorithm, the left (lo$6 of $an
(e redu$ed to the identity matrix !, &hi$h leaves A
8 1
in the right (lo$6 of .
)f the algorithm is una(le to redu$e A to triangular form, then A is not inverti(le.
)n pra$ti$e, inverting a matrix is rarely required. Nost of the time, one is really after the solution of a
parti$ular system of linear equations.
The &ene'al al&o'ithm to com#ute 'an)s and bases
The +aussian elimination algorithm $an (e applied to any matrix A. )f &e get Ostu$6O in a given
$olumn, &e move to the next $olumn. )n this &ay, for example, any matrix $an (e transformed to
a matrix that has a redu$ed ro& e$helon form li6e
(the I,s are ar(itrary entries". This e$helon matrix " $ontains a &ealth of information a(out A* the ran6
of A is / sin$e there are / non%zero ro&s in "G the ve$tor spa$e spanned (y the $olumns of A has as (asis
the first, third, fourth, seventh and ninth $olumn of A (the $olumns of the ones in "", and the I,s tell you
ho& the other $olumns of A $an (e &ritten as linear $om(inations of the (asis $olumns.
Anal,sis! The $omputational $omplexity of +aussian elimination is #(n
-
" (see Big C notation"G that is,
the num(er of operations required is (approximately" proportional to n
-
if the matrix size is n P n.
This algorithm $an (e used on a $omputer for systems &ith thousands of equations and un6no&ns.
Mo&ever, the $ost (e$omes prohi(itive for systems &ith millions of equations. These large systems are
generally solved using iterative methods. Spe$ifi$ methods exist for systems &hose $oeffi$ients follo& a
regular pattern (see system of linear equations". The +aussian elimination $an (e performed over any
field.
Bac) Substitution
Deinition -U##e'-T'ian&ula' Mat'i".% 2n matrix is $alled upper-triangular provided
that the elements satisfy &henever .
)f A is an upper%triangular matrix, then is said to (e an upper%triangular system of linear
equations.

Theo'em -Bac) Substitution.% Suppose that is an upper%triangular system &ith the form given
a(ove in (1". )f for then there exists a unique solution.
The bac) substitution al&o'ithm% To solve the upper%triangular system (y the method of (a$6%
su(stitution. 4ro$eed &ith the method only if all the diagonal elements are nonzero. 0irst $ompute

and then use the rule
for

Cr, use the Ogeneralized ruleO
for
&here the Ounderstood $onventionO is that is an Oempty summationO (e$ause the lo&er
index of summation is greater than the upper index of summation.
/ema')% The loop $ontrol stru$ture &ill permit us to use one formula.
Com#ute' 0'o&'ams T'ian&ula' S,stems and Bac) Substitution T'ian&ula' S,stems and Bac)
Substitution
Mathematica Sub'outine -Bac) Substitution.%
4edagogi$al version for Oprinting all the details.O
Lo1e'-t'ian&ula' s,stems%
7e &ill no& develop the lower$substitution al%orit&m, &hi$h is useful for solving a linear system of
equations that has a lo&er%triangular $oeffi$ient matrix.
Deinition -Lo1e'-T'ian&ula' Mat'i".% 2n matrix is $alled lower-triangular provided
that the elements satisfy &henever .
)f A is an lo&er%triangular matrix, then is said to (e a lo&er%triangular system of linear
equations.


Theo'em -*o'1a'd Substitution.% Suppose that is an lo&er%triangular system &ith the form
given a(ove in (!". )f for then there exists a unique solution.
0'oo T'ian&ula' S,stems and Bac) Substitution T'ian&ula' S,stems and Bac) Substitution
The o'1a'd substitution al&o'ithm% To solve the lo&er%triangular system (y the method of
for&ard%su(stitution. 4ro$eed &ith the method only if all the diagonal elements are nonzero. 0irst
$ompute

and then use the rule
for .

/ema')% The loop $ontrol stru$ture &ill permit us to use one formula
for .
is x < =!?, y < =1-, and z < !.
Forward Difference Method
)n mathemati$s, a die'ence o#e'ato' maps a fun$tion, f(x", to another fun$tion, f(x ' a" 8 f(x ' b".
The o'1a'd die'ence o#e'ato'
o$$urs frequently in the $al$ulus of finite differen$es, &here it plays a role formally similar to that of the
derivative, (ut used in dis$rete $ir$umstan$es. Kifferen$e equations $an often (e solved &ith te$hniques
very similar to those for solving differential equations. 2nalogously &e $an have the bac)1a'd
die'ence o#e'ato'
7hen restri$ted to polynomial fun$tions f, the for&ard differen$e operator is a delta operator, i.e., a
shift%equivariant linear operator on polynomials that redu$es degree (y 1.
n-th difference
The n%th for&ard differen$e of a fun$tion f(x" is given (y
&here is the (inomial $oeffi$ient. 0or&ard differen$es applied to a sequen$e are sometimes $alled
the (inomial transform of the sequen$e, and, as su$h, have a num(er of interesting $om(inatorial
properties.
0or&ard differen$es may (e evaluated using the :Qrlund%9i$e integral. The integral representation for
these types of series is interesting (e$ause the integral $an often (e evaluated using asymptoti$
expansion or saddle%point te$hniquesG (y $ontrast, the for&ard differen$e series $an (e extremely hard to
evaluate numeri$ally, (e$ause the (inomial $oeffi$ients gro& rapidly for large n.
DeditF
Newton series
The $e1ton se'ies or $e1ton o'1a'd die'ence e2uation is the relationship
&hi$h holds for any polynomial fun$tion f and for some, (ut not all, analyti$ fun$tions. Mere,
is the (inomial $oeffi$ient, and
is the Ofalling fa$torialO or Olo&er fa$torialO and the empty produ$t (x". defined to (e 1. :ote also the
formal similarity of this result to Taylor,s theoremG this is one of the o(servations that lead to the idea of
um(ral $al$ulus.
)n analysis &ith p%adi$ num(ers, Nahler,s theorem states that the assumption that f is a polynomial
fun$tion $an (e &ea6ened all the &ay to the assumption that f is merely $ontinuous.
3arlson,s theorem provides ne$essary and suffi$ient $onditions for a :e&ton series to (e unique, if it
exists. Mo&ever, a :e&ton series &ill not in general exist.
The :e&ton series, together &ith the Stirling series and the Sel(erg series, is a spe$ial $ase of the
general differen$e series, all of &hi$h are defined in terms of s$aled for&ard differen$es.
DeditF
Rules for finding the difference applied to a combination of functions
2nalogous to rules for finding the derivative, &e have*
Constant 'ule* )f c is a $onstant, then
Linea'it,* if a and b are $onstants,
2ll of the a(ove rules apply equally &ell to any differen$e operator, in$luding as to .
0'oduct 'ule*
3outient 'ule*
or
Summation 'ules*
Finite Difference Method
)n mathemati$s, more pre$isely in numeri$al analysis, finite differen$es play an important role, they are
one of the simplest &ays of approximating a differential operator, and are extensively used in solving
differential equations.
0or example, $onsider the ordinary differential equation
The #uler method for solving this equation uses the finite differen$e
to approximate the differential equation (y
The last equation is $alled a inite-die'ence e2uation. Solving this equation gives an approximate
solution to the differential equation.
The error (et&een the approximate solution and the true solution is determined (y the error that is made
(y going from a differential operator to a differen$e operator. This error is $alled the (iscretization error
or truncation error (the term truncation error refle$ts the fa$t that a differen$e operator $an (e vie&ed as
a finite part of the infinite Taylor series of the differential operator".
Example: the heat equation
3onsider the normalized heat equation in one dimension, &ith homogeneous Kiri$hlet (oundary
$onditions
((oundary $ondition"
(initial $ondition"
Cne &ay to numeri$ally solve this equation is to approximate all the derivatives (y finite differen$es.
7e partition the domain in spa$e using a mesh x.,...,xJ and in time using a mesh t.,....,tN. 7e assume a
uniform partition (oth in spa$e and in time, so the differen$e (et&een t&o $onse$utive spa$e points &ill
(e & and (et&een t&o $onse$utive time points &ill (e k. The points
&ill represent the numeri$al approximation of U(xj,tn".
Explicit method
>sing a for&ard differen$e at time tn and a se$ond%order $entral differen$e for the spa$e derivative at
position xj &e get the re$urren$e equation*
This is an expli$it method for solving the one%dimensional heat equation.
7e $an o(tain from the other values this &ay*
&here r < k R &
!
.
So, 6no&ing the values at time n you $an o(tain the $orresponding ones at time n;1 using this
re$urren$e relation. and must (e repla$ed (y the (order $onditions, in this example they are (oth
..
This expli$it method is 6no&n to (e numeri$ally sta(le and $onvergent &henever . The
numeri$al errors are proportional to the time step and the square of the spa$e step*
Implicit method
)f &e use the (a$6&ard differen$e at time tn ; 1 and a se$ond%order $entral differen$e for the spa$e
derivative at position xj &e get the re$urren$e equation*
This is an impli$it method for solving the one%dimensional heat equation.
7e $an o(tain from solving a system of linear equations*
The s$heme is al&ays numeri$ally sta(le and $onvergent (ut usually more numeri$ally intensive then
the expli$it method as it requires solving a system of numeri$al equations on ea$h time step. The errors
are linear over the time step and quadrati$ over the spa$e step.
Crank-Nicolson method
0inally if &e use the $entral differen$e at time tn ; 1 R ! and a se$ond%order $entral differen$e for the spa$e
derivative at position xj &e get the re$urren$e equation*
This formula is 6no&n as the 3ran6%:i$olson method.
7e $an o(tain from solving a system of linear equations*
The s$heme is al&ays numeri$ally sta(le and $onvergent (ut usually more numeri$ally intensive as it
requires solving a system of numeri$al equations on ea$h time step. The errors are quadrati$ over the
time step and formally are of the fourth degree regarding the spa$e step*
Mo&ever, near the (oundaries, the error is often C(&
!
" instead of C(&
?
".
>sually the 3ran6%:i$olson s$heme is the most a$$urate s$heme for small time steps. The expli$it
s$heme is the least a$$urate and $an (e unsta(le, (ut is also the easiest to implement and the least
numeri$ally intensive. The impli$it s$heme &or6s the (est for large time steps.