Numerical Methods So, 0.02 b= = 4.468 x 10-8 VB x 2.24 Hence the system is ill-conditioned. Example: 3.2.2 Find the condition number of the system SJE Solution: = VITFTTR = 2.766, sy = 8.157, 0.03, k= = 0.001329, Hence the system is ill-conditioned. 3.3 Iterative Methods We start with an initial approximation to the solution vector x = zp, to solve the system of equations Ax = , and obtain a sequence of approximate vectors 19,71; -yTky--, Which in the limit as k + oo, converges to the exact solution veetor x = Ab, A general linear iterative method for the solution of the system of equations Ar = b, can be written in matrix form as 2) = Hal + 6k =0,1,2, where 2") and 2) are the approximations for x at the (k + 1)!* and k'* itera tions respectively. HT is called the iteration matrix, which depends on A and c is, a column vector, which depends on A and 6 Manoj Kumar 43SS Nrmerrical Methods When to stop the iteration: We stop the iteration procedure when the magni- tudes of the differences between the two successive iterates of all the variables are smaller than a given accuracy or error tolerance or an error bound e, that is, (ea) ce VE For example, if we require two decimal places of accuracy, then we iterate until [cf — 2!) < 0.005, Yi. If we require three decimal places of accuracy, then ce) we iterate |! 2!) < 0.0005, va. 3.3.1 Gauss-Seidel Iteration Method It is also called method of successive displacements. Consider 3x3 system of equations, ant) + ante + dist 03321 + agy9 + aging = by 3121 + agnt2 + Aas We write the Gauss-Seidel method as aya? = a2 — agaf? +b, (esa) (een) ic ane) 4 ayy angel) +b age aga) + ase?) = by In matrix notation, 2) = (D+ LW) + (D+ Lb = Hal +6, k=0,1,2, where H = —(D+L)'U and ¢ = (D + L)%, and the matrices L and U are respectively lower and upper triangular matrices with zero diagonal entries, the matrix is the diagonal matrix, such that A= L+D+U. Manoj Kumar aNumerical Methods Example: 3.3.1 Solve the system 2a — 42 +0053 =7 =n, +2ny—25=1 Oxy — 2, +225 =1 using Gauss-Seidel iterative method and perform three iterations Solution: We have, o 00 200 o 10 7 0 of, D=|o 2 0],U=|o o -a}, andb= |i 0 -10 002 oo 1 The Gauss-Seidel iterative method gives 2®) — (D+ 1) U2 +(D+L) 1b 0 05 0 35 0 025 05] 2+ | 225 0 0.125 0.25 1.625, Starting with the zero initial vector, we get 35 4.625 5.3125 x= | 2.295] ,20 = | 3.625 |, and 2 = |43125 1.625 2.3125 2.6502 Example: 3.3.2 Show that Gauss-Seidel method applied to the system of equa: tions 3x1 — 6 + 2r3 = 23 dey + 22-23 = -8 ay ~ 3x24 Tay = 17 diverges. Take the initial approximations as a = 0.9, = ~3.1,5 = 0.9. Inter- change the first and second equations and solve the resulting system by the Gauss- ‘eidel method. Again take the initial approvimations as 2, = 0.9,r = ~3.1, 25 = Manoj Kumar 45Numerical Methods 0.9, and obtain the resull correct to two decimal places. The exact solution is 2 = 1.0, 22 =-3.0,25 = 10. Solution: (a) Without Pivoting: We have 0 0 0 300 0-6 2 23, L=|-4 0 0], D=|o 1 0J,U=Jo 0 ~1], andd=|-8 1-3 0 007 o 0 0 7 The Gauss-Seidel iterative method gives a) = (p+ Lytue® + (D+ Ly 0 2 0.6666 7.6666 =]o 8 ~1.6666] 2+ |22.666 0 3.142 —0.619 11.047 Starting with the initial vector, we get 0.8666 0.099 7.073 x!) = | 3.633] ,«°) = |-7.647] , and 2 = |-37.12 0.7497 —0.831, =12.46 It can be observed that the iterations are diverging very fast (b) With Pivoting: Now, we exchange the first and second equations to obtain the system dy + 22 — 25 = -8 Bry — Gary + 2xy = 23 ay — 3xy + Try = 17 We have 000 400 for a 8 b=|3 0 of, D=]o -6 oJ, 7=|o 0 2], ando= Jas 1-30 0 0 00 0 " Manoj Kumar 46Numerical Methods The Gauss-Seidel iterative method gives e+) = (p+ Ly Ue + (D+ Ly 0 0.25 ~0.25 2 0 0.125 0.208 | 2" + | 2.833 0 0017 0.125 0.9285, Starting with the initial vector, 09 2 = | 31 09 we get 1 0.9946 0.9984 0.9994 2 = |_3.933| ,2? = |-s.006] .2 = |~3.000] , and 2 = |-2.999 0.9883 1.0008 1.0024 1.0028 We find, 0.0038 0.0012] jn) — af) 0.0062] ¢ 0.005, |x) — x) = |o.o012| < 0.005, 0.0022 0.0004 Rounding to two decimal places, we get 1 = 1.0,.r2 = -3.0,25 = 1.0. 3.3.2 Successive Over Relaxation (SOR) Method Seidel Iteration Method. Consider It is also called method of generalized 3x3 system of equations yay + aye, + ag03 = ag.) + agate + aaxts = by Manoj Kumar ayNumerical Methods Step 1: We write down the equations for Gauss-Seidel Iteration Method en a) os ant} sayz}? — ays2}9 +b, (ey on) anti? + ach) = ania" + aya + asset Reform it, such as dey 1 a o af) = —[-ayrf — ase! +0) an wey 1 (en c 28) = [anal — agsey? + ba] an syd 7 1) 29 = [anal — agae? + ba} a Step 2: Now multiply the right hand side by the parameter w and add it the vector 2* from the previous iteration multiplied by the factor of (1 — w), then we have a2) — ays2%) + 63] + [Haga — anal) + ba] ayuett — ayaa! 40 ane? = (= w)mal + sef-a2l? — aye +8] ant + war rl = (1 — w)azarl” + wars!) + by] Oe ant" + wane + wast aga(1 — wx) + wy Step 3: In matrix notation, (D+ wh)x* = (1 w) D2 — wa + wb, or 2?) = (D+ wh) (1 — w)D — w]e + w(D + wh) =He® +0, k=0,1,2 Manoj Kumar 4BNumerical Methods where H = (D+wL)""[(1—w)D—wU] and ¢ = w(D-+wL)~1, and the matrices L and U are respectively lower and upper triangular matrices with zero diagonal entries, the matrix D is the diagonal matrix, such that A= L~D+U. When w = 1, the SOR method reduces to the Gauss-Seidel method. The quantity w is called the relaxation parameter. If w > 1, then the method is called an over relaxation method and if w < 1 then it is called an under relaxation method. ¢ The rate of convergence of Gauss Seidel method is improved through the over relaxation, ‘¢ Convergence of SOR is possible for 0 < w < 2. Example: 3.3.3 Consider a linear system Ax = b, where Complete the four iteration by the SOR method with w = 1.25 starting at the point 2° = (0.0 0)". Step 1: First, write down the equations for the GS iteration Step 2: Now multiply the right hand side by the parameter w and add to it the vector «* from the previous iteration multiplied by the factor of (1 — w) Af) 1 wal 4% 2 = wal 4 pt 4 a 7) A =u }e + E28) gale? 7] Manoj Kumar 49Yel Numerical Methods whl + Spx — 2 ol? = (1—w)2 4 2 4 2 27 3 a) = (we +3 For w = 1.25, a) = (1 —1,25)0 + Fo 0-1) =~0.4167 af) = (1 = 1.25)0 I 0.4167 +0 + 7] = 2.7431 = (= 1.25)0+ PF poanor + 2.7431 — 7] = -1.6001. Therefore 0.4167 1.4972 0494 0.9428 2) = | o7asi | 2 = | aisso | 2° = | ievs2 | 2 = | 2.0007 1.6001 2.2288 2.0141 1.9723 3.4 Convergence Analysis Tho iteration method 20) = He +e, b= 0,1,2, (3.1) ‘The exact solution x will satisfy Hz+e, (3.2) Subtracting (3.1) from (3.2) and substituting «®) ee k = He k=1,2, from which follows Ma=HO 46, k=1,2, (3.3) 50 Manoj KumarSS Nrmerrical Methods ‘Theorem 3.4.1 Let A be a square matrix, Then if |Al| <1. Proof: If \\A\] <1, we have Am < Al" and || im, ar] stim Al” = Theorem 3.4.2 The infinite series T+ Atay converges if litt soo A” = 0. The series converges to (I — A)! Proof: If lithyy-y A” = 0, then by theorem, ||Al] <1. Hence | — A| #0 and (1A) exists Consider the identity (+ A+ APt e+ A= A) = T- a Post multiplying by (I — A)", we have (4 At AME +A") = (F- AMY At Asm > 00, we get (I+ A444...) =A) ‘Theorem 3.4.3 The iteration method of the form (3.1) for the solution of Aa converges to the exact solution for any initial vector, af ||| <1 Proof: Without loss of generality, we take initial vector x = 0. We have 2?) = He +e= (H+ Te a8) = He -o= (H+ H+ Te 2) = He 4 ¢= (HY + HE +4... +H+ De Manoj Kumar 51Numerical Methods fim ah) jm (H+ HEY 4. + H+ De =(I- Hye if || <1 In the case of the Gauss-Seidel method, we have (I= Byte = [P+ (D+ UP (D + Ly = [D+ (D+L+ Uy 4 1) =(D+L+US (D+ 1)(D+ 1) b =A Similar result can be proved for the SOR methods. Remark: A sufficient condition for convergence of the iterative method is that the system of equations is diagonally dominant, that is, the coefficient matrix A is, Giagonally dominant, This implies that convergence may be obtained even if the system is not diagonally dominant. If the system is not diagonally dominant, we may exchange the equations, if possible, such that the new system is diagonally dominant and convergence is guaranteed. ‘The necessary and sufficient condition for convergence is that |||) <1 ‘Theorem 3.4.4 A necessary and sufficient condition for convergence of an iter- ative method of the form is that the eigenvalues of the iteration matriv satisfy M(H) <1 i Example: 3.4.1 For the solution of the system of equations 2 -1 0] |x] |7 -1 2 -1] Jer o -1 2] fxs} [t Manoj Kumar 52Numerical Methods set up the SOR method (i) w =0.5 (it) w= 1 (iti) w= 15. We have 0 00 200 en) 7 L=|-1 0 of, D=Jo 2 0],U-]o o ~a], andb-|a 0-10 002 o 0 0 1 The SOR scheme is obtained as 2) = (D+ wh) (1 — w)D — wr + w(D + wh) Ho +e, k=0,1,2, ew = 05, 2 0 0 105 0 (D+05L)=|-05 2 0] osp-o5sv=|o 1 05 0 05 2 oo 1 2) = (D+0.5L)(05D — 0.5U]2 + (D+ 0.5L)" 05 025 0 175 0.125 0.5625 0.25 | 2 + |o.6875 0.0312 0.1406 0.5625 0.4218 Starting with the initial vector, 09 2 = |_31 09 2.7968 3.498 4.0055 2 = |1 3984] 2) = |2.0262 2.5548 0.8103 1.1614 1.4691 owed, 2 00 o10 (D+L)=|-1 2 of, ool 0 -12 O00 Manoj Kumar 53Numerical Methods 2) = (D+ L)[-U]z™ + (D+ L) 0 05 35 =|o 025 05) 2+ | 2.25 0 0.125 0.25 1.625 Starting with the initial vector, oy a we get 1.95 4.4625 5.2312 5.6156 2 | 1995 | 2% = |3.4625 a2ai2| 2 = |4156 1.4625 2.2312 2.6156 2.8078 3.5 Eigen Value Problem Consider the eigen value problem Ar = dx. where x is an eigenvector corresponding to the eigenvalue A of A. The eigenvalues of a matrix A are given by the roots of the characteristic equation A-X\=0. 3.6 Rayleigh’s Power Method Power method is used to determine the largest eigenvalue (in magnitude) and the corresponding eigenvector of the system Ax = dx. Manoj Kumar 54SS Nrmerrical Methods Let A, A2,...,An be the distinct eigenvalues such that [Ar] > [Aa > > Pl and v;,02,..., 0, be the corresponding eigenvectors. The procedure is applicable if a complete system of n independent eigenvectors exists, even though some of eigenvalues Az,...,n may not be distinct. ‘Then any eigenvector v in the space of eigenvectors v1, ¥2,...,ty can be written as vs ey teats to Feat Premultiplying by A and substituting Avy = Av, Avr = Ayva ete., we obtain Av Avy + epAzde +H GAnts =1s [ome (3) rote( Premultiplying by A again and simplifying we get : : won feota() arte)’ a] »2\* an)" Aart fon sa (B) aera) s ay\t a, Aly = tt [>» +e (2) uate ten (2) Uy aL ay Jim Ate = Aan Jim AM ty = Xfi The eigenvalue A; is obtained as the ratio of the corresponding components of Aw and AN y, Abe Ay = Jim fim Gaye To beon where the sullix r denotes the r!* component of the vector. If |z| << As), then faster convergence is obtained. In order to keep the roundoff error in control, we Manoj Kumar 55Numerical Methods normalize (such that the largest element is unity) the vector before premultipl by A. We can write the method as follows. Let vy be an arbitrary initial vector and find esr = Avg ve = where mcr is the largest element in magnitude of yz41. Then (vesa)e Dr = Jim SEE 2.0m rm (Us)e and vy is the required eigenvector. The initial vector is usually chosen a vector with all components equal to unity. ‘* Power method gives the largest cigen value in magnitude. If the sign of the eigen value is required, then we substitute this value in the determinant |A — AiI| and find its value. If this value is approximately zero, then the eigen value is of positive sign. Otherwise, it is of negative sign Example: 3.6.1 Find the largest eigenvalue in modulus and the corresponding eigenvector of the matrix, 6 4 3 A=|10 -12 6 2 -4 2 using the power method. Solution: We start the iteration using the unit vector as the initial vector 1 w= {a l we find f-us 4 a] fi 8 8/18 asad n=Aw=|10 -12 6 fif=]4f, u=4=| ans | = | 0222 2» -4 af fa] [as 1/18| | 1.0000 Manoj Kumar 56Numerical Methods 15 4 3] [-o4aa 1 n= Av.=| 10 12 6| | 0.2222 | = == 20 4 2| | 1.0000 1978 14 38 17.6316 0.9305 w= A=] 10-12 6 =| 6.848 20 4 2| [-o7s68} | 18.9476 15 4 3] [-09305) —[ 18.4019 u=Avs=] 10 -12 6| | ose | = | -7.7382 20-4 2{ | 10000} [-180540 =0.9811 -15 4 3] [1.0000] — [-19.6258 0.9952 ys=4u=] 10 —12 6| |-0.4205] = | 9.1594 0.4645 20-4 2| |-oosi 19.7198 © | 1.0000 15 4 3] [-0.9952] [19.7860 1 w= Avs=| 10 12 6] | oss | =|-9.5200], = = |-o4sia mg 20-4 2] | 1.0000] — |-19.762 0.9988 -15 4 3}f 1 —19.922 0.9997 w=Avs=|10 ~12 6] |-oasia] =| 9.780 |, = =| og 20-4 2] [09088] | 19.928 1.0000 15 4 3] [-0.9997] [ 19.9599 1 w=Avy=] 10-12 6] | o4on | =| 9.8902], v= = | 0495s, 20 4 2 | 1.0000 19,9581 0.9999 3 4 3)f a 19,9817 w= v=] 10 12 6) |-0.4955] = | 9.9466 20-4 2| [0.9000] | 19.9822 Manoj KumarNumerical Methods AL this step, the approsimations to the largest eigenvalue in modulus are 1,2,3. = 19.9817, 20.0738, 19.9842 If we roundoff to ¥ digits, we have |As| = 20. The approzimate eigenvector is 1 0.4955 0.9999 The exact eigenvalue is 20 and its eigenvector is Manoj Kumar 58