Solving Systems of Linear Equations

Feng-Nan Hwang

Department of Mathematics
National Central University,
Jhongli District, Taoyuan City, Taiwan
Email: [email protected]
Homepage: http:/www.math.ncu.edu.tw/~hwangf

1 / 143
Problem statement: Linear systems

2 / 143
I We are interested in solving systems of linear equations having
the form:


 a11 x1 + a12 x2 + a13 x3 · · · + a1n xn = b1
 a21 x1 + a22 x2 + a23 x3 · · · + a2n xn = b2



a31 x1 + a32 x2 + a33 x3 · · · + a3n xn = b3
 ..
.




an1 x1 + an2 x2 + an3 x3 · · · + ann xn = bn


I This is a system of n equations in the n unknowns,

x1 , x2 , . . . , xn .
I The elements aij and bi are assumed to be prescribed real
numbers.

3 / 143
I We can rewrite this as a system of linear equations:
  x   b 
a11 a12 a13 · · · a1n 1 1
 a21 a22 a23 · · · a2n   2   2
 x   b 

x3  =  b3
 
 a31 a32 a33 · · · a3n    
 
  ...   .

···  ..
   

an1 an2 an3 · · · ann xn bn
I We can denote these matrices by A, x, and b, giving the
simpler equation:

Ax = b

4 / 143
In-class exercise
Consider the 3 × 3 system

x + 2y − 1 = 0
2x + y + z − 3 = 0
x − y + 2z − 2 = 0
In matrix form, the system is
    
x 1
  y  =  3 .
z 2
The augmented matrix is
 
 

5 / 143
Matrix Algebra

6 / 143
I A matrix is a rectangular array of numbers such as:
 
3.0 1.1 −0.12
 6.2 0.0 0.15 
 
 0.6 −4.0 1.3 
9.3 2.1 8.2
I This is an example of a 4 x 3 matrix.

7 / 143
Matrix properties

I If A is a matrix, the notation aij , (A)ij , or A(i, j) is used to

denote the element at the intersection of the ith row and the
jth column.
I The transpose of a matrix is denoted by AT and is the
matrix defined by (AT )ij = aji . The transpose of the matrix
on the previous slide is:
 
3.0 6.2 0.6 9.3
AT =  1.1 0.0 −4.0 2.1 
−0.12 0.15 1.3 8.2
I If a matrix A has the property A = AT , we say that A is
symmetric

8 / 143
Algebraic operations

I Matrix * Scalar: If A is a matrix and λ is a scalar, then λA is

defined by (λA)ij = λaij .
I Matrix + Matrix: If A = (aij ) and B = (bij ) are m × n
matrices, then A + B is defined by (A + B)ij = aij + bij .
I Matrix * Matrix: If A is an m × p matrix and B is a p × n
matrix, then AB is an m × n matrix defined by:
p
X
(AB)ij = aik bkj (1 ≤ i ≤ m, 1 ≤ j ≤ n)
k=1
I In-class exercise. What is the cost of AB?

9 / 143
In-class exercise

I Let A, B, and C be the square matrix of dimension n. Let u

and v be the vectors of dimension n.
I Complete the following table.
Operations Cost ((flops), (big O)) Storage (big O) Comments
u·v
v = Au
C = AB
I 1 flop = one multiplication + one addition.
I Recall that Big notation, Big O: xn = O(αn ) for two
sequences [xn ] and [αn ] if there exists C and n0 such that
|xn | ≤ C |αn | when n ≥ n0 .

10 / 143
 I Let two systems be given, each consisting of n equations with
n unknowns:
Ax = b Bx = d
I If the two systems have precisely the same solutions, we call
them equivalent systems.
Note A and B can be very different.

11 / 143
Elementary operations

I Let εi denote the ith equation in the system. The following

are the elementary operations which can be performed:
1. Interchanging two equations in the system: εi ↔ εj
2. Multiplying an equation by a nonzero number: λεi → εi
3. Adding to an equation a multiple of some other equation:
εi + λεj → εi
I Theorem on Equivalent Systems
If one system of equations is obtained from another by a finite
sequence of elementary operations, then the two systems are
equivalent.

12 / 143
In-class exercise
Solve the system
x1 + x2 − x3 = 1,
2x1 + x2 + x3 = 0
−x1 − 2x2 + 3x3 = 2.
Row reduce the augmented matrix to upper-triangular form.
   
1 1 −1 1 1 1 −1 1
 2 1 1 0  −→  0 
R2 −2R1 →R2
−1 −2 3 2 −1 −2 3 2
   
1 1 −1 1 1 1 −1 1
 0  −→  0 
R3 −(−1)R1 →R3
−1 −2 3 2 0
   
1 1 −1 1 1 1 −1 1
 0  −→  0 
R3 −(1)R2 →R3
0 0 0
Execute back substitution.
(−1)x3 = 5, x3 =
−x2 + 3( ) = −2, x2 =
x1 + (1)( ) − ( ) = 1, x1 = .
13 / 143
Matrix properties

 
1 0 0 ··· 0

 0 1 0 ··· 0 

I = In = In×n =
 0 0 1 ··· 0 

 ··· 
0 0 0 ··· 1
This matrix is called the identity matrix. IA = A = AI for any
matrix A.

14 / 143
I If A and B are two matrices such that AB = I , then we say
that B is a right inverse of A and that A is a left inverse of
B. For example:
 
  1 0  
1 0 0  1 0
0 1 =
0 1 0 0 1
α β
I Theorem on Right Inverse A square matrix can possess at
most one right inverse.
I Theorem on Matrix Inverse If A and B are square matrices
such that AB = I , then BA = I .
[hint:] Let C = BA − I + B then show C = B.

15 / 143
I If a square matrix A has a right inverse B, then B is unique
and BA = AB = I . We then call B the inverse of A and say
that A is invertible or nonsingular.
I We denote B = A−1
I Example:
       
−2 1 1 2 1 2 −2 1 1 0
3 = =
2 − 21 3 4 3 4 3
2 − 12 0 1

I If A is invertible, then the system of equations Ax = b has the

solution x = A−1 b. If A−1 is not available, then in general,
A−1 should not be computed solely for the purpose of
obtaining x.
I How do we get this A−1 ?

16 / 143
An elementary matrix is defined to be an n × n matrix that arises
when an elementary operation is applied to the n × n identity
matrix. The elementary operations expressed in terms of the rows
of matrix A are:
1. The interchange of two rows: As ↔ At
2. Multiplying one row by a nonzero constant: λAs → As
3. Adding to one row a multiple of another: As + λAt → As

17 / 143
Examples of the elementary operations:

    
1 00 a11 a12 a13 a11 a12 a13
 0 0
1  a21 a22 a23  =  a31 a32 a33 
 0 1
0  a31 a32 a33   a21 a22 a23 
1 00 a11 a12 a13 a11 a12 a13
 0 λ
0  a21 a22 a23  =  λa21 λa22 λa23 
 0 0
1  a31 a32 a33  a31 a32 a33
1 00 a11 a12 a13
 0 1
0  a21 a22 a23 
0
 1λ a31 a32 a33 
a11 a12 a13
= a21 a22 a23 
λa21 + a31 λa22 + a32 λa23 + a33

18 / 143
 I If a matrix is invertible, such a sequence of elementary row
operations can be applied to A, reducing it to I .

Em Em−1 · · · E2 E1 A = I
I This gives us an equation for computing the inverse of a
matrix:
A−1 = Em Em−1 · · · E2 E1
Remark: This is not a practical method to compute A−1

19 / 143
In-class exercise

Given a nonsingular 2 by 2 matrix

 
a b
A=
c d

Show that A−1 is given by

  
−1 1 d −b
A =
ad − bc −c a

by using Gaussian eliminiation.

20 / 143