Singular Systems versus

Non-Singular Systems
• A singular system has no solution or
infinitely many solution
– Row Picture: two line are parallel or the same
– Column Picture: Two column vectors are co-
linear
• A non-singular system has a unique solution
– Row Picture: two non-parallel lines
– Column Picture: two non-colinear column
vectors
 Gaussian Elimination in 3D
2x  4 y  2z  2
4 x  9 y  3z  8
 2 x  3 y  7 z  10
• Using the first pivot to eliminate x from the
next two equations
 Gaussian Elimination in 3D
2x  4 y  2z  2
y  z  4
y  5 z  12
• Using the second pivot to eliminate y from
the third equation
 Gaussian Elimination in 3D
2x  4 y  2z  2
y  z  4
4z  8
• Using the second pivot to eliminate y from
the third equation
Now We Have a Triangular System
2x  4 y  2z  2
y  z  4
4z  8
• From the last equation, we have
 Backward Substitution
2x  4 y  2z  2
y  z  4
z  2
• And substitute z to the first two equations
 Backward Substitution
2x  4 y  4  2
y  2  4
z  2
• We can solve y
 Backward Substitution
2x  4 y  4  2
y  2
z  2
• Substitute to the first equation
 Backward Substitution
2x  8  4  2
y  2
z  2
• We can solve the first equation
 Backward Substitution
x  1
y  2
z  2
• We can solve the first equation
 Generalization
• How to generalize to higher dimensions?

• What is the complexity of the algorithm?

• Answer:

Express Elimination with Matrices

 Step 1
Build Augmented Matrix
2x  4 y  2z  2
4 x  9 y  3z  8
 2 x  3 y  7 z  10
Ax = b

 2 4 2 2
A b   4 9  3 8 
[A b]
 2  3 7 10
Pivot 1: The elimination of column 1
 2 4 2 2
 4 9 3 8 
 
 2  3 7 10
 
 2 4 2 2  2
 0 1 1 4  
   1 
 2  3 7 10

2 4  2 2 
0 1 1 4 
 
0 1 5 12
Pivot 2: The elimination of column 2
2 4  2 2 
0 1 1 4 
 
0 1 5 12
 
 
 
 2 4  2 2 1
0 1 1 4
 
0 0 4 8 

Upper triangular matrix

 Backward Substitution 1: from the
last column to the first
Upper triangular matrix
 2 4  2 2  2 4  2 2  2 4  2 2
0 1 1 4 0 1 1 4 0 1 0 2
     
0 0 4 8  0 0 1 2 0 0 1 2

1 0 0  1  2 0 0  2 2 4 0 6
0 1 0 2  0 1 0 2  0 1 0 2
     
0 0 1 2  0 0 1 2  0 0 1 2
Expressing Elimination by
Matrix Multiplication
Elementary or Elimination Matrix
Ei , j
• The elementary or elimination matrix Ei , j
That subtracts a multiple l of row j from row i
can be obtained from the identity matrix I
by adding (-l) in the i,j position
 1 0 0
E3,1   0 1 0
 l 0 1
Elementary or Elimination Matrix
 a1,1 a1, 2 a1,3   1 0 0  a1,1 a1, 2 a1,3 
     
E3,1 a2,1 a2, 2 a2,3    0 1 0 a2,1 a2, 2 a2,3 
 a3,1 a3, 2 a3,3   l 0 1  a3,1 a3, 2 a3,3 

 a1,1 a1, 2 a1,3 

 
  a2,1 a2 , 2 a2 , 3 
 la1,1  a3,1  la1, 2  a3, 2  la1,3  a3,3 
Pivot 1: The elimination of column 1
   2 4 2 2 2 4  2 2 
 2  4 9 3 8  0 1 1 4 
     
 1   2  3 7 10 0 1 5 12
Elimination matrix
 1 0 0  2 4  2 2   2 4 2 2
  2 1 0  4 9  3 8    0 1 1 4
    
 0 0 1  2  3 7 10  2  3 7 10

1 0 0  2 4  2 2  2 4  2 2 
0 1 0   0 1 1 4    0 1 1 4 
    
1 0 1  2  3 7 10 0 1 5 12
The Product of Elimination Matrices
1 0 0  1 0 0  1 0 0
0 1 0    2 1 0     2 1 0 
    
1 0 1  0 0 1  1 0 1

1 0 0  1 0 0  1 0 0
0 1 0    2 1 0     2 1 0 
    
0  1 1  1 0 1  1  1 1
Elimination by Matrix Multiplication

 1 0 0  2 4  2 2   2 4  2 2
  2 1 0  4 9  3 8   0 1 1 4
    
 1  1 1  2  3 7 10 0 0 4 8
Linear Systems in Higher Dimensions

1 1 1 1 0
1 2 3 4   2 
 x 
1 3 6 10    5
   
1 4 10 20  9
Linear Systems in Higher Dimensions
1 1 1 1 0 1 1 1 1 0
1 2 3 4  2 0 1 2 3  2
 
1 3 6 10  5 0 2 5 9  5
   
1 4 10 20  9 0 3 9 19  9 

1 1 1 1 0 1 1 1 1 0
0 3  2 0
 1 2
 1 2 3  2
0 0 3 6  3 0 0 3 6  3
   
0 0 0 4 0 0 0 3 10  3
Linear Systems in Higher Dimensions
1 1 1 1 0 1 1 1 1 0
0 1 1 1 0 0
1 2 3  2 0 3  2 0
  1 2
 1 2 0  2
0 0 3 6  3 0 0 3 6  3
  0 0 3 0  3
 
0 0 0 4 0 0 0 0 1 0  
0 0 0 1 0

1 0 0 0 1 1 1 0 0 1 1 1 1 0 0
0 0 0  2
 1 0 0 0  0
 1 0 0 0   1 2
0 0 1 0  1 0 0 1 0  1 0 0 1 0  1
     
0 0 0 1 0 0 0 0 1 0 0 0 0 1 0
Booking with Elimination Matrices
10
 00
1
 11101 1110

110
0
1 234
2 0 123
2
   

101
0
1 3610

5 0 259
5
   

1001
1
 410
20
9
0 3919
9

1 0 0 0  1 1 1 1 0  1 1 1 1 0 
0 1 0 0  0 1 2 3  2   0 1 2 3  2 
 
0  2 1 0  0 2 5 9  5  0 0 3 6  3
    
0  3 0 1  0 3 9 19  9   0 0 3 10  3 

1 0 0 0  1 1 1 1 0  1 1 1 1 0 
0 1 0 0  0 1 2 3  2   0 1 2 3  2 
 
0 0 1 0  0 0 3 6  3  0 0 3 6  3
    
0 0  1 1  0 0 3 10  3   0 0 0 4 0 
Multiplying Elimination Matrices

1 0 0 0  1 1 1 10  1 1 1 1 0 
 1 1 0 0  1 2 3 4  2   0 1 2 3  2 
 
 1  2 1 0  1 3 6 10  5   0 0 3 6  3
    
 1  3  1 1  1 4 10 20  9   0 0 0 4 0 
 Inverse Matrices
• In 1 dimension

3x  9
x  3 9  3
1

3 3  33  1
1 1
 Inverse Matrices
• In high dimensions

Ax  b
Can we write?
1
xA b
1
Is there a matrix A such that?
1 1
A A  AA  I
 Inverse Matrices
• In 1 dimension
1
0 does not exist
1
a exists iff a  0
• In higher dimensions
1
When does A not exist?
singular matrices!!!
Some Special Matrices and
Their Inverses
I 1  I

1
d1  1 / d1 
     
   
 d n   1 / d n 
 Inverses in Two Dimensions
1
a b  1  d  b
 c d   ad  bc  c a 
   
Proof:
1  d  b  a b  1 ad  bc 0 
       I
ad  bc  c a   c d  ad  bc  0 ad  bc

a b  1  d  b   1 ad  bc 0 
 c d  ad  bc  c a    ad  bc  0 
ad  bc 
I
    
 Uniqueness of Inverse Matrices
BA  I and AC  I then
B  C

Proof :
B  BI  B AC   BAC  BAC  IC  C
Inverse and Linear System
if A is invertible then
Ax  b
has a unique solution given by
1
A b

Proof :
1 1
A Ax  A b
Ix  A1b
x  A1b
 Inverse and Linear System
• Therefore, the inverse of A exists if and
only if elimination produces n non-zero
pivots (row exchanges allowed)
 Inverse, Singular Matrix and
Degeneracy
Suppose there is a nonzero vector x such that Ax =
0 [column vectors of A co-linear] then A cannot
have an inverse
Proof :
1 1
A Ax  A 0
x0
Contradiction:
So if A is invertible, then Ax =0 can only have
the zero solution x=0
 One More Property
AB  1
B A 1 1

Proof
 
B 1 A1  AB   B 1 A1 A B  B 1 B  I
So

ABC  1 1
C B A 1 1
 Gauss-Jordan Elimination for
Computing A -1

• 1D
1
ax  1 implies x  a
• 2D
 a11 a12   x1  1  a11 a12   y1  0
a a   x   0 and a a   y   1
 21 22   2     21 22   2   
then
 a11 a12   x1 y1  1 0
 a a   x y   0 1 
 21 22   2 2   
 Gauss-Jordan Elimination for
Computing A -1

• 3D
 a11 a12 a13   x  1  a11 a12 a13   y  0
a a a   1   0 , a a a   1   1  and
 21 22 23   x2     21 22 23   y2   
a31 a32 a33   x3  0  a31 a32 a33   y3  0
 a11 a12 a13   z  0
 a a a   z 1   0 
 21 22 23   2   
 a31 a32 a33   z3  1 
then
 a11 a12 a13   x1 y1 z1  1 0 0
 a a a   x y z   0 1 0 
 21 22 23   2 2 2   
 a31 a32 a33   x3 y3 z3  0 0 1
 Gauss-Jordan Elimination for
Computing A -1

• 3D: Solving three linear equations defined

by A simultaneously

• n dimensions: Solving n linear equations

defined by A simultaneously

A 1
A I   I , A 1

 Example:Gauss-Jordan
Elimination for Computing A -1

 2 1 0  1 0 0
 1 2  1 X  0 1 0
   
 0  1 2  0 0 1
• Make a Big Augmented Matrix
 2  1 0 1 0 0
 
  1 2  1 0 1 0 
 0  1 2 0 0 1
 Example:Gauss-Jordan
Elimination for Computing A -1

 2  1 0 1 0 0 2  1 0 1 0 0
   
  1 2  1 0 1 0 0 3 / 2   1 1 / 2  1 0
 0  1 2 0 0 1 0  1 2 0 0 1

2  1 0 1 0 0
 
0 3 / 2  1 1 / 2 1 0
0 0 4 / 3 1 / 3 2 / 3 1
 Example:Gauss-Jordan
Elimination for Computing A -1
2  1 0 1 0 0
 
 0 3 / 2   1 1 / 2  1 0 
0 0 4 / 3 1 / 3 2 / 3 1

2  1 0 1 0 0 
 
 0 3 / 2  0 3 / 4  3 / 2  3 / 4 
0 0 4 / 3 1 / 3 2 / 3 1 

2 0 0 3 / 2 1 1 / 2
 
0 3 / 2 0 3 / 4 3 / 2 3 / 4 
0 0 4 / 3 1 / 3 2 / 3 1 
 Example:Gauss-Jordan
Elimination for Computing A -1

2 0 0 3 / 2 1 1 / 2
 
0 3 / 2  0 3 / 4 3 / 2 3 / 4
0 0 4 / 3 1 / 3 2 / 3 1 

2 0 0 3 / 4 1 / 2 1 / 4
 
0 1 0 1 / 2  1 1 / 2
0 0 1 1 / 4  1 / 2 3 / 4