Theorem on nonsingular matrix properties

For an n × n matrix, A, the following properties are equivalent:

1. The inverse of A exists; that is, A is nonsingular.
2. The determinant of A is nonzero.
3. The rows of A form a basis for R n .
4. The columns of A form a basis for R n .
5. As a map from R n to R n , A is injective (one to one).
6. As a map from R n to R n , A is surjective (onto).
7. The equation Ax = 0 implies x = 0.
8. For each b ∈ R n , there is exactly one x ∈ R n such that
Ax = b.
9. A is a product of elementary matrices.
10. 0 is not an eigenvalue of A.

21 / 143
Review: eigenvalue problems

I Recall the definition of eigenvalue problem:

I What the geometrical interpretation of the eignvalue

problems?

I (In-class exercise). Find the eigenvalue of A given by

 
2 −1
−1 2 2×2

by hand.
I Is the matrix A nonsingular?

22 / 143
I Positive Definite: A matrix A is positive definite if
x T Ax > 0 for every nonzero vector x.
 
2 1
I Example: A = is positive definite, since
1 2
  
T 2 1 x1
= (x1 +x2 )2 +x12 +x22 > 0
 
x Ax = x1 x2
1 2 x2

for all x1 and x2 except x1 = x2 = 0.

I In-class exercise. If A is positive definite and symmetric,
then its eigenvalues are positive.
Note that all eigenvalues of the SPD matrix A are real.

23 / 143
LU and Cholesky Factorizations

24 / 143
I How to solve linear systems?
I Let us begin with the Easy-to-Solve Systems
1. Diagonal Structure
  x   b1

a11 0 0 ··· 0 1
 0 a22 0 · · ·  x2   b2
0 
 
  
  x3   b3
 0 0 a · · · 0 =

33

  ..   ..

 ···  .   .


0 0 0 · · · ann xn bn

The solution is: x = [b1 /a11 b2 /a22 b3 /a33 · · · bn /ann ]T

I If aii = 0 for some index i, and if bi = 0 also, then xi can be
any real number. The number of solutions is infinity
I If aii = 0 and bi 6= 0, no solution of the system exists
I What is the complexity of the method? n divisions

25 / 143
2. Triangular linear systems
   
  x1 b1
a11 0 0 ··· 0  x2   b2 
 a21 a22 0 ··· 0 
   
  x3 =
  b3 
 a31 a32 a33 ··· 0 
 
..   .. 
an1 an2 an3 ··· ann
 .   . 
xn bn
I Some simple observations:
If a11 6= 0, then we have x1 = b1 /a11 .
I Once we have x1 , we can simplify the second equation

a21 x1 + a22 x2 = b2

x2 = (b2 − a21 x1 )/a22

provided that a22 6= 0
I Similarly
x3 = (b3 − a31 x1 − a32 x2 )/a33
26 / 143
I In general, to find the solution to this system, we use
forward substitution:
input n, (aij ), (bi )
for i = 1 to n do Pi−1 
xi ← bi − j=1 aij xj /aii
end do
output (xi )
I Complexity of forward substitution:
I n divisions
I The number of multiplications: 0 for x1 , 1 for x2 , 2 for x3 , ...
Total =
0 + 1 + 2 + · · · + (n − 1) ≈ 1 + 2 + · · · + n = (n + 1)n/2, or
Total = O(n2 )
I The number of additions: Same as the number of
multiplications = O(n2 )

27 / 143
I The complexity of an algorithm is often measured using the
unit called flop
One flop = one addition + one multiplication
I Forward substitution is an O(n2 ) algorithm
I A remark: Forward substitution is a sequential algorithm (not
parallel at all)

28 / 143
3 Another Simple System If we re-order the equations, we can
get a lower triangular system:
   
a11 a12 0 x1 b1
 a21 a22 a23   x2   b2 
a31 0 0 x3 b3

I How do we solve Ax = b if A is a permuted lower or upper

triangular matrix?
I Assuming that the permutation vector (p1 , p2 , . . . , pn ) is
known, we modify the forward substitution algorithm for a
permuted lower triangular system:
input n, (aij ), (bi ), (pi )
for i = 1 to n doP 
xi ← bpi − nj=i+1 api j xj /api i
end do
output (xi )

29 / 143
LU Factorizations

Suppose that A can be factored into the product of a lower

triangular matrix L and an upper triangular matrix U:

A = LU

Then, to solve the system of equations Ax = b, it can be done in

two stages.

Lz = b solve for z
Ux = z solve for x

30 / 143
I We begin with an n × n matrix A and search for matrices:

l11 0 0 ··· 0 u11 u12 u13 ··· u1n

   
 l21 l22 0 ··· 0   0 u22 u23 ··· u2n 
L =  l31 l32 l33 ··· 0 , U =  0 0 u33 ··· u3n 
   
   
ln1 ln2 ln3 ··· lnn 0 0 0 ··· unn

such that A = LU. When this is possible, we say that A has

an LU-decomposition.
I It turns out if we compare A = LU, we have more unknowns
(n2 + n) than equations (n2 ).
I One simple choice is to make L unit lower triangular (lii = 1
for each i).
I Another obvious choice is to make U unit upper triangular
(uii = 1 for each i).

31 / 143
I Using the formula for matrix multiplication, we have:

n min(i,j)
X X
aij = lis usj = lis usj (∗)
s=1 s=1
I At each new step k, we know rows 1, 2, ..., k − 1 for U and
columns 1, 2, ..., k − 1 for L. Setting i = j = k in (∗), we have
k−1
X
akk = lks usk + lkk ukk
s=1

I If ukk or lkk has be specified then the above equation can be

used to determine the other, e.g., if ukk = 1, then
k−1
X
lkk = akk − lks usk
s=1

32 / 143
I Use (∗) in the previous slide again to get the k-th row (i = k)
and the k-th column (j = k)
k−1
X
akj = lks usj + lkk ukj (k + 1 ≤ j ≤ n)
s=1
|{z}
unknown

k−1
X
aik = lis usk + lik ukk (k + 1 ≤ i ≤ n)
|{z}
s=1 unkown

I This algorithm is known as Doolittle’s factorization when L

us unit lower triangular and as Crout’s factorization when U
is unit upper triangular.
I When U = LT , so that lii = uii for 1 ≤ i ≤ n, the algorithm is
called Cholesky’s factorization

33 / 143
The algorithm for the general LU-factorization is as follows:
input n, (aij )
for k = 1 to n do
Specify a nonzero value for either
lkk or ukk and compute the other from
lkk ukk = akk − k−1
P
s=1 lks usk
for j = k + 1to n do 
Pk−1
ukj ← akj − s=1 lks usj /lkk
end do
for i = k + 1 to n do
Pk−1 
lik ← aik − s=1 lis usk /ukk
end do
end do
output (lij ), (uij )

34 / 143
I In-class exercise. Find the Doolittle, Crout, and Cholesky
factorizations of the matrix
 
60 30 20
A =  30 20 15 
20 15 12

35 / 143
Theorem on LU-Decomposition

I If all n leading principal minors of the n × n matrix A are

nonsingular, then A has an LU-decomposition.
[proof] See textbook p. 156.
I Recall that the kth leading principal minor of the matrix A is
the matrix:
 
a11 a22 · · · a1k
Ak =  a21 a22 · · · a2k 
ak1 ak2 · · · akk

36 / 143
Cholesky Theorem on LLT -Factorization

I If A is a real, symmetric, and positive definite matrix, then it

has a unique factorization, A = LLT , in which L is a lower
triangular matrix with positive diagonal.
[proof] See blackboard
I Recall that a matrix A is symmetric and positive definite if
A = AT and x T Ax > 0 for every nonzero vector x.
I Some key steps in the proof.
I Prove that A has an LU-decomposition by showing that all
leading principal minors of A are SPD, which implies the
non-singularity of Ak .
I Show that A = LDLT by considering LU = A = AT = U T LT .
−1 −1 T −1
Since U(LT ) = L | {zU } ∃D s.t. D = U(LT )
| {z }
upper ∆ lower ∆
I Show that dii > 0

37 / 143
Basic steps for solving a linear system

I Consider
Ax = b
I Obtain a LU decomposition

A = LU

I Solve a lower triangular system

Lz = b

I Solve an upper triangular system

Ux = z

38 / 143
Complexity analysis
I Consider the number of multiplications in the general
LU-decomposition:
k = 1, 0
k = 2, 1 + ((n − 1) ∗ 1) ∗ 2
k = 3, 2 + ((n − 2) ∗ 2) ∗ 2
......
k = n, (n − 1) + ((n − (n − 1)) ∗ (n − 1)) ∗ 2
I Total:
n−1
X n−1
X n
X X n
= i +2 (n − i) ∗ i ≈ i +2 (n − i) ∗ i
i=1 i=1 i=1 i=1
n
X Xn n
X n
X n
X
= i + 2n i −2 i 2 = (2n + 1) i −2 i2
i=1 i=1 i=1 i=1 i=1
(2n + 1)n(n + 1) 2n(n + 1)(2n + 1)
= −
2 6
1
= n(n + 1)(2n + 1)
6
39 / 143
I In the LU decomposition phase, the cost is O(n3 )
I In the triangular solve phases, the cost is O(n2 )
I Total cost is O(n3 ) or more precisely

1 3
n + O(n2 )
3
I Remark: Once L and U are obtained, A is no longer needed.
One can over-write A with L and U.

40 / 143