Matrix norm and low-rank approximation

Second, for any k ∈ R,

kkAk = max k(kA)uk = |k| · max kAuk = |k| · kAk.

u∈Rn :kuk=1 u∈Rn :kuk=1

Lastly, for any two matrices A, B ∈ Rm×n ,

kA + Bk = max k(A + B)uk = max kAu + Buk

u∈Rn :kuk=1 u∈Rn :kuk=1

≤ max (kAuk + kBuk)

u∈Rn :kuk=1

≤ max kAuk + max kBuk

u∈Rn :kuk=1 u∈Rn :kuk=1

= kAk + kBk.

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 13/49
 Matrix norm and low-rank approximation

Theorem 0.4. For any norm on Euclidean spaces and its induced matrix operator
norm, we have

kAxk ≤ kAk · kxk for all A ∈ Rm×n , x ∈ Rn

Proof. For all x 6= 0 ∈ Rn , by definition,

kAxk
≤ kAk
kxk

This implies that

kAxk ≤ kAk · kxk.

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 14/49
 Matrix norm and low-rank approximation

Remark. More generally, the matrix operator norm satisfies

kABk ≤ kAk · kBk, for all A ∈ Rm×n , B ∈ Rn×p

Matrix norms with such a property are called sub-multiplicative matrix norms.

The proof of this result is left to you in the homework (you will need to apply the
theorem on the preceding slide).

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 15/49
 Matrix norm and low-rank approximation

When the Euclidean norm (i.e., 2-norm) is used, the induced matrix operator
norm is called the spectral norm.
Def 0.2. The spectral norm of a matrix A ∈ Rm×n is defined as
kAxk2
kAk2 = max = max kAuk2
x6=0 kxk2 u∈Rn :kuk2 =1

Theorem 0.5. The spectral norm of any matrix coincides with its largest singular
value:
kAk2 = σ1 (A).

Proof.
kAxk22 xT AT Ax
kAk22 = max = max = λ1 (AT A) = σ1 (A)2 .
x6=0 kxk22 x6=0 xT x
The maximizer is the largest right singular vector of A, i.e. v1 .

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 16/49
 Matrix norm and low-rank approximation

Example 0.3. For the matrix

 
1 −1
X = 0 1 ,
 
1 0
√
we have kXk2 = 3.

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 17/49
 Matrix norm and low-rank approximation

We note that the Frobenius and spectral norms of a matrix correspond to the 2-
and ∞-norms of the vector of singular values (σ = (σ1 , . . . , σr )):

kAkF = kσk2 , kAk2 = kσk∞

The 1-norm of the singular value vector is called the nuclear norm of A, which is
very useful in convex programming.

Def 0.3. The nuclear norm of a matrix A ∈ Rn×d is defined as

X
kAk∗ = kσk1 = σi .

√
Example 0.4. In the last example, kXk∗ = 3 + 1.

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 18/49
 Matrix norm and low-rank approximation

MATLAB function for matrix/vector norm

norm – Matrix or vector norm.

norm(X,2) returns the 2-norm of X.

norm(X) is the same as norm(X,2).

norm(X,’fro’) returns the Frobenius norm of X.

In addition, for vectors...

norm(V,P) returns the p-norm of V defined as SUM(ABS(V).ˆP)ˆ(1/P).

norm(V,Inf) returns the largest element of ABS(V).

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 19/49
 Matrix norm and low-rank approximation

Condition number of a matrix

Briefly speaking, the condition number of a square, invertible matrix is a measure
of its near-singularity.

Def 0.4. Let A be any square, invertible matrix. For a given matrix operator
norm k · k, the condition number of A is defined as

κ(A) = kAkkA−1 k

kIxk
Remark. κ(A) ≥ kAA−1 k = kIk = maxx6=0 kxk = 1.

Remark. Under the matrix spectral norm,

1 σmax (A)
κ(A) = kAk2 kA−1 k2 = σmax (A) · =
σmin (A) σmin (A)

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 20/49
 Matrix norm and low-rank approximation

1st perspective (for understanding the matrix condition number): Let

kAxk
M = max = kAk
x6=0kxk
kAxk kyk 1 1
m = min = min −1
= −1 yk =
x6=0 kxk y6=0 kA yk kA
maxy6=0 kyk kA−1 k

which respectively represent how much the matrix A can stretch or shrink vectors.
Then
M
κ(A) =
m
If A is singular, then there exists a nonzero x such that Ax = 0. Thus, m = 0
and the condition number is infinity.

In general, a finite, large condition number means that the matrix is close to being
singular. In this case, we say that the matrix A is ill-conditioned (for inversion).

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 21/49
 Matrix norm and low-rank approximation

2nd perspective: Consider solving a system of linear equations subject to

measurement error:

Ax = b + |{z}
e (A is assumed to be exact)
error

We would like to see how the error term e (along with b) affects the solution x
(via the matrix A).

Since A is invertible, the linear system has the following solution

x = A−1 (b + e) = A −1
| {z b} + A −1
| {z e}
true solution error in solution

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 22/49
 Matrix norm and low-rank approximation

The ratio of the relative error in the solution to the relative error in b is
kA−1 ek / kA−1 bk kA−1 ek kbk
= ·
kek / kbk kek kA−1 bk

where k · k represents a given vector norm.

The maximum possible value of the above ratio (for nonzero b, e) is then
 −1   −1   
kA ek kbk kA ek kbk
max · = max max
b6=0,e6=0 kek kA−1 bk e6=0 kek b6=0 kA−1 bk
 −1   
kA ek kAyk
= max max
e6=0 kek y6=0 kyk
= kA−1 k · kAk
= κ(A).

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 23/49
 Matrix norm and low-rank approximation

Remark.

• If the condition number of A is large (i.e., ill-conditioned), then the relative

error in the solution x is much larger than the relative error in b, and thus
even a small error in b may cause a large error in x.

• On the other hand, if this number is small, then the relative error in x will
not be much bigger than the relative error in b.

• In the special case when the condition number is exactly one, the solution
has the same relative error with the data b.

See a MATLAB demonstration by C. Moler.1

1
https://blogs.mathworks.com/cleve/2017/07/17/
what-is-the-condition-number-of-a-matrix/

Dr. Guangliang Chen | Mathematics & Statistics, San José State University 24/49