Digital Image Processing

(Code: 455)

National Institute of Technology Delhi

Contents
• Linear Algebra Basics
Book: Linear Algebra. An Introduction. Second Edition. RICHARD
BRONSON
Introduction to Linear Algebra: G.Strang
• Image Transforms
Space (mathematics)
Linear Algebra Basics: Vector space

• Vector space: a set that is closed under finite vector addition and
scalar multiplication. (also called a linear space)
Vector space
Vector space
• If the scalars are restricted to be real numbers, then V is called a real
vector space;
• If the scalars are allowed to be complex numbers, then V is called a
complex vector space.
• Here assume that all scalars are real and that we are dealing with real
vector space.
SUBSPACE

• We define a nonempty subset S of a vector space V as a subspace of V

if S is itself a vector space under the same operations of vector
addition and scalar multiplication defined on V.
• LINEAR INDEPENDENCE
BASIS AND DIMENSION

• A set of vectors S in a vector space V as a spanning set for V if every

vector in V can be written as a linear combination of the vectors in S;
that is, if V = span{S}.
BASIS AND DIMENSION

• A basis for a vector space V is a set of vectors that is

linearly independent and also spans V.
• The dimension of a vector space is the number of vectors in
a basis for that vector space.
 SVD - Overview

A technique for handling matrices (sets of equations) that do not have

an inverse. This includes square matrices whose determinant is zero
and all rectangular matrices.

Common usages include computing the least-squares solutions, rank,

range (column space), null space and pseudoinverse of a matrix.
SVD - Basics

The SVD of a m-by-n matrix A is given by the formula :

A = UWV T

Where :
U is a m-by-m matrix of the orthonormal eigenvectors of
AAT
VT is the transpose of a n-by-n matrix containing the
orthonormal eigenvectors of ATA
W is a m-by-n Diagonal matrix of the singular values
which are the square roots of the eigenvalues of ATA
SVD

𝑤1 0 0 T
𝐀 = 𝐔 0 ⋱ 0 𝐕
0 0 𝑤𝑛

• Treat as black box: code widely available

In Matlab: [U,W,V]=svd(A,0)
SVD
• The wi are called the singular values of A
• If A is singular, some of the wi will be 0
• In general rank(A) = number of nonzero wi
• SVD is mostly unique (up to permutation of singular values, or if some
wi are equal)
Singular Value Decomposition
For an m n matrix A of rank r there exists a factorization
(Singular Value Decomposition = SVD) as follows:
A = UV T

m m m n V is nn

The columns of U are orthogonal eigenvectors of AAT.

The columns of V are orthogonal eigenvectors of ATA.
Eigenvalues 1 … r of AAT are the eigenvalues of ATA.
 i = i
 = diag( 1... r ) Singular values.
Singular Value Decomposition
• Illustration of SVD dimensions and sparseness
SVD example
1 − 1
Let 
A= 0 1 
 1 0 
Thus m=3, n=2. Its SVD is

 0 2/ 6 1/ 3  1 0 
   1 / 2 1/ 2 
1 / 2 −1/ 6 1 / 3  0 3 
1 / 2  1/ 2 −1/ 2 
 1/ 6 − 1 / 3   0 0 

Typically, the singular values arranged in decreasing order.

Low-rank Approximation
• SVD can be used to compute optimal low-rank
approximations.
• Approximation problem: Find Ak of rank k such that

Ak = min A− X F Frobenius norm

X :rank ( X ) = k

Ak and X are both mn matrices.

Typically, want k << r.
Low-rank Approximation
• Solution via SVD
Ak = U diag ( 1 ,..., k ,0,...,0)V T
set smallest r-k
singular values to zero

Ak = i =1 i ui viT
k
column notation: sum
of rank 1 matrices
Approximation error
• How good (bad) is this approximation?
• It’s the best possible, measured by the Frobenius norm of the error:

min A− X F
= A − Ak F
=  k +1
X :rank ( X ) = k

where the i are ordered such that i  i+1.

Suggests why Frobenius error drops as k increased.
The Singular Value Decomposition
(SVD)
Data matrices have n rows (one for each
object) and d columns (one for each
feature).

feature 2
Rows: vectors in a Euclidean space,
Object d

Two objects are “close” if the angle (d,x) Object x

between their corresponding vectors is
small. feature 1
SVD: Example
Input: 2-d dimensional points

5 Output:

2nd (right)
singular 1st (right) singular vector:
vector direction of maximal variance,
4

2nd (right) singular vector:

direction of maximal variance, after
3 removing the projection of the
data along the first singular vector.
1st (right) singular
vector
2
4.0 4.5 5.0 5.5 6.0
Singular values
5

2nd (right)
singular 1: measures how much of the
4
vector data variance is explained by the
first singular vector.

2: measures how much of the

3 data variance is explained by the
1 second singular vector.
1st (right) singular
vector
2
4.0 4.5 5.0 5.5 6.0
SVD decomposition

nxd nxℓ ℓxℓ ℓxd

U (V): orthogonal matrix containing the left (right) singular

vectors of A.
: diagonal matrix containing the singular values of A:
(1 ≥ 2 ≥ … ≥ ℓ )

Exact computation of the SVD takes O(min{mn2 , m2n}) time.

The top k left/right singular vectors/values can be computed
faster using Lanczos/Arnoldi methods.
Why are orthonormal bases good?
• ( )-1=( )T
• Implication:
• Matrix inversion
A = UV T
• Ax=b −1
A = UV ( T −1
) = V −1U T
A = UV T
Ax = b
(UV )x = b
T

(V x ) = (U b )... diagonal system

T T

49
More on U and V
 2
 T V: eigenvectors of ATA
A A = V U UV = V 
T T T T 1
2
V
 2 
U: eigenvectors of AAT
Similarly,
 2
 T
AA = UV V U = U 
T T T T 1
2
U
 2 
[Find vi first, then use Avi to find ui

This is the
key to solve
SVD
50
SVD: A=UVT
• The singular values are the diagonal entries of the  matrix and are
arranged in descending order
• The singular values are always real (non-negative) numbers
• If A is real matrix, U and V are also real

51
Example (2x2, full rank)
 2 2
A= 
 − 1 1 
5 3 1 2 − 1 2 
A A=
T
 , v1 =   , v2 =  
3 5 1 2 1 2 
2 2  1 1 STEPS:
Av1 =   =  1  , u1 =   1. Find e-vectors of ATA;
 0  0  0  normalize the basis
2. Compute Avi, get i
0  0  0 
Av2 =   =  2  , u2 =   If i  0, get ui
 2 1 1 Else find ui from N(AT)

1 0 2 2 0  1 2 1 2
A = UV = 
T
  
 0 1  0 2  − 1 2 1 2
52
SVD Theory
AV = U
→ Av j =  j u j , j = 1,2, , r
• If j=0, Avj=0→vj is in N(A)
• The corresponding uj in N(AT)
• [UTA=VT=0]
• Else, vj in C(AT)
• The corresponding uj in C(A)
• #of nonzero j = rank

53
Example 2(2x2, rank deficient)
2  1 1  1 
A=  , r = 1, C ( A ) basis :  ; v1 =
T
1
 1 1  
1 2
Can also be
Av1 =  1u1 obtained from
2 2 1 1 1  2 e-vectors of ATA
1 1  1 =  1 1 →  1 = 10
  2 5 
1 1
v2 ⊥ v1 → v2  N ( A) → Av2 = 0 → v2 = − 1
2  
1 1
u2 ⊥ u1 → u2  N ( A ) → A u2 = 0 → u A = 0 → u1 =
T T T
 
5  − 2
2

 2 2 1 2 1   10 0 1 1 1 
1 1 = UV =
T
1 − 2    1 − 1
  5   0 0 2  
54
Example (cont)
 2 2 1 2 1   10 0 1 1 1 
1 1 = UV =
T
1 − 2    1 − 1
  5   0 0  2  

 1 0 v1T 
A = UV T = u1 u2    T
 0 0  v2 
 1v1T 
= u1 u2   =  u v
1 1 1
T

 0 

Bases of N(A) and N(AT) (u2 and v2 here) do not contribute

the final result. The are computed to make U and V
orthogonal.
55
Extend to Amxn
Avi =  i ui , i = 1,, r
 1 
  
Av1  vr vr +1  vn  = u1  ur ur +1 um  
 r 
 
Basis of N(A) Basis of N(AT) 0
Dimension Check
AV = U
Amn = U mm  mnVnTn
( A A)
T
nn (
= UV ) (UV ) = V ( ) V
T T T
nn
T
nn
T
nn

(AA ) T
mm = (UV )(UV ) = U (  ) V
T T T
mm
T
mm
T
mm 56
Extend to Amxn (cont)
 1 
  
Av1  vr vr +1  vn  = u1  ur ur +1  um  
 r 
 
 0

Summation of r
v  T
1 rank-one matrices!
 
  
 vrT 
A =  1u1   r ur 0 0 T  =  1u1v1T +  +  r ur vrT
vr +1 
  
Bases of N(A) and N(AT)  T  They are useful only for
vm 
do not contribute nullspace solutions
57
 1 1 0
A=  , r = 2, A23V33 = U 22  23
0 1 1 
1 1 0

V : A A = 1 2 1  1 3 3 2 1
Av1 =  = 1,  1 = 3
T 1

6 3 6 2

0 1 1
1 1
Av2 = − 1,  2 = 1
2 
1 
1    
1 = 3, v1 =  2
1 1 1

6 1 1 0  6 2 3
 1 1 1   3 
=
−1
1 0 1 1  
2
0 1 − 1  
   
6 3
1 −1 1  2 1 
1  6 2 3 
1  
2 = 1, v2 =  0
2 C(AT) N(A) C(A)
− 1  1 2 1 
1 1 0 1 1 1   3  6 6 2

  0 1 1  =
−1
1 − 1  
1
1 0 
   
2 2
1   2 1  
3 = 0, v3 = − 1 (nullspace) 
1 −1 1

3  3 3 3
 1 

58
 1 2 1  6 10 6
   
Example :A = 2 3 2 , AA T = A T A = 10 17 10 
1 2 1  6 10 6 
 
Eigen values of A T A ,A T A
28 .86 
 
 0 .14 , Number of non - zero singularvalues = rank of A
 0 
 
Eigen vectors of A T A ,A T A
0 .454   0 .542  − 0 .707 
     
u 1 = v 1 = 0 .776  u 2 = v 2 = − 0 .643  u 3 = v 3 =  0 
0 .454   0 .542  − 0 .707 
     
Expansion of A
2
A = 
i
 iu iv Ti
=1

59
Summary
• SVD chooses the right basis for the 4 subspaces
• AV=U
• v1…vr: orthonormal basis in Rn for C(AT)
• vr+1…vn: N(A)
• u1…ur: in Rm C(A)
• ur+1…um: N(AT)
• These bases are not only ⊥, but also Avi=iui
• High points of Linear Algebra
• Dimension, rank, orthogonality, basis, diagonalization, …

60
SVD Applications
• Using SVD in computation, rather than A, has the advantage of being more
robust to numerical error
• Many applications:
• Inverse of matrix A
• Conditions of matrix
• Image compression
• Solve Ax=b for all cases (unique, many, no solutions; least square solutions)
• rank determination, matrix approximation, …
• SVD usually found by iterative methods (see Numerical Recipe, Chap.2)

61
Matrix Inverse
A isnonsingular iff i  0 foralli
A = U V T or A −1 = V  −1U T where  −1 = diag (1/ 1,1/ 2 ,....,
1/ n )
A issingularor ill
- conditioned

(
A −1 = U V T )
−1
 V 0−1U T 1/ i if  i  t
where 0−1 = 
 0 otherwise

Degree of singularit y of a matrix

System of linear equations
Ax = b
Ill - conditioned when small changes in b produces large changes in x
Degree of singularit y A := σ1 / σ n

62
SVD and Ax=b (mn)
• Check for existence of solution

UV x = b → Vx = U
T
b
T T

z d

z = d
If  i = 0 but d i  0,
solution does not exist

63
Ax=b (inconsistent)

 2 2 8
A=  ,b =  
1 1  3
 2 2 1 2 1   10 0 1 1 1 
1 1 = UV =
T
1 − 2    1 − 1
  5   0 0  2  
 10 0  z1  1 2 1  8 20 5 
  = U b = =
T
     
 0 0  2 
z 5  1 − 2    2 5 
3

No solution!

64
Ax=b (underdetermined)
 2 2 8 
A=  ,b =  
1 1   4
 2 2 1 2 1   10 0 1 1 1 
1 1 = UV =
T
1 − 2    1 − 1
  5   0 0  2  
 10 0  z1  1 2 1  8 20 5 
  = U b = =
T
     
 0 0  z 2  5 1 − 2   4   0 
 z1  2 2  1 1 1  2 2  2
z  =   → x particular = Vz = 1 − 1  = 
 2  0  2   0   2
 2  1 2 
xcomplete = x particular + xnull =   + c  
  − 1 2 
2
65
Pseudo Inverse
(Sec7.4, p.395)
• The role of A:
• Takes a vector vi from row space to iui in the column space
• The role of A-1 (if it exists):
• Does the opposite: takes a vector ui from column space to row space vi

Av i =  i ui
v i =  i A−1ui
A−1ui = 1i v i

66
Pseudo Inverse (cont)
• While A-1 may not exist, a matrix that takes ui back to vi/i does exist. It is
denoted as A+, the pseudo inverse
• A+: dimension n by m
+ 1
A ui = vi for i  r and A+ ui = 0 for i  r
i
A+ = Vnn  +nmU mT m
 1−1  T
  
= v1  vr  vn    u  u  u 
  r−1  1 r m

   
 
67
Pseudo Inverse and Ax=b
Ax = b A panacea for Ax=b
+ +
x = A b = V U b T

• Overdetermined case: find the solution that minimize the error r=|Ax–
b|, the least square solution

Compare A T A xˆ = A T b

68
Ex: full rank
 2 2 0
A=  ,b =  
− 1 1   − 2
1 0 2 2 0   1 2 1 2 
A = UV = 
T
  
 0 1  0 2  − 1 2 1 2 
Ax = b → UV T x = b → x = Vdiag(1 /  i )U T b
1 −1 2  2 2 0  1 0  0   1 
1
2
x=  0 1     = 
1 2 1 2  2 
0 1 − 2 − 1

69
Ex: over-determined
1 0 1
A = 1 1, b =  2 , r = 2, A32V22 = U 33 32
0 1 − 1

1 0  16 12 13   3 
1 1 =  2 0 −1    1 1 1 
3 
   6 1 1 − 1
0 1  16 −12 13    2  
  
A = UV T
A+ = V +U T
Will show this need not
be computed…

70
Over-determined (cont)
x = V +U T b Compare A T A xˆ = A T b
 16 26 16   1  2 1  x 1  3 
1 1 1   1
 1 −1       =  
=  
3
 2 0 2 
2   1 2  x 2   1 
2 1 − 1  1   x y z  − 1
    x 1   53 
 4
   =  −1 
1 1 1   1
 6
 x 2   3 
= 1 − 1 
3

2

 
2
2 1 x + 2 y − z
 
1 1 1   2 3 2   53  Same
= 1 − 1   =  −1 
2   2   3 
result!!

71
Ex: general case, no solution
 2 2 8
A=  ,b =  
1 1  3
 2 2 1 2 1   10 0 1 1 1 
1 1  = UV =
T
1 − 2    1 − 1
  5   0 0 2  
x = V +U T b
2 x + 2 y = 8
 1
x 1
0  2 1
 8 
=   x+ y =3
2

0  x y  3
10 5 5
1
 2
y  0
 2 1 5 0  25 1
 8  15 1
 8  1910 
= 1  
5
  = 1 10
   =  19 
 2 5 0   x y  3  5  3  10 
1
10

72
Matrix Approximation
Ai = U iV T
 i : the rank i version of  (by setting last m − i  ' s to zero)
Ai : the best rank i approximation to A in the sence of Euclidean distance

A =  1u1v1T +  2u2 v2T +  +  mum vmT

Storage save : rank one matrix (m + n) numbers
Operation save : m + n

making small ’s to zero and back substitute

(see next page for application in image compression)

73
Image Compression
• As described in text p.352
• For grey scale images: mn bytes
After SVD, taking the most significan t r terms :
A =  1u1v1T +  2u2v2T +  +  r ur vrT
• Only need to store r(m+n+1)

Original
r = 1,3,5,10,16 (no perceivable difference afterwards)
6464 74
75
76
Euclidean inner product

• The inner product of two vectors x and y in 𝑅𝑛 is a real number

determined by multiplying corresponding components of x and y and
then summing the resulting products.
Euclidean inner product
Euclidean inner product
Euclidean inner product
Dot product geometric intuition:
“Overlap” of 2 vectors
Euclidean inner product
Euclidean inner product
Element-by-element product (Hadamard
product)

• Element-wise multiplication (.* in MATLAB)

Multiplication: Outer product
Matrix times a vector

MX1 MXN NX1

Matrix times a vector:
inner product interpretation

• Rule: the ith element of y is the dot product of

the ith row of W with x
Matrix times a vector:
inner product interpretation

• Rule: the ith element of y is the dot product of

the ith row of W with x
Matrix times a vector:
inner product interpretation

• Rule: the ith element of y is the dot product of the ith

row of W with x
Matrix times a vector:
inner product interpretation

• Rule: the ith element of y is the dot product of the ith

row of W with x
Basic concept Vector norms
• Vector in Rn is an ordered set
of n real numbers. 1
 
• e.g. v = (1,6,3,4) is in R4 6
 3
• A column vector:  
 4
• A row vector:  
• m-by-n matrix is an object in (1 6 3 4)
Rmxn with m rows and n
columns, each entry filled
with a (typically) real 1 2 8
number:  
 4 78 6 
9 3 2
 
Basic concepts
Vector norms: A norm of a vector ||x|| is informally a
measure of the “length” of the vector.

– Common norms: L1, L2 (Euclidean)

– Linfinity - L0 Norm
- 𝑥 0 = counts the total
number of nonzero
vector X= [-6, 4, 2], the elements of a vector.
L-infinity norm is 6.
Basic concepts
We will use lower case letters for vectors The elements are
referred by xi.
• Vector dot (inner) product:

If u•v=0, ||u||2 != 0, ||v||2 != 0 → u and v are orthogonal

• Vector outer product:
If u•v=0, ||u||2 = 1, ||v||2 = 1 → u and v are orthonormal
Basic concepts
We will use upper case letters for matrices. The elements
are referred by Ai,j.
• Matrix product:

e.g.  a11 a12   b11 b12 

A =  , B =  
 a21 a22   b21 b22 
 a11b11 + a12b21 a11b12 + a12b22 
AB =  
 a21b11 + a22b21 a21b12 + a22b22 
 Vector Norms
DEF: A norm is a function  : Rn → R that satisfies

p-norms: The most important class Example: 2

x =  5 
of vector norms
 − 3
x 1 = 10

x2= 4 + 25 + 9  6.1644

x 
= 5

x p
= p
2 p + 5p + 3p
 Vector Norms

Example: S = {x  R n : x  1}

{x  R n : x 2  1} {x  R n : x  1}


{x  R : x 1  1}
n {x  R n : x p
 1}
 Matrix Norm Induced by Vector Norm
DEF: the matrix norm of A (induced by the vector norm) is defined to be

Ax
A m,n = sup m A m,n = sup Ax m
0  xR n x x n =1
n

DEF: If the matrix A is a square matrix

A n = sup Ax n
x n =1
 Matrix Norm Induced by Vector Norm
DEF: If the matrix A is a square matrix
A n = sup Ax n
x n =1

Example: 1 2
A= 
0 2 
The unit vector x that is
amplified most by A is
[0,1]^T, the amplification
factor is 4.
A1=4
Special matrices
 a 0 0 a b c
   
 0 b 0 diagonal 0 d e upper-triangular
0 0 c 0 0
   f 

a b 0 0 a 0 0
   
c d e 0
b c 0  lower-triangular
0  tri-diagonal

f g h

d e f 
0 
 0 i j 

1 0 0
 
 0 1 0 I (identity matrix)
0 0 1
 
Basic concepts
Transpose: You can think of it as
• “flipping” the rows and columns
OR
• “reflecting” vector/matrix on line

T
e.g. a
  = (a b )
b
T
a b  a c 
  =  
c d  b d 
 Matrix Norm Induced by Vector Norm
DEF: If the matrix A is a square matrix
A n = sup Ax n
x n =1

Example: 1 2
A= 
0 2 
The unit vector x that is
amplified most by A is
[0,1]^T, the amplification
factor is 4.
A1=4
 Matrix Norm Induced by Vector Norm
DEF: If the matrix A is a square matrix
A n = sup Ax n
x n =1

Example: 1 2
A= 
0 2 

The unit vector x that is amplified most by A is the vector

indicated by the dashed line, the amplification factor is 2.9208. A 2 = 2.9208
 Holder Inequalities
Rem
Ax n
 An x n

Cauchy-Schwarz:
𝑥𝑇𝑦 ≤ 𝑥 2 𝑦 2

Holder Inequality: 1  p, q   1 1
+ =1
p q

𝑥𝑇𝑦 ≤ 𝑥 𝑝 𝑦 𝑞
Image Transforms
• Image Transforms
Image Transforms
• Transform is basically a mathematical tool, which allows us to move
from one domain to another domain (time domain to the frequency
domain).
• The reason to migrate from one domain to another domain is to
perform the task at hand in an easier manner. Image transforms are
useful for fast computation of convolution and correlation.
• Transforms change the representation of a signal by projecting it onto
a set of basis functions.
• The transforms do not change the information content present in the
signal.
NEED FOR TRANSFORM

• (i) Mathematical Convenience: Every action in time domain will have

an impact in the frequency domain.

• (ii) To Extract more Information: Transforms allow us to extract more

relevant information.
NEED FOR TRANSFORM

• First, the transformation may isolate critical components of the image

pattern so that they are directly accessible for analysis.
• Second, the transformation may place the image data in a more
compact form so that they can be stored and transmitted efficiently.
• The different types of image transforms discussed in this section
are
1. Fourier transform 2. Walsh transform 3. Hadamard transform
4. Slant transform 5. Discrete cosine transform
6. KL transform 7. Radon transform 8. Wavelet transform
Why Do Transforms?
• Fast computation
• E.g., convolution vs. multiplication for filter with wide support
• Conceptual insights for various image processing
• E.g., spatial frequency info. (smooth, moderate change, fast change, etc.)
• Obtain transformed data as measurement
• E.g., blurred images, radiology images (medical and astrophysics)
• Often need inverse transform
• May need to get assistance from other transforms
• For efficient storage and transmission
• Pick a few “representatives” (basis)
• Just store/send the “contribution” from each basis
Classification of Image Transforms

• Image transforms can be classified based on the nature of the basis

functions as
• (i) transforms with orthogonal basis functions,
• (ii) transforms with non-sinusoidal orthogonal basis functions,
• (iii) transforms whose basis functions depend on the statistics of the
input data, and
• (iv) directional transformation, that is transforms whose basis
functions are capable of representing the directional information
present in the image.
 Basis for a Vector Space
• Every vector in the space is a linear combination of
basis vectors.
• n independent vectors from an nth dimensional
vector space form a basis.
• Orthogonal Basis: Every two basis vectors are
Orthogonal.
• Orthonormal Basis: The absolute value of all basis
vectors is 1.
1 1 1 − 1 − 1 − 1 − 1 1 
1  , 1 
− 1
,  1  ,  1 − 1
 1   1   
 i
Complex Numbers
b


a
a + bi = R  ei

i
e = cos( ) + i sin( )

Absolute Value: Phase:

R = (a + b ) b
 = tan  
−1
2 2

a
i1 i 2 i (1 + 2 )
R1e  R2e = R1R2e
Fourier Spectrum
Fourier: F (u) = R(u) + iI (u)

Fourier Spectrum F (u ) = R 2 (u ) + I 2 (u )

Fourier Phase  (u) = tan−1 (I (u) / R(u))

Fourier: F (u ) = F (u ) exp( i )
 Discrete Fourier Transform
Fourier Transform

N −1 −2iux N −1
F (u ) =  f ( x) e F (0) =  f ( x) e = f
1 1 0
N
N N
x =0 x =0

Inverse Fourier Transform

N −1 2iux
f ( x) = 11  F (u ) e N
u =0

Complexity: O(N2) (106 1012)

FFT: O(N logN) (106 107)

Unitary Matrix and 1D Unitary Transform
Definition conjugate
transpose
A matrix A is called unitary if A-1=A*T

When the   matrix A is unitary, the defined

transform
transform y = Ax is called unitary transform

Example

1  1 1 −1 1 1 − 1
A= − 1 1 , A = 1 1  = A T

2   2  
For a real matrix A, it is unitary if A-1=AT
Inverse of Unitary Transform
For a unitary transform, its inverse is defined by

 −1  *T 
x=A y=A y
Inverse Transform

 x1   a11*   a*N 1   y1 
x    
 2=      y2 
       
   * *  
 N  a1N   a NN   N 
x y

 N  
x =  yi bi , bi = [a1*i ,..., a*Ni ]T
i =1
basis vectors corresponding to inverse transform
1-D Unitary Transform
• Linear invertible transform
• 1-D sequence { x(0), x(1), …, x(N-1) } as a vector
• y = A x and A is invertible

• Unitary matrix ~ A-1 = A*T

• Denote A*T as AH ~ “Hermitian”
• x = A-1 y = A*T y =  ai*T y(i)
• Hermitian of row vectors of A form a set of orthonormal basis vectors
ai*T = [a*(i,0), …, a*(i,N-1)] T

• Orthogonal matrix ~ A-1 = AT

• Real-valued unitary matrix is also an orthogonal matrix
• Row vectors of real orthogonal matrix A form orthonormal basis vectors
Multiply with bases
Use def above
Shift x,y inside
Given orthonorm.
Basis Vectors and Basis Images
• A basis for a vector space ~ a set of vectors
• Linearly independent ~  ai vi = 0 if and only if all ai=0
• Uniquely represent every vector in the space by their linear combination
~  bi vi ( “spanning set” {vi} )
• Orthonormal basis
• Orthogonality ~ inner product <x, y> = y*T x= 0
• Normalized length ~ || x ||2 = <x, x> = x*T x= 1

• Inner product for 2-D arrays

• <F, G> = m n f(m,n) g*(m,n) = G1*T F1 (rewrite matrix into vector)
• !! Don’t do FG ~ may not even be a valid operation for MxN matrices!
• 2D Basis Matrices (Basis Images) 
• Represent any images of the same size as a linear combination of basis images
Properties of Unitary Transform
• Energy compaction: only few transform coefficients have large
magnitude
• Such property is related to the decorrelating role of unitary transform
• Energy conservation: unitary transform preserves the 2-norm of input
vectors
• Such property essentially comes from the fact that rotating coordinates does
not affect Euclidean distance
 Properties of 1-D Unitary Transform y = A x
• Energy Conservation
• || y ||2 = || x ||2

• || y ||2 = || Ax ||2= (Ax)*T (Ax)= x*T A*T A x = x*T x = || x ||2

• Rotation
• The angles between vectors are preserved

• A unitary transformation is a rotation of a vector in an

N-dimension space, i.e., a rotation of basis coordinates
Energy Compaction Example

1 1 1 1 100
Hadamard matrix    
1 1 − 1 1 − 1   98 
A= ,x =
2 1 1 − 1 − 1  98 
   
1 − 1 − 1 1  100
significant
1 1 1 1  100 198
1 − 1 1 − 1  98   0 
  1   =  
y = Ax =
2 1 1 − 1 − 1  98   0 
    
1 − 1 − 1 1  100  2 
insignificant
Energy Conservation*
 2  2
 
A is unitary
|| y || =|| x ||
y = Ax
Proof

 2 N
2  *T   *T 
|| y || =  | yi | = y y = ( Ax ) ( Ax )
i =1

 *T *T   *T  N  2
= x ( A A) x = x x =  | xi | = || x ||
2

i =1
Numerical Example
1  1 1  3
A= − 1 1, x = 4
2   

  1  1 1 3 1 7
y = Ax = − 1 1 4 = 1 
2   2 
Check:

 2 2  7 2
+ 12
|| x || = 3 + 4 2 = 25, || y ||2 = = 25
2
Properties of 1-D Unitary Transform y = A x
• Energy Conservation: || y || = || x ||
2 2

Proof: || y ||2 = || Ax ||2= (Ax)*T (Ax)= x*T A*T A x = x*T x = || x ||2

• Interpretation:
• The angles between vectors are preserved

• A unitary transformation is a rotation of a vector in an

N-dimension space, i.e., a rotation of basis coordinates
 Properties of 1-D Unitary Transform
(cont’d)
• Energy Compaction and Decorrelation
• Many commonly used unitary transforms tend to pack a large fraction of signal
energy into just a few transform coefficients

• Highly correlated input elements ➔ quite uncorrelated output coefficients

• Covariance matrix E[ ( y – E(y) ) ( y – E(y) )*T ]
small correlation implies small off-diagonal terms

Example: recall the effect of DFT

Question: What unitary transform gives the best compaction and decorrelation?

=> Will revisit this issue

 1-D DFT and Vector Form
 1 N −1


Z ( k ) = 
N n =0
z ( n )  W nk
N

• { z(n) }  { Z(k) }  N −1
 z ( n) = 1
n, k = 0, 1, …, N-1



N k =0
Z ( k )  W − nk
N
WN = exp{ - j2 / N }
~ complex conjugate of primitive Nth root of unity
• Vector form and interpretation for inverse transform
z = k Z(k) ak
ak = [ 1 WN-k WN-2k … WN-(N-1)k ]T /  N
• Basis vectors ➔
• akH = ak* T = [ 1 WNk WN2k … WN(N-1)k ] /  N

• Use akH as row vectors to construct a matrix F

• Z = F z  z = F*T Z = F* Z

• F is symmetric (FT=F) and unitary (F-1 = FH where FH = F*T )

  1 N −1
Matrix/Vector Form of 1-D DFT Z (k ) =


N n =0
z ( n)  WNnk
 N −1
 z ( n) = 1
• { z(n) }  { Z(k) } 


N k =0
Z ( k )  W − nk
N

n, k = 0, 1, …, N-1, WN = exp{ - j2 / N }

~ complex conjugate of primitive Nth root of unity

 Z (0)  1 1 1  1   z (0)   *T 
 Z (1)  1 e − j 2 / N e − j 2  2 / N
 e − j 2 ( N −1) / N    a0
 *T 
     z (1) 
− j 2  2 / N
e − j 2  4 / N  e − j 2 2( N −1) / N    z (2) a 
 Z (2)  = 1 e  =  1 z
    
    
        *T 
 Z ( N − 1) 1 e − j 2 ( N −1) / N e − j 2 2( N −1) / N  e − j 2 ( N −1) / N   z ( N − 1)  N −1 
2 a


 z (0)  1 1 1  1   Z (0) 
 1 e j 2 / N  Z (0) 
 z (1) e j 2  2 / N  e j 2 ( N −1) / N   Z (1) 
 Z (1) 
    
 z (2)  = 1 e
j 2  2 / N
e j 2  4 / N
 e j 2 2 ( N −1) / N
   Z (2)  = a 0 a1  a N −1   
       
           
 Z ( N − 1) 
 z ( N − 1) 1 e j 2 ( N −1) / N e j 2 2( N −1) / N  e j 2 ( N −1) / N   Z ( N − 1)
2


Matrix/Vector Form of 1-D DFT (cont’d)
 1 N −1
• { z(n) }  { Z(k) } 

Z ( k ) = 
N n =0
z ( n )  W nk
N

n, k = 0, 1, …, N-1, WN = exp{ - j2 / N }  N −1

 z ( n) = 1
“transform kernels” are complex exponentials 


N k =0
Z ( k )  WN− nk
• Vector form and interpretation for inverse transform
z = k Z(k) ak
where ak = [ 1, WN-k , WN-2k , … WN-(N-1)k ]T /  N
• Basis vectors ➔
• akH = ak* T = [ 1, WNk , WN2k , … WN(N-1)k ] /  N

Use akH as row vectors to construct a matrix F

Z = F z  z = F*T Z = F* Z

• F is symmetric (FT=F) and unitary (F-1 = FH where FH = F*T ) ➔

1-D Unitary Transform
Consider linear invertible transform
• 1-D sequence { x(0), x(1), …, x(N-1) } as a vector
• y = A x and A is invertible
• Unitary matrix: A is unitary if A-1 = A*T = AH
• Denote A*T as AH ~ “Hermitian”
• x = A-1 y = A*T y =  ai*T y(i)
• Hermitian of row vectors of unitary matrix A form
a set of orthonormal basis vectors {ai*T }
Think: how about column vectors of A?
• Orthogonal matrix ~ A-1 = AT
• Real-valued unitary matrix is also an orthogonal matrix
• Row vectors of real orthogonal matrix A form orthonormal basis vectors
Exercise:
• Is each matrix here unitary or orthogonal?
• If yes, what are the basis vectors?

 2 j 1 1 j 
A1 =  A2 =  
 2  j 1
− j 2

 1 1 
 2 3  2   cos sin  
A3 =  A4 =  2 A5 = 
 1  cos 
− sin 
1
 1 2   −
 2 2 

1: n; 2: y. inv(A3) = [2, –3; -1, 2]; Check A A’ = I ?

 Review: 1-D Discrete Cosine Transform
(DCT)
 N −1
  ( 2n + 1) k 


Z ( k ) = 
n =0
z ( n )   ( k ) cos  2N 

 z ( n) =
N −1
  ( 2n + 1) k 
  Z ( k )   (k ) cos
 
 k =0 2N
1 2
 (0) = , (k ) =
N N

• Transform matrix C
• c(k,n) = (0) for k=0
• c(k,n) = (k) cos[(2n+1)/2N] for k>0
• C is real and orthogonal
• Rows of C form orthonormal basis
• C is not symmetric!
• DCT is not the real part of unitary DFT!
• related to DFT of a symmetrically extended signal
Periodicity Implied by DFT and DCT
 From Ken Lam’s DCT talk 2001 (HK Polytech)

Example of 1-D DCT

100 100

50 50
z(n) Z(k)
0 0
DCT
-50 -50

-100 -100
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
n k
 From Ken Lam’s DCT talk 2001 (HK Polytech)

Example of 1-D DCT (cont’d)

1.0 1.0 100 100

UMCP ENEE631 Slides (created by M.Wu © 2001) 0.0 0.0 0 0

-1.0 -1.0 -100 u=0 -100 u=0 to 4

z(n)
1.0 1.0 100 100

n 0.0 0.0 0 0

Original signal -1.0 -1.0 -100 u=0 to 1 -100 u=0 to 5

1.0 1.0 100 100

0.0 0.0 0 0

-1.0 -1.0 -100 u=0 to 2 -100 u=0 to 6

Z(k)
1.0 1.0 100 100

k 0.0 0.0 0 0

-1.0 -1.0 -100 u=0 to 3 -100 u=0 to 7

Transform coeff.
Basis vectors Reconstructions
2D Transform=Two Sequential 1D Transforms
Y = AXA T

column transform Y1 = AX (left matrix multiplication first)

row transform Y = Y1A T = ( AY1T )T

row transform
Y2 = XAT = (AXT )T(right matrix multiplication first)
column transform
Y = AY2
Conclusion:
• 2D separable transform can be decomposed into two sequential
• The ordering of 1D transforms does not matter
Energy Compaction Property of
2D Unitary Transform
• Example 1 1 1 1 
 
1 1 − 1 1 − 1
A=
2 1 1 − 1 − 1
 
1 − 1 − 1 1 
100 100 98 99  391.5 0 5.5 1 
100 100 94 94  Y = AXA
T
 2.5 − 2 − 4.5 2 
X=  Y= 
 98 97 96 100  1 − 0.5 2 − 0.5
   
100 99 97 94   2 1.5 0 − 1.5 
A coefficient is called significant if its magnitude
is above a pre-selected threshold th insignificant coefficients (th=64)
Energy Conservation Property of
2D Unitary Transform
N N
X =  | xij |2
2
2-norm of a matrix X
i =1 j =1

Y = X
2 2
Y = AXA T A unitary

Example: 1 / 2 1/ 2 
A= 
1 / 2 −1/ 2 

1 2 Y = AXA T
 5 − 1
X=  Y= 
3 4   − 2 0 
X = 12 + 2 2 + 32 + 4 2 = 30 = 52 + 2 2 + 12 + 0 2 =|| Y ||2
2

You are asked to prove such property in your homework

Important 2D Unitary Transforms
• Discrete Fourier Transform
• Widely used in non-coding applications (frequency-domain approaches)
• Discrete Cosine Transform
• Used in JPEG standard
• Hadamard Transform
• All entries are 1
• N=2: Haar Transform (simplest wavelet transform for multi-resolution analysis)
Image Transforms
• Image transforms usually refers to a class of unitary matrices used for
representing images.
• Just as a one-dimensional signal can be represented by an
orthogonal series of basis functions, an image can also be expanded
in terms of a
Discrete set of basis arrays called basis images. These basis images
can be generated by unitary matrices.