Gram-Schmidt Orthogonalization Procedure

For a set of M signals, {s0 (t), s1 (t), . . . , sM −1 (t)}, defined on some interval [a, b], find a set of K orthonormal basis
signals, {φ0 (t), φ1 (t), . . . , φK−1 (t)}, such that each sm (t) can be represented as a linear combination of the basis signals.
Also, find the appropriate weights {sm,k } so that
K−1
X
sm (t) = sm,k φk (t)
k=0
for all 0 ≤ m ≤ M − 1.
Note: K is the minimum number of signals required. If {sm (t)} are linearly independent then K = M . Otherwise
K < M.
Description of Algorithm
Let Em be the energy of sm (t):
2 Z b  2
Em = sm (t) = sm (t) dt
 
a

1. Find the first basis signal:

s0 (t) s (t)
φ0 (t) = = √0
s0 (t)
E0

Clearly,
p √
s0 (t) = E0 φ0 (t) E0 , for k = 0
s0,k =
0 , for 1 ≤ k ≤ K − 1
= s0,0 φ0 (t)
2. Find the second basis signal:
Calculate the projection of s1 (t) onto φ0 (t):
Z b
s1,0 = hs1 (t), φ0 (t)i = s1 (t)φ∗0 (t) dt
a
Calculate the error signal:
g1 (t) = s1 (t) − s1,0 φ0 (t)
Note: g1 (t) is orthogonal to φ0 (t).
The second basis signal is
g1 (t)
φ1 (t) =
g1 (t)

Clearly, 
 hs1 (t), φ0 (t)i ,
 for k = 0
s1 (t) = s1,0 φ0 (t) + g1 (t) s1,k = g1 (t) ,

for k = 1
= s1,0 φ0 (t) + s1,1 φ1 (t)

0 , for 2 ≤ k ≤ K − 1


3. Find the mth basis signal:

Calculate the projection of sm (t) onto φk (t) for k = 0, 1, . . . , m − 1:
Z b
sm,k = hsm (t), φk (t)i = sm (t)φ∗k (t) dt
a
Calculate the error signal:
m−1
X
gm (t) = sm (t) − sm,k φk (t)
k=0

Note: gm (t) is orthogonal to {φ0 (t), φ1 (t), . . . , φm−1 (t)}. i.e. hgm (t), φk (t)i = 0 for all 0 ≤ k ≤ m − 1.

SYSC 3503 1 Winter 2009/10

 The mth basis signal is
gm (t)
φm (t) =
gm (t)

Clearly,
m−1
X 
sm (t) = sm,k φk (t) + gm (t)  hsm (t), φk (t)i ,
 for 0 ≤ k ≤ m − 1
k=0 sm,k = gm (t) ,

for k = m
m 
0 , for m + 1 ≤ k ≤ K − 1
X 
= sm,k φk (t)
k=0
Note: If gm (t) = 0 for all t ∈ [a, b], then sm (t) can be expressed as a linear combination of
{s0 (t), s1 (t), . . . , sm−1 (t)} .
Therefore {sm (t) | 0 ≤ m ≤ M − 1} are not linearly independent. In this case no additional basis signal
is required to represent sm (t), so K ≤ M .
Example:
s0 (t) s1 (t) s2 (t)
6 6 6
A A A
- - -
T T t T T t T T t
2 2 2
−A −A −A
2 Z b  2
E0 = s0 (t) =

s0 (t) dt
  T
a E1 = A2 E 2 = A2 T
2
= A2 T
1. Find the first basis signal:
φ0 (t)
√1
6
T
s0 (t) s0 (t)
φ0 (t) = √ = √ -
E0 A T T T t
− √1T 2

s
s0 (t) = s0,0 φ0 (t) 1
2
p √ φ0 (t) =

√ T =1 (normal)
s0,0 = E0 = A T T
2. Find the second basis signal:
Calculate the projection of s1 (t) onto φ0 (t):
Z T
s1,0 = hs1 (t), φ0 (t)i = s1 (t)φ∗0 (t) dt
0
T
−A T A√
Z 2 1
= (−A) √ dt = √ =− T
0 T T 2 2

SYSC 3503 2 Winter 2009/10

 Calculate the error signal:
g1 (t)
g1 (t) = s1 (t) − s1,0 φ0 (t) A 6
  √  2
 −A − −A T √1 , 0 ≤ t < T
=  2√  T 2
-
 0 − −A T √1 , T ≤ t < T
2 T 2 T T t
2
 A
−2 , 0≤t< 2 T − A2
= A T
2 , 2 ≤t<T
s 2  2 r
A T A T A2 T A√
g1 (t) = − + = = T

2 2 2 2 4 2
The second basis signal is
φ1 (t)
√1
6
T
g1 (t)
φ1 (t) = -
g1 (t)

T T t
− √1T 2

s 2  2
−1 T 1 T
φ1 (t) = √ + √ =1 (normal)

T 2 T 2
Note: φ0 (t) and φ1 (t) are orthonormal:
Z T
hφ0 (t), φ1 (t)i = φ0 (t)φ∗1 (t) dt = 0
0

A√
s1,0 = − T
2
s1 (t) = s1,0 φ0 (t) + s1,1 φ1 (t) A√
s1,1 = T
2
3. Find the third basis signal:
Calculate the projection of s2 (t) onto φ0 (t):
Z T
s2,0 = hs2 (t), φ0 (t)i = s2 (t)φ∗0 (t) dt
0
T
T
−A T
Z Z
2 1 1 A T
= (−A) √ dt + (A) √ dt = √ +√ =0
0 T T
2
T T 2 T 2
Calculate the projection of s2 (t) onto φ1 (t):
Z T
s2,1 = hs2 (t), φ1 (t)i = s2 (t)φ∗1 (t) dt
0
T
T √
   
−1
Z Z
2 1 A T A T
= (−A) √ dt + (A) √ dt = √ +√ =A T
0 T T
2
T T 2 T 2
Calculate the error signal
√
g2 (t) = s2 (t) − s2,0 φ0 (t) − s2,1 φ1 (t) = s2 (t) − A T φ1 (t)
( √ −1
−A − A T √ , 0 ≤ t < T2
= √ T
1
A − A T √T , T2 ≤ t < T
0 , 0 ≤ t < T2

=
0 , T2 ≤ t < T
=0
Therefore
√
s2 (t) = s2,0 φ0 (t) + s2,1 φ1 (t) = A T φ1 (t)
so a third basis signal is not required to represent s2 (t).

SYSC 3503 3 Winter 2009/10

 Summary:
φ0 (t) φ1 (t)
√1 √1
6 6
T T

- -
T T t T T t
− √1T 2
− √1T 2

√  √ 
s0 (t) = A T φ0 (t) s0 = A T,0
√ √  √ √ 
s1 (t) = − A2 T φ0 (t) + A
2 T φ1 (t) s1 = − A2 T , A2 T
√  √ 
s2 (t) = A T φ1 (t) s2 = 0, A T
Signal Space Diagram:
φ1 (t)
6 √
s2 @ A T

s1 √
A
@ 2 T

s0
 -
√ √ φ0 (t)
@
− A2 T A T

?
Proof of the Gram-Schmidt Orthogonalization Procedure
Show that {φ0 (t), φ1 (t), . . . , φK−1 (t)} are orthonormal when determined by the Gram-Schmidt Procedure.
Assume that {φ0 (t), φ1 (t), . . . , φk−1 (t)} are orthonormal for some k < K − 1.
Then, for any l, 0 ≤ l ≤ k − 1, !
1 gk (t)
hφk (t), φl (t)i =
g (t) hgk (t), φl (t)i φk (t) =
g (t)
k k
   !
k−1 k−1
1
P P
=
g (t) s k (t) − s φ
k,i i (t) , φl (t) gk (t) = sk (t) − sk,i φi (t)
k i=0 i=0
 k−1

1
P
=
g (t) hs k (t), φ l (t)i − s k,i hφ i (t), φl (t)i
k i=0
  !
k−1
1
P
=
g (t) hs k (t), φ l (t)i − s δ
k,i l−i hφi (t), φl (t)i = δl−i
k i=0
!
1
g (t) [sk,l − sk,l ]
= hsk (t), φl (t)i = sk,l
k
=0
Therefore φk (t) is orthogonal with {φl (t)|0 ≤ l ≤ k − 1}. Since

(t)

gk (t) gk
φk (t) = =⇒ φk (t) = =1,

gk (t) gk (t)

{φ0 (t), φ1 (t), . . . , φk (t)} are orthonormal.

Since {φ0 (t)} forms a one-dimensional orthonormal basis, it follows by recursion that
{φ0 (t), φ1 (t), . . . , φK−1 (t)}
are orthonormal.

SYSC 3503 4 Winter 2009/10

 Also, since
m−1
X
gm (t) = sm (t) − sm,k φk (t)
k=0
the signals are given by
m−1
X
sm (t) = gm (t) + sm,k φk (t)
k=0
m−1
X
= gm (t) φm (t) + sm,k φk (t)

k=0
m
X
= sm,k φk (t) with sm,m = gm (t)

k=0

Note: When using the Gram-Schmidt Procedure it is sometimes helpful to use the following property:
2
2 m−1
X
gm (t) = sm (t) − sm,k φk (t)


k=0
Z b  m−1
2
X 
= sm (t) − sm,k φk (t) dt
 
a  k=0

Z b 2 Z b m−1
X Z b m−1
X
∗ ∗
= sm (t) dt − sm (t) sm,k φk (t) dt − s∗m (t) sm,k φk (t) dt
 
a a k=0 a k=0
 2
b m−1
Z 
X
+ sm,k φk (t) dt


a  k=0

2 m−1
X m−1
X
∗
= sm (t) − sm,k hsm (t), φk (t)i − sm,k hs∗m (t)φ∗k (t)i

k=0 k=0
m−1
X m−1
X Z b
+ sm,k s∗m,l φk (t)φ∗l (t) dt
k=0 l=0 a

2 m−1
X m−1
X m−1
X m−1
X
= sm (t) − s∗m,k sm,k − sm,k s∗m,k + sm,k s∗m,l δl−k

k=0 k=0 k=0 l=0
2 m−1
X  2 m−1
X  2
= sm (t) − 2 sm,k  +
 
sm,k 
k=0 k=0
m−1
X 2
= Em −
 
sm,k 
k=0

SYSC 3503 5 Winter 2009/10