6

CHAPTER

Inner Product Spaces

When we introduced vectors in chapter 4, we briefly discussed the notion of an

inner product. In this chapter we will investigate this notion in more detail. We
begin with a formal definition.
Let V be a vector space. To each pair of vectors u, v ∈ V there is a number
that we denote (u, v) that is called the inner product, if it satisfies the following:

1. Linearity. For a real vector space, the inner product is a real number and
the inner product satisfies

(au + bv, w) = a (u, w) + b (v, w)

If the vector space is complex, the inner product is a complex num-

ber. We will define it in the following way. It is antilinear in the first
argument

(au + bv, w) = a ∗ (u, w) + b∗ (v, w)

120
Copyright © 2006 by The McGraw-Hill Companies, Inc. Click here for terms of use.
CHAPTER 6 Inner Product Spaces 121

but is linear in the second argument

(u, av + bw) = a (u, v) + b (u, w)

2. Symmetry. For a real vector space, the inner product is symmetric

(u, v) = (v, u)

If the vector space is complex, then the inner product is conjugate

symmetric

(u, v) = (v, u)∗

3. Positive Definiteness. This means that the inner product satisfies

(u, u) ≥ 0

with equality if and only if u = 0.

EXAMPLE 6-1
Suppose that V is a real vector space and that

(u, v) = −2
(u, w) = 5

Calculate (3v − 6w, u).

SOLUTION 6-1
First we use the linearity property. The vector space is real, and so we have

(3v − 6w, u) = 3 (v, u) − 6 (w, u)

We also know that a real vector space obeys the symmetry property. Therefore
we can rewrite this as

3 (v, u) − 6 (w, u) = 3 (u, v) − 6 (u, w)

Now, using the given information, we find

(3v − 6w, u) = 3 (u, v) − 6 (u, w) = (3) (−2) − (6) (5) = −6 − 30 = −36

 122 CHAPTER 6 Inner Product Spaces

The Vector Space Rn

We define a vector u in Rn as the n-tuple (u 1 , u 2 , . . . , u n ). The inner product
for the Euclidean space Rn is given by

(u, v) = u 1 v 1 + u 2 v 2 + · · · + u n v n

The norm of a vector is denoted by

u
and is calculated using
 

u
= (u, u) = u 21 + u 22 + · · · + u 2n

EXAMPLE 6-2
Let u = (−3, 4, 1) , and v = (2, 1, 1) be vectors in R3 . Find the norm of each
vector.
SOLUTION 6-2
Using the formula with n = 3, we have
  

u
= (u, u) = + + u 21 u 22 u 23 = (−3)2 + (4)2 + (1)2
√ √
= 9 + 16 + 1 = 27

and
   √ √

v
= (v,v) = v 12 + v 22 + v 32 = (2)2 + (1)2 + (1)2 = 4+1+1= 6

EXAMPLE 6-3
Suppose that u = (−1, 3, 2) , and v = (2, 0, 1) are vectors in R3 . Find the angle
between these two vectors.
SOLUTION 6-3
In Chapter 4 we learned that the angle between two vectors can be found from
the inner product using

(u, v)
cos θ =

u

v

The inner product is

(u, v) = (−1) (2) + (3) (0) + (2) (1) = −2 + 0 + 2 = 0

CHAPTER 6 Inner Product Spaces 123

Therefore we have

cos θ = 0

Which leads to
π
θ=
2

We have found that this pair of vectors is orthogonal. For ordinary vectors
in Euclidean space, the vectors are perpendicular, as the calculation of angle
shows.

Inner Products on Function Spaces

Looking at the formula for the inner product, one can see that we can generalize
this notion to a function by letting summations go to integrals. The vector space
C [a, b] is the space of all continuous functions on the closed interval a ≤ x ≤ b.
Supposing that f (x) and g(x) are two functions that belong to C [a, b], the inner
product is given by
$ b
( f, g) = f (x) g(x) dx
a

EXAMPLE 6-4
Let C [0, 1] be the function space of polynomials defined on the closed interval
0 ≤ x ≤ 1 and let

f (x) = −2x + 1, g (x) = 5x 2 − 2x

Find the norm of each function and then compute their inner product.

SOLUTION 6-4
First we compute the norm of f (x) = −2x + 1, which is shown in Fig. 6-1.
The norm is given by
$ 1 $ 1 $ 1
( f, f ) = f (x) dx =
2
(−2x + 1) dx = 2
(4x 2 − 4x + 1) dx
0 0 0
124 CHAPTER 6 Inner Product Spaces

0.5

0.2 0.4 0.6 0.8 1

−0.5

−1

Fig. 6-1. The function −2x + 1 which belongs to the vector space C [0, 1].

Integrating term by term, we find

4 3  4 1
( f, f ) = x − 2x 2 + x 10 = − 2 + 1 =
3 3 3

Now we consider g (x) = 5x 2 − 2x. The function is shown in Fig. 6-2.

The norm is given by

$ $ $
1 1
2 1
(g, g) = g (x) dx =
2
5x − 2x dx =
2
(25x 4 − 20x 3 + 4x 2 ) dx
0 0 0

1.5

0.5

0.2 0.4 0.6 0.8 1

Fig. 6-2. g(x) = 5x 2 − 2x is also continuous over the interval and so belongs to C [0, 1].
CHAPTER 6 Inner Product Spaces 125

Integrating term by term, we obtain

4  4
(g, g) = 5x 5 − 5x 4 + x 3 10 =
3 3
Finally, for the inner product we obtain
$ 1 $ 1

( f, g) = f (x) g (x) dx = (−2x + 1) 2x 2 − x dx
0 0
$ 1
= (−4x 3 + 4x 2 − x) dx
0

Integrating term by term, we obtain

4 1  4 1 1
( f, g) = −x 4 + x 3 − x 2 10 = −1 + − = −
3 2 3 2 6
EXAMPLE 6-4
Are the functions used in the previous example orthogonal?
SOLUTION 6-4
The functions are not orthogonal because ( f, g) = 0.
EXAMPLE 6-5
The functions cos θ and sin θ belong to C [0, 2π ]. What are their norms? Are
they orthonormal?
SOLUTION 6-5
A plot of cos θ over the given range is shown in Fig. 6-3.
The norm is found by calculating
$ 2π $ 2π
1 + cos 2θ θ 1 
cos θ dθ =
2
dθ = + sin 2θ 2π 0 =π
0 0 2 2 4
√
and so the norm is π . The sin function is shown in Fig. 6-4.
We have
$ 2π $ 2π
1 − cos 2θ θ 1 
sin θ dθ =
2
dθ = − sin 2θ 2π 0 =π
0 0 2 2 4
√
and so again, the norm is π .
Recall that we can normalize a vector (in this case a function) by dividing by
the norm. Therefore we see that the normalized functions for the vector space
126 CHAPTER 6 Inner Product Spaces

0.75

0.5

0.25

1 2 3 4 5 6
−0.25

−0.5

−0.75

−1

Fig. 6-3. The cos function in the interval defined for C [0, 2π].

C [0, 2π ], found by dividing each function by the norm would be

cos θ sin θ
f = √ , g= √
π π

If these functions are orthonormal, then

$ 2π
fg dθ = 0
0

0.75

0.5

0.25

1 2 3 4 5 6
− 0.25

− 0.5

− 0.75

−1

Fig. 6-4. The sin function over the interval defined by the vector space C [0, 2π ].
CHAPTER 6 Inner Product Spaces 127

In this case we have

$ 2π
1
cos θ sin θ dθ
π 0

Choosing u = sin θ, we have du = cos θ dθ and

$
1 2π
1 sin2 θ 2π
cos θ sin θ dθ = =0
π 0 π 2 0

Therefore, the functions f = √ θ,

cos
g= √θ
sin
are orthonormal on C [0, 2π ].
π π

Properties of the Norm

In Chapter 4 we stated the Cauchy–Schwarz and triangle inequalities. These
relations can be used to derive properties of the norm. If a vector space V is an
inner product space, then the norm satisfies

•
u
≥ 0 with
u
= 0 if and only if u = 0
•
αu
= |α|
u

•
u + v
≤
u
+
v

You may recall that the last property is the triangle inequality. This is an abstrac-
tion of the notion from ordinary geometry that the length of one side of a triangle
cannot be longer than the lengths of the other two sides summed together. Using
ordinary vectors, we can visualize this by using vector addition (see Fig. 6-5).

u+v

v
u

Fig. 6-5. An illustration of the triangle inequality using ordinary vectors.

 128 CHAPTER 6 Inner Product Spaces

An Inner Product for Matrix Spaces

The set of m × n matrices form a vector space which we denote Mm,n . Suppose
that A, B ∈ Mm,n are two m × n matrices. An inner product exists for this space
and is calculated in the following way:


(A, B) = tr B T A

EXAMPLE 6-5
Find the angle between two matrices, cos θ, where

−2 1 5 0
A= , B=
4 1 1 2

SOLUTION 6-5
The inner product is given by


(A, B) = tr B T A

First we compute the transpose of B

5 1
B = T
0 2

And so we have

5 1 −2 1 −6 6
B A=
T
=
0 2 4 1 8 2

We calculate the trace by summing the diagonal elements


(A, B) = tr B T A = −6 + 2 = −4

The transpose of A is

−2 4
A =
T
1 1
CHAPTER 6 Inner Product Spaces 129

And so we have

−2 4 −2 1 20 2
A A=
T
B=
1 1 4 1 2 2

Therefore we find that


(A, A) = tr A T A = 20 + 2 = 22

The norm of A is found by taking the square root of the inner product:
 √

A
= (A, A) = 22

For B we have

5 1 5 0 26 2
B B=
T
=
0 2 1 2 2 4

and so


(B, B) = tr B T B = 26 + 4 = 30

The norm of B is
 √

B
= (B, B) = 30

Putting these results together, we find

(A, B) −4
cos θ = =√ √

A

B
22 30

The Gram-Schmidt Procedure

An orthonormal basis can be produced from an arbitrary basis by application
of the Gram-Schmidt orthogonalization process. Let {v 1 , v 2 , . . . , v n } be a basis
for some inner product space V . The Gram-Schmidt process constructs an
130 CHAPTER 6 Inner Product Spaces

orthogonal basis w i as follows:

w 1 = v1
(w 1 , v 2 )
w 2 = v2 − w1
(w 1 , w 1 )
..
.
(w 1 , v n ) (w 2 , v n ) (w n−1 , v n )
w n = vn − w1 − wn − · · · − w n−1
(w 1 , w 1 ) (w 2 , w 2 ) (w n−1 , w n−1 )

To form an orthonormal set using this procedure, divide each vector by its norm.
EXAMPLE 6-6
Use the Gram-Schmidt process to construct an orthonormal basis set from
     
1 0 3
v 1 =  2  , v 2 =  1  , v 3 =  −7 
−1 −1 1

SOLUTION 6-6
We use a tilde character to denote the unnormalized vectors. The first basis
vector is

w̃ 1 = v 1

Now let’s normalize this vector

 
1
(v 1 , v 1 ) = (1 2 − 1)  2  = 1 × 1 + 2 × 2 + (−1) × (−1)
−1
=1+4+1=6
 
w̃ 1 1 1
⇒ w1 =  =√  2 
(v 1 , v 1 ) 6 −1

To find the second vector, first we compute

 
0
(w̃ 1 , v 2 ) = (1 2 − 1)  1  = [1∗ 0 + 2∗ 1 + (−1)∗ (−1)] = 3
−1
CHAPTER 6 Inner Product Spaces 131

The first vector is already normalized, so

    1
0 1 −2
(w̃ 1 , v 2 )   3  
w̃ 2 = v 2 − w̃ 1 = 1 − 2 = 0 
(w̃ 1 , w̃ 1 ) −1 6 −1 −1 2

Now we normalize
 
  − 12
1 1  0  = 1 +0+ 1 = 1
w̃ 2 , w̃ 2 = − 0 −
2 2 − 12 4 4 2

and so a second normalized vector is

  √1  
− 1 − 2
√ 2  
√
w 2 = (w̃ 2 ,w̃ 2 ) w̃ 2 = 2  0  =  0 
1

− 12 − √12

Finally, the third vector is found from

(w̃ 1 , v 3 ) (w̃ 2 , v 3 )
w̃ 3 = v 3 − w̃ 1 − w̃ 2
(w̃ 1 , w̃ 1 ) (w̃ 2 , w̃ 2 )

Now
 
  3
1 1  −7  = − 3 − 1 = − 4 = −2
(w̃ 2 , v 3 ) = − 0 −
2 2 1 2 2 2

and so
    1
3 1 −
  12   2  2
w̃ 3 = −7 + 2 +
1 0
6 −1
1 2 − 12
       
3 2 −2 3
=  −7  +  4  +  0  =  −3 
1 −2 −2 −3
 132 CHAPTER 6 Inner Product Spaces

Normalizing we find


3
(w̃ 3 , w̃ 3 ) = 3 −3 −3  −3  = 9 + 9 + 9 = 27
−3

and so the last normalized basis vector is

     
1 1 3 1 3 1 1
w3 =  w̃ 3 = √  −3  = √  −3  = √  −1 
(w̃ 3 , w̃ 3 ) 27 −3 3 3 −3 3 −1

Quiz
1. For the vector space C2 , the inner product is defined by

(u, v) = u ∗1 v 1 + u ∗2 v 2

Show that (u, v) = (v, u)∗ and that the inner product is antilinear in the
first argument, but linear in the second argument.
2. Consider the vector space C2 . Let u, v, w ∈ C2 and suppose that

(u, v) = 2i
(u, w) = 1 + 9i

1.5

1.25

0.75

0.5

0.25

0.2 0.4 0.6 0.8 1

Fig. 6-6. cos−1 (x).

CHAPTER 6 Inner Product Spaces 133

−1 −0.5 0.5 1

−2

−4

−6

Fig. 6-7. f (x) = 3x3 −2x2 + x − 1 shown on C [ −1, 1].

Find (v − 2w, u) and 2 (3iu, v) − (u, iw).

3. Find the inner product of the matrices

−1 1 2 3
A= , and B=
1 1 4 5

4. Consider the vector space of continuous functions C [0, 1]. The function
is cos−1 (x), which is shown in Fig. 6-6.
Is it possible to find the norm of cos−1 (x).
5. Find the norm of f (x) = 3x 3 − 2x 2 + x − 1 on C [−1, 1] (see Fig. 6-7).

−1 −0.5 0.5 1

Fig. 6-8. −x3 + 6x2 − x shown over the interval defined by C [−1, 1].
134 CHAPTER 6 Inner Product Spaces

0.5

0.5 1 1.5 2

−0.5

−1

Fig. 6-9. The functions f (x) = x2 − 2x and g(x) = −x + 1 are orthogonal on C [0, 2].

6. Is f (x) = 3x 3 − 2x 2 + x − 1 orthogonal to −x 3 + 6x 2 − x on
C [−1, 1] (see Fig. 6-8)?
7. Are the columns of
 
1 0 4
A = 2 2 −5
3 5 2

orthogonal?
8. Show that the functions f (x) = x 2 − 2x and g(x) = −x + 1 are orthog-
onal on C [0, 2] (see Fig. 6-9). Normalize these functions.