Inner product spaces

Examples. Verify that the properties of the inner product hold in each case.
1. 1 =
n
and x, y) := x

y =
n

i=1
x
i
y
i
.
2. We can dene an inner product on
n
using a given invertible matrix, A, by,
x, y) := x

Ay.
3. 1 =
mn
, the space of mn matrices. Given A, B
mn
,
A, B) := trace
_
A
T
B
_
(or trace (A

B) for (
mn
)
4. Complex valued functions: x(w), y(w) ((, ).
x, y) :=
_

x(w)y(w) dw.
Roy Smith: ECE 210a: 6.2
Inner product spaces
Inner product spaces
Dened for a pair of elements of a vector space, x, y A,
x, y)
A
: A A (or possibly ().
Dening properties:
1. x, x) , x, x) 0 and x, x) = 0 x = 0.
2. x, y) = x, y), for all scalars, .
3. x, y + z) = x, y) +x, z).
4. x, y) = y, x). ( y, x) denotes the complex conjugate).
If the vector space is clear we will drop the explicit subscript.
The pair, 1, and , )
1
are known as an inner product space.
A complete inner product space is called a Hilbert space.
Roy Smith: ECE 210a: 6.1
Inner product spaces
Compatible norm
If we have an inner product space, we can dene a compatible norm by,
|x| :=
_
x, x).
This is not the only possible norm, but compatibility with the inner product is required to
generalize
3
intuition about distances and angles.
Notice that this norm looks a lot like a Euclidean norm (or 2-norm).
In
n
it is the Euclidean norm.
Norm properties:
1. |x| 0 and |x| = 0 x = 0 comes from the inner product properties.
2. |x| = x, x)
1/2
= (x, x))
1/2
=
_
[[
2
x, x)
_
1/2
= [[|x|.
3. Triangle inequality. This one is trickier ...
Roy Smith: ECE 210a: 6.4
Inner product spaces
Key idea:
Inner products convey the idea of an angle between vectors.
We will see that we can dene such an angle by,
cos =
x, y)
|x||y|
.
This gives the idea of orthogonality; vectors can be perpendicular to one-another.
x y x, y) = 0 (i.e. = /2).
This will allow us to generalize our geometric intuition in
3
to higher dimensional spaces.
Roy Smith: ECE 210a: 6.3
Cauchy-Bunyakovskii-Schwarz inequality
Proof continued ...
From before,
y, y) y, x) 0.
This is equal to,
y, y) y, x) = y, y) y, x) = y, y)
x, y)
|x|
2
y, x) =
|y|
2
|x|
2
x, y)y, x)
|x|
2
0.
As the denominator is positive the numerator must also be positive and we get,
|y|
2
|x|
2
x, y)y, x).
As x, y) = y, x), the term, x, y)y, x) = [x, y)[
2
. So now,
|y|
2
|x|
2
[x, y)[
2
which, by taking square-roots, is [x, y)[ |y||x|.
Holder inequality: generalization for p-norms.
If we have real-numbers, p > 1 and q > 1 such that,
1
p
+
1
q
= 1 then [x

y[ |x|
p
|y|
q
.
Roy Smith: ECE 210a: 6.6
Cauchy-Bunyakovskii-Schwarz inequality
Cauchy-Bunyakovskii-Schwarz inequality
(the inequality previously known as Cauchy-Schwarz)
[x, y)[ |x||y|
Note that the norm in question is the one thats compatible with the inner product.
This clearly makes the angle between x and y well dened.
Proof:
Assume that x ,= 0 and choose as,
=
x, y)
|x|
2
which means that |x|
2
= x, y) or x, x y) = 0.
Now look at |x y|
2
(which is 0).
|x y|
2
= x y, x y) = x, x y)
. .
= 0
y, x y).
This means that,
y, x y) 0, or equivalently, y, y) y, x) 0.
Roy Smith: ECE 210a: 6.5
Parallelogram identity
Given a norm, can we always dene an inner product?
No, not always.
Parallelogram identity
We can dene an inner product if and only if the parallelogram identity is true.
|x + y|
2
+|x y|
2
= 2(|x|
2
+|y|
2
).
y
x
x+y
x-y
The inner product can be dened via,
x, y) :=
1
4
(|x + y|
2
|x y|
2
). (see Meyer for details)
Roy Smith: ECE 210a: 6.8
Compatible norms
Triangle inequality
The CBS inequality can be used to show that the triangle inequality holds for the norm
compatible with the inner product.
|x + y|
2
= x + y, x + y)
= x, x)+ x, y) +y, x)
. .
+y, y)
2[x, y)[
x, x) + 2[x, y)[ +y, y)
|x|
2
+ 2|x||y| +|y|
2
(|x| +|y|)
2
,
and taking square-roots gives the triangle inequality.
So every inner product space is also a normed space.
Roy Smith: ECE 210a: 6.7
Orthogonality
Orthogonality
Consider a real inner product space (x, y) )
x y x, y) = 0.
Pythagoras: Right angle triangles
x
x+y
y
x-y
=/2
|x + y|
2
= |x y|
2
= |x|
2
+|y|
2
So 0 = |x|
2
+|y|
2
|x y|
2
= x, x) +y, y) x y, x y)
= x, x) +y, y) x, x) y, y) +x, y) +y, x) which implies that x, y) = 0.
Roy Smith: ECE 210a: 6.10
Parallelogram identity
Inner products from other norms?
Can we form an inner product from ||
1
or ||

?
No. The parallelogram identity doesnt hold in either case.
So to retain the concept of an angle in
n
or (
n
we need to use the ||
2
(or something close
to it).
Elliptical norms
Suppose we are given a square, invertible matrix, A (
nn
.
Dene the norm:
|x|
A
= |Ax|
2
, where ||
2
is the usual Euclidean norm.
Can we dene an inner product here?
Does the obvious denition satisfy the parallelogram identity?
Roy Smith: ECE 210a: 6.9
Orthogonormal bases
Orthogonal bases
A basis,
B = x
1
, x
2
, . . . , is orthogonal if x
i
, x
j
) = 0 for all i ,= j.
The vectors, x
i
, are at right angles to one-another.
Orthonormal bases
A basis,
B = x
1
, x
2
, . . . , is orthonormal if x
i
, x
j
) =
_
1 i = j
0 i ,= j
The vectors, x
i
, are at right angles to one-another and |x
i
| = 1.
Wherever possible we choose orthonormal bases for our vector spaces.
This allows us to extend our intuition in
3
to higher dimensional spaces.
Roy Smith: ECE 210a: 6.12
Orthogonality
Angle between vectors
y
x
x-y

|x|
2
+|y|
2
= |x y|
2
+ 2 cos |x||y|.
Solving for cos gives the general angle formula,
cos =
|x|
2
+|y|
2
|x y|
2
2|x||y|
=
|x|
2
+|y|
2
x y, x y)
2|x||y|
=
|x|
2
+|y|
2
x, x) y, y) +x, y) +y, x)
2|x||y|
=
2x, y)
2|x||y|
=
x, y)
|x||y|
The CBS inequality guarantees that this is in the interval [-1,1] and so is uniquely dened.
Roy Smith: ECE 210a: 6.11
Fourier coecients
Fourier coecients and orthogonal bases
Suppose we have a basis,
B
1
= v
1
, v
2
, . . . , v
n
span(B
1
) = 1.
We can express any vector, v 1 with respect to this basis by,
v =
n

i=1

i
v
i
with
i
=
v
i
, v)
v
i
, v
i
)
. (If the basis is orthonormal,
i
= v
i
, v))
To see this,
v
j
, v) = v
j
,
n

i=1

i
v
i
)
=
n

i=1

i
v
j
, v
i
)
=
j
v
j
, v
j
) (all the other terms are zero)
So,
j
=
v
j
, v)
v
j
, v
j
)
.
Roy Smith: ECE 210a: 6.14
Orthogonal projection
Orthogonal projection
Given two vectors, x and y in an inner product space, the orthogonal projection of y on
spanx is the vector,
x where (x y) x.
y
x
y - x
x
(x y) x = x, x y) = 0 = x, x) x, y) = 0
= =
x, y)
x, x)
is called the Fourier coecient of y with respect to x.
Roy Smith: ECE 210a: 6.13
Fourier coecients
Coordinate matrix representations of linear transformations (see 2.16 and 4.10)
Say we have an operator, T : 1 J, with an orthogonal basis for each space,
B
1
= v
1
, v
2
, . . . , v
n
and B
J
= w
1
, w
2
, . . . , w
n

Recall that the matrix representation gives,

BW
=
BW
[T ]
BV
_
v

BV
=
_
[T (v
1
)]
BW
[T (v
2
)]
BW
. . . [T (v
n
)]
BW
_
v

BV
Examine the th column of the matrix. To express this in the B
J
basis we project it onto
each of the basis elements,
[T (v
j
)]
BW
=
_

_
w
1
, T (v
j
))
w
1
, w
1
)
.
.
.
w
m
, T (v
j
))
w
m
, w
m
)
_

_
so the th element is, T
ij
=
w
i
, T (v
j
))
w
i
, w
i
)
And so for an orthonormal basis: T =
_
_
w
1
, T (v
1
)) . . . w
1
, T (v
n
))
.
.
.
.
.
.
w
m
, T (v
1
)) . . . w
m
, T (v
n
))
_
_
.
Roy Smith: ECE 210a: 6.16
Fourier coecients
Fourier coecients and orthogonal bases
An
2
example with an orthogonal (but not orthonormal) basis.
v
v
2
v
1
, v
v
2
, v
v
1
, v
1
v
2
, v
2
v
1
v
1
v
2
v =
v, v
1
)
v
1
, v
1
)
v
1
+
v, v
2
)
v
2
, v
2
)
v
2
.
Roy Smith: ECE 210a: 6.15