0% found this document useful (0 votes)
150 views3 pages

Linear Algebra Inner Product

This document discusses several topics in linear algebra: 1. It defines inner products on vector spaces and gives examples, including the standard inner products on Rn and Cn. 2. It proves the Cauchy-Schwarz inequality that relates the inner product of two vectors to their lengths. 3. It describes the Gram-Schmidt process for orthogonalizing a set of linearly independent vectors to obtain an orthogonal basis for the span of the vectors. 4. It discusses best approximations of vectors by vectors in a subspace and characterizes them as vectors whose difference is orthogonal to all vectors in the subspace.

Uploaded by

kamlesh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
150 views3 pages

Linear Algebra Inner Product

This document discusses several topics in linear algebra: 1. It defines inner products on vector spaces and gives examples, including the standard inner products on Rn and Cn. 2. It proves the Cauchy-Schwarz inequality that relates the inner product of two vectors to their lengths. 3. It describes the Gram-Schmidt process for orthogonalizing a set of linearly independent vectors to obtain an orthogonal basis for the span of the vectors. 4. It discusses best approximations of vectors by vectors in a subspace and characterizes them as vectors whose difference is orthogonal to all vectors in the subspace.

Uploaded by

kamlesh
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

21

MTL101:::Lecture 17 and 18
(Inner product, Cauchy-Schwarz Ineq, Gram-Schmidt orthogonalization,best approximation)
(54) Let V be a vector space over F (where F = R or C). A map V ×V → F denoted by (u, v) 7→ hu|vi
is called an inner product on V if the following properties hold:
(a) hu|ui ∈ R and ≥ 0 for each u ∈ V ;
(b) hu|ui = 0 if and only if u = 0;
(c) hau + bv|wi = ahu|wi + bhv|wi;
(d) hu|vi = hv|ui (the complex conjugate).
A vector space together with an inner product is called an inner product space. Observe
that hu|av + bwi = hav + bw|ui = ahv|ui + bhw|ui = āhv|ui + b̄hw|ui = āhu|vi + b̄hu|wi.
Example: 1. Standard inner product on Rn and Cn : The dot product on R3 is an
inner products. Recall
(a1 , a2 , a3 ) · (b1 , b2 , b3 ) = a1 b1 + a2 b2 + a3 b3 .
n
P
In fact for every n ∈ N, the map h(x1 , x2 , . . . , xn )|(y1 , y2 , . . . , yn )i = xi yi is an inner product
i=1
n
on Rn . On Cn , h(x1 , x2 , . . . , xn )|(y1 , y2 , . . . , yn )i =
P
xi yi is an inner product. We leave the
i=1
verification to the students.
Example: 2. Suppose CC [0, 1] is the collection of complex valued continuous mapsR 1 on [0, 1]
which is a complex vector space (i.e. a vector space over C). The map hf |gi := 0 f (s)g(s)ds
is an inner product on C[0, 1]. Guess, how to get an inner product on the vector space CR [0, 1]
of real valued continuous functions on [0, 1].
Lemma: Suppose a > 0. Then at2 + bt + c > 0 for every t ∈ R if and only if b2 − 4ac < 0.
Hint: The quadratic expression aX 2 + bX + c has a real zero if and only if the discriminant
b2 − 4ac ≥ 0. Therefore this expression has no real zero if and only b2 − 4ac < 0. Since
s 7→ as2 + bs + c is a continuous map on R, it does not change sign on R if aX 2 + bX + c has
no real zero. Since a > 0, as2 + bs + c > 0 if aX 2 + bX + c has no real zero. 
Excercise: Suppose a > 0. Show that at2 +bt+c ≥ 0 for every t ∈ R if and only if b2 −4ac ≤ 0.
2
3. What are all inner products on  R :  
 a b x2
Define h(x1 , y1 )|(x2 , y2 )i := x1 y1 = ax1 x2 + bx1 y2 + cx2 y1 + dy1 y2 (observe
c d y2
that term containing x1 y1 , x2 y2 , x2i or yi2 is not acceptable!).
Assertion:h(x1 , y1 )|(x2 , y2 )i = ax1 x2 + bx1 y2 + cx2 y1 + dy1 y2 defines an inner product on R2 if
and only if a > 0, b = c and ad − bc > 0.
To prove this if and only if statement, first assume that the expression defines an inner product.
Then h(1, 0)|(1, 0)i > 0 ⇒ a > 0 (by the first property). By symmetry (the last property) we
get b = c (to get this explicitly write down the property.). For any (x, y) ∈ R2 \ {(0, 0)}, we
have ax2 +2bxy +dy 2 = h(x, y)|(x, y)i > 0. Hence we must have (2by)2 −4a·(dy 2 ) < 0 (∀ y 6= 0)
i.e., ad − b2 > 0. We leave the “if” part for the students as an exercise. 
Remark: All the inner products on Rn can be described. But we skip this part since it is
beyond the scope of this course.
Example: 3. Suppose Pn be the real Rvector space of polynomials in X with real coefficients
b
of degree < n. Then the map hf |gi := a f (s)g(s)ds (for f, g ∈ Pn ) is an inner product on Pn
for any a < b ∈ R.
A map V → F denoted by v 7→ ||v|| is called a norm on V if the following conditions are
satisfied.
(a) ||v|| ≥ 0;
(b) ||v|| = 0 if and only if v = 0;
(c) ||av|| = |a|||v|| for any a ∈ F;
(d) ||u + v|| ≤ ||u|| + |v||.
A vector space with a norm is called a normed space.
22

Theorem: (Cauchy-Schwarz inequality) Suppose V is an inner product space. Then for any
u, v ∈ V we have
|hu|vi| ≤ ||u||||v||.

Proof: Suppose F = R. Then 0 ≤ hu + tv|u + tvi = ||u||2 + 2thu|vi + t2 ||v||2 . Using the Lemma
above we have (2hu|vi)2 − 4||u||2 ||v||2 ≤ 0 or |hu|vi| ≤ ||u||||v||.
If F = C, the let α = hu|vi. 0 ≤ hu + tαv|u + tαvi = ||u||2 + t(ᾱhu|vi + αhu|vi) + t2 |α|2 ||v||2 =
||u||2 +2t(|hu|vi|2 )+t2 |hu|vi|2 |2 ||v||2 . Hence, by Lemma above (2|hu|vi|2 )2 −4||u||2 ||v||2 |hu|vi|2 ≤
0 which implies |hu|vi| ≤ ||u||||v||.
Every inner p product space is a normed space. Suppose V is an inner product space, then the
map ||v|| = hv|vi is a norm. Verification of the first three properties is straight forward. We
verify the last property ||u + v|| ≤ ||u|| + ||v||.
||u+v||2 = ||u||2 +(hu|vi+hv|ui)+||v||2 = ||u||2 +2<(hu|vi)+||v||2 ≤ ||u||2 +2|hu|vi|+||v||2 (since
the real part of z= <(z) ≤ |z|). Use Cauchy-Schwarz inequality and get ||u+v||2 ≤ (||u||+||v||)2 .
Remark: If we know the concept of “metric space” we may observe the following: A norm on
a vector spaces defines the following metric: d(u, v) = ||u − v||. Observe that the properties of
a metric follows from the properties of a norm.
(55) A set {u1 , u2 , . . . , un } of an inner product space is called an orthogonal set of vectors if
hui |uj i = 0 if i 6= j and hui |ui i =6 0 for each i and each j. It is not difficult to show that a
orthogonal set of vectors is linearly independent (exercise for the students). Although not every
linearly independet set of vectors is orthogonal, we have a very interesting and usual theorem:
Theorem: (Gram-Schmidt Orthogonalization) Suppose {u1 , u2 , . . . , un } is a linearly inde-
pendent set of vectors in an inner product space. Then there is an orthogonal set {v1 , v2 , . . . , vn }
such that for each i,
span{u1 , u2 , . . . , ui } = span{v1 , v2 , . . . , vi }.
Indeed, v1 = u1 and for 2 ≤ i ≤ n,
i−1
X hui , vj i
vi = ui − vj .
j=1
||vj ||2

Remark: We expect students to verify for each i, span{u1 , u2 , . . . , ui } = span{v1 , v2 , . . . , vi }


and observe that hvi , vj i = 0 for i 6= j. Note that the description of vi uses each vj for j < i. vi is
constucted so that vi is orthogonal to the space span{v1 , v2 , . . . , vi−1 } and span{u1 , u2 , . . . , ui } =
span{v1 , v2 , . . . , vi }. Since by induction,
span{u1 , u2 , . . . , ui−1 } = span{v1 , v2 , . . . , vi−1 }
i−1
P
so we have span{u1 , u2 , . . . , ui } = span{v1 , v2 , . . . , vi−1 , ui }. Define vi = ui + xj vj for xj ∈ F.
k=1
Determine xj so that vi is orthogonal to each of vj (j < i). Take inner product of vi with vj to
hu ,v i
get 0 = hui |vj i + xj ||vj ||2 . Hence xj = − ||vij ||j2 for j < i.

Application: Every finite dimensional inner product space has an orthogonal basis.
Proof: Suppose {u1 , u2 , . . . , un } be a basis of the inner product space V (so that the dimension
is n). Apply Gram-Schmidt process on this set to get the orthogonal set {v1 , v2 , . . . , vn } which
also spans V .
Example: Find an orthogonal basis of R2 with the inner product given by h(x1 , y1 )|(x2 , y2 )i =
x1 x2 + 2x1 y2 + 2x2 y1 + 5y1 y2 . We know that {e1 , e2 } is a basis of R2 . Since he1 , e2 i = 2 6= 0, the
standard basis is not an orthogonal basis. To get an orthogonal basis we use Gram-Schmidt
process. v1 = e1 and v2 = e2 − he||e21|e||12i e1 = e2 − 2e1 . Thus {e1 , e2 − 2e1 } is an orthogonal basis.
23

(56) Suppose V is an inner product space and W is a proper subspace of V . Given v ∈ V , a vector
w0 ∈ W is said to be a best approximation of v by a vector belonging to W if for each
w ∈ W we have
||v − w0 || ≤ ||v − w||.
Theorem: A vector w0 ∈ W is a best approximation of v ∈ V by a vector belonging to W if
and only if v − w0 is orthogonal to every vector belong to W .
Proof: Let w ∈ W . Then ||v − w||2 = ||(v − w0 ) + (w − w0 )||2 = ||v − w0 ||2 + ||w0 − w||2 +
2<(hv − w0 |w0 − wi). Suppose v − w0 is orthogonal to every vector belong to W . Then
||v − w||2 = ||v − w0 ||2 + ||w0 − w||2 Hence ||v − w|| ≥ ||v − w0 ||.
Conversely suppose w0 is a best approximation of v by a vector belonging to W . Since
||v − w|| ≥ ||v − w0 ||, we have ||w0 − w||2 + 2<(hv − w0 |w0 − wi) ≥ 0. Since any vector u in W
can be expressed in the form w − w0 for some w ∈ W , we have ||u||2 + 2<(hv − w0 |ui) ≥ 0 for
every u ∈ W . Choose u = − hv−w 0 |w0 −wi
||w0 −w||2
(w0 − w). Then you would get

|hv − w0 |w0 − wi|2 hv − w0 |w0 − wihv − w0 |w0 − wi


− 2< ≥0
||w0 − w||2 ||w0 − w||2
2
or − |hv−w 0 |w0 −wi|
||w0 −w||2
≥ 0 which implies hv − w0 |w0 − wi = 0 so that v − w0 is orthogonal to w − w0
for every w ∈ W . 
Proposition: If w1 , w2 are both best approximations of v ∈ V by a vector in W then w1 = w2 .
In particular, best approximation of a vector by a vector in a subspace of an inner product
space is unique.
Proof: By the preceding theorem v − w1 and v − w2 are both orthogonal to every vector in
W . Hence hv − w1 |w1 − w2 i = 0 and hv − w2 |w1 − w2 i = 0. Hence hw1 − w2 |w1 − w2 i =
h(v − w2 ) − (v − w1 )|(w1 − w2 )i = 0 
Theorem: Let W be a subspace of V . Suppose {w1 , w2 , . . . , wn } be an orthogonal basis of W .
Then the best approximation of v ∈ V is given by
hv|wi i
w0 = wi .
||wi ||2
n
P
Proof: Let w0 = xi wi be the best approximation of v by a vector belonging to W . Then
i=1
by the preceding theorem, v −w0 is orthogonal to every vector in W . Therefore, hv −w0 |wj i = 0
n
P
for each 1 ≤ j ≤ n. So hv|wj i − xi hwi |wj i = 0. Since {w1 , w2 , . . . , wn } is an orthogonal basis
i=1
hv|wj i
we have hv|wj i − xj hwj |wj i = 0 so that xj = ||wj ||2
. 
Exercise: Suppose R3 is the inner product space with inner product defined by
{(x1 , y1 , z1 )|(x2 , y2 , z2 )} = x1 x2 + 2x1 y2 + 2x2 y1 + 5y1 y2 + z1 z2 .
Find the shorest distance of v = e1 + e2 + e3 from the subspace W = {(x, y, z) ∈ R3 : z = 0}.
Solution: The set {e1 , e2 } is a basis of W . Use Gram-Schmidt process to get an orthog-
onal basis of W . For instance, {e1 , e2 − 2e1 } is an orthogonal basis of W . Thus the best
approximation of (e1 + e2 + e3 ) is given by w0 = he1 +e||e21+e ||2
3 |e1 i
e1 + he1 +e||e22+e 3 |e2 −2e1 i
−2e1 ||2
(e2 − 2e1 ) =
3 1
e + 1 (e2 − 2e1 ) = e1 + e2 .
1 1
Alternatively, Observe that v − (e1 + e2 ) = e3 is orthogonal to W with respect to the given
inner product space, then by the other theorem for shortest distance we must have w0 = e1 +e2 .
Exercise: Which of the following are inner products on R2 ?
(a) h(x1 , y1 )|(x2 , y2 )i = x1 y1 − x1 y2 − x2 y1 + 2y1 y2 ,
(b) h(x1 , y1 )|(x2 , y2 )i = x1 y1 + 2x1 y2 + 2x2 y1 + y1 y2 ,
(c) h(x1 , y1 )|(x2 , y2 )i = 54x1 y1 + 74x1 y2 + 74x2 y1 + 101y1 y2 ,
(d) h(x1 , y1 )|(x2 , y2 )i = x1 y1 − x1 y2 + x2 y1 + 2y1 y2 .
:::END:::

You might also like