6.

Inner Product Spaces

Learning Objectives

After completing this module, students should be able to :

1. Explain the definition of inner product spaces;
2. Find the norm of a vector and the angle between two vectors;
3. Explain the relation between null space, row space and column space;
4. Apply the Gram-Schmidt procedure on a basis of a finite-dimension
vector space;
5. Find the QR-decomposition of a matrix where its columns are linearly
independent;
6. Find the least square solution from the normal system of an SLE.
Scope

Review: Dot Product in Rn

Inner Product

Norm and distance

between 2 vectors
Angle between two vectors
and orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

Pre-test Module
Pre-test

• Answer the following questions:

1. Do you think that vectors are a mathematical element that can be drawn
as a line segment that has a direction?

2. Is matrix considered not to be a vector because you cannot determine

its magnitude and direction?
How is your pre-test?

Compare your answers.

If both of your answers are: YES
it is possible that you are still not familiar with these new concepts. It is
difficult to change the knowledge we know since ages ago. Try to review
again on general vector spaces, You will understand that a matrix can be
considered as a vector and it is not always the case that we can draw
vectors as line segments that has direction (arrow).

If both of your answers are: NO

CONGRATULATIONS, vector is an element of a vector space. Vector can be
any object such as matrices, continuous functions and even real
numbers; these kind of vector cannot be represented as a line segment
with direction.
 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

Review: Dot Product in Rn

Dot Product in Rn

• Is a.b = b.a?
• Write two different formulas of : a.b

• Can you give a sample to validate the formula a.b =

b.a ?

• Note:
A sample helps to bring an abstract concept to be more concrete.
A sample may help to think more generally, but it is not enough to
validate a general concept
Dot Product in Rn

• Is (ka).b =k( b.a)?

 Use some examples/samples to guess if it is true

or false
 Then, prove that the property is generally
applicable.
Dot Product in Rn

• Is a.(b + c) = a.b + a.c ?

 Use some examples/samples to guess if it is true

or false
 Then, prove that the property is generally
applicable.
Dot Product in Rn

• Find a.a
• In which cases does a.a = 0?

 Use some examples/samples to guess if it is true

or false
 Then, prove that the property is generally
applicable.
4 Properties of the Dot Product

• Symmetry
• Homogeneity
• Additive
• Positive
Norm of a Vector, Angle between 2
Vectors
• Find the norm of vector a in Rn

• Find the angle between a and b in Rn

 Euclidean Vector Space
V

a (a1, a2) (a1, a2, a3) (a1, a2, a3, a4) (a1, a2, …, an)
+ addition
….
Scalar
R multiplication
R R2 R3 R4 …. Rn
Inner Product Space
u.v <…,…>
Dot Product Inner Product

Length (norm) Angle Distance Length (norm) Angle Distance

Properties of the Dot Product
Rn
The dot product in Rn satisfies 4 properties, i.e.
<..,.>
1. u.v = v.u (symmetry)
+
2. (k u).v = k (u.v) (homogeneity)

3. u.(v + w) = u.v + u.w (additive) R

Euclid vector
4. v.v = 0, v.v = 0 if and only if v = 0 (positive) space
<…,…>
Dot product

Proyy x

length(norm) angle distance Next, we will define the inner product as a

generalization of the dot product.
 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

6.1 Inner Product

Inner Product

Definition 6.1.: Inner Product

An inner product of a vector space V over real numbers is a function that associates a real
number <u,v> with every pair of vectors u and v in V in such a way that the following axioms
are satisfied for all vectors u, v, and w in V and all scalars k :

1.<u, v> = <v, u> (symmetry axiom)

2.<u + v, w> = <u, w> + <v, w> (additive axiom)
3.<k u, v> = k <u, v> (homogeneity axiom)
4.<v, v> > 0 where <v, v> = 0 if and only if v = 0 (positivity axiom)

V symmetry
<..,..> additive

+ homogeneity
positive

R
A vector space over real numbers with an inner product is called an inner product space
Euclid Inner Product
Let u and v be any two vectors in R3, and it is defined that: R3
<u, v> = u.v (dot product) <a, b> = a.b Dot product

As it is previously explained, the 4 following properties are satisfied:

1. <u, v> = <v, u>
+
u, v  u 1 v 1  u 2 v 2  u 3 v 3
R
 v 1 u1  v 2u 2  v 3 u 3
Inner product space (on R)
 v .u
It is proved that <u, v> = <v, u>

2. <(u + v), w> = <u, w> + <v, w>

u  v , w  u1  v 1 , u 2  v 2 , u 3  v 3 .w 1 , w 2 , w 3 
 u 1  v 1 w 1  u 2  v 2 w 2  u 3  v 3 w 3
 u 1 w 1  v 1 w 1   u 2 w 2  v 2 w 2   u 3 w 3  v 3 w 3 
 u, w  v , w
It is proved that <(u + v), w> = <u, w> + <v, w>

Proving properties 3 and 4 are for your exercise/homework. R3 is an inner product space (Euclidean n-
space)
Euclidean Inner Product

The dot product in Rn satisfies the 4 properties: Rn

<a, b> = a.b Dot product
1. <u, v> = <v, u>
2. <(u + v), w> = <u, w> + <v, w> +
3. <ku, v> = k <u, v> R
4. <v, v> > 0 where <v, v> = 0 if and only if v = 0 Euclidean n-Space

The dot product is an inner product of Rn, and it is called as the Euclidean inner
product. Rn with a Euclidean inner product is called the Euclidean n-space.
Weighted Inner Product
If u and v are any two vectors in R3, it is defined
that: R3
u, v  3u1 v 1  2u 2 v 2  5u 3 v 3
u, v  3u1 v 1  2u 2 v 2  5u 3 v 3
Weighted dot product
+
Try to find whether these 4 axioms are satisfied or
R
not.
Inner product space (over R)
1. <u, v> = <v, u>
2. <(u + v), w> = <u, w> + <v, w>
3. <k u, v> = k <u, v>
4. <v, v> > 0 where <v, v> = 0 if and only if v = 0
 Proof : Positive-weighted Inner Product

u, v  3u1 v 1  2u 2 v 2  5u 3 v 3

1 . u, v  3 u 1 v 1  2 u 2 v 2  5 u 3 v 3
 3 v 1 u1  2 v 2u 2  5 v 3 u 3
 v, u

2 . u, v , w  u1  v 1 , u 2  v 2 , u 3  v 3 , w 1 , w 2 , w 3 

 3 u 1  v 1 w 1  2 u 2  v 2 w 2  5 u 3  v 3 w 3
 3 u 1 w 1  2 u 2 w 2  5 u 3 w 3   3 v 1 w 1  2 v 2 w 2  5 v 3 w 3 
 u, w ,  v , w 
3 . k u, v  k (u 1 , u 2 , u 3 ), (v 1 , v 2 , v 3 )
Conclusion: a positive-weighted inner product
 ku 1 , ku 2 , ku 3 , v 1 , v 2 , v 3 
 3 ku 1 v 1  2 ku 2 v 2  5 ku 3 v 3  function is an inner product in R3, it is called
 k 3 u 1 v 1  2 u 2 v 2  5 u 3 v 3  as a weighted Euclidean inner product
 k u, v
4. <v, v> > 0 where <v, v> = 0 if and only if v = 0
(try to prove it as your exercise)
Exercise 1: negative weights
1. Let u and v be any two vectors in R3, and it is defined that:
f u, v   3u1 v1  2u2v 2  5u3 v 3
Show that f is not an inner product.
Answer:
Let v = (1, 1, 1)
f (v, v) = 3.1.1.+ 2.1.1 -5.1.1 = 0 where v ≠ 0.

The positive axiom is NOT satisfied, so f is not an inner product. If there is a

negative weight, then that function is not an inner product.

2. Give an example of a function that maps f(R4xR4)  R which is also a weighted

inner product.

3. . Give an example of a function that maps f(R4xR4)  R which is NOT an inner

product.
Weighted Inner Products in Rn

Definition 6.2.: Weighted inner product in Rn

If u = (u1, u2, u3,…, un) and v = (v1, v2, v3,…, vn) are vectors in Rn, then the weighted
inner product with weights w1, w2, w3,…, wn where each weights is a real positive
number, is defined as:

u, v  w 1 u1 v 1  w 2u 2 v 2  ...  w nu nv n

Rn

u, v Weighted inner product

+
R
Weighted inner product space
Inner Product on Rn generated by
Matrices
Rn
A is a nxn matrix that is invertible. The inner
product is defined as <u, v> = Au.Av u, v  A u. A v

+
4 properties are satisfied, i.e.:
1. Au. Av = Av.Au R
2. A (u + v). Aw = Au.Aw + Av.Aw
3. A (k u.Av) = k (Au.Av)
4. Av.Av > 0 where Av.Av = 0 if and only if v = 0

Note:
Au and Av are vectors in Rn. Such that, Au. Av is a scalar. It is easy to show that
the 4 axioms are satisfied by this inner product.
That vector space Rn is said to be an inner product space Rn that is generated by
matrix A.
Exercise 2

1. For the inner product that is generated by matrix A, if A is the identity

matrix, what kind of inner product will you get?

2. If all weights (in the weighted Euclidean inner product) are 1, what kind
of inner product will you get?
Inner Product on P3 (1)
Any two vectors in P3 P3
2 3
p  a0  a1 x  a2 x  a3 x
p, q  a0 b0  a1 b1  a2 b2  a3 b3
2 3
q  b0  b1 x  b2 x  b3 x 2
p  a0  a1 x  a2 x  a3 x
3

2 3
q  b0  b1 x  b2 x  b3 x

The inner product is defined as:

p, q  a0 b0  a1 b1  a2 b2  a3 b3 R +
Inner product space

The following properties are satisfied:

1. <u, v> = <v, u>
2. <(u + v), w> = <u, w> + <v, w>
3. <k u, v> = k <u, v>
4. <v, v> > 0 where <v, v> = 0 if and only if v = 0

Conclusion:
Function <p, q> is an inner product on P3
Inner Product on P3 (2)
Any two vectors in P3 P3
2 3
p  a0  a1 x  a2 x  a3 x
p, q  a0 b0  a1 b1  a2 b2  a3 b3
2 3
q  b0  b1 x  b2 x  b3 x 2
p  a0  a1 x  a2 x  a3 x
3

2 3
q  b0  b1 x  b2 x  b3 x

The inner product space is defined as:

p, q  a0 b0  2 a1 b1  7 a2 b2  3 a3 b3
R +
Inner product space

The following four properties are satisfied:

1. <u, v> = <v, u>
2. <(u + v), w> = <u, w> + <v, w>
3. <k u, v> = k <u, v>
4. <v, v> > 0 where <v, v> = 0 if and only if v = 0

Conclusion:
The function <p, q> is an inner product on P3
Not an inner product in P3
Any two vectors in P3 P3
2 3
p  a0  a1 x  a2 x  a3 x
p, q  a0 b0  a1 b1  a2 b2  a3 b3
2 3
q  b0  b1 x  b2 x  b3 x 2
p  a0  a1 x  a2 x  a3 x
3

2 3
q  b0  b1 x  b2 x  b3 x

The inner product is defined as:

p, q  2 a0 b0  2 a1 b1  7 a2 b2  3 a3 b3
R +
Inner product space

Is <p, q> an inner product on P3?

[Hint: try for p  1  x  x 2  x 3 show that the 4th axiom is

not satisfied]
Inner product on P3 (3)
Any two vectors in P3 P3 1
2 3
p  a0  a1 x  a2 x  a3 x p, q   p( x )q ( x )dx
2 3
q  b0  b1 x  b2 x  b3 x
1
p  p x 
q  q x 

The inner product is defined as:

1
R +
p, q   p( x )q( x )dx Inner product space
1
The following four properties are satisfied:
1. <u, v> = <v, u>
2. <(u + v), w> = <u, w> + <v, w>
3. <k u, v> = k <u, v>
4. <v, v> > 0 where <v, v> = 0 if and only if v = 0

Conclusion:
<p, q> is an inner product on P3
The definite integral: inner product on
C[a,b]

C[a.b] is a vector space that consists of continuous functions on the interval

[a, b], b > a.

f and g are two vectors in C[a, b], where f = f(x), g = g(x)

b C[a,b]
f , g   f ( x ) g ( x )dx b

a
f, g   f ( x ) g ( x )dx
a
p  p x 
q  q x 

R +
Inner product space
Inner product in Mnxn

Mnxn is a vector space that consists of nxn matrices

If u = A, v = B are matrices in Mnxn, the inner product is defined as
<u, v>= trace(ATB).
The function above is an inner product because the 4 axioms are satisfied.

1.<u, v> = <v, u>

2.<u + v, w> = <u, w> + <v, w>
3.<k u, v> = k <u, v>
4.<v, v> > 0 where <v, v> = 0 if and only if v = 0

So, Mnxn with the inner product above is an inner product space.
 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

6.2 Norm and Distance

between 2 Vectors
Norm (length) and Distance

Definition 6.3: Length and distance in an inner product space

Given an inner product space V, the norm (length) of a vector v, is defined as
follows v  v, v

The distance between u and v is d u, v   u  v

Example 1:
(a) u = (2, 3, 1) in the Euclidean space R3
2 2 2
u  2  3  1  14
(b) u = (2, 3, 1) in R3 where it has a weighted inner product, with weights 2, 1, 1
consecutively.
2 2 2
u  2 .2  1 .3  1 .1  18
Distance between two vectors in the
Euclidean vector space

Distance between two vectors is the norm of the vector that is the difference
between the two vectors
y

(b1, b2)
a-b
b (a1, a2)
a
x

Distance between a and b is

2 2
a b  (b1  a1 )  (b2  a2 )
Example 2: norm of matrix

Mnxn is a vector space that consists of nxn matrices. If u = A, v = B are

matrices in Mnxn, the inner product is defined as <u, v>= trace(ATB)
Let there be a matrix: 1 2
 
A
2 1 

The norm of A is obtained as follows.

1 2
A u
2 0 
u, u  a11 a11  a12 a12  a21 a21  a22 a22
1
u  u, u 2
 a11 2  a12 2  a21 2  a22 2
2 2 2 2
 1 2 2 0
3
Example 3: norm of a polynomial

p = p(x) =x2 + 1 in P3 with the inner product defined as the definite integral
from -1 to 1. Then, the norm of p is

p   ( x  1) dx 
2 2
 1
5
x 
5 2
3
x x
3

1
1
 ....
1
Example 4: distance between two
polynomials
Based on the definition of inner product on P3:
2 3
p  a0  a1 x  a2 x  a3 x
2 3
q  b0  b1 x  b2 x  b3 x
p, q  a0 b0  a1 b1  a2 b2  a3 b3

Distance between two vectors:

3
p 2 x x
2
q  1  3x  x
2 3
p  q  1  2x 1x  x

pq  p  q, p  q
 1  4 1 1  7
Properties of the Norm

Theorem 6.1: Properties of the norm

If u and v are vectors in the inner product space V, if k is a scalar, then
(1) u  0

(2) u  0, if and only if u  0

(3) k u  k u

(4) u  v  u  v
 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

6.3 Angle between 2 vectors and

orthogonality
The angle between two vectors in the
Euclidean n-space
If u and v are nonzero vectors in R2, then:
u.v  u1 v1  u2 v 2  u v cos 

where α is the angle between u and v

u u. v
cos  
α u v
v
The angle between two vectors in the
general vector space

Definition 6.4.: Angle between 2 vectors in an inner product space V

If u and v are nonzero vectors in an inner product space, then:

u, v
cos  
u v

As it is in the Euclidean vector space, -1 ≤ cos α ≤ 1.

This is guaranteed by the following theorem:
Theorem 6.2.: Cauchy- Schwartz Inequality
If u and v are two vectors in an inner product space over real numbers, then
u, v  u v
Example 3: the cosinus of the angle
between two vectors
Let u = (0, 1, 0), v = (2, 1, 2), where the inner product is defined as follows:
<u, v> = u1v1+u2v2+u3v3

Determine the cosinus angle between u and v.

Answer:
u. v 0 .2  1 .1  2 .0 1
cos    
u v 1 .3 3
Example 4: cosinus angle between two
vectors
Let u = (0, 1, 0), v = (2, 1, 2) where the weighted inner product on Rn is
defined as follows:

<u, v> = 1u1v1 + 2u2v2 + 3u3v3

Determine the cosinus angle between u and v in the Euclidean vector

space R3

u. v 1 .0 .2  2 .1 .1  3 .2 .0 2
Answer: cos    
u v 1 .3 3
Example 5: cosinus of the angle
between two vectors
An inner product is defined in Mnxn: <u, v> = trace(ATB)

Let there be two vectors: 2 3 2 1

A B
1 1  3 
1

The cosinus angle between those two vectors is

u. v 2 .2  3 .1  1 .1  1 .3 11
cos    
u v  15 
2
15

Note:
u  2 .2  3 .3  1 .1  1 .1  15
v  2 .2  1 .1  1 .1  3 .3  15
Orthogonal Vectors

Definition 6.5.: orthogonal vectors

Two vectors a, b in an inner product space V are called orthogonal if and
only if <a, b> = 0

Example 6:
Let p = p(x) = x dan q =q(x) = x2 in P3 where <p.q> is defined as:
1

p, q   p  x q  x dx
1
1 1
 1
Such that, p , q   xx dx 
2 4
x 0
4 
 1
1

Hence, p and q are orthogonal

Pythagorean Theorem
C

AC2 = AB2 + BC2

A B
The Pythagorean Theorem also applies to inner product spaces.
Let u and v be orthogonal vectors, <u, v> = 0
2
u  v  u  v, u  v (norm definition)
 u  v, u  u  v, v (inner product property)
 u, u  v , u  u, v  v , v
(inner product property)
2 2 (<u, v> = 0 because they are orthogonal)
 u  2 u, v  v
2 2
 u  v

Theorem 6.3: Pythagorean Theorem

If u and v are orthogonal vectors in an inner product space, then:
2 2 2
uv  u  v
Example 7: Applying the Pythagorean
Theorem
Let there be two orthogonal vectors in R4 :

u   0 ,1 ,0 ,0 
v  1 ,0 ,1 ,2 

then,

 u  v, u  v  1,1,1,2 , 1,1,1,2   7
2
uv

2
u  u, u  0  1  0  0  1
2
v  v, v  1  0  1  4  6
2 2 2
It is shown that: u  v  u  v  7
Orthogonal Complement

Definition 6.6.:
Let W be a subspace of the inner product space V.
Vector v is orthogonal to W if v is orthogonal to every other vector in W.

The set of all vectors that are orthogonal to W is called the orthogonal
complement of W, and it is denoted by W ┴ (read: W perp)

L = W┴

W
Orthogonal Complement (cont)

Line L and the plane W are subspaces of R3. Every vector on L is

orthogonal to every vector on the xy-plane. If two subspaces are
orthogonal complement to each other (e.g. L is orthogonal
complement of W and vice versa), then L = W ┴

In geometry, the symbol ┴ means “orthogonal”.

W ┴ is read as “W perp”, from the word “perpendicular”.
The Basic Property of Orthogonal
Complement
If W is a subspace of the finite dimensional vector space V, then:
1. W ┴ is a subspace of V
2. The vector which is a member of both W and W ┴ is only the zero vector 0.
3. Orthogonal complement of W ┴ is W

To show that W ┴ is a subspace, we need to show that W ┴ is not empty, closed

under addition, and closed under scalar multiplication.

If a is a vector in W and W ┴, then <a, a> = 0, is satisfied if and only if a = 0

(positivity axiom of an inner product space)

0
Example 5
1. If V is an inner product space, determine the orthogonal complement of
{0}.
Answer: V.

2. If V is an inner product space, and W is a subspace of V.

If W = W ┴ . What can you conclude based on that info?
Answer: V = {0}

3. W is the y = -x line on the coordinate system of R2. Determine W ┴.

Answer: the y = x line.

4. If W is the yz-plane on the coordinate system of R3. Determine W ┴.

Answer: the x-axis
Null(A) and Row (A) are Orthogonal
Complement to each other
Proof:
1. Let there be any vector v that is orthogonal to every vector in Row(A), show
that Av = 0, where v is an element of Null(A)
2. If Av = 0, show that v is orthogonal to every vector in Row(A).

v is orthogonal to every vector in Row(A), then v is orthogonal to the row

space vectors r1, r2,…, rm
r1 .v  r2 .v  ...  rm v  0

 r1 .v   0 
 r .v  0 
LS Ax = 0
 2  
   0 
v is element of Null(A)
   
 m  0 
r . v
Null(A) and Row (A) are Orthogonal Complement
to each other
Proof:
1. Let there be any vector v that is orthogonal to every vector in Row(A), show
that Av = 0, where v is an element of Null(A)
2. If Av = 0, show that v is orthogonal to every vector in Row(A).

Av = 0, then r1 .v  r2 .v  ...  rm v  0

If k is any vector in Row(A), then k can be written as a linear combination of

the vectors in Row(A): k  c1r1  c2r2    c mrm

then k .v  c1 r1  c 2r2    c m rm .v

 c 1 r1 .v   c 2 r2 . v   ...  c m rm .v 
 c 1 .0  c 2 . 0  ...  c m .0  0
v is orthogonal to every vector in Row(A)
Null(A) and Row (A) are Orthogonal Complement to
each other

• Coll(AT) = Row(A)
• Based on the previous explanation, substituting A with AT, we obtain
that Coll(A) and (Null(AT)) are orthogonal complement to each other
Row(A), Coll(A), Null(A)

 a11 a12 a13 ... a1 .. n 

 
 a21 a22 a23 ... a2 .. n 
A   a31 a32 a33 ... a3 .. n 
 
 ... ... ... ... a4 .. n 
a an ..2 an ..3 ... am .. n 
 n ..1

Coll(A) Null(AT) Null(A)

Row(A)
Rm Rn

Coll(A) 0 Null(AT)
0 Row(A) 0 Null(A)
Relation between Row(A), Coll(A), Null(A)

Null(A) and Row(A) are orthogonal complements in Rn

Rn

Null(A) = [Row(A)]┴
Row(A) 0 Null(A)

Null(AT) and Coll(A) are orthogonal complements in Rm

Rm
Null(AT) = [Coll(A)]┴
Coll(A) 00 Null(AT)
 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

6.4 Gram-Schmidt Procedure

Orthogonal and Orthonormal Basis
Definition 6.7.a.: Orthogonal Set
A set S in the inner product space V is said to be orthogonal if and only if every pair
of its different vectors are orthogonal.

Definition 6.7.b: Orthogonal and Orthonormal Basis

V is a vector space, B is a basis of V. B is an orthogonal basis if B is an orthogonal
set. If B is orthogonal and the norm of every vector in B is 1, then B is an
orthonormal basis.
z

Example 8: k
(0, 0, 1)
1. { i, j, k } orthogonal basis in R3 j
i y
2 3
(0, 1, 0)
2. {1, x, x , x } Orthogonal basis in P3 (1, 0, 0)
x

Given a basis of a finite-dimensional vector space, we can obtain an orthonormal basis using the
Gram-Schmidt procedure.
Example 9: orthonormal basis in P3

Let there be an orthonormal basis of P3, i.e. B = {1, x, x2, x3} where its inner
product is:
p, q  a0 b0  a1 b1  a2 b2  a3 b3

• Proof:
B  v 1 , v 2 , v 3 , v 4  Every pair of its vector is orthogonal:
2 3
v1  1  0 x  0 x  0 x v 1 , v 2  1 .0  0 .1  0 .0  0 .0  0
v2  0  1x  0x  0x
2 3
v 1 , v 3  1 .0  0 .0  0 .1  0 .0  0
v3  0  0x  1x  0x
2 3
v1 , v 4  v 2, v 3  v 2, v 4  v 3 , v 4  0
2 3
v4  0  0x  0x  1x The norm of every vector = 1
v 1  v 1 , v 1  1 .1  0  0  0  1 v3  v4  1
v 2  v 2 , v 2  0  1 .1  0  0  1
Coordinate relative to an orthonormal
basis
 1 1 
u 1   0, 2, 2
 2 2 
1 1 1 
u2   6 , 6, 6
3 6 6 
1 1 1 
u3   3, 3 , 3
3 3 3 

a  1,1,1 

B = {u1, u2, u3}

write a as a coordinate relative to B
Coordinate relative to an orthonormal
basis
Theorem 6.4:
If B  u1 , u2 ,..., u n  is an orthonormal basis of vector space V, and a is an element
of V, then
a  a, u1 u1  a, u2 u 2  ...  a, un u n

Every vector in V can be expressed as a linear combination of the vectors in basis

where the coefficients are the inner product of that vector with the vectors in the
basis.

Example 10:
 1 1  B = {u1, u2, u3} is an orthonormal basis of V, a is a vector in V
u 1   0, 2, 2
 2 2 
then a can be expressed as a linear combination of the vectors
1 1 1 
u2  
3
6 ,
6
6,
6
6

in basis.
1 1
1
u3   3,
1
3 ,
1 
3
a 2u 1  6u 2  3u 3
3 3 3  3 3
1 1
a  1,1,1  Coordinate matrix a relative to B: a B   2 , 6,

3
 3 3 
Norm, Angle and Distance Formula
Theorem 6.5:
If B is an orthonormal basis of an inner product space, and if
u B  u1 , u 2 ,..., u n  and v B   v 1 , v 2 ,..., v n 
then
(a) u  u1 2  u 2 2  ...  u n 2

(b) d u, v   u1  v 1 2  u 2  v 2 2  ...  u n  v n 2

(c)  u , v  u1 v 1  u 2 v 2  ...  u n v n

Example 11: determine the norm of a = (1, 1, 1).

Option 1. using dot product
1
a  a.a 2 a  1 1 1  3
Option 2. using coordinate vector relative to the orthonormal basis
1 1
a B   2 , 6,

3 a  u1 2  u 2 2  u 3 2  2
6

3
 3
 3 3  9 9
Orthogonal Basis

Theorem 6.6.:
If B  u1 , u 2 ,..., u n  is an orthogonal basis of the inner product space V,
and a is an element of V, then
a, u 1 a, u 2 a, u n
a  2
u1  2
u 2  ...  2
un
u1 u2 un

Example 12:
  1 1 2 2 2 
B    0 ,1 ,1 ,  1 ,  , ,  , ,    a  1,1,1 
  2 2 3 3 3 
B is an orthogonal basis of V, a is a vector in V then a can be expressed as a linear
combination of vectors of basis.
2 1
2 1 1 12 2 2
a  1  0,1 ,1    1 ,  ,    , ,   a B   1 , , 
3 2 2 23 3 3  3 2
Projection Theorem
u u
w2 w2
W O
O w1 w1
W
If W is a line or plane that goes through the main axis, then every vector u in
the euclidean vector space can be expressed as: u = w1+w2
where w1 is along (inside of) W, and w2 is perpendicular to W

Theorem 6.7.: Projection Theorem

If W is a finite-dimensional subspace of an inner product space V, then every vector u
in V can be expressed in exactly one way as:
u = w1+w2
where w1 is in W, and w2 is in W ┴.
w1 is the orthogonal projection of u on W (projwu), w2 is the component of u orthogonal
to W.
Orthogonal Projection in a Subspace

u
w2
W
O w 1  proj w u

Theorem 6.8: Orthogonal Projection in a Subspace

Let W be a finite-dimensional subspace of an inner product space V.
(a) If {v1, v2, …, vr} is an orthonormal basis for W, and u is any vector in V, then
proj W u  u, v 1 v 1  u, v 2 v 2  ...  u, v r v r
(b) If {v1, v2, …, vr} is an orthogonal basis for W, and u is any vector in V, then
u, v 1 u, v 2 u, v r
proj W u  2
v1  2
v 2  ...  2
vr
v1 v2 vr
 Finding Orthogonal and Orthonormal
Basis
Orthonormalization proses using Gram-Schmidt:
B  {u1 , u 2 ,  , u n } is a basis for inner product space V

Jika v 1  u1
〈 u 2 , v 1〉 v 2  u 2 - proj W1 u 2
v 2  u 2  proj W1 u 2  u 2  2
v1
v1
u2
〈 u 3 , v 1〉 〈 u 3 , v 2〉
v 3  u 3  proj W2 u 3  u 3  2
v1  2
v2
v1 v2 W1

O v 1= u 1 proj W1 u 2
n 1
〈 v n , v i〉 w 1  span({ v 1 })
v n  u n  proj W n 1 u n  u n   2
vi
vi
i 1
w 2  span({ v 1 , v 2 })
 B'  { v 1 , v 2 ,  , v n } Is an orthogonal basis
v1 v2 vn Is an orthonormal basis
 B''  { , ,, }
v1 v2 vn
Example 13: Gram-Schmidt Process
Given a basis of R3 B = {(0, 1, 1), (1, 0, 1), (1, 1, 0)}. Change the basis into an
orthonormal basis using Gram-Schmidt process.

〈 u 3 , v 1〉 〈 u 3 , v 2〉
1. Take v1  u1 3 . v 3  u 3  proj W2 u 3  u 3  2
v1  2
v2
v1 v2
v 1  0,1,1 
1 1  1 1
〈 u 2 , v 1〉 v 3  1,1,0   0,1,1    1, , 
2 . v 2  u 2  proj W1 u 2  u 2  v1 2 3  2 2
v1
2 2
2 2 2
v 3   , , 
3 3 3
0 .1  1 .0  1 .1
v 2  1 ,0,1   2 2 2
0,1 ,1  4. Orthogonal basis is obtained : B   v 1 , v 2 , v 3 
0 1 1

 v v v  
1 5. Orthonorma l basis : B    1 , 2 , 3 
v 2  1 ,0,1   0,1 ,1 
2 
 v1 v 2 v 3 

 1 1  1 1  1 1 1 
v 2   1 , ,    0, 2, 2 ,  6 , 6, 6 ,
 2 2  3 6 6 
 2 2 B    
 1 3 , 1 3 , 1 3  

 3 3 3  

 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

6.5 QR-Decomposition
Gram-Schmidt Process on matrices with
linearly independent columns
• Let matrix A have linearly independent columns, such that it forms a
basis for Coll(A):

B  u1 , u2 ,..., u n 

• After applying Gram-Schmidt proses, orthonormal basis is obtained

B*  p 1 , p 2 ,..., p n 

• Express every column vector as a linear combination of orthonormal

basis
u 1  u 1 , p 1 p 1  u 1 , p 2 p 2  ...  u 1 , p n p n
u 2  u 2 , p 1 p 1  u 2 , p 2 p 2  ...  u 2 , p n p n

u n  u n , p 1 p 1  u n , p 2 p 2  ...  u n , p n p n
G-S (cont’d)

u 1  u 1 , p 1 p 1  u 1 , p 2 p 2  ...  u 1 , p n p n
u 2  u 2 , p 1 p 1  u 2 , p 2 p 2  ...  u 2 , p n p n

u n  u n , p 1 p 1  u n , p 2 p 2  ...  u n , p n p n

Can be written as:

 u1 , p 1 u2 ,p 1  un,p1 
 
u1 , p 2 u2 ,p 2  un,p 2 
A  u 1 u 2  u n   p1 p 2  p n 
     
 
 u 1 , p n u2 ,p n  u n , p n 

A = QR, R is an upper triangular matrix, because uj is orthogonal to pk for

k>j, the inner product is 0.
QR-Decomposition

Theorem 6.9:
If A is a matrix with linearly independent column vectors, then A can be
decomposed into a product of two matrices, i.e. A = QR; where Q is a matrix with
orthonormal columns and R is an invertible upper triangular matrix.

QR-Decomposition can be applied on numerical algorithms; e.g. to find an

eigenvalue of a large matrix.
 Example 14: QR-Decomposition

1 0 0  12   3
12
 0
1 1 0  GramSchmid t process & normalizat ion 1   1   2 
A            v 1   1 , v 2  
2 12 , v   6 ,
1 1 1 2  1
 3  16 
12
  1    1 
1 1 1 1
2  12   6

3
 12 12
0
1 1 2 

Q  1
2 12 6

2 1
12
1
6

1 1 1 
 2 12 6

Determine R (where its inner product is the dot product).

Find QQT and QTQ
3
1 0 0  12 12
0
1 1 2 
0 T
1

A
1
 Q  1
2 12 6 Q QI
1 1 1 2 1 1
 T
QQ  I
12 6
  1 1 
1 1 1 1
 2 12 6

R  IR  Q T QR  Q T QR   Q T A

2 3
2 1 
 
R  0 3
12
2
12 
0 0 2 
 6 
QR-Decomposition Procedure

• Let A have linearly independent columns.

• B = basis of Coll(A) which consists of columns of A

• Apply Gram-Schmidt process on B and normalize to obtain orthonormal

basis of Coll(A)

• Let Q be the matrix with orthonormal columns

• R is an invertible upper triangular matrix, i.e. R = QTA

 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

6.6 Orthogonal Matrix

Gram-Schmidt Process on Invertible
Square Matrices
• A is an invertible square matrix

• A set which consists of columns of A, i.e. {c1, c2, …,cn} forms a basis for
Coll(A)

• Coll(A) = Rn

• After applying the G-S process and normalize B, then we obtain the
normal standard basis for Rn
Orthogonal Matrix

Definisi 6.8:
A nxn matrix A is orthogonal if and only if AT = A-1

Which means
ATA= I A = [ c1 c2 … cn]
(I)ij = ciT cj = ci.cj
So ci.cj = 0 if i ≠ j
ci.cj= 1 if i = j, where ci.ci = ║ci║2.

Conclusion:
A set which consists of column vectors of A is orthonormal; a set which
consist of the rows of A is also orthonormal.
Orthogonal Matrix Property

If A is an orthogonal matrix, then

• Det(A) = ± 1
• A-1 is orthogonal
• If x a vector in Rn then ║x║=║Ax║
(vector x and its transformation has the same norm)

Proof:
• Det(A) = det(AT) = det(A-1) = 1/det(A). Then det(A) = 1 or -1

• (A-1 )-1 = A = (AT )T= (A-1) T

T T T T T
• ║Ax║= ( Ax ) Ax  x A Ax  x Ix  x x  x
Example 15: Orthogonal Matrices

0 1 0  cos  sin  
   sin 

1 0 0
   cos  
 0 0 1 

1 0 1
 1
2 2
3
 1  
  2 
0 2 0
 2
3
 2
1
  1 0 1 
Properties of Orthogonal Matrices

1. The inverse of an orthogonal matrix is also an orthogonal matrix.

2. A product of orthogonal matrices will result in an orthogonal matrix.
3. If A is orthogonal, then det(A) = 1 or det(A) = -1.
4. Orthogonal matrices have orthonormal columns (prove it!)

A  [ a 1 a 2 ... a n ]
 1, if i  j

[ I ]ij  A A
T
  ai
T
aj  
 0 , if i  j
ij
 Review: Dot Product in Rn

Inner Product

Norm and Distance between 2

vectors

Angles between 2 vectors and

Orthogonality

Gram-Schmidt Procedure

QR-Decomposition

Orthogonal Matrix Least Square Solution

6.7 Least Square Solution

Match a graph on a data
One of the most frequent problems we encounter is finding the relation between two
factors (e.g. two variables x and y) and express it into an equation y = f(x) that
corresponds to (x, y) which are raw data we obtain beforehand, e.g. (x1, y1), (x2, y2), (x3,
y3), …, (xn, yn)

Straight Line
y y
(2, 7)

(1, 2) (3, 2)
(0, 1) (3, 1)
(0, 0) x x
(2, 0)
y  ax  b 2
y  ax  bx  c
 Match a straight line on a data
y
Let there be points on a plane. Find a line equation (2, 7)
that is most suitable to describe the data.

(1, 2)

(0, 0) x
We obtain the following linear system
0 = a.0 + b 0 1 0 
  a  
2 = a. 1 + b where its matrix equation is: 1 1 b   2 
   
7 = a. 2 + b  2 1  7 

The linear system does not have a solution (inconsistent) because the points are
not on the same line. Question: how to find the most suitable line that describes
the data?
Brief Review

Two principles are needed in this process.

1. Consistency Theorem: LS Ax = b is consistent if

and only if b is in the column space of A.

2. Null(AT) =(Coll(A))┴
Least Square Solution

Least Square Solution:

Let Ax = b be a linear system that has m equations with n unknowns,

determine x (if possible) that minimizes ||Ax –b|| in the euclidean n-
space. Vector x is called the least square solution of Ax = b.

b e

Ax Coll(A)

Least square solution x produces the vector Ax on Coll(A) that is the

closest to b.
Finding the best approximation

b e

Ax W= Coll(A)

Finding the solution of an approximation is

finding x* such that Ax* = proyColl(A) b

Problem : A and b is known, find x such that Ax = proyColl(A) b. It is not

easy to find proyColl(A) b
Normal System
b e

Ax W= Coll(A)

Given A and b, finding the best approximation with the least error means finding x that satisfies:
Ax = proyColl(A)b
Ax = proyColl(A)b. (shown in the picture above)
b - Ax = b - proyColl(A)b
b - Ax orthogonal to W, and W┴ = Null(AT), the set of solutions of AT x = 0. b-Ax is inside of
the nullspace of AT.
Hence, AT(b-Ax) = 0 or ATb= ATAx which is known as the normal system of Ax = b.
Such that, finding the approximation of the solution of Ax = b, is the same as finding the
solution of the consistent linear system, i.e. the normal system.
Definition: Least Square Solution
If A is a mxn matrix, the least square solution of Ax = b is the solution x* from the normal
system ATAx* = ATb
Normal System (cont’d)
Given a linear system Ax = b with m equations with n unknowns each, then:
ATb= ATAx is the normal system of said linear system.
1. The normal system consist of n equations with n unknowns.
2. The normal system is guaranteed to be consistent and the solution of a normal
system is called as the least square solution.
3. If vector x is the least square solution, then: proyColl(A)b = Ax
4. The normal system can have infinitely many solutions

If the columns of A are linearly independent, then the least square solution is
exactly one.
If the columns of A is linearly independent, then ATA has an inverse, such that
the least square solution is as follows
ATAx = ATb (multiply with (ATA)-1 )
(ATA)- 1ATAx = (ATA)-1 ATb (associative)
x = (ATA)-1 ATb (least square solution)
Example 16: a straight line (1)
Given points (0, 0), (1, 2), and (2, 7). We will find the line equation that most represents these
data. We construct a linear system from the data we have.
A line equation has the general form of y = ax + b. We need to find a and b.
If the points are all on the line, then a and b satisfy the following linear system:
0 = a. 0 + b 0 1  0 
  a  
2 = a. 1 + b where the matrix equation is: 1 1   b    2 
7 = a. 2 + b  2 1     7 

Give notations to the matrices

0 1 0 
   a    y = (7/2)x –1/2
M  1 1  , v   , b  2 
 2 b 
1   7  y
0 1 0 
0 1 2  5 3 0 1 2 (2, 7)
T
M M

1     16 
1
T
1  3  M b 2    9 
1 1 3 1 1 1 
 2 1   7   
The normal system:
 5 3   a  16  (1, 2)
T
M Mv    b    9 
 3 3     (0, 0) x
Solution: a= 7/2, b = -1/2. Such that, y = 7/2x –1/2
A Straight Line
Let there be n data points ( x1 , y 1 ), ( x 2 , y 2 ), ( x 3 , y 3 ),..., ( x n , y n )
And we want to match a straight line y = ax + b that will represent the data above.
If the points are on the line, then a and b satisfy
y1  ax 1  b  x1 1  y1 
   
y2  ax 2  b x 1 y2
 2   
a  
y3  ax 3  b w ritten  x 3 1     y3 
  b   
       
yn  ax n  b  x n 1  y n 
The normal system M u y
T T
M M .u  M y
 x1 1  y1 
x 1 y 
xn     a  x1 xn   
2 2
 x1 x2 x3 ... x2 x3 ...
1  x3 1     y3 
 1 1 ... 1    b 1 1 1 ... 1   
 ... ...      ... 
 x n 1   y n 
Finding the solution to the normal system, we obtain the values for the line equation.
Example 17: parabola (1)
Given data points (0, 1), (2, 0), (3, 1), and (3, 2)

(3, 2)
(0, 1) (3, 1)
(2, 0) x

The general form of a parabola is: y = ax2 + bx + c

Substitute the variables in the equation based on the data points we have to
form a linear system, then find the normal system.
Solution: parabola (cont’d)
y1 = ax12 + bx1 + c 1 = a(0)2 + b(0) + c 0 0 1 1 
  a   
y2 = ax22 + bx2 + c 0 = a(2)2 + b(2) + c 4 2 1    0 
Written as:
9 b 
y3 = ax32 + bx3 + c 1 = a(3)2 + b(3) + c 3 1    1 
  c  
y4 = ax42 + bx4 + c 2 = a(3)2 + b(3) + c 9 3 1   2
  
Normal system:
0 0 1 1 
0 4 9 9   a   0 4 9 9  
  4 2 1     0 
0 2 3 3  b  0 2 3 3  
1  9 3 1     1
1 1 1   c 1 1 1 1   

 1    
9 3 2
The solution of normal system:
1
 0 0 1  1
a 0 4 9 9  0 4 9 9   2 / 3 
    4 2 1    0  
b  0 2 3 3   0 2 3 3     11 / 6
   9 3 1   1  
c   1 1   1 1     1 
 
  1 1   1 1
 9 3 1   2
Parabola (cont’d)
Given data points (0, 1), (2, 0), (3, 1), and (3, 2)
The general form of a parabola is y = ax2 + bx + c

The solution: y

a = 2/3
b = -11/6
y = -3x2 +5x +2 (3, 2)
c=1
(0, 1) (3, 1)
(2, 0) x

The parabola equation:

y = -2x2/3 -11x/6 +1
Exercise 6

• What is the least square solution of LS Ax = b if the linear system itself

is consistent?

Answer:
It is the same as the solution of said linear system.

• Let Ax = b be an inconsistent linear system. When can we determine the

least square solution by calculating x  AT A  AT b  ?
1

Answer:
When columns of A are linearly independent, or when ( AT A) 1 exists
Reflection

1. Summarize the material you just learned.

2. List 5 new things that you learned from this module where you have
not learned it before anywhere else.
3. Find 2 real world problems that you can solve by using the method you
just learned in this module.
Summary
• Check your summary, Does it contain the following key concepts?
 Inner product
 Weighted inner product
 Norm of a vector and distance between two vectors
 Angle between two vectors
 The generalized pythagorean theorem
 Orthogonal basis
 Orthonormal basis
 Cauchy-Swartz inequality
 Orthogonal complement and its properties
 Relation (orthogonal complement) between row space, column space, null
space
 Gram-Schmidt Process
 QR-Decomposition
 Least square solution
 Normal system

96
Post-test module

Copyright © 2004-2005 NameOfTheOrganization. All rights reserved.

Post-test

• Answer the following questions.

Complete the following sentences.
1. The inner product space is.....................
2. In the Euclidean vector space R2, and given vectors a= (-1, 2), b = (4, -2),
a. Draw the vectors in that space
b. determine <a, b>
2 2
3. If p  a0  a1 x  a2 x and q  b0  b1 x  b2 x is any vector in P2.
A function is defined as f p, q   p, q  a0 b0  a1b1  a2 b2

The angle between x2 and 2x in P2 is ………

Determine whether these statements are True/False.

1. The least square solution of Ax = b is (ATA)-1ATb = A-1b
2. If ||a + b||2 = ||a||2 +||b||2, then a and b orthogonal.
3. A consistent linear system does not have a least square solution.
Congratulations, you have finished module 6.
Prepare for the next module