Scribd
Upload
5 views7 pages

Mathematics of Modern Engineering I Lecture 5

Uploaded by

lzgcd99
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views7 pages

Mathematics of Modern Engineering I Lecture 5

Uploaded by

lzgcd99
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

ESE 501 Mathematics of Modern Engineering I

Lecture 5

”Inner product in a vector space. Distance between vectors and norm of a


vector. Orthogonality.”

The concept of distance between elements in a vector space is of crucial


importance.

One motivation for that arises when we try to solve a linear system

Ax = b. (1)

We know: system (1) has a solution if and only if b ∈ Col(A).

Given b ∈
/ Col(A) (otherwise expressed, the system (1) is inconsistent),
in what sense should we try then to ”solve” (1)?

Typically, one looks then for an ”approximate” solution x so that the


distance between x and Col(A) is minimal (if (1) has exact solution, then
that distance is zero).

First, we assume V = IRn and introduce the concept of distance in IRn


in the following way.

Definition 1. For any two vectors v, u ∈ IRn , we define


n
X
v • u := vi ui (1)
i=1

called the inner product (another name is the dot product) between elements
v and u.

◦ We notice that the inner product is a function from IRn × IRn into IR and
satisfies the following properties:

1) u • v = v • u;

1
2) (u + v) • w = u • w + v • w;
3) u • u ≥ 0 and u • u = 0 if and only if u = 0;

Definition 2. For any vector v ∈ IRn , we define


v
u n
√ uX
∥v∥ := v • v = t vi2 (2)
i=1

called the length (or norm) of v.

Definition 3. For any two vectors u, v ∈ IRn , the number


v
u n
uX
∥u − v∥ = t (ui − vi )2 (3)
i=1

is called the distance between u and v.

Definition 4. If for any two vectors u, v ∈ IRn , it follows that u • v = 0,


then u and v are said to be orthogonal.

Remark 1. a) For n ≤ 3, the elements from IRn can be visualized and in-
terpreted geometrically as ”usual” vectors so that the definitions of a norm
in (2) and the distance in (3) reflect nothing but well-known Pythagorean
Theorem.

b) The concept of orthogonality reflects the geometric fact that two or-
thogonal vectors are perpendicular to each other in the sense that cosine
of the angle between them is zero. Indeed, in can be proven that if vectors
u and v are from IR3 or IR2 , then the Cosine theorem holds:

u • v = ∥u∥∥v∥ cos α

where α is the angle between u and v.

c) If vectors u and v are orthogonal, then

∥u + v∥2 = ∥u∥2 + ∥v∥2

2
(again, the algebraic reflection of the fact expressed in the Pythagorean
theorem )

Definition 5. A set of vectors {u1 , ..., up } from IRn is said to be an or-


thogonal set if ui • uj = 0 for all i ̸= j.

◦ It can easily be seen that the following fact is true: Any orthogonal set
of non-zero vectors {u1 , ..., up } is a set of linearly independent vectors.

Indeed, for that to be true the equation


c1 u1 + c2 u2 + ... + cp up = 0 (4)
must have only the trivial solution
c1 = c2 = ... = cp = 0.
The latter can be verified if we dot-multiply (4) by the vector ui , i =
1, 2, ..., p and use the orthogonality of the vectors {u1 , ..., up }.

◦ Assume that the set {u1 , ..., up } is an orthogonal basis of a subspace


W ⊂ IRn . Then, any w ∈ W can uniquely be represented in the form
w = c1 u1 + c2 u2 + ... + cp up (5)
where
w • ui
ci = , i = 1, 2, ..., p. (6)
ui • ui
Indeed, (5) holds because {u1 , ..., up } is a basis in W and (6) follows if we
dot-multiply (5) by the vector ui , i = 1, 2, ..., p and use the orthogonality
of vectors {u1 , ..., up }.

An application: ”orthogonal projection of one vector onto another”

◦ Problem: Choose vectors y, u ∈ IRn and find two vectors ȳ and z such
that

1) y = ȳ + z;
2) ȳ and u are parallel;

3
3) ȳ and z are orthogonal;

(geometrically, we have to find the orthogonal projection ȳ of the vector y


onto the vector u)

It follows from the geometrical interpretation of the problem that


z = y − ȳ
and
ȳ = αu
where α is a constant to be determined.

The last two relations imply


y•u
α=
u•u
so that
y•u
ȳ = proju y =
u. (7)
u•u
Example 1. Find proju y when y = [1, 7] and u = [−4, 2].
Solution: We calculate u • u = 16 + 4 = 20 and y • u = −4 + 14 = 10 so
that α = 1/2 and ȳ = 1/2[−4, 2] = [−2, 1].

The above idea can be easily generalized to project a given vector y ∈ IRn
onto a subspace W ⊂ IRn .

Generalization: Let W = Span{u1 , u2 , .., up } and {u1 , u2 , ..., up } is an


orthogonal basis of W . Then, the orthogonal projection ȳ of a vector y ∈ IRn
onto the subspace W is given by
y • u1 y • up
ȳ = u1 + ... + up
u1 • u1 up • up
(the proof is similar to the projection onto a subspace spanned by one
vector)

Remark 2. It follows from the construction that


min ∥y − w∥ = ∥y − ȳ∥
w∈W

4
where ȳ = projW y.

◦ Finally, often we are interested to find projW y of a given vector y ∈ IRn


onto a given subspace W ⊂ IRn when W = Span{u1 , ..., up } and we know
that {u1 , .., up } is a basis of W but not necessarily an orthogonal one.

The following procedure called Gram-Schmidt orthogonalization process


allows to construct from the basis {u1 , .., up } an orthogonal basis {v1 , ..., vp }
as follows:
v1 = u1 ,
u2 • v1
v2 = u2 − v1 ,
v1 • v1
u3 • v2 u3 • v1
v3 = u3 − v2 − v1 ,
v2 • v2 v1 • v1
...........................
up • vp−1 up • v1
vp = up − vp−1 − .... − v1 ,
vp−1 • vp−1 v1 • v1
Example 2. Consider the vectors u1 = [−1, 1, 1], u2 = [2, 0, 3], and u3 =
[4, 2, 3] in IR3 . Construct an orthogonal basis in W = Span{u1 , u2 , u3 }.

Solution: 1) First we want to check that the given vectors are linearly
independent and thus build a basis in IR3 .
2) One uses then the Gram-Schmidt procedure to obtain the orthogonal
vectors {v1 , v2 , v3 }.

Application to an inconsistent system Ax = b:

Goal: Find a vector x̃ such that

∥Ax̃ − b∥ ≤ ∥Ax − b∥ (8)

for all x ∈ IRn .

To solve the above problem, we define

b̃ := projCol(A) b.

5
Then, it follows from the previous discussions that any solution of (8)
should satisfy
Ax̃ = b̃. (9)
Since b̃ ∈ Col(A), there is at least one solution x̃ of (8).

It is not hard to show (it would be a good exercise to verify it!) that
solving the equation (9) is equivalent to solving the system
AT Ax̃ = AT b (10)
which is called the system of normal equations corresponding to the prob-
lem (8).

Practically, it is what one does: one solves the system (10).

Example 3. Solve the system Ax = b, where A is given by


 
2 1
A = −2 0
2 3
and b = [−5, 8, 2]T .
Solution: First, one can easily see that the given system is inconsistent.
That is, there are no ”exact solutions” of the system. Then, we will solve
the system approximately. In other words, we will find a solution in the
sense (8). For that, it is enough for us to solve the corresponding system
of normal equations. We calculate
 
  2 1  
2 −2 2 12 8
AT A = · −2 0 =
1 0 3 8 10
2 3
and
 
  −5  
2 −2 2 −22
AT b = · 8 =
1 0 3 1
2
Since the matrix AT A is invertible, the only solution is x̃ = (AT A)−1 AT b.

6
We find det(AT A) = 56 and
   
1 10 −8 1 10 −8
(AT A)−1 = = .
det(AT A) −8 12 56 −8 12

You might also like