Algebra

Nguyen Thieu Huy

Hanoi University of Science and Technology

Nguyen Thieu Huy (HUST) Algebra 1 / 13

III. Orthogonality in (V , h·, ·i)
3.1. Definition:
The vectors u, v ∈ V are said to be orthogonal (write u ⊥ v )
⇐⇒ hu, v i = 0, (also say: u is orthogonal to v )

Nguyen Thieu Huy (HUST) Algebra 2 / 13

III. Orthogonality in (V , h·, ·i)
3.1. Definition:
The vectors u, v ∈ V are said to be orthogonal (write u ⊥ v )
⇐⇒ hu, v i = 0, (also say: u is orthogonal to v )

Note: 1) O ⊥ v ∀v ∈ V .

Nguyen Thieu Huy (HUST) Algebra 2 / 13

III. Orthogonality in (V , h·, ·i)
3.1. Definition:
The vectors u, v ∈ V are said to be orthogonal (write u ⊥ v )
⇐⇒ hu, v i = 0, (also say: u is orthogonal to v )

Note: 1) O ⊥ v ∀v ∈ V . If u ⊥ v ∀v ∈ V , then u = O.

Nguyen Thieu Huy (HUST) Algebra 2 / 13

III. Orthogonality in (V , h·, ·i)
3.1. Definition:
The vectors u, v ∈ V are said to be orthogonal (write u ⊥ v )
⇐⇒ hu, v i = 0, (also say: u is orthogonal to v )

Note: 1) O ⊥ v ∀v ∈ V . If u ⊥ v ∀v ∈ V , then u = O.
2) For u, v 6= O; if u ⊥ v , the angle between u and v is π2

Nguyen Thieu Huy (HUST) Algebra 2 / 13

III. Orthogonality in (V , h·, ·i)
3.1. Definition:
The vectors u, v ∈ V are said to be orthogonal (write u ⊥ v )
⇐⇒ hu, v i = 0, (also say: u is orthogonal to v )

Note: 1) O ⊥ v ∀v ∈ V . If u ⊥ v ∀v ∈ V , then u = O.
2) For u, v 6= O; if u ⊥ v , the angle between u and v is π2
3) Pythagorean theorem: u ⊥ v ⇔ kuk2 + kv k2 = kv + uk2

Nguyen Thieu Huy (HUST) Algebra 2 / 13

III. Orthogonality in (V , h·, ·i)
3.1. Definition:
The vectors u, v ∈ V are said to be orthogonal (write u ⊥ v )
⇐⇒ hu, v i = 0, (also say: u is orthogonal to v )

Note: 1) O ⊥ v ∀v ∈ V . If u ⊥ v ∀v ∈ V , then u = O.
2) For u, v 6= O; if u ⊥ v , the angle between u and v is π2
3) Pythagorean theorem: u ⊥ v ⇔ kuk2 + kv k2 = kv + uk2
3.2. Definition:
∅ 6= S ⊂ V . The orthogonal complement of S, denoted by S ⊥ , is defined
as S ⊥ = {v ∈ V | hv , ui = 0 ∀u ∈ S} .

Examples. {(1, −2, 1); (−2, 1, 1)}⊥

Nguyen Thieu Huy (HUST) Algebra 2 / 13

III. Orthogonality in (V , h·, ·i)
3.1. Definition:
The vectors u, v ∈ V are said to be orthogonal (write u ⊥ v )
⇐⇒ hu, v i = 0, (also say: u is orthogonal to v )

Note: 1) O ⊥ v ∀v ∈ V . If u ⊥ v ∀v ∈ V , then u = O.
2) For u, v 6= O; if u ⊥ v , the angle between u and v is π2
3) Pythagorean theorem: u ⊥ v ⇔ kuk2 + kv k2 = kv + uk2
3.2. Definition:
∅ 6= S ⊂ V . The orthogonal complement of S, denoted by S ⊥ , is defined
as S ⊥ = {v ∈ V | hv , ui = 0 ∀u ∈ S} .

Examples. {(1, −2, 1); (−2, 1, 1)}⊥

Note:
1 For ∅ 6= S ⊂ V we have: S ⊥ is a subspace of V and S ⊥ ∩ S ⊂ {O}.

2 If S is a subspace of V then S ⊥ ∩ S = {O}.

3 For u ∈ V , denote u ⊥ = {u} .

⊥

Examples.
Nguyen Thieu Huy (HUST) Algebra 2 / 13
Orthogonal and orthonormal set

3.3. Definition. Consider (V , h·, ·i): IPS

The set S = {u1 , u2 , · · · , um } ⊂ V is called:
1 orthogonal ⇔ hui , uj i = 0 if i =
6 j; where 1 6 i 6 m; 1 6 j 6 m
(
0 if i 6= j
2 orthonormal ⇔ hui , uj i = where 1 6 i 6 m; 1 6 j 6 m.
1 if i = j

So, an orthonormal set is an orthogonal set of unit vectors. Examples.

Nguyen Thieu Huy (HUST) Algebra 3 / 13

Orthogonal and orthonormal set

3.3. Definition. Consider (V , h·, ·i): IPS

The set S = {u1 , u2 , · · · , um } ⊂ V is called:
1 orthogonal ⇔ hui , uj i = 0 if i =
6 j; where 1 6 i 6 m; 1 6 j 6 m
(
0 if i 6= j
2 orthonormal ⇔ hui , uj i = where 1 6 i 6 m; 1 6 j 6 m.
1 if i = j

So, an orthonormal set is an orthogonal set of unit vectors. Examples.

Remark: If S is an orthogonal set and O ∈/ S, then S is linealy
independent.

Nguyen Thieu Huy (HUST) Algebra 3 / 13

Orthogonal and orthonormal set

3.3. Definition. Consider (V , h·, ·i): IPS

The set S = {u1 , u2 , · · · , um } ⊂ V is called:
1 orthogonal ⇔ hui , uj i = 0 if i =
6 j; where 1 6 i 6 m; 1 6 j 6 m
(
0 if i 6= j
2 orthonormal ⇔ hui , uj i = where 1 6 i 6 m; 1 6 j 6 m.
1 if i = j

So, an orthonormal set is an orthogonal set of unit vectors. Examples.

Remark: If S is an orthogonal set and O ∈ / S, then S is linealy
independent.
Therefore, an orthonormal set is linealy independent.

Nguyen Thieu Huy (HUST) Algebra 3 / 13

Orthogonal and orthonormal set

3.3. Definition. Consider (V , h·, ·i): IPS

The set S = {u1 , u2 , · · · , um } ⊂ V is called:
1 orthogonal ⇔ hui , uj i = 0 if i =
6 j; where 1 6 i 6 m; 1 6 j 6 m
(
0 if i 6= j
2 orthonormal ⇔ hui , uj i = where 1 6 i 6 m; 1 6 j 6 m.
1 if i = j

So, an orthonormal set is an orthogonal set of unit vectors. Examples.

Remark: If S is an orthogonal set and O ∈ / S, then S is linealy
independent.
Therefore, an orthonormal set is linealy independent.
3.4. Definition: Consider Euclidean space (V , h·, ·i), V 6= {O}
A basis S of V is called
1 an orthogonal basis (OG-basis) if S is an orthogonal set.
2 an orthonormal basis (ON-basis) if S is an orthonormal set.

Nguyen Thieu Huy (HUST) Algebra 3 / 13

Remark: a) Consider Euclidean space V with dim V = n: Then an
orthogonal set S of n nonzero vectors is OG-basis. So, an orthonormal set
S of n vectors is ON-basis.

Nguyen Thieu Huy (HUST) Algebra 4 / 13

Remark: a) Consider Euclidean space V with dim V = n: Then an
orthogonal set S of n nonzero vectors is OG-basis. So, an orthonormal set
S of n vectors is ON-basis.
b) u = 1 ∀u 6= 0. So, if B = {u1 , u2 , · · · , un } is OG-basis, then
n kuk o
u1 u2 un
,
ku1 k ku2 k , · · · , kun k is ON-basis.
Examples.

Nguyen Thieu Huy (HUST) Algebra 4 / 13

Remark: a) Consider Euclidean space V with dim V = n: Then an
orthogonal set S of n nonzero vectors is OG-basis. So, an orthonormal set
S of n vectors is ON-basis.
b) u = 1 ∀u 6= 0. So, if B = {u1 , u2 , · · · , un } is OG-basis, then
n kuk o
u1 u2 un
,
ku1 k ku2 k , · · · , kun k is ON-basis.
Examples.
3.3 Theorem (Coordinates relative to OG-basis, or ON-basis)
(V , h·, ·i); B = {e1 , e2 , · · · , en } is OG-basis of V . Then, ∀u ∈ V , we have
n
P hu,ek i
u= hek ,ek i ek .
k=1  
hu,e1 i
 he 1 ,e1 i 
 hu,e2 i 
 he ,e i 
So, [u]B =  2. 2 
 .. 
 
hu,en i
hen ,en i
n
P
In case B is ON-basis, we have: u = hu, ek iek ∀u ∈ V .
k=1
Nguyen Thieu Huy (HUST) Algebra 4 / 13
Proof: Since B is a basis V , we can write
Pn
u= λk ek . To compute λk .
k=1

Nguyen Thieu Huy (HUST) Algebra 5 / 13

Proof: Since B is a basis V , we can write
Pn
u= λk ek . To compute λk .
k=1
Fix any j ∈ {1, 2, · · · , n}
 we nhave:
Pn P
hu, ej i = λk ek , ej = λk hek , ej i = λj hej , ej i
k=1 k=1
(here hek , ej i = 0 ∀k 6= j).

Nguyen Thieu Huy (HUST) Algebra 5 / 13

Proof: Since B is a basis V , we can write
Pn
u= λk ek . To compute λk .
k=1
Fix any j ∈ {1, 2, · · · , n}
 we nhave:
Pn P
hu, ej i = λk ek , ej = λk hek , ej i = λj hej , ej i
k=1 k=1
(here hek , ej i = 0 ∀k 6= j).
hu,ej i
Therefore, λj = e ,e ∀j ∈ {1, 2, · · · , n}. Just replacing letter j by k we
h j ji
obtain the wanted formula.

Nguyen Thieu Huy (HUST) Algebra 5 / 13

Proof: Since B is a basis V , we can write
Pn
u= λk ek . To compute λk .
k=1
Fix any j ∈ {1, 2, · · · , n}
 we nhave:
Pn P
hu, ej i = λk ek , ej = λk hek , ej i = λj hej , ej i
k=1 k=1
(here hek , ej i = 0 ∀k 6= j).
hu,ej i
Therefore, λj = e ,e ∀j ∈ {1, 2, · · · , n}. Just replacing letter j by k we
h j ji
obtain the wanted formula.
3.4 Theorem (Gram-Schmidt orthonormalization process:)
Euclidean space (V , h·, ·i), dim V = n > 0. B = {v1 , v2 , · · · , vn } is a basis
of V . Then, ∃ ON-basis T = {e1 , e2 , · · · , en } of V such that
Span {e1 , e2 , · · · , ek } = Span {v1 , v2 , · · · , vk } ∀k = 1, 2, · · · , n.

Nguyen Thieu Huy (HUST) Algebra 5 / 13

Proof: Since B is a basis V , we can write
Pn
u= λk ek . To compute λk .
k=1
Fix any j ∈ {1, 2, · · · , n}
 we nhave:
Pn P
hu, ej i = λk ek , ej = λk hek , ej i = λj hej , ej i
k=1 k=1
(here hek , ej i = 0 ∀k 6= j).
hu,ej i
Therefore, λj = e ,e ∀j ∈ {1, 2, · · · , n}. Just replacing letter j by k we
h j ji
obtain the wanted formula.
3.4 Theorem (Gram-Schmidt orthonormalization process:)
Euclidean space (V , h·, ·i), dim V = n > 0. B = {v1 , v2 , · · · , vn } is a basis
of V . Then, ∃ ON-basis T = {e1 , e2 , · · · , en } of V such that
Span {e1 , e2 , · · · , ek } = Span {v1 , v2 , · · · , vk } ∀k = 1, 2, · · · , n.

Proof: Firstly, we look for an OG-basis U = {u1 , u2 , · · · , un } such that

Span {u1 , u2 , · · · , uk } = Span {v1 , v2 , · · · , vk } ∀k = 1, 2, · · · , n.

Nguyen Thieu Huy (HUST) Algebra 5 / 13

k = 1 : Put u1 = v1

Nguyen Thieu Huy (HUST) Algebra 6 / 13

k = 1 : Put u1 = v1
k = 2 : Put u2 = v2 + αu1 , clearly Span {u1 , u2 } = Span {v1 , v2 }, we find
α ∈ R such that hu1 , u2 i = 0.

Nguyen Thieu Huy (HUST) Algebra 6 / 13

k = 1 : Put u1 = v1
k = 2 : Put u2 = v2 + αu1 , clearly Span {u1 , u2 } = Span {v1 , v2 }, we find
α ∈ R such that hu1 , u2 i = 0.
hv2 ,u1 i
⇔ hu1 , v2 + αu1 i = 0 ⇔ hu1 , v2 i + α hu1 , u1 i = 0, vy α = − hu1 ,u1 i
,
hv2 ,u1 i
therefore, u2 = v2 − hu1 ,u1 i u1

Nguyen Thieu Huy (HUST) Algebra 6 / 13

k = 1 : Put u1 = v1
k = 2 : Put u2 = v2 + αu1 , clearly Span {u1 , u2 } = Span {v1 , v2 }, we find
α ∈ R such that hu1 , u2 i = 0.
hv2 ,u1 i
⇔ hu1 , v2 + αu1 i = 0 ⇔ hu1 , v2 i + α hu1 , u1 i = 0, vy α = − hu1 ,u1 i
,
hv2 ,u1 i
therefore, u2 = v2 − hu 1 ,u1 i
u1
Generally, 1 < k 6 n: after obtaining OG- set {u1 , u2 , · · · , uk−1 }, we then
find uk = vk + α1 u1 + α2 u2 + · · · + αk−1 uk−1 and use huk , uj i = 0 to
compute αj , it yields:
hvk ,u1 i hvk ,u2 i hvk ,uk−1 i
uk = vk − hu 1 ,u1 i
u1 − hu 2 ,u2 i
u2 − · · · − huk−1 ,uk−1 i uk−1 ∀k = 2, · · · , n.

Nguyen Thieu Huy (HUST) Algebra 6 / 13

k = 1 : Put u1 = v1
k = 2 : Put u2 = v2 + αu1 , clearly Span {u1 , u2 } = Span {v1 , v2 }, we find
α ∈ R such that hu1 , u2 i = 0.
hv2 ,u1 i
⇔ hu1 , v2 + αu1 i = 0 ⇔ hu1 , v2 i + α hu1 , u1 i = 0, vy α = − hu1 ,u1 i
,
hv2 ,u1 i
therefore, u2 = v2 − hu 1 ,u1 i
u1
Generally, 1 < k 6 n: after obtaining OG- set {u1 , u2 , · · · , uk−1 }, we then
find uk = vk + α1 u1 + α2 u2 + · · · + αk−1 uk−1 and use huk , uj i = 0 to
compute αj , it yields:
hvk ,u1 i hvk ,u2 i hvk ,uk−1 i
uk = vk − hu 1 ,u1 i
u1 − hu 2 ,u2 i
u2 − · · · − huk−1 ,uk−1 i uk−1 ∀k = 2, · · · , n.
After obtaining OG-basis U = {u1 , u2 , · · · , un } as above, we put
u
ej := uj ∀j = 1, · · · , n then we get the wanted ON-basis
k jk
T = {e1 , e2 , · · · , en } .

Nguyen Thieu Huy (HUST) Algebra 6 / 13

3.5 Corollary
Consider non-trivial Euclidean space (E , h·, ·i), dim E = n > 0:
1 E has at least one orthonormal basis (ON-basis).
2 For orthonormal set S = {e1 , e2 , · · · , ek } , k < n,, we can extend S
to an ON – basis {e1 , e2 , · · · , ek , ek+1 , · · · , en } of E .

Nguyen Thieu Huy (HUST) Algebra 7 / 13

3.5 Corollary
Consider non-trivial Euclidean space (E , h·, ·i), dim E = n > 0:
1 E has at least one orthonormal basis (ON-basis).
2 For orthonormal set S = {e1 , e2 , · · · , ek } , k < n,, we can extend S
to an ON – basis {e1 , e2 , · · · , ek , ek+1 , · · · , en } of E .
3 Let {0} =
6 F be a subspace of E . Then for each v ∈ E , there exists a
unique couple of vectors w1 ∈ F and w2 ∈ F ⊥ such that v = w1 + w2 .
(We write F ⊕ F ⊥ = E and call E the direct sum of F and F ⊥ ).

Nguyen Thieu Huy (HUST) Algebra 7 / 13

3.5 Corollary
Consider non-trivial Euclidean space (E , h·, ·i), dim E = n > 0:
1 E has at least one orthonormal basis (ON-basis).
2 For orthonormal set S = {e1 , e2 , · · · , ek } , k < n,, we can extend S
to an ON – basis {e1 , e2 , · · · , ek , ek+1 , · · · , en } of E .
3 Let {0} =
6 F be a subspace of E . Then for each v ∈ E , there exists a
unique couple of vectors w1 ∈ F and w2 ∈ F ⊥ such that v = w1 + w2 .
(We write F ⊕ F ⊥ = E and call E the direct sum of F and F ⊥ ).

Proof.
Let {e1 , e2 , · · · , ek } be an ON-basis of F . Extend it to obtain ON-basis
{e1 , e2 , · · · , ek , ek+1 , · · · , en } of E . It’s easy to see that {ek+1 , · · · , en } is
ON-basis of F ⊥ (homework). So, for v ∈ E we have
n X k Xn
P
v= hv , ej iej = hv , ej iej + hv , ej iej .
j=1 j=1 j=k+1
| {z } | {z }
w1 w2

Nguyen Thieu Huy (HUST) Algebra 7 / 13

Proof (continued).
To prove that the expression v = w1 + w2 for w1 ∈ F and w2 ∈ F ⊥ is
unique, we let v = w10 + w20 be another expression such that w10 ∈ F and
w20 ∈ F ⊥ .

Nguyen Thieu Huy (HUST) Algebra 8 / 13

Proof (continued).
To prove that the expression v = w1 + w2 for w1 ∈ F and w2 ∈ F ⊥ is
unique, we let v = w10 + w20 be another expression such that w10 ∈ F and
w20 ∈ F ⊥ . It follows that v = w1 + w2 = w10 + w20 ⇔ w1 − w10 = w20 − w2 .
| {z } | {z }
∈F ∈F ⊥

Nguyen Thieu Huy (HUST) Algebra 8 / 13

Proof (continued).
To prove that the expression v = w1 + w2 for w1 ∈ F and w2 ∈ F ⊥ is
unique, we let v = w10 + w20 be another expression such that w10 ∈ F and
w20 ∈ F ⊥ . It follows that v = w1 + w2 = w10 + w20 ⇔ w1 − w10 = w20 − w2 .
| {z } | {z }
∈F ∈F ⊥
Hence,w1 − w10 = w20 − w2 ∈ F ∩ F ⊥ = {O} ⇒ w1 = w10 and w20 = w2 .

Nguyen Thieu Huy (HUST) Algebra 8 / 13

Proof (continued).
To prove that the expression v = w1 + w2 for w1 ∈ F and w2 ∈ F ⊥ is
unique, we let v = w10 + w20 be another expression such that w10 ∈ F and
w20 ∈ F ⊥ . It follows that v = w1 + w2 = w10 + w20 ⇔ w1 − w10 = w20 − w2 .
| {z } | {z }
∈F ∈F ⊥
Hence,w1 − w10 = w20 − w2 ∈ F ∩ F ⊥ = {O} ⇒ w1 = w10 and w20 = w2 .

3.6 Definition: Consider (E , h·, ·i), dim E = n > 0

Let F be a nontrivial subspace of E . We define the mapping P : E → F
by P(v ) = w1 if v = w1 + w2 where w1 ∈ F and w2 ∈ F ⊥ . Then, P is
called the orthogonal projection on F .

Nguyen Thieu Huy (HUST) Algebra 8 / 13

Proof (continued).
To prove that the expression v = w1 + w2 for w1 ∈ F and w2 ∈ F ⊥ is
unique, we let v = w10 + w20 be another expression such that w10 ∈ F and
w20 ∈ F ⊥ . It follows that v = w1 + w2 = w10 + w20 ⇔ w1 − w10 = w20 − w2 .
| {z } | {z }
∈F ∈F ⊥
Hence,w1 − w10 = w20 − w2 ∈ F ∩ F ⊥ = {O} ⇒ w1 = w10 and w20 = w2 .

3.6 Definition: Consider (E , h·, ·i), dim E = n > 0

Let F be a nontrivial subspace of E . We define the mapping P : E → F
by P(v ) = w1 if v = w1 + w2 where w1 ∈ F and w2 ∈ F ⊥ . Then, P is
called the orthogonal projection on F .

Remarks: 1)For an ON-basis S = {e1 , ..., ek } of F , the orthogonal

Pk
projection P : E → F can be computed as P(v ) = hv , ej iej ∀v ∈ E
j=1

Nguyen Thieu Huy (HUST) Algebra 8 / 13

Proof (continued).
To prove that the expression v = w1 + w2 for w1 ∈ F and w2 ∈ F ⊥ is
unique, we let v = w10 + w20 be another expression such that w10 ∈ F and
w20 ∈ F ⊥ . It follows that v = w1 + w2 = w10 + w20 ⇔ w1 − w10 = w20 − w2 .
| {z } | {z }
∈F ∈F ⊥
Hence,w1 − w10 = w20 − w2 ∈ F ∩ F ⊥ = {O} ⇒ w1 = w10 and w20 = w2 .

3.6 Definition: Consider (E , h·, ·i), dim E = n > 0

Let F be a nontrivial subspace of E . We define the mapping P : E → F
by P(v ) = w1 if v = w1 + w2 where w1 ∈ F and w2 ∈ F ⊥ . Then, P is
called the orthogonal projection on F .

Remarks: 1)For an ON-basis S = {e1 , ..., ek } of F , the orthogonal

Pk
projection P : E → F can be computed as P(v ) = hv , ej iej ∀v ∈ E
j=1
2) For an orthogonal projection P : E → F we have that, P is a linear
operator, P 2 = P; and Im P = F ; Ker P = F ⊥ .
Nguyen Thieu Huy (HUST) Algebra 8 / 13
3.7 Lemma: Consider (E , h·, ·i), dim E = n > 0
{O} =6 F : a subspace of E ; P : E → F : orthogonal projection on F .
Then kx-yk > kx − P(x)k for all x ∈ E and y ∈ F . (The “=” occurs if and
only if y=P(x))

Nguyen Thieu Huy (HUST) Algebra 9 / 13

3.7 Lemma: Consider (E , h·, ·i), dim E = n > 0
{O} =6 F : a subspace of E ; P : E → F : orthogonal projection on F .
Then kx-yk > kx − P(x)k for all x ∈ E and y ∈ F . (The “=” occurs if and
only if y=P(x))

Proof.
∀y ∈ F we have kx-yk2 = kx-P(x)+P(x)-yk2 .

Nguyen Thieu Huy (HUST) Algebra 9 / 13

3.7 Lemma: Consider (E , h·, ·i), dim E = n > 0
{O} =6 F : a subspace of E ; P : E → F : orthogonal projection on F .
Then kx-yk > kx − P(x)k for all x ∈ E and y ∈ F . (The “=” occurs if and
only if y=P(x))

Proof.
∀y ∈ F we have kx-yk2 = kx-P(x)+P(x)-yk2 .
Since x – P(x) ∈ F ⊥ and P(x) –y ∈ F , using Pythagorean Theorem we
have kx-P(x)+P(x)-yk2 = kx-P(x)k2 + kP(x)-yk2 ≥ kx-P(x)k2
(“=” occurs ⇔ P(x)=y). (Take square roots, we are done!)

Nguyen Thieu Huy (HUST) Algebra 9 / 13

IV. Least square approximation
e = [A...B]
Consider AX = B for A ∈ Mm×n (R); B ∈ Mm×1 (R); A
In case rank A 6= rank A
e ⇐⇒ AX = B has no solution

Nguyen Thieu Huy (HUST) Algebra 10 / 13

IV. Least square approximation
Consider AX = B for A ∈ Mm×n (R); B ∈ Mm×1 (R); A e = [A...B]
In case rank A 6= rank A
e ⇐⇒ AX = B has no solution
Q: Which vector X̃ ∈ Mn×1 (R) will minimize the norm kAX − Bk2 for
X ∈ Mn×1 (R)?

Nguyen Thieu Huy (HUST) Algebra 10 / 13

IV. Least square approximation
Consider AX = B for A ∈ Mm×n (R); B ∈ Mm×1 (R); A e = [A...B]
In case rank A 6= rank A
e ⇐⇒ AX = B has no solution
Q: Which vector X̃ ∈ Mn×1 (R) will minimize the norm kAX − Bk2 for
X ∈ Mn×1 (R)? (This means, kAX̃ − Bk2 = min kAX − Bk2 )
X ∈Mn×1 (R)

Nguyen Thieu Huy (HUST) Algebra 10 / 13

IV. Least square approximation
Consider AX = B for A ∈ Mm×n (R); B ∈ Mm×1 (R); A e = [A...B]
In case rank A 6= rank A
e ⇐⇒ AX = B has no solution
Q: Which vector X̃ ∈ Mn×1 (R) will minimize the norm kAX − Bk2 for
X ∈ Mn×1 (R)? (This means, kAX̃ − Bk2 = min kAX − Bk2 )
X ∈Mn×1 (R)
Such a vector X̃ , which minimizes the norm kAX − Bk,
is called a least square solution to system AX = B.

Nguyen Thieu Huy (HUST) Algebra 10 / 13

IV. Least square approximation
Consider AX = B for A ∈ Mm×n (R); B ∈ Mm×1 (R); A e = [A...B]
In case rank A 6= rank A
e ⇐⇒ AX = B has no solution
Q: Which vector X̃ ∈ Mn×1 (R) will minimize the norm kAX − Bk2 for
X ∈ Mn×1 (R)? (This means, kAX̃ − Bk2 = min kAX − Bk2 )
X ∈Mn×1 (R)
Such a vector X̃ , which minimizes the norm kAX − Bk,
is called a least square solution to system AX = B.

4.1 Theorem: The least square solution X

e is a solution to system
T
A AX e =A BT

Nguyen Thieu Huy (HUST) Algebra 10 / 13

IV. Least square approximation
Consider AX = B for A ∈ Mm×n (R); B ∈ Mm×1 (R); A e = [A...B]
In case rank A 6= rank A
e ⇐⇒ AX = B has no solution
Q: Which vector X̃ ∈ Mn×1 (R) will minimize the norm kAX − Bk2 for
X ∈ Mn×1 (R)? (This means, kAX̃ − Bk2 = min kAX − Bk2 )
X ∈Mn×1 (R)
Such a vector X̃ , which minimizes the norm kAX − Bk,
is called a least square solution to system AX = B.

4.1 Theorem: The least square solution X

e is a solution to system
T
A AX e =A BT

Proof. Put E := Mm×1 (R) with usual hX , Y i = X T Y ; F := Colsp(A).

Nguyen Thieu Huy (HUST) Algebra 10 / 13

IV. Least square approximation
Consider AX = B for A ∈ Mm×n (R); B ∈ Mm×1 (R); A e = [A...B]
In case rank A 6= rank A
e ⇐⇒ AX = B has no solution
Q: Which vector X̃ ∈ Mn×1 (R) will minimize the norm kAX − Bk2 for
X ∈ Mn×1 (R)? (This means, kAX̃ − Bk2 = min kAX − Bk2 )
X ∈Mn×1 (R)
Such a vector X̃ , which minimizes the norm kAX − Bk,
is called a least square solution to system AX = B.

4.1 Theorem: The least square solution X

e is a solution to system
T
A AX e =A BT

Proof. Put E := Mm×1 (R) with usual hX , Y i = X T Y ; F := Colsp(A).

Taking into account that AX ∈ F for all X ∈ Mn×1 (R), by Lemma 3.7 we
obtain that kAX − Bk2 is smallest for such X = X̃ that AX̃ = P(B),
where P : E → F is orthogonal projection on F , (since
kAX − Bk2 = kB − AX k2 > kB − P(B)k2 and “=” occurs if and only if
AX = P(B)).
Nguyen Thieu Huy (HUST) Algebra 10 / 13
Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)
⇔ AX̃ –B ⊥ Ci ∀i = 1, 2, ..., n (where Ci is the i th column of A)

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)
⇔D AX̃ –B ⊥ CiE∀i = 1, 2, ..., n (where Ci is the i th column of A)
⇔ AX̃ –B, Ci = 0 ∀i = 1, 2, ..., n.

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)
⇔D AX̃ –B ⊥ CiE∀i = 1, 2, ..., n (where Ci is the i th column of A)
⇔ AX̃ –B, Ci = 0 ∀i = 1, 2, ..., n.
⇔ CiT (AX̃ –B) = 0 ∀i = 1, 2, ..., n.

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)
⇔D AX̃ –B ⊥ CiE∀i = 1, 2, ..., n (where Ci is the i th column of A)
⇔ AX̃ –B, Ci = 0 ∀i = 1, 2, ..., n.
⇔ CiT (AX̃ –B) = 0 ∀i = 1, 2, ..., n.
⇔ AT (AX̃ –B) = O ⇔ AT AX̃ –AT B = O

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)
⇔D AX̃ –B ⊥ CiE∀i = 1, 2, ..., n (where Ci is the i th column of A)
⇔ AX̃ –B, Ci = 0 ∀i = 1, 2, ..., n.
⇔ CiT (AX̃ –B) = 0 ∀i = 1, 2, ..., n.
⇔ AT (AX̃ –B) = O ⇔ AT AX̃ –AT B = O
⇔ AT AX̃ = AT B

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)
⇔D AX̃ –B ⊥ CiE∀i = 1, 2, ..., n (where Ci is the i th column of A)
⇔ AX̃ –B, Ci = 0 ∀i = 1, 2, ..., n.
⇔ CiT (AX̃ –B) = 0 ∀i = 1, 2, ..., n.
⇔ AT (AX̃ –B) = O ⇔ AT AX̃ –AT B = O
⇔ AT AX̃ = AT B
    
1 1 2 x1 1
Example. Consider 1 −1 0x2  = 1
0 1 1 x3 2
| {z }| {z } | {z }
  A X B
1 1 2 1
e = 1 −1 0 1, rank A 6= rank A
A e ⇒ system has no solution.
0 1 1 2

Nguyen Thieu Huy (HUST) Algebra 11 / 13

Proof (continued).
We now find X̃ such that AX̃ = P(B). To do this, we write
AX̃ –B = P(B) − B ∈ F ⊥ .
This is equivalent to AX̃ –B ⊥ U for all U ∈ F = Colsp(A)
⇔D AX̃ –B ⊥ CiE∀i = 1, 2, ..., n (where Ci is the i th column of A)
⇔ AX̃ –B, Ci = 0 ∀i = 1, 2, ..., n.
⇔ CiT (AX̃ –B) = 0 ∀i = 1, 2, ..., n.
⇔ AT (AX̃ –B) = O ⇔ AT AX̃ –AT B = O
⇔ AT AX̃ = AT B
    
1 1 2 x1 1
Example. Consider 1 −1 0x2  = 1
0 1 1 x3 2
| {z }| {z } | {z }
  A X B
1 1 2 1
e = 1 −1 0 1, rank A 6= rank A
A e ⇒ system has no solution.
0 1 1 2
To find the least square solution Xe we solve AT AX
e = AT B.
Nguyen Thieu Huy (HUST) Algebra 11 / 13
  
x̃1
Putting X := x̃2  we obtain
e 
x̃3
      
1 1 0 1 1 2 x̃1 1 1 0 1
1 −1 1 1 −1 0 x̃2  = 1 −1 1 1
2 0 1 0 1 1 x̃3 2 0 1 2

Nguyen Thieu Huy (HUST) Algebra 12 / 13

  
x̃1
Putting X := x̃2  we obtain
e 
x̃3
      
1 1 0 1 1 2 x̃1 1 1 0 1
1 −1 1 1 −1 0 x̃2  = 1 −1 1 1
2 0 1 0 1 1 x̃3 2 0 1 2
    
2 0 2 x̃1 2
⇔ 0 3 3 x̃2  = 2
2 3 5 x̃3 4

Nguyen Thieu Huy (HUST) Algebra 12 / 13

  
x̃1
Putting X := x̃2  we obtain
e 
x̃3
      
1 1 0 1 1 2 x̃1 1 1 0 1
1 −1 1 1 −1 0 x̃2  = 1 −1 1 1
2 0 1 0 1 1 x̃3 2 0 1 2
    
2 0 2 x̃1 2
⇔ 0 3 3 x̃2  = 2
2 3 5 x̃3 4
2
⇔ x̃3 is arbitrary in R; x̃2 = − x̃3 ; x̃1 = 1 − x̃3 ;
3
   
x̃1 1−t
e := x̃2  =  2 − t  ∀t ∈ R.
So, X 3
x̃3 t

Nguyen Thieu Huy (HUST) Algebra 12 / 13

V. Orthogonal matrices
5.1 Definition
Matrix A ∈ Mn (R) is said to be orthogonal if AT A = I (i.e., A is invertible
and A−1 = AT )

Nguyen Thieu Huy (HUST) Algebra 13 / 13

V. Orthogonal matrices
5.1 Definition
Matrix A ∈ Mn (R) is said to be orthogonal if AT A = I (i.e., A is invertible
and A−1 = AT )
 
cos α sin α
Example. 1) A =
− sin α cos α
    
T cos α − sin α cos α sin α 1 0
⇒A A= =
sin α cos α − sin α cos α 0 1

Nguyen Thieu Huy (HUST) Algebra 13 / 13

V. Orthogonal matrices
5.1 Definition
Matrix A ∈ Mn (R) is said to be orthogonal if AT A = I (i.e., A is invertible
and A−1 = AT )
 
cos α sin α
Example. 1) A =
− sin α cos α
    
T cos α − sin α cos α sin α 1 0
⇒A A= =
sin α cos α − sin α cos α 0 1
2)  1
− √2 √16 √13


B =  √12 √1 √1  ⇒ BT B = I .

6 3
−2 √1
0 √
6 3

Nguyen Thieu Huy (HUST) Algebra 13 / 13

V. Orthogonal matrices
5.1 Definition
Matrix A ∈ Mn (R) is said to be orthogonal if AT A = I (i.e., A is invertible
and A−1 = AT )
 
cos α sin α
Example. 1) A =
− sin α cos α
    
T cos α − sin α cos α sin α 1 0
⇒A A= =
sin α cos α − sin α cos α 0 1
2)  1
− √2 √16 √13


B =  √12 √1 √1  ⇒ BT B = I .

6 3
−2 √1
0 √
6 3

5.2 Theorem
E : Euclidean space 6= {0}. The change-of-basis matrix from an ON-basis
to another ON-basis is orthogonal.
Nguyen Thieu Huy (HUST) Algebra 13 / 13