Algebra

Associate Professor Nguyen Trung Thanh

Faculty of Data Science and Artificial Intelligence

Vector space

1. Euclidean vector space

2. Linear independence and Spanning set

3. Subspaces and dimension

4. General vector space

2/34
Euclidean vector space
Euclidean n-space

Definition
Euclidean n-space, also called Rn , is formally written as

Rn = {(x1 , x2 , . . . , xn ) | xi ∈ R}

We call x = (x1 , x2 , . . . , xn ) a vector in Rn , and n is the dimension of the space.

Examples
• (1, 2) ∈ R2 .
• (−2, 1, 0) ∈ R3 .
√
• (−1, 0, 2, π, 105 ) ∈ R5 .

3/34
Vector Operations

Let x = (x1 , x2 , . . . , xn ) and y = (y1 , y2 , . . . , yn ) be vectors in Rn and r ∈ R.

• Equality:
(x1 , x2 , . . . , xn ) = (y1 , y2 , . . . , yn ) means xi = yi ∀ i.
• Addition:

(x1 , x2 , . . . , xn ) + (y1 , y2 , . . . , yn ) = (x1 + y1 , x2 + y2 , . . . , xn + yn ).

• Scalar multiplication:

r · (x1 , x2 , . . . , xn ) = (rx1 , rx2 , . . . , rxn ).

Note: If we look at the vectors as column vectors, then equality, addition and scalar
multiplication are the same as matrix equality, addition and scalar multiplication.
4/34
Properties of vectors

Let x = (x1 , . . . , xn ), y = (y1 , . . . , yn ) and z = (z1 , . . . , zn ) be vectors in Rn and r, s ∈ R

be scalars. In addition, let 0 = (0, . . . , 0) and 1 = (1, . . . , 1), and −x = (−x1 , . . . , −xn ).
Addition Multiplication
(A1) x + y ∈ Rn (M1) rx ∈ Rn
(A2) x + y = y + x (M2) r(x + y) = rx + ry
(A3) x + 0 = x (M3) (r + s)x = rx + sx
(A4) x + (−x) = 0 (M4) (rs)x = r(sx)
(A5) x + (y + z) = (x + y) + z (M5) 1x = x

5/34
The dot product of vectors in Rn

Definition
Let x = (x1 , x2 , . . . , xn ), y = (y1 , y2 , . . . , yn ) be vectors in Rn , then the dot product (or
inner product) of x and y is

x · y = x1 y1 + x2 y2 + · · · + xn yn .

If x and y are column vectors (represented as n × 1 matrices), then

   
x1 y1
x2   y2 
   
x =  . , y =  . , x · y = x⊤ y = y⊤ x.
 ..   .. 
xn yn
6/34
The length of a vector in Rn

Definition
√
• in R2 : ||x|| = x21 + x22
√
• in R3 : ||x|| = x21 + x22 + x23
√ √
• in Rn : ||x|| = x21 + x22 + · · · + x2n = x · x ≥ 0

Property

1. ||rx|| = |r| ||x||

2. |x · y| ≤ ||x|| ||y|| (Cauchy-Schwarz inequality)
3. ||x + y||2 + ||x − y||2 = 2(||x||2 + ||y||2 )) (Parallelogram equality)
4. ||x + y|| ≤ ||x|| + ||y|| (Triangle inequality)
7/34
The angle between vectors in Rn

Definition
Let x, y be nonzero vectors in Rn , then the angle θ between x and y is determined by
x·y
cos θ = .
||x|| ||y||

8/34
The distance between two vectors in Rn

Euclidean distance
Let x, y be vectors in Rn , then the Euclidean distance between x and y is

de (x, y) = ||x − y||.

Property

1. de (x, y) ≥ 0 with equality if and only if x = y

2. de (x, y) = de (y, x)
3. de (x, z) ≤ de (x, y) + de (y, z) (Triangle inequality)

9/34
The distance between two vectors in Rn

Cosine similarity and cosine distance

Let x, y be vectors in Rn , then the cosine similarity between x and y is

sc (x, y) = cos θ.

and the cosine distance between x and y is

dc (x, y) = 1 − cos θ.

10/34
Cosine Similarity

11/34
Cosine Similarity vs. Euclidean Distance

12/34
What are the purposes of using vector spaces

Vector space model are to consider the relationship between data that are
represented by vectors.
• Each object in the dataset is represented as a vector of features.
• The dimension of a vector is the number of features.

13/34
Cosine similarity

Model cars as a 3-dimensional space

Every car is expressed as a vector x = (x1 , x2 , x3 ), where
• x1 represents max_speed (maximum speed in kmph)
• x2 represents acceleration_time (number seconds to reach 100 kmph)
• x3 represents price

Example:
• porsche = (304, 4.4, 99200) porsche tesla bmw
• tesla = (250, 3.8, 79990) porsche 1.00 0.43 -0.93
• bmw = (160, 7, 60000) tesla 0.43 1.00 -0.73
bmw -0.93 -0.73 1.00
14/34
Cosine similarity

x = “That is a very happy person” =⇒ (1, 1, 1, 1, 1, 1, 0, 0, 0, 0)

y = “That is a happy person” =⇒ (1, 1, 1, 0, 1, 1, 0, 0, 0, 0)
z = “That is a happy dog” =⇒ (1, 1, 1, 0, 1, 0, 1, 0, 0, 0)
t = “Today is a sunny day” =⇒ (0, 1, 1, 0, 0, 0, 0, 1, 1, 1)

That is a very happry person dog today sunny day

x 1 1 1 1 1 1 0 0 0 0
y 1 1 1 0 1 1 0 0 0 0
z 1 1 1 0 1 0 1 0 0 0
t 0 1 1 0 0 0 0 1 1 1

15/34
Cosine similarity

16/34
Cosine distance and Euclidean distance

https://machinelearningmastery.com
17/34
Cosine distance and Euclidean distance

Sorted by Euclidean distance Sorted by cosine distance

Euclidean consine Euclidean consine

Australia 0.0000e+00 -2.2204e-16 Australia 0.0000e+00 -2.2204e-16
Mexico 1.5338e+11 7.9495e-03 Colombia 8.9811e+11 1.7206e-03
Spain 3.4119e+11 3.0579e-03 Cuba 1.1260e+12 2.4839e-03
Turkey 3.7982e+11 3.5028e-03 Italy 1.0883e+12 2.6777e-03
Indonesia 4.0835e+11 7.4176e-03 Argentina 7.5723e+11 2.9301e-03

Compare between Australia to all other countries on the list based on the
selected economic metrics
https://machinelearningmastery.com

18/34
Vector embedding space

19/34
Further applications

• Document Similarity.
• Recommendation Systems.
• Text Classification.
• Information Retrieval.
• Clustering.

20/34
Linear independence and Spanning set
Linear combination of vectors

Definition
If x1 , x2 , . . . , xm are vectors in Rn , then the vector

y = r1 x1 + r2 x2 · · · + rm xm

is called a linear combination of x1 , x2 , . . . , xm .

Examples
• (4, 2) is a linear combination of (1, 1) and (2, 0).
• [ ][ ] [ ] [ ] [ ] [ ]
1 3 x1 4 1 3 4
= =⇒ x1 + x2 = .
2 2 x2 0 2 2 0
21/34
Linear independence

Definition
The set of vectors P = {x1 , x2 , . . . , xm } ⊆ Rn is linearly independent if

0 = r1 x1 + r2 x2 · · · + rm xm (⋆)

then it must be that r1 = r2 = . . . = rm = 0. If there is at least one ri ̸= 0 so that (⋆) is

satisfied then the set P is called linearly dependent.

Examples
Verify that
• P = {x1 = (1, 2, 0), x2 = (0, 1, 1), x3 = (−1, 0, 2)} is linearly independent.
• P ∪ {x4 = (1, 1, 1)} is linearly dependent.
22/34
Linear independence

Property
• If P = {x1 , x2 , . . . , xm } ⊆ Rn is linearly independent then m ≤ n.
• Every set P = {x1 , x2 , . . . , xm } ⊆ Rn with m > n is linearly dependent.

Proof
Hint. Write r1 x1 + r2 x2 · · · + rm xm = 0 ⇐⇒ Ar = 0, where
 
r1
[ ]  r2 
 
A = [x1 ] [x2 ] · · · [xm ] n×m
, and r =  . 
 .. 
rm
23/34
Spanning sets

Spanning vectors
A set of vectors P = {x1 , x2 , . . . , xm } ⊆ Rn is spans Rn if every y ∈ Rn is a linear
combination of vectors in P, i.e., there exist r1 , r2 , . . . , rm so that

y = r1 x1 + r2 x2 · · · + rm xm .

We denote Rn = Span(P).

Examples
Verify that
• P = {x1 = (1, 2, 0), x2 = (0, 1, 1)} does not span R3 .
• P = {x1 = (1, 2, 0), x2 = (0, 1, 1), x3 = (−1, 0, 2)} spans R3 .
24/34
Bases of Rn

Definition
If a set of vectors P = {x1 , x2 , . . . , xm } ⊆ Rn spans Rn and m = n then we call P a
basic of Rn .

Examples
• P = {x1 = (1, 2, 0), x2 = (0, 1, 1), x3 = (−1, 0, 2)} is a basic of R3 .
• P = {x1 = (1, 0, 0), x2 = (0, 1, 0), x3 = (0, 0, 1)} is a standard basic of R3 .

25/34
Bases of Rn

Properties
• If P is a basic of Rn then P is linearly independent
• There are infinitely many bases of Rn
• Every basic has exactly n vectors
• dim(Rn ) = n

26/34
Spanning sets

Definition
For a set of vectors P = {x1 , x2 , . . . , xm } ⊆ Rn , we denote by Span(P) the set of all
linear combination of vectors in P. That is,

Span(P) = {y | y = r1 x1 + r2 x2 · · · + rm xm , ri ∈ R}.

27/34
Subspaces and dimension
Subspaces of Rn

Definition
A nonempty subset S ⊆ Rn is a subspace of Rn if it satisfies the following conditions
• If x, y ∈ S then x + y ∈ S
• If x ∈ S and r ∈ R then rx ∈ S

Example
• {0} is zero subspace of Rn
• {(x, 0) | x ∈ R} and {(0, y) | y ∈ R} are a subspaces of R2
• {(x, y) | x − y = 0, x, y ∈ R} is a subspace of R2
• {(x, y) | x + y = 1, x, y ∈ R} is not a subspace of R2
• For any nonempty set of vectors P, Span(P) is a subspace of Rn
28/34
Examples of Subspaces of Rn

Subspaces in R2 Subspaces in R3

29/34
Bases and dimension of subspaces

Definition
Let S be a subspace of Rn . A set P = {x1 , x2 , . . . , xm } ⊆ S is a basic of S if
• If P is linear independent
• Span(P) = S
and m is called the dimension of the subspace S.

Example
• The zero space has no basic and has a dimension of 0
• (1, 0) is a basic of the subspace Span({(1, 0)})
• Find a basic of the subspace {(x, y) | x − y = 0, x, y ∈ R}

30/34
Hyperplanes as subspaces in Rn

Hyperplanes
∑
For a given a number b ∈ R and a tuple (r1 , r2 , . . . , rn ) ∈ Rn , where i r2i ̸= 0, the
subset S ⊆ Rn containing all the vectors (x1 , x2 , . . . , xn ) ∈ Rn such that

r1 x2 + r2 x2 + · · · + rn xn = b

is called a hyperplane.

Hyperplanes as subspaces
A hyperplane H is a subspace of Rn if and only if it passes through the origin 0. In this
case the dimension of H is n − 1.

31/34
Properties of subspaces

Properties
Let S1 , S2 are subspaces of Rn .
• S1 ∩ S2 = {x | x ∈ S1 , x ∈ S2 } is a subspace of both S1 , S2
• S1 + S2 = {x + y | x ∈ S1 , y ∈ S2 } is a subspace of Rn

dim(S1 + S2 ) = dim(S1 ) + dim(S2 ) − dim(S1 ∩ S2 )

• S1 ∪ S2 = {x | x ∈ S1 or x ∈ S2 } may not a subspace of Rn

Example
Find a basic of the subspace which is the intersection of two hyperplanes in R3 :

(H1 ) : 2x1 + x2 − x3 = 0 and (H2 ) : x1 + 3x2 + x3 = 0. 32/34

General vector space
General vector space

A vector space V is a non-empty set equipped with an addition operation and a

scalar multiplication operation such that for all x, y, z ∈ V and for all r, s ∈ R:
Addition Multiplication
(A1) x + y ∈ V (M1) rx ∈ V
(A2) x + y = y + x (M2) r(x + y) = rx + ry
(A3) ∃ 0 ∈ V : x + 0 = x (M3) (r + s)x = rx + sx
(A4) ∃ − x ∈ V : x + (−x) = 0 (M4) (rs)x = r(sx)
(A5) x + (y + z) = (x + y) + z (M5) 1x = x

33/34
General vector space

Example
• The set Rn×1 of all column matrices is a vector space
• The set R1×n of all row matrices is a vector space
• The set Rn×m of all n × m matrices is a vector space
• The set Pn of all polynomials of degree at most n with real coefficients is a
vector space with the foregoing addition and scalar multiplication

34/34