Lecture 2

Math Foundations Team

Vector Spaces

▶ We would like to shift focus from systems of linear equations

and their solutions to vector spaces.
▶ Vector spaces are structured spaces in which vectors live.
▶ What do we mean by ”structure” here?
Groups

▶ We already have some notion about the structure of a vector

space, i.e adding two vectors returns a vector, multiplying a
vector by a scalar also returns a vector.
▶ To formalize these notions, we need the concept of a Group
▶ A Group is a set G and an operation ⊗ : G × G → G defined
on G . Then (G , ⊗) is called a group if the following
properties hold:
▶ Closure of G under ⊗: ∀x, y ∈ G , x ⊗ y ∈ G
▶ Associativity: ∀x, y , z ∈ G , (x ⊗ y ) ⊗ z = x ⊗ (y ⊗ z)
▶ Neutral or Identity element: ∋ e ∈ G , ∀x ∈ G x ⊗ e = x
▶ Inverse element: ∀x ∈ G , ∋ y ∈ G , x ⊗ y = y ⊗ x = e
▶ If in addition to all the above properties we have
∀x, y ∈ G , x ⊗ y = y ⊗ x then (G , ⊗) is an Abelian group
Some examples

▶ (Z , +) is an Abelian group. We can see that all the

aforementioned properties hold.
▶ What about (N0 , +)? This is not a group since there is no
inverse element for an arbitrary element in it.
▶ (Z , .) where Z is the set of integers, and . is product. It has
the identity element but there exist elements in it that don’t
have an inverse.
Vector Spaces

▶ We will now consider sets that are just like groups in terms of
its properties with respect to an inner operation +, and also
have an outer operation called . which denotes the
multiplication of a vector x ∈ G by a scalar λ ∈ R.
▶ The inner operation can be viewed as a form of addition while
the outer operation can be viewed as a form of scaling.
▶ A real-valued vector space is a set V = (V, +, .) with
operations +, . such that + : V × V → V and . : R × V → V
where (V, +) is an Abelian group, and the following properties
hold:
▶ Distributivity
▶ associativity with respect to the outer operation.
▶ there exists a neutral or identity element with respect to the
outer operation.
Vector Spaces

Let us look at the properties of a Vector Space more carefully:

▶ Distributivity: ∀λ ∈ R, x, y ∈ V, λ.(x + y) = λ.x + λ.y and
∀λ, ψ ∈ R, (λ + ψ).x = λ.x + ψ.x
▶ Associativity with respect to the outer operation:
∀λ, ψ ∈ R, x ∈ V, λ.(ψ.x) = (λψ).x
▶ Neutral element with respect to the outer operation: ∀x ∈ V,
1.x = x
▶ The elements x ∈ V are called vectors.
▶ The neutral element with respect to (V, +) is the zero vector
[0, 0, . . . 0]T and the inner operation is called vector addition.
Examples of vector spaces

Consider some examples of vector spaces, for example V = Rn

▶ We can define addition:
x + y=(x1 , x2 , . . . xn ) + (y1 , y2 , . . . yn ) =
(x1 + y1 , x2 + y2 , . . . xn + yn )
▶ Multiplication by scalars:
λx=λ(x1 , x2 , . . . xn ) = (λx1 , λx2 , . . . λxn ), ∀λ ∈ R, x ∈ Rn .
Another example

What we call a vector need not be the standard column vector that
we are accustomed to treating as a vector. We can think of m × n
matrices as vectors and create a vector space out of them. Thus
V = Rm×n with addition and multiplication defined as below:
 
a11 + b11 . . . a1n + b1n
▶ Addition A + B =  ..
 is defined

 .
an1 + bn1 . . . ann + bnn
element-wise for two matrices A, B ∈ V.
 
λa11 . . . λa1n
▶ Multiplication by scalars: λA =  .. 
 . 
λan1 . . . λann
Vector subspaces

▶ Let V = (V, +, .) be a vector space and let U ⊆ V, U ̸= Φ.

then U = (U, +, .) is called a vector subspace of V if U is a
vector space with the vector space operations + and .
restricted to U × U and R × U.
▶ We use the notation U ⊆ V to denote that U is a vector
subspace of V .
▶ If U ⊆ V and V is a vector space, then U naturally inherits
many properties directly from V because they hold for all x
∈ V , and in particular for all x ∈ U ⊆ V . These properties
include the Abelian group property, associativity, distributivity
and the neutral element.
Establishing vector subspaces

How do we show if (U, +, .) is subspace of V ?

We need to show that
▶ U ̸= Φ
▶ Closure of U with respect to the outer and inner operations,
i.e ∀λ ∈ R, ∀x, ∈ U, λx ∈ U and ∀x , y ∈ U, x + y ∈ U.
Examples

▶ Let V = R2 and U be the y -axis. . Is U = (U, +, .) a

subspace of V = (V, +, .)? Answer: Yes, because the addition
of any two vectors on the y -axis remains on the y -axis so
closure with respect to the addition operation is satisfied.
Also any vector on the y -axis when scaled by any real number
(including 0) will yield a vector on y -axis.
▶ What about when we shift the y -axis one unit to the right, i.e
U = {x = 1}? Answer: We no longer have a vector subspace
since scaling with respect to the outer operation does not
have the closure property.
More examples

▶ What about the subset of R2 that represents a square around

the origin , i.e −1 ≤ x ≤ 1, −1 ≤ y ≤ 1? This is again not a
subspace.
▶ This subspace is of particular interest to us called the
nullspace of a matrix, i.e the set of solutions to a linear
system of equations Ax = 0. Why is this a vector subspace?
Linear combination and linear independence

▶ We know that we can stay in a vector space by adding vectors

belonging to the space, and scaling them?
▶ We are now interested in a different question → can we come
up with a set of vectors such that every vector in the vector
space can be represented as a sum of these vectors with
scaling as necessary?
▶ The answer is yes. A set of vectors capable of representing all
vectors in a vector space is called a basis.
▶ To explore the question of finding a basis, we need to learn
the concepts of linear combination and linear independence.
Linear combination

▶ Consider a vector space V and a set of vectors x1 , x2 , . . .

xk ∈ V . Then every v∈ V that is of the form v = λ1 x1 +
λ2 x2 . . . + λk xk is a linear combination of the vectors x1 , x2 ,
. . . xk .
▶ Note that the 0 can be written trivially as a linear combination
of the given k vectors xi s. We are however interested in
non-trivial linear combinations of vectors to get the 0-vector.
Linear (in)dependence

▶ Consider a vector space V and k vectors x1 , x2 , . . . xk ∈ V .

Then if there is a non-trivial
Pi=k linear combination of the given
vectors such that 0 = i=1 λi xi where at least one of the λi s
is non-zero, then the vectors x1 , x2 , . . . xk are linearly
dependent.
▶ Put another way, if the only way we can combine vectors to
get the 0-vector is by letting all the λi be zero, then the
vectors are linearly independent.
Linear (in)dependence

▶ What does the concept of linear independence capture? If the

vectors are linearly dependent we can write one of the vectors
in terms of the others, so that vector is redundant. On the
other hand, when the vectors are linearly independent, each
vector brings something to the table which the other vectors
collectively cannot replace.
Linear independence

▶ k vectors in a vector space are either linearly independent or

linearly dependent. There is no third option.
▶ If at least one of the vectors x1 , x2 , . . . xk is the 0-vector,
then the vectors are linearly dependent. Why? Apply the
definition to see that this is the case. We can choose a
non-zero λ for the 0-vector and zero λs for all the other
vectors to get the linear combination to be zero.
▶ If all the vectors x1 , x2 , . . . xk are non-zero, then the vectors
are linearly dependent if and only if one of them is a linear
combination of the others.
Using Gaussian elimination to check for linear
independence

▶ A practical way of checking whether a bunch of vectors are

linearly independent is to fill out the columns of a matrix with
the given vectors and then perform Gaussian elimination to
get the row-echelon form
▶ The pivot columns indicate all the vectors that are linearly
independent, and they are all on the left.
▶ The non-pivot columns can be expressed as a linear
combination of the pivot columns
Example

▶ Let us start with the matrix

 
1 2 3
2 4 4

▶ Gaussian elimination of this matrix will give rise to the

following row-echelon form
 
1 2 3
0 0 −2

▶ The pivot columns are the first and the third column and we
see that the second column which is a non-pivot column can
be expressed as a linear combination of the pivot columns to
its left, which is just twice the first column.
Another example

     
1 1 −1
2 1 −2
Consider the vectors x1 = −3, x2 = 0, x3 =  1 
    

4 2 1
To check for linear independence we set up the equation with λs as
follows: λ1 x1 + λ2 x2 + λ3 x3 = 0.
As before we put the given vectors into a matrix to get:
 
1 1 −1
 2 1 −2
 
−3 0 1 
4 2 1
.
Another example continued

We now perform Gaussian elimination on this matrix to identify

pivot and non-pivot columns. If all columns are pivot columns then
the given three vectors are linearly independent. After Gaussian
elimination we get  
1 1 −1
0 1 0 
 
0 0 1 
0 0 0
.
Note that all columns are pivot columns, so we conclude that the
original three vectors are linearly independent. The only way to
combine those vectors and get the 0-vector is to take 0s as the λs.
More on linear independence

▶ Let us say that we have a bunch of k independent vectors b1 ,

b2 , . . . bk
▶ We now have m vectors x1 , x2 , . . . xm , each of which is some
linear combination of the vectors bi .
▶ The goal is to find under what conditions the vectors xi are
linearly independent.
▶ Can we express the criterion of linear independence of the xi s
in terms of the way in which we combine the bi s to get the
xi s?
More on linear independence

We have the following equations:

i=k
X
x1 = λi1 bi ;
i=1
i=k
X
x2 = λi2 bi ;
i=1
..
.
i=k
X
xm = λim bi ;
i=1
More on linear independence

▶ We can define a matrix B = [b1 , b2 , . . . bk ] whose columns are

the original linearly independent vectors b1 , b2 . . . bk and then
we see that xj = Bλj where λj is a vector of coefficients.
▶ Now to test whether x1 , x2 . . . xm are linearly independent, we
set up the equation j=m j=0 ψj xj = 0 and check whether the
P
ψj s have to be zero. This means
j=m j=m j=m
ψj xj = ψj B λj = B
X X X
ψj λj
j=1 j=1 j=1
More on linear independence

Now j=m j=1 ψj λj = v , a vector so Bv is a linear combination of

P
the column vectors of B , and the only way a linear combination of
the columns of B can be equal to zero if all theP combining
coefficients are zero. This means that v = 0 = j=m ψj λj .
Pj=m j=1
The fact that j=0 ψj xj = 0 implies that j=1 ψj λj = 0 means
Pj=m
that x1 , x2 , . . . xm are linearly independent if and only if
λ1 , λ2 , . . . λm are linearly independent.
Example

Consider the following set of equations:

x1 = b1 − 2b2 + b3 − b4
x2 = −4b1 − 2b2 + 4b4
x3 = 2b1 + 3b2 − b3 − 3b4
x4 = 17b1 − 10b2 + 11b3 + b4
Example

We would like to find if the vectors x1 , x2 , x3 , x4 are linearly

independent. Following the theory established in the last slides we
check if the vectors corresponding to the λj s are linearly
independent.
We therefore need to check whether the vectors
       
1 −4 2 17
−2 −2  3  −10
 , , ,  (1)
 1   0  −1  11 
−1 4 −3 1

are linearly independent.

Example

We form a matrix consisting of all these column vectors:

 
1 −4 2 17
−2 −2 3 −10
A =  
1 0 −1 11 
−1 4 −3 1

and perform Gaussian elimination to get the reduced row-echelon

form.
Example

We get the following matrix:

 
1 0 0 −7
0 1 0 −15
B =  
0 0 1 −18
0 0 0 0

Clearly there is a non-trivial way of combining the columns of the

above matrix to get the 0-vector , i.e take 7 × the first column,
add it to 15 × the second column, add the result to 18 × the third
column and add the result to the fourth column to get 0. Thus the
original set of vectors x1 , x2 , x3 , x4 are not linearly independent.
Insight from the example

▶ We had a set of k vectors bi which we combined linearly to

get m vectors xi .
▶ Determining whether the m vectors xi were linearly
independent boiled down to checking if the column vectors of
a k × m matrix were linearly independent.
▶ We performed Gaussian elimination on this matrix to get the
reduced row-echelon form.
Insight from the example

▶ Here is an interesting insight: if m > k, we have more

columns than rows and there can only be as many pivots as
there are non-zero rows, so the number of pivots and therefore
the number of pivot columns is less than k.
▶ We are therefore guaranteed to have non-pivot columns in
this case, so that we can express the non-pivot columns in
terms of the pivot columns to their left and get a non-trivial
linear combination of the columns to get the 0 vector. This
would make the m vectors xi linearly dependent.
Basis of vector space

▶ Do there exist a set of vectors in a vector space which ”span”

the entire space?
▶ What we mean here is that any vector v ∈ V can be
generated as a linear combination of the vectors in question.
▶ Is such a set of vectors unique to the vector space?
▶ Are all sets capable of generating all vectors in a vector space
of the same size?
Generating set and basis

▶ Consider a vector space V = (V, +, .) and a set of vectors

A = {x1 , x2 , . . . xk } ⊆ V .
▶ If every vector v ∈ V can be expressed as a linear combination
of the vectors x1 , x2 , . . . xk , then A is called a generating set
of V.
▶ The set of all linear combinations of the vectors in A is known
as the span of A.
▶ If A spans the vector space V we write V = span[A].
Basis as smallest generating set

▶ Consider a vector space V = (V, +, .) and A ⊆ V.

▶ A generating set A of V is called minimal if there is no
smaller set à such that à ⊂ A ⊆ V that spans V .
▶ Every linearly independent generating set of V is minimal and
is called a basis of V .
Characterizations of a basis

The following statements are equivalent:

▶ B is a basis of V .
▶ B is a minimal generating set.
▶ B is a maximal linearly independent set of vectors in V such
that adding a vector to it will make it a linearly dependent set.
▶ Every vector x ∈ V is a linear combination of vectors from B,
and every
P linear combination is unique. This means that if
x = i=k b b
Pi=k
λ
i=1 i i = i=1 i i then λi = ψi , 1 ≤ i ≤ k.
ψ ,
Basis is not unique

     
1 0 0
▶ Consider the space R3 . The canonical basis is 0 , 1 , 0
0 0 1
     
1 1 1
▶ Another basis for R3 is 0 , 1 , 1
0 0 1
     
0.5 1.8 −2.2
▶ Yet another basis for R3 is 0.8 , 0.3 , −3.3
0.4 0.3 1.5
Not a basis

▶ Is any linear independent set a basis?

▶ No, we can have a linearly independent set that has too few
vectors to become a basis
▶ As an example, consider the following set of vectors in R4
     
1 2 1
2 −1 1
 , , 
3  0  0
4 2 4
▶ Why is the above not a basis?
Some properties

▶ Vector spaces can be finite or infinite-dimensional.

▶ We consider only finite-dimensional vector spaces.
▶ In a finite-deimensional vector space V , the number of vectors
in the basis is known as dim(V ).
▶ If U ⊆ V , then dim(U) ≤ dim(V ).
▶ dim(U) = dim(V ) if and only if U = V .
Dimension of a vector space

▶ Is the size of the basis merely the number of components in

the vector?
▶ So far, that has been the case.
 
0
▶ But consider the vector space span( ) - the vector has two
1
components, but the basis is just one vector. All vectors in
0
this vector space are multiples of the vector .
1
Finding a basis

There are three steps to finding a basis for a vector space. A basis
of a subspace U = span(x1 , . . . , xm ) ⊆ Rn can be found by
executing the following steps:
▶ Write the spanning vectors as columns of a matrix A.
▶ Determine the row-echelon form of A.
▶ The spanning vectors associated with the pivot columns are a
basis of U.
Example

▶ Consider the vector subspace U ⊆ R5 which is spanned by the

following vectors:
       
1 2 3 −1
2 −1 −4 8
x1 = −1 , x2 =  1  , x3 =  3  , x4 −5
       
−1 2 5 −6
−1 −2 −3 1

▶ We would like to check if the vectors x1 , x2 , x3 , x4 constitute a

basis for the subspace U.
▶ We therefore need to check whether the given vectors are
linearly independent or not.