Applied Linear Algebra for Signal Processing, Data Analytics and Machine

Learning
Professor Aditya K. Jagannatham
Department of Electrical Engineering
Indian Institute of Technology, Kanpur
Lecture- 5
Matrix: Column Space, Linear Independence, Rank of Matrix, Gaussian
elimination
Hello, welcome to another module in this massive open online course. In today's module,
let us start our discussion or let us start looking at another very important concept or
property of a matrix that is known as the rank of a matrix, alright?

(Refer Slide Time: 00:35)

So, let us start our discussion on another very important aspect which is the rank of a
matrix. So, in order to answer and this is a very important concept or this notion of rank is
a very important concept and is one of the fundamental property. So, let us start this
discussion, let us again begin with an 𝑚 × 𝑛 matrix. I hope all of you remember we have
this 𝑚 × 𝑛 matrix 𝐀 as

𝑎11 𝑎12 … 𝑎1𝑛

𝑎21 𝑎22 … 𝑎2𝑛
𝐀=[ ⋮ ⋮ ⋮ ⋮ ]
𝑎𝑚1 𝑎𝑚2 … 𝑎𝑚𝑛
 ̅1
And this is your 𝐀 = [𝒂 ̅2
𝒂 ̅𝑛 ]. Let us recall that we can represent matrix 𝐀 as a
… 𝒂
collection of 𝑛 columns where each column is basically represents a vector of size 𝑚 where
𝑚 denotes the number of rows of the matrix 𝐀. So, these are 𝑛 columns of size 𝑚 or
basically you can think of these are column vectors of size 𝑚 × 1.

(Refer Slide Time: 2:22)

Now, we define this notion of what is known as the column space which is essentially

𝑥1
𝑥2
̅1
𝐲̅ = 𝐀𝐱̅ = [𝒂 ̅2
𝒂 ̅𝑛 ] [ ⋮ ].
… 𝒂
𝑥𝑛
This can be now essentially represented as a linear combination of the columns of the
matrix 𝐀 with the elements of the vector 𝐱̅. You can view it as nothing but a linear
̅1 + 𝑥2 𝒂
combination of the columns of the matrix 𝐀 as 𝑥1 𝒂 ̅2 + ⋯ + 𝑥𝑛 𝒂
̅𝑛 .

So, this is basically nothing but you can see, this is essentially a linear combination. You
have already seen this concept of linear combination. This is a linear combination of
̅1 + 𝑥2 𝒂
columns of 𝐀 and we call this vector 𝐲̅ = 𝐀𝐱̅ = 𝑥1 𝒂 ̅2 + ⋯ + 𝑥𝑛 𝒂
̅𝑛 , which is
basically a linear combination of columns of 𝐀, and is belonging to the column space, that
is, we say this belongs to the column space of 𝐀.

(Refer Slide Time: 04:12)

So, this 𝐶(𝐀), this is essentially the column space of the matrix 𝐀. This is essentially is a
subspace. So, 𝐶(𝐀) this is the subspace spanned by columns of 𝐀 that is essentially
it contains all possible linear combinations of the columns of 𝐀.
(Refer Slide Time: 5:55)

Now, we come to another important concept that is linear independence. What is linear
independence? Now linear independence, that is, if we have these vectors
̅1 , 𝒂
𝒂 ̅2 , … , 𝒂
̅𝑛 , and we say that this set of vectors is linearly independent. This is linearly
̅1 + 𝑥2 𝒂
independent if there is a linear combination 𝑥1 𝒂 ̅2 + ⋯ + 𝑥𝑛 𝒂
̅𝑛 =0. This is possible
if and only if all the 𝑥𝑖 ’s are identical and equal to 0, that is, what it means essentially
̅1 + 𝑥2 𝒂
saying is, that is, any linear combination 𝑥1 𝒂 ̅2 + ⋯ + 𝑥𝑛 𝒂
̅𝑛 , this is equal to 0, only
if each 𝑥𝑖 that is 𝑥1 = 𝑥2 = ⋯ = 𝑥𝑛 = 0, and if any of these coefficients is nonzero then
the linear combination is not 0. So, essentially it means that the only linear combination of
̅1 , 𝒂
these vectors 𝒂 ̅2, … , 𝒂
̅𝑛 that is 0, when all the coefficients are identical and equal to 0.
So, to put the same thing in other words it says, that is, there do not exist scalars
̅1 + 𝑥2 𝒂
𝑥1 , 𝑥2 , … , 𝑥𝑛 not all 0, this is an important step, not all zeros such that 𝑥1 𝒂 ̅2 + ⋯ +
̅𝑛 = 0.
𝑥𝑛 𝒂

̅1 +
That is there does not exist any set of any scalars 𝑥1 , 𝑥2 , … , 𝑥𝑛 not all 0 such that 𝑥1 𝒂
̅2 + ⋯ + 𝑥𝑛 𝒂
𝑥2 𝒂 ̅𝑛 = 0, that is, the only time this is 0 is when all the scalars are 0. Which
means even if one or a few of them are nonzero, then the linear combination is not 0. Such
a set of vectors is known as a linearly independent set of vectors. We call this as basically
linearly independent. So, this is essentially known as your linearly independent set of
vectors.

(Refer Slide Time: 9:49)

And for example, let us take a look at a simple example. Let us take

1 2 7
̅ ̅ ̅
𝒂̅1 = [−1] , 𝒂̅2 = [ 1 ] , and 𝒂3 = [−1].
1 −3 −3

̅1 , 𝒂
Now, let us ask the question are these set of vectors 𝒂 ̅2 , and 𝒂
̅3 , we asked the question
̅1 , 𝒂
that 𝒂 ̅2 , and 𝒂
̅3 are these linearly independent?

And we will show that they are not linearly independent. In fact, there exists a nonzero
linear combination that exists, a linear combination with nonzero combining coefficients
such that the linear combination equals 0. That is essentially how we are going to
demonstrate that these are not linearly independent.

1 2
̅1 + 2 𝒂
For instance, if you look at 3𝒂 ̅2 , which is basically 3 [−1] + 2 [ 1 ], then you can
1 −3
see, this is equal to essentially

1 2 7
3 [−1] + 2 [ 1 ] = [−1] = ̅𝒂3 ,
1 −3 −3

which is essentially nothing but which is equal to ̅𝒂̅3 . So,

̅1 + 2 𝒂
3𝒂 ̅2 = ̅𝒂3 ⇒ 3𝒂
̅1 + 2 𝒂
̅ 2 − ̅𝒂3 = 0.
 ̅1 + 𝑥2 𝒂
So, if your set 𝑥1 = 3, 𝑥2 = 2, and 𝑥3 = −1. So, we have 𝑥1 𝒂 ̅2 + ⋯ + 𝑥𝑛 𝒂
̅𝑛 = 0,
where not all are 0. In fact, you can see neither is 0, in fact you can say all are not 0 in this
case. So, therefore, these are not linearly independent.

(Refer Slide Time: 14:18)

̅1 and 𝒂
On the other hand, if you take 𝒂 ̅2 now, you can show that these are linearly
independent, these are linearly independent. Not very difficult to show that these are
linearly independent, that is, there do not exist, there are no scalars 𝑥1 , 𝑥2 not both 0 such
̅1 + 𝑥2 𝒂
that 𝑥1 𝒂 ̅2 = 0. So, that is, there do not exist scalars 𝑥1 , 𝑥2 not both 0 so that is at
̅1 + 𝑥2 𝒂
least one of them is nonzero, says that 𝑥1 𝒂 ̅ 2 = 0.

Now, in fact how do we systematically determine the rank? And this is why our row and
column elimination, this is what we are going to come to. So, for a system, so, in just a
little bit of course, we have not illustrated at this point a clear procedure to determine if the
columns are linearly dependent or the columns are linearly independent. Previously, we
simply showed a linear combination of the column vectors which reduces to 0. In this case,
we are saying that there is no linear combination which is equal to 0 with at least one of
the coefficients nonzero.

So, how to systematically determine linear dependence or linear independence that is why
a row and column elimination is useful, which we are going to see shortly. So, this brings
us to the next important concept which is essentially that of the rank which is what we set
out to discuss.

(Refer Slide Time: 17:00)

So, now, we come to the notion of the rank of a matrix 𝐀. This is equal to the maximum
number of linearly independent columns which is equal to the and this can be shown, we
are not going to explicitly show this, that is, the maximum number of linearly independent
columns is also equal to the maximum number of linearly independent rows.

The maximum number of linearly independent columns, this is what we call as the column
rank. And the maximum number of linearly independent rows, this is what we call as more
specifically the row rank. And we can show that for any matrix 𝐀, the column rank equals
row rank which is what we call as simply the rank of matrix 𝐀, which is less than or equal
to the minimum of the number of linearly independent rows and columns.

So, the rank has to be less than or equal to minimum of the number of rows or columns.
So, therefore, it has to be less than both the number of rows as well as the number of
columns, the maximum it can achieve is the minimum of the number of rows and columns
it cannot be greater than that. The rank of a matrix any matrix 𝐀, the maximum value of
that can be the minimum of the number of rows and columns.

So, it is always less than or equal to the minimum of both the number of rows and columns.
And what is the rank? The rank is the minimum number. It is basically the maximum
number of linearly independent columns or the maximum number of linearly independent
rows and both these are actually the same.

This basically, maximum number of linearly independent columns is the column rank
maximum number of linearly independent rows is the row rank. And actually, it can be
shown that both these quantities are the same. And this is what we are terming as the rank
of the matrix 𝐀.

(Refer Slide Time: 20:24)

How to find the rank, we now come to this concept of row operations. How to determine
rank? This is a very important concept. This is why are the row operations you can simply
do column operations to get what is known as the row, so this is basically known as the
Gaussian elimination procedure.

(Refer Slide Time: 21:19)

For instance, let us again go back to our matrix 𝐀 . Let us take the three columns that we
̅1 , 𝒂
looked at before, let us put them in, does look at our 𝒂 ̅2 , and 𝒂
̅3 that we previously
looked at, let us put them together as this is the first column as

1 2 7
̅̅1 = [−1] , 𝒂
𝒂 ̅̅2 = [ 1 ] , and ̅𝒂3 = [−1].
1 −3 −3

So, these are your three columns. So, this essentially becomes your 3 cross 3 matrix. So
the matrix 𝐀 is

1 2 7
[𝒂
𝐀 = ̅1 𝒂
̅2 𝒂 ]
̅ 3 = [−1 1 −1].
1 −3 −3

So, you know that the rank has to be less than or equal to minimum of number of rows and
number of columns, 𝑟𝑎𝑛𝑘(𝐴) ≤ 𝑚𝑖𝑛(3,3), which in this case, essentially 3, so the
maximum rank equals 3, so rank can equal to three, but cannot be more than that.
(Refer Slide Time: 22:22)
Now, let us start with the row operations. Now, what we are going to do is we have the
matrix

1 2 7 → 𝑅1
𝐀 = 1 𝒂2 𝒂3 = [−1 1 −1] → 𝑅2 .
[𝒂
̅ ̅ ̅ ]
1 −3 −3 → 𝑅3

The first operation that we are going to do, is to add 𝑅1 and 𝑅2 and replace 𝑅2 with this,
that is, 𝑅2 ← 𝑅1 + 𝑅2 . Essentially, what we are going to see is, we have to essentially make
the leading entries of each row as 0, so essentially that this gives me, what does this give
me. So, you have
 1 2 7
[0 3 6 ].
1 −3 −3

𝑅2
Third row 𝑅3 remains as it is. Now, you replace 𝑅2 by 𝑅2 divide by 3, that is, 𝑅2 ← . So
3

this becomes

1 2 7
[0 1 2]
1 −3 −3

Now, we are essentially going to replace, now this leading entry in 𝑅2 is reduced to 0. Now,
the next row we can target to reduce the leading entry of 𝑅3 to 0. So we can do 𝑅3 ← 𝑅3 −
𝑅1 .

And that essentially means that you are going to have

1 2 7
[0 1 2 ],
0 −5 −10

and now we do 𝑅3 ← 𝑅3 + 5𝑅2 which is essentially becomes

1 2 7
[0 1 2 ],
0 −5 −10

and you can clearly see the entire third row 𝑅3 becomes 0. So, the third row is completely
0 and therefore, you have two nonzero rows essentially, plus a rank equal to 2. Or
essentially and we had seen this before that in fact, there are 3 columns only two of them
are linearly independent. So, maximum number of linearly independent columns, which
will also be equal to maximum number of linearly independent rows, this is equal to 2. We
had seen that earlier.

So, let me write it clearly, maximum number of linearly independent columns which is
equal to maximum number of linearly independent rows and in fact, these leading nonzero
elements of each nonzero row these have an interesting name these are what are known as
the pivots.
So, you essentially have 2 pivots you can also see it in terms of the pivots. So, you have 2
nonzero pivots essentially the rank of this matrix is 2, maximum number of linearly
independent columns is 2, maximum number of linearly independent rows is 2 and when
you do the row elimination, you can see that the number of nonzero rows is also 2.

Similarly, you can go to column elimination also, although row elimination is what is
usually preferred and this is known as the Row Echelon form, the reduced row echelon
form. This is known as the Row Echelon form of the matrix which you get via Gaussian
elimination with the pivots.

So, from the Row Echelon form, the number of nonzero pivots essentially gives you the
rank of the matrix which is a very important property as we go into subsequent modules.
So, let us stop our discussion here. We will continue our discussion on matrices. In fact, as
I told you matrices form a very important component.

In fact, linear algebra is sometimes also known as matrix algebra. So, our discussion of
matrices and various types of matrices, the properties and the various manipulations, the
use for the various uses of these is going to form essentially the bulk of this course. So, let
us stop here and we will continue our discussion in the next month. Thank you very much.