Applied Linear Algebra for Signal Processing, Data Analytics and Machine Learning

Professor Aditya K. Jagannatham

Department of Electrical Engineering
Indian Institute of Technology, Kanpur
Lecture 01
Vector properties: addition, linear combination, inner product, orthogonality, norm

Hello welcome to another module in this massive open online course on Applied Linear Algebra
for Signal Processing, Data Analytics and Machine Learning. So, let us start our lecture, so we are
going to start with the fundamental concepts of vectors and how vectors can be used to represent
data.

(Refer Slide Time: 0:41)

So, this is the massive open online course on Applied Linear Algebra for SP that is Signal
Processing and you have the DA that is your Data Analytics and also you have the ML which is
basically Machine Learning. So, as you can see essentially linear algebra and its applications, the
concepts of linear algebra are so fundamental that they have diverse applications, in fact they are
applied in almost all the fields of science and engineering. And, in particular, we are going to look
at the interesting applications of linear algebra in signal processing, data analysis, data analytics
and machine learning.
(Refer Slide Time: 2:25)
So, the notion of linear algebra, when you talk about the linear algebra the notion or concept of
vectors, so vector or basically 𝑛 dimensional point is fundamental. This is fundamental to represent
̅ as this is, an 𝑛 dimensional vector comprising of the
data. For instance, we can define a vector 𝒖
elements or we say the components 𝑢1 , 𝑢2 ,.., 𝑢𝑛 . This is we say 𝑛 dimensional vector. This is
fundamental to represent the data which is what we are seeing and this belongs to the 𝑛 dimensional
real space.

So, this notion is basically your 𝑛 dimensional real space if 𝑢1 , 𝑢2 , so on up to 𝑢𝑛 if these are real,
else this belongs to the 𝑛 dimensional complex space, so this is 𝑛 dimensional complex space, if
̅ , 𝑢1 , 𝑢2 , 𝑢𝑛 are real then
your 𝑢1 , 𝑢2 , 𝑢𝑛 are, complex. So, you have the 𝑛 dimensional vector 𝒖
this belongs to the 𝑛 dimensional real space if 𝑢1 , 𝑢2 , 𝑢𝑛 are complex this belongs to the 𝑛
dimensional complex space.
(Refer Slide Time: 4:56)

̅ represents, this essentially represents an 𝑛 tuple, or essentially, you can

And you can also say 𝒖
also say this is a point in 𝑛 dimensional space. So, by default we will consider real vectors. For
example, a classic example of that for our course would be an 𝑛 dimensional signal, 𝑛 samples of
a signal, so you take continuous time analog signal, you sample it and you consider 𝑛 samples of
the signal that would be a vector.

(Refer Slide Time: 6:20)

So, you take for instance a classic example, a very useful example for us. So this is your signal
and then you take the 𝑛 samples. So 𝑛 samples of a signal in space, sorry 𝑛 samples of a signal in
time or we can generally say 𝑛 samples of a signal, so that will be our signal vector.

So, for instance you have 𝑛 samples of noise that can be your noise vector. So, vectors have a lot
of applications, of course, you can consider 𝑛 samples of a spatial signal such as an image, so of
course, an image is 2 dimensional but you can also take the samples and put them as a vector. So,
essentially vectors are very useful in representing data signals and various other quantities.

(Refer Slide Time: 7:54)

Therefore, now for instance, you can also have the 𝑛 observations of a sensor, so that can also be
another interesting vector. So, 𝑛 observations of a sensor in time such as temperature pressure. For
instance, let us write this another example, you can have 𝑛 observations of a sensor such as
temperature, pressure, etc so that is your, there are a lot there are basically many many examples
of such vectors.
(Refer Slide Time: 8:41)

Now, for instance a location, 3D space, that can be represented by or you have a location in 3D
space that can be represented by a 3-dimensional vector. Any location in 3D space that can be
represented by the 3-dimensional vector 𝑢1 , 𝑢2 , 𝑢3 or what we call as the 𝑥 𝑦 𝑧 coordinates. So,
this is basically a 3-dimensional vector, this is the physical space that we are used to, this is your
location in 3-dimensional space.

(Refer Slide Time: 9:31)

Now, consider 2 𝑛 dimensional vectors, now let us look at vector addition. Consider 2 𝑛
̅ equals 𝑢1 , 𝑢2 , . . , 𝑢𝑛 and we have 𝒗
dimensional vectors that is we have, let us say, 𝒖 ̅ equals
𝑣1 , 𝑣2 , … , 𝑣𝑛 then the vector addition is simply an element-by-element addition of these 2 vectors.

̅ plus 𝒗
So, essentially 𝒖 ̅ will also be an 𝑛 dimensional vector with the first element 𝑢1 plus 𝑣1 ,
second element 𝑢2 plus 𝑣2 , so on, 𝑛th element 𝑢𝑛 plus 𝑣𝑛 .
(Refer Slide Time: 11:09)

̅ plus 𝒗
And let us now consider the scalar product, let us consider the scalar product you have 𝒖 ̅
̅ you take the scalar multiplied
or you have 𝑘 that is a scalar, so 𝑘 is a scalar coefficient, so 𝑘 times 𝒖
by each element of the vector, so this will be 𝑘𝑢1 , 𝑘𝑢2 so on up to 𝑘𝑢𝑛 . And then we have the
notion of a linear, so this basically completes the basic operations that is your what we call as your
vector addition and the scalar product. And now let us look at another fundamental operation of
vectors that is basically a linear combination.
(Refer Slide Time: 12:22)
So, let us now look at a linear combination, let us now look at a linear combination of vectors, so
̅1, 𝒖
you have the vectors 𝒖 ̅ 2 ,.., 𝒖
̅ 𝑚 . So, consider the vectors 𝒖
̅1, 𝒖
̅ 2 ,.., 𝒖
̅ 𝑚 , these are the vectors
and then you have the scalar quantities 𝑘1 , 𝑘2 , … , 𝑘𝑚 these are your scalars. Then your linear
combination of vectors, the linear combination is basically given as multiply each scalar
coefficient by the corresponding vector.

So, this is essentially the concept of the linear combination of the vectors. Then we have the notion
of an inner product, so that is basically and this is a very important concept the linear combination
of the vectors.
(Refer Slide Time: 14:42)

Now, let us consider another concept that is the inner product, what about the inner product and
this is another fundamental concept that is the inner product when you have 2 𝑛 dimensional
̅,
vectors what happens to the inner product of these 2 vectors. So, consider arbitrary vectors 𝒖
̅ that belong to the 𝑛 dimensional space of real vectors that is you have 𝒖
comma 𝒗 ̅ equals 𝑢1 , 𝑢2 ,
̅ equals 𝑣1 , 𝑣2 , up to 𝑣𝑛 .
up to 𝑢𝑛 , and 𝒗
(Refer Slide Time: 16:02)
 ̅ 𝑇 , this becomes a row vector 𝒖
Then we have 𝒖 ̅ 𝑇 when you take the transpose of a column vector
̅ 𝑇 and then so this is 𝒖
that becomes a row vector. So this is 𝒖 ̅ 𝑇 is essentially a, this is essentially a
row vector and then we define the inner product, this is a very important definition, we define the
̅, 𝒗
inner product of 𝒖 ̅𝑇 𝒗
̅, we define it as 𝒖 ̅ which is essentially if you look at this, this is the product
of the row vector 𝑢1 , 𝑢2 , … , 𝑢𝑛 times the column vectors 𝑣1 , 𝑣2 , … , 𝑣𝑛 which is equal to summation
𝑖 equal to 1 to n, I can write it as 𝑢𝑖 𝑣𝑖 or essentially this is 𝑢1 , 𝑣1 , write it in the expanded form
𝑢1 𝑣1 plus 𝑢2 𝑣2 , plus 𝑢𝑛 𝑣𝑛 so this is basically the notion of the inner product.

(Refer Slide Time: 17:54)

So, this is the inner product, this is a very important concept, the concept of an inner product
̅ equal to 2 1 minus 1 and 𝒗
between 2 vectors. For example, let us take these 2 vectors, example 𝒖 ̅
̅, 𝒗
equal to 1 minus 1 3 and then what you can say is you have 𝒖 ̅𝑇 𝒗
̅ the inner product is 𝒖 ̅ which is
basically equal to 2 1 minus 1 and 1 minus 1 3 which is essentially equal to 2 minus 1 minus 3
which is equal to minus 2, so that is a simple example for the inner product which is essentially
you can also think of it as an element wise multiplication of 2 vectors and then an addition that is
𝑢1 𝑣1 plus 𝑢2 𝑣2 so on until 𝑢𝑛 𝑣𝑛 .

(Refer Slide Time: 19:10)

 ̅ , the inner product of 𝒖
Now, for complex vectors the definition becomes we have 𝒖 ̅, 𝒗
̅. We will
̅𝐻 𝒗
define this as 𝒖 ̅ that is essentially you have to take the transpose that is when you look at the
Hermitian you essentially have to take the transpose and also the complex conjugate 𝑢1∗ , 𝑢2∗ , … , 𝑢𝑛∗
times 𝑣1 , 𝑣2 , … , 𝑣𝑛 that is sigma equal to 1 to 𝑛 that is 𝑢𝑖∗ 𝑣𝑖 , so this is the notion of inner product
for complex vectors 𝑢1∗ to 𝑣1 plus 𝑢2∗ 𝑣2 plus 𝑢𝑛∗ 𝑣𝑛 .

(Refer Slide Time: 20:24)

Further now there is a notion of orthogonality which is very important, the notion of orthogonality
of 2 vectors, 2 vectors are, and this is a very important property 2 vectors are orthogonal if their
inner product is 0, so orthogonally of 2 vectors are orthogonal or perpendicular, orthogonal, I am
sorry orthogonal if their inner product equal to 0.

̅𝑇 𝒗
So, the 2 vectors are orthogonal that is in real dimensions we have 𝒖 ̅ equal to 0 that is for the 𝑟
̅𝐻 𝒗
dimensional space, for the 𝑛 dimensional real space, for the complex space we have 𝒖 ̅ equal to
0, so this is essentially the definition of orthogonality.
(Refer Slide Time: 22:00)

̅ = [1, −1,1, −1] and 𝒗

Let us take a look another again at another simple example 𝒖 ̅=
̅𝑇 𝒗
[2,2, −2, −2] and we have 𝒖 ̅𝑇 𝒗
̅ if we do 𝒖 ̅ this will be 2 − 2 − 2 + 2, which is equal to 0.

So, these 2 vectors that you have over here, so these 2 vectors are orthogonal, essentially what we
also say informally as that as these 2 vectors are perpendicular and that makes more sense in 3-
dimensional space that is we have 2 vectors which are at angle with respect to each other, more
generally the notion is these 2 vectors are orthogonal because their inner product is 0. Let us look
at another interesting example and this time with respect to complex signals and complex
sinusoids.
(Refer Slide Time: 23:39)

So, another very important example I would also say consider the complex sinusoids for instance
𝑗2𝜋 𝑗4𝜋 𝑗(𝑁−1)𝜋
̅ , let us call this as your 1, 𝑒
you have the sinusoids 𝒖 𝑁 ,𝑒 𝑁 so on and so forth 𝑒 𝑁 , so this is
𝑗3𝜋 𝑗6𝜋
your first sinusoid. And then you have another which is essentially 1, 𝑒 𝑁 ,𝑒 𝑁 and so on and so
𝑗4(𝑁−1)𝜋
forth 𝑒 𝑁 .

So, you can see these are essentially 2 complex sinusoids, I am sorry let me make this again, this
𝑗4𝜋 𝑗8𝜋
has to be 𝑒 𝑁 ,𝑒 𝑁 and essentially what you have, yeah, essentially these are 2 complex sinusoids,
these are 2 complex sinusoids and if you look at their inner product so these are 2 complex
sinusoids corresponding to the frequencies if you look at the frequency of this that is 𝑓1 this will
be equal to 1 over 𝑛 and this will be a complex sinusoid of frequency 𝑓2 this will be equal to 2
over 𝑛, so you are talking about the complex sinusoid 𝑒 𝑗2𝜋𝑓1 𝑛 and 𝑒 𝑗2𝜋𝑓2𝑛 .

(Refer Slide Time: 27:16)

 ̅𝐻 𝒗
And when you look at their inner product, their inner product 𝒖 ̅ is basically if you look at this
𝑗2𝜋𝑙 𝑗4𝜋𝑙
−
that will be the summation 𝑙 equal to the ∑𝑁−1
𝑙=0 𝑒 𝑁 𝑒 𝑁 . And therefore, this is going to be the
𝑗2𝜋𝑁
𝑗2𝜋𝑙
1−𝑒 𝑁
∑𝑁−1
𝑙=0 𝑒 𝑁 which is essentially 𝑗2𝜋 .
1−𝑒 𝑁

𝑗2𝜋𝑁
1−1
And if you look at this 𝑒 𝑁 is nothing but 𝑒 𝑗2 𝜋 which is 1. so this is 𝑗2𝜋 . So, therefore, you
1−𝑒 𝑁

have this interesting property where you have the complex sinusoids at the frequency
1 2 𝑁−1
0, 𝑁 , 𝑁 , … , , these are essentially orthogonal.
𝑁

(Refer Slide Time: 29:09)

So, we have the interesting property complex sinusoids at frequencies, so you have complex
1
sinusoids at the frequencies, at the frequency 0, 𝑁, so these frequencies, these are orthogonal,

complex sinusoids edges and this property, this orthogonality of this complex sinusoids, this
property is extremely important in Fourier analysis, this forms the basis of all Fourier analysis.

So it is going to be very important. Naturally it is important for all of signal processing because
remember the Fourier transform or the spectrum of a signal, the decomposition of the signal is
very important in signal processing and in general, of course, naturally when you apply to images
and audio signals and so on, it becomes also very important in machine learning and also naturally
in data analytics, all of these are interrelated.
(Refer Slide Time: 31:10)

Let us now go to this notion of the norm of a vector, so what we mean by the norm of a vector?
̅ ||, which is nothing but the square root of the inner
So, the norm of a vector this is denoted by ||𝒖
̅ with itself which is essentially for a real vector, you can clearly see the inner
product of a vector 𝒖
̅ inner product with itself this is 𝑢12 , 𝑢22 , … , 𝑢𝑛2 , which is
̅ with itself for a real vector 𝒖
product of 𝒖
̅𝑇 𝒖
nothing but 𝒖 ̅.

(Refer Slide Time: 32:30)

 ̅ || or what you also call as 𝑙2 norm of 𝒖
And therefore, this ||𝒖 ̅ which is what we are going to use

by default is going to be √𝑢12 + 𝑢22 +. . +𝑢𝑛2 . And this is also what we call as the length of a vector
in 3-dimensional space or in general length, this is essentially the length of a vector.

And for complex vectors, so this is for real vectors where 𝑢1 , 𝑢2 , … , 𝑢𝑛 are real, for complex
vectors this will be simply the magnitude, this is simply the √|𝑢|12 + |𝑢|22 +. . +|𝑢|2𝑛 . And so, this
is for a complex vector, this definition is for a complex vector.
(Refer Slide Time: 34:04)

2
̅ || that is ||𝒖
And it is very easy to see ||𝒖 ̅ || is greater than equal to 0 and this equal to 0, ||𝒖
̅ ||
̅ equal to zero that is the norm or length
equal to 0 if and only if this is implies and is implied by 𝒖
̅ equal to 0 which means every 𝑢𝑖 which means essentially every 𝑢𝑖
of a vector equal to 0 only if 𝒖
equal to 0.

So, this is essentially the norm of a vector which is defined as √𝑢12 + 𝑢22 +. . +𝑢𝑛2 for a real vector

and √|𝑢|12 + |𝑢|22 +. . +|𝑢|2𝑛 for a complex vector. And this is always greater than equal to 0 except
̅ which is identically equal to 0, that is 𝑢1 , 𝑢2 , … , 𝑢𝑛 all the elements are 0. So, let us
for a vector 𝒖
stop this module here and let us continue in the next module with further discussion on these
various concepts of applied linear algebra. Thank you very much.