Machine Learning

Chapter 2
MATHEMATICAL FOUNDATION FOR ML
Linear Algebra is an essential field of mathematics, which defines the study of
vectors, matrices, planes, mapping, and lines required for linear transformation.
 Linear Algebra for Machine learning

● In the context of Machine Learning, linear algebra is employed to model

and analyze relationships within data.
● It enables the representation of data points as vectors and allows for
efficient computation of operations on these vectors.
● Linear transformations, matrices, and vector spaces play a significant
role in defining and solving problems in ML.
What is a System of Linear Equations

 A system of linear equations consists of two or more linear equations that share the
same variables.
 A system of linear equations involves multiple linear equations that share common
variables.
 Each equation can represent a line, plane, or higher-dimensional surface,
depending on the number of variables involved.
 The solution to the system is the set of variable values that satisfy all equations
simultaneously, usually corresponding to points of intersection among the lines or
planes represented by these equations.
Solution of a System of Linear Equation
Linear equations can have three kinds of possible solutions:
•No Solution
•Unique Solution
•Infinite Solution
How to Solve System of Linear Equations?
The subsequent techniques for solving the system of linear equations AX = B are viable
solely under the condition that the coefficient matrix A is non-singular, meaning |A| ≠
0.we can then further solve the equation using following methods.
These methods include:
•Cramer’s Rule
•Inverse Method
•Gauss-Jordan Method
•Gauss Elimination Method
•LU Decomposition Method of Factorization (also known as Method of
Triangularization)

We will solve some examples and see

VECTORS:

Definition: A vector is a mathematical object that has both magnitude (or length) and direction.

• The Fig. 2.1.1 shows a vector.

• Two vectors are the same if they have the same magnitude and direction. This means that if you
take a vector and translate it to a new position (without rotating it), then the vector you obtain at
the end of this process is the same vector you had initially.
• Two examples of vectors are those that represent force and velocity. Both force and velocity are
in a particular direction. The magnitude of the vector would indicate the strength of the force, or
the speed associated with the velocity.
• You denote vectors using boldface as in a or b. Especially when writing by hand, where
you cannot easily write in boldface, you denote vectors using arrows as in a or b, or use
other markings.
• The magnitude of the vector a is denoted by ||a||.
• When you want to refer to a number and stress that it is not a vector, you can call the
number a scalar.
• You denote scalars with italics, as in a or b.
• Definition: A vector space is a set of vectors that can be added and multiplied (scaled)
by numbers, known
as scalars.
Solve the problems from book page 2-8
 length of a vector
The length of a vector (commonly known as the magnitude) allows us to quantify the property
of a given vector. To find the length of a vector, simply add the square of its components then
take the square root of the result. In this article, we'll extend our understanding of magnitude
to vectors in three dimensions.

Solve sum from page number 2-3

Solve example from Page number: 2-10
 Eigenvalues and Eigenvectors
● Eigenvalues and Eigenvectors are the scalar and vector quantities associated
with Matrix used for linear transformation.
● The vector that does not change even after applying transformations is
called the Eigenvector and the scalar value attached to Eigenvectors is
called Eigenvalues.
● Eigenvectors are the vectors that are associated with a set of linear
equations.
● For a matrix, eigenvectors are also called characteristic vectors, and we can
find the eigenvector of only square matrices.
● Eigenvectors are very useful in solving various problems of matrices and
differential equations.
What are Eigenvalues?
● Eigenvalues are the scalar values associated with the eigenvectors in linear
transformation.
● The word ‘Eigen’ is of German Origin which means ‘characteristic’. Hence,
these are the characteristic value that indicates the factor by which
eigenvectors are stretched in their direction.
● It doesn’t involve the change in the direction of the vector except when the
eigenvalue is negative.
● When the eigenvalue is negative the direction is just reversed.
● The equation for eigenvalue is given by
where A is the matrix
v is associated
Av = λv eigenvector
λ is scalar eigenvalue
What are Eigenvectors?
● Eigenvectors for square matrices are defined as non-zero vector values
which when multiplied by the square matrices give the scaler multiple of the
vector, i.e. we define an eigenvector for matrix A to be “v” if it specifies the
condition, Av = λv
● The scaler multiple λ in the above case is called the eigenvalue of the square
matrix.
● We always have to find the eigenvalues of the square matrix first before
finding the eigenvectors of the matrix. where,
● The eigenvector equation is, A is the given square matrix
v is the eigenvector of matrix
Av = λv A
λ is any scaler multiple
How to Find an Eigenvector?
Step 1: Find the eigenvalues of the matrix A, using the equation det |(A – λI| =0,
where “I” is the identity matrix of order similar to matrix A

Step 2: The value obtained in Step 2 are named as, λ1, λ2, λ3….

Step 3: Find the eigenvector (X) associated with the eigenvalue λ1 using the
equation, (A – λ1I) X = 0

Step 4: Repeat step 3 to find the eigenvector associated with other remaining
eigenvalues λ2, λ3….
Types of Eigenvector
The eigenvectors calculated for the square matrix are of two types which are,

Right Eigenvector
The eigenvector which is multiplied by the given square matrix from the
right-hand side is called the right eigenvector. It is calculated by using the
following equation,

AVR = λVR

where
A is given square matrix of order n×n
λ is one of the eigenvalues
VR is the column vector matrix
Types of Eigenvector
The eigenvectors calculated for the square matrix are of two types which are,

Left Eigenvector :The eigenvector which is multiplied by the given square

matrix from the left-hand side is called the left eigenvector. It is calculated by
using the following equation,

VLA = VLλ
where
A is given square matrix of order n×n The value of VL is,
λ is one of the eigenvalues
VL is the row vector matrix VL = [v1, v2, v3,…, vn]
Singular Value Decomposition (SVD)
● The Singular Value Decomposition (SVD) of a matrix is a factorization of
that matrix into three matrices.
● It has some interesting algebraic properties and conveys important
geometrical and theoretical insights about linear transformations. It also has
some important applications in data science.
● In this article, I will try to explain the mathematical intuition behind SVD
and its geometrical meaning.

Mathematics behind SVD:

The SVD of mxn matrix A is given by the formula

 Mathematics behind SVD:

The SVD of mxn matrix A is given by the formula

● U: mxm matrix of the orthonormal eigenvectors of .

● VT: transpose of a nxn matrix containing the orthonormal eigenvectors of
● : diagonal matrix with r elements equal to the root of the positive
eigenvalues of AAᵀ or Aᵀ A (both matrics have the same positive
eigenvalues anyway).
Applications

Calculation of Pseudo-inverse:
● Pseudo inverse or Moore-Penrose inverse is the generalization of
the matrix inverse that may not be invertible (such as low-rank
matrices).
● If the matrix is invertible then its inverse will be equal to Pseudo
inverse but pseudo inverse exists for the matrix that is not
invertible.
● It is denoted by A+.
Applications
Calculation of Pseudo-inverse:
● Pseudo inverse or Moore-Penrose inverse is the generalization of the
matrix inverse that may not be invertible (such as low-rank matrices).
● If the matrix is invertible then its inverse will be equal to Pseudo inverse
but pseudo inverse exists for the matrix that is not invertible.
● It is denoted by A+.

The equation gives the pseudo-inverse.