Upto 75% Off

Courses TutorialsData SciencePractice

Courses upto 75% Off Python for Machine Learning Machine Learning with R Machine Learning Algorithms

Linear Algebra Operations For Machine

Learning
Last Updated : 06 Jun, 2024
Linear algebra is the backbone of many machine learning algorithms and
techniques. Understanding the fundamental operations of linear algebra is
crucial for anyone aspiring to delve deep into the world of machine learning.
At its core, linear algebra provides a framework for handling and
manipulating data, which is often represented as vectors and matrices.
These mathematical constructs enable efficient computation and provide
insights into the underlying patterns and structures within the data.

In machine learning, linear algebra operations are used extensively in

various stages, from data preprocessing to model training and evaluation.
For instance, operations such as matrix multiplication, eigenvalue
decomposition, and singular value decomposition are pivotal in
dimensionality reduction techniques like Principal Component Analysis
(PCA). Similarly, the concepts of vector spaces and linear transformations
are integral to understanding neural networks and optimization algorithms.

Linear Algebra in Machine learning

This article aims to explore the key linear algebra operations that form the
foundation of many machine learning models. By grasping these operations,
you will gain a deeper appreciation of the mathematical elegance and
computational power that drive machine learning innovations. Whether you
are a beginner or an experienced practitioner, a solid understanding of linear
algebra will enhance your ability to develop and implement more effective
machine learning solutions.

Table of Content
Basics of Linear Algebra
Linear Transformations
Matrix Operations
Eigenvalues and Eigenvectors
Solving Linear Systems
Applications of Linear Algebra in Machine Learning

Basics of Linear Algebra

Linear algebra serves as the backbone of machine learning, providing the
mathematical foundation for understanding and implementing various
algorithms. In this section, we delve into the fundamental concepts of linear
algebra, including vectors, matrices, and scalar operations.

A. Definition of Linear Algebra

Linear algebra is the branch of mathematics that deals with vector spaces
and linear mappings between these spaces. It encompasses the study of
vectors, matrices, linear equations, and their properties.

B. Fundamental Concepts

1. Vectors
Vectors are quantities that have both magnitude and direction, often
represented as arrows in space.
2
v = −1​ ​ ​

2. Matrices
Matrices are rectangular arrays of numbers, arranged in rows and
columns.
Matrices are used to represent linear transformations, systems of linear
equations, and data transformations in machine learning.
 1 2 3
Example: 4 ​ ​ 5 ​ 6 u = [3, 4] v = [-1, 2]
​

7 8 9

3. Scalars
Scalars are single numerical values, without direction, magnitude only.
Scalars are used to scale vectors or matrices through operations like
multiplication.
2
Example: Let’s consider a scalar, k= 3, and a vector [v = −1 ] ​ ​ ​

4
Scalar multiplication involves multiplying each component of the vector
by the scalar. So, if we multiply the vector v by the scalar k=3 we get:
2 3⋅2 6
k ⋅ v = 3 ⋅ −1 = 3 ⋅ (−1) = −3
​ ​ ​ ​ ​ ​ ​ ​

4 3⋅4 12

C. Operations in Linear Algebra

1. Addition and Subtraction

Addition and subtraction of vectors or matrices involve adding or
subtracting corresponding elements.
2 3
Example: [u = −1 , ​ ​ ​ v= 0 ]
​ ​ ​

4 −2

2 3 2+3 5
addition: [u + v = −1 + ​ ​ ​ 0 = −1 + 0 = −1 ]
​ ​ ​ ​ ​ ​ ​ ​ ​

4 −2 4 + (−2) 2

2 3 2−3 −1
subtraction: [u–v = −1 – 0 = −1 − 0 = −1 ] ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

4 −2 4 − (−2) 6

2. Scalar Multiplication
Scalar multiplication involves multiplying each element of a vector or
matrix by a scalar.
2
Example: Consider the scalar k=3 and a vector v = −1 ​ ​ ​

4
scalar multiplication involves multiplying each component of the vector
by the scalar. So, if we multiply the vector v by the scalar k=3 , we get :
2 3⋅2 6
k ⋅ v = 3 ⋅ −1 = 3 ⋅ (−1) = −3
​ ​ ​ ​ ​ ​ ​ ​

4 3⋅4 12

3. Dot Product (Vector Multiplication)

 The dot product of two vectors measures the similarity of their directions.
It is computed by multiplying corresponding elements of two vectors and
summing the results.
Example: For example, given two vectors(u = [u1 , u2 , u3 ] and v = [v1 , v2 , v3 ]), ​ ​ ​ ​ ​

their dot product is calculated as:

u ⋅ v = u1 ⋅ v1 + u2 ⋅ v2 + u3 ⋅ v3
​ ​ ​ ​ ​ ​

4. Cross Product (Vector Multiplication)

The cross product of two vectors in three-dimensional space produces a
vector orthogonal to the plane containing the original vectors.
It is used less frequently in machine learning compared to the dot
product.
Example: Given two vectors u and v, their cross product u×v is calculated
as:
u1 v1 ​ ​ u2 v3 –u3 v2
​ ​ ​ ​

u × v = u2 × v2​ ​ ​ ​ ​ ​ ​ ​ = u3 v1 –u1 v3
​ ​ ​ ​ ​ ​ ​

u3 v3 ​ ​ u1 v2 –u2 v1
​ ​ ​ ​

Linear Transformations
Linear transformations are fundamental operations in linear algebra that
involve the transformation of vectors and matrices while preserving certain
properties such as linearity and proportionality. In the context of machine
learning, linear transformations play a crucial role in data preprocessing,
feature engineering, and model training. In this section, we explore the
definition, types, and applications of linear transformations.

A. Definition and Explanation

Linear transformations are functions that map vectors from one vector space
to another in a linear manner. Formally, a transformation TTT is considered
linear if it satisfies two properties:

1. Additivity: T(u+v)=T(u)+T(v) for all vectors u and v.

2. Homogeneity: T(kv)=kT(v) for all vectors v and scalars k.

Linear transformations can be represented by matrices, and their properties

are closely related to the properties of matrices.

B. Common Linear Transformations in Machine Learning

1. Translation:
Translation involves shifting the position of vectors without changing
their orientation or magnitude.
In machine learning, translation is commonly used for data
normalization and centering, where the mean of the data is subtracted
from each data point.

2. Scaling:
Scaling involves stretching or compressing vectors along each
dimension.
Scaling is frequently applied in feature scaling, where features are
scaled to have similar ranges to prevent dominance of certain features
in machine learning models.

3. Rotation:
Rotation involves rotating vectors around an axis or point in space.
While less common in basic machine learning algorithms, rotation can
be useful in advanced applications such as computer vision and
robotics.

Matrix Operations
Matrix operations form the cornerstone of linear algebra, providing essential
tools for manipulating and analyzing data in machine learning. In this
section, we explore key matrix operations, including multiplication,
transpose, inverse, and determinant, along with their significance and
applications.

A. Matrix Multiplication

Matrix multiplication is a fundamental operation in linear algebra, involving

the multiplication of two matrices to produce a new matrix. The resulting
matrix’s dimensions are determined by the number of rows in the first matrix
and the number of columns in the second matrix.

Definition: Given two matrices A and B, the product matrix C=A⋅B is

computed by taking the dot product of each row of matrix A with each
column of matrix B.
Significance: Matrix multiplication is widely used in machine learning for
various tasks, including transformation of feature vectors, computation of
 model parameters, and neural network operations such as feedforward
and backpropagation.
2 1 3 0
Example: [A = [ ] and B = [ ]]
1 2 1 2
​ ​ ​ ​

to multiply matrices (A )and(B) we have to perform row-column

multiplication. the element at the row (i) and the column (j) of the
resulting matrix (c)is calculated by taking the dot product of the (i)th row
of matrix (A)and the (j)th column of matrix (B).

for example , the element c11 of matrix (c) is calculated as:

​

[c12 = a11 ⋅ b12 + a12 ⋅ b22 = 2 ⋅ 0 + 1 ⋅ 2 = 2]

​ ​ ​ ​ ​

Following this pattern, we can calculate all elements of matrix (C):

7 2
[C = [ ]]
5 4
​ ​

So, the result of the matrix multiplication (A × B) is (c).

B. Transpose and Inverse of Matrices

1. Transpose:
The transpose of a matrix involves flipping its rows and columns,
resulting in a new matrix where the rows become columns and vice
versa.
It is denoted by AT, and its dimensions are the reverse of the original
matrix.
Transpose is used in applications such as solving systems of linear
equations, computing matrix derivatives, and performing matrix
factorization.

2. Inverse:
The inverse of a square matrix A is another matrix denoted by A−1
such that A⋅A−1=A−1⋅A= I ,where I is the identity matrix.
Not all matrices have inverses, and square matrices with a determinant
not equal to zero are invertible.
Inverse matrices are used in solving systems of linear equations,
computing solutions to optimization problems, and performing
 transformations.

C. Determinants

The determinant of a square matrix is a scalar value that encodes various

properties of the matrix, such as its volume, orientation, and invertibility.

Significance: The determinant is used to determine whether a matrix is

invertible, calculate the volume of parallelepiped spanned by vectors, and
analyze the stability of numerical algorithms.
Properties: The determinant satisfies several properties, including
linearity, multiplicativity, and the property that a matrix is invertible if and
only if its determinant is non-zero.

Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are fundamental concepts in linear algebra
that play a significant role in machine learning algorithms and applications.
In this section, we explore the definition, significance, and applications of
eigenvalues and eigenvectors.

A. Definition and Significance

1. Eigenvalues:
Eigenvalues of a square matrix AAA are scalar values that represent
how a transformation represented by AAA stretches or compresses
vectors in certain directions.
Eigenvalues quantify the scale of transformation along the
corresponding eigenvectors and are crucial for understanding the
behavior of linear transformations.
Example: Consider the matrix:
2 1
[A = [ ]]
1 2
​ ​

To find the eigenvalues λ , we solve the characteristic equation:

[det(A–λI) = 0]
Substituting the values:
2−λ 1
[det ([ ]) = 0]
1 2−λ
​ ​

This simplifies to:

[(2 − λ)2 –1 = 0]
Solving this, we find (λ1 = 1) and (λ2 = 3)
​ ​

[ ]
 1
for (λ1 = 1), solving ((A–λ1 I )v1 = 0) , we find the eigenvector (v1 = [ ])
−1
​ ​ ​ ​ ​

1
for (λ2 = 3), solving ((A–λ2 I)v2 = 0) , we find the eigenvector (v2 = [ ])
1
​ ​ ​ ​ ​

2. Eigenvectors:
Eigenvectors are non-zero vectors that are transformed by a matrix
only by a scalar factor, known as the eigenvalue.
They represent the directions in which a linear transformation
represented by a matrix stretches or compresses space.
Eigenvectors corresponding to distinct eigenvalues are linearly
independent and form a basis for the vector space.

B. Applications in Machine Learning

1. Dimensionality Reduction:
Techniques such as Principal Component Analysis (PCA) utilize
eigenvalues and eigenvectors to identify the principal components
(directions of maximum variance) in high-dimensional data and project
it onto a lower-dimensional subspace.
Eigenvalues represent the amount of variance explained by each
principal component, allowing for effective dimensionality reduction
while preserving as much information as possible.

2. Graph-based Algorithms:
Eigenvalues and eigenvectors play a crucial role in graph-based
algorithms such as spectral clustering and PageRank.
In spectral clustering, eigenvalues and eigenvectors of the graph
Laplacian matrix are used to partition data into clusters based on
spectral properties.

3. Matrix Factorization:
Techniques like Singular Value Decomposition (SVD) and Non-
negative Matrix Factorization (NMF) rely on eigenvalue decomposition
to factorize matrices into lower-dimensional representations.
Eigenvalue decomposition facilitates the extraction of meaningful
features or components from high-dimensional data matrices, enabling
efficient data representation and analysis.

C. Eigen Decomposition
Eigen decomposition is the process of decomposing a square matrix into its
eigenvalues and eigenvectors. It is expressed as A = QΛQ−1 , where Q is a
matrix whose columns are the eigenvectors of A, and λ is a diagonal matrix
containing the corresponding eigenvalues.

Eigen decomposition provides insights into the structure and behavior of

linear transformations, facilitating various matrix operations and
applications in machine learning.

Solving Linear Systems

Linear systems of equations arise frequently in machine learning tasks, such
as parameter estimation, model fitting, and optimization. In this section, we
explore methods for solving linear systems, including Gaussian elimination,
LU decomposition, and QR decomposition, along with their significance and
applications.

A. Gaussian Elimination

Gaussian elimination is a fundamental method for solving systems of linear

equations by transforming the augmented matrix into row-echelon form
through a sequence of elementary row operations. It involves three main
steps:

1. Forward Elimination:
Reduce the augmented matrix to upper triangular form by eliminating
variables below the main diagonal using row operations.

2. Back Substitution:
Solve for the variables starting from the last equation and working
upward, substituting known values to solve for unknowns.

3. Pivoting:
To improve numerical stability and avoid division by zero, pivoting may
be applied to interchange rows and ensure non-zero pivot elements.

B. LU Decomposition

LU decomposition, also known as LU factorization, decomposes a square

matrix into the product of a lower triangular matrix (L) and an upper
triangular matrix (U). It simplifies the process of solving linear systems and
computing determinants. The steps involved in LU decomposition are:

1. Decomposition:
Decompose the original matrix A into the product of lower triangular
matrix L and upper triangular matrix U such that A = LU Forward and
Back Substitution:
Use the LU decomposition to solve linear systems more
efficiently by performing forward and back substitution steps.

C. QR Decomposition

QR decomposition decomposes a matrix into the product of an orthogonal

matrix (Q) and an upper triangular matrix (R). It is particularly useful for
solving least squares problems and computing eigenvalues. The steps
involved in QR decomposition are:

1. Factorization:
Factorize the original matrix AAA into the product of orthogonal matrix
Q and upper triangular matrix R, such that A = QRA=QR.XTXβ = XT Y

2. Orthogonalization:
Orthogonalize the columns of A to obtain the orthogonal matrix Q
using techniques such as Gram-Schmidt orthogonalization. XTXβ = XTY

Applications of Linear Algebra in Machine Learning

Linear algebra serves as the backbone of many machine learning algorithms,
providing powerful tools for data manipulation, model representation, and
optimization. In this section, we explore some of the key applications of
linear algebra in machine learning, including principal component analysis
(PCA), singular value decomposition (SVD), linear regression, support vector
machines (SVM), and neural networks.

A. Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is a dimensionality reduction technique

that utilizes linear algebra to identify the principal components in high-
dimensional data. The main steps of PCA involve:

1. Covariance Matrix Calculation:

 Compute the covariance matrix of the data to understand the
relationships between different features.
2. Eigenvalue Decomposition:
Decompose the covariance matrix into its eigenvalues and
eigenvectors to identify the principal components.

3. Projection onto Principal Components:

Project the original data onto the principal components to reduce the
dimensionality while preserving the maximum variance.

B. Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) is a matrix factorization technique

widely used in machine learning for dimensionality reduction, data
compression, and noise reduction. The key steps of SVD include:

1. Decomposition:
Decompose the original matrix into the product of three matrices: A =
U ΣV T where U and V are orthogonal matrices, and σ is a diagonal
matrix of singular values.

2. Dimensionality Reduction:
Retain only the most significant singular values and their
corresponding columns of U and V to reduce the dimensionality of the
data.

C. Linear Regression

Linear regression is a supervised learning algorithm used for modeling the

relationship between a dependent variable and one or more independent
variables. Linear algebra plays a crucial role in solving the linear regression
problem efficiently through techniques such as:

1. Matrix Formulation:
Representing the linear regression problem in matrix formY = Xβ + ϵ
where Y is the dependent variable, X is the matrix of independent
variables, β is the vector of coefficients, and ϵ\epsilonϵ is the error term.

2. Normal Equation:
 Solving the normal equation X T Xβ = X T Y using linear algebra to
obtain the optimal coefficients β.

D. Support Vector Machines (SVM)

Support Vector Machines (SVM) are powerful supervised learning models

used for classification and regression tasks. Linear algebra plays a crucial
role in SVMs through:

1. Kernel Trick:
Utilizing linear algebraic operations efficiently in kernelized SVMs to
map data into higher-dimensional feature spaces for nonlinear
classification.

2. Optimization:
Formulating the SVM optimization problem as a quadratic
programming problem and solving it efficiently using linear algebraic
techniques such as convex optimization and quadratic programming
solvers.

E. Neural Networks

Neural networks, especially deep learning models, heavily rely on linear

algebra for model representation, parameter optimization, and
forward/backward propagation. Key linear algebraic operations in neural
networks include:

1. Matrix Multiplication:
Performing matrix multiplication operations between input features
and weight matrices in different layers of the neural network during
the forward pass.

2. Gradient Descent:
Computing gradients efficiently using backpropagation and updating
network parameters using gradient descent optimization algorithms,
which involve various linear algebraic operations.

3. Weight Initialization:
Initializing network weights using techniques such as Xavier
initialization and He initialization, which rely on linear algebraic
 properties for proper scaling of weight matrices.

Conclusion
Linear algebra underpins machine learning, providing crucial tools for data
manipulation and algorithmic implementation. From basic concepts like
vectors and matrices to advanced techniques such as eigenvalue
decomposition and singular value decomposition, it forms the foundation of
various algorithms. Applications include dimensionality reduction, regression
analysis, classification, and neural network training. Mastering linear algebra
is essential for excelling in machine learning and driving innovation in
artificial intelligence. Its importance will continue to grow as the field
evolves, making it a vital area of study for aspiring data scientists and
machine learning engineers.

Linear Algebra In Machine Learning – FAQs

What is the importance of linear algebra in machine learning?

Linear algebra forms the foundation of many machine learning

algorithms and techniques. It is used extensively for data
preprocessing, feature engineering, model training, and optimization.

What are matrices and vectors in the context of machine learning?

Matrices and vectors are fundamental data structures in machine

learning. A vector represents a one-dimensional array of numbers,
while a matrix represents a two-dimensional array of numbers. In
machine learning, vectors are commonly used to represent features or
data points, while matrices are used to represent datasets or
transformations.

How are matrices and vectors used in machine learning algorithms?

 Matrices and vectors are used to represent data, parameters, and
operations in machine learning algorithms. For example, datasets are
represented as matrices where each row corresponds to a data point
and each column corresponds to a feature. Model parameters, such as
weights and biases, are represented as vectors. Linear transformations,
such as matrix multiplication, are used in various machine learning
operations, including feature scaling, dimensionality reduction, and
neural network computations.

What are eigenvalues and eigenvectors, and how are they used in
machine learning?

Eigenvalues and eigenvectors are important concepts in linear algebra

that have various applications in machine learning. Eigenvalues
represent the scaling factor of eigenvectors in linear transformations.
In machine learning, they are used in techniques such as principal
component analysis (PCA) for dimensionality reduction and spectral
clustering for clustering algorithms.

How does linear algebra contribute to optimization in machine learning?

Optimization is a key component of many machine learning algorithms,

where the goal is to find the optimal parameters that minimize a cost
function. Linear algebra provides the mathematical framework for
optimization techniques such as gradient descent, which involves
computing gradients (derivatives) of the cost function with respect to
the model parameters. These gradients are typically represented as
vectors and matrices, and linear algebra operations are used to update
the model parameters iteratively until convergence.
 Never before seen offer on GeeksforGeeks Courses! Get up-to 75% off on all
premium courses such as Full Stack Web Dev, Backend Dev, DSA for Interview and
more. Avail discount now!

ritura… 10

Next Article
Best Python libraries for Machine
Learning

Similar Reads
Locally Linear Embedding in machine learning
LLE(Locally Linear Embedding) is an unsupervised approach designed to
transform data from its original high-dimensional space into a lower-…
8 min read

Linear Regression in Machine learning

Machine Learning is a branch of Artificial intelligence that focuses on the
development of algorithms and statistical models that can learn from and…
15+ min read

Linear Discriminant Analysis in Machine Learning

As we know that while dealing with a high dimensional dataset then we must
apply some dimensionality reduction techniques to the data at hand so, that…
8 min read

Loss function for Linear regression in Machine Learning

The loss function quantifies the disparity between the prediction value and the
actual value. In the case of linear regression, the aim is to fit a linear equation…
7 min read

Support vector machine in Machine Learning

In this article, we are going to discuss the support vector machine in machine
learning. We will also cover the advantages and disadvantages and…
9 min read
 View More Articles

Article Tags : Technical Scripter 2019 Data Science Math Machine Learning

Practice Tags : Machine Learning

A-143, 9th Floor, Sovereign Corporate

Tower, Sector-136, Noida, Uttar Pradesh -
201305

Company Explore
About Us Job-A-Thon Hiring Challenge
Legal Hack-A-Thon
Careers GfG Weekly Contest
In Media Offline Classes (Delhi/NCR)
Contact Us DSA in JAVA/C++
Advertise with us Master System Design
GFG Corporate Solution Master CP
Placement Training Program GeeksforGeeks Videos
Geeks Community

Languages DSA
Python Data Structures
Java Algorithms
C++ DSA for Beginners
PHP Basic DSA Problems
 GoLang DSA Roadmap
SQL DSA Interview Questions
R Language Competitive Programming
Android Tutorial

Data Science & ML Web Technologies

Data Science With Python HTML
Data Science For Beginner CSS
Machine Learning Tutorial JavaScript
ML Maths TypeScript
Data Visualisation Tutorial ReactJS
Pandas Tutorial NextJS
NumPy Tutorial NodeJs
NLP Tutorial Bootstrap
Deep Learning Tutorial Tailwind CSS

Python Tutorial Computer Science

Python Programming Examples GATE CS Notes
Django Tutorial Operating Systems
Python Projects Computer Network
Python Tkinter Database Management System
Web Scraping Software Engineering
OpenCV Tutorial Digital Logic Design
Python Interview Question Engineering Maths

DevOps System Design

Git High Level Design
AWS Low Level Design
Docker UML Diagrams
Kubernetes Interview Guide
Azure Design Patterns
GCP OOAD
DevOps Roadmap System Design Bootcamp
Interview Questions

School Subjects Commerce

Mathematics Accountancy
Physics Business Studies
Chemistry Economics
Biology Management
Social Science HR Management
English Grammar Finance
Income Tax
 Databases Preparation Corner
SQL Company-Wise Recruitment Process
MYSQL Resume Templates
PostgreSQL Aptitude Preparation
PL/SQL Puzzles
MongoDB Company-Wise Preparation
Companies
Colleges

Competitive Exams More Tutorials

JEE Advanced Software Development
UGC NET Software Testing
UPSC Product Management
SSC CGL Project Management
SBI PO Linux
SBI Clerk Excel
IBPS PO All Cheat Sheets
IBPS Clerk

Free Online Tools Write & Earn

Typing Test Write an Article
Image Editor Improve an Article
Code Formatters Pick Topics to Write
Code Converters Share your Experiences
Currency Converter Internships
Random Number Generator
Random Password Generator

@GeeksforGeeks, Sanchhaya Education Private Limited, All rights reserved