Support Vector Machine

(SVM)
 Introduction
▪ Support Vector Machine (SVM) is one of the most popular
Supervised Learning algorithms, which is used for Classification as
well as Regression problems.

▪ However, primarily, it is used for Classification problems in Machine

Learning.
 Introduction
▪ SVM chooses the extreme points/vectors that help in creating
the hyper-plane.
▪ These extreme cases are called as support vectors, and
hence algorithm is termed as Support Vector Machine.
 Basic Goal
▪ To create the best line or decision boundary that can
segregate n-dimensional space into classes so that we can
easily put the new data point in the correct category in the
future.
▪ This best decision boundary is called a hyperplane.
 Types of SVM
SVM can be of two types:

Linear SVM
▪ Linear SVM is used for linearly separable data, which means if a dataset
can be classified into two classes by using a single straight line, then such
data is termed as linearly separable data, and classifier is used called as
Linear SVM classifier.
Non-linear SVM
▪ Non-Linear SVM is used for non-linearly separated data, which means if a
dataset cannot be classified by using a straight line, then such data is
termed as non-linear data and classifier used is called as Non-linear SVM
classifier.
 Hyper-plane
▪ There can be multiple lines/decision boundaries to segregate the classes in
n-dimensional space, but we need to find out the best decision boundary
that helps to classify the data points. This best boundary is known as the
hyper-plane of SVM.
▪ The dimensions of the hyper-plane depend on the features present in the
dataset, which means if there are 2 features (as shown in image), then
hyper-plane will be a straight line. And if there are 3 features, then hyper-
plane will be a 2-dimension plane. We always create a hyper-plane that has
a maximum margin, which means the maximum distance between the data
points.
 Support Vectors
▪ The data points or vectors that are the closest to the hyper-
plane and which affect the position of the hyper-plane are
termed as Support Vector.

▪ Since these vectors support the hyper-plane, hence called a

Support vector.
 Working of SVM
▪ An SVM model is basically a representation of different
classes in a hyperplane in multidimensional space.

▪ The hyperplane will be generated in an iterative manner by

SVM so that the error can be minimized.

▪ The goal is to divide the datasets into classes to find a

maximum marginal hyperplane (MMH).
 Concept of SVM
The followings are important concepts in SVM −
▪ Support Vectors
▪ Hyper-plane
▪ Margin
▪ Support Vectors − Data points that are closest to the hyper-plane is called support
vectors. Separating line will be defined with the help of these data points.
▪ Hyperplane − As we can see in the above diagram, it is a decision plane or space
which is divided between a set of objects having different classes.
▪ Margin − It may be defined as the gap between two lines on the closet data points
of different classes. It can be calculated as the perpendicular distance from the line
to the support vectors. Large margin is considered as a good margin and small
margin is considered as a bad margin.
▪ The main goal of SVM is to divide the datasets into classes to find a maximum
marginal hyper-plane (MMH) and it can be done in the following two steps −
▪ First, SVM will generate hyper-planes iteratively that segregates the classes
in best way.
▪ Then, it will choose the hyper-plane that separates the classes correctly.
 Concept of SVM
The followings are important concepts in SVM −
▪ Support Vectors
▪ Hyper-plane
▪ Margin
▪ Support Vectors − Data points that are closest to the hyper-plane is called support
vectors. Separating line will be defined with the help of these data points.
▪ Hyperplane − As we can see in the above diagram, it is a decision plane or space
which is divided between a set of objects having different classes.
▪ Margin − It may be defined as the gap between two lines on the closet data points
of different classes. It can be calculated as the perpendicular distance from the line
to the support vectors. Large margin is considered as a good margin and small
margin is considered as a bad margin.
▪ The main goal of SVM is to divide the datasets into classes to find a maximum
marginal hyper-plane (MMH) and it can be done in the following two steps −
▪ First, SVM will generate hyper-planes iteratively that segregates the classes
in best way.
▪ Then, it will choose the hyper-plane that separates the classes correctly.
 Concept of SVM
▪ Hence, the SVM algorithm helps to find the best line or decision boundary; this
best boundary or region is called as a hyperplane.
▪ SVM algorithm finds the closest point of the lines from both the classes. These
points are called support vectors.
▪ The distance between the vectors and the hyperplane is called as margin. And the
goal of SVM is to maximize this margin.
▪ The hyperplane with maximum margin is called the optimal hyperplane.
 Working Non-Linear SVM
▪ If data is linearly arranged, then we can separate it by using a straight line,
but for non-linear data, we cannot draw a single straight line. Consider the
below image:
 Concept of SVM
▪ So to separate these data points, we need to add one more dimension.
▪ For linear data, we have used two dimensions x and y, so for non-linear data, we
will add a third dimension z.
▪ It can be calculated as:
z=x2 +y2

By adding the third dimension, the sample space will become as below image:
 Concept of SVM
▪ So now, SVM will divide the datasets into classes in the following way. Consider
the below image:
▪ Since we are in 3-d Space, hence it is looking like a plane parallel to the x-axis.
▪ If we convert it in 2d space with z=1, then it will become as:
 Concept of SVM
▪ Since we are in 3-d Space, hence it is looking like a plane parallel to the x-axis.
▪ If we convert it in 2d space with z=1, then it will become as:
▪ Hence we get a circumference of radius 1 in case of non-linear data.
SVM Pros/Cons
Pros of SVM
▪ SVM classifiers offers great accuracy and work well with high
dimensional space. SVM classifiers basically use a subset of
training points hence in result uses very less memory.

Cons of SVM classifiers

▪ They have high training time hence in practice not suitable for large
datasets. Another disadvantage is that SVM classifiers do not work
well with overlapping classes.
 How Does Identify Right Hyperplane

▪ Scenario-1 Here we have 3 hyperplanes (A, B, and C). Now

identify the right hyperplane for classify the stars and circles.

A
B

▪ You need to remember the thumb rule to identify the right

hyperplane: select the hyperplane that segregates the two classes
better.
▪ In this scenario hyperplane B perform their job excellently.
 How Does Identify Right Hyperplane

▪ Scenario-2 Here we have 3 hyperplanes (A, B, and C)and all are segregating the
classes well. Now how can we identify the right Hyperplane.

A
B
C

▪ Here, maximizing the distance between nearest data points and Hyperplane will help us to decide
the right Hyperplane.
▪ In above picture margins of Hyperplane C is more than the others. So C has selected as the best
Hyperplane.
 How Does Identify Right Hyperplane

▪ Scenario-3 Here we have 2 hyperplanes (A, and B). Use the rules as discussed in
previous section to identify the right Hyperplane

▪ SVM selects the Hyperplane which classifies the classes accurately prior to maximizing the
margins.
▪ Hyperplane B has a classification error and A has classified all accurately. Therefore right
Hyperplane is A.
How Does Identify Right Hyperplane
Advantages of SVM
Disadvantages of SVM
 How Does Identify Right Hyperplane

▪ Scenario-4 Below we are not able to segregate the two classes using straight line, as
one of the star lies on the territory of circle.

▪ One star at other end is like an outlier for star class. The svm algorithm has the feature of ignoring
the outlier and find the Hyperplane with maximum margins. Hence we can say that the svm
classification is robust to outliers.
 Non-Linear SVM

It is a form of data structure where the data elements don't stay arranged
linearly or sequentially. Since the data structure is non-linear, it does not involve
a single level.
 Kernel Function
In machine learning, a kernel refers to a method that allows us to
apply linear classifiers to non-linear problems by mapping non-linear
data into a higher-dimensional space without the need to visit or
understand that higher-dimensional space.
Kernel Trick
 Types of Kernel Function
▪ Polynomial Kernel
▪ RBF Kernel
▪ Sigmoid Kernel
 Polynomial Kernel
▪ The polynomial kernel is a kernel function commonly used with support
vector machines and other kernelized models, that represents the similarity
of vectors in a feature space over polynomials of the original variables
▪ It is popular in image processing.

K ( xi , x j ) = ( xi .x j + 1)
T d

▪ Where d is the degree of polynomial.

▪ T is the transpose.
 Radial-Basis Kernel Function
▪ It is general purpose kernel: used when their is no prior knowledge about
the data.

 − | x − y |2 
K ( x, y ) = exp  
 2
2

 Hyperbolic Tangent Kernel
▪ Mainly used in neural networks.

K ( xi , x j ) = tanh ( kxi .x j + c )

▪ Where k>0 and c<0

SVM Solved Numerical
SVM Solved Numerical
SVM Solved Numerical
SVM Solved Numerical
SVM Solved Numerical
SVM Solved Numerical