SUPPORT VECTOR MACHINE (SVM)

Presented by

M. TAYYAB YAQUB
21107004-003
KHALID MAHMOOD
21107004-004
Presented to
MS Electrical Engineering
Sir Dr. Adven

UNIVERSITY OF SIALKOT
 What is Support Vector Machin (SVM)
A support vector machine (SVM) is a type of deep learning algorithm that
performs supervised learning for classification or regression of data groups.
However,  it is mostly used in classification problems. 

Types of Support Vector Machin (SVM)

Linear SVM: Linear SVM is used for linearly separable data, which means if a dataset can be
classified into two classes by using a single straight line, then such data is termed as linearly
separable data, and classifier is used called as Linear SVM classifier.
Non-linear SVM: Non-Linear SVM is used for non-linearly separable data, which means if a
dataset cannot be classified by using a straight line, then such data is termed as non-linear data
and classifier used is called as Non-linear SVM classifier.
 BASIC TERMS OF
SUPPORT VECTOR MACHINE (SVM)
 What is Hyper Plane

Hyperplanes are decision boundaries that help classify the data

points. Data points falling on either side of the hyperplane can
be attributed to different classes.
There can be multiple lines/decision boundaries to segregate the classes, but we need to
find out the best decision boundary that helps to classify the data points. This best
boundary is known as the hyperplane of SVM.

The dimension of the hyperplane depends upon the number

of features. If the number of input features is 2, then the
hyperplane is just a line.
If the number of input features is 3, then the hyperplane
becomes a two-dimensional plane.
It becomes difficult to imagine when the number of features
exceeds 3.
 What is Support Vector

The data points or vectors that are

the closest to the hyperplane and
which affect the position of the
hyperplane are termed as Support
Vector. Since these vectors support
the hyperplane, hence called a
Support vector.

Support Vectors are simply the coordinates of

individual observation.
 What is Margin

The width that the boundary could

be increased by before hitting a
data point.

A Support Vector Machine (SVM) performs classification

by finding the hyperplane that maximizes the margin
between the two classes.
 Hard Margin & Soft Margin

When the data is linearly separable, and we don’t

want to have any misclassifications, we use SVM
with a Hard margin. 

When a linear boundary is not feasible, or we

want to allow some misclassifications in the hope
of achieving better generality, we can opt for a
Soft margin for our classifier.

Sometimes, the data is linearly separable, but the margin is so small that the
model becomes prone to overfitting or being too sensitive to outliers. Also, in
this case, we can opt for a larger margin by using soft margin SVM in order to
help the model generalize better.
 How does SVM Work?

At first approximation, SVM finds a separating line (or hyperplane) between

data of two classes. SVM is an algorithm that takes the data as an input and
outputs a line that separates those classes if possible.

Suppose we have a dataset as shown and we

need to classify the red rectangles from the blue
ellipses. So our task is to find an ideal line that
separates this dataset in two classes (say red
and blue).
 How does SVM Work (Contd.)

Not a big task, right…?

 How does SVM Work (Contd.)

Not a big task, right…?

But, as we notice there isn’t a unique line that

does the job. In fact, we have an infinite lines
that can separate these two classes. So how
does SVM find the ideal one???
Let’s take some probable candidates and
figure it out ourselves.
 How does SVM Work (Contd.)

We have two candidates here, the green colored line

and the yellow colored line. Which line according to
you best separates the data?

The green line in the image above is quite close to

the red class. Though it classifies the current
datasets it is not a generalized line.

If we select the yellow line then it’s visually quite

intuitive in this case that the yellow line classifies better.
But, we need something concrete to fix our line.
 How does SVM Work (Contd.)

According to the SVM algorithm we find the

points closest to the line from both the classes.
These points are called support vectors. Now,
we compute the distance between the line and
the support vectors. This distance is called the
margin. Our goal is to maximize the margin.
The hyperplane for which the margin is
maximum is the optimal hyperplane.
 How does SVM Work (Contd.)

Let’s consider a bit complex dataset, which

is not linearly separable.

This data is clearly not linearly separable. We

cannot draw a straight line that can classify this
data. But, this data can be converted to linearly
separable data in higher dimension. Lets add one
more dimension and call it z-axis. Let the co-
ordinates on z-axis be governed by the constraint,
z = x²+y²

Basically z co-ordinate is the square of distance of the

point from origin. Let’s plot the data on z-axis.
 How does SVM Work (Contd.)

Now the data is clearly linearly separable. Let the purple line separating the data in higher dimension be
z=k, where k is a constant. Since, z=x²+y² we get x² + y² = k; which is an equation of a circle. So, we can
project this linear separator in higher dimension back in original dimensions using this transformation.

Remember that…. This feature is not added manually.

This is done by Kernel trick. 
 Kernel Method

Kernel methods represent the techniques that are used to deal with linearly inseparable data or non-linear
data set shown figure below. The idea is to create nonlinear combinations of the original features to
project them onto a higher-dimensional space via a mapping function, where the data becomes linearly
separable. In the diagram given below, the two-dimensional dataset (X1, X2) is projected into a new three-
dimensional feature space (Z1, Z2, Z3) where the classes become separable.
 SVM
Kernel
Functions
 Advantages of SVM

• Training of the model is relatively easy

• The model scales relatively well to high dimensional data
• SVM is a useful alternative to neural networks
• Trade-off amongst classifier complexity and error can be
controlled explicitly
• It is useful for both Linearly Separable and Non-linearly
Separable data
• Assured Optimality: The solution is guaranteed to be the
global minimum due to the nature of Convex Optimization
 Disadvantages of SVM

• Picking right kernel and parameters can be computationally

intensive
• In Natural Language Processing (NLP), a structured
representation of text yields better performance. However,
SVMs cannot accommodate such structures (word
embedding)
 Applications of SVM
Geostatistics: It is a branch of statistics concentrating on spatial or spatiotemporal datasets. It was originally created to
predict the probability distributions of ore grading at mining operations. Now it is applied in diverse disciplines
including petroleum geology, hydrogeology, hydrology, meteorology, oceanography, geochemistry, geometallurgy,
geography, forestry, environmental control, landscape ecology, soil science, and agriculture (specifically in precision
farming).
Inverse distance weighting: Type of deterministic method for multivariate interpolation with a known scattered set of
points. The values assigned to unknown points are calculated with a weighted average of the values existing at the
known points.
3D Reconstruction: Process of capturing the shape and appearance of real objects.
Bioinformatics: An interdisciplinary field that involves molecular biology, genetics, computer science, mathematics and
statistics. Software tools and methods are developed to understand biological data better.
Chemoinformatics: Application of computational and informational techniques over the field of chemistry to solve a
wide range of problems.
Information Extraction: Acronym as IE, It is a method of automated extraction or retrieval of structured information
from an unstructured and semi-structured text documents, databases and websites.
Handwriting Recognition: Acronym as HWR, It is the ability of a computer system to receive and interpret the
handwritten input comprehensibly from different sources such as a letter paper documents, photographs, touch-
screens and other devices.