Support Vector Machines and Kernel

based methods
 Support Vector Machines --
Introduction
Nonlinear SVM classifiers,
Kernel Functions for
Course Outline nonlinear SVMs
Mercer and positive
definite Kernels,
Support Vector Regression
and &epsilon
Insensitive Loss function, examples of
SVM learning
 Support Vector Machines --
Introduction

• Support Vector Machine or SVM is a supervised machine learning algorithm used for both regression
and classification problems.
• There are two types of SVM algorithm
• Linear SVM classify the data points in 2 classes using a straight line in a 2D space.
• Non-Linear SVM cannot classify the data points in 2 classes with a straight line; technique like kernel
trick is used to classify the data points.

3
 Support Vector Machines:
Hyperplane
• SVM ALGORITHM IS BASED ON THE CONCEPT OF 'DECISION PLANES' WHERE LINES CLASSIFY A SET OF GIVEN OBJECTS.
• LINES ARE KNOWN AS DECISION BOUNDARY.
• SEVERAL DECISION BOUNDARIES CAN DIVIDE THE DATA POINTS WITHOUT ANY ERRORS.
• BUT, HOW DO WE PICK THE BEST DECISION BOUNDARY? BEST DECISION BOUNDARY IS THE LINE THAT HAS THE GREATEST
DISTANCE TO THE BOUNDARY POINTS OF EACH CLASS.
• SVM ALGORITHM CREATES THE BEST DECISION BOUNDARY THAT SEGREGATES A N DIMENSIONAL SPACE INTO CLASSES SO
THAT THE UNSEEN DATA POINTS CAN BE CATEGORIZED.
• FOR THE NUMBER OF INPUT FEATURES = 2, DECISION BOUNDARY IS JUST A STRAIGHT LINE BUT WHEN NUMBER OF INPUT
FEATURES IS GREATER THAN 2, THEN THE DECISION BOUNDARY IS A HYPERPLANE.
 Support Vectors and Margin
• Support vectors are the co-ordinates of data points that lie closest to the decision surface/
hyperplane.
• Support vectors have direct bearing on the optimum location of the decision surface.
• Points of a data set, if removed, alter the position of the hyperplane.
• Margin is the distance between hyperplane and support vectors closest to hyperplane;
margin is like a buffer/cushion between the separating line and the points
• There are two types of margins
In Hard Margin, SVM classifies all the data points accurately, often lead to overfitting.
Soft margin allows SVM to misclassify few data points by keeping large margin-width.

5
 SVM with Slack Variables
Slack variables allow some misclassifications or errors to occur in the training data to handle non-linearly
separable data. This is known as a soft margin SVM.
Formulation
1.Objective Function: Minimize the following function:

where:
• w is the weight vector.
• C is the regularization parameter that
controls the trade-off between maximizing the
margin and minimizing the classification error.
•ξi are the slack variables.
Constraints: For each training example (Xj,
Yj )
where:
• Yj is the class label (+1+1+1 or −1-1−1).
6
• X is the feature vector.
 Kernal Method
• Kernel: Kernel is a mathematical function used in SVM to map the
original input data points into high-dimensional feature spaces,
allowing the hyperplane to be located even when the data points are
not linearly separable in the original input space.

8
9
Mercer and positive definite Kernels

10
Advantages of SVM:

1. Handling high-dimensional data: SVMs effectively handle high-dimensional data, which is common in many
applications such as image and text classification.
2. Handling small datasets: SVMs can perform well with small datasets, as they only require a small number of
support vectors to define the boundary.
3. Modeling non-linear decision boundaries: SVMs can model non-linear decision boundaries by using the
kernel trick, which maps the data into a higher-dimensional space where the data becomes linearly separable.
4. Robustness to noise: SVMs are robust to noise in the data, as the decision boundary is determined by the
support vectors, which are the closest data points to the boundary.
5. Generalization: SVMs have good generalization performance, which means that they can classify new, unseen
data well.

Disadvantages of SVM:

1. Computationally expensive: SVMs can be computationally expensive for large datasets, as the algorithm
requires solving a quadratic optimization problem.
2. Choice of the kernel: The choice of the kernel can greatly affect the performance of an SVM, and it can be
difficult to determine the best kernel for a given dataset.
3. Sensitivity to the choice of parameters: SVMs can be sensitive to the choice of parameters, such as the
regularization parameter, and it can be difficult to determine the optimal parameter values for a given dataset.
Applications of SVM in Real World:
As we have seen, SVMs depend on supervised learning algorithms. The aim of using SVM is to classify unseen
data correctly. SVMs have several applications in several fields. Some common applications of SVM are-

Face detection – SVMc classifies parts of the image as a face and non-face and creates a square boundary around
the face.
Text and hypertext categorization – SVMs allow Text and hypertext categorization for both inductive and
transductive models. They use training data to classify documents into different categories. It categorizes based on
the score generated and then compares with the threshold value.
Classification of images – The use of SVMs provides better search accuracy for image classification. It provides
better accuracy in comparison to traditional query-based searching techniques.
Bioinformatics – It includes protein classification and cancer classification. We use SVM for identifying the
classification of genes, and patients based on genes and other biological problems.
THANK YOU!