Scribd
Upload
144 views4 pages

SVM Overview

SVM uses nonlinear mapping to transform input data into a higher dimensional feature space, finding a linear separating hyperplane with maximum margin between classes in that space. The hyperplane is defined by support vectors, and kernel functions allow computing dot products in feature space without explicitly transforming data, avoiding high computational costs. SVM advantages include accurate classification and robustness to noise, while disadvantages are that it only performs binary classification and can be computationally expensive.

Uploaded by

webdaxter
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
144 views4 pages

SVM Overview

SVM uses nonlinear mapping to transform input data into a higher dimensional feature space, finding a linear separating hyperplane with maximum margin between classes in that space. The hyperplane is defined by support vectors, and kernel functions allow computing dot products in feature space without explicitly transforming data, avoiding high computational costs. SVM advantages include accurate classification and robustness to noise, while disadvantages are that it only performs binary classification and can be computationally expensive.

Uploaded by

webdaxter
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Brief Introduction to Support Vector Machines

1. Linearly Separable models


If all patterns in a dataset can be separated by a straight line or a hyper-plane, the dataset is said
to be linearly separable.

However, there are many problems, such as XOR, which are not linearly separable.

2. SVM Approach
SVM uses linear models to implement nonlinear class boundaries. It transforms the input space
using a nonlinear mapping into a new space (F feature space). Then a linear model constructed in
the new space can represent a nonlinear decision boundary in the original space:

3. The maximum margin hyperplane


Another component in the SVM approach is the maximum margin hyperplane.

a. What is a maximum margin hyperplane?


Suppose a two-class dataset is linearly separable. The maximum margin hyperplane is the one
that gives the greatest separation between the classes. Among all hyperplanes that separate the
classes, the maximum margin hyperplane is the one that is as far away as possible from the two
convex hulls, each of which is formed by connecting the instances of a class.

The instances that are closest to the maximum hyperplane are called support vectors. There is at
least one support vector for each class, and often there are more.
A set of support vectors can uniquely defines the maximum margin hyperplane for the learning
problem. All other training instances are irrelevant; they can be removed without changing the
position and orientation of the hyperplane.
b. The equation of the maximum margin hyperplane
In general, a hyperplane separating the two classes in the two-attribute case, can be written as
F ( x) = w0 + w1 a1 + w2 a 2 , where a1 and a2 are attribute values, and w0, w1, w2 are weights.
The equation for the maximum margin hyperplane can also be written in terms of support
vectors.
l

f ( x) = b + i y i < xi x >
i =1

where

i is the support vector;


yi is the class value of a training instance: either 1 or -1;
xi is the ith support vector;
x is a vector which represents a test instance;
< xi x > denotes the dot product of x and xi ; and
b and s are parameters that determine the hyperplane (just as the weights w0, w1, w2).
Finding support vectors and the maximum margin hyperplane (i.e., b and s) belongs to a
standard class of optimization problem known as constraint quadratic optimization. There are
several off-the-shelf tools for solving those problems.

4. Kernel Functions

According to the equation of the maximum margin hyperplane above, every time an instance is
classified, its dot product with all support vectors must be calculated. In a high-dimensional
space resulting after a non-linear transformation is applied to the instances, the number of
attributes in the new space could become huge and the computation of dot product becomes very
expensive. The same problem occurs during training as well. Working in such a highdimensional space becomes intractable very quickly.
l

f ( x ) = b + i y i < ( xi ) ( x ) >

(1)

i =1

where is a non-linear transformation function.


Fortunately, we dont need to explicitly construct a high dimensional features space. By using
the kernel function, it is possible to calculate the dot product before the non-linear transformation
is performed. In other words, a kernel function can be applied to instances in the original input
space which brings out the same effect as the linear transformation without expanding the feature
space by non-linear transformation kernel trick.
Implicit mapping into the transformed feature space

A kernel is a function K such that, for all x, z X (input space)


K(x, z) = < ( x ) ( z ) >

By using K, the equation (1) can be written as:


l

f ( x) = b + i y i K ( xi , x)
i =1

(2)

Polynomial Kernel

The function < x z > represents a polynomial function of degree n applied to the do product
of two vector x and z. In the case when n = 2 and the number of attributes is 2, the function
becomes:
n

< x z >2
2

= ( xi z i ) 2
i =1

= ( x1 z1 + x 2 z 2 ) 2
= ( x12 z12 + 2 x1 x 2 z1 z 2 + x 22 z 22 )
=< ( x12 , 2 x1 x 2 , x 22 ) ( z12 , 2 z1 z 2 , z 22 ) >
=< ( x ) ( z ) >
Therefore, the kernel K becomes
K(x, z) =< x z > 2 = < ( x ) ( z ) > , where ( x) = ( x12 , 2 x1 x 2 , x 22 ) .
Other Kernels

In addition to the polynomial kernel, RBF (radial basis function) and sigmoid kernels are often
used. Using which kernel depends on the specific data and applications.

5. Advantage & Disadvantages of SVM

Advantages:
1. Produce very accurate classifiers.
2. Less overfitting, robust to noise.
Disadvantages:
1. SVM is a binary classifier. To do a multi-class classification, pair-wise classifications
can be used (one class against all others, for all classes).
2. Computationally expensive, thus runs slow.

You might also like