Support Vector and Kernel Machines

Nello Cristianini BIOwulf Technologies [email protected] http://www.support-vector.net/tutorial.html

ICML 2001

A Little History
SVMs introduced in COLT-92 by Boser, Guyon, Vapnik. Greatly developed ever since. Initially popularized in the NIPS community, now an important and active field of all Machine Learning research. Special issues of Machine Learning Journal, and Journal of Machine Learning Research. Kernel Machines: large class of learning algorithms, SVMs a particular instance.
www.support-vector.net

A Little History
Annual workshop at NIPS Centralized website: www.kernel-machines.org Textbook (2000): see www.support-vector.net Now: a large and diverse community: from machine learning, optimization, statistics, neural networks, functional analysis, etc. etc Successful applications in many fields (bioinformatics, text, handwriting recognition, etc) Fast expanding field, EVERYBODY WELCOME !

www.support-vector.net

Preliminaries
Task of this class of algorithms: detect and exploit complex patterns in data (eg: by clustering, classifying, ranking, cleaning, etc. the data) Typical problems: how to represent complex patterns; and how to exclude spurious (unstable) patterns (= overfitting) The first is a computational problem; the second a statistical problem.
www.support-vector.net

Very Informal Reasoning

The class of kernel methods implicitly defines the class of possible patterns by introducing a notion of similarity between data Example: similarity between documents
By length By topic By language

Choice of similarity features

Choice of relevant

www.support-vector.net

More formal reasoning

Kernel methods exploit information about the inner products between data items Many standard algorithms can be rewritten so that they only require inner products between data (inputs) Kernel functions = inner products in some feature space (potentially very complex) If kernel given, no need to specify what features of the data are being used

www.support-vector.net

Just in case
Inner product between vectors
i i x, z = xz

Hyperplane:

w, x + b = 0

x o x

w
o x

o o o o

www.support-vector.net

Overview of the Tutorial

Introduce basic concepts with extended example of Kernel Perceptron Derive Support Vector Machines Other kernel based algorithms Properties and Limitations of Kernels On Kernel Alignment On Optimizing Kernel Alignment
www.support-vector.net

Parts I and II: overview

Linear Learning Machines (LLM) Kernel Induced Feature Spaces Generalization Theory Optimization Theory Support Vector Machines (SVM)

www.support-vector.net

Modularity

IMPORTANT CONCEPT

Any kernel-based learning algorithm composed of two modules:

A general purpose learning machine A problem specific kernel function

Any K-B algorithm can be fitted with any kernel Kernels themselves can be constructed in a modular way Great for software engineering (and for analysis)

www.support-vector.net

1-Linear Learning Machines

Simplest case: classification. Decision function is a hyperplane in input space The Perceptron Algorithm (Rosenblatt, 57) Useful to analyze the Perceptron algorithm, before looking at SVMs and Kernel Methods in general

www.support-vector.net

Basic Notation
Input space Output space Hypothesis Real-valued: Training Set Test error Dot product
x X y Y = {1,+1} h H f:X R S = {( x1, y1),...,( xi, yi ),....} x, z

www.support-vector.net

Perceptron
Linear Separation of the input space
x x o x x

f ( x ) = w, x + b h( x ) = sign( f ( x ))

w
o x

o o o o

www.support-vector.net

Perceptron Algorithm
Update rule (ignoring threshold): if yi( wk, xi ) 0 then

x x o x x

wk + 1 wk + yixi k k +1

w
o x

o o o o

www.support-vector.net

Observations
Solution is a linear combination of training points w = i y ix i

i 0

Only used informative points (mistake driven) The coefficient of a point in combination reflects its difficulty
www.support-vector.net

Observations - 2
Mistake bound:
x

 R M

2
o

x x x

x o x

o o o o

coefficients are non-negative possible to rewrite the algorithm using this alternative representation
www.support-vector.net

Dual Representation

IMPORTANT CONCEPT

The decision function can be re-written as follows:

w = iyixi

f (x) = w, x + b = iyi xi,x + b

www.support-vector.net

Dual Representation
And also the update rule can be rewritten as follows: if

yi

! y x , x + b& 0
j j j i

then

i i +

Note: in dual representation, data appears only inside dot products

www.support-vector.net

Duality: First Property of SVMs

DUALITY is the first feature of Support Vector Machines SVMs are Linear Learning Machines represented in a dual fashion

f (x) = w, x + b = iyi xi,x + b

Data appear only within dot products (in decision function and in training algorithm)
www.support-vector.net

Limitations of LLMs
Linear classifiers cannot deal with Non-linearly separable data Noisy data + this formulation only deals with vectorial data

www.support-vector.net

Non-Linear Classifiers
One solution: creating a net of simple linear classifiers (neurons): a Neural Network (problems: local minima; many parameters; heuristics needed to train; etc) Other solution: map data into a richer feature space including non-linear features, then use a linear classifier
www.support-vector.net

Learning in the Feature Space

Map data into a feature space where they are linearly separable x ( x)
f

x x o x o o x X f (o) f (o)

f (x) f (x) f (x) f (o) f (o)

www.support-vector.net

f (x) F

Problems with Feature Space

Working in high dimensional feature spaces solves the problem of expressing complex functions BUT: There is a computational problem (working with very large vectors) And a generalization theory problem (curse of dimensionality)
www.support-vector.net

Implicit Mapping to Feature Space

We will introduce Kernels: Solve the computational problem of working with many dimensions Can make it possible to use infinite dimensions efficiently in time / space Other advantages, both practical and conceptual
www.support-vector.net

Kernel-Induced Feature Spaces

In the dual representation, the data points only appear inside dot products:

f (x) = iyi (xi),(x) + b

The dimensionality of space F not necessarily important. May not even know the map
www.support-vector.net

Kernels

IMPORTANT CONCEPT

A function that returns the value of the dot product between the images of the two arguments

K ( x1, x 2 ) = ( x1), ( x 2 )

Given a function K, it is possible to verify that it is a kernel

www.support-vector.net

Kernels
One can use LLMs in a feature space by simply rewriting it in dual representation and replacing dot products with kernels:

x1, x 2 K ( x1, x 2 ) = ( x1), ( x 2 )

www.support-vector.net

The Kernel Matrix

(aka the Gram matrix):
K(1,1) K(2,1) K(1,2) K(2,2) K(1,3) K(2,3) K(1,m) K(2,m)

IMPORTANT CONCEPT

K=
K(m,3) K(m,m) K(m,1) K(m,2)

www.support-vector.net

The Kernel Matrix

The central structure in kernel machines Information bottleneck: contains all necessary information for the learning algorithm Fuses information about the data AND the kernel Many interesting properties:

www.support-vector.net

Mercers Theorem
The kernel matrix is Symmetric Positive Definite Any symmetric positive definite matrix can be regarded as a kernel matrix, that is as an inner product matrix in some space

www.support-vector.net

More Formally: Mercers Theorem

Every (semi) positive definite, symmetric function is a kernel: i.e. there exists a mapping

such that it is possible to write:

K ( x1, x 2 ) = ( x1), ( x 2 )
Pos. Def.

f L2
www.support-vector.net

K ( x , z ) f ( x ) f (z )dxdz 0

Mercers Theorem
Eigenvalues expansion of Mercers Kernels:

K ( x1, x 2 ) = i i ( x 1)i ( x 2 )
i

That is: the eigenfunctions act as features !

www.support-vector.net

Examples of Kernels
Simple examples of kernels are:

K ( x, z ) = x, z K ( x, z ) = e

x z 2 / 2

www.support-vector.net

Example: Polynomial Kernels

x = ( x1, x 2 ); z = ( z1, z 2 ); x, z
2

= ( x 1 z1 + x 2 z 2 ) 2 =

2 2 z 2 + 2 x 1 z1 x 2 z 2 = = x12 z12 + x2 2 2 , 2 x1 x 2 ),( z12 , z2 , 2 z1 z 2 ) = = ( x12 , x2

= ( x ), ( z )

www.support-vector.net

Example: Polynomial Kernels

www.support-vector.net

Example: the two spirals

Separated by a hyperplane in feature space (gaussian kernels)

www.support-vector.net

Making Kernels

IMPORTANT CONCEPT

The set of kernels is closed under some operations. If K, K are kernels, then: K+K is a kernel cK is a kernel, if c>0 aK+bK is a kernel, for a,b >0 Etc etc etc can make complex kernels from simple ones: modularity !
www.support-vector.net

Second Property of SVMs:

SVMs are Linear Learning Machines, that Use a dual representation AND Operate in a kernel induced feature space f (x) = iyi (xi),(x) + b (that is: is a linear function in the feature space implicitely defined by K)

www.support-vector.net

Kernels over General Structures

Haussler, Watkins, etc: kernels over sets, over sequences, over trees, etc. Applied in text categorization, bioinformatics, etc

www.support-vector.net

A bad kernel
would be a kernel whose kernel matrix is mostly diagonal: all points orthogonal to each other, no clusters, no structure 1 0 0 0 1 0 0 0 1 0 1 0 0

www.support-vector.net

No Free Kernel

IMPORTANT CONCEPT

If mapping in a space with too many irrelevant features, kernel matrix becomes diagonal Need some prior knowledge of target so choose a good kernel

www.support-vector.net

Other Kernel-based algorithms

Note: other algorithms can use kernels, not just LLMs (e.g. clustering; PCA; etc). Dual representation often possible (in optimization problems, by Representers theorem).

www.support-vector.net

#

www.support-vector.net

NEW TOPIC

The Generalization Problem

The curse of dimensionality: easy to overfit in high dimensional spaces
(=regularities could be found in the training set that are accidental, that is that would not be found again in a test set)

The SVM problem is ill posed (finding one hyperplane that separates the data: many such hyperplanes exist) Need principled way to choose the best possible hyperplane

www.support-vector.net

The Generalization Problem

Many methods exist to choose a good hyperplane (inductive principles) Bayes, statistical learning theory / pac, MDL, Each can be used, we will focus on a simple case motivated by statistical learning theory (will give the basic SVM)

www.support-vector.net

Statistical (Computational) Learning Theory

Generalization bounds on the risk of overfitting (in a p.a.c. setting: assumption of I.I.d. data; etc) Standard bounds from VC theory give upper and lower bound proportional to VC dimension VC dimension of LLMs proportional to dimension of space (can be huge)

www.support-vector.net

Assumptions and Definitions

distribution D over input space X train and test points drawn randomly (I.I.d.) from D training error of h: fraction of points in S misclassifed by h test error of h: probability under D to misclassify a point x VC dimension: size of largest subset of X shattered by H (every dichotomy implemented)

www.support-vector.net

VC Bounds

~ VC = O m

VC = (number of dimensions of X) +1 Typically VC >> m, so not useful Does not tell us which hyperplane to choose

www.support-vector.net

Margin Based Bounds

= = ~ ( R / ) 2 O m y if ( x min i f i )

Note: also compression bounds exist; and online bounds.

www.support-vector.net

Margin Based Bounds

IMPORTANT CONCEPT

(The worst case bound still holds, but if lucky (margin is large)) the other bound can be applied and better generalization can be achieved:

~ ( R / ) = O m
Best hyperplane: the maximal margin one Margin is large is kernel chosen well

www.support-vector.net

Maximal Margin Classifier

Minimize the risk of overfitting by choosing the maximal margin hyperplane in feature space Third feature of SVMs: maximize the margin SVMs control capacity by increasing the margin, not by reducing the number of degrees of freedom (dimension free capacity control).

www.support-vector.net

Two kinds of margin

Functional and geometric margin:

funct = min yif ( xi )

x

g
o

x x x

x o x

yif ( xi ) geom = min f

o o o o

www.support-vector.net

Two kinds of margin

www.support-vector.net

Max Margin = Minimal Norm

If we fix the functional margin to 1, the geometric margin equal 1/||w|| Hence, maximize the margin by minimizing the norm

www.support-vector.net

Max Margin = Minimal Norm

Distance between The two convex hulls
x

w, x w, x

+ b = +1 + b = 1

g
o

x x x

x o x

w,( x + x ) = 2 w 2 + ,( x x ) = w w

o o o o

www.support-vector.net

The primal problem

Minimize: subject to:

IMPORTANT STEP

w, w yi " w, xi + b ' 1

www.support-vector.net

Optimization Theory
The problem of finding the maximal margin hyperplane: constrained optimization (quadratic programming) Use Lagrange theory (or Kuhn-Tucker Theory) Lagrangian: 1  i + b 1 w, w iyi  w , x   2

0
www.support-vector.net

From Primal to Dual

1  i + b 1 L( w ) = w, w iyi  w , x   2

i 0

Differentiate and substitute:

L =0 b L =0 w

www.support-vector.net

The Dual Problem

Maximize:

IMPORTANT STEP

1 W ( ) = i i i jyiyj xi, xj ,j 2 i Subject to: i 0 iyi = 0

i

The duality again ! Can use kernels !

www.support-vector.net

Convexity

IMPORTANT CONCEPT

This is a Quadratic Optimization problem: convex, no local minima (second effect of Mercers conditions) Solvable in polynomial time (convexity is another fundamental property of SVMs)

www.support-vector.net

Kuhn-Tucker Theorem
Properties of the solution: Duality: can use kernels KKT conditions: i yi w, xi + b 1 = 0

Sparseness: only the points nearest to the hyperplane (margin = 1) have positive weight i i i

w = y x

They are called support vectors

www.support-vector.net

KKT Conditions Imply Sparseness

Sparseness: another fundamental property of SVMs
x

g
o

x x x

x o x

o o o o

www.support-vector.net

Properties of SVMs - Summary

' ' ' ' '

Duality Kernels Margin Convexity Sparseness

www.support-vector.net

Dealing with noise

In the case of non-separable data in feature space, the margin distribution 2 can be optimized 2

1 R + 2 m

yi " w, xi + b ' 1 i
www.support-vector.net

The Soft-Margin Classifier

Minimize:

Or:

1 w, w + C i 2 i 2 1 w, w + C i 2 i yi " w, xi + b ' 1 i
www.support-vector.net

Subject to:

Slack Variables
1 R + 2 m

yi " w, xi + b ' 1 i
www.support-vector.net

Soft Margin-Dual Lagrangian

1 i jyiyj xi, xj Box constraints W( ) = i 2 i ,j i 0 i C iyi = 0
i

Diagonal

1 1 i jyiyj xi, xj i i jj ,j 2 2C i 0 i iyi 0

i

www.support-vector.net

The regression case

For regression, all the above properties are retained, introducing epsilon-insensitive loss:
L

e 0 yi-<w,xi>+b

www.support-vector.net

Regression: the -tube

www.support-vector.net

Implementation Techniques
Maximizing a quadratic function, subject to a linear equality constraint (and inequalities as well) 1

W ( ) = i
i

y y K ( x , x ) 2
i, j i j i j i j

i 0 iyi = 0
i

www.support-vector.net

Simple Approximation
Initially complex QP pachages were used. Stochastic Gradient Ascent (sequentially update 1 weight at the time) gives excellent approximation in most cases
1 i i + 1 yi iyiK ( xi, xj ) K ( xi, xi )

www.support-vector.net

Full Solution: S.M.O.

SMO: update two weights simultaneously Realizes gradient descent without leaving the linear constraint (J. Platt).

Online versions exist (Li-Long; Gentile)

www.support-vector.net

Other kernelized Algorithms

Adatron, nearest neighbour, fisher discriminant, bayes classifier, ridge regression, etc. etc Much work in past years into designing kernel based algorithms Now: more work on designing good kernels (for any algorithm)

www.support-vector.net

On Combining Kernels
When is it advantageous to combine kernels ? Too many features leads to overfitting also in kernel methods Kernel combination needs to be based on principles Alignment

www.support-vector.net

Kernel Alignment

IMPORTANT CONCEPT

Notion of similarity between kernels: Alignment (= similarity between Gram matrices)

A( K1, K 2) =

K1, K 2 K1, K1 K 2, K 2

www.support-vector.net

Many interpretations
As measure of clustering in data As Correlation coefficient between oracles Basic idea: the ultimate kernel should be YY, that is should be given by the labels vector (after all: target is the only relevant feature !)

www.support-vector.net

The ideal kernel

YY=

1 1 -1 -1

1 1 -1 -1

-1 -1 1 1

-1 -1 1 1

www.support-vector.net

Combining Kernels
Alignment in increased by combining kernels that are aligned to the target and not aligned to each other.

A( K 1, YY' ) =

K 1, YY' K 1, K 1 YY' , YY'

www.support-vector.net

Spectral Machines
Can (approximately) maximize the alignment of a set of labels to a given kernel yKy By solving this problem:

y = arg max yi {1,+1}

yy'

Approximated by principal eigenvector (thresholded) (see courant-hilbert theorem)

www.support-vector.net

Courant-Hilbert theorem
A: symmetric and positive definite, Principal Eigenvalue / Eigenvector characterized by:

vAv = max v vv'

www.support-vector.net

Optimizing Kernel Alignment

One can either adapt the kernel to the labels or vice versa In the first case: model selection method Second case: clustering / transduction method

www.support-vector.net

Applications of SVMs
Bioinformatics Machine Vision Text Categorization Handwritten Character Recognition Time series analysis

www.support-vector.net

Text Kernels
Joachims (bag of words) Latent semantic kernels (icml2001) String matching kernels See KerMIT project

www.support-vector.net

Bioinformatics
Gene Expression Protein sequences Phylogenetic Information Promoters

www.support-vector.net

Conclusions:
Much more than just a replacement for neural networks. General and rich class of pattern recognition methods
44 43 $'8  8:55479
;0.947309

Kernel machines website www.kernel-machines.org www.NeuroCOLT.org

www.support-vector.net