Scribd
Upload
2 views16 pages

VC Dim

The document discusses the Vapnik-Chervonenkis (VC) dimension, a key concept in computational learning theory that helps determine if a function is PAC learnable. It defines VC-dimension through the concept of set shattering and provides examples, including the VC-dimensions of linear functions and rectangles. The document also includes bounds on sample complexity and classification error based on VC-dimension, along with references for further reading.

Uploaded by

sangeetha.cse
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views16 pages

VC Dim

The document discusses the Vapnik-Chervonenkis (VC) dimension, a key concept in computational learning theory that helps determine if a function is PAC learnable. It defines VC-dimension through the concept of set shattering and provides examples, including the VC-dimensions of linear functions and rectangles. The document also includes bounds on sample complexity and classification error based on VC-dimension, along with references for further reading.

Uploaded by

sangeetha.cse
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

MACHINE LEARNING

Vapnik-Chervonenkis (VC) Dimension

Alessandro Moschitti

Department of Information Engineering and Computer Science


University of Trento
Email: [email protected]
Computational Learning Theory

The approach used in rectangular hypotheses is just


one case:
Medium-built people
No general rule has been derived
Is there any means to determine if a function is PAC
learnable and derive the right bound?
The answer is yes and it is based on theVapnik-
Chervonenkis dimension (VC-dimension, [Vapnik
95])
VC-Dimension definition (1)

Def.1: (set shattering): a subset S of instances of a set


X is shattered by a collection of function F if ∀ S'⊆ S
there is a function f ∈ F such data:
f (x) = 1 x ∈ S′
{ 0 x ∈ S − S′

€
VC-Dimension definition (2)

Def. 2: the VC-dimension of a function set F (VC-


dim(F)) is the cardinality of the largest dataset that can
be shattered by F.
Observation: the type of the functions used for
shattering data determines the VC-dim
VC-Dim of linear functions (hyperplane)

In the plane (hyperplane = line):


VC(Hiperlpanes) is at least 3
VC(Hiperlpanes)< 4 since there is no set of 4 points, which can be shattered by a
line.
⇒ VC(H)=3. In general, for a k-dimension space VC(H)=k+1
NB: It is useless selecting a set of linealy independent points
Upper Bound on Sample Complexity
Lower Bound on Sample Complexity
Bound on the Classification error using
VC-dimension
Example: Rectangles have VC-dim > 4

We must choose 4-point set, which can be shattered in


all possible ways
Given such 4 points, we assign them the {+,-} labels,
in all possible ways.
For each labeling it must exist a rectangle which
produces such assignment, i.e. such classification
Example (cont’d)

Our classifier: inside the recatagle positive and outside negative


examples, respectively
Given 4 points (linearly independent), we have the following
assignments:
a) All points are “+” ⇒ use a rectangle that includes them
b) All points are “-” ⇒ use a empty rectangle
c) 3 points “-” and 1 “+” ⇒ use a rectangle centered on the “+”
points
Example (cont’d)

d) 3 points “+” and one “-” ⇒ we can always find a rectangle


which exclude the “-” points
e) 2 points “+” and 2 points “-” ⇒ we can define a rectangle
which includes the 2 “+” and excludes the 2 “-”.
To show d) and e) we should check all possibilities
For example, to prove e)

Given 4 points
VC-dim cannot be 5

For any 5-point set, we can define a rectangle which


has the most extern points as vertices
If we assign to such vertices the “+” label and to the
internal point the “-” label, there will not be any
rectangle which reproduces such assigment
Bound Comparison

m > (4/ ε) ⋅ ln(4/δ) (ad hoc bound)


m > (1/ ε) ⋅ ln(1/δ) + 4/ ε = (lower bound based on VC-dim)
(4/ ε) ⋅ ln(4/δ) > (1/ ε) ⋅ ln(1/δ) + 4/ ε
4 ⋅ ln(4/δ) > ln(1/δ) + 4
ln(4/δ) > ln((1/δ)1/4) + 1
4/δ > (1/δ)1/4 ⋅ e
4 > δ 3/4 ⋅ e
4 > (<1) ⋅ (<3) verified
References

VC-dimension:
MY SLIDES: http://disi.unitn.it/moschitti/
teaching.html
MY BOOK:
Automatic text categorization: from information retrieval
to support vector learning
Roberto Basili and Alessandro Moschitti
References
A tutorial on Support Vector Machines for Pattern Recognition
Downlodable from the web
The Vapnik-Chervonenkis Dimension and the Learning
Capability of Neural Nets
Downlodable from the web
Computational Learning Theory
(Sally A Goldman Washington University St. Louis Missouri)
Downlodable from the web
AN INTRODUCTION TO SUPPORT VECTOR MACHINES
(and other kernel-based learning methods)
N. Cristianini and J. Shawe-Taylor Cambridge University Press
You can buy it also on line

You might also like