SSVM : A Simple SVM Algorithm

S.V.N. Vishwanathan, M. Narasimha Murty

{vishy, mnm}@csa.iisc.ernet.in
Dept. of Comp. Sci. and Automation,
Indian Institute of Science,
Bangalore 560 012,
INDIA

Abstract - We present a fast iterative algorithm for identifying the inseparable case [8]. The Karush-Kuhn-Tucker ( KKT )
the Support Vectors of a given set of points. Our algorithm works conditions for the dual can be expressed as
by maintaining a candidate Support Vector set. It uses a greedy
approach to pick points for inclusion in the candidate set. When dW =
the addition of a point to the candidate set is blocked because
9'I -
-
-
dai ;
Qijaj + yib - 1 = y i f ( ~ i )- 1 (2)
of other points already present in the set we use a backtracking
approach to prune away such points. To speed up convergence and
we initialize our algorithm with the nearest pair of points from
opposite classes. We then use an optimization based approach (3)
to increment or prune the candidate Support Vector set. The al-
gorithm makes repeated passes over the data to satisfy the KKT
constraints. The memory requirements of our algorithm scale as This partitions the training set into S the Support Vector set
O(lSlz) in the average case, where IS1 is the size of the Support (O<ai<C,gi=O),Etheerrorset(ai = C,gi < 0 )
Vector set. We show that the algorithm is extremely competitive and R the well classified set ( ai = 0, gi > 0 ) [2].
as compared to other conventional iterative algorithms like SMO If the points in error are penalized quadratically with a
and the NPA. We present results on a variety of real life datasets penalty factor C', then, it has been shown that the problem
to validate our claims. reduces to that of a separable case with C = 00 [3]. The
kernel function is modified as
I. Introduction
Support Vector Machines ( SVM ) have recently gained
prominence in the field of machine learning and pattern clas-
sification [8]. Classification is achieved by realizing a linear where 6 i j = 1if i = j and S i j = 0 otherwise. The advantage
or non-linear separation surface in the input space. of this formulation is that the SVM problem reduces to that of
In Support Vector classification,the separating function can a linearly separable case [4].
be expressed as a linear combination of kernels associated It can be seen that training the SVM involves solving a
with the Support Vectors as quadratic optimization problem which requires the use of op-
timization routines from numerical libraries. This step is com-
putationally intensive, can be subject to stability problems and
Xj ES is non-trivial to implement [5]. Attractive iterative algorithms
like the Sequential Minimal Optimization ( SMO ), Nearest
where z i denotes the training patterns, y i E {+l,-1) de- Point Algorithm ( NPA ) etc. have been proposed to overcome
notes the corresponding class labels and S denotes the set of this problem [ 5 ] , [4]. This paper makes another contribution
Support Vectors [8]. in this direction.
The dual formulation yields
1 A. DirectSVM and Geometric SVM
min W = -
o<ai<c
aiQijaj - ai +b yiai (1)
i,j i i The DirectSVM is an intuitively appealing algorithm, which
where a i are the corresponding coefficients, b is the offset, builds the Support Vector set incrementally [6]. Recently it
Q t3
.. - y i y j K ( z i , z j ) is a symmetric positive definite kernel has been proved that the closest pair of points of the opposite
matrix and C is the parameter used to penalize error points in class are always Support Vectors [7]. DirectSVM starts off
with this pair of points in the candidate Support Vector set. It
Corresponding author - M. Narasimha Murty has been conjectured that the maximum violator during each

0-7803-7278-6/02/$10.00 02002 IEEE 2393

iteration is a Support Vector [7]. The algorithm finds the max- addition of a new Support Vector. We prune away those points
imum violator during each iteration and rotates the candidate using the Decremental technique described in [2]. It may also
Support Plane to make the maximum violator a Support Vec- happen that other points in the dataset may become violators
tor. In case the dimension of the space is exceeded or all the or Support Vectors. The Incremental algorithm handles such
data points are used up, without convergence, the algorithm points by doing costly book keeping in the current iteration it-
reinitializes with the next closest pair of points from opposite self and hence does not need to make repeated passes over the
classes [6]. The advantage of the DirectSVM algorithm is that dataset [2]. On the other hand our algorithm does not handle
it is geometrically motivated and simple to understand. such points in the current iteration, it makes repeated passes
The Geometric SVM proposed by us improves the scaling over the'dataset to identify and satisfy such points.
behavior of the DirectSVM by using an optimization based
approach to add points to the candidate Support Vector set C. Outline of Our Paper
~91.
We observed that neither of the algorithms has a provision In this paper we present a fast iterative algorithm which com-
to backtrack, i.e. once they decide to include a point in the can- bines the the theoretically motivated foundation of the Incre-
didate Support Vector set they cannot discard it. During each mental algorithm with the speed typical of iterative algorithms
step, both the algorithms spend their maximum effort in find- like NPA or the SMO. We borrow the initialization technique
ing the maximum violator. If S is the current candidate Sup- from the DirectSVM algorithm to speed up convergence of
port Vector set, and n the size of the dataset, the algorithms our algorithm. Section I1 talks about our algorithm and pro-
spend (n - s)(SIkernel evaluations to locate the maximum vides proofs. We also talk about the memory requirements
violator. Caching schemes have been proposed to increase the of our algorithm. We present and discuss results obtained on
efficiency of this step [9]. real life datasets in Section 111. We conclude in Section IV
Our observations led us to a greedy algorithm which picks by pointing out advantages of our algorithm and indicate sit-
uations under which it can perform extremely well. We also
the next immediately available violating point for inclusion
in the candidate Support Vector set. Some stray non Support provide pointers to our ongoing and future work.
Vectors may be picked up for inclusion in the candidate set 11. Simple S V M
because of this greedy approach. Our algorithm backtracks
later by pruning away such stray points from the candidate Our algorithm maintains a candidate Support Vector set. It
Support Vector Set. In order to satisfl the KKT conditions for initializes the set with the closest pair of points from opposite
all points it has to make repeated passes though the dataset. classes like the DirectSVM algorithm. As soon as the algo-
rithm finds a violating point in the dataset it greedily adds it
to the candidate set. It may so happen that addition of the vi-
B. Incremental Algorithm olating point as a Support Vector may be prevented by other
candidate Support Vectors already present in the set. We sim-
Recently some work has also been done on incremental SVM ply prune away all such points from the candidate set. To en-
algorithms which can converge to exact solutions and also
sure that the KKT conditions are satisfied we make repeated
efficiently calculate leave one out errors [2]. The algorithm
passes through the dataset until no violators can be found. We
works by adding a point at a time to the current dataset. It is
use the quadratic penalty formulation to ensure linear separa-
shown that the addition of a point converges in a finite num-
bility of the data points in the kernel space.
ber of steps. The algorithm calculates the change in the KKT
conditions of each one of the current data points in order to
do book keeping. In case the size of the current data set is n A. Finding the Closest Pair of Points
and the number of Support Vectors is (SI,then, this algorithm First of all, we observe that, finding the closest pair of points
has to perform nlSl kemel operations for addition of the next in kernel space requires n2 kernel computations where n rep-
point. The authors suggest a practical online variant where resents the total number of data points. But, in case we use
they introduce an 6 margin and concentrate only those points a distancepreserving kernel like the exponential kernel the
which are within the 6 margin of the boundary [2]. But, it is nearest neighbors in the feature space are the same as the near-
clear that the results may vary by varying the value of 6. est neighbors in the kernel space. Hence we need not perform
Instead of concentratingon the entire current dataset, which any costly kernel evaluations for the initialization step.
may be costly, our algorithm focuses its attention only on the
current Support Vector set. It then picks data points greedily B. Adding a Point to the Support Vector Set
for inclusion in the current Support Vector set. While aug-
menting the current Support Vector set, some already existing Given a set S which contains only Support Vectors, we wish
Support Vectors may become well classified because of the to add another Support Vector c to S. From Equations 2 and

0-7803-7278-6/02/$10.00 (92002 JEEE 2394

~~
3 we get the change in gi due to addition of a new point c as c. Pruning
Agi = QicAac + Qij Aaj +yiAb (4) In the discussion above we tacitly assumed that (a+ A a ) >
j€S 0 V p E S. But, this condition may be violated if any point
and in S blocks c. When we say that a point p E S is blocking
0 = ycAac + YjAaj (5)
the addition of c to S what we mean is that ap of that point
may become negative due to the addition of c to S. What it
jES
physically implies is that p is making a transition from S to the
where Aai is the change in the value of ai and Ab is the
well classified set R. Because of the presence of such points
change in the value of b. We start off with a, = 0 and update
we may not be able to update a, by the amount specified by
ac as we go along.
Equation 8. In such a case we can prune away p from S by
Because all the vectors in S are Support Vectors we know
using [2]
from Equation 2 that gi = 0 Vi. If none of the current Sup-
port Vectors blocks the addition of c to S then all the vectors a.-- R.. - R - ~ & , R .
3 SS 83 Y Ra3. .E R
in S continue to remain Support Vectors in S U { c} and hence
we require that Agi = 0 for all vectors in S. It is shown that We remove the alpha entry corresponding to p from S so that
[21 all the other points in S continue to remain Support Vectors.
Ab = pAa, We now try to add c to this reduced S. We keep pruning points
from S till c can actually become a Support Vector.
and
Aaj = pjAac (7)
D. OurAlgorithm
If we define
-1 Using the ideas we discussed above an iterative algorithm can
0 y1 ... YS
be designed which scans through the dataset looking for vi-
YI Qii - - . &is olators. Using ideas presented in Section 11-B the violator
R=
is made a Support Vector. Blocking points are identified and
pruned away by using the ideas presented in Section 11-C. The
then

[ :]=-R[ $1
Qw

We want gc = 0 so that c can become a Support Vector. Some

algorithm stops when all points are classified within an error
bound i.e. y i f ( z i ) > 1- e Vi. The outline of our algorithm
is presented in Algorithm 1.

Algorithm 1 Simde SVM

candidateSV = { closest pair from opposite classes }
while there are violating points do
algebra yields Find a violator
+ ycb - 1
C j E S Qcjaj
U
candidateSV = candidateSV violator
Aa, = - (8) if any ap < 0 due to addition of c to S then
Qcc + CjEs Q c j P j + Y C P
candidateSV = candidateSV \ p
If we define repeat till all such points are pruned
end if
~c = Qcc + QcjFlj +~ c F l end while
j€S
From [2] we know that we can expand R as
E. Scaling
As is clear from the discussion in Section 11-B and 11-C,
Bs 1 3 (9) the algorithm needs to store the values of a s corresponding to
the candidate Support Vector set (S). This occupies at most
O(lS() space. Storing the value of b requires O(1) space. It
Thus our new a is also needs to store the R matrix corresponding to the candi-
date Support Vector set. Since R is a IS( x IS1 matrix its size
scales up as O(lSI2). Thus we can conclude that the mem-
ory requirements of the algorithm scale up as O(lS(2)in the
and our new b is b + Ab. average case.

O-7803-7278-6/02/%10.00 Q2002 IEEE 2395

 111. Results Simple SVM NPA
Recently Keerthi et. al. have suggested using the number of
C' Itr 1 SV I KernelEval SV I KemelEval
kernel evaluations performed by a Support Vector algorithm
0.03 665 I 652 I 0.490 652 I 1.21
as an effective measure of its speed [4]. Comparisons of the
NPA algorithm with various other conventional iterative algo-
rithms shows that NPA performs significantly faster [4]. We
compared our algorithm with the NPA on 5 publicly avail-
able datasets. All experiments were run on a 800 MHz, Intel
Pentium I11 machine with 128MB RAM running Linux Man-
drake 8.0 unless mentioned otherwise. The code was written
in C++ as well as in MATLAB'. We uniformly used a value
of 0.001 for the error bound i.e. we stop the algorithm when
yif(zi) > 0.999 Vi. The NPA results are those reported in
~41.
First we used the Spiral dataset proposed by Alexis Wieland 1 500 I 356 I 296 I 0.443 1 298 1 0.69
of the MITRE Corporation and available from the CMU Arti-
ficial Intelligence repository. We use the Gaussian kernel TABLE I1
NPA VS SIMPLE SVM ON WPBC DATASET. KERNEL
~ ( z , y )= ezp(-0.5(lz - y1I2/a2) EVALUATIONS
ARE X lo6

with 'a = 0.5 [4]. We vary the value of C' and reproduce our
results in table I.
We used the Adult- 1 dataset from the UCI Machine Learn-
Simple SVM NPA ing repository [ 11. This is a sparse dataset which consists of
C' I& I I
SV KernelEval SV KernelEval I 1605 data points, each having a dimension of 123. We used
0.03 192 I 194 I 0.038 195 I 0.104 the Gaussian kernel with c2 = 10.0 [5]. We vary the value of
0.1 192 194 0.038 195 0.121 C' and reproduce our results in table III.
0.2 192 194 0.038 195 0.134 The Adult-4 dataset is also from the UCI Machine Learning
0.3 192 194 0.038 195 0.143 repository [l]. This is a sparse dataset and consists of 4781
0.6 192 194 0.038 195 0.162 data points, each having a dimension of 123. We used the
1.0 192 194 0.038 195 0.188 Gaussian kernel with n' = 10.0 [5]. We vary the value of C'
2.0 199 194 0.038 195 0.238 and reproduce our results in table IV.
I I
I I

3.0 I 201 1 194 I 0.038 I 194 I 0.276

5.0 204 188 0.040 188 0.340
10 204 184 0.039 185 0.452
1.094
1.300

2.210

TABLE I
NPA VS SIMPLE SVM ON SPIRALS DATASET. KERNEL
EVALUATIONS ARE x lo6

We used the WPBC dataset from the UCI Machine Learn-

ing repository [l]. This dataset consists of 683 data points,
each having a dimension of 9. We used the Gaussian kernel C' ->
with n2 = 4.0 [4]. We vary the value of C' and reproduce ow
Fig. 1. Comparison of the Kernel Evaluations performed by Simple
results in table 11. S V M and the NPA on the Adult-4 Dataset.
'Working code and datasets used for the experiments are available at
http://wwwl.csa.iisc.ernet.in/-vishy The Adult-7 dataset from the UCI Machine Learning repos-

0-7803-7278-6/02/$10.00 02002 lEEE 2396

 itory is a sparse data set of 16,100 points, each having a
dimension of 123 [l]. We used the Gaussian kernel with
I I Simde SVM I NPA u2 = 10.0 [5]. The experiments were conducted on a 800
C' Itr 1SV I KernelEval SV I KernelEval- MHz PI11 machine with 256 MB RAM,running Windows
0.03 1603 1 1602 I 2.58 1602 I 7.4 2000. We vary the value of C' and reproduce our results in
table V.
Simple SVM
2.99 1191 7.9
2.83 1118 8.9 10181 11.31
1.0 1077 1272 2.73 1077 9.6 9343 20.53
2.0 1257 1032 2.77 1032 10.6 3.92 8393 5 1.05
3.0
-~ 1252 996 2.90 996 11.5
5.0 1217 951 2.23 950 11.9 TABLE V
10 1191 898 2.88 898 14.1 NPA VS SIMPLE SVM ON ADULT-I DATASET. KERNEL
50 1176 776 2.82 776 26.2 EVALUATIONSARE X 10'
100 1188 735 3.18 734 33.7
500 1200 687 3.08 687 56.9
1000 1224 677 3.20 677 66.0
A. Discussion of the Results
As can be seen the Simple SVM algorithm outperforms the
NPA on each one of the datasets tested. For example on the
Spiral dataset the Simple S V M is an order of magnitude faster
than the NPA. On the Adult4 dataset with C' = 1000 the
Simple SVM algorithm is nearly 50 times faster than the NPA.
On the average the Simple S V M is seen to be around 3 to 5
times faster than the NPA.
The kernel computations performed by NPA, SMO and
other iterative algorithms tend to vary with the value of C'
and the total number of Support Vectors. On the other hand
the kernel evaluations of the Simple S V M seem to be a func-
tion of the dataset only. They are fairly independent of the
value of C' or the total number of Support Vectors. This is
illustrated by the graph in Fig 1.
I
The difference between the number of iterations and the
0.6 3605 3201 2.40 3199 8.26
number of final Support Vectors indicates the number of times
1.0 3552 3117 2.36 3116 8.96
the algorithm had to backtrack. It can be seen that the Sim-
2.0 3517 2999 2.44 2998 9.88
ple SVM algorithm needs to backtrack only around 20% of
3.0 3466 2911 2.66 2915 9.87 the times and hence the penalty incurred due to a greedy ap-
5.0 3448 2882 2.63 2824 11.51 proach is not very significant. This validates the observation
10 3486 2679 2.64 2680 15.06 that a greedy approach taken by SMO like algorithms pro-
50 3336 2313 2.64 2371 35.55 duces faster convergence.
100 3420 2194 3.06 2242 46.08 The number of Support Vectors found by our algorithm and
500 3590 2032 2.74 2034 102.33 the NPA is different in the case of Adult-7 dataset. One possi-
1000 3662 1967 3.02 1968 146.89 ble explanation for this observation is that the stopping crite-
rion employed by the two algorithms are different. Round off
TABLE IV errors due to differences in the precisions used for the compu-
NPA VS SIMPLE SVM ON ADULT-4 DATASET. KERNEL tations may further aggravate the problem.
EVALUATIONSARE x io7
IV. Conclusion and Future Work
We have presented a new algorithm that is efficient, intu-
itive and fast. We show that the algorithm significantly outper-

0-7803-7278-6/02/$10.00 02002 IEEE 2397

forms other iterative algorithms like the NPA in terms of the
number of kernel computations. Because of the approach used
to build the Support Vector, set our algorithm does not suffer
from numerical instabilities and round off errors that plague
other numerical algorithms for the SVM problem. Our algo-
rithm currently does not use any kind of kernel cache to reuse
kernel computations. We are currently investigating meth-
ods to speed up the algorithm using some efficient caching
scheme.
We also observe that the memory utilization of the algo-
rithm is governed by the R matrix which scales as O(lS12).
Hence, our algorithm does not scale well for those problems
where the R matrix cannot be held in main memory. But, this
is not a serious limitation, for example on a machine with 256
MB of main memory we can store the R matrix corresponding
to as many as 10,000 Support Vectors. We are investigating
methods to store a compact representation of R in order to
reduce this memory overhead.
It can be observed that the addition of a vector to the Sup-
port Vector set is entirely reversible. Using this property Pog-
gio et. al. [2] have calculated the leave one out error. We pro-
pose to use similar techniques to calculate the leave one out
error based on the order in which the data points were added
to Support Vector set.

References
[I] C.L. Blake and C.J. Men. UCI repository of machine learning databases,
1998.
[2] Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental
support vector machine learning. In Advances in Neural Information
Processing System, (NIPS*2000), volume 13. NIPS, Cambridge MA:
MIT Press, 2001.
[3] T. Friess, N. Cristianini, and C. Campbell. The kernel adatron algo-
rittun: a fast and simple learning procedure for support vector machine.
In Proceedings of ISth lntemational Conference on Machine Learning.
Morgan Kaufinan, 1998. ,

[4] S. S. Keerthi, S. K. Shevade, C. Bhattacharyya,andK. R. K. Murthy. A

fast iterative nearest point algorithm for support vector machine classifier
design. IEEE Transactions on Neural Networks, 11(1):124-136,2000.
[5] J. C. Platt. Fast training of support vector machines using sequential
minimal optimization. In B. Scholkopf,C. Burges, and A. Smola, editors,
Advances in Kernel Methods: Support VectorMachines. Cambridge MA:
MIT Press, December 1998.
[6] Danny Roobaert. DirectSVM: A fast and simple support vector machine
perceptron. In Proceedings of IEEE International Workshop on Neural
Networksfor Signal Processing, Sydney,Australia, December 2000.
[7] Danny Roobaert. DirectSVM: A simple support vector machine percep-
tron. Journal of VLSISignal Processing Systems, 2001. To appear.
[8] V. N.Vapnik. The Nature of Statistical Learning Theory. Springer, New
York,2"d edition, 2000.
[9] S. V. N. Vishwanathan and M. Narasimha Murty. Geometric SVM: A
fast and intuitive SVM algorithm. Technical Report IISC-CSA-2001-14,
Dept. of CSA, Indian Institute of Science, Bangalore, India, November
2001. Submitted to ICPR 2002.

0-7803-7278-6/02/$10.00 02002 lEEE 2398

_ _ _ _ ~