2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC 2019)

Research and Application of AdaBoost Algorithm

Based on SVM
Yanqiu Zhang1, Ming Ni1,*, Chengwu Zhang1, Shuang Liang1, Sheng Fang1, Ruijie Li1, Zhouyu Tan1
1.Sichuan Agricultural University, College of Information Engineering, Yaan, China
[email protected]

Abstract—In this paper, we use AdaBoost algorithm as a Both SVM and Adaboost play an important role in the
framework to design a new classification algorithm. The classification algorithm. SVM is designed for two-category
algorithm combines the characteristics of SVM and AdaBoost tasks. For small samples, so how to make it suitable for large-
algorithms. The SVM algorithm is used as a weak classifier scale data is the focus of research. The Adaboost algorithm, as
algorithm. AdaBoost is used to cascade multiple weak classifiers one of the integrated algorithms, has the advantage of being
to form a strong classifier, thus achieving more accurate able to change the weight of the classified error data and
classification. In the experiment, this paper uses SVM algorithm, reclassify it in the next training.
AdaBoost algorithm and the algorithm of this paper to carry out In this paper, a classification method based on SVM as weak
several comparison experiments. The experimental results show classifier and Adaboost as strong classification is proposed.
that the algorithm can achieve higher accuracy classification in
data sets of various sizes. At the same time, the algorithm is II. ADABOOST ALGORITHM AND FEATURES
moderate in calculation speed, so the new algorithm proposed in
this paper is better than the traditional SVM or AdaBoost A. AdaBoost
classification algorithm.

AdaBoost (Adaptive Boosting) [1] is a Boosting family of

Keywords—SVM; AdaBoost; Classifier algorithms. This type of learner is characterized by more
attention on the wrongly categorized samples during training,
I. INTRODUCTION
adjusting the sample distribution, and repeating this operation
Nowadays, information data is exploding. However only until the number of trainings of the weak classifier reaches a
0.5% of the total is useful. In order to find useful data quickly pre-specified value, completing the learning. And exit the loop.
and accurately in massive data, many classification algorithms AdaBoost algorithm process:
have emerged.

Table Ⅰ
Input: Training sample set N= {(X1 , Y1 ), ( X2 , 𝑌𝑌2 )…( X𝑁𝑁 , 𝑌𝑌𝑁𝑁 )}, iteration number T and the weak classifier algorithm

1 Initialize the weight distribution of the training samples:𝐷𝐷1 (𝑥𝑥) =

1
𝑛𝑛

2 for t = 1,2…T：

3 Learning a training data set with a weight distribution 𝐷𝐷𝑡𝑡 to obtain a weak classifier 𝐺𝐺𝑡𝑡 (𝑥𝑥)

Estimate the classification error rate of 𝐺𝐺𝑡𝑡 (𝑥𝑥) on the training data set: 𝜀𝜀𝑡𝑡 , go to the next step when 𝜀𝜀𝑡𝑡 is less
4 than 0.5, otherwise the loop ends (𝐺𝐺𝑡𝑡 (𝑥𝑥) ≠ f(𝑥𝑥), f(𝑥𝑥) is the real function)

978-1-5386-8178-7/19/$31.00 ©2019 IEEE 662

 1 1−𝜀𝜀𝑡𝑡
5 Calculate the weight of 𝐺𝐺𝑡𝑡 (𝑥𝑥): 𝛼𝛼𝑡𝑡 = ln
2 𝜀𝜀𝑡𝑡

𝐷𝐷𝑡𝑡 (𝑥𝑥) 𝑒𝑒𝑒𝑒𝑒𝑒�−𝛼𝛼𝑡𝑡 𝑓𝑓(𝑥𝑥)𝐺𝐺𝑡𝑡 (𝑥𝑥)�

Next weight distribution: 𝐷𝐷𝑡𝑡+1 (𝑥𝑥) =
𝑍𝑍𝑡𝑡
6
(𝑍𝑍𝑡𝑡 is normalization factor)

7 Get the final classifier: sign(∑𝑇𝑇𝑡𝑡=1 𝛼𝛼𝑡𝑡 𝐺𝐺𝑡𝑡 (𝑥𝑥))

AdaBoost algorithm flow chart

better than the next guess. Because the resulting weak
classifiers are linearly superimposed, they can be assigned
coefficients, which can also improve the classification effect.
The end of the loop is determined by the error rate and the
number of iterations. Usually, when the training reaches the
number of iterations, the learning is completed. The end of the
loop is determined by the error rate and the number of iterations.
Usually, when the training reaches the number of iterations,
the learning is completed.
However, the AdaBoost method is sensitive to noise data and
abnormal data. This means that if there is more noise data, the
time spent by the AdaBoost algorithm will increase and the
efficiency will decrease.
Many scholars have studied AdaBoost. Junming Chen,
Chunhua Yang, Hongqiu Zhu, Yonggang Li, Juan Gong [2]
fig. 1 found that the UV-visible spectroscopy and Adaboost can be
used simultaneously to determine the trace copper and cobalt
B. The Features and Application Of Adaboost in high concentration zinc solution. When Mücahid Barstuğan,
Rahime Ceylan [3] studied “The effect of dictionary learning
on weight update of AdaBoost and ECG classification”, they
The adaptation of AdaBoost is that at the beginning, all
used AdaBoost as a classifier to obtain an AdaBoost classifier
training samples have the same weight. The sample of the
based on dictionary learning, and classified ECG
previous classifier error will be used to train the next classifier,
(electrocardiogram) signals according to this classifier. The
that is, the probability of selecting the correct error sample to
dictionary-based learning method called DL-AdaBoost-SVM
enter the next weak classifier, and the probability of selecting
on the feature subset obtained by the DWT and Wilcoxon test
the paired sample will decrease. This approach enables
methods achieves very high optimal classification accuracy.
AdaBoost to focus on those samples that have been
misclassified. For AdaBoost, don't worry about overfitting. The
III. SVM ALGORITHM AND ITS APPLICATION
reason is that the test error exists independently of T. The larger
the margin on the training set, the lower the test error. And the
A. SVM
classification effect of the weak classifier used by AdaBoost is
better than the random guess (0.5 correct).
As long as it is greater than 0.5, it is considered to be slightly The SVM (Support Vector Machine) algorithm [4] is a two-
class model. As a common discriminating method, SVM

663
technology has many advantages in solving small sample, machine, where 𝑙𝑙2 is the most reliable hyperplane at this time.
nonlinear and high latitude pattern recognition problems. It has
been successfully applied in many fields such as text B. Application and Optimization Of SVM
categorization, image recognition and bioinformatics. The
most basic idea of the SVM algorithm is to find the partition
Now, SVM is being used in many areas. Abbas, Raza, Zaidi,
hyperplane in the sample space based on the training set and
Khan, Heinemann [5] applied SVM to classify epileptic signals
separate the samples of different categories. The most basic
and produced good results. Kiyoumars Roushangar, Roghayeh
idea of the SVM algorithm is to find a partitioning hyperplane
Ghasempour [6] confirmed the ability of the SVM model to
in the sample space based on the training set, separating the
predict relative energy dissipation.
samples of different categories. The division of the hyperplane
The SVM algorithm needs to find a kernel function that can
can be described by the following linear equation:
be mapped. When the dimension is too high, it is particularly
𝝎𝝎𝑻𝑻 𝒙𝒙 + 𝑏𝑏 = 0 (1)
difficult to find a qualifying kernel function. So, the SVM
As can be seen from the above expression, the hyperplane is
algorithm can only play its own advantages when training
determined by the normal vector ω and the displacement b.
small-scale data. In addition, the SVM algorithm is very
Now assume that a point p is known on the plane, the distance
sensitive to the missing data of the sample, and direct use can
from p to the hyperplane is represented by d, and the formula
lead to over-fitting of the training. In order to ensure that the
for calculating d is as follows:
data is 100% categorizable, we also need to discuss slack
�𝝎𝝎𝑇𝑇 𝒙𝒙+𝑏𝑏�
𝑑𝑑 = (2) variables [7], kernel functions, regularization and other very
‖𝝎𝝎‖
complex problems on the SVM model.
In order to allow the hyperplane to classify the training
In the improvement of SVM algorithm, many scholars have
samples correctly, we hope that the two heterogeneous support
proposed a very mature and improved algorithm and
vectors on both sides of the hyperplane can have a "maximum
successfully implemented the algorithm. Liu Xintong [8] and
interval:
her team proposed a short-term bus passenger flow prediction
𝟐𝟐
max = ‖𝝎𝝎‖ (3) model based on SVM-KNN, which further exploited the
advantages of fast classification of support vector machine
For the convenience of calculation, it is usually rewritten
(SVM) and high prediction accuracy of K-nearest neighbor
as:
algorithm (KNN). Experimental results show that The
1
min = 2 ‖𝝎𝝎‖2 (4) comprehensive mean absolute error (MAE) of the SVM-KNN
algorithm is 8.437, the integrated mean square error (MSE) is
This is the basic model of the support vector machine.
10.725, the combined mean relative error (MAPE) is 8.8%, and
the integrated mean square relative error (MSPE) is 11.3%. The
commonly used RBF models have reduced the errors of MAE,
MSE, MAPE and MSPE by 3.646, 3.631, 4.6% and 5.5%,
respectively. Dr. Chen Zuoqi [9] found that the regional growth
support vector machine-bidirectional Markov random field
(RSVM-BMRF) model can effectively realize large space in
the "multi-scale night light remote sensing image based multi-
scale urban spatial morphological structure analysis" The long-
term urban built-up area under the scale is extracted and has a
more stable performance in time series.

fig. 2: The figure above is an illustration of a support vector IV. SVM- ADABOOST ALGORITHM

664
 selected in the next classification.
Typically, when using SVM, the model used is a support f) Repeat the operations of C, D, and E. When the
vector machine with a kernel function + soft interval. If the number of iterations is reached, or the error rate
selection of the kernel function is too powerful, or if the reaches the required level, complete the training.
requirement interval is too large, insisting that the data be g) Linearly combine each weak classifier to form a
completely separated, there will be a fitting problem at this time. strong classifier.
At the same time, SVM has a strong generalization ability, in h) Test the test set with a strong classifier.
the processing of small sample data, in order to best play the
generalization ability. AdaBoost fully considered the weight of B. Experiment
each weak classifier, and made good use of the weak classifier
for cascading.
This part will experiment with the SVM-AdaBoost
Based on the above points, this paper proposes an SVM
algorithm proposed in this paper, and compare it with SVM and
algorithm and AdaBoost algorithm, and proposes an algorithm
AdaBoost separately in terms of training time and accuracy.
that uses SVM as a weak classifier to integrate weak classifiers
a) Experimental Environment
into a strong classifier using AdaBoost algorithm. It is called
Operating System: Windows 10 Home Chinese Edition
SVM-AdaBoost algorithm.
CPU: intel core i5-7300HQ
GPU: GeForce GTX 1050
A. SVM-Adaboost Algorithm Process Memory: 8G
Experimental operating environment: MATLAB R2016a
a) Perform a simple scale operation on the sample b) Experimental Results
points in the determined total training sample set N, The characteristics of the training data set used in this
and select the appropriate kernel function. experiment are that the number of features and the number of
b) Initialize the weight distribution, enter the weak samples are relatively large, so the SVM kernel function and
classifier algorithm SVM, and the number of the kernel function of the SVM-AdaBoost weak classifier in
iterations T. this paper select the linear kernel. There are 2000 data sets in
c) Select a sample point with the appropriate number the data set, and each category is evenly distributed. Each time
of sample points and the same sample point to form the weak classifier selects 200 data for training to ensure that
a small sample set during training. the SVM exerts its advantages, the number of iterations is 15
d) Use cross-validation to choose the best parameters. times. This experiment does not set the minimum error rate,
Train the small sample set with the best parameters that is, the conditions for ending the training. It is to generate
obtained to get the SVM model. 15 weak classifiers.
e) Change the weight distribution of the erroneous data The training model results are as follows:
in the last small sample set so that it is easier to be

Table Ⅱ

Algorithm Training time (s) Accuracy

SVM 18.6 85.15%

665
 AdaBoost 23.8 81.23%

SVM- AdaBoost 19.2 91.86%

The training results show that the SVM-AdaBoost algorithm using AdaBoost algorithm, and tests the performance of the
is more efficient than the SVM or AdaBoost algorithm alone algorithm. Satisfactory results are obtained. Compared with the
when training large data sets, considering the training time and other two algorithms alone, it is more accurate and the
accuracy. calculation time is medium. Therefore, the subsequent research
The trained strong classifier is used for the test set of can be based on the algorithm to improve the calculation speed
different data quantities, and the results are as follows: of the algorithm. The result is only the simulation result, and
the effect of the actual application needs to be studied again.
References

[1] Freund, Y. and Schapire, R. (1997) A Decision-Theoretic Generalization

of Online Learning and an Application to Boosting. Journal of Computer
and System Sciences, 55, 119-139.
[2] Junming Chen, Chunhua Yang, Hongqiu Zhu, Yonggang Li,Juan Gong.
Simultaneous determination of trace amounts of copper and cobalt in
high concentration zinc solution using UV–vis spectrometry and
Adaboost[J]. Optik,2019,181.
[3] Mücahid Barstuğan,Rahime Ceylan. The effect of dictionary learning on
weight update of AdaBoost and ECG classification[J]. Journal of King
Saud University - Computer and Information Sciences,2018,

fig. 3 [4] Vapnik, V.N. and Lerner, A.Y., 1963. Recognition of patterns with help

After several experiments, the SVM-AdaBoost algorithm of generalized portraits. Avtomat. i Telemekh, 24(6), pp.774-780.

proposed in this paper is compared with the SVM and [5] Abbas, Raza, Zaidi, Khan, Heinemann. Seizure forecasting using single

AdaBoost algorithms. When the training model is trained by a robust linear feature as correlation vector of seizure-like events in brain

large sample set, the SVM-AdaBoost algorithm remains good slices preparation in vitro[J]. Neurological Research,2019,41(2).

in both the training set and the test set efficient features. [6] Kiyoumars Roushangar, Roghayeh Ghasempour. Evaluation of the

However, the accuracy of the SVM in the test set due to the impact of channel geometry and rough elements arrangement in

over-fitting problem decreases with the increase of the sample. hydraulic jump energy dissipation via SVM[J]. Journal of

Although the accuracy rate of the AdaBoost algorithm is not as Hydroinformatics,2019,21(1).

fast as that of the SVM, it takes too much time for training and [7] Smith, F.W., 1968. Pattern classifier design by linear programming.

the accuracy rate is also not ideal. IEEE Transactions on Computers, 100(4), pp.367-372.
[8] [8] Liu Xintong, Huang Xiaolong, Xie Binglei. Prediction of Short-term

V. CONCLUSIONS Bus Passenger Flow Based on SVM-KNN[J]. Traffic Information and

Security, 2018, 36(05): 117-123.

This paper proposes an SVM algorithm and AdaBoost [9] Chen Zuoqi. Multi-scale urban spatial morphological structure analysis

algorithm, and proposes an algorithm that uses SVM as a weak based on multi-source nighttime remote sensing image[D]. East China

classifier to integrate weak classifiers into a strong classifier Normal University, 2017

666