FULL PAPER

International Journal of Recent Trends in Engineering, Vol 2, No. 5, November 2009

Ensemble Voting System for Anomaly

Based Network Intrusion Detection
Mrutyunjaya Panda1 and Manas Ranjan Patra2
1
Department of ECE, Gandhi Institute of Engineering and Technology, Gunupur, Orissa-765022, India
Email: [email protected]
2
Department of Computer Science, Berhampur University, Orissa-760007, India
Email: [email protected]

Abstract— The growing dependence of modern One of the most active areas of research in
society on telecommunication and information supervised learning has been to study methods for
networks has become inevitable. Therefore, the constructing good ensemble of classifiers. It has
security aspects of such networks play a strategic been observed that when certain classifiers are
role in ensuring protection of data against misuse. ensembled, the performance is phenomenal
Intrusion Detection systems (IDS) are meant to
detect intruders who elude the “first line”
compared to the performance of the individual
protection. Data mining techniques are being used classifiers. Here, we propose a voting ensemble
for building effective IDS. In this paper we analyze classifier algorithm which is tested and results are
the performance of some data classifiers in a compared with other ensemble machine learning
heterogeneous environment using voting ensemble algorithms, including AdaBoost, MultiBoost, and
system with the purpose of detecting anomaly based Decorate with various base learner algorithms like
network intrusions. Experimental results using J48, SMO, Rule Learner. KDDCup 1999
KDDCup 1999 benchmark dataset demonstrate that benchmark dataset is used for the experimentation
the voting ensemble technique yield significantly and the results show that the proposed algorithm is
better results in detecting intrusions when compared
to other techniques.
promising and greatly outperforms existing
methods, achieving high detection rate with low
Index Terms— Intrusion Detection, Ensemble false alarm rate and more importantly take less
Learning, Voting Ensemble, ROC time to build the model.
The outline of the paper is as follows. A review of
I. INTRODUCTION the state-of-the-art on Intrusion Detection systems
(IDSs) is given in Section 2. Section 3 introduces
Computer networks are usually protected against technical analysis of the various machine learning
attacks by Intrusion Detection Systems (IDS). The approaches. The proposed method is presented in
traditional prevention techniques such as user Section 4. Various base learners used in this paper
authentication, data encryption, avoidance of are explained in Section 5. The evaluation on the
programming errors, and firewalls are only used proposed algorithm is carried out in Section 6 by
as the fist line of defense. But, if a password is comparing it with other ensembles. The paper is
weak and is compromised, user authentication concluded in Section 7.
cannot prevent unauthorized use. Similarly,
firewalls are vulnerable to errors in configuration II. RELATED WORK
and sometimes have ambiguous/undefined
security policies. They fail to protect against In [2], the authors have proposed various feature
malicious mobile code, insider attacks and reduction techniques like Principal component
unsecured modems. Therefore, intrusion detection analysis (PCA), Linear Discriminate Analysis
is required as an additional wall for protecting (LDA) and Independent Component Analysis
systems [1]. (ICA) in order to build an efficient network
Intrusion detection attempts to detect computer intrusion detection model in terms of detection
attacks by examining various process data on the accuracy and computation time. PCA and ICA
network. It is split into two categories, anomaly feature extraction approaches with Pareto-Optimal
detection systems and misuse detection systems. optimization is used in [3] to obtain a high
Anomaly detection is an attempt to search for performance intrusion detection system. The
malicious behaviour that deviates from established authors show that their proposed system
normal patterns. Misuse detection is used to outperforms standard SVM, PCA SVM and ICA
identify intrusions that match known attack SVM. In [4], the authors have proposed support
scenarios. In this paper, we propose a scalable vector machines (SVM) and neural networks (NN)
solution for detecting anomaly based network for intrusion detection. An evolutionary support
intrusion. vector machine for intrusion detection is proposed

© 2009 ACADEMY PUBLISHER

 FULL PAPER
International Journal of Recent Trends in Engineering, Vol 2, No. 5, November 2009
in [5]. In this, the authors have combined the performance of their base models in certain
evolutionary programming into support vector situations.
machines. They concluded that their model is able AdaBoost has performed very well in practice and
to detect new attacks as well as experienced is one of the few theoretically motivated
attacks. In [6], ensemble learning with various algorithms that have turned into a practical
base learning algorithms for detecting rare attacks algorithm. However, AdaBoost can perform
is proposed. In [7], the authors have proposed poorly when the training data is noisy, i.e. the
Bayesian approach in intrusion detection system. inputs and outputs have been randomly
It consists of building a reference model and the contaminated [11]. Noisy examples are normally
use of a Bayesian classification procedure difficult to learn. More details can be found in
associated to unsupervised learning algorithm to [12], [13].
evaluate the deviation between current and
reference behaviour. The authors have evaluated MultiBoosting
various machines learning algorithm for detecting It is another method of the same category that can
network intrusions in [8]. A novel ensemble of be considered as wagging committees formed by
classifiers for micro array data classification is AdaBoost [14]. Wagging is a variant of bagging;
done in [9]. They used this method using Particle bagging uses re-sampling to get the datasets for
swarm optimization (PSO) and EDAs (Estimation training and producing a weak hypothesis,
of Distribution Algorithms) on four benchmark whereas wagging uses re-weighting for each
datasets to produce the best recognition rates. In training example, pursuing the effect of bagging in
[10], the authors have modeled an intrusion a different way.
detection system using hybrid intelligent systems
using Decision Trees and SVMs in order to Decorate
maximize detection accuracy and minimize A new meta-learner DECORATE (Diverse
computational complexity. Ensemble Creation by Oppositional Re-labeling of
Artificial Training Examples) reported in [15]
III. TECHNICAL ANALYSIS uses an existing “strong” learner (one that
provides high accuracy on the training data) to
A. Ensemble learning build a diverse committee.
Ensembles of classifiers can perform better than
any individual classifier; this performance IV. PROPOSED METHODOLOGY
advantage can be attributed to three key factors In this section, first we discuss on the
[11]. disadvantages of the existing ensembles and then
The learning procedure for ensemble algorithms present the proposed approach which utilizes their
can be divided into the following parts. positive aspects while subduing their weaknesses.

Constructing base classifiers/base models: A. Analysis of Weakness of Existing Ensembles

The main tasks at this stage are:
• Data processing: prepare the input training The accuracy of boosting models remain the same
data for building base classifiers by perturbing after specific numbers of base models are
the original training data, and established, because of the checking mechanism
• Base classifier constructions: build base after each construction of base classifiers. The
classifiers on the perturbed data with a specific criterion in boosting stops further
learning algorithm as the base learner. In this construction while its error rate equals to 0 or
work, we have used SMO, J48, and ZeroR as greater than 0.5. Therefore, if the sequential
the base learners. construction halts after building 6 base classifiers,
Voting: The second stage of an ensemble same result will be obtained on evaluating over
method is to combine the base models built in the boosting models with any number greater than 6,
previous step into a final ensemble model. There because fundamentally these models are all
are different types of voting systems, the identical. As our intrusion detection dataset
frequently used ones are: weighted voting and un- consist of large amounts of intrusion classes, it
weighted voting. In the weighted voting system, helps the learning methods to generate a more
each base classifier holds different voting power. precise classifier that fits exactly on the training
On the other hand, in the un-weighted system, dataset. The checking criterion seriously
individual base classifier has equal weight, and the influences the diversity of boosted models by
winner is the one with most number of votes. forbidding further construction of base models.
B. Boosting and Decorate It was thought that using base learners can lead to
a better performance, but in [16], “many could be
Boosting algorithms are a class of algorithms that better than all” theorem indicates that this may not
have been mathematically proven to improve upon be the fact. It was shown that after generating a set

© 2009 ACADEMY PUBLISHER

 FULL PAPER
International Journal of Recent Trends in Engineering, Vol 2, No. 5, November 2009
of base learners, selecting some base learners Combining classifiers with this method is simple;
instead of using all of them to compose an it does not require any previous knowledge of the
ensemble is a better choice. behaviour of the classifiers nor does it require any
It is also worth noting that the computational cost complex methodology to decide. It only counts the
for building an ensemble comprising T base number of classifiers that agree in their decision
learners is roughly T times the cost of training a and accordingly decides the class to which the
single learner. So, from the computational input pattern belongs. This simplicity has a
complexity point of view, training an ensemble is drawback, however; the weight of the decision of
almost as efficient as training a single classifier. all the classifiers is equal, even when some of the
B. Design of Voting Ensemble system classifiers are much more accurate than others.
In this paper, we have used the un-weighted The voting ensemble is illustrated in Fig. 1.
majority voting for detecting network intrusions.

KDDCup’99
Intrusion detection Dataset

P S F SB DB AC

CL1 CL3 CL4 CL5 CL6

CL2

VOTING SYSTEM

Figure 1. The framework for the Ensemble of Voting System with six features.

Where, CLi refers to the ith classifier, P for N

⎧ 1, Cd = i
Protocol, S for Service, F for Flag, SB for Source Vi = ∑ α ∗ wd , α = ⎨ (1)
Byte, DB for Destination Byte and AC for Attack d =1 ⎩0, otherwise
Class.
Where N is the number of classifiers, i=1, 2... C
a. Feature Selection: In this proposed is the class label, Cd is the predicted class label
framework, six features were extracted by the d classifier, and wd is the weight of the d
from the full dataset. The features are classifier. For a given unknown pattern, the final
normalized to [0, 1]. class to be classified is determined by
b. Classification: Three Classifiers namely C
maximizing arg max j =1 V j .
AdaBoost, MultiBoost, Decorate Meta
classifiers were used with J48, SMO, ZeroR
as base learners. The details are given in the
next section. V. BASE LEARNERS USED IN OUR
c. Voting System: Different results will be EXPERIMENTATION
obtained from the different ensemble of
classifiers by using different features As mentioned above, we plan to design an
extracted from the KDDCup’99 intrusion ensemble machine learning classifier to address
detection dataset, and then these results are the network intrusion detection problem. The
put into the voting system. Each classifier critical factor to achieve this goal is the selection
has a weight to denote the contributions of of classifiers, since it is vital to achieve high
the classifier to the voting system. For each accuracy from ensemble techniques. We have
class to be identified, a weighted sum of used Decision Trees (J48), SMO (Sequential
base learners can be calculated as: Minimal Optimization), Rule Learning (ZeroR)

10

© 2009 ACADEMY PUBLISHER

 FULL PAPER
International Journal of Recent Trends in Engineering, Vol 2, No. 5, November 2009
specifically for this problem. This process is illustrated in Fig.2.

Boosting
Zero
R
Training dataset

Ensemble
Voting

Classifier based Output

Merge on the previous Classification
classifiers
Decorate

Testing data set

SMO
J48

Figure 2. Proposed Ensemble process with various base learners used

LAMSTAR IDS, SVM +Rocchio Bundling and

VI. EXPERIMENTAL SETUP AND EVALUATION OF SVM+DGSOT. However, the proposed methods
RESULTS provide better accuracy in detecting Probe and
R2L attacks in comparison to Hybrid DT+SVM,
In this work, we have used five class SMO, AdaBoost+SMO, LAMSTAR IDS and
classification methods to build our intrusion SOM IDS. It is also proposed to compare the
detection model. We have used a randomly systems performance in terms of Receiver
selected subset of KDDCup’99 intrusion
operating characteristics (ROC), which is a plot
detection dataset, which contains 1000 instances between detection rates (DR) with false positive
from each class proportional to size, except that rate (FPR), which is shown in Fig. 3. Compared
the smallest class is completely included. Full
to other measurements, ROC provides a visual
dataset is used for training and 10 fold cross tool for examining the trade off between the
validation for testing purpose. We have carried ability of a classifier to correctly identify positive
out our experiment on a Pentium 4 CPU 2.8GHz
cases that are incorrectly classified. At the same
with 512MB RAM. time, other evaluation metrics like root mean
It can be observed from Table 1 that average square error (RMSE), false negative rate (FNR)
accuracy of our proposed voting ensembles with
and F-Score are evaluated for different ensemble
ZeroR provides the best results in comparison to classifiers in Fig. 4. From, all these comparisons,
other ensemble classifiers already available. It it is imperative that our proposed voting
can also be observed that it is very fast which
ensemble classifiers with AdaBoost+48 and
takes only 0.03 second to build the network J48+ZeroR rule learner performs well in building
intrusion detection model. an efficient network intrusion detection model.
We have also compared our five class
classification results individually with other
approaches used by different authors in building
an efficient intrusion detection model in Table 2.
In that, we could see that our proposed voting
ensembles Voting+J48+ZeroR and
Voting+AdaBoost+SMO are amongst the best to
detect the normal classes. While detecting DoS

and U2R attacks, our method is the best in

comparison to Hybrid DT+SVM, SOM IDS,

11

© 2009 ACADEMY PUBLISHER

 FULL PAPER
International Journal of Recent Trends in Engineering, Vol 2, No. 5, November 2009
Table.1. Performance Comparison of various Classifiers optimal optimisation”, Information Technology
Classifier Avg. Build Time Journal, Vol.7, No.3, pp.510-515, 2008. ISSN-
Accura (Seconds) 1812-5638.
cy (%) [4] Srinivas Mukkamala, Guadalupe janoski, and
SVM [3] 95.56 Not Provided Andrew Sung, “Intrusion detection using support
PCA+SVM [3] 96.54 Not Provided vector machines and neural networks”, in Proc. Of
ICA+SVM [3] 87.14 Not Provided IEEE International conference on Neural
PARETO OPTIMAL [3] 96.56 Not Provided
Networks, IEEE Computer society Press,
pp.1702-1707, 2002.
SMO [17] 75.97 1962.25
[5] Sung-Hae Jun and Kyung-Whan Oh, “An
SVM Light [17] 88.55 24.10
Evolutionary support vector machine for intrusion
ISVM [17] 88.12 26.0
detection”, Asian journal Information
Tree SVM [17] 85.99 28.2 Technology, Vol.5, No.7, pp.778-783, 2006.
Array SVM [17] 90.78 45.0 [6] M.Panda and M.R.Patra, “Ensemble learning for
AdaBoost + SMO 96.63 230.52 detection of rare attacks”, in Proc. Of
MultiBoost + SMO 95.66 367.58 International conference on advances in computer,
Vote+AdaBoost+J48 (ours) 97.38 0.28 communication and control, India, pp.510-515,
Vote+AdaBoost+SMO(ours) 91.47 315.52 2009, ACM Press, USA. ISBN: 978-1-60558-
Vote+J48+ZeroR(ours) 97.47 0.03 315-8.
Vote+MultiBoost+J48(ours) 96.97 0.63 [7] M.Mehdi, S.Zair, A.Anou and M.Bensebti, “A
Bayesian networks on intrusion detection
Vote+SMO(ours) 96.97 42.06
system”, Journal of computer Science, Vol.3,
Vote+Decorate+J48(ours) 97.28 2.86
No.5, pp.259-263, 2007. ISSN: 1549-3636.
SVM+Rochilo Bundling 51.6 26.7 [8] M.Panda and M.R.Patra, “Evaluating machine
[18]
learning algorithms in detecting network
Clustering Tree +SVM [18] 69.8 13.18
intrusions”, International Journal of Recent
ESVM [5] 96.4 Not Provided
Trends in Engineering”, Vol.1, No.1, pp.472-
ESVM with bootstrap [5] 95.6 Not Provided 477, 2009. Academy Publisher, Finland.
Logistic [5] 89.8 Not Provided [9] Yuehui Chen, Yaou Zhao, “A novel ensemble of
Sigmoid [5] 80.4 Not Provided classifiers for micro array data classification”,
Applied soft computing, No.8, pp.1664-1669,
VII. CONCLUSION 2008. Elsevier.
[10] Sandhya Peddabachigari, Ajith Abraham, Crina
Grosan, and Johnson Thomas, “Modelling
In this paper, after investigating voting ensemble
intrusion detection system using hybrid intelligent
of classifiers, it is observed that classifiers based systems”, Journal of network and computer
on voting+J48+Rule learner and applications, Vol.30, no.1, pp.114-132, 2007.
voting+AdaBoost+J48 performs efficiently in Elsevier.
terms of high detection rate, low false positive [11] Dietterich, T.G., “Ensemble methods in machine
rate, less time taken to build the model, high F- learning”, Lecture notes in computer science
score, reasonably low RMSE in comparison to (LNCS), Vol.1857, pp.1-15, 2000.
existing ensemble classifiers, at the expense of [12] N.C.Oza, “AdaBoost2: Boosting with noisy data”,
some what high false negative rate. In F.Roli, J.Kittler, and T.windeatt (Edns.), Proc.
Of the 5th international workshop on multiple
A serious deficiency of ensemble methods is its
classifier systems, pp.31-40, 2004. Springer-
lack of comprehensibility, i.e., the knowledge Verlag, Berlin.
learned by ensembles is not understandable to the [13] Ratsch,G., Onoda, T., and Muller,K.R., “Soft
user. Improving the comprehensibility of margins for AdaBoost”, machine Learning,
ensembles is an important yet largely Vol.42, pp.287-320, 2001.
understudied direction. Exploration of [14] Webb,G.I., “MultyiBoosting: A technique for
methodology towards this will be our future combining boosting and wagging”, Machine
research direction. Learning, Vol.40,pp.159-196,2000.
[15] P.Melville and R.J.Mooney, “Constructing
diverse classifier ensembles using artificial
REFERENCES
training examples”, in Proc. Of IJCAI,
Acapulco, Mexico, pp.505-510, 2003.
[1] Denning D., “An Intrusion Detection model”, IEEE [16] Z-H Zhou, J.Wu and W.Tang, “Ensembling
transaction on S/W Engineering, Vol.8E-13, No.2, neural networks: many could be better than
pp.222-232, 1987. all”Artificial intelligence, Vol.137, No.1-2,
[2] V.Venkatchalam and S.Selvan, “Performance pp.239-263, 2002.
comparison of intrusion detection system [17] John Mill and Atsushi Inoue, “support vector
classification using various feature reduction classifiers network intrusion detection”, in Proc.
techniques”, International journal of simulation, Of 2004 IEEE international conference on fuzzy
Vol.9, no.1, ISSN-1473-8031(print), ISSN-1473- systems, WA, USA, Vol.1, pp.407-410, 2004.
804x (online). ISSN: 1098-7584.
[3] Yu, Gu, Bo Zhou and Jiashu Zhao, “PCA-ICA [18] Latifur Khan, Momouri Awad and Bhavani
ensembled intrusion detection system by Pareto- Thurasingham, “A new intrusion detection

12

© 2009 ACADEMY PUBLISHER

 FULL PAPER
International Journal of Recent Trends in Engineering, Vol 2, No. 5, November 2009
system using SVM and hierarchical clustering”, support vector classifiers”, In Proc. Of 2009
the VLDB journal, Vol.16, pp.507-521, 2007. IEEE international advance computing
DOI-10.1007/s00778-006-0002-5. conference (IACC-09)”, India, 2009, pp.926-
[19] M. Panda and M.R.Patra, “Anomaly based 931. ISBN: 978-981-08-246-5. IEEE, USA.
network intrusion detection using boosting

Receiver Operating Characteristics of different Vote+AdaBoost+J48

Ensemble Classifiers Vote+AdaBoost+ZeroR

1.2 Vote+SMO
Detection Rate(DR)

1
Vote+Decorate+J48
0.8
0.6 SVM+Rocchio Bundling
0.4 SVM+DGSOT
0.2
0 SOM IDS
-0.2 0 0.2 0.4 0.6 0.8 1 1.2 LAMSTAR IDS
False Alarm Rate(FPR)

Figure 3. ROC comparison among different classifier ensembles

Evaluation of classifiers Vote+AdaBoost+J48

Vote+AB+SMO
Vote+J48+ZeroR
Vote+MultiBoost+J48
Vote+SMO
Vote+Decorate+J48
SVM
PCA+SVM
ICA+SVM
FNR RMSE F-Score Pareto Optimal
SVM+Rocchio Bundling
CT+SVM

Figure: 4. Comparison of Evaluation metrics for different Ensemble Classifiers

Table.2. Performance Evaluation of different Ensemble of Classifiers

Hybr SMO AB+ SVM + SVM SOM LAMSTAR Vote + Vote+ Vote + Vote
id [19] SMO Rocchio + IDS IDS AB+ J48+ AB+ +Decor
DT+ [19] [18] D [2] [2] J48 ZeroR SMO ate+
SVM G (ours) (ours) (ours) J48
[10] S (ours)
O
T
[18]
Normal 99.7 97.4 97.8 98.0 95.0 93.6 99.4 99.0 99.6 99.6 99.3

Probe 98.5 66.2 71.0 34.0 97.0 60.5 95.6 78.1 65.6 66.0 63.0
7
DoS 99.9 100 99.0 11.0 23.0 95.9 98.6 100.0 100.0 100.0 100.0
2
U2R 48.0 54.3 67.0 27.0 43.0 20.6 31.6 67.0 67.0 67.0 67.0
R2L 37.8 38.1 30.0 88.0 91.0 10.9 37.7 59.0 59.0 47.0 59.0

13

© 2009 ACADEMY PUBLISHER