Research Proposal

Topic
A Review on Intrusion Detection System in Big Data environment

Author: Jamali Liaquat Ali

Bachelor of Computer Science Sindh University Pakistan 2016
Master of Software Engineering Nankai University China 2019

PhD applicant in the School of Computer Science and Technology

Xidian University China. 2021-2022

PhD Advisor: Doctoral Supervisor. Professor. Shen Yulong School

of Computer Science and Technology Xidian University China
Abstract
The rapid growth of Internet and Networking Technology has advantages and
disadvantages. Even though the business solutions would be faster and easier to
perform better than before, the security and privacy issues might be suffered along
with the massive growth of data over the Internet and Network nowadays. Data
security and network security have become the highest concerns because attackers
are doing various cyber-attacks by using advanced tools for intruding to the network
in order to steal confidential information or to make the network unavailable.
Intrusion detection system can monitor network traffic and flag when it detect a
malicious activity. This study report highlighted the importance of Big Data and the
security concerns. A review on the application of Intrusion Detection System and
the applications of Data Science and Machine Learning Algorithms was performed
in this study as well.
Keyword: Intrusion detection system, big data, Network Security, Data Security

2
Table of Contents
Abstract ......................................................................................................................2
Chapter (1) Introduction.............................................................................................4
1.1 Background and Overview ...............................................................................4
1.2 Aims and Objectives .........................................................................................5
1.3 Big Data Formation ..........................................................................................5
1.3.1 Big Data Concept and Characteristics ........................................................5
1.3.2 Sources of Big Data ....................................................................................5
1.3.3 Big Data Concerns ......................................................................................6
1.4 Types of Intrusions ...........................................................................................6
1.5 Intrusion Detection System...............................................................................7
CHAPTER (2) Related Work ....................................................................................8
2.1 Literature Review on Application of Intrusion Detection System on Big Data
Environment ............................................................................................................8
2.2 Literature Review on Application of Data Science and Machine Learning
Models on Intrusion Detection System...................................................................8
2.3 Discussion .........................................................................................................9
CHAPTER-3 Future Work and Conclusion ............................................................10
References ................................................................................................................11

3
 Chapter (1)
Introduction
1.1 Background and Overview
Nowadays, internet has been the main role in our every day's life. With the advanced
of new technologies and emerging of new devices that make use of big data flow
over the Internet has made network security become an important domain. While
trying to become a smart world with the utilization of smart technologies and
devices, the occurrence of intrusions in computing environment has caused a lot of
problems and issues for people, business or government that need to deal with
information privacy. As a result, the data security becomes the highest concern. Big
Data security is the processing of guarding data and analytics processes, both in the
cloud and on-premise, from any number of factors that could compromise their
confidentiality.
A huge amount of business companies has already been faced with many types of
malicious activities to lose money by being exposed of important private data and
confidential information. An intrusion compromises a computer system or a network
system by breaking the security and millions of computers and networks nowadays
are being victims of such practice. Thus, the network security has become a
prominent issues for various organizations.
An intrusion detection system has therefore become an important mechanism to
detect different types of malicious activities. It is the first line of defense system for
any network system. The system monitors users or network traffic and flags
suspicious and malicious activities.
Data science, in its fundamental form, involves studying, processing, and extracting
valuable insights from a set of information. By processing the enough network
transmission packets log data, malicious activity can be learnt with the help of data
science processes.
Therefore, many computer scientists, data scientists, and researchers have performed
the statistical analysis of network huge data to develop algorithms and models for
intrusion detection system. A wide number of intrusion detection and prevention
system and tools has been built in order to detect anomalies and abnormal traffic on
a network system or a networking device. A literature review on big data analytics
for intrusion detection system has done in this paper in order to meet one of the main

4
objectives which is to research how big data analytics has been applied in developing
intrusion detection system together with data science best practices.

1.2 Aims and Objectives

Combining my own research interest together with the knowledge and experience
learnt from the course, this study report has been carried out by the guidance of the
following aims and objectives:
• To research on how big data has been concerned with today's data security
and network security issues
• To know the importance of intrusion detection systems for the entire
networking system
• To know the importance of big data analytics on Intrusion detection system
• To find the challenges of Intrusion Detection System
• To research on the application of Data Science and Machine Learning
Algorithms for Intrusion Detection System

1.3 Big Data Formation

1.3.1 Big Data Concept and Characteristics
Big data can be referred as a complex and large data that has to be processed and
analyzed to apply in extraction valuable information for business and organizations.
Big data is growing massively and exponentially with time. It is so voluminous that
it cannot be processed or analyzed using conventional processing techniques. It
actually has a 4Vs' characteristics: Volume, Variety, Velocity and Veracity.
1.3.2 Sources of Big Data
Big data can mainly be generated from three primary sources: social data, machine
data and transactional data. Data has become enormous on social media platforms
because people nowadays cannot stay away from social media. Lots of demographic
information as well as other types of information can be found on social media as a
huge source. Moreover, the machine generated data become larger and larger with
the development of new technological devices such as IoT devices. In addition, in a
financial domain and business domain, transaction-typed data such as financial
electronic transactions and network transactions are generating enormously day by
day as well. This also depends on the development of technology.

5
1.3.3 Big Data Concerns
Data security and network security have become the main concerns with the huge
amount of data is flowing over the networks. On social media platforms, people are
updating their status timely without having awareness of their information privacy.
Many demographic related information can be found on social media. In business
related organizations as well as in governmental organizations, huge amount of
confidential information are flowing over the internet through the various networks
according to the needs of their business solutions. Therefore, the issues on the
privacy of personal data as well as the security of organizations' confidential data
has to be solved in an urgent and timely manner. Information leakage can be
occurred because of unsecure or less secure network system in an organization.
Therefore, the consideration on how to secure a network system is also a high
concern.

1.4 Types of Intrusions

Because of the less security on the networking and computing system in an
organization, the attackers can make intrusions to their system and get important
information. As a result, the confidential information can become misused by those
attacker for various purposes. According to [1], types of intrusion can be categorized
as follows:
• Active Attacks
• Passive Attacks
• Fraud
• Sniffer Attacks
• Side Channels Attacks
• Low Rate TCP Attacks
• Close in Attacks
• Exploit Attacks
• Insider Attacks
• Malicious Attacks
• DDOS/DOS Attacks
• Cyber Harassment
• Vulnerabilities Report
• Probing
• Content Related
6
 • SQL Injection
• Spam
• Remote to Local User
• Distributed Attacks
• Application Layer
• Compromised Key
• Password Attacks
An intrusion attacked by attackers may fall on one of the above mentioned
categories.

1.5 Intrusion Detection System

An intrusion detection system (IDS) monitors network traffic for suspicious activity
or malicious activity and issues warnings or alerts when such an activity occurs. It
can be a device or a software. According to Jyothsna Et.al [2], intrusion detection
systems can be any of these three types: Signature Based Detection (SBS), Anomaly
Based detection (ABS), and Network-based Intrusion Detection System (NIDS).
The mechanism of SBS normally searches through the network traffic and compare
a series of data packets to a signature or pattern which is stored in the signature
database. And it will issue alerts while it detects similar malicious bytes or packet
sequences. SBS therefore has a constraint that activities which has the same
signatures with malicious attacks' signatures previously stored in the database. [2]
Unlike SBS, the anomaly based detection can detect new types of unseen attacks.
The system learns normal behaviors from the network system and predefined those
behaviors on the system. The system can give alarms when it detects abnormal
behaviors or anomalies in the network system. Therefore, defining rules becomes
the major limitation for anomaly detection system. Rule defining process can be
affected by different protocols based on the utilization of different vendors. [2]
Network-based Intrusion Detection System can detect abnormal behaviors or a
change in the traffic load. The system normally operates at the network level and
performs monitoring on network traffic from all devices over the network. It
captures the network packets and compare to the patterns or signatures of the
database by scanning through network traffic. The NIDS is usually composed of two
components: the sensor and the management station [3] . The sensor on the network

7
segments monitors for suspicious traffic and sends alarms to the management station
then later shows to network operators.

CHAPTER (2)
Related Work
2.1 Literature Review on Application of Intrusion Detection System on Big
Data Environment
Cloud computing is a buzz technology for big data that has been applied widely
nowadays and normally it has distributed and open structures. These structures make
attackers targeted for intruding into the clouds. Ahmed Et.al [4] mentioned that
traditional intrusion detection system are not suitable to apply in cloud computing
environments because of its openness and specific essence. They performed a
systematic review for intrusion detection system for cloud computing. They
researched on possible solutions and informed the researchers the requirements of
IDS for cloud computing.
Since web is also the main source of big data, an intrusion detection system is
necessary for the web applications. Nancy Et.al [5] identified various dimensions for
comparing different web-based IDS from the perspective of functionalities and
architectural designs. They proposed a conceptual framework for web IDS and
compared its performance and functionalities to the existing web intrusion detection
systems: AppSensor, PHPIDS, ModSecurity, Shadow Daemon, and AQTRONIX
WebKnight.
Apart from clouds and webs, IoTs are also a source of big data. Wireless sensors are
very useful and highly applied in various fields including smart logistics in military
applications. Doumit Et.al [6] applied hidden Markov model in their proposed
system based on the structure of naturally occurring events. The system adapted the
norm of dynamics in its natural surroundings to detect abnormal activities.
2.2 Literature Review on Application of Data Science and Machine Learning
Models on Intrusion Detection System
Although a traditional intrusion detection system can detect many types of
suspicious activities, the system has a limitation in detection of zero-day attacks. It
can concern with the reduction of relatively high false alarm rates. Moustafa Et.al
[7]a proposed a scalable framework of lightweight and effective intrusion detection
system with three functionalities: capturing and logging, pre-processing and a new

8
statistical decision engine, called the Dirichlet mixture model based anomaly
detection technique. Their empirical results showed that the mixture model yielded
a higher detection rate and lower false alarm rate while compared to other three
techniques which were based on the correlation and distance measures that mimic
normal activities.
Sometimes, traditional supervised machine learning algorithms may slow for
analyzing of highly scalable analysis environment. Therefore, Junlong Et.al [8]
proposed the use of Extreme Learning Machine (ELM) and massively parallel
algorithm (MR-ELM) trained on KDDcup99 dataset for Network Intrusion
Detection System (NIDS). The result showed that ELM could achieve high accuracy
and could decrease training time. Traditional ELM could not handle with big data
whereas MR-ELM could handle without any losses. A MR-ELM outperformed a
normal ELM according to their experiment result.
Lidong Et.al [9] introduced some data mining and machine learning methods for
intrusion detection system. They performed big data analytics on huge and
heterogeneous featured of big data. The proposed data mining and machine learning
methods are emphasized on hybrid solutions and detection classification-nearest
neighbors (k-NN), Naï ve Bayes, boosted decision tree, and Support Vector Machine
(SVM) are presented in their study. Hybrid models for those algorithms were
presented as well. A hybrid k-means and k-NN model performed better than normal
k-NN. A hybrid DT-SVM model improves or yields equal performance than normal
SVM.
2.3 Discussion
After the completion of literature part, the following facts are extracted:
• The rapid growth of Internet has brought Data security and Network security
concerns together with the advancement of technology and the formation of
big data.
• Research on Intrusion Detection has become a hot topic among researchers.
• A wide range of research was also done on the application of Machine
Learning and Data Science models for Intrusion Detection System.
• Generally, hybrid models performed better than normal models.

9
 CHAPTER-3
Conclusion and Future Work
Data becomes massive along with the advancement in Internet and Networking
Technology. As a result, not only governmental organizations and other business
organizations but also individuals are facing with the privacy of their personal and
confidential information. The intruders are using various tools to penetrate the
organizations' network system and stealing confidential data for various purposes.
Since technology has been growth rapidly, the attackers are using advanced
technology to make intrusions as well. Therefore, the improvement of Intrusion
Detection is necessary in timely manner along with the explosion of big data. The
paper is conducted in research on the Intrusion Detection System and Big Data
Concerns. The study make a literature review on the applications of Intrusion
Detection System and the algorithms and models used for Intrusion Detection
System. The study found out that some traditional intrusion detection systems would
not perform well on handling some malicious activities such as Zero-day attack.
Therefore, more research is needed to improve the system.
My future work would be doing more research on the applications of Data Science
and Machine Learning Algorithms for Intrusion Detection System by performing
detail analytics of Big Data. In addition, making getting to know how the attacker
perform intrusions and understanding detail on types of intrusions would be my
future research as well.

10
References

[1] Anwar Shahid & Mohamad Zain, Jasni & Zolkipli, Mohamad & Inayat, Zakira
& Khan, Suleman & Anthony Jnr, Bokolo & Chang, Victor, "From Intrusion
Detection to an Intrusion Response System: Fundamentals, Requirements, and
Future Directions," Algorithms, vol. 10, no. 39, 2017.

[2] V.Jyothsna, V.V Rama Prasad, K. Mumivara Prasad, "A Review of Anomaly
based Intrusion Detection Systems," International Journal of Computer
Applications, vol. 28, 2011.

[3] Neyole Misiko Jacob, Muchelule Yusuf Wanjala, "A Review of Intrusion
Detection Systems," International Journal of Computer Science and
Information Technology Research, vol. 5, no. 4, pp. 1-5, 2017.

[4] Ahmed Patel, Mona Taghavi, Kaveh Baktiyari, Joaquim Celestino Junior, "An
intrusion detection and prevention system in cloud computing: A systematic
review," Journal of Network and Computer Applications, vol. 36, no. 1, pp.
25-41, 2013.

[5] Nancy Agarwal,Syed Zeeshan Hussain, "A Closer Look at Intrusion Detection
System for Web Applications," Security and Communication Netowrks, vol.
2018, 2018.

[6] S.S. Doumit, D.P Agrwal, "Self-organized criticality and stochastic learning
based intrusion detection system for wireless sensor networks," in Military
Communications Conference, 2003.

[7] Nour Moustafa, Gideon Creech, Jill Slay, "Big Data Analytics for Intrusion
Detection System: Statistical Decision-Making Using Finite Dirichlet Mixture
Models," Data Analytics and Decision Support for Cybersecurity, pp. 127-156,
2017.

11
[8] Junlong Xiang, Magnus Westerlund, Dusan Sovilj, Goran Pulkkis, "Using
Extreme Learning Machine for Intrusion Detection in a Big Data
Environment," 2014.

[9] L. Wang, "Big Data in Intrusion Detection Systems and Intrusion Prevention,"
Journal of Computer Networks, vol. 4, no. 1, pp. 48-55, 2017.

12