2016 Cybersecurity Symposium

Neural Network Analysis of System Call Timing

Abstract—In the realm of cybersecurity, rootkits pose a study. Our novel method uses neural networks and behavior
credible threat to individuals, corporations, and governments. based analysis to classify data as coming from a system that
Through various techniques, rootkits are not only able to infect is either not infected or infected with a rootkit. We conclude
computer systems, but often times are able to remain undetected
in a host for an extended amount of time by manipulating with a discussion of results, limitations, and future work.
system software. The purpose of this paper is to describe what a
rootkit is, how they operate, and how they relate to other types II. W HAT A RE ROOTKITS
of malware. Historical data and statistics will be presented in Rootkits, also known as stealth malware, differ from other
order to show how rootkits have been employed in cyber attacks. forms of malware because they have the ability to gain root
Different types of rootkits, including user, kernel, and hypervisor
rootkits will be described, as well as the various methods used to access to an operating system while remaining undetected [4].
defend against rootkits. We will then present a case study where The method by which rootkits are able to remain undetected is
neural networks were used to analyze the behavior of a system called “hooking”. The Microsoft Developers Network defines
both not infected and infected with a rootkit, and categorize the a hook as “a point in the system message-handling mechanism
resulting system calls as anomalous or not. where an application can install a subroutine to monitor the
message traffic in the system and process certain types of
I. I NTRODUCTION
messages before they reach the target window procedure” [25].
In this paper we focus on the threat rootkits pose from a By using a hooking strategy, a rootkit can record, simulate,
cybersecurity perspective. The term rootkit refers to a program or intercept keystrokes or mouse clicks, intercept system
that has root privileges on a computer, along with the tools it communications, and perform a variety of other actions.
needs to execute. Having root privileges means a program has Other methods used to remain concealed include replacing
access to the root account on a system, and gives the program system binaries, replacing libraries, and subverting kernel data
the ability to modify files. This is akin to “admin permission” structures [32]. Here, kernel refers to the main component of
on a Windows operating system. The negative connotation an operating system. The kernel is a program responsible for
associated with rootkits is not on the rootkit itself, but on managing I/O request, data management, and communication
cyberattacks that use rootkits to accomplish their goal. Rootkit with other devices.
technology has been used by US law enforcement, such as the There are two primary reasons rootkits are deployed in a
FBI’s Magic Lantern for legitimate reasons. The complexity cyberattack. The first reason is to establish remote command
and use of rootkits has increased over time. Between 2001 and and control, and the second is to eavesdrop [13]. Once an
2005, the use of rootkits on Windows based systems increased attacker has remote control over a computer they are able
by 2300%, and from 2000 to 2005, the complexity of rootkits to execute processes and controlling system files. If the goal
grew by 400% [29]. In 2004, of the 15000 trojans reported by is to eavesdrop, the attacker will simply spy on the infected
McAfee, 87% were rootkits [29]. computer. Rootkits that eavesdrop are able to read emails and
In this paper we present a general overview of what rootk- record keystrokes, which the attacker can use to acquire sen-
its are and the various methods used in designing rootkits. sitive information. In both scenarios, the attacker is interested
Keywords will be defined, different types of rootkits will be in long term access to the system. Rootkits are considered
described, and classification methods will be discussed. A post-exploits because they are brought in toward the end of
brief survey of platforms of deployment will be discussed, and the attack cycle. An example of this is the use of rootkits in
statistical data will be provided in order to show the magnitude advanced persistent threats (APT). One of the most recent and
of the risk posed by rootkits. A survey of the current methods buzz worthy APT style attacks that employed a rootkit was
used to defend and detect against rootkits will be provided. We Stuxnet. Stuxnet used a rootkit to hide itself on a Windows
will then present a novel approach to identifying the presence operating systems, and was the first known rootkit that could
of a rootkit in a system, and display its effectiveness in a case hide injected code located on a PLC [10].

978-1-5090-5771-9/16 $31.00 © 2016 IEEE 1

Fig. 1: Feed Forward Network Analyzing Call ”open”.

A widely used method to classify rootkits was proposed hooking methods to remain concealed [35]. User level rootkits
by Joanna Rutkowska [30]. This taxonomy classifies malware have a robust attack vector, and the techniques used are
into 4 categories, 0-III, and is based on the behavior of the somewhat dependent on the host operating system. Injection
malware in a system. Type 0 rootkits are the least dangerous techniques include injection by application extension, injection
because they do not compromise the operating system or any by message filtering, injection by debugging subsystems, and
applications. They can also be easily detected by anti-forensic injection by application vulnerability.
technology. Type I and II rootkits take steps to conceal their
presence. Type I affects static resources, such as in-memory
code sections, and type II affects dynamic resources, such as B. Kernel Level Rootkits
data sections of running processes [30]. Type III rootkits do Unlike user level rootkits, kernel level rootkits have unre-
not rely on the host operating system because they run on stricted access to a system. In order for a kernel level rootkit
virtualized hardware. Because they are not using any of the to execute, the code must be loaded directly to the kernel.
host software, everything, including drivers has to be built This is usually achieved by installing a kernel mode device
in. Various other methods have been used to classify rootkits. driver [1]. Once the driver is installed, the rootkit can hook
A common method is to classify rootkits as either user or API calls to remain concealed. Delivery techniques used by
kernel, while other methods classify rootkits as kernel, library, kernel level rootkits include system service descriptor table
or binary [32]. modification, direct kernel object modification (DKOM), filter
device drivers, kernel mode system service handler modifica-
A. User Level Rootkits
tion, runtime detour patching, or I/O request packet function
User level rootkits only affect processes and applications. table modification [1]. In direct kernel object manipulation,
They are able to hide from process viewers by hooking the rootkit hides system resources by manipulating data the
the viewer process [1]. User level rootkits use a method operating system uses to keep track of resources [34]. This
called dynamic linked library (DLL) injection to infect a is often times accomplished by unlinking objects from list
host system. The host system becomes infected when an that are managed by the operating system. Once the object
application or process calls the infected DLL. The DLL, is unlinked from the list, it is no longer visible to the API.
including the infected code, is executed in the process or In kernel object hooking, rootkits take over the control flow
applications reserved memory. At that point, the rootkit is able of the kernel, while direct kernel object manipulation subverts
to infect other applications and processes while using various the kernel by directly modifying dynamic data objects [37].

2
C. Hypervisor Rootkits havioral techniques attempt to analyze how a system operates
Rootkits have also begun to infect virtual machines. Hard- and try to identify deviations. Integrity detection looks for
ware based virtual machine (HVM) rootkits are able to turn an changes in files or system components in memory. This
operating system into a virtual machine and take full control technique also collects data and compares it to a baseline.
of the system [5]. HVM rootkits are hypervisors making use Signature detection looks for code in the system that has been
of the hardware virtualization features of the CPU. They used and recorded in previous attacks. A problem with this
are not flawed by the shortcomings of software-based virtual method is it is only able to detect rootkits after they have
machines [12]. The first known hypervisor rootkit was called been observed and recorded.
Blue Pill [26]. Blue pill was able to change the state of the Kruegel et al. [18] proposed a method of detecting rootkits
operating system, control timing resources, and monitor inputs through binary analysis. In there method they statistically
and outputs without using hooking strategies. analyze kernel module binaries to determine if the kernel
module interface has been modified. In [36] Wang et al.
D. Rootkits on Smartphones suggest detecting hooks in a system and protecting the hooks
Smartphones have also become a target for attackers. A from the rootkit. If a rootkit is unable to use a hook it will
2008 report by McAfee Security stated almost 14% of mobile be easily detected by system administrators or anti-malware
users worldwide have been infected or know someone who software. In a similar paper, Wang et al. [37] use a hypervisor
has been infected by mobile malware [23]. In a 2012 article based system that is a able to identify and defend thousands
in Network World, researchers stated “One-quarter of more of kernel hooks in an operating system with a minimal impact
than 400,000 Android apps examined in the Google Play store on the performance of the system. This method identifies a
pose security risks to mobile device users” [24]. An interesting solution to the so called protection granularity gap, which
difference between rootkits on smartphones and rootkits on refers to the fact that kernel hook protection requires byte
personal computers is the use and implications of user level level protection, but hardware provides page level protection.
rootkits. They can be more severe on smartphones because In [27], Riley et al. note that most methods that defend
of the nature of the sensitive information commonly found against rootkits are focused on detection, which requires an
on user applications on smartphones. The threat of infection attack to have already occurred on a system. Their paper fo-
is even greater on smartphones because of the growing trend cuses on a rootkit prevention system they refer to as NICKLE.
of free, open source platforms and third party applications NICKLE is a virtual machine monitor system that prevents
available for download at little or no cost to the user. Rootkits unauthorized code execution for guest operating systems.
on smartphones are able to access interfaces and information Kunk et al. use a similar VMM based detection method
that are unique to smartphones, including GPS, messaging, for smart phones [19]. Their method, which they refer to
and voice [20]. Unfortunately, due to the restrictions on as a “level-below approach”, uses a virtual machine monitor
resources, power, and functionality, many of the traditional to establish a kernel module that executes as an extension
techniques used to defend personal computers against rootkits of the Linux operating system, which implements various
are not effective for use on a smartphone. In [20] Polla et security mechanisms used to detect rootkits without the risk of
al. classify the methodologies used in smartphone attacks into compromise. This method is able to not only defend against
6 categories: wireless, break in, infrastructure based, worm kernel level rootkits, but also user level rootkits.
based, botnet, and user based. Of these methodologies, the Methods have also been proposed for detecting virtual
one that most pertains to rootkits is wireless. An alternative machine based rootkits. Xie et al. [38] suggest a method
classification approach is suggested by Felt et al. [11]. They of anomaly detection through cross verification of compo-
classify malware by the behavior it exhibits. They note the nents in a virtual machine. The hypervisor can reconstruct
most common behaviors exhibited by malicious software are the virtual machine’s execution states and learn information
collecting user information and sending premium rate mes- such as running processes, active network connections, and
sages. opened files. It then makes cross verification checks to detect
In [8], Dixon and Mishra state techniques used to defend anomalies. Desnos et al. [5] suggest a statistical analysis of
smart phones can be categorized into three classes: monitoring system behavior to detect the presence of a rootkit.
the volume of traffic, provide system level defense, and use of
IV. C ASE S TUDY: D ETECTING ROOTKITS WITH N EURAL
sophisticated admission control mechanisms. To overcome the
N ETWORKS BASED ON S YSTEM C ALL T IMES
limitations mentioned above, they suggest running detection
and prevention software through a personal computer that is Due to the various hooking methods discussed in this paper,
connected to and synchronized with the smartphone. Dixon et rootkits are able to hide within a host system without detection
al. also approached rootkit and malware detection by analyzing from traditional security. However, they must still make system
the power consumption of the smartphone [6][9]. calls like any other process. In our case study we attempt
to identify the presence of rootkits within a system using
III. D EFENDING AGAINST AND D ETECTING ROOTKITS a behavior based analysis of system call times. We initially
There are many methods for identifying rootkits, including attempted to use standard statistical measures on the data sets,
behavioral, integrity, and signature based detection [33]. Be- such as threshold based identifiers on values such as mean

3
 (a) ROC (b) Confusion Matrix

Fig. 2: Feed Forward Network Analyzing multiple system calls.

and variance, but due to the nonlinear nature of the data, these types. This data set contained approximately 2000 calls. Of
methods were unsuccessful. Therefore, we turned to the use of the 2000 calls, approximately 33% were infected. All training,
neural networks, which are capable of approximating nonlinear test, and validation sets were randomly separated from the data
data, to classify system calls as infected or not infected. sets in the amounts of 70%, 15%, and 15% respectively.
The software used for our analysis was MATLAB release
A. Analysis
R2015bSV and the MATLAB Neural Network Toolbox. Our
The first stage of our case study involved collecting sys- experiment involved two network architectures. The first was
tem calls. The computing lab at the University of South a static feed forward architecture, and the second was a
Alabama has a hypervisor based system where we are able recurrent nonlinear auto-regressive architecture. Both networks
to collect system calls and there corresponding execution used the Levenberg Marquardt algorithm (LMA) for training,
times on an operating system. We are then able to infect which is traditionally used for solving nonlinear least squares.
the system with a rootkit, and collect system calls from the The error function used was Mean Squared Error (MSE).
infected operating system. For further details on the hypervisor Various amounts of hidden layers and neurons were used in the
based call collection environment see [15]. For our analysis, analysis. However, due to the computational complexity and
we used the KBeast rootkit. More information on KBeast memory requirements of LMA, we were required to limit our
can be found at [14]. A total of six collections were run. networks to a relatively small size. The best results came from
Four of the experiments collected uninfected data, and two a combination of radial base and hyperbolic tangent transfer
of the experiments collected infected data. All experiments functions.
were performed independent of the other experiments, and
all collections came from the same Ubuntu operating system.
B. Results
For training purposes, once all calls were collected they were
assigned a value of zero for infected or one for uninfected. The first network tested was a feed forward network.
All calls were pooled together and we generated three data Initially it was tested against only one call (open). Figure one
sets. The first set contained infected and uninfected calls and depicts the results. The network was able to correctly classify
their corresponding execution times from a single call type. 82.8% of the calls. The lowest MSE for validation was .1615,
This data set consisted of approximately 300 uninfected and and was slightly lower for the test case. The algorithm trained
200 infected calls. The second data set contained three call on 70% of the data set, validated on 15%, and tested on 15%.
types, which consisted of approximately 600 uninfected and This network consisted of two hidden layers. The first hidden
400 infected calls. The third set contained all forty four call layer had five neurons with radial base transfer functions, and

4
 (a) ROC (b) Confusion Matrix

Fig. 3: Recurrent Network Analyzing multiple system calls.

the second had three neurons with hyperbolic tangent transfer radial base/hyperbolic tangent transfer function respectively.
functions. The testing accuracy for the recurrent network was 95.9%
Next, the same network was tested against a data set and a MSE of .036. The improvement in performance is
containing three different call types. Figure two depicts the possibly due to the fact that this type of network architecture
results. The network was able to correctly classify 67.7% of is historically better at modeling nonlinear dynamical systems
the calls. There are several possible reasons for the reduction than the static feed forward methods [17][22][39]. This is due
in accuracy. One reason is the possibility that some system to the fact that the network trains not only on the current data
calls are more affected or used by the rootkit than others. For points, but also the results of previous data points. In essence,
example, we consider it reasonable to assume calls such as the network considers the previous state when determining the
open, close, read, and write are commonly used by a rootkit, current state. Maintaining internal states allows the network to
while other calls, such as futex or nanosleep might not be used model dynamical temporal behavior more effectively.
as often. Another reason is complexity and dimensionality. The
more complex a data set and the more dimensions or features V. F UTURE W ORK
contained within the data set, the larger the neural network Further analysis with neural networks should be conducted.
must be in order to make accurate decisions. For this data As mentioned, we were very restricted with the size of the neu-
set, we increased the size of the network to two hidden layers ral network. The use of parallel computers with large amounts
each with ten neurons. It is possible better results could have of memory and processing power such as state funded super
been obtained with a larger network. The data set containing computers should be used to accommodate larger networks and
all forty four calls was also tested, however there was further data sets. Other machine learning techniques, such as decision
decline in performance. trees, support vector machines, and unsupervised methods
The second network tested was the recurrent network. The such as random forest should be tested. Other behavioral
same multi-call data set used on the feed forward network techniques should also be considered, such as CPU power
was used to test the performance of the recurrent network. consumption.
Far better results were achieved with the recurrent network
versus the feed forward network. The recurrent network was VI. C ONCLUSION
then tested against the data set containing all forty four calls, In this paper we have presented a general overview of what a
and again showed far better performance as depicted in figure rootkit is, and the various methods used in designing rootkits to
three. The network had two hidden layers with 5/5 neurons and achieve the goals of an attacker. A brief survey of the platforms

5
rootkits can be deployed on was presented. When available, [21] Levine, John, Julian Grizzard, and Henry Owen. “A methodology to
detect and characterize kernel level rootkit exploits involving redirection
statistical data was provided in order to show the magnitude of the system call table.” Information Assurance Workshop, 2004. Pro-
of the risk posed by rootkits. A survey of the current methods ceedings. Second IEEE International. IEEE, 2004.
used to defend and detect against rootkits was reviewed, and [22] Lin, Tsungnam, et al. “Learning long-term dependencies in NARX
recurrent neural networks.” Neural Networks, IEEE Transactions on 7.6
results were provided. Further, we present a novel approach (1996): 1329-1338.
to detecting rootkits within a system. Using neural networks, [23] McAfee Security McAfee Mobile Security Report 2008, McAfee,
we were able to successfully identify the presence of rootkits www.mcafee.com/us/research, 2008.
[24] Messmer Ellen Security research labels more than 290,000
within an infected system with a high degree of accuracy. Our Google Play Android apps as ’high-risk’, Network World,
case study proved machine learning techniques, such as neural http://www.networkworld.com/news/2012/110112-google-play-apps-
networks, provide a reasonable alternative to signature based 263805.html, 2012-11-12.
[25] Microsoft Developers Network, Hooks, https://msdn.microsoft.com/en-
detection methods by focusing on the behavior of the system. us/library/ms63258928VS.8529.aspx
[26] Ou, George. “Blue Pill: The first effective Hypervisor Rootkit”,
R EFERENCES http://www.zdnet.com/article/blue-pill-the-first-effective-hypervisor-
rootkit/!
[1] Aditya Kapoor, Ahmed Sallam, Rootkits Part 2 of 3: A Technical Primer, [27] Riley, Ryan, Xuxian Jiang, and Dongyan Xu. “Guest-transparent pre-
McAfee, 2007-04. vention of kernel rootkits with vmm-based memory shadowing.” Recent
[2] Bickford, Jeffrey, et al. “Rootkits on smart phones: attacks, implications Advances in Intrusion Detection. Springer Berlin Heidelberg, 2008.
and opportunities.” Proceedings of the eleventh workshop on mobile [28] Rodionov, DH Eugene, Aleksandr Matrosov, and David Harley. “Bootk-
computing systems and applications. ACM, 2010. its: Past, Present and Future.” VB Conference. 2014.
[3] Blunden, Bill. The Rootkit arsenal: Escape and evasion in the dark corners [29] Rootkits Part 1 of 3: The Growing Threat, McAfee, 2006-04.
of the system. Jones and Bartlett Publishers, 2012. [30] Rutkowska, Joanna. “Introducing stealth malware taxonomy.” COSEINC
[4] Bray, Rory, Daniel Cid, and Andrew Hay. OSSEC host-based intrusion Advanced Malware Labs (2006): 1-9.
detection guide. Syngress, 2008. [31] Schuster, Andreas. “Searching for processes and threads in Microsoft
[5] Desnos, Anthony, ric Filiol, and Ivan Lefou. “Detecting (and creating!) Windows memory dumps.” digital investigation 3 (2006): 10-16.
a HVM rootkit (aka BluePill-like).” Journal in Computer Virology 7.1 [32] Shah, Alkesh, and J. Giffin. Analysis of rootkits: Attack approaches and
(2011): 23-49. detection mechanisms. Technical report, Georgia Institute of Technology,
[6] Dixon, Bryan, et al. “Location based power analysis to detect malicious 2008.
code in smartphones.” Proceedings of the 1st ACM workshop on Security [33] Sparks, Sherri, and Jamie Butler. “Shadow Walker: Raising the bar for
and privacy in smartphones and mobile devices. ACM, 2011. rootkit detection.” Black Hat Japan 11.63 (2005): 504-533.
[7] Dixon, Bryan, and Shivakant Mishra. “Power based malicious code [34] Tsaur, Woei-Jiunn, Yuh-Chen Chen, and Being-Yu Tsai. “A new win-
detection techniques for smartphones.” Trust, Security and Privacy in dows driver-hidden rootkit based on direct kernel object manipulation.”
Computing and Communications (TrustCom), 2013 12th IEEE Interna- Algorithms and Architectures for Parallel Processing. Springer Berlin
tional Conference on. IEEE, 2013. Heidelberg, 2009. 202-213.
[8] Dixon, Bryan, and Shivakant Mishra. “On rootkit and malware detection [35] Vikram Kumar, Rootkit- An Intruder Living in Your Kernel,
in smartphones.” Dependable Systems and Networks Workshops (DSN- http://www.symantec.com/connect/articles/rootkit-intruder-living-your-
W), 2010 International Conference on. IEEE, 2010. kernel, Symantic, 2009-16-08.
[9] Dixon, Bryan, Shivakant Mishra, and Jeannette Pepin. “Time and location [36] Wang, Zhi, et al. “Countering persistent kernel rootkits through system-
power based malicious code detection techniques for smartphones.” Net- atic hook discovery.” Recent Advances in Intrusion Detection. Springer
work Computing and Applications (NCA), 2014 IEEE 13th International Berlin Heidelberg, 2008.
Symposium on. IEEE, 2014. [37] Wang, Zhi, et al. “Countering kernel rootkits with lightweight hook
[10] Falliere, Nicolas. “Stuxnet introduces the first known rootkit for protection.” Proceedings of the 16th ACM conference on Computer and
industrial control systems.” Published online at http://www. symantec. communications security. ACM, 2009.
com/connect/blogs/stuxnet-introduces-first-known-rootkit-scada-devices. [38] Xie, Xiongwei, and Weichao Wang. “Rootkit detection on virtual
Last accessed on February 10 (2011). machines through deep information extraction at hypervisor-level.” Com-
[11] Felt, Adrienne Porter, et al. “A survey of mobile malware in the wild.” munications and Network Security (CNS), 2013 IEEE Conference on.
Proceedings of the 1st ACM workshop on Security and privacy in IEEE, 2013.
smartphones and mobile devices. ACM, 2011. [39] Yu, Wen. “Nonlinear system identification using discrete-time recurrent
[12] Fritsch, Hagen. “Analysis and detection of virtualization-based rootkits.” neural networks with stable learning algorithms.” Information sciences
Munchen: Technische Universitat (2008). 158 (2004): 131-147.
[13] Hoglund, Greg, and James Butler. Rootkits: subverting the Windows
kernel. Addison-Wesley Professional, 2006.
[14] http://core.ipsecs.com/rootkit/kernel-rootkit/kbeast-v1/
[15] Hubbard, Charles. Data Collection for Cyber Anomaly Event Detection.
University of South Alabama, 2015.
[16] Hutchins, Eric M., Michael J. Cloppert, and Rohan M. Amin.
“Intelligence-driven computer network defense informed by analysis
of adversary campaigns and intrusion kill chains.” Leading Issues in
Information Warfare and Security Research 1 (2011): 80.
[17] Kechriotis, George, Evangelos Zervas, and Elias S. Manolakos. “Using
recurrent neural networks for adaptive communication channel equaliza-
tion.” Neural Networks, IEEE Transactions on 5.2 (1994): 267-278.
[18] Kruegel, Christopher, William Robertson, and Giovanni Vigna. “De-
tecting kernel-level rootkits through binary analysis.” Computer Security
Applications Conference, 2004. 20th Annual. IEEE, 2004.
[19] Kunk, Adam, Pete Bohman, and Erik Shaw. “VMM based rootkit de-
tection on Android.” 2011-05-10]. http://cs523 sp2011 bjks. googlecode.
com/files/cs5 23 final report. pdf (2011).
[20] La Polla, Mariantonietta, Fabio Martinelli, and Daniele Sgandurra. “A
survey on security for mobile devices.” Communications surveys and
tutorials, IEEE 15.1 (2013): 446-471.