Implications of Rootkit to User Privacy

Term Paper

Submitted to Dr. Mario Garcia

The Department of Computing Sciences

Texas A&M University-Corpus Christi

Corpus Christi, Texas

By

Srikanth Padakanti

Fall 2010

Course Instructor
Dr. Mario Garcia

______________________________
ABSTRACT:

Methods and techniques currently exist to detect whether a rootkit has exploited a
system or not. But, these methods and techniques gives us result that, whether the system
has been exploited by rootkit or not. This paper deals with detection of the rootkits using
the behavioral analysis. Rootkit is a type of malware (software) which is operated in a
stealth mode. My research is divided into two stages where in first stage; i studied a wide
variety of rootkits and their characteristics and enter those characteristics into database
with unique ids and names. In second stage, i used some methodology and proposed a
tool which is used to detect the rootkit present on the host system, is an existing rootkit or
new rootkit.

2
 TABLE OF CONTENTS

Abstract….........................................................................................................................2

Table of Contents..............................................................................................................3

List of Figures…...............................................................................................................4

1. Introduction…..............................................................................................................5

1.1. Attacking process of rootkits …...........................................................................6

1.2. Motivation ………………………………………………………………………7

2. Proposed Research ………...........................................................................................7

3. Methodology ………....................................................................................................7

3.1 Stage 1 …………………………………………………………………………..8

3.2 Stage 2 …...……………………………………………………………………...9

4. Conclusion and Future Work ……...............................................................................13

5. References ……………………………………………………………………………13

3
 LIST OF FIGURES

1. Rootkits Characteristic Table ………………………………...................................... 9

2. Showing the common features of two rootkits f1 and f2 ……..…........................... 10

3. Explains how to calculate f2’ ……….........................................................................11

4. Shows f2 is an extension of existing rootkit ………………………………………..12

5. Shows f2 is subset of existing rootkit ………………………………………………12

4
1. Introduction:
Rootkits are software which is used to hide its presence and activity on the host
system. Rootkits are the procedures that have attracted attention recently. There is a
huge increase in the count and complexity of rootkits in recent years. Rootkits are sets of
tools used by an intruder to maintain root level access for a system in a covert way [1].
To know whether an attacker has installed any rootkit and up to what extent the system
is compromised, the administrators need to trust the host based techniques. Before
rootkits, system tools could be trusted to provide an administrator with accurate
information. Modern cookies have developed some methods to conceal their activities.
The rootkits are considered as serious threat to security of a networked system.
The major malware attacks focuses on the commercial markets and enterprises
with an intention of producing huge profits for the malware authors, but the visibility of
these types of threats is in gradually sinking because of the stealth techniques such as
rootkits are installed on the host system to hide the malicious activities. Detection and
classification of these rootkits, i believe, will help us to reduce the future attacks on
business, commercial markets and even we can detect them before it harms our system
completely [2].
Some rootkits attacks on windows operating system and some attack on
LINUX/UNIX operating systems. Installation of the rootkits is the first and major step of
an intruder after gaining access to the system, as it ensures that attack is not detected.
Rootkits also open a backdoor by which an attacker can eavesdrop on the activities of
the system. Once the rootkit gets installed then the attacker can capture any kind of
information from the system like credit card numbers, bank account details, passwords
etc.
Rootkits are considered as Trojans introduced into our machines. Generally
rootkits are of two types, user mode rootkits and kernel mode rootkits. User level
rootkits are also known as application level rootkits because they usually replace the
system services with the modified versions which hide the intruder’s files, network
connections and processes. But, it is very simple to detect these rootkits because these
rootkits are not capable to modify the operating system kernel. Kernel level rootkits are

5
considered as dangerous and most powerful rootkits. These rootkits uses a custom kernel
module which modifies the system call of a kernel to hide the processes and files.
Besides this, it also provides the back door for intruder to visit the root level
permissions. The custom kernel module mainly uses hooking techniques such as inline
hooking, Process Control Block (PCB) hooking etc. In this paper we mainly deal with
inline hooking. It overwrites the first five bytes of an application function with first byte
as JUMP instruction (E9) and remaining four bytes with a 32-bit address of some
malicious code [3].

1.1 Attacking process of rootkits:

Researchers analyzed the attacking method of the kernel level rootkits, and
divided the course of attack into four steps:
1. Collect the information of the target: Before attacking, attackers will scan and
observe the target system completely and gains the knowledge about the flaws and
the versions of the system.
2. Get Root level access: In this stage, intruder gets only temporary access to the
target system. By knowing the flaws of the target system in step 1, intruder gets
the root level access to the target.
3. Installation of rootkits: After gaining the root level access, the intruders will
install rootkits into the target systems which hide its processing and activities.
4. Control: Once the intruder completes the above three steps completely then, the
last stage is just controlling the target system and compromise the target without
the detection of the owner [4].
Rootkits cannot propagate by themselves. It is software which typically consists
of three snippets of code called as dropper, loader and rootkit. Dropper is the code that
gets the rootkit installation started. It requires the human intervention like clicking
emails. After that it loads a loader program into the system and deletes itself. Loader
causes a buffer overflow which loads the rootkit into the memory. Lastly rootkit starts
working on the host computer and hides from administrator [5].

6
 1.2 Motivation:
Rootkits are very hard to detect and its presence in our machines is very
dangerous. For example, the information collected by intruder through rootkits is sold to
some companies for huge profits. We have some tools to say whether a rootkit exists are
not but it wouldn’t assure completely about the removal of the rootkit. So, detection and
removal of rootkits from the computers which are connected in the network decreases a
big task in security. The important reason to be considered is, rootkit is invisible when it
is executing. These are very small software but, causes very major threats to the
confidentiality of data. The best example is Sony BMG stealth Digital Right
Management (DRM) rootkit. This is software with Extended Copy Protection (XCP)
installed in the machines. This is done in attempt to stop the music copy right violations.
This rootkit creates vulnerabilities for other malware to exploit.
In this paper, i mainly concentrated on the rootkits that affects the operating
system processes and resources which mainly uses the inline function hooking. There
are valid reasons why it is important to do research in the area of rootkits:
 It is known from a survey that 85 percent of malicious software is being
developed with an intention to generate profits for malware developers.
 There is a huge increase in the number and complexity of rootkits in recent
years [6].

2. Proposed research:
By using behavioral detection the subset and extended version of the existing
rootkits are found. This is achieved by observing and analyzing the characteristics of the
rootkits. It is also extended to other types of rootkits.

3. Methodology:
The research work is divided into two stages where in first stage, reviewed
almost eleven famous rootkits which are freely available online. Among eleven, two are
adware, remaining are rootkits. Clearly, their characteristics and behavior is analyzed
and i maintained a table in the database with unique ids and names for this rootkits. This

7
is used in future for querying. In second stage, using set theory, i tried to found the
affected target system has a rootkit which is either new or exactly the same or extended
or subset of the existing rootkit.
3.1 Stage 1:
I observed some rootkits characteristics which are open source tools. Some of
them are:
I. Hacker Defender: It installs a backdoor technique and downloads a malware on
to the target system and the initialization part starts when the Hxdef100.ini starts
executed. Immediately after execution it deletes the kernel mode driver called
hxdef100rv.sys. It mainly sniffs the data to a highly trafficked path to get the
commands from the remote machine [7].

II. Shadow Walker: The name itself conveys its characteristic feature. It behaves
like shadow means almost invisible. It works by hooking the Virtual Memory
Manager (VMM) and editing the page-fault handler. Page-fault is considered as a
virtual memory interrupt. When the next instruction or item of data is not in
physical memory and must be swapped back in from the disk. If the required page
on disk cannot be found, then a page fault error occurs, which means that either the
operating system or an application has corrupted the virtual memory. If such an
error occurs, the user has to reload the application. The execution of the shadow
walker is invisible to signature-based scanners because it exists in the memory and
it never leaves any of its contents on the hard drive [8].

III. Futo: It uses a method to hide the process. The method is Direct Kernel Object
Manipulation (DKOM) of PspCidTable. It is a memory resident kernel data
structure which contains information of both active and inactive processes. There is
a tool Known as Rootkit Analysis Identification Elimination (RAIDE), which is
under beta testing [9].

IV. Apropos Rootkit: This is famous and dangerous rootkit. It installs a kernel
mode driver into the operating system and hooks the user mode and kernel mode

8
 data structures. Besides that, it installs a spyware which is started before the
operating system gets started. It doesn’t have a standard name for their files, it
keeps on renaming them which is difficult for an administrator to detect them and
uninstall them [10].

V. Pe386: The processes of pe386 run in background and are hidden. Apart from
that the files are located in the Alternate Data Stream (ADS) of system32 folder.
ADS are a special storage area where the infrequently accessed files are stored and
they are not normally viewable without using a special program [11].
In this way, i studied the characteristics of eleven rootkits and inserted a table
into the database with unique ids. The table name is RootChartable which is shown in
Table 1.
Table 1. Rootkits characteristic Table.

3.2 Stage 2:
In my research identification of rootkit is done by using set theory. So, before
entering into the proposed model we will see some basics of set theory. If we have two
sets say A and B, then
 It is said to be A U B, if we have both the features of A and B combinable.
 It is said to be A ∩ B, if we have the common features of A and B, but not
9
 everything. It is shown in Figure 2.
If {U} is the universal set, then A’ is computed by deleting A from the
universal set i.e., A’= {U}-{A}. We can explain the concepts using Venn diagrams.

Shaded part indicates (f1 ∩ f2)

Figure 1. Showing the common features of two rootkits f1 and f2.

Let us consider an example, Let fi be different functionalities of a rootkit where

{i=1, 2, 3, …, n} and we suppose that f1 is subset of f2 and f1 is not equal to f2.
Difference between f1 and f2 contains only those elements belong to f2 (but not in f1). It
is represented by Δ (f1, f2) = f2/f1. Now, if we consider f1 be the existing rootkit and f2
be the new one and we don’t know whether f2 is exactly same as f1 or subset or
extended or modified.
Here, use the characteristics from the database which are done in stage 1 to know
whether it is same as existing or not. Now observe the behavior of the new rootkit in the
target system and tabulate them in a separate database table. We should find whether the
new rootkit characteristics matches with other rootkit characteristics which are already
exist in table. The comparison is shown in Figure 3. If we remove the intersection part
from the f2 we will get the features that are only in f2 and name it as f2’. If it doesn’t
match any of the existing characteristics with the new rootkit characteristics. Then, we
should give a unique id to that rootkit and enter it into the RootChartable.
Now, calculate the common features for f2’ with all existing rootkits and name
those features as {F}.

10
 {F} = (f1 ∩ f2’) U (f2 ∩ f2’) U (f3 ∩ f2’) U………. U (fn ∩ f2’) ……. (Eqn 1)

f2’ = f2 – (f1 ∩ f2)

Figure 2. Explains how to calculate f2’

The results for the proposed tool depend on the {F} value. If we have NULL
value in {F} i.e., {F} = {Φ} and (fi ∩ f2) = {Φ} where i= 1, 2, 3, 4,….., n. Then that
rootkit is said to be new rootkit. Add that rootkit features to the existing table as (rn+1)
th rootkit. If Δ (f1, f2) = 0 then, the new rootkit f2 is exactly same as the existing rootkit
f1.

If we have NOT NULL value in {F} i.e., {F} ≠ {Φ} and (fi ∩ f2) ≠ {Φ} where
i= 1, 2, 3, 4,….., n. Then that rootkit is said to be a modified or extended or subset of the
existing rootkit. So, to decide whether it is modified or subset or extended we follow the
following rules.

 If Δ (f1, f2) = 0 then, the new rootkit f2 is exactly same as the new rootkit.
 If (f1∩ f2) = f1 and (f1U f2) = f2 then, the new rootkit is the extension of
the existing rootkit which is shown in Figure 4.
 If (f1∩ f2) = f2 and (f1U f2) = f1 then, the new rootkit is the subset of the
existing rootkit which is shown in Figure 5.

11
 Shaded part indicates (f1 ∩ f2)

Figure 3. Shows f2 is an extension of existing rootkit.

Shaded part indicates (f1 ∩ f2)

Figure 4. Shows f2 is a subset of existing rootkit.

The proposed tool only works for inline hooking because it just manipulates the
address in the instruction. But, other hooking techniques don’t affect the address part.
But, they hold the control of the kernel processes by hooking their kernel environment.

12
4. Conclusion and Future work:
The proposed tool detects the rootkit and tells us that it is a new rootkit or the
existing type. The proposed tool is successful to determine whether the new rootkit is
subset or extension of an existing rootkit theoretically. The future work for this research
work is to implement the rules and start designing the tool and apply the rules to check
the real world simulation. Another enhancement can be made to this tool, which can be
extended to detect the modified versions of the rootkits.

5. References:

[1] A. Emigh, “The Crimeware Landscape: Malware, Phishing, Identity Theft and
Beyond”, Journal of Digital Forensic Practice, 2006, pp. 245-260.
[2] ] D. Dettrich, "Root Kits" and hiding files directories/processes after a break-in.
[Online]. Available: http://staff.washington.edu/dittrich/misc/faqs/rootkits.faq, 2002.
[3] M. Russinovich, D. Solomon, Windows Internals, Microsoft Press, U.S, 2004.
[4] J. Hao, Y. Hao, Z. Ding, L. Tao, “A Methodology to Detect Kernel Level Rootkits
Based on Detecting Hidden Processes”, in proceedings with International Conference on
Apperceiving Computing and Intelligent Analysis, 2008, pp. 359-361.
[5] J. Grizzard, J. Levine, J. Owen, : Detecting and categorizing kernel-level rootkits to
aid future detection”, Security & Privacy, IEEE, 2006, pp. 24-32.

[6] J. Levine, J. Grizzard, P. Hutto, H Owen, "A Methodology to Characterize Kernel

Level Rootkits Exploits that Overwrite the System Call Table", Security & Privacy,
IEEE, 2004, pp 24-31.
[7] www.f-secure.com, http://www.f-secure.com/v-descs/hacdef.shtml, last visited
12/09/2010.
[8] www.securityfocus.com, http://www.securityfocus.com/print/infocus/1851, last
visited 12/09/2010.
[9] www.openrce.org, https://www.openrce.org/articles/full_view/19, last visited
12/09/2010.
[10]www.symantec.com, http://www.symantec.com/security_response/writeup.jsp?
docid=2005-102112-2934-99, last visited, 12/09/2010.

13
[11] www.f-secure.com, http://www.f-secure.com/v-descs/mailbot_az.shtml, last visited
12/09/2010.

14