Early Detection of Remote Access Trojan

by Software Network Behavior

Masatsugu Oya1 and Kazumasa Omote2(B)

1
JAIST, Nomi, Ishikawa 923-1292, Japan
[email protected]
2
University of Tsukuba, Tsukuba 305-8573, Japan
[email protected]

Abstract. APT (Advanced Persistent Threat) attack is increasing in

recent years. APT attackers usually utilize malware called RAT (Remote
Access Trojan) to access and control computers by stealth. The invasion
method of RAT has been refined and it is extremely difficult to prevent
its infection beforehand. Hence, an approach to detect RAT infection
at the early stage after infection is important. However, there are two
drawbacks in the existing early detection methods of RAT; (1) they
do not become early detection in some circumstances; (2) they do not
consider the RAT-like healthy software (e.g., system related software
and antivirus software) for evaluation experiments. In this paper, we
propose a detection method of RAT based on the new mechanism of early
detection. Our evaluation experiments show that the proposed method
can distinguish between RAT and the RAT-like healthy software with
great accuracy.

Keywords: Advanced Persistent Threat (APT) attack ·

Remote Access Trojan (RAT) · Machine learning ·
Host-based detection

1 Introduction

Along with the rapid development of networking technologies and information

systems, most organizations store information in their computers as digitalized
data. At the same time, APT (Advanced Persistent Threat) attack, which is a
type of cyber-crime targeting high-value information assets owned by a specific
organization, is increasing [11]. Today, APT attack is one of the biggest threat
to companies and government agencies. The ultimate objective of APT attacks
is to steal high-value information assets such as technical property, financial
information, and personal information of customers. Once confidential informa-
tion is leaked, organization must incur a great loss, therefore, strong protection
measures against APT attacks are extremely important.
APT attackers usually use malware called RAT (Remote Access Trojan) to
achieve their ultimate goal. RAT is a type of malware that enables the attacker
to access and control remote computers by stealth. After a computer in the
c Springer Nature Switzerland AG 2019
F. Guo et al. (Eds.): Inscrypt 2018, LNCS 11449, pp. 658–671, 2019.
https://doi.org/10.1007/978-3-030-14234-6_37
 Early Detection of Remote Access Trojan by Software Network Behavior 659

target organization is infected with RAT, it establishes a connection with C&C

(Command and Control) server prepared by the attacker. The attacker sends
commands through C&C server to control infected computers and RAT conducts
intelligent activities such as downloading tools, exploring the network, searching
and gathering files in the computers, and sending them to the attacker.
RAT infects computers by targeted emails in most cases [12]. The attacker
guides the targets to open the attachments or URLs by utilizing any measures
including social engineering. The methods of targeted email has become more
and more sophisticated, so that it is extremely difficult to completely block the
intrusion of RAT at the entrance. Therefore, an approach to detect infection of
RAT at the early stage of post-infection activity is important.
One of the traditional detection methods of malware is signature based detec-
tion. In signature based detection method, malicious network communications
are judged based on predefined signatures. However, with this method, it is
not possible to detect unknown malware or variants of known malware, so that
detection can be easily avoided [7]. On the other hand, in behavior based mal-
ware detection method, it is possible to detect unknown malware and variants of
known malware by modeling behaviors peculiar to malware and different from
normal state. Also, malware detection methods are classified into two types
depending on the place where the system is introduced: network based detection
method and host based detection method. In a host based detection method, it
is necessary to consider the influence on the host computer and the operation
management of the system since the system is installed in each computer, so
that the operation is difficult as compared with network base detection method.
However, since the host based detection method is much richer in the amount
of information that can be used for malware detection, it is possible to detect
malware more accurately. Moreover, the host based detection method has the
advantage that it is relatively easy to identify the illegal software on the host
when detecting malware.
As a related research on our approach, Adachi et al. [1] proposed a host based
early detection method of RAT based on network behavior on a computer. How-
ever, this method has a drawback that much time may be required for detection
according to behavior of RAT or benign communication. As a result, it does
not become early detection in some circumstances. Furthermore, only several
specific benign software selected by the authors were used for the evaluation
experiments. As a result, it incorrectly detects the communication of system
related software and antivirus software as RAT. Therefore, the practicality of
this method on a real environment is unclear.
Our contributions are as follows.
– We propose a detection method of RAT based on the new mechanism of
early detection. The mechanism is different from the existing methods [1,3].
While the early stage may get longer in the existing methods, our early stage
finishes after a fixed period of time. Thanks to this improvement, it achieves
true early detection.
660 M. Oya and K. Omote

– Our method is greatly different from existing researches in that features

extracted from multiple connections in the initial stage communication are
combined to generate features for one process. As a result, diversity of various
communication by the same process can be expressed as a feature, and also,
it is possible to learn a more accurate detection model of RAT.
– We perform evaluation experiments considering the RAT-like healthy software
such as system related software and antivirus software. The network behavior
of such software resembles RAT. As for benign features, we capture the data
of communication and process from seven computers in the campus network
of JAIST. Nevertheless, our method distinguishes between such software and
RAT with great accuracy. Our method is superior to the previous method [1].
More precisely, our method can detect RAT with F-value of 91.5% while the
previous method can detect it with F-value of 69.6% in our experiment.

2 Related Work

As related researches on behavior based malware detection method, methods

of learning a model classifying malignant behavior and benign behavior using a
machine learning algorithm have been proposed. Tang et al. [10] and Ozsoy et al.
[4] proposed systems that detect malware by using machine learning techniques
for the behavior of hardware such as CPU. In the research by Sangkatsanee
et al. [9] and the research by Bekerman et al. [2], techniques for detecting attacks
by extracting network features such as the number of packets, the number of
destination ports, the number of TCP packets with an ACK flag was proposed.
However, it is not clear whether these methods are effective for RAT.
As a research on RAT detection, Li et al. [5] proposed a method of calculat-
ing feature values from sessions and learning detection models using clustering
method. Liang et al. [6] showed that a highly accurate detection model can be
realized by using features, such as the number of destination IP addresses and
the number of connections for each application, that can be acquired on each
computer. However, these methods use all network packets from the beginning
to the end of a session or software to judge whether the communication is gen-
erated by RAT or not. It means that confidential information can has been
already stolen at the point when the system detects the communication of RAT.
Yamada et al. [14] proposed the method to detect reconnaissance activities of
RAT. However, it is also unclear whether the method contribute to the preven-
tion of information leakage. Wu et al. [13] proposed the network-based detection
framework of human controlled RAT session.
Jiang et al. [3] proposed a method to detect RAT before the information
leakage occurs by using the communication data extracted from the early stage
of the session. Adachi et al. [1] improved the detection rate of RAT by adding
features that can be acquired on the computer to the approach of using the data
from the early stage of the session. However, in common with these two studies,
they have a drawback that much time may be required for detection according
to behavior of RAT or benign communication. As a result, it does not become
 Early Detection of Remote Access Trojan by Software Network Behavior 661

early detection for many situations. Furthermore, only several specific benign
software selected by the authors were used for the evaluation experiments. As a
result, it incorrectly detects the communication of system related software and
antivirus software as RAT. Thus, these methods [1,3] are not practical.
The existing methods [1,3] of RAT do not become early detection in some
circumstances, since the early stage does not finish as long as the packets are
transmitted in the existing methods. In other words, the early stage finishes
when the observed packets are not transmitted for a fixed period of time. Thus,
much time may be required for RAT detection. On the other hand, in our new
mechanism of early detection, the early stage inevitably finishes after a fixed
period of time.

3 Proposed Method
From the viewpoint of countermeasures against information leakage, detection
of RAT should be realized as early as possible after infection. We propose a
detection method of RAT based on the new mechanism of early detection. The
mechanism is different from the existing methods [1,3]. While the early stage
may get longer in the existing methods, our early stage finishes after a fixed
period of time. Thanks to this improvement, it achieves true early detection.
Our proposed method consists of two stages: learning phase and detection
phase. In both stages, feature extraction is a main of processing. In this section,
we first describe the details of the feature extraction processing and then explain
the each stage. For simplicity, the detection engine running on the host is
assumed to be trustworthy.

3.1 Our Approach

We observed benign and RAT communications, and then discovered the following
important innate characteristics of RAT:
– RAT tends to behave secretly, and then the communication data are low.
– Benign communication including the RAT-like healthy software has greater
traffic from the beginning because of non-stealthiness. Note that it does not
always has greater traffic.
Our results confirm that the above innate feature of RAT practically exists and
that the difference of network behavior between RAT and the RAT-like healthy
software is obvious during the early stage. If the communication data of RAT are
large, much witness are captured by network devices such as IDS. Our approach
is based on the above characteristics.

3.2 Feature Extraction

In feature extraction processing, features are calculated from monitored com-
munication for each process and a feature vector of the process is generated.
The target network communication is TCP connections that is started from the
662 M. Oya and K. Omote

Fig. 1. Monitoring of target communications

target computer to the external network. Therefore, UDP connections, com-

munication from the target computer to a computer in the same network, and
connection initialized from the outside are not monitored. This is because most
RATs communicate by TCP packets and a connection is established from the
infected computer to the C&C server when RAT infects the computer. Also,
considering the privacy of users and the scalability of the system, TCP payloads
are discarded and only the header parts are acquired.
Since the purpose of this research is to detect RAT at the early stage, features
are calculated using only a part of the first few connections for each process. The
new definition of the early stage of communication in this research is as follows.

Definition 1 (Early Stage). The early stage of communication is t seconds

from the first packet for same 5-tuple (Source IP address, Destination IP address,
Source port number, Destination Port number, and Protocol) network packets.

For feature extraction, the first few early stage of connections in each process
are used. Specifically, for each process, the early stage of connections initiated
during the early stage of the first connection of the process are monitored. Thus,
the monitoring time in each process is 2t seconds at maximum. In the case of the
example of Fig. 1, the system monitors the early stage of Connection 1, which is
the first connection of the process, and Connection 2 and Connection 3, which
are the connections started during the early stage of Connection 1.
From the early stage of monitoring connections, eleven types of features
shown in Table 1 are extracted. Then, after obtaining features from all mon-
itoring early stage connections of the process, “Mean”, “Standard Deviation
(SD)”, “Maximum value (Man)”, “Minimum value (Min)”, and “Range” of each
connection features, and also, “Sum” of “PacNum”, “OutPac”, “InPac”, “Out-
Byte” and “InByte” are calculated. Further,“Number of Connections (Conn)”
and “Number of Destination IP addresses (DstIP)” are counted. Finally, a 62
dimensional vector is outputted as a feature vector of the process. A specific
calculation procedure of feature vectors is as follows.
 Early Detection of Remote Access Trojan by Software Network Behavior 663

Table 1. Connection features to extract

Feature Explanation
PacNum Total number of packets
OutPac Number of outbound packets
InPac Number of inbound packets
OutByte Data size of outbound packets
InByte Data size of inbound packets
OutByteSD Standard deviation of data size of outbound packets
InByteSD Standard deviation of data size of outbound packets
O/Ipac Rate of OutPac to InPac
O/Ibyte Rate of OutByte to InByte
OB/OP Average data size per one outbound packet
IB/IP Average data size per one inbound packet

1. Read a packet.
2. Judge whether the packet is TCP and the destination is toward the outside
of the network.
3. Identify the connection from the set of source IP address, destination IP
address, source port number, and destination port number.
4. If it is a new connection, identify the process ID, the process name, and the
path of the executable file from the source port number, and store them in a
correspondence table, otherwise, refer the correspondence table and identify
the process.
5. Judge whether the packet is monitoring target.
6. If it is a monitoring target, update connection features (PacNum, OutPac,
InPac, OutByte, InByte).
7. When all monitoring connections of a process are captured, calculate remain-
ing connection features (OutByteSD, InByteSD, O/Ipac, O/Ibyte, OB/OP,
IB/IP).
8. Calculate Mean, SD, Max, Min, Range of each connection feature and Sum
of PacNum, OutPac, InPac, OutByte, InByte.
9. Calculate “Conn” and “DstIP”.
10. Output a feature vector of the process.

Our proposed method differs greatly from existing researches in the point
that the feature vector of a process is generated based on the multiple early
stage connections of the process. Therefore, the diversity of connection, such
as differences and variations in communication sizes for each connection, by
the same process can be captured as features. RAT communication is often
single session and single connection, and due to its characteristics, there is a
time lag between the establishment of the connection to the C&C server and
the attacker who confirms the connection starts attacks. Thus, we can guess
that RAT communication in the early stage is monotonous. On the other hand,
664 M. Oya and K. Omote

benign software will generate a wide variety of connections even in the early
stage. Therefore, we expect that the difference between RAT and benign software
become clear.

3.3 Learning Phase

In this phase, at first, feature vectors of processes of benign software and RATs
are collected, and then the detection model of RAT is learned using those data.
Feature vectors of benign software are gathered by performing the feature extrac-
tion processing described in the previous section on computers which can be
guaranteed that they are not infected with malware. Also, feature vectors of
RAT are collected by running prepared RAT samples. After that, for each col-
lected feature vector, 0 is assigned if the feature vector is for benign software,
and 1 if it is RAT as a label. Finally, the detection model of RAT is generated
by applying supervised learning algorithm of machine learning for this dataset.

3.4 Detection Phase

At the detection phase, the feature extraction processing is executed at the

monitor target computer, and the calculated feature vector is given as an input
to the detection model generated at the learning phase. When the output is 0,
it is assumed that the process is generated by benign software, and when the
output is 1, the process is judged to be RAT.
When a process of RAT is detected, by referring to the location data of
the execution file saved during the feature extraction processing, it is possible
to check and analyze the actual substance. Therefore, it becomes possible to
quickly respond to attacks in the first place, and it makes a time for implementing
countermeasures before information leakage due to APT attack occurs.

4 Evaluation Experiments

4.1 Overview

In this section, we perform two types of evaluation experiments to confirm the

effectiveness of the RAT detection method proposed in the previous section. In
the first experiment, we mainly evaluate the detection performance of RAT by
K-fold cross-validation for feature vectors of RAT and benign software. K-fold
cross-validation is a method of dividing data into k groups and verifying each
group using a model learned with all data of other groups. K-fold cross-validation
enables to evaluate the detection performance against unknown RAT with lim-
ited data since RAT used for learning is not used at the time of evaluation. In
this experiment, we set k = 5.
In the second experiment, the detection model of RAT was learned using all
of the feature vectors of RAT and a part of the feature vector of benign software,
and we evaluate the false detection of the proposed system by using the remaining
 Early Detection of Remote Access Trojan by Software Network Behavior 665

Table 2. List of RAT samples

BX Bandook Bozok Cerberus CyberGate

DarkComet DarkNET Ghost LeGeNd Mega
Netbus NovaLite Nuclear OptixPro Orion
Pandora PoisonIvy ProRAT Turkojan WiRAT
dalethRAT deamondRAT jSpy njRAT ucuL

feature vectors of benign software. In the case that many false detections are
created by the system, the cost for examining alerts increases during operation,
which increases the burden on the system administrators, so that the system
is not practical. Therefore, in the second experiment, the practicality of the
proposed system is evaluated from the viewpoint of the amount of false detection.
Before these experiments, we perform a preliminary experiment to deter-
mine the parameter t of the early stage of communication and the combination
of the features to use in the experiments. In all experiments, we use seven type
of machine learning classification algorithms: Decision Tree (DT), Random For-
est (RF), Support Vector Machine (SVM), Naive Bayes (NB), Gradient Tree
Boosting (GTB), AdaBoost (AB), and Multi-layer Perceptron (MLP).

4.2 Evaluation Indices

We use Accuracy (ACC), FPR, FNR, Precision (PRC), Recall (RCL), and
F-measure (F1) as evaluation indices of the experiments. ACC is the accuracy
of the overall identification, FPR is the false detection rate of the benign pro-
cess, and FNR is the index showing the overlook rate of RAT. Also, PRC is the
reliability of malignancy judgment, RCL means the detection rate of RAT, and
F1 is the harmonic mean of PRC and RCL, which is a comprehensive measure
of accuracy and completeness.

4.3 Dataset
As a benign dataset, we use daily communications including system related soft-
ware and anti-virus software from active PCs to generate new dataset. The fea-
ture vectors of benign software used in the experiments are generated by collect-
ing logs of communication and processes from seven computers in the campus
network of JAIST and then by executing the feature extraction program. We
acquired about 24 h (weekday) of communication logs from each computer. The
usage purpose of each computer is creation of documents, browsing of emails
and Web sites, and access to file servers, so it has no big difference from the
usage of most organizations.
Feature vectors of RAT are collected by running RAT samples in a isolated
network environment, and executing the feature extraction program. We use 25
kinds of RATs for evaluation. Table 2 shows the list of RAT samples used in
experiments.
666 M. Oya and K. Omote

4.4 Preliminary Experiment

For evaluation, it is necessary to derive the optimum value of t, the parameter of
the early stage communication, and the combination of features. However, it is
impossible to test all possible combinations from the viewpoint of computational
cost. Therefore, we determine t and the combination of features by the following
procedure.

Table 3. Evaluation results of Experiment 1

DATA AO’16[1] Proposed

ACC FPR FNR PRC RCL F1 ACC FPR FNR PRC RCL F1
PC1+RAT 0.883 0.100 0.200 0.625 0.800 0.702 0.983 0.011 0.040 0.960 0.960 0.960
PC2+RAT 0.733 0.289 0.200 0.476 0.800 0.597 0.970 0.027 0.040 0.923 0.960 0.941
PC3+RAT 0.822 0.185 0.160 0.636 0.840 0.724 0.938 0.054 0.080 0.885 0.920 0.902
PC4+RAT 0.933 0.055 0.160 0.677 0.840 0.750 0.972 0.019 0.080 0.885 0.920 0.902
PC5+RAT 0.894 0.099 0.160 0.538 0.840 0.656 0.984 0.006 0.080 0.958 0.920 0.939
PC6+RAT 0.868 0.126 0.160 0.600 0.840 0.700 0.960 0.020 0.120 0.917 0.880 0.898
PC7+RAT 0.893 0.080 0.200 0.741 0.800 0.769 0.935 0.048 0.120 0.846 0.880 0.863
Total 0.874 0.115 0.177 0.603 0.823 0.696 0.967 0.022 0.080 0.910 0.920 0.915

When t becomes smaller, the earlier detection is possible, but the amount of
features is smaller. So we first set the values from 1 to 10 as candidate values of t.
For each candidate value, we evaluate the detection performance using 15 useful
features determined by F value in analysis of variance. At this time, we consider t
at which the evaluation result is the best as the optimum value. Next, we fix t is to
the optimum value and find the best combination of features by brute force.
As a result of the above processing, the optimum value of t became 4, and
the optimum combination of features became “InByteSDMin + IB/IPMean +
IB/IPMax + IB/IPMin + OutByteSDMax + OutByteSDMean + OB/OPMin
+ IB/IPRange”. In the subsequent experiments, we use the value of t and the
combination of features obtained.

4.5 Experiment 1
In this experiment, we evaluate the detection performance of the proposed
method using collected benign communication and RAT communication. We
use feature vectors extracted from each computer and feature vectors of RAT as
one dataset, and calculate performance indices by 5-fold cross validation. Table 3
shows the results of this experiment. In evaluation experiments using the method
of Adachi et al. [1], the performance was the best on average when SVM was
used as a learning algorithm1 In the proposed method of this research, the best
performance was obtained when GTB was used. Note that we implemented the
method of Adachi et al. [1] by ourselves for comparison.
1
Naive Bayes was the best algorithm in [1]. But, as a result of our code and feature
refining, the result by SVM was the best in our experiment.
 Early Detection of Remote Access Trojan by Software Network Behavior 667

Table 4. Evaluation results of Experiment 2

TrainData TestData AO’16 [1] Proposed

FP FPR FP FPR
PC1+RAT PC2+PC3+PC4+PC5+PC6+PC7 77 0.109 49 0.078
PC2+RAT PC1+PC3+PC4+PC5+PC6+PC7 84 0.112 44 0.068
PC3+RAT PC1+PC2+PC4+PC5+PC6+PC7 78 0.103 29 0.044
PC4+RAT PC1+PC2+PC3+PC5+PC6+PC7 108 0.168 26 0.046
PC5+RAT PC1+PC2+PC3+PC4+PC6+PC7 81 0.126 42 0.075
PC6+RAT PC1+PC2+PC3+PC4+PC5+PC7 64 0.090 15 0.024
PC7+RAT PC1+PC2+PC3+PC4+PC5+PC6 99 0.134 14 0.022
Total 591 0.120 219 0.051

Comparing the results, our method is better than the method by Adachi
et al. for all evaluation indices. The detection rate (RCL) of RAT in the proposed
method is 0.920 as a whole, and it becomes clear that most RAT can be detected.
On the other hand, the detection rate by Adachi et al. method is 0.823, indicating
that the detection performance of our method is high by about 0.1. As for FPR,
the proposed method was 0.022 as a whole, while that of Adachi et al. was 0.115.
In PRC which is an index of reliability of RAT detection, the proposed method
is about 0.3 better than the method by Adachi et al.

4.6 Experiment 2

In this experiment, the detection model learned using communication of the

benign process of one computer and all communication of RAT is tested by
the data collected from the remaining computers. At that time, we derive the
false detection rate and evaluate whether it can be suppressed to a level that can
withstand practical use. Also, the collected environment of data used for learning
and data used for testing is different in this experiment, so the versatility of the
proposed method can also be evaluated.
Table 4 shows the evaluation result of number of false positives and the FPR.
In the method of Adachi et al., the FPR is as high as 0.120 as a whole, and many
benign processes are falsely detected as RAT. Note that we implemented the
method of Adachi et al. [1] by ourselves for comparison. On the other hand, in
the proposed method of this study, the number of processes misjudged as RAT
is relatively small, and the FPR is 0.051 as a whole. Therefore, it can be said
that the proposed method is more practical in terms of false detection rate. Also,
this experimental result shows that the proposed method is effective even when
the collection environment of learning data and evaluation data is different.
668 M. Oya and K. Omote

5 Discussion
5.1 False Detection

In the previous section, we show that the proposed method can detect infec-
tion of RAT with high accuracy, and also the possibility of false detection of
benign software is greatly improved compared to existing research. However, in
a large organization that owns a large number of computers, even if there are few
false detection alerts on one computer, the burden on the system administrators
becomes large since the number of alerts as a whole increases.

Table 5. Processes frequently judged as RAT (RAT-like healthy software)

Process Count Type Distributor

backgroundTaskHost.exe 40 System Microsoft
ActionUriServer.exe 30 System Microsoft
svchost.exe 14 System Microsoft
McPltCmd.exe 10 Antivirus McAfee
OneDrive.exe 9 Cloud Microsoft
explorer.exe 8 System Microsoft
System 7 System Microsoft
avgnt.exe 6 Antivirus Avira
Avira.ServiceHost.exe 6 Antivirus Avira
Avira.Systray.exe 6 Antivirus Avira
Dropbox.exe 6 Cloud Dropbox
OSE.EXE 6 Update Microsoft
Shogidokoro.exe 6 Other Individual
TeamViewer Service.exe 6 Development TeamViewer
wsqmcons.exe 6 System Microsoft

Table 5 shows processes that are erroneously detected as RAT many times by
our method. From the results, many erroneous detections occurred for Windows
system related processes, antivirus software related processes, software update
related processes, and cloud storage services. We consider that the characteris-
tics of these processes are similar to RAT since they do not perform vigorous
communication at the early stage compared with other benign processes.
We assume that it is possible to avoid false detection by pre-registering pro-
cesses often judged as RAT in the whitelist. In the proposed method, when
an alert occurs, the location of the executable file of the process can be easily
specified, so the whitelist can be realized by confirming the digital signature
of executable files. Table 6 shows the FPR and the FP, which are calculated
based on the experiment in the previous section, depending on the presence
 Early Detection of Remote Access Trojan by Software Network Behavior 669

Table 6. False detection without whitelist and with whitelist

Without WL With WL: 4 Signaturesa With WL: 8 Signaturesb

FPR 0.051 0.007 0.003
Average FP on 36.500 5.167 2.167
1-PC for 7 days
a
Microsoft, McAfee, Avira, Dropbox
b
Microsoft, McAfee, Avira, Dropbox, TeamViewer, Opera, Oracle, ASUS

of the whitelist. Note that the FP is the estimated average value on 1-PC for
7 days. The results indicates that it is possible to reduce the number of false
detections by about 90% by registering digital signatures of Microsoft, McAfee,
Avira, Dropbox in the whitelist. Also, the FPR decreases to 0.003 when eight
publishers, which are Microsoft, McAfee, Avira, Dropbox, TeamViewer, Opera,
Oracle, ASUS, are registered in the whitelist.

5.2 Evasion
We discuss whether our method can be avoided by attackers. As a detection
avoidance, it is conceivable that attackers make RAT to generate communication
similar to the features of a benign process. In this case, RAT needs to establish
some connections with different traffic amounts and make each packet size widely
varied. However, artificially generating unnecessary communication leads to an
increase in the possibility that RAT communication will be detected by another
attack detection system. In particular, mechanically generated communication
is expected to be regarded as communication of Bots. Also, unnecessary commu-
nication increases the amount of traces of the attack. In APT attacks, attackers
need to complete tasks without noticing the targets, therefore, we consider that
attackers are not willing to attempt such means. Furthermore, since this method
learns the detection model by machine learning, the detection criterion is not
clear compared with signature based detection methods. Thus, it is extremely
difficult to analyze and generate communication that the detection model misses.
From the above viewpoint, we conclude that the risk that attackers intentionally
avoid the detection model by such a method is low.
As another detection avoidance method, injecting RAT into a running benign
process (e.g., cross-process injection) is conceivable. If the RAT program is
injected into a process under execution, the RAT apparently functions as the
benign process, so it may be excluded from detection by the whitelist. Also, if the
injection process already communicates and the monitoring time of the proposed
method has passed, the proposed method cannot detect RAT infection. There-
fore, it is necessary to detect RAT infected by injection by another method. In
this case, since the occurrence of code injection itself can be regarded as suspi-
cious activity, a system that can detect injection can be effective. As measures
against injection, there is a method that previously records the address where
Windows API call instructions on the software executable file are described, and
detects the injection by checking the record when the API is actually called [8].
670 M. Oya and K. Omote

6 Conclusion
We proposed a detection method of RAT based on the new mechanism of early
detection. The mechanism is different from the existing methods [1,3]. While
the early stage may get longer in the existing methods, our early stage finishes
after a fixed period of time. Thanks to this improvement, it achieves true early
detection. This makes it possible for the proposed method to more clearly distin-
guish between RAT and the RAT-like healthy software communication including
system related one. Evaluation experiments show that the proposed method can
detect RAT in early stage of post-infection activity with a detection rate of
92%, FPR of 2.2% and FNR of 8.0%. Therefore, we conclude that the proposed
method is sufficiently practical as an early detection system of RAT. As a future
work, we will conduct the experimental evaluation using new RATs in various
networks.

Acknowledgements. This work was partly supported by Grant-in-Aid for Scientific

Research (C) (16K00183).

References
1. Adachi, D., Omote, K.: A host-based detection method of remote access trojan in
the early stage. In: Bao, F., Chen, L., Deng, R.H., Wang, G. (eds.) ISPEC 2016.
LNCS, vol. 10060, pp. 110–121. Springer, Cham (2016). https://doi.org/10.1007/
978-3-319-49151-6 8
2. Bekerman, D., Shapira, B., Rokach, L., Bar, A.: Unknown malware detection using
network traffic classification. In: CNS 2015, pp. 134–142. IEEE (2015)
3. Jiang, D., Omote, K.: A RAT detection method based on network behavior of the
communication’s early stage. IEICE Trans. Fundam. E99.A(1), 145–153 (2016)
4. Khasawneh, K.N., Ozsoy, M., Donovick, C., Abu-Ghazaleh, N., Ponomarev, D.:
Ensemble learning for low-level hardware-supported malware detection. In: Bos,
H., Monrose, F., Blanc, G. (eds.) RAID 2015. LNCS, vol. 9404, pp. 3–25. Springer,
Cham (2015). https://doi.org/10.1007/978-3-319-26362-5 1
5. Li, S., Yun, X., Zhang, Y., Xiao, J., Wang, Y.: A general framework of trojan
communication detection based on network traces. In: NAS 2012, pp. 49–58. IEEE
(2012)
6. Liang, Y., Peng, G., Zhang, H., Wang, Y.: An unknown trojan detection method
based on software network behavior. Wuhan Univ. J. Nat. Sci, 18(5), 369–376
(2013)
7. Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection.
In: ACSAC 2007, pp. 421–430. IEEE (2007)
8. Rabek, J.C., Khazan, R.I., Lewandowski, S.M., Cunningham, R.K.: Detection of
injected, dynamically generated, and obfuscated malicious code. In: ACM work-
shop on Rapid Malcode, pp. 76–82. ACM (2003)
9. Sangkatsanee, P., Wattanapongsakrn, N., Charnsripinyo, C.: Practical real-time
intrusion detection using machine learning approaches. Comput. Commun. 34(18),
2227–2235 (2011)
 Early Detection of Remote Access Trojan by Software Network Behavior 671

10. Tang, A., Sethumadhavan, S., Stolfo, S.J.: Unsupervised anomaly-based malware
detection using hardware features. In: Stavrou, A., Bos, H., Portokalidis, G. (eds.)
RAID 2014. LNCS, vol. 8688, pp. 109–129. Springer, Cham (2014). https://doi.
org/10.1007/978-3-319-11379-1 6
11. Tankard, C.: Advanced persistent threats and how to monitor and deter them.
Netw. Secur. 2011(8), 16–19 (2011)
12. Check Point Software Technologies Ltd., Grobal Cyber Attack Treands Report
(2017)
13. Wu, S., Liu, S., Lin, W., Zhao, X., Chen, S.: Detecting remote access trojans
through external control at area network borders. In: ANCS 2017, pp. 131–141.
ACM/IEEE (2017)
14. Yamada, M., Morinaga, M., Unno, Y., Torii, S., Takenaka, M.: RAT-based mali-
cious activities detection on enterprise internal networks. In: ICITST 2015, pp.
321–325. IEEE (2015)