776 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 19, NO.

2, MARCH/APRIL 2022

Open Source Intelligence for Malicious Behavior

Discovery and Interpretation
Yi-Ting Huang , Chi Yu Lin , Ying-Ren Guo , Kai-Chieh Lo, Yeali S. Sun, and Meng Chang Chen

Abstract—Cyber threats are one of the most pressing issues in the digital age. There has been a consensus on deploying a proactive
defense to effectively detect and respond to adversary threats. The key to success is understanding the characteristics of malware,
including their activities and manipulated resources on the target machines. The MITRE ATT&CK framework (ATT&CK), a popular
source of open source intelligence (OSINT), provides rich information and knowledge about adversary lifecycles and attack behaviors.
The main challenges of this study involve knowledge collection from ATT&CK, malicious behavior identification using deep learning,
and the identification of associated API calls. A MITRE ATT&CK based Malicious Behavior Analysis system (MAMBA) for Windows
malware is proposed, which incorporates ATT&CK knowledge and considers attentions on manipulated resources and malicious
activities in the neural network model. To synchronize ATT&CK updates in a timely manner, knowledge collection can be an automatic
and incremental process. Given these features, MAMBA achieves the best performance of malicious behavior discovery among all the
compared learning-based methods and rule-based approaches on all datasets; it also yields a highly interpretable mapping from the
discovered malicious behaviors to relevant ATT&CK techniques, as well as to the related API calls.

Index Terms—Cyber threat intelligence, dynamic analysis, malware behavior analysis, MITRE ATT&CK framework

1 INTRODUCTION derive a clear picture of the attack, and take the necessary
action to stop or mitigate the attack. The strength of ATT&CK,
attacks have proliferated recently, incurring dam-
C YBER
ages that cost individuals and companies dearly. A
powerful proactive defense collects information about
one of most popular OSINTs, is its structure and openness in
collecting and sharing cyber threat intelligence. In this study,
we crawl the contents of ATT&CK to build the needed knowl-
known attacks and comprehensively understands malicious
edge about malware behavior to facilitate dynamic malware
behaviors, and further exploits this knowledge to interdict
analysis via deep learning.
and disrupt attacks or preparations for attack [1], [2]. Thus
Information about adversaries is commonly published in
it is crucial to grasp the characteristics of malicious behavior
cyber threat intelligence (CTI) reports presented with seman-
and the resources used therein. Open source intelligence
tic descriptions and lists of manipulated resources. Compre-
(OSINT) assimilates experience and knowledge from the
hension of CTI is a large-scale data-driven process that
cybersecurity community to form a common knowledge
involves systematic analysis of observations, including mal-
base for cyber threat studies that best supports a proactive
ware, suspicious events, and other rapidly evolving cyberse-
defense [3].
curity data. To facilitate CTI usage, many studies [7], [8], [9],
The attack development life cycle, such as Lockheed
[10], [11] focus on collecting, analyzing, and extracting evi-
Martin’s cyber kill chain [4], the MITRE ATT&CK (Adversar-
dence such as indicators of compromise (IoCs) in CTI reports.
ial Tactics, Techniques and Common Knowledge) framework
Dealing with increasingly sophisticated cyber threats and
(hereafter referred to as ATT&CK) [5], and Mandiant’s adver-
obtaining a overall picture of the fast-evolving attack scenario
sary life cycle [6], describes the adversary process at each
from OSINT CTI helps cybersecurity analysts handle poten-
stage of the attack. Take for example ATT&CK: the frame-
tial attacks as they are unveiled.
work is designed to describe the attacker intent and malicious
Holmes [12] and RapSheet [13] are state-of-the-art sys-
behavior at each tactic stage. Once all malicious behaviors are
tems that apply manually crafted expert rules to discover
compiled, the cybersecurity analyst can correlate them to
advanced persistent threats or tactics, techniques, and pro-
cedures (TTPs) to detect potential attacks on their host sys-

Yi-Ting Huang, Kai-Chieh Lo, and Meng Chang Chen are with the tems. In this paper, instead of investigating a computer’s
Academia Sinica, Taipei 115024, Taiwan. E-mail: {ythuang, sage66730, system log, we focus on analyzing the dynamic behavior of
mcc}@iis.sinica.edu.tw. malware using the knowledge from ATT&CK and neural

Chi Yu Lin, Ying-Ren Guo, and Yeali S. Sun are with the National Taiwan networks.
University, Taipei 10617, Taiwan. E-mail: {r07946012, r09921a01,
sunny}@ntu.edu.tw. To analyze malware activity, dynamic analysis tools such
Manuscript received 28 Dec. 2020; revised 23 July 2021; accepted 26 Sept. 2021.
as Cuckoo Sandbox [14], CWSandbox [15], and APIf [16]
Date of publication 11 Oct. 2021; date of current version 14 Mar. 2022. record execution steps in detail to generate execution traces.
This work was supported in part by CITI, Academia Sinica, and by MOST Cuckoo Sandbox further applies ATT&CK with rules contrib-
under Grants 110-2218-E-001-001-MBK and 109-2221-E-001-010-MY3. uted by volunteers to detect malicious behavior. However,
(Corresponding author: Yi-Ting Huang.)
Recommended for acceptance by Special Issue on XAI-CTI. due to the crowd-sourced nature of Cuckoo Sandbox, the
Digital Object Identifier no. 10.1109/TDSC.2021.3119008 completeness and timeliness of the contributed rules (called
1545-5971 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See ht_tps://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
HUANG ET AL.: OPEN SOURCE INTELLIGENCE FOR MALICIOUS BEHAVIOR DISCOVERY AND INTERPRETATION 777

Cuckoo Signatures) may not be consistent with ATT&CK.

Therefore, in this study, we construct regular expression
rules to implement knowledge within ATT&CK for use as a
labeling method, in addition to the Cuckoo Signatures, for
later use in deep learning. We crawl the MITRE website to
extract and organize the relations of TTPs and malware that
can be used as another labeling method. To account for
MITRE website updates, all labeling processes must be auto-
matic and incremental.
With the rapid development of artificial intelligence (AI),
data-driven methods (i.e., machine learning and deep learn-
ing approaches) can be used for cyber threat analyses such as
malware analysis [17], [18] or attack analysis [19], [20]. How-
ever, AI methods are seen as black boxes, which can create
confusion and doubt [21]. For example, when security ana-
lysts analyze malware using an AI model, questions may arise
such as “How can I trust the decision-making of this model?”
or “How has the model come to this decision for a given mal-
ware sample?” To understand how a model learns from data,
studies in disciplines as varied as image caption genera-
tion [22], sentiment analysis [23], and electronic health record
applications [24] incorporate attention mechanisms in neural
network models to interpret model outcomes. Generally, the
attention mechanism, which calculates the probability distri- Fig. 1. Mapping knowledge from ATT&CK to a malware trace. The top
MITRE webpage is about sub-technique T1547.001 and the bottom
bution over inputs, is intuitively seen as an indicator of the shows the API calls of JCry to partially carry out the technique.
model’s focus, as a convincing explanation is to humans. In
this study, we similarly employ the attention mechanism to
via a well-crafted neural network model. Within an execution
interpret the model outcome. These two questions can be
trace, malicious behaviors (TTPs) are undertaken by one or
answered if the outcome of the model is not merely a decision
many API calls and may be described by CTI reports.
but also concerns semantics, as is the case with TTPs, API
MAMBA makes novel use of the information presented in
calls, and the associated resources in this study.
ATT&CK as the pivotal reference in addressing the challenges
In this study, we examine whether OSINT has a role to
in malware behavior interpretation. For instance, in Fig. 1, the
play in using intelligence to better interpret malware. Our
sub-technique T1547.001 Boot or Logon Autostart Execution:
goal is to discover malicious behavior based on the analysis
Registry Run Keys / Startup Folder refers to adding an execut-
of an execution trace of Windows malware, to interpret the
able program to a startup folder to maintain a foothold. This
discovered behavior as a collection of techniques (TTPs),
sub-technique can be identified when a malware sample
and to find the API calls and system resources associated
attempts to add a malicious payload to the startup folder.
with these TTPs. Three main aspects of this study are as
High-level descriptions of TTPs in ATT&CK serve as explana-
follows:
tions of malicious behavior, and can be used to link to low-

OSINT for cyber threat intelligence: Assimilating level execution traces of malware by MAMBA. MAMBA is
threat intelligence from OSINT to intercept malicious thus:
behavior requires an information extraction mecha-
Explainable. In contrast to traditional malware detec-
nism and a competent neural network model. tion and malware classification tasks, we discover

Malicious behavior discovery: Linking a low-level high-level semantic TTPs associated with low-level
malware execution trace to a high-level description API calls for a malware sample.
of malicious behavior (i.e., TTPs) requires that we
Comprehensive. Malicious behavior is composed
close the semantic gap between them. of a series of operations and resources. By taking

Malicious behavior explanation: Helping the security into account resource dependencies, MAMBA finds
analyst to better understand the captured malicious related TTPs and their API calls.
behavior, the associated API calls and manipulated
Extendable. MITRE ATT&CK keeps up with the con-
system resources constitute observable evidence. stant evolution of cybersecurity threats. MAMBA is
As dynamic malware analysis has been widely used for designed to automatically retrieve the ATT&CK con-
malware analysis [17], [25], [26], [27], we present MAMBA tents to reflect these changes.
(MITRE ATT&CK based Malicious Behavior Analysis), a sys- To summarize, our work offers the following
tem that addresses the above aspects. An execution trace contributions:
includes the sequences of the invoked API calls during execu-
tion of a malware sample. MAMBA starts by extracting the
MAMBA incorporates knowledge from ATT&CK in
TTPs and their corresponding resources from the MITRE deep learning analysis to discover malicious behavior.
website and cited references, and discovers TTPs from mal-
The MAMBA design and methodology are examined
ware and their corresponding API execution call sequences extensively using the contents of MITRE as well as
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
778 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 19, NO. 2, MARCH/APRIL 2022

correspond to techniques in ATT&CK. For example, the

command “cmd.exe /c powershell -WindowStyle Hidden Start-
Process Dec.exe -WindowStyle maximized” can be found on the
ATT&CK TTP webpages (T1059.001 and T1059.003). While
such malicious behavior is traditionally represented by indi-
cators of compromise (IoCs) or signatures in intrusion detec-
tion systems (IDSs), in ATT&CK they are presented using
natural language descriptions. In this study, the abundance
and openness of the ATT&CK information facilitates the use
of information retrieval techniques to collect and convert this
data into knowledge for later use.

2.2 MITRE ATT&CK Framework

ATT&CK is a document source of post-compromise adver-
sarial tactics and techniques based on real-world observa-
tions. From the contents of [5], ATT&CK is a behavioral
Fig. 2. Life cycle of a malware sample from malware family JCry. model that consists of adversary tactics, techniques, and
procedures (TTPs).
real-world data. The evaluation outcomes meet the
A tactic represents the goals of an adversary. It cate-
challenges. gorizes the attack life cycle into different stages.

The study shows that the open-source intelligence of
A technique/sub-technique represents the technical
the MITRE ATT&CK framework facilitates cyberse- means through which goals are accomplished. A
curity applications. sub-technique, inheriting a technique, corresponds
to more specific action.
2 BACKGROUND AND MOTIVATION
A procedure in ATT&CK is exemplified by real-world
In this section, we introduce a motivating example and examples, either software or an adversary group, to
present insight into using ATT&CK to interpret the mali- show their use of techniques or sub-techniques.
cious behavior lifecycle from an execution trace. Each tactic serves as a class of techniques implemented by
software to accomplish the tactic. For example, to establish
persistence (tactic), JCry (malware) may add a downloaded
2.1 Motivating Example payload to the startup folder (sub-technique T1547.001).
We analyze a malware sample (MD5 c86c75804435efc380- In recent years, this framework has become popular for
d7fc436e344898) classified as a member of the JCry fam- describing the attack life cycle of either malware or an adver-
ily [28], [29]. Fig. 2 depicts the JCry life cycle with an sary group. This paper will focus on the techniques of all
emphasis on its created processes, discovered TTPs, and the stages of Windows malware samples from ATT&CK. In this
manipulated resources. JCry is ransomware disguised as an study, techniques refer to techniques as well as sub-techni-
Adobe flash player update installer. Once it is clicked, it cre- ques (hereafter techniques) and resources refer to files, librar-
ates malicious files msg.vbs (~), Enc.exe (), and Dec.exe (&), ies (modules), registries, processes, and networks. Malicious
and stores these in the startup folder to maintain its persis- behavior of a malware sample can be represented by one or
tence (in ATT&CK this is identified as T1547.001 Boot or more techniques; the attack life cycle (kill chain) of malware
Logon Autostart Execution: Registry Run Keys / Startup Folder). is composed of a series of techniques.
These programs are executed when the user logs in. Execut-
ing msg.vbs displays an “Access Denied” message to warn
that the Adobe Flash Player failed to update (T1059.005 Com- 2.3 Techniques and Execution Trace
mand and Scripting Interpreter: Visual Basic). The executable The MITRE website provides descriptions of techniques for
file Enc.exe encrypts the user’s files for ransom (T1486 Data which MAMBA extracts resources and matches them with
Encrypted for Impact), and also deletes shadow copies using a arguments of the API calls. This strategy is also supported by
command to prevent recovery (T1490 Inhibit System Recov- [30], in which a comprehensive analysis demonstrates a
ery), after which it launches Dec.exe using PowerShell to dis- strong correlation between ATT&CK techniques and Win-
play the ransom note (T1059.003 Command and Scripting dows API calls. As shown in Fig. 1, the resource mentioned in
Interpreter: Windows Command Shell, T1059.001 Command and the webpage for technique T1547.001 Registry Run Keys /
Scripting Interpreter: PowerShell). Startup Folder indicates that T1547.001 may be discovered if
We offer two observations. First, the manipulated resour- resource “C:\\Users\\...\\Startup\\Enc.exe” is accessed
ces are useful to group processes and API calls which work in an execution trace. As the figure shows, this specific
together to carry out malicious activities. For example, resource can be found in the API calls “NtCreateFile” and
the manipulated resource Enc.exe () is used by the “NtWriteFile”, in which the connection constitutes
“malware.exe (PID=2932)”, “enc.exe (PID=912)”, and “dec. an important clue to understanding the malicious activity.
exe (PID=3572)” processes for its creation, execution, and Following this procedure, MAMBA’s neural network model
deletion. Second, the malicious activities associated with is designed to learn the associations between TTPs and execu-
these manipulated resources, e.g., files and commands, may tion traces.
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
HUANG ET AL.: OPEN SOURCE INTELLIGENCE FOR MALICIOUS BEHAVIOR DISCOVERY AND INTERPRETATION 779

addition, in resource-technique binding we use a neural net-

work to learn the connection between resources and techni-
ques from ATT&CK to enable the proposed neural network
to associate the embedding of resources from traces to tech-
niques from ATT&CK.
Once the extraction phase and the fusion phase are com-
plete, threats are identified by detecting techniques from a
malware sample. First, API call embeddings are generated
from the output of the fusion phase and are processed by
gated recurrent units (GRUs) to obtain a sequential hidden
Fig. 3. MAMBA workflow. vector. Attention mechanisms are applied to highlight the
relevance between resources and API calls as well as depen-
3 SYSTEM DESIGN dencies among the bindings and API calls. Finally, threat
The main design goal of MAMBA is to align a resource anno- identification yields the compromised techniques.
tated with a TTP in ATT&CK to a manipulated resource used
by malware. In this paper, matrices are represented using 3.2 Knowledge Extraction From MITRE ATT&CK
uppercase characters and vectors are represented in boldface Framework
using lowercase characters. The first step of knowledge extraction is to extract a disclosed
resource r related to a technique y as a tuple fr; yg from
the webpage for every technique in the MITRE ATT&CK
3.1 Overview framework. The regular expressions for r extraction from a
A high-level overview of the MAMBA workflow is shown in token, a shadowed token (a token with gray background), or
Fig. 3: it is composed of an extraction phase, a fusion phase, a sentence in the MITRE website are expressed in Table 1. A
and a threat identification phase. The extraction phase shadowed token is a complete path of a resource or com-
includes technique extraction by extracting knowledge tuples mand line; for example, the filename “C:\\Users[Username]
from ATT&CK, and malware execution trace generation \\...\\Startup” in Fig. 1 is a shadowed token, which can be
from a sandbox. The technique pages in ATT&CK present recognized as the regular expression for directory (fd). Some
use cases performing the corresponding techniques. These resources in a sentence are not marked with gray back-
use cases are treated as observable clues by which to detect ground. For example, the sentence “... usage of the Windows
techniques, and are extracted by MAMBA as the technique Script Host (typically cscript.exe or wscript.exe) ...” from the
knowledge. We also consider series of API calls and sets of MITRE webpage of T1059.005 Command and Scripting Inter-
manipulated resources from an execution trace as a sequence preter: Visual Basic consists of two non-shadowed resources
of operations executed by malware. ATT&CK technique- “cscript.exe” and “wscript.exe”, which can be recognized by
related knowledge as well as execution-trace API calls and the composition of the regular expressions for filename (fn)
resources are collected in the extraction phase. and extension (fe). In summary, we collect from the MITRE
The fusion phase involves resource embedding and website 988 resources associated with 229 techniques, form-
resource-technique binding. Even if the collected knowledge ing 2100 fr; yg pairs.
from ATT&CK and execution traces indicate the same mali- To evaluate the accuracy of knowledge extraction from the
cious behavior, their constitutions may be different. The MITRE ATT&CK framework, we randomly selected 50 tech-
designed embedding mechanism maps resources to fixed- niques to manually label technique-related resources from
size vectors while preserving their semantic properties. In the corresponding webpages, and measured the accuracy of

TABLE 1
Regular Expressions for Resource Categories

y
Abbreviations C, F, L, R, P, and N represent category, file, library, registry, process, and network, respectively.
z
fd = directory, fn = filename, fe = extension, le = library extension, rk = root key, sk = subkey path, CLSID = class id, cmd = command, domain = domain name.
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
780 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 19, NO. 2, MARCH/APRIL 2022

resource extraction. Two authors read every technique web- 1X

max log pðyjer ; Wz Þ: (1)
page on ATT&CK to find related resources and label a N
resource with a word or a phrase, such as a filepath, a registry, We apply Wz to derive the hidden vector zr for each
or a command, used to implement the technique. Their agree- resource r, computed as
ment (Cohen’s kappa = 0.739) [31] is considered substantial.
When they disagreed on a labeled resource, the third author zr ¼ sðWz er Þ; (2)
joined the discussion to make a decision. This resulted in a
total of 1159 resources annotated among 50 techniques. The where s is the activation function. For a manipulated
average number of resources per technique was 23.18 (SD = resource extracted from an API call, we use the same
20.44). To verify the MAMBA extraction capability, we com- embedding function in Section 3.3 to transform r into er ,
pared MAMBA with the shadowed tokens presented in the and further compute the hidden vector z in (2), which can
ATT&CK web pages: it outperforms (p < 0:001 by be considered its contribution to TTPs for later neural net-
McNemar’s test [32]) the ATT&CK shadowed tokens. work processing.
MAMBA achieves a precision, recall, and F1 of 0.906, 0.770,
and 0.833, respectively, with 19.72 (SD = 18.5) discovered 3.5 Threat Identification Phase
resources, whereas the ATT&CK shadowed tokens achieve The goal of the threat identification phase is to identify mali-
only 0.924, 0.475, and 0.627, respectively, and discover 11.90 cious behavior (TTPs) y from a malware execution trace
(SD = 11.98) resources. with API calls x ¼ fx1 ; x2 ; . . . xpjT j g. Formally, given a
training set of M pairs of fx; yg, the objective of the learning
function is to maximize the average log probability with
3.3 Resource Representation respect to the MAMBA neural network with all trainable
In this step, each resource found from ATT&CK and the weights u, including Wc ; Wn ; Wv , and Wd (which will be
execution trace is embedded into a resource embedding er . defined later):
An embedding maps a variable-length resource to a fixed-
length feature vector in the embedding domain. As resour- 1 X
max log pðyjx; uÞ: (3)
ces are not necessarily represented in the same way between M m
ATT&CK and the execution traces, we seek to preserve their
closeness in the embedding domain for later neural network The attack life cycle can be recognized by a series of techni-
processing. For instance, as shown in the example in Fig. 1, ques identified from API calls with their arguments.
the startup folder path in ATT&CK consists of the token A resource-based API call group is defined as a collection
“Users[Username]”, which is slightly different from the of the related API calls that share the same resource. Given a
“UsersnnBaka” in the execution trace. To ensure that the malware execution trace, the threat identification phase pro-
neural network learns properly, their embeddings should duces resource-based API call groups for each process, after
be close. which it compares resource-based API call groups with
We employ the paragraph vector distributed memory other call groups in all processes and predicts possible tech-
method (PV-DM) [33] to transform a resource into an n- niques. The structure of the threat identification phase is
dimensional real-valued vector. PV-DM is an unsupervised shown in Fig. 4.
learning algorithm to transform a sentence, a paragraph, or An execution trace is composed of the traces of all pro-
a document into a fixed-length vector. As it is based on cesses; each process trace is a sequence of API calls. A single
skip-gram embedding techniques, it preserves semantics API call x consists of a category c, an API function name n,
and word ordering to facilitate the use of embeddings for and one or more argument values (i.e., resources). In Fig. 1,
similarity computation while maintaining the closeness for instance, API call “NtCreateFile” belongs to the “file”
property. In this study, we tokenize each resource and treat category and has argument values such as “C:\\Users\\...
each token as a word in the PV-DM model. To reduce the \\Startup\\Enc.exe”. The embedding of API call x is a con-
influence of unseen words, we build a resource vocabulary catenation of the embeddings of category ec , API name en ,
set by excluding out-of-vocabulary and rare words whose and resources er1 , er2 , er3 (only three resources are consid-
frequency is lower than a given threshold. Once the learning ered):
of the PV-DM for resource is completed, the resource
ex ¼ ½ec ; en ; ½er1 ; er2 ; er3 ; (4)
embedding function is ready.
where ½;  is concatenation, and er1 ; er2 ; er3 are from the PV-
3.4 Resource-Technique Binding DM model in Section 3.3.
Once resource embedding er is generated, the next step is to
build a neural network to learn the relation between a ec ¼ Wc xc (5)
resource and a technique. A resource can be seen as a plausi- en ¼ Wn xn ; (6)
ble clue to the implementation of a technique y to achieve its
tactical intent. A multiple layer perceptron (MLP) is trained where Wc and Wn are the weight matrices of category c and
using the pairs fer ; yg from ATT&CK, used to predict the like- API name n, and xc and xn are one-hot encodings of cate-
lihood of techniques given a resource from an execution trace. gory and API name. Wc and Wn are trained during the train-
Formally, given a set of N pairs of fer ; yg from ATT&CK, ing phase of the MAMBA neural network model.
the objective of the learning function is to maximize the To preserve ordinal information, the sequence of the API
average log probability with respect to the MLP weights Wz : call embeddings in a process is handled using gated recurrent
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
HUANG ET AL.: OPEN SOURCE INTELLIGENCE FOR MALICIOUS BEHAVIOR DISCOVERY AND INTERPRETATION 781

Also, a binding embedding zi for a resource ri can be

acquired in (2) as a feature corresponding to technique y. The
group vector gi is combined with the binding embedding zi
to yield the binding group embedding bi for each resource:
bi ¼ ½gi ; zi : (11)

For each resource of a process, the binding group embedding

b includes information not only from API calls but also from
ATT&CK. At this step, each process is represented by a collec-
tion of binding group embeddings.
The next step is to aggregate the binding group
embeddings from each process and produce a malware
representation d for prediction. As shown in the example in
Fig. 2, resources may be manipulated among processes; thus
we apply a self-attention mechanism to highlight dependen-
cies among the binding group embeddings. The self-atten-
tion mechanism allows each binding group embedding to
interact with the other embeddings to determine which
should get more attention:
vi ¼ softmaxðWv bi Þ; (12)

where Wv is weight matrix of the two-layer dense network.

The malware representation d is the aggregation of the group
attention scores v and the binding group embeddings b:
Fig. 4. MAMBA neural network model. d ¼ vb: (13)
The technique prediction task is a multi-label classification
units (GRUs). A member of the recurrent neural network fam- problem with a sigmoid layer at the end of the classifier.
ily, GRUs compute efficiently with performance comparable The predicted probability of each technique produced by
to LSTMs [34]. As GRUs are commonly used in malware anal- the sigmoid function is independent of the others:
ysis [35], [36], MAMBA considers GRUs to process the vari-
able-length input sequence ex ¼ fex1 ; ex2 ; . . . ; exjT j g and y ¼ sigmoidðWd dÞ: (14)
produce a hidden state h. At time step t, the hidden state ht of
the GRU is updated by Algorithm 1 concludes the operations of MAMBA neural
network model described above.
ht ¼ GRUðht1 ; ext Þ: (7)
Algorithm 1. MAMBA Neural Network
GRUs learn a probability distribution over an input sequence
Input: an execution trace x
such that the output h encodes sequential information from Output: a set of TTPs y
the first API call to the current API call. 1: while u has not converged do
To find the connection between each pair of API call xt 2: Forward Propagation:
and system resource ri in a process, we use a resource atten- 3: for each process p do
tion mechanism as the score function that is the maximum 4: r Extract a set of resources from xp
value of the inner product of resource embedding eri against 5: er Get resource embedding in Section 3.3
the three resource embeddings er;t of API call xt in (8): 6: zr Get binding embedding in (2)
  7: ex API_call_embedding(x x) in (4)
eri er1 t eri er2 t eri er3 t
scoreðeri ; xt Þ ¼ max ; ; : (8) 8: h GRU(eex ) in (7)
jeri jjer1 t j jeri jjer2 t j jeri jjer3 t j 9: for each er in er do
10: ser ;hh resource_attention(er , h) in (9)
The result is normalized to derive the resource attention 11: gr group_embedding(ser ;hh , h ) in (10)
weights sit as a distribution over all API calls: 12: br binding_group(gr , zr ) in (11)
expðscoreðeri ; xt ÞÞ 13: end for
sit ¼ PjT j : (9) 14: end for
t0 ¼1 expðscoreðeri ; xt ÞÞ
0 15: v group_attention(bb) in (12)
16: d malware_representation(vv; b) in (13)
Given these attention weights, we compute a group vector gi 17: y sigmoid(d) in (14)
as the weighted sum of the API call hidden states h for a cer- 18: Backward Propagation:
tain resource ri : 19: Conduct backward propagation with Adam;
20: end while
X
jT j 21: # Use the trained network to discover TTPs y of an
gi ¼ sit ht : (10) execution trace x
t¼1
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
782 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 19, NO. 2, MARCH/APRIL 2022

4 EMPIRICAL STUDIES TABLE 2

Dataset Statistics
We designed experiments to answer the following critical
questions. Source ATT&CK MalShare

Q1: How effectively does MITRE knowledge improve Sample 2335 23655
TTPs extraction? Process 3.8223.91 3.1716.12
API call 2023.4723415.45 4346.2926399.03

Q2: How effectively are the true TTPs extracted from
Selected API calls 514.541199.05 642.051203.08
a given malware sample using MAMBA? Resource 329.551296.19 531.216563.29

Q3: What makes MAMBA capable of identifying TTPs per sample 5.943.78 2.802:98x, 12.705:89{
TTPs?
x

Q4: How well does MAMBA perform against realis- Cuckoo Signatures {RegExp
tic attack campaigns?

Q5: How well does MAMBA locate API calls associ- provided by crowd intelligence, and RegExp, a regular
ated with the predicted TTPs? expression set generated based on the TTP descriptions in
For Q1 and Q2, we collected two datasets from MITRE ATT&CK which recognizes 169 TTPs. To label each malware
and MalShare [37] and used three labeling methods: MITRE, sample, we applied these label methods to the Big dataset.
Cuckoo, and RegExp. We compared the performance of We randomly divided the datasets into a training set
MAMBA, two rule-based methods, five traditional machine (80%), a development set (10%), and a testing set (10%). We
learning methods, and three deep learning approaches in continued the above process until the F-test on the TTP dis-
Evaluation 1. To answer Q3 and understand the contribu- tributions of the three sets showed no significant differences.
tions of each component, we further conducted an ablation
study. To answer Q4, we analyzed malware samples pro- 4.2 Implementation Settings
vided in the ATT&CK APT29 description to examine We used Cuckoo Sandbox [14] to obtain execution traces of
MAMBA’s capabilities. Finally, to answer Q5, we present a malware samples. In the MAMBA implementation, the PV-
case study showing that MAMBA locates the API calls and DM model in Section 3.3 for resource embedding used the
manipulated resources associated with the predicted TTPs. Gensim library [39] to produce a 100-dimension embedding
vector as er . For the PV-DM model parameters, the mini-
4.1 Data Collection mum frequency threshold for each resource token was set
Here we describe the collection of samples and labels used in to 5, and the size of the context window was 2. For training
the evaluations. The MITRE ATT&CK framework (version 7) both resource-technique binding in Section 3.4 and the
for Windows includes 12 tactics, 148 techniques, 214 sub- MAMBA neural network in Section 3.5, we used the loss
techniques, and 378 pieces of software. We gathered malware function with cross entropy and the Adam optimizer to
samples and their corresponding TTPs presented in update the parameters, with an initial learning rate of 0.01.
ATT&CK as the ground truth (note that this association is The size of binding embedding zr was set to 50. The iden-
termed ATT&CK labeling). That is, each sample in the tity function was used for the s function in (2). The weight
ATT&CK dataset is collected based on the documents refer- matrix Wz was for the two-layer dense networks, set to
enced on each technique page on the MITRE ATT&CK web- R100100 and R10050 . We set each API call and GRU hidden
site; its labels (TTPs) are determined accordingly. For each state size to 400 and 100 respectively, and set the maximum
technique page on the MITRE ATT&CK website, the mali- timestamp t to 500. For category and API name embedding,
cious activity is described by one or more referenced docu- the weight matrices Wc and Wn were R1007 R10036 . Both of
ments. We accessed these documents and used regular the weight matrices Wv and Wd were two-layer dense net-
expressions to crawl and extract the MD5, SHA1, and works: Wv1 and Wv2 were set to R15064 and R641 , and Wd1
SHA256 hashes of the associated malware samples. To vali- and Wd2 were R15064 and R64jyj .
date the extracted hashes, we uploaded the hashes to Viru-
sTotal [38] for verification. If a reference document involved 4.3 Evaluation 1: MAMBA Evaluations
more than one technique, we discarded it to eliminate ambi- In this section, we compare the performance of MAMBA
guity. We also discarded inaccessible references such as those and other methods using the ATT&CK and Big datasets to
with anti-crawler prevention, machine-unreadable content, answer Q1 and Q2. Table 3 compares the performance of
and broken links. A total of 2,335 malware samples (referred MAMBA with two rule-based systems (Cuckoo Signatures
to as the ATT&CK dataset) were collected corresponding to and RegExp), five traditional machine learning methods,
67 techniques. We also collected 23,655 malware samples i.e., LinearSVC (Linear Support Vector Classifier), Random
from MalShare [37] verified as malware by VirusTotal [38] Forest, Decision Tree, GaussianNB (Gaussian Naive Bayes),
from January 2018 to April 2019. The combination of the and KNeighbors (K-nearest Neighbors) in Scikit-learn [40],
ATT&CK and MalShare datasets is called the Big dataset. and three conventional neural networks, i.e., MLP (multi-
The statistics of the two datasets are shown in Table 2. For layer perceptron), RNN (recurrent neural network), and
instance, the average number of process per malware sample LSTM (long-short term memory). Machine learning-based
is 3.82, and the average API calls and resources per process and deep learning-based methods are commonly used in
are 2,023.47 and 329.55 respectively, for the ATT&CK dataset. malware analysis [35], [41], [42], [43]. As traditional
Since samples from MalShare lack TTP labels, we consid- machine learning methods could not accept a complete exe-
ered two rule-based label methods: Cuckoo Signatures (Ver- cution trace as input, we took the first five hundred API
sion 2.0.7), which recognizes 43 TTPs using signatures calls (with API categories and API function names only) of
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
HUANG ET AL.: OPEN SOURCE INTELLIGENCE FOR MALICIOUS BEHAVIOR DISCOVERY AND INTERPRETATION 783

TABLE 3
Comparison of Various Methods for TTP Discovery

y
p <0.05, z p <0.01, ? p <0.001.
Mann–Whitney U test is applied to compare the performance of MAMBA against that of each compared method.

an execution trace and used PCA (principle component as it ignores the semantics and structure of malware. To
analysis) [44] to reduce the dimensions of the execution sum up, MAMBA yields superior performance due to the
trace. For the traditional machine learning methods, the resource attention and group attention mechanisms as well
reduced API call sequences and associated TTPs were used as ATT&CK knowledge and resource embeddings.
as input, whereas for the conventional deep learning mod- For the second and third set of comparisons, first, the
els, MAMBA’s API call embeddings and associated TTPs agreement (Krippendorff’s a ¼ 0:120) [46] between
were used as input. Cuckoo Signatures and RegExp is low, which demonstrates
Evaluation 1 has 3 sets of comparisons using ATT&CK, the subjective nature of rule-based systems. Conventional
Cuckoo Signatures, and RegExp labels sequentially as rule-based approaches exhibit limited performance on TTP
ground truth for comparisons; the first set is used with the discovery, as shown in Table 3. Moreover, given sufficient
ATT&CK dataset, and the other two are used with the Big training data, learning-based approaches learn from latent
dataset. Table 3 compares various methods for TTP discov- information in the data, and yield better predictions using
ery in terms of precision (P), recall (R), F1 score, false positive the Big dataset than when using the ATT&CK dataset.
rate (FPR), and false negative rate (FNR), including the statis- When using these two labeling methods MAMBA achieves
tical significance level as per the Mann–Whitney U test [45] around 90% in terms of precision, recall, and F1 score, and
in comparison to MAMBA. MAMBA has significant perfor- produces a 0.6% false positive rate and a 6% false negative
mance differences with most of the compared methods. Both rate, which is the best performance of all the methods.
Cuckoo Signatures and RegExp perform poorly as the TTPs From Table 3, MAMBA outperforms in terms of preci-
cover only part of ATT&CK labels on the ATT&CK dataset; sion, recall, and F1 at 0.667, 0.569, and 0.591 respectively. To
as a result, they achieve F1 scores of 0.049 and 0.099 and false answer Q1, the result shows the ATT&CK labeling and
negative rates of 0.858 and 0.659. The five traditional dataset does provide useful knowledge to extract TTPs
machine learning methods perform slightly better than the from execution traces, but due to the limited number of mal-
rule-based approaches, with F1 scores ranging from 0.389 to ware samples and TTP labels, the resultant performance is
0.260, as they learn the relationship between API calls and moderate. For question Q2, we conclude that: 1) MAMBA
TTPs. The conventional neural networks learn from the accurately identifies TTPs compared to rule-based and other
embeddings of API calls, yielding high precision, i.e., 0.556, learning-based approaches on the Big dataset using both
0.461 and 0.552 for MLP, RNN, and LSTM, and relatively labeling methods. 2) Comparing the results on the ATT&CK
low false positive rates, i.e., 0.010, 0.009, and 0.006. Com- dataset against those on the Big dataset, given sufficient
pared to the deep learning models, MAMBA demonstrates samples and labels, MAMBA achieves high precision, recall,
significant improvements due to its preservation of resource and F1, attesting the efficacy of the MAMBA neural network
and group dependencies in the attention mechanism and its model.
utilization of resource and binding embeddings.
Take JCry as an example: as shown in Section 2.1, JCry is 4.4 Evaluation 2: Ablation Test
identified with six malicious behaviors: T1547.001, T1486, MAMBA includes knowledge from ATT&CK (binding
T1490, T1059.001, T1059.003, and T1059.005. Whereas the embeddings), group dependencies (group attention), and
MLP model correctly labels T1547.001 only, MAMBA addi- API calls (resource attention). We conducted an ablation
tionally identifies T1059.001 and T1059.003 because it makes study to understand the contributions of each component to
use of ATT&CK resource knowledge such as “powershell”, TTP identification using the RegExp labels on the Big dataset.
“-WindowStyle Hidden” and “cmd.exe” (see Section 4.6.2 for Table 4 shows that after the removal of one or two compo-
details). For the LSTM model, although it handles sequen- nents, MAMBA still performs well, and indeed demonstrates
tial data, it fails to identify any of JCry’s malicious behavior statistically significant improvements by the Mann–Whitney
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
784 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 19, NO. 2, MARCH/APRIL 2022

TABLE 4
Ablation Test Results on Big Dataset

P R F1
MAMBA 0.945 0.954 0.949
? ? ?
– group attention 0:942 0:935 0:938
– resource attention 0.911 0:920y 0:915y
?
– binding embedding 0:940 0.933 0.936
0:949z
? ?
– (resource + group attention) 0:925 0:936
? ? ?
– (binding embedding + group attention) 0:886 0:932 0:908
? ? ?
– (binding embedding + resource attention) 0:934 0:912 0:919
y
p <0.05, z p <0.01, ? p <0.001. Mann–Whitney U test is applied to compare
the performance of MAMBA against that of each compared method.

U test [45] against most of the models. All components have

positive effects on the F1 score; in particular, resource atten-
tion, that is, measuring the association between manipulated
resources and API calls, has an obvious impact. For instance,
MAMBA discovers that a sample (MD5 b9f7ff253e508d01c- Fig. 5. Comparisons of MAMBA and security vendors on 56 TTPs listed
fa6defccdbad400) extensively manipulates the resource in APT29 evaluations.
“aevce.exe” among invoked API calls such as RegSetValue,
NtCreateFile, and CreateProcessInternalW to implement the larger the circle is, the more vendors recognize the TTP;
T1547.001; when resource attention is removed, MAMBA true positives and false negatives of MAMBA prediction are
does not discover this TTP. This verifies that resource atten- represented with different colors. In addition, MAMBA rec-
tion measures the correlation between resources and API ognizes TTPs—e.g., T1056.001 Input Capture: Keylogging and
calls, enhancing the ability of MAMBA to discover TTPs. T1059.003 Command and Scripting Interpreter: Windows Com-
Another interesting finding is that precision increases when mand Shell—beyond the 56 TTPs in the ATP29 Evaluation,
only considering binding embedding, (“– (resource attention although the discovery of these two TTPs is consistent with
+ group attention)”); one reasons is because it generates the [49]. However, MAMBA does produce false positive TTPs,
fewest TTP predictions to increase precision. Taking T1057 such as T1546.010 Event Triggered Execution: AppInit DLLs,
Process Discovery as an example, 33 resources such as “tasklist” which is misidentified because MAMBA treats the registry
are extracted from ATT&CK. MAMBA recognizes that sam- subkey “... Windows\LoadAppInit_DLLs” in an execution
ple MD5 203d4f3541012300368ee97420f46f5f attempts to list trace as “...\AppInit_DLLs” in the MITRE webpage.
processes with a command, tasklist /fi “imagename eq rfusclient. To answer Q4, MAMBA demonstrates the feasibility of
exe”, whereas MAMBA with fewer components fails to iden- capturing TTPs on malware samples used in a threat group.
tify the TTP. This shows that the relation between the com- However, there are still some shortcomings due to the sta-
mand and the extracted resource transfers successfully to the tistical characteristics of deep learning and the size limita-
binding embedding, boosting performance. tion of ATT&CK dataset.
To answer Q3, each component of MAMBA, binding
embedding, group attention and resource attention helps to 4.6 Evaluation 4: Resource and API Locating
discover TTPs. This section presents a post-processing heuristic to locate
the APIs and manipulated resources of malicious behaviors,
4.5 Evaluation 3: APT29 Case Study
and discusses a case study that demonstrates the effective-
The ATT&CK evaluations use known attack methods of ness of API call locating.
APT groups such as APT29 [47] to evaluate cybersecurity
products. In 2019, 21 security vendors participated in the
evaluation using this emulated adversary environment. 4.6.1 Inference and API Locating
With this experiment we examined the capability of At inference time, MAMBA predicts TTPs y^ and locates
MAMBA trained with ATT&CK dataset and ATT&CK related API calls x  x for a given execution trace x . Given
labeling in dealing with malware samples used in a well- the group attentions in (12) and the resource attentions in
known APT29 adversary, and compared the predicted TTPs (9), we find the dominant resources for discovering TTPs
with the ATT&CK APT29 Evaluation results [48]. The mal- and locating the related API calls in a process. More specifi-
ware samples deployed in APT29 are well-documented in cally, a set of manipulated resources ry^ is selected based on
[49], [50], [51]; we collected these 310 malware samples for two criteria: i) the similarity between the resources extracted
the evaluation and compared the outcome with those of the from ATT&CK and the manipulated resources, and ii) the
attending vendors. group attention. The similarity scores reveal the likelihood
Taking 310 execution traces as inputs, MAMBA discov- that a certain resource is being manipulated to implement a
ered 67 TTPs among 9 tactics. (As a side note, when trained TTP. The group attention measures how much information
with the Big dataset, MAMBA discovered 90 TTPs among 10 a resource provides, that is, whether it is a common or rare
tactics.) Whereas 56 TTPs are listed in the APT29 Evalua- resource across API calls. A large group attention value for
tions, Fig. 5 shows that 20 TTPs are recognized by MAMBA a resource indicates that the resource is frequently used
against those results from the security vendors [48]. In Fig. 5, among API calls or processes; in contrast, a resource with a
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
HUANG ET AL.: OPEN SOURCE INTELLIGENCE FOR MALICIOUS BEHAVIOR DISCOVERY AND INTERPRETATION 785

small group value means that it is uniquely representative,

or is used only by chance. Security analysts use this to select
observable resources by setting a threshold thd for the corre-
sponding similarity scores and k as the number of highest
and lowest attention values. Once the resources are selected,
malicious behavior can be located via the API calls whose
resource attention values are larger than the largest atten-
tion value minus a times the standard deviation. Algo-
rithm 2 describes the locating process for the alignment of
API calls and resources.

Algorithm 2. MAMBA API Call Locating

Input: an execution trace x , a set of group attentions v , a set of
resource attentions s , a set of predicted TTPs y^ from the
MAMBA neural network, knowledge pairs of {resource r, TTP
y} from Section 3.2
Output: a set of selected manipulated resources ry^ and the cor-
responding API call subsequences xy^j
1: for each TTP y^ do
2: # Select possible resources ry^ for a certain TTP y^
3: i Extract resources r from knowledge pairs fr; yg given
TTP y^
4: for each resource i do
5: for each manipulated resource j in x do
6: scoreði; jÞ ¼ simðei ; ej Þ
7: end for
8: end for
9: ry^ Extract j when scoreði; jÞ > thdy^
10: ry^ Extract top and bottom k of sort(vv)
11: # Locate API call xj for a certain resource j. Fig. 6. Group attention and resource attention diagram in JCry analysis.
12: for each resource j in ry^ do
13: for each resource attention s in sj do
attention in Fig. 6, such as APIs “RegEnumKeyW” and
14: xy^j Extract x when s maxðssj Þ  astdðssj Þ
“RegOpenKeyExW” that enumerate and attempt to open
15: end for
subkeys, support the discovery of the TTP. The behavior
16: end for
17: end for
meets the description of T1082; “RegEnumKeyW” and
“RegOpenKeyExW” are the associated API calls.
The algorithm then finds 3572_Enc.exe, with which we
find the TTP T1070.004 Indicator Removal on Host: File Dele-
4.6.2 Case Study tion discovered by MAMBA but not documented on the
As there is no benchmark for a quantitative evaluation of the MITRE website [28]. The group attention (3572_Enc.exe)
efficacy of associated API call locating, we here present a case and its highest resource attentions “NtDeleteFile” together
study on a JCry malware sample to demonstrate MAMBA’s support the discovery of TTP T1070.004. This malicious
ability to align API calls. The malicious activities of JCry were behavior can be observed in the execution trace as well: it
presented as a motivating example in Section 2.1. The mal- deletes the self-created files to evade detection.
ware sample manipulates 8,440 resource groups in seven Fig. 6 depicts the discovery of TTPs T1547.001 and
processes. Based on the MITRE website [28], JCry is labeled T1059.001, which are listed on the MITRE website. The group
with seven TTPs: T1547.001, T1059.001, T1059.003, T1059.005, attention of 2932_Enc.exe is high and the resource attent-
T1486, T1490, and T1204.002. Nine techniques are predi- ions of associated API calls such as ”NtCreateFile” and
cted by MAMBA, among which T1547.001, T1059.001, and ”GetFileAttributesExW” are also high, suggesting T1547.001
T1059.003 are consistent with the content on the MITRE web- Boot or Logon Autostart Execution: Registry Run Keys/Startup
site; T1033, T1070.004, T1082, T1016, T1218.010, and T1220 Folder. The next group attention for the command line
are not listed. 3420_PS and its resource attentions ”NtCreateSection” and
Fig. 6 shows the sorted group attentions and their associ- ”CreateProcessInternalW” contribute to the identification of
ated resource attentions of selected resources found by TTP T1059.001 Command and Scripting Interpreter: PowerShell.
Algorithm 2. The highest group attention refers to subkey However, MAMBA incorrectly recognizes TTPs T1033
2392_regkey1 “HKEY_LOCAL_MACHINE\SOFTWARE System Owner/User Discovery and T1218.010 Signed Binary
\Classes”, which is heavily manipulated (443 times). Its high Proxy Execution:Regsvr32, as their behaviors are not found in
group attention scores and high associated resource atten- the Cuckoo Sandbox execution trace. TTP T1220 XSL Script
tions lead to the discovery of TTP T1082 System Information Processing is mistakenly recognized when JCry renames and
Discovery. The 2392_regkey1 subkey and its many high encrypts XML files. Moreover, T1204.002 User Execution:
resource attentions, depicted as the first row of resource Malicious File is not recognized because it involves human
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
786 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 19, NO. 2, MARCH/APRIL 2022

TABLE 5 et al. [62] develop rcATT to identify tactics and techniques

Discovered Life Cycle of JCry for textual CTI reports based on ATT&CK content. Al-Shaer
et al. [63] analyze and cluster the APT and software data
reported by ATT&CK to predict techniques. These works
concern only the natural-language contents of the MITRE or
CTI reports, and can be seen as a foundation that facilitates
the use of ATT&CK knowledge in MAMBA.
TTP Identification. Holmes [12] and RapSheet [13], the most
(*) indicates the TTP is not listed in [28]. closely related works, attempt to connect adversarial behav-
ior to ATT&CK techniques. Both leverage provenance
action. Finally, MAMBA does not recognize TTPs T1059.005, graphs and rules to map audit logs to advanced persistent
T1486, and T1490. threat (APT) stages and TTPs on a host. These works differ
Following the MITRE ATT&CK framework, Table 5 from the proposed method in three ways. First, the scope of
presents the life cycle of the associated TTPs of the JCry MAMBA is different from the two studies, as they focus on
analysis, indicating correspondences between the discov- either APT or endpoint detection and not malware behavior.
ered TTPs and TTPs listed in [28]. Also, the focus of this work is to develop a neural network
To answer Q5, the group and resource attention mecha- and integrate ATT&CK knowledge to discover TTPs, which
nisms indeed capture the relations among the predicted differs greatly from their pattern matching approaches.
TTPs, the manipulated resources, and the corresponding Finally, as ATT&CK evolves, MAMBA is easily adapted to
API calls; some mistakes are made because they are not new versions of the framework, whereas their approaches
found in the execution trace, some require human interac- require human involvement for pattern development.
tion, and some are not explainable. Embedding Techniques. Rather than manually developing
solutions for security-related events, the use of embedding
techniques has brought significant benefits to the field of
5 RELATED WORK cybersecurity. Mimura et al. [19] embed proxy server logs to
Open Source Intelligence and Cyber Threat Intelligence. detect unknown malicious communication. ATTCK2vec [20]
Open Source Intelligence (OSINT) involves information learns attack embeddings which correlate a collection of bil-
gathering, collection, processing, and correlation from open lions of security events with common vulnerabilities and
data sources such as technical reports, blogs, forums, or exposures (CVEs). PROVDETECTOR [17] embeds malware
social networks [52]. In recent years, OSINT and CTI reports execution paths in the provenance graphs to detect stealthy
have been utilized to provide threat information to proac- malware. SDAC [18] embeds API call paths from each
tively mitigate potential attacks [53]. Whereas a number of Android app process to detect malware. Similar to [19] and
studies have discussed the mechanism and standards of [17], MAMBA uses a PV-DM model to embed resource val-
threat intelligence extraction [7], [8], [10], [11] and shar- ues from both ATT&CK and execution traces. With advan-
ing [54], [55], another line of work investigates the integra- ces in embedding techniques, we preserve the closeness in
tion of threat intelligence for detecting network threats [56] the embedding domain and use it to bind components and
and advanced persistent threats [57]. In this study, MAMBA identify threats.
integrates MITRE ATT&CK into a neural network model to
leverage the ever-increasing OSINT and CTI reports for
malware behavior analysis. 6 CONCLUSION
Malware Behavior Discovery. Behavior-based malware anal- In MAMBA, the proposed system, the key drivers to discov-
ysis detection learns behavior first and detects malware ering MITRE techniques include 1) incorporating knowl-
later [27], [58], while other studies focus directly on extracting edge from the MITRE ATT&CK framework, 2) considering
malicious behaviors or common behaviors of a family. Chris- the relation between resources and API calls, and 3) leverag-
todorescu et al. [59], Fredrikson et al. [60], and Palahan et al. ing resource dependencies among processes. Based on these
[61] mine malicious behavior by comparing dependence drivers, the design of the MAMBA neural network includes
graphs extracted from the execution behavior of malware 1) binding embeddings, 2) resource attention, and 3) group
against that of benign software, and produce sub-graphs spe- attention. These ensure that MAMBA achieves the best per-
cific to malicious behavior. Bayer et al. [26] develop clustering formance on both ATT&CK and Big datasets. In addition,
algorithms to discover behaviors that are characteristic of a this study demonstrates a usage of the MITRE ATT&CK
family in a group of malware. Rieck et al. [25] learn malware framework in cybersecurity applications in general that
behavior using support vector machines, in which API calls increases the interpretability of the deep learning outcomes.
with high corresponding weights represent malware behav- The information collected from ATT&CK has limitations,
ior. Similar to these works, MAMBA discovers malware as the data collection process of the MITRE ATT&CK frame-
behavior. However, whereas their discovered behavior must work relies heavily on contributions from security experts
be further interpreted by security analysts, MAMBA presents and organizations; as a result the data may be neither timely
them as mappings to MITRE ATT&CK TTPs. nor complete. This limits the capability of cybersecurity sys-
MITRE ATT&CK Techniques.Several recent studies lever- tems that rely solely on the MITRE ATT&CK framework as a
age ATT&CK as a rich source of knowledge. TTPDrill [7] knowledge source. In this case, performance can be improved
constructs an ontology of attack patterns that maps threat if the system adopts more OSINTs and other reliable sources.
actions to TTPs based on a collection of CTI reports. Legoy In this work, we focus only on Windows malware and its
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
HUANG ET AL.: OPEN SOURCE INTELLIGENCE FOR MALICIOUS BEHAVIOR DISCOVERY AND INTERPRETATION 787

associated TTPs, but the concept of our approach is not lim- [3] R. Azevedo, I. Medeiros, and A. Bessani, “PURE: Generating qual-
ity threat intelligence by clustering and correlating OSINT,” in
ited to certain operating systems since malicious behaviors Proc. 18th IEEE Int. Conf. Trust Secur. Privacy Comput. Commun/
could be discovered by aligning the manipulated resources 13th IEEE Int. Conf. Big Data Sci. Eng., 2019, pp. 483–490.
to ATT&CK knowledge. [4] Lockheed Martin Corporation, “Gaining the advantage: Applying
When an adversary with knowledge of MAMBA regular cyber kill chain methodology to network defense,” 2015.
[Online]. Available: https://www.lockheedmartin.com/content/
expressions creates new malware variants to avoid TTP dis-
dam/lockheed-martin/rms/documents/cyber/Gaining_the_Ad-
covery, there are two ways that MAMBA could still recog- vantage_Cyber_Kill_Chain.pdf
nize the technique. First, besides extracting knowledge [5] B. E. Strom, A. Applebaum, D. P. Miller, K. C. Nickels, A. G. Pen-
directly from ATT&CK, MAMBA can learn malicious nington, and C. B. Thomas, “MITRE ATT&CK: Design and phil-
osophy,” 2018. [Online]. Available: https://www.mitre.org/sites/
behaviors from malware execution traces. For example, the
default/files/publications/pr-18-0944-11-mitre-attack-design-
TTP T1070.004 File Deletion contains ten resources, including and-philosophy.pdf
the command “rm -rf”, on ATT&CK. Malware sample [6] A. Mandiant, “Exposing one of China’s cyber espionage uni-
MD5 cd1c95aa6f45101735d444aeb447225c does not use ts,” 2013. Accessed: Aug. 24, 2020. [Online]. Available: http://
intelreport.mandiant.com/Mandiant_APT1_Report.pdf
any extracted resource appearing on ATT&CK such as “rm [7] G. Husari, E. Al-Shaer, M. Ahmed, B. Chu, and X. Niu,
-rf”, but implements the technique with an API call, “TTPDRILL: Automatic and accurate extraction of threat actions
“DeleteFileW”. In this case MAMBA still identifies the TTP from unstructured text of CTI sources,” in Proc. 33rd Annu. Com-
T1070.004 File Deletion by learning the association between put. Secur. Appl. Conf., 2017, pp. 103–115.
[8] G. Husari, X. Niu, B. Chu, and E. Al-Shaer , “Using entropy and
“rm -rf” and “DeleteFileW” from data. This shows MAMBA
mutual information to extract threat actions from cyber threat
can learn the relationship between TTPs and API calls from intelligence,” in Proc. IEEE Int. Conf. Intell. Secur. Inform., 2018,
execution traces. Second, we leverage the fact that open- pp. 1–6.
source intelligence reports are constantly updated. When an [9] S. Zhou, Z. Long, L. Tan, and H. Guo, “Automatic identification of
indicators of compromise using neural-based sequence labelling,”
additional resource is used to implement a technique,
in Proc. 32nd Pacific Asia Conf. Lang. Inf. Comput., 2018.
OSINT knowledge (e.g., ATT&CK) may be updated accord- [10] Z. Zhu and T. Dumitras, “ChainSmith: Automatically learning
ingly. For example, resource powershell.exe -windowstyle hid- the semantics of malicious campaigns by mining threat intelli-
den -exec bypass -file ”%appdata%\onedrive.ps1” used to gence reports,” in Proc. IEEE Eur. Symp. Secur. Privacy, 2018,
implement TTP T1059.001 Command and Scripting Interpreter: pp. 458–472.
[11] Y. Gao, X. LI, H. PENG, B. Fang, and P. Yu, “HinCTI: A Cyber
PowerShell is not available in version 61 but is in version 72. threat intelligence modeling and identification system based on
Given the latest version of ATT&CK, MAMBA can keep up heterogeneous information network,” IEEE Trans. Knowl. Data
with the behavior of JCry. To summarize, MAMBA can still Eng., 2020, doi: 10.1109/TKDE.2020.2987019.
[12] S. M. Milajerdi, R. Gjomemo, B. Eshete, R. Sekar, and V. N. Venka-
discover malicious behavior of new malware variants by takrishnan, “HOLMES: Real-time APT detection through correla-
not only learning from data but also from the up-to-date tion of suspicious information flows,” in Proc. IEEE Symp. Secur.
CTI knowledge. Detecting adversarial examples in malware Privacy, 2019, pp. 1137–1152.
analysis, such as [64] and [65], remains an active research [13] W. U. Hassan, A. Bates, and D. Marino, “Tactical provenance
analysis for endpoint detection and response systems,” in Proc.
area in which MAMBA has room to improve. IEEE Symp. Secur. Privacy, 2020, pp. 1172–1189.
This is the first attempt to leverage the knowledge col- [14] Cuckoo Sandbox. [Online]. Available: https://cuckoosandbox.org/
lected from OSINTs in deep learning analysis of malicious [15] The Sandbox. [Online]. Available: https://cwsandbox.org/
[16] S.-W. Hsiao, Y. S. Sun, and M. C. Chen, “Hardware-assisted MMU
behavior. Still, MAMBA has several drawbacks. For instance, redirection for in-guest monitoring and API profiling,” IEEE
some discovered TTPs cannot be explained and some TTPs Trans. Inf. Forensics Security, vol. 15, pp. 2402–2416, Jan. 2020.
cannot be found. One reason for these limitations is insuffi- [17] Q. Wang et al., “You are what you do: Hunting stealthy malware
cient or imbalanced annotated resources on the MITRE web- via data provenance analysis,” in Proc. Symp. Netw. Distrib. Syst.
Secur., 2020, doi: 10.14722/ndss.2020.24167.
site [62]. In future work, we plan to enrich the knowledge [18] J. Xu, Y. Li, R. Deng, and K. Xu, “SDAC: A slow-aging solution for
associated with TTPs via data augmentation or by adopting android malware detection using semantic distance based API
more OSINTs. On the other hand, we observed that the clustering,” IEEE Trans. Dependable Secure Comput., 2020, doi:
results on the attention distribution in our case study indi- 10.1109/TDSC.2020.3005088}.
[19] M. Mimura and H. Tanaka, “Heavy log reader: Learning the con-
cate only a human-understandable explanation and do not text of cyber attacks automatically with paragraph vector,” in
reflect the model’s actual reasoning process for the model’s Proc. Int. Conf. Inf. Syst. Secur., 2017, pp. 146–163.
outcome [66]. Another direction of future work is to improve [20] Y. Shen and G. Stringhini, “ATTACK2VEC: Leveraging temporal
word embeddings to understand the evolution of cyberattacks,”
the faithfulness of attention-based explanations for mali- in Proc. 28th USENIX Conf. Secur. Symp., 2019, pp. 905–921.
cious behavior discovery, such as in [67]. [21] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and
D. Pedreschi, “A survey of methods for explaining black box mod-
els,” ACM Comput. Surv., vol. 51, no. 5, pp. 1–42, 2018.
REFERENCES [22] K. Xu et al.“Show, attend and tell: Neural image caption genera-
[1] Executive Office of the President, “Artificial Intelligence, Automa- tion with visual attention,” in Proc. Int. Conf. Mach. Learn., 2015,
tion, and the Economy,” 2016. [Online]. Available: https://oba- pp. 2048–2057.
mawhitehouse.archives.gov/sites/whitehouse.gov/files/docu- [23] J. Li, W. Monroe, and D. Jurafsky, “Understanding neural net-
ments/Artificial-Intelligence-Automation-Economy.PDF works through representation erasure,” 2017, arXiv:1612.08220.
[2] N. Kaloudi and J. Li, “The AI-based cyber threat landscape: A [24] E. Choi, M. T. Bahadori, J. A. Kulas, A. Schuetz, W. F. Stewart, and
survey,” ACM Comput. Surv., vol. 53, no. 1, pp. 1–34, 2020. J. Sun, “RETAIN: An interpretable predictive model for healthcare
using reverse time attention mechanism,” in Proc. 30th Int. Conf.
Neural Inf. Process. Syst., 2016, pp. 3512–3520.
[25] K. Rieck, T. Holz, C. Willems, P. D€ ussel, and P. Laskov,
“Learning and classification of malware behavior,” in Proc. Int.
1. https://attack.mitre.org/versions/v6/techniques/T1086/ Conf. Detection Intrusions Malware Vulnerability Assessment, 2008,
2. https://attack.mitre.org/versions/v7/techniques/T1059/001/ pp. 108–125.
Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
788 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 19, NO. 2, MARCH/APRIL 2022

[26] U. Bayer, P. M. Comparetti, C. Hlauschek, C. Kruegel, and E. Kirda, [53] W. Tounsi and H. Rais, “A survey on technical threat intelligence
“Scalable, behavior-based malware clustering,” in Proc. Netw. Dis- in the age of sophisticated cyber attacks,” Comput. Secur., vol. 72,
trib. Syst. Secur. Symp., 2009, pp. 8–11. no. C, pp. 212–233, 2018.
[27] T. W€ uchner, A. Cis»ak, M. Ochoa, and A. Pretschner, “Leveraging [54] S. Qamar, Z. Anwar, M. A. Rahman, E. Al-Shaer, and B.-T. Chu,
compression-based graph mining for behavior-based malware “Data-driven analytics for cyber-threat intelligence and informa-
detection,” IEEE Trans. Dependable Secure Comput., vol. 16, no. 1, tion sharing,” Comput. Secur., vol. 100, no. 67, pp. 35–58, 2017.
pp. 99–112, Jan./Feb. 2017. [55] T. D. Wagner, K. Mahbub, E. Palomar, and A. E. Abdallah, “Cyber
[28] JCry. [Online]. Available: https://attack.mitre.org/software/S0389/ threat intelligence sharing: Survey and research directions,” Com-
[29] S. L. Lee, “CB TAU threat intelligence notification: JCry ransom- put. Secur., vol. 87, 2019, Art. no. 101589.
ware pretends to be adobe flash player update installer.” [Online]. [56] I. Vacas, I. Medeiros, and N. Neves, “Detecting network threats
Available: https://www.carbonblack.com/blog/cb-tau-threat- using OSINT knowledge-based IDS,” in Proc. 14th Eur. Dependable
intelligence-notification-jcry-ransomware-pretends- to-be-adobe- Comput. Conf., 2018, pp. 128–135.
flash-player-update-installer/ [57] Y. Li, W. Dai, J. Bai, X. Gan, J. Wang, and X. Wang, “An intelli-
[30] K. Oosthoek and C. Doerr, “SoK: ATT&CK techniques and trends gence-driven security-aware defense mechanism for advanced
in windows malware,” in Proc. Int. Conf. Secur. Privacy Commun. persistent threats,” IEEE Trans. Inf. Forensics Security, vol. 14, no. 3,
Syst., 2019, pp. 406–425. pp. 646–661, Mar. 2019.
[31] A. J. Viera et al., “Understanding interobserver agreement: The [58] A. Saracino, D. Sgandurra, G. Dini, and F. Martinelli, “MADAM:
kappa statistic,” Family Med., vol. 37, no. 5, pp. 360–363, 2005. Effective and efficient behavior-based android malware detection
[32] B. S. Everitt, The Analysis of Contingency Tables. London, U.K./Boca and prevention,” IEEE Trans. Dependable Secure Comput., vol. 15,
Raton, FL, USA: Chapman and Hall/CRC, 2019. no. 1, pp. 83–97, Jan./Feb. 2018.
[33] Q. Le and T. Mikolov, “Distributed representations of sentences and [59] M. Christodorescu, S. Jha, and C. Kruegel, “Mining specifications
documents,” in Proc. Int. Conf. Mach. Learn., 2014, pp. 1188–1196. of malicious behavior,” in Proc. 6th Joint Meeting Eur. Softw. Eng.
[34] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evalua- Conf. ACM Symp. Found. Softw. Eng., 2007, pp. 5–14.
tion of gated recurrent neural networks on sequence modeling,” [60] M. Fredrikson, S. Jha, M. Christodorescu, R. Sailer, and X. Yan,
in Proc. NIPS Workshop Deep Learn., 2014. “Synthesizing near-optimal malware specifications from suspicious
[35] B. Athiwaratkun and J. W. Stokes, “Malware classification with behaviors,” in Proc. IEEE Symp. Secur. Privacy, 2010, pp. 45–60.
LSTM and GRU language models and a character-level CNN,” in [61] S. Palahan, D. Babic, S. Chaudhuri, and D. Kifer, “Extraction of
Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 2017, pp. 2482–2486. statistically significant malware behaviors,” in Proc. 29th Annu.
[36] H. Zhou, X. Yang, H. Pan, and W. Guo, “An android malware Comput. Secur. Appl. Conf., 2013, pp. 69–78.
detection approach based on SIMGRU,” IEEE Access, vol. 8, [62] V. Legoy, M. Caselli, C. Seifert, and A. Peter, “Automated
pp. 148404–148410, 2020. retrieval of ATT&CK tactics and techniques for cyber threat
[37] MalShare. [Online]. Available: https://malshare.com/ reports,” 2020, arXiv:2004.14322.
[38] VirusTotal. [Online]. Available: https://www.virustotal.com/ [63] R. Al-Shaer, J. M. Spring, and E. Christou, “Learning the associations
[39] GenSim. Doc2vec paragraph embeddings. [Online]. Available: of MITRE ATT&CK adversarial techniques,” in Proc. IEEE Conf. Com-
https://radimrehurek.com/gensim/models/doc2vec.html mun. Netw. Sec., 2020, pp. 1–9, doi: 10.1109/CNS48642.2020.9162207.
[40] F. Pedregosa et al.“Scikit-learn: Machine learning in Python,” J. [64] A. Al-Dujaili, A. Huang, E. Hemberg, and U.-M. OReilly ,
Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011. “Adversarial deep learning for robust detection of binary encoded
[41] G. E. Dahl, J. W. Stokes, L. Deng, and D. Yu, “Large-scale mal- malware,” in Proc. IEEE Secur. Privacy Workshops, 2018, pp. 76–82.
ware classification using random projections and neural [65] H. Li, S. Zhou, W. Yuan, X. Luo, C. Gao, and S. Chen, “Robust
networks,” in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., android malware detection against adversarial example attacks,”
2013, pp. 3422–3426. in Proc. Web Conf., 2021, pp. 3603–3612.
[42] R. Pascanu, J. W. Stokes, H. Sanossian, M. Marinescu, and A. [66] S. Wiegreffe and Y. Pinter, “Attention is not not explanation,” in
Thomas, “Malware classification with recurrent networks,” in Proc. Proc. Conf. Empir. Methods Natural Lang. Process. 9th Int. Joint Conf.
IEEE Int. Conf. Acoust. Speech Signal Process., 2015, pp. 1916–1920. Natural Lang. Process., 2019, pp. 11–20.
[43] K. Sethi, R. Kumar, L. Sethi, P. Bera, and P. K. Patra, “A novel [67] G. Chrysostomou and N. Aletras, “Improving the faithfulness of
machine learning based malware detection and classification attention-based explanations with task-specific information for
framework,” in Proc. Int. Conf. Cyber Secur. Protection Digit. Serv- text classification,” in Proc. 59th Ann. Meeting Assoc. Comput. Lin-
ices, 2019, pp. 1–4. guistics 11th Int. Joint Conf. Natural Language Process. (Volume 1:
[44] I. Jolliffe, “Principal component analysis,” Technometrics, vol. 45, Long Papers, 2021, pp. 477–488. [Online]. Available: https://aclan-
no. 3, 2003, Art. no. 276. thology.org/2021.acl-long.40, doi: 10.18653/v1/2021.acl-long.40.
[45] H. B. Mann and D. R. Whitney, “On a test of whether one of two
random variables is stochastically larger than the other,” Ann.
Math. Statist., vol. 18, no. 1, pp. 50–60, 1947. Yi-Ting Huang received the PhD degree in
[46] K. Krippendorff, “Reliability in content analysis: Some common information management from National Taiwan
misconceptions and recommendations,” Hum. Commun. Res., University, Taipei, Taiwan, in 2015, and is now a
vol. 30, no. 3, pp. 411–433, 2004.
postdoctoral fellow with the Institute of Informa-
[47] APT29 Group. [Online]. Available: https://attack.mitre.org/ tion Science, Academia Sinica. Her research
groups/G0016/ interests include malware analysis, deep learn-
[48] APT29 Emulation - Enterprise Evaluation, 2019. [Online]. Available: ing, and natural language processing in educa-
https://attackevals.mitre-engenuity.org/APT29/ tional applications.
[49] F-Secure Labs., “The Dukes: 7 years of Russian cyberespionage,”
Retrieved Aug., 2015. [Online]. Available: https://blog-assets.f-
secure.com/wp-content/uploads/2020/03/18122307/F-Secure_-
Dukes_Whitepaper.pdf
[50] S. Adair, “PowerDuke: widespread post-election spear phishing
campaigns targeting think tanks and NGOs.” 2016. [Online]. Chi Yu Lin is currently working toward the mas-
Available: https://www.volexity.com/blog/2016/11/09/ ter’s degree in the Data Science Degree Program,
powerduke-post-election-spear-phishing-campaigns-targeting- National Taiwan University, Taipei, Taiwan, and
think-tanks-and-ngos/ Academia Sinica, Taipei, Taiwan, devoted to res-
[51] M. Faou, M. Tartare, and T. Dupuy, “ESET research White earch methods in data mining, deep learning, and
papers: Operation ghost. The Dukes aren’t back – they never cybersecurity.
left.” 2019. [Online]. Available: https://www.welivesecurity.com/
wp-content/uploads/2019/10/ESET_Operation_Ghost_Dukes.pdf
[52] J. Pastor-Galindo, P. Nespoli, F. G. Marmol, and G. M. Perez, “The
not yet exploited goldmine of OSINT: Opportunities, open chal-
lenges and future trends,” IEEE Access, vol. 8, pp. 10282–10304, 2020.

Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.
HUANG ET AL.: OPEN SOURCE INTELLIGENCE FOR MALICIOUS BEHAVIOR DISCOVERY AND INTERPRETATION 789

Ying-Ren Guo is currently working toward the Yeali S. Sun received the BS degree from the
master’s degree in the Department of Electrical Computer Science and Information Engineering
Engineering: Master Program of Cybersecurity, Department, National Taiwan University, Taipei,
National Taiwan University, Taipei, Taiwan. His Taiwan, and the MS and PhD degrees in com-
research interests include malware reverse engi- puter science from the University of California,
neering and analysis, and cybersecurity. Los Angeles (UCLA), Los Angeles, California, in
1984 and 1988, respectively. From 1988 to 1993,
she was with Bell Communications Research Inc.
In August 1993, she jointed National Taiwan Uni-
versity and is currently a professor with the
Department of Information Management. Her
research interests include Internet security and forensics, quality of ser-
Kai-Chieh Lo received the graduate (MS pro- vice (QoS), cloud computing and services, and performance modeling
gram) degree in computer science from the and evaluation.
National Tsing Hua University, Hsinchu, Taiwan,
in 2019. He is currently a research assistant with
Academia Sinica. His research interests include Meng Chang Chen received the PhD degree
image processing and information security with in computer science from the University of
deep learning. California, Los Angeles, California, in 1989. He
was with AT&T Bell Labs and currently he is a
research fellow/professor with the Institute of
Information Science, Academia Sinica, Taiwan.
His research interests include computer and net-
work security, wireless network, deep learning for
complicated applications, data and knowledge
engineering.

" For more information on this or any other computing topic,

please visit our Digital Library at www.computer.org/csdl.

Authorized licensed use limited to: Dalian University of Technology. Downloaded on October 28,2023 at 15:39:05 UTC from IEEE Xplore. Restrictions apply.