Predicting malware threat intelligence using KGs

Nidhi Rastogi, Sharmishtha Dutta, Ryan Charu Aggarwal

Christian, Jared Gridley, Alex Gittens, IBM TJ Watson
Mohammaed Zaki, Charu Aggarwal New York, USA
Rensselaer Polytechnic Institute
Troy, New York, USA

ABSTRACT 1 INTRODUCTION
Large amounts of threat intelligence information about malware Malware threat intelligence informs security analysts and researchers
arXiv:2102.05571v3 [cs.CR] 24 May 2021

attacks are available in disparate, typically unstructured, formats. about trending attacks and defense mechanisms. Rooted in data,
Knowledge graphs can capture this information and its context threat intelligence can help analysts’ analyze zero-day attacks,
using RDF triples represented by entities and relations. Sparse or newly identified vulnerabilities, threats, and attack patterns to pro-
inaccurate threat information, however, leads to challenges such tect intellectual property and prevent intrusions from adversaries.
as incomplete or erroneous triples. Generic information extraction Extracting relevant information from heterogeneous sources re-
(IE) models used to populate the knowledge graph cannot fully quires correlation, disambiguation, and structuring before it can
guarantee domain-specific context. This paper proposes a system become actionable. Due to the sheer volume and inevitable noise in
to generate a Malware Knowledge Graph called MalKG, the first threat feeds, a central goal for security threat researchers is to over-
open-source automated knowledge graph for malware threat intel- come this issue and automate threat detection and prevention with
ligence. MalKG dataset (MT40K1 ) contains approximately 40,000 minimal expert involvement. AI-enabled technology, together with
triples generated from 27,354 unique entities and 34 relations. For big data about threats, has the potential to deliver on this to a great
ground truth, we manually curate a knowledge graph called MT3K, extent. Increasing sources of threat information on the Internet,
with 3,027 triples generated from 5,741 unique entities and 22 rela- such as analysis reports, blogs, common vulnerabilities, and expo-
tions. We demonstrate the intelligence prediction of MalKG using sure databases, provide enough data to evaluate the current threat
two use cases. Predicting malware threat information using bench- landscape and prepare our defenses against future ones. Several
mark model achieves 80.4 for the hits@10 metric (predicts the top research institutions, security organizations, government agencies,
10 options for an information class), and 0.75 for the MRR (mean and experts publish this as threat reports, largely in the form of
reciprocal rank). We also propose an automated, contextual frame- unstructured text. MITRE, NIST, OASIS Open have spearheaded
work for information extraction, both manually and automatically, initiatives that make security information available in structured
at the sentence level from 1,100 malware threat reports and from formats through standards like common vulnerabilities and expo-
the common vulnerabilities and exposures (CVE) database. sures (CVE) [43] and national vulnerability database (NVD) [32],
and structured threat information eXpression (STIX) [6]. Struc-
CCS CONCEPTS turing and distributing the aforementioned unstructured threat
• Computer systems organization → Embedded systems; Re- reports is critical to disseminating threat intelligence; however, this
dundancy; Robotics; • Networks → Network reliability. approach solves only one aspect of the problem. We need to address
the following challenges as well:
KEYWORDS (1) Whereas the current taxonomy and standards provide ex-
Knowledge Graphs, Security Intelligence, Contextualization, Mal- tensive coverage of threat concepts, they miss the semantic
ware. and contextual associations from the captured data. The ex-
tracted entities and relations ignore provenance that allows
ACM Reference Format:
referencing to their source of publication.
Nidhi Rastogi, Sharmishtha Dutta, Ryan Christian, Jared Gridley, Alex Git-
tens, Mohammaed Zaki, Charu Aggarwal and Charu Aggarwal. 2021. Pre-
(2) Limitations in data extraction models for domain-specific
dicting malware threat intelligence using KGs. In Seoul’21: ACM Conference threat reports can lead to capturing fragmentary and some-
on Computer and Communications Security (CCS), November 14–19, 2021. times fallacious threat information.
ACM, New York, NY, USA, 13 pages. (3) Referencing and linking of data sets can address the concerns
around the reliability of source information. Consequently,
1 Anonymous GitHub link: https://github.com/malkg-researcher/MalKG security analysts are required to correlate, disambiguate, and
structure heterogeneous formats and quality of data before
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
it can become actionable.
for profit or commercial advantage and that copies bear this notice and the full citation In this paper, we address the first two challenges by proposing
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, MalKG, a knowledge graph for malware threat intelligence, and
to post on servers or to redistribute to lists, requires prior specific permission and/or a defer the third challenge for future research. MalKG extends mal-
fee. Request permissions from [email protected].
ware ontologies [33, 39] by capturing threat information in the
Seoul’21, November 14–19, 2021, Seoul, South Korea
© 2021 Association for Computing Machinery. form of RDF triples using entities and relations. Entities are inter-
linked descriptions of pre-defined ontology classes such as malware,
Seoul’21, November 14–19, 2021, Seoul, South Korea Anonymous

organizations, vulnerability, attack patterns, malware behavioral

characteristics, and timeline of the attack. Relations connect en-
tities and provide context for their interpretation that describes
the relationship between entities. MalKG encompasses the best of
both the worlds, databases (data exploration via structured queries)
and graphs (analyze like a network data structure), as well as bears
formal semantics used to interpret data and infer new facts. Specif-
ically, the main contributions of this paper are:
(1) We present MalKG, the first end-to-end, automated, ontology-
defined knowledge graph generation framework for malware Figure 1: MalKG has captured two indicators of Malware-1.
threat intelligence. MalKG is open source and comprises When MalKG is queried for attribution on Malware-2, it can
40,000 RDF triples2 extracted from 1,100 unstructured mal- find the associations between the two malware.
ware threat reports and CVE vulnerability descriptions for
almost one thousand malware seen since 1999 (see Section 6
for details).
(2) We predict malware threat intelligence through two use such as information about the malware author, associated cam-
cases to demonstrate to power of our knowledge graph. paign, and many other attributes that can reveal the malicious
For example, MalKG predicts previously unknown threat program’s objectives or be able to recreate it. Analyzing malware
information such as vulnerabilities, associations between and attributing it is very challenging due to the wide application of
malware, malware author. This is because the triples are code obfuscation, polymorphism, and deformation changes in the
encoded into vector embeddings that can extract latent in- structure of malicious code. The goal of MalKG is to automate the
formation from graphs and reveal latent malware informa- prediction of these features associated with a given malware (see
tion unknown during training the prediction model. This Figure 1. For instance, for a newly discovered malware with sparse
is a crucial goal for cybersecurity analysts because running information available in the published and publicly available threat
machine learning models on vector embeddings can produce reports, MalKG can predict the attack campaigns responsible for
more accurate and contextual predictions for autonomous the new malware along with a confidence score of the prediction.
intrusion detection systems. Threat reports3 about the recent attacks on the SolarWinds software
(3) We present a novel, challenging corpus of extracted 27,354 captured detailed analysis on the Sunspot malware that inserted
unique named entities and 34 relations. This is a new bench- the Sunburst malware into SolarWinds’ Orion software. However,
mark framework for novel contextual security research prob- in the reports published before January 2021, no information was
lems. available about the threat actor or the campaign behind Sunburst.
Previous work in malware threat intelligence leans on syntac- MalKG can predict malware features like attack actors and other
tic or semantic similarities, API call sequences to detect malware properties that are similar to Sunburst. For instance, Sunburst and
or their variants [2, 4, 11, 21]. To the best of our knowledge, our a backdoor linked to the Turla advanced persistent threat (APT)
research is the first that utilizes semantic similarities to predict mal- group share similarities, and MalKG can predict this information.
ware features using an unstructured text corpus. We first provide
two use cases that describe the predictive abilities of MalKG in Sec- 2.2 Vulnerability Prediction
tion 2 for vulnerability and malware identification. We then present
One-fourth of the zero-day exploits detected in the year 2020 were
MalKG’s overall architecture and describe each of its component,
either variants of previously publicly disclosed vulnerabilities4 or
including the corpus comprising the RDF triples, and malware fea-
those vulnerabilities that were incompletely patched [16]. MalKG
ture prediction. We also discuss the challenges encountered in this
can predict the list of potential vulnerabilities exploited by a new
process. We then discuss our evaluation and experimental results
malware or a variant of an existing virus. For example, given a
and outline ongoing and future work.
malware impacted software system (such as Microsoft Office, Linux
Operating system, Windows NT Server), we are interested in pre-
2 USE CASES dicting all the vulnerabilities (such as spoofed route pointer, multi-
In this section, we illustrate two example scenarios to demonstrate ple eval injection vulnerabilities, heap overflow) or the associated
the capabilities of MalKG in predicting malware threat intelligence. CVE IDs that may have impacted the system. MalKG can predict an
ordered list of vulnerabilities or exploits based on (a) available data
2.1 Malware Attribution about the software system, (b) preliminary information about the
Malware attribution is the careful process of answering the 6-Ws malware, and (c) latent information captured from the knowledge
(who, how, what, where, why, when) of a given malware attack graph.
on a system under observation. It requires assembling sufficient
malware-related features to build a fingerprint of its origination,
3 https://www.cyberscoop.com/turla-solarwinds-kaspersky/
2 as of April 2021 4 https://googleprojectzero.blogspot.com/2021/02/deja-vu-lnerability.html
Predicting malware threat intelligence using KGs Seoul’21, November 14–19, 2021, Seoul, South Korea

3 CHALLENGES ⟨𝑒ℎ𝑒𝑎𝑑 , 𝑟, 𝑒𝑡𝑎𝑖𝑙 ⟩ ∈ T indicates that there is a relation r ∈ R between

In this section, we detail the major challenges encountered during 𝑒ℎ𝑒𝑎𝑑 ∈ E and 𝑒𝑡𝑎𝑖𝑙 ∈ E. Knowledge graph completion, sometimes
the process of generating MalKG for making threat intelligence also known as entity prediction, is the task of predicting the best can-
predictions: didate for a missing entity. Formally, the task of entity-prediction is
to predict the value for h given ⟨?, r, t⟩ or t given ⟨h, r, ?⟩, where
(a) Missing cybersecurity specific information extraction
“?" indicates a missing entity.
models - Threat reports contain both generic and domain-
For the entities and relation 𝑒ℎ𝑒𝑎𝑑 , 𝑒𝑡𝑎𝑖𝑙 , 𝑟 we use the bold face
specific threat data. Named entity recognizer (NER) models
h, t, r to indicate their low-dimensional embedding vectors, re-
such as StanfordNER5 trained on generic datasets or UMLS
spectively. We use 𝑑𝑒 and 𝑑𝑟 to denote the embedding dimensions
for specific domains such as health cannot be used for cy-
of entities and relations, respectively. The symbols 𝑛𝑒 , 𝑛𝑟 , and 𝑛𝑡
bersecurity threat reports. Besides, using existing domain-
denote the number of entities, relations, and triples of the KG, re-
specific concepts sometimes have titles that can mislead NER
spectively. We use the 𝑓𝜙 notation for the scoring function of each
models into categorizing them as generic concepts. For ex-
triple ⟨h, r, t⟩. The scoring function measures the plausibility of
ample, when a CVE description mentions “Yosemite backup
a fact in T, based on translational distance or semantic similarity
8.7,” a generic NER does not refer to Yosemite as a server
[24]. A summary of the notation appears in Table 1.
backup software, but as a national park in California.
(b) Missing context in extracting key phrases - Cybersecu-
rity standards and taxonomies overlook the importance of Table 1: Notations and Descriptions
context in extracted concepts, which can be favorably added
via linking and semantic metadata. Text data is not a mere Notation Description
aggregation of words. There is a meaning in the information E the set of all entities
and linking of concepts to outside sources is crucial to its R the set of all relations
effective use. For example, a human security analyst can T the set of all triples
correctly comprehend the meaning of the term “Aurora” in ⟨𝑒ℎ𝑒𝑎𝑑 , 𝑟, 𝑒𝑡𝑎𝑖𝑙 ⟩ head entity, relation and tail entity
a threat report. It refers to the series of cyber attacks con- ⟨h, r, t ⟩ embedding of a h, r and a t
ducted by Advanced Persistent Threats (APTs) by China, 𝑑 𝑒 , 𝑑𝑟 embedding dimension of entities, relations
first detected by Google in 2009. However, a computer or an 𝑛 𝑒 , 𝑛𝑟 , 𝑛 𝑡 number of h+t, r, triples
AI model collecting this information might mistake it for the 𝑓𝜙 scoring function

Northern Lights unless the security context is provided.

(c) Lack of standard format of malware threat corpus - Structure of MalKG: The MalKG comprises hierarchical enti-
Malware threat reports cover technical and social aspects ties. For example, a Malware class can have sub-classes such as
of a malware attack in the form of unstructured text. Gen- TrojanHorse and Virus. A hierarchy captures the categories of mal-
erating a knowledge graph requires extracting key phrases ware based on how they spread and infect computers. The relations
from these reports, which is not a trivial task. Unlike for a in MalKG, on the other hand, are not hierarchical. For instance, a
small curated benchmark dataset, we require an information Virus (sub-class of Malware) known as WickedRose exploits a Mi-
extraction systems that is scalable and replicable for a large crosoft PowerPoint vulnerability and installs malicious code. Here
amount of text corpus specific to malware threat domain. “exploits” is the relation between the entities Virus (a subclass) and
The varying degrees of attack and defence description and Software, and therefore does not require a sub-relation to describe
the changing landscape of the information can render prior a different category of exploit. MalKG stores facts like these in the
information defunct, which is an additional challenge. form of a triple.
(d) Missing validation techniques - Knowledge graphs for Instantiating MalKG: The malware threat intelligence knowl-
cybersecurity is a relatively new approach for providing edge graph, MalKG, is instantiated from threat reports collected
structured and semantically rich information to people and from the Internet and CVE (see Figure 2). We tokenize approxi-
machines. However, since this is an area of novel research, mately 1,100 threat reports in PDF format and from the CVE vul-
there was no publicly available “ground truth” to validate nerability description database to extract entities covering malware
MalKG and measure if the prediction from MalKG were threat intelligence. The instantiating process is described in later
semantically correct and correspond to the so-called "real" sections and includes algorithms for named entity extraction (NER)
world. and relation extraction (RE). Together, both NER and RE algorithms
are deployed and automated to generate thousands of RDF triples
4 BACKGROUND that capture malware threat information into units of a knowledge
4.1 Malware Knowledge Graph (MalKG) graph.
Definitions: The malware knowledge graph, MalKG, expresses het- MalKG Embeddings: Triples of MalKG are represented by a
erogeneous multi-modal malware threat data as a directed graph, relatively low (e.g., 30-200) dimensional vector space embeddings
MalKG = {E, R, T}, where E, R and T indicate the sets of entities, [5, 41, 44, 47, 48], without losing information about the structure
relations and triples (also called facts), respectively. Each triple and preserving key features. We construct embeddings for all the
triples in MalKG where each triple has three vectors, e1, e2 ∈ R𝑑𝑒 ,
5 https://nlp.stanford.edu/software/CRF-NER.shtml 𝑟 ∈ R𝑑𝑟 and the entities are hierarchical. Therefore, to narrow down
Seoul’21, November 14–19, 2021, Seoul, South Korea Anonymous

would predict its missing head h[44], i.e., a campaign that involved
this Hydraq malware family. MalKG can predict missing entities
from an incomplete triple, which forms the smallest malware threat
intelligence unit. Section 7.1 elaborates on this topic.

5 MALKG: SYSTEM ARCHITECTURE

The input to our proposed framework for MalKG construction, as
Figure 2: An excerpt of a threat report on Backdoor.Winnti shown in Figure 3, consists of a training corpus and a malware
malware family. threat ontology (e.g.,[33]) that has classes and relations to provide
structure to MalKG. The training corpus contains malware threat
intelligence reports gathered from the Internet, manual annotations
of the entities and relations in those reports using the ontology,
the choice of embedding models, we ensured that the embedding
and the CVE vulnerability database. The training corpus and the
model meets the following requirements:
ontology classes are fed into entity extractors such as SetExpan [34]
(1) An embedding model has to be expressive enough to handle to provide entity instances from the unstructured malware reports.
different kinds of relations (as opposed to general graphs A relation extraction model [49] uses these entity instances and
where all edges are equivalent). It should create separate em- ontology relations as input and predicts relations among the en-
beddings for entities that can be involved in the same relation. tity pairs. Entity pairs and the corresponding relations are called
To elaborate, for two triples- ⟨𝑒 1, hasVulnerability, 𝑒 3 ⟩ and triples and they form an initial knowledge graph. We use a tensor
⟨𝑒 2, hasVulnerability, 𝑒 3 ⟩, the embedding of an entity 𝑒 1 ∈ factorization-based approach, TuckER[5], to create vector embed-
class:ExploitTarget is different than that of an entity 𝑒 2 ∈ dings to predict the missing entities and obtaining the final version
class:Software. of MalKG with the completed RDF triples.
(2) An embedding model should also be able to create different
embeddings for an entity when it is involved in different
kinds of relations [45]. This requirement ensures creating
embedding of triples with 1-to-n, n-to-1, and n-to-n relations. 5.1 Generating Contextual and Structured
For example, a straightforward and expressive model TransE Information
[9] has flaws in dealing with 1-to-n, n-to-1, and n-to-n rela- This section describes the process of generating contextual and
tions [25, 45]. structured malware threat information from unstructured data.
(3) Entities that belong to the same class should be projected in Corpus - Threat reports, blogs, threat advisories, tweets, and
the same neighborhood in the embedding space [14]. This reverse engineered code can be investigated to gather details about
requirement allows using an embedding’s neighborhood malware characteristics. Our work relies on threat reports to cap-
information when predicting the class label of an unlabeled ture information on the detailed account of malware attacks and
entity (this task is called entity classification). CVE vulnerability descriptions. Security organizations publish in-
depth information on how cyber-criminals abuse various software
4.2 Threat Intelligence Prediction and hardware platforms and security tools. See Figure 2 for an
Predicting malware threat information can be modeled in MalKG excerpt of a threat report on the Backdoor.Winnti malware fam-
by defining that unknown threat information as missing from the ily. Information collected from a corpus of threat reports includes
knowledge graph. For example, consider “Adobe Flash Player” as an vulnerabilities, platforms impacted, modus operandi, attackers, the
entity belonging to the class Software in MalKG. A user is interested first sighting of the attack, and facts such as indicators of compro-
in learning all the vulnerabilities impacting “Adobe Flash Player.” mise. These reports serve as research material for other security
which are mostly unknown so far. It can be written as an incomplete analysts who acquire essential information to study and analyze
triple ⟨Adobe Flash Player, hasVulnerability, ?⟩. The knowledge malware samples, adversary techniques, learn about zero-day vul-
graph completion task would predict its missing tail th[44], i.e., a nerabilities exploited, and much more. However, not all reports are
vulnerability of the software titled “Adobe Flash Player.” MalKG consistent, accurate, or comprehensive and can vary in technical
can predict missing entities from an incomplete triple, which forms breadth of the malware attack description.
the smallest malware threat intelligence unit. Section 7.1 elaborates Contextualizing using Linked Data - Each entity in MalKG
on this topic. has a unique identifier, which captures the provenance information
In another example, a user is interested in learning information comprising references to the triples’ threat sources. For MalKG,
associated with a malware family named “Hydraq”. Some infor- this provenance information comes from two sources, a URL of the
mation on this malware is known and therefore contained in the threat reports from which it was extracted, the unique CVE IDs for
training set. However, some unknown information (e.g., campaign vulnerabilities. For general concepts, entities are linked to DBPedia
that involved this malware) may be missing from the KG. This [3].
missing information can be written as an incomplete triple ⟨?, in- Generating Triples - MalKG captures threat information in a
volvesMalware, Hydraq⟩. The knowledge graph completion task triple, which is the smallest unit of information in the knowledge
Predicting malware threat intelligence using KGs Seoul’21, November 14–19, 2021, Seoul, South Korea

Figure 3: System Architecture for MalKG construction and showing entity prediction.

graph [26]. For instance, consider the following snippet from a

threat report:6
“... DUSTMAN can be considered as a new variant of ZeroCleare
malware ... both DUSTMAN and ZeroCleare utilized a skeleton of
the modified “Turla Driver Loader (TDL)” ... The malware executable
file “dustman.exe” is not the actual wiper, however, it contains all
the needed resources and drops three other files [assistant.sys, el-
rawdisk.sys, agent.exe] upon execution...”
Table 2 shows a set of triples generated from this text. Together,
many such triples form the malware knowledge graph where the
entities and relations model nodes and directed edges, respectively.
Figure 4 shows the sample knowledge graph from the text. A knowl-
Figure 4: Sample MalKG. Nodes and edges represent entities
edge graph contains 1-to-1, 1-to-n, n-to-1, and n-to-n relations
and relations, respectively. The edge labels denote relation
among entities. In Figure 4, the entity “dustman.exe” is an instance
type.
of a Malware File. It drops three different files. Entities DUSTMAN
and ZeroCleare both *involve* the Turla Driver Loader (TDL) file,
and elaborate how n-to-1 relations exist in a knowledge graph;
[29]. For example, there are classes named Malware and Location
n-to-n works similarly.
in MALOnt. According to the relations specified in MALOnt, an
entity of Malware class can have a relation “similarTo” with another
Table 2: Triples corresponding to the threat report. Malware class entity. But due to the rule engine, Malware class can
never be related to an entity of Location class.
Head Relation Tail Creating Ground Truth - We create ground truth triples by
⟨DUSTMAN, similarTo, ZeroCleare ⟩ manually annotating threat reports. We generated MT3K, a knowl-
⟨DUSTMAN, involves, Turla Driver Loader(TDL) ⟩ edge graph for malware threat intelligence using hand annotated
⟨ZeroCleare, involves, Turla Driver Loader(TDL) ⟩
triples. While this was an essential step for creating ground truth,
⟨DUSTMAN, involves, dustman.exe ⟩
the hand-annotated approach had several limitations, but is invalu-
⟨dustman.exe, drops, assistant.sys ⟩
able for testing large-scale models. Details of creating the ground
⟨dustman.exe, drops, elrawdisk.sys ⟩
truth knowledge graph are covered in Section 6.2.
⟨dustman.exe, drops, agent.exe ⟩

5.2 Predicting Missing Threat Intelligence

Inference Engine - We extend the classes and relations de-
Knowledge graph completion enables inferring missing facts based
fined in MALOnt [33] and malware ontology [39] to extract triples
on existing triples in MalKG when information (entities and rela-
from unstructured text. While using an ontology is not the only
tions) about a malware threat is sparse. For MalKG, we perform
approach to instantiate MalKG, it can map disparate data from mul-
extensive experimental evaluation and predict entities through
tiple sources into a shared structure and maintains a logic in that
TuckER [5] and TransH [45]. Standard datasets such as FB15K [9],
structure using rules. An ontology uses a common vocabulary that
FB15k-237 [40], WN18 [9] , and WN18RR [12] were used to compare
can facilitate the collection, aggregation, and analysis of that data
with other models and create baselines for other more elaborate
6 https://www.lastwatchdog.com/wp/wp-content/uploads/Saudi-Arabia-Dustman- models. However, the datasets that were used for comparison dif-
report.pdf fered starkly form our MT3KG and MT40K triple dataset.
Seoul’21, November 14–19, 2021, Seoul, South Korea Anonymous

Defining MalKG protocol for Entity Prediction: The follow- Table 3: Description of Datasets
ing protocol describes the prediction for missing entities in the
malware knowledge graph: Dataset 𝑛𝑒 𝑛𝑟 𝑛𝑡 docs avgDeg density
MT3K 5,741 22 3,027 81 0.5273 0.00009
a The head entity h is replaced with all other entities in the
knowledge graph for each triple in the test set. This generates MT40K 27,354 34 40,000 1,100 1.46 5.34 ×
10−5
a set of candidate triples to compare with each true test triple.
This step is known as corrupting the head of a test triple, FB15K 14,951 1345 592213 - 39.61 0.00265
and the resultant triple is called a corrupted triple. WN18 40943 18 151,442 - 3.70 0.00009
b The scoring function, 𝑓𝜙 (ℎ, 𝑟, 𝑡), returns a score for each FB15K- 14,541 237 310,116 - 21.32 0.00147
true test triple and its corresponding corrupted triples. The 237
scores are sorted to obtain the rank of the true test triple in WN18RR 40,943 11 93,003 - 2.27 0.00005
the ordered set. This rank assesses the model’s performance
in predicting ℎ𝑒𝑎𝑑 given ⟨ ?, relation, tail ⟩. A higher rank
indicates the model’s efficiency in learning embeddings for • 1 security researcher trained 1 graduate and 5 undergraduate
the entities and relations. student to annotate the threat reports. The annotated triples
c The previous two steps are repeated for the tail entity t, were later manually verified by the same expert.
and true test triples ranks are stored. This rank assesses We perform several pre-processing steps on the raw corpus data.
the model’s performance in predicting the missing t given For provenance, we provide a unique identifier (URL of threat re-
⟨h, r, t⟩. port) to every individual entity and the ID of the CVE description to
d Mean Rank, Mean Reciprocal Rank (MRR), and Hits@n are vulnerability related entities. Individual sentences were extracted
calculated from the ranks of all true test triples. Mean rank from each document and assigned a sentence ID. Identification of
signifies the average rank of all true test triples. Mean rank documents and sentences also provides provenance for future work
(MR) is significantly affected when a single test triple has a in trustworthiness, contextual entity classification, and relation ex-
bad rank. Therefore, recent works [5, 22] report mean recip- traction. We tokenize and parse the plain text sentences into words
rocal rank (MRR), which is the average of the inverse of the with start and end positions using the spaCy default tokenizer8
ranks of all the true test triples, for more robust estimation. library and save them in sequential order for each document.
Hits@n denotes the ratio of true test triples in the top n po-
sitions of the ranking. When comparing embedding models 6.2 Benchmark dataset
for prediction, smaller values of MR and larger values of In absence of a benchmark dataset for malware threat intelligence
MRR and Hits@n indicate a better model. and to train and test MalKG, we manually generate MT3K triple
dataset. The MT3K dataset is a collection of 5,741 triples hand-
6 EXPERIMENTAL EVALUATION annotated by our research team and manually validated by a secu-
This section describes our experimental setup, results, and imple- rity expert. 81 threat reports were annotated (see Figure 5) using the
mentation details for knowledge graph completion. Brat annotation tool[37]. In Figure 5, “PowerPoint file” and “installs
malicious code” are labeled as classes Software and Vulnerability
respectively. The arrow from Software to Vulnerability denotes the
6.1 Datasets semantic relationship hasVulnerability between them. The annota-
In this paper, we extract relavant key phrases from malware threat tion followed rules defined in malware threat ontologies [33, 39]. 22
intelligence reports and CVE dataset. The threat reports were writ- other relation types (𝑛𝑟 ) are defined in malware threat intelligence
ten between 2006-2021 and all CVE vulnerability descriptions cre- ontology[33].
ated between 1990 to 2021. The threat reports are formal investiga-
tive reports on malware attacks. Each report starts with a table
of content followed by a detailed analysis of the attack patterns,
actors involved, systems impacted. Images, text embedded within
images, tables, and snapshots of code snippets frequently appear in
these reports. Each report often focuses on the technical analysis
and mitigation of one malware campaign. Below we detail further
information about the threat reports:
• 1,100 threat advisory reports from all the major organiza- Figure 5: Annotation using Brat annotation tool.
tions were included in this dataset. For example, Symantec,
Kaspersky, Microsoft, IBM, FireEye, TrendMicro, McAfee,
The MT3K dataset serves as the benchmark dataset for evalu-
and several smaller players. The reports are made available
ating malware threat intelligence using our automated approach
for download as well. 7
for triples generation. Other triple datasets such as the FB15K [9],
FB15k-237 [40], WN18 [9] , and WN18RR [12] have been used to
7 https://github.com/malkg-researcher/MalKG 8 https://spacy.io/api/tokenizer
Predicting malware threat intelligence using KGs Seoul’21, November 14–19, 2021, Seoul, South Korea

evaluate the performance of generic knowledge graphs and their An explanation for this phenomenon is the variation and gram-
applications, such as link-prediction, and knowledge graph com- matical inconsistency in different threat reports and their writing
pletion. However, there is no open-source knowledge graph for style. For example, the entity “linux” was classified as Software
malware threat intelligence, and therefore there do no exist triples and Organization for the same position in a document. Since the
for malware KG evaluation. We hope that MT3K and MT40K will entity extraction models are running independently of each other,
serve as benchmark datasets for future security research. it was likely for close class types to have overlaps. We followed
a three step approach for resolving the disambiguation in entity
classification:
6.3 Automated Dataset: MT40K
(1) We compared the confidence score of entities occurring at
The MT40K dataset is a collection of 40,000 triples generated from
the same position in the same document and between classes.
27,354 unique entities and 34 relations. The corpus consists of ap-
(2) For the same confidence scores, we marked those classes as
proximately 1,100 de-identified plain text threat reports written
ambiguous. We retained these instances until the relation
between 2006-2021 and all CVE vulnerability descriptions created
extraction phase. At that time, we relied on the rule engine
between 1990 to 2021. The annotated keyphrases were classified
from the ontology to determine the correct class of the head
into entities derived from semantic categories defined in malware
and tail entity for a given relation (described in the next
threat ontologies [33, 39]. The inference rule engine was applied
section).
when forming relations between entities to produce triples used to
(3) For entities that passed the first two steps, we classified them
build the MalKG.
using the broader semantic class.
Our information extraction framework comprises entity and
relation extraction and is grounded in the cybersecurity domain 6.3.3 Relation Extraction. We perform relation extraction for MalKG
requirements. The framework takes as input the plain text of threat using a distantly supervised artificial neural network model [49]
reports and raw CVE vulnerability descriptions. It locates and classi- pre-trained on the Wiki data and Wikipedia, originally with three
fies atomic elements (mentions) in the text belonging to pre-defined features. The MT3K triples and text corpus form the training dataset
categories such as Attacker, Vulnerability, and so on. We split the and the entities generated in the previous steps, and the processed
tasks of entity extraction and recognition. The entity extraction text corpus the testing dataset. As described in section 3, while
(EE) step combines an ensemble of state-of-the-art entity extrac- training and testing was not a straightforward implementation, we
tion techniques. The entity classification (EC) step addresses the generated 40,000 triples using 34 relations between 27,354 entities.
disambiguation of an entity classified into more than one semantic The instantiated entities, their classification, and position in the
category, and assigns only one category based on an empirically threat report or CVE description formed the RE model’s input. The
learned algorithm. Results obtained from both EE and EC are com- testing batch size was 16 documents. Due to scalability issues in
bined to give the final extraction results. Details are provided in DocRED, the number of entities per document was limited to less
the next section. than 80. Text documents with a higher number of entities were
further split into smaller sizes. The training ran for 4-6 hours, for
6.3.1 Entity Extraction (EE). We segregate the extraction of to- an epoch size of 984, and the threshold set to 0.5.
kenized words into two categories: domain-specific key-phrase
extraction and generic key-phrase extraction (class:Location). The 6.4 Predicting Missing Information
cybersecurity domain-specific EE task was further subdivided into Knowledge graph completion (or entity prediction) predicts miss-
factual key-phrase extraction (class:Malware) and contextual key- ing information in MalKG. It also evaluates the entity prediction
phrase extraction (class:Vulnerability). We use precision and recall accuracy of a given triple and recommends a ranking of predicted
measures as evaluation criteria to narrow down the EE models for entities. For the missing entities in the MalKG triples, a tensor
different classes. Specifically, the Flair framework [1] gave high factorization-based approach [5] can predict the missing entities.
precision scores (0.88-0.98) for classification of generic and factual
domain-specific key-phrases such as - Person, Date, Geo-Political 6.4.1 Comparing with Benchmark Dataset. Due to the lack of a
Entity, Product (Software, Hardware). The diversity in writing style benchmark dataset in the cybersecurity domain, we rely on other
for the threat reports and the noise introduced during reports con- datasets to observe and compare existing embedding models’ perfor-
version to text led to different precision scores for classes but within mances. The most commonly used benchmark datasets are FB15K,
the range of 0.9-0.99. For contextual entity extraction, we expanded WN18, FB15K-237, WN18RR. Collected from the freebase knowl-
the SetExpan [34] model to generate all entities in the text cor- edge graph, FB15K and FB15K-237 are collaborative knowledge
pus. The feedback loop that input seed values of entities based bases containing general facts such as famous people, locations,
on confidence score allowed for a better training dataset. Some of events. WN18 and WN18RR are datasets from Wordnet, a lexical
the semantic classes used in this approach are Malware, Malware database of the English language. Simpler models perform better on
Campaign, Attacker Organization, Vulnerability. these datasets where a test triple’s corresponding inverse relation
exists in the training set. Such instances are eliminated from FB15K
and WN18 to generate FB15K-237 and WNRR. Experimental results
6.3.2 Entity Classification (EC). A shortcoming of using multiple
by OpenKE9 for entity prediction (or knowledge graph completion)
EE models is that some entities can get classified multiple times,
are summarized in Table 5.
which might even be inaccurate in some cases. We observed empir-
ically that this phenomenon was not limited to any specific class. 9 https://github.com/thunlp/OpenKE
Seoul’21, November 14–19, 2021, Seoul, South Korea Anonymous

The comparison between the structure of the graphs we gen- this set of 10 possible choices, the analyst finds the accurate
erated - MalKG and hand-annotated, their properties, and corpus information at the second item. By definition, the Hits@3
details are described in Table 3. For example, it compares the num- and Hits@10 metrics increase as the true entity exist in the
ber of entities, relations, and triples (facts), the average degree of top 3 and certainly in the top 10. As seen in table 7, we
entities, and graph density of our datasets (MT3K, MT40K) and can say 75.9% times the true entity will appear in the top 3
benchmark datasets. Note, the average degree across all relations predictions of the model.
and graph density are defined as 𝑛𝑡 /𝑛𝑒 and 𝑛𝑡 /𝑛𝑒2 , respectively [30]. (2) Case study 2, Predicting a Campaign: A security analyst
A lower score of these two metrics indicates a sparser graph. wants to identify which campaign is involved with the mal-
ware family named Hydraq. The following query is posed to
6.4.2 Validation. MT3K validation set is used for tuning the hyper-
the MalKG in the form of an incomplete triple:
parameters for MT40K. For the TuckER experiment on MT3K, we
achieved the best performance with 𝑑𝑒 = 200 and 𝑑𝑟 = 30, learning ⟨?, 𝑖𝑛𝑣𝑜𝑙𝑣𝑒𝑠𝑀𝑎𝑙𝑤𝑎𝑟𝑒, 𝐻𝑦𝑑𝑟𝑎𝑞⟩
rate = 0.0005, batch size of 128 and 500 iterations. For the TransH The training set had some triples involving Hydraq such as -
experiment, 𝑑𝑒 = 𝑑𝑟 = 200, learning rate = 0.001 gave us the best
results. For MT3K and MT40K, we split each dataset into 70% train, ⟨𝑂𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛_𝐴𝑢𝑟𝑜𝑟𝑎, 𝑢𝑠𝑒𝑠𝑀𝑎𝑙𝑤𝑎𝑟𝑒, 𝐻𝑦𝑑𝑟𝑎𝑞⟩
15% validation, and 15% test data. Due to limited hand-annotated ⟨𝑂𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛_𝐴𝑢𝑟𝑜𝑟𝑎, 𝑡𝑎𝑟𝑔𝑒𝑡𝑠, 𝐺𝑜𝑜𝑔𝑙𝑒⟩
dataset, we tested the MT40K dataset using Mt3K. ⟨𝐺𝑜𝑜𝑔𝑙𝑒_𝐻𝑎𝑐𝑘_𝐴𝑡𝑡𝑎𝑐𝑘, 𝑡𝑎𝑟𝑔𝑒𝑡𝑠, 𝐺𝑜𝑜𝑔𝑙𝑒⟩
Note that the training set contains facts from different re-
7 RESULTS AND ANALYSIS ports, and the terms ‘Google Hack Attack’ and ‘Operation
7.1 Threat Predictions Aurora’ refer to the same campaign. The model learns latent
We present detailed results and analysis of two malware threat features of these entities and relations during training and
predictions that demonstrate new security findings and detect pre- later uses these features to predict a set of campaign names
viously unknown malware-related information to the MalKG. For highly probable to involve the Hydraq malware family.
this, we use queries to acquire security findings from the training Table 4 demonstrates the predictions for this query. We can
model. Each query (formed as an incomplete test triple) results in see that the true answer (Google Hack Attack) is ranked #7.
a set of predictions where each prediction has a confidence score In order to validate this finding, we refer the reader to the
associated with it. The confidence score of a predicted entity 𝑒𝑖 threat report titled “In-depth Analysis of Hydraq”11 .
denotes the probability of 𝑒𝑖 being the accurate candidate for the This result contributes to increasing only the Hits@10 met-
incomplete triple. The predictions are sorted in descending order ric (since the rank is within the top 10), and it does not
by this score, and we observe the top 10 predictions as potential contribute to Hits@1 and Hits@3. We obtain an intuition
predictions. about what the Hits@n metric on the overall test dataset
represents. Hits@10 score reported in table 7 can be inter-
preted that 80.4% time the true entity would appear in the top
(1) Case Study 1, Predicting a malware family: A security
10 predictions. We elaborate on this topic in the following
analyst wants to identify which malware family is associated
section.
with the indicator 𝑖𝑛𝑡𝑒𝑙 − 𝑢𝑝𝑑𝑎𝑡𝑒 [.]𝑐𝑜𝑚 in a MalKG. The
following query is posed to the MalKG in the form of an
incomplete triple: 7.2 Analysing MalKG predictions
⟨𝑖𝑛𝑡𝑒𝑙 − 𝑢𝑝𝑑𝑎𝑡𝑒 [.]𝑐𝑜𝑚, 𝑖𝑛𝑑𝑖𝑐𝑎𝑡𝑒𝑠, ?⟩ Table 5 displays the results of entity prediction existing embedding
Here, the accurate malware family associated with the do- models on the benchmark datasets. An upward arrow (↑) next to
main 𝑖𝑛𝑡𝑒𝑙 − 𝑢𝑝𝑑𝑎𝑡𝑒 [.]𝑐𝑜𝑚 is named Stealer. We refer the an evaluation metric indicates that a larger value is preferred for
reader to the FireEye report titled ‘Operation Saffron Rose’10 that metric and vice versa. For ease of interpretation, we present
for validating this fact. The training set had some triples Hits@n measures as a percentage value instead of a ratio between
involving Stealer, such as: 0 and 1. TransE has the largest Hits@10 and MRR scores on the
FB15K-237 dataset, whereas TransR performs better on the rest
of the three datasets. However, TransH is more expressive (capa-
⟨𝑜 𝑓 𝑓 𝑖𝑐𝑒.𝑤𝑖𝑛𝑑𝑜𝑤𝑠𝑒𝑠𝑠𝑒𝑛𝑡𝑖𝑎𝑙𝑠 [.]𝑡𝑘, 𝑖𝑛𝑑𝑖𝑐𝑎𝑡𝑒𝑠, 𝑆𝑡𝑒𝑎𝑙𝑒𝑟 ⟩ ble of modeling more relations) than TransE despite having the
⟨𝑆𝑎𝑓 𝑓 𝑟𝑜𝑛_𝑅𝑜𝑠𝑒, 𝑖𝑛𝑣𝑜𝑙𝑣𝑒𝑠𝑀𝑎𝑙𝑤𝑎𝑟𝑒, 𝑆𝑡𝑒𝑎𝑙𝑒𝑟 ⟩ simplicity of TransE (time complexity O(𝑑𝑒 )). The space complex-
The model learns latent features of Stealer, its indicators ity of TransR (O(𝑛𝑒 𝑑𝑒 + 𝑛𝑟 𝑑𝑒 𝑑𝑟 )) is more expensive than TransH
of compromise, and other related entities (e.g., campaigns (O(𝑛𝑒 𝑑𝑒 + 𝑛𝑟 𝑑𝑒 ))[44]. Additionally, on average, it takes six times
that involved Stealer) present in the training set. Table 4 more time to train TransR15 compared to TransH[25]. Consider-
shows the ordered set of predictions returned by the model, ing these model capabilities and complexity trade-offs, we choose
and the true answer is ranked #2. This is a significant result TransH as a representational translation-based model. DistMult
as it can simplify the job of an analyst to a great extent by performs better than TuckER by a small margin of 0.37% in only
narrowing down the scope for further analysis. Starting with
11 https://paper.seebug.org/papers/APT/APT_CyberCriminal_Campagin/2010/in-
10 https://www.fireeye.com/content/dam/fireeye-www/global/en/current- depth_analysis_of_hydraq_final_231538.pdf
threats/pdfs/rpt-operation-saffron-rose.pdf 15 Onbenchmark datasets
Predicting malware threat intelligence using KGs Seoul’21, November 14–19, 2021, Seoul, South Korea

Table 4: Detailed result of Entity prediction case studies

Case study 1: Predicting a malware family Case study 2: Predicting a Campaign

⟨𝑖𝑛𝑡𝑒𝑙 − 𝑢𝑝𝑑𝑎𝑡𝑒 [.]𝑐𝑜𝑚, 𝑖𝑛𝑑𝑖𝑐𝑎𝑡𝑒𝑠, ?⟩ ⟨?, 𝑖𝑛𝑣𝑜𝑙𝑣𝑒𝑠𝑀𝑎𝑙𝑤𝑎𝑟𝑒, 𝐻𝑦𝑑𝑟𝑎𝑞⟩
Rank Predicted entity Confidence score Predicted entity Confidence score
1 A malware hash12 0.5769 Night Dragon 0.0085
2 Stealer 0.5694 Google 0.0084
3 A malware hash13 0.5679 IXESHE 0.0072
4 Gholee 0.5679 Hydraq 0.0053
5 200.63.46.33 0.5668 MiniDuke 0.0053
6 26978_ns2._aeroconf2014[.]org 0.5662 Poison Ivy 0.0052
7 A malware hash14 0.5620 Google Hack Attack 0.0052
8 Capstone Turbine 0.5602 IRC 0.0044
9 2012/06/06 0.5592 Pitty Tiger 0.0043
10 Google Hack Attack 0.5566 Miniduke 0.0039
Bold text denotes the true entity for the corresponding test triple

Table 5: Entity Prediction for different Data sets

FB15K WN18 FB15K-237 WN18-RR

Model Hits@10↑ MRR↑ Hits@10↑ MRR↑ Hits@10↑ MRR↑ Hits@10↑ MRR↑
TransE 44.3 0.227 75.4 0.395 32.2 0.169 47.2 0.176
TransH 45.5 0.177 76.2 0.434 30.9 0.157 46.9 0.178
TransR 48.8 0.236 77.8 0.441 31.4 0.164 48.1 0.184
DistMult 45.1 0.206 80.9 0.531 30.3 0.151 46.2 0.264
ComplEx 43.8 0.205 81.5 0.597 29.9 0.158 46.7 0.276
TuckER 51.3 0.260 80.6 0.576 35.4 0.197 46.8 0.272
The top three rows are models from the translation-based approach, rest are from tensor factorization-based approach
Bold numbers denote the best scores in each category

one case (Hits@10 on WN18). TuckER and ComplEx both achieve models do[45]. For larger knowledge graphs, such as MalKG,
very close results on multiple cases. this property is especially pertinent because it may be miss-
Therefore, we look closely at the results of the recently refined ing entities from the KG due to shortcomings of entity ex-
datasets (FB15K-237 and WN18RR) as our MT3K dataset shares traction models or for predicting information for a sparsely
similar properties with these two datasets (MT3K has hierarchical populated graph. Accurate predictions help in both scenarios,
relations as in WN18RR and no redundant triples as in FB15K-237). especially in the field of cybersecurity.
Since the space complexity is very close (ComplEx: O(𝑛𝑒 𝑑𝑒 + 𝑛𝑟 𝑑𝑒 ) (2) We show that TuckER does not store all the latent knowl-
and TuckER: O(𝑛𝑒 𝑑𝑒 + 𝑛𝑟 𝑑𝑟 )) we choose TuckER for evaluation on edge into separate entity and relation embeddings. Instead,
MT3K, as it subsumes ComplEx [5]. it enforces parameter sharing using its core tensor so that all
We evaluate the performance of TransH [45] and TuckER [5] on entities and relations learn the common knowledge stored
our datasets. Tables 6 and 7 display the performance of TransH and in the core tensor. This is called multi-task learning, which
TuckER on MT3K and MT40K. TransH’s MRR score on MT3K im- no other model except Complex[42] employs. Therefore,
proved when compared to the benchmarks. Tucker’s performance TuckER creates embeddings that reflect the complex rela-
improved in all measures when compared to the other benchmark tions (1-n, n-1, n-n) between entities in the MalKG.
results. Evidently, TuckER outperforms TransH by a large margin (3) Moreover, we almost certainly ensure there are no inverse
on the MT3K dataset. Four reasons underlie this result: relations and redundant information in the validation and
test data sets. It is hard for simple yet expressive models
(1) MT3K has only 22 relations in comparison to 5,741 unique en-
like TransH to perform well in such datasets compared to
tities. This makes the knowledge graph dense with a higher
bi-linear models like TuckER.
entities-to-relation ratio, which implies a strong correlation
(4) The number of parameters of TransH grows linearly with
among entities. Tensor-factorization based approaches use
the number of unique entities (𝑛𝑒 ), whereas for TuckER, it
tensors and non-linear transformations that capture such
grows linearly for the embedding dimensions (𝑑𝑒 and 𝑑𝑟 ).
correlations into embeddings more accurately than linear
A TuckER low-dimensional embedding can be learned and
12 01da7213940a74c292d09ebe17f1bd01 reused across various machine learning models and performs
13 a476dd10d34064514af906fc37fc12a3
14 deeac56026f3804968348c8afa5b7aba10900aeabee05751c0fcac2b88cff71e
Seoul’21, November 14–19, 2021, Seoul, South Korea Anonymous

Table 6: Experimental evaluation of MT3K for entity predic- are constructed using competency questions that pave way for sys-
tion. tematic identification of semantic categories (called classes) and
properties (also called relations) [33]. An ontology also avoids gen-
Model Hits@1↑ Hits@3↑ Hits@10↑ MR↓ MRR↑ eration of redundant triples and accelerates the triple generation
process.
TransH 26.8 50.5 65.2 32.34 0.414
Several generic and specific knowledge graphs such as Freebase[8],
TuckER 64.3 70.5 81.21 7.6 0.697 Google’s Knowledge Graph17 , and WordNet[13] are available as
open source. Linked Open data[7] refers to all the data available
Table 7: Experimental evaluation of MT40K for entity pre- on the Web, accessible using standards such as Uniform Resource
diction. Identifiers (URIs) to identify entities, and Resource Description
Framework (RDF) to unify entities into a data model. However,
it does not contain domain-specific entities that deal with cyber
Model Hits@1↑ Hits@3↑ Hits@10↑ MR↓ MRR↑
threats and intelligence. Other existing knowledge graphs are ei-
TuckER 73.9 75.9 80.4 202 0.75 ther not open source, too specific, or do not cater to the breadth
of information our research intends to gather from intelligence
reports.
tasks such as classification and anomaly detection. We ex- Creating and completing a knowledge graph from raw text com-
plore anomaly detection as future work in the MalKG to prises multiple tasks, e.g., entity discovery, relation extraction, and
identify trustworthy triples. designing embedding models. In this section, we review notable
Since TuckER performed significantly better than TransH on research works to accomplish these tasks. Entity discovery refers
the ground truth dataset, we employ only TuckER on the MT40K to categorizing entities in a text according to a predefined set of
dataset. It is noteworthy that all the measures except mean rank semantic categories.
(MR) improved on MT40K compared to MT3K. It confirms the gen- Named Entity Recognition (NER) is a machine learning task for
eral intuition that as we provide more data, the model learns the entity discovery. The machine learning model learns features of
embeddings more effectively. MR is not a stable measure and fluc- entities from an annotated training corpus and aims to identify and
tuates even when a single entity is ranked poorly. We see that the classify all named entities in a given text. Deep neural network-
MRR score improved even though MR worsened, which confirms based approaches have become a popular choice in NER for their
the assumption that only a few poor predictions drastically affected ability to identify complex latent features of text corpora without
the MR on MT40K. Hits@10 being 80.4 indicates that when the any human intervention for feature design[24]. Long Short Term
entity prediction model predicted an ordered set of entities, the Memory (LSTM), a recurrent neural network architecture, in NER
true entity appeared in its top 10 for 80.4% times. Hits@1 mea- can learn beyond the limited context window of a component and
sure denotes that 73.9% times, the model ranked the true entity can learn long dependencies among words. Convolutional Neural
at position 1, i.e., for 80.4% cases, the model’s best prediction was Network (CNN) is a class of neural networks that learns character-
the corresponding incomplete triple’s missing entity. We can inter- level features like prefixes, and suffixes when applied to the task
pret Hits@3 similarly. We translate the overall entity prediction of NER. Chiu et al. [10] combine LSTM and CNN’s strengths to
result on MT40K to be sufficient such that the trained model can learn word-level and character-level features. Lample et al.[23] use
be improved and further used to predict unseen facts. a layer of conditional random field over LSTM to recognize entities
with multiple words. MGNER[46] uses a self-attention mechanism
8 RELATED WORK to determine how much context information of a component should
Knowledge graphs store large amount of domain specific informa- be used, and is capable of capturing nested entities within a text.
tion in the form of triples using entities, and relationships between In addition, pre-trained off-the-shelf NER tools such as Stanford-
them [26]. General purpose knowledge graphs such as Freebase[8], CoreNLP18 , Polyglot19 , NLTK20 , and spaCy21 are available. These
Google’s Knowledge Graph16 , and WordNet[13] have emerged in tools can recognize entities representing universal concepts such
the past decade. They enhance “big-data” results with semantically as person, organization, and time. These tools categorize domain-
structured information that is interpretable by computers, a prop- specific concepts (e.g., malware campaign, software vulnerability
erty deemed indispensable to build more intelligent machines. No in threat intelligence domain) as other or miscellaneous because
open-source malware knowledge graph exists, and therefore, in this the models are pre-trained on corpora created from news articles,
research, we propose the first of its kind automatically generated Wikipedia articles, and tweets. While it is possible to train these
knowledge graph for malware threat intelligence. models on a domain-specific corpus, a large amount of labeled
The MalKG generation process uses a combination of curated training data is required.
and automated unstructured data [26] to populate the KG. This has
helped us in achieving the high-accuracy of a curated approach and
the efficiency of an automated approach. MalKG was constructed
17 https://developers.google.com/knowledge-graph
using existing ontologies [33, 39] as they provide a structure to 18 https://stanfordnlp.github.io/CoreNLP/
MalKG generation with achievable end goals in mind. Ontologies 19 https://polyglot.readthedocs.io/en/latest/
20 https://www.nltk.org/
16 https://developers.google.com/knowledge-graph 21 https://spacy.io/api/entityrecognizer
Predicting malware threat intelligence using KGs Seoul’21, November 14–19, 2021, Seoul, South Korea

The absence of sufficient training data in a specific domain such 𝑓𝜙 (h, t) = −|h + r − t| 1/2
as malware threat intelligence poses a challenge to use the algo-
rithms mentioned above and tools to extract technical terms of that where | · | 1/2 means either the 𝑙 1 or the 𝑙 2 norm.
domain. Corpus-based semantic class mining is ideal for identifying TransE [9] creates embeddings such that for a correct triple
domain-specific named entities when there are very few labeled ⟨ℎ, 𝑟, 𝑡⟩, h + r ≈ t. However, it has the limitation of represent-
data instances. This approach requires two inputs: an unlabeled ing only 1-to-1 relations as it projects all relations in the same
corpus and a small set of seed entities appearing in the corpus vector space. Thus, TransE cannot represent 1-to-n, n-to-1,
and belonging to the same class. The system then learns patterns and n-to-n relations. TransH [45] overcomes the drawback of
from the seed entities and extracts more entities from the corpus TransE without increasing computational complexity. It uses
with similar patterns. Shi et al. [36], and Pantel et al.[31] prepare a separate hyperplane for each kind of relation, allowing
a candidate entity set by calculating the distributional similarity an entity to have different representations in different hy-
with seed entities, which often introduces entity intrusion error (in- perplanes when involved in different relations. TransR [25]
cludes entities from a different, often more general, semantic class). proposes a separate space to embed each kind of relation,
Alternatively, Gupta et al.[15], and Shi et al.[35] use an iterative making it a complex and expensive model. TransD [17] sim-
approach, which sometimes leads to semantic drift (shifting away plifies TransR by further decomposing the projection matrix
from the seed semantic class as undesired entities get included dur- into a product of two vectors. TranSparse [18] is another
ing iterations). SetExpan[34] overcomes both of these shortcomings model that simplifies TransR by enforcing sparseness on the
by using selective context features of the seeds and resetting the projection matrix. These models combined are referred to as
feature pool at the start of each iteration. TransX models. An important assumption in these models is
Following the entity discovery phase, extracting relations be- that an embedding vector is the same when an entity appears
tween entity pairs takes place. Deep neural network-based ap- as a triple’s head or tail.
proaches eliminate the need for manual feature engineering. To (2) Tensor based approaches: Tensor factorization-based ap-
extract relations for constructing a large scale knowledge graph, proaches consider a knowledge graph as a tensor of order
manual annotations to generate training data is costly and time- three, where the head and tail entities form the first two
consuming. Subasic et al. [38] use deep learning to extract relations modes and relations form the third mode. For each observed
from plain text and later align with Wikidata22 to select domain- triple ⟨𝑒𝑖 , 𝑟𝑘 , 𝑒 𝑗 ⟩, the tensor entry 𝑋𝑖 𝑗𝑘 is set to 1; and other-
specific relations. Jones et al. [20] propose a semi-supervised algo- wise 0. The tensor is factorized by parameterizing the enti-
rithm that starts with a seed dataset. The algorithm learns patterns ties and relations as low dimensional vectors. For each entry
from the context of the seeds and extracts similar relations. A con- 𝑋𝑖 𝑗𝑘 = 1, the corresponding triple is reconstructed so that the
fidence scoring method prevents the model from drifting away value returned by the scroing function on the reconstructed
from pertinent relations and patterns. Ji et al.[19] propose APCNN, triple is very close to 𝑋𝑖 𝑗𝑘 , i.e., 𝑋𝑖 𝑗𝑘 ≈ 𝑓𝜙 (⟨𝑒𝑖 , 𝑟𝑘 , 𝑒 𝑗 ⟩). The
a sentence-level attention module, which gains improvement on model learns the entity and relation embeddings with the
prediction accuracy from using entity descriptions as an input. In objective to minimize the triple reconstruction error:
applications where labeled data is scarce, distant supervision, a
learning approach for automatically generating training data using
heuristic matching from few labeled data instances is favored. Yao 𝐿 = Σ𝑖,𝑗,𝑘 (𝑋𝑖 𝑗𝑘 − 𝑓𝜙 (𝑒𝑖 , 𝑟𝑘 , 𝑒 𝑗 )) 2
et al. [49] use distant supervision to extract features beyond the
sentence level, and address document level relation extraction. where 𝑓𝜙 is a triple scoring function.
Vector embeddings address the task of knowledge representation Another model, RESCAL [28], comprises powerful bi-linear
by enabling computing on a knowledge graph. MalKG has highly models, but may suffer from overfitting due to large number
structured and contextual data and can be leveraged statistically for of parameters. DistMult [48] is a special case of RESCAL
classification, clustering, and ranking. However, the implementa- which fails to create embeddings for asymmetric relations.
tions of embedding vectors for the various models are very diverse. ComplEx [42] overcomes the drawback of DistMult by ex-
We can broadly categorize them into three categories: translation- tending the embedding space into complex numbers. HolE [27]
based, neural-network-based, or trilinear-product-based [41]. For combines the expressive power of RESCAL with the effi-
MalKG, we evaluate two embedding approaches that directly relate ciency and simplicity of DistMult. TuckER [5] is a fully ex-
to our work: pressive model based on Tucker decomposition that enables
(1) Latent-distance based approach: These models measure multi-task learning by storing some learned knowledge in
the distance between the latent representation of entities the core tensor, allowing parameter sharing across relations.
where relations act as translation vectors. For a triple ⟨h, r, t⟩, TuckER represents several tensor factorization models pre-
they aim to create embeddings h, r, t such that the distance ceding it as its special case. The model has a linear growth
between h and t is low if ⟨ℎ, 𝑟, 𝑡⟩ is a correct fact (or triple) of parameters as 𝑑𝑒 and 𝑑𝑟 increases.
and high otherwise. A correct fact is a triple where a relation (3) Artificial neural network based models: Neural network
exists between the head h and tail t. The simplest application based models are expressive and have more complexity as
of this idea is: they require many parameters. For medium-sized datasets,
they are not appropriate as they are likely to over fit the data,
22 https://www.wikidata.org/wiki/Wikidata:Main_Page which leads to below-par performance [44].
Seoul’21, November 14–19, 2021, Seoul, South Korea Anonymous

9 CONCLUSION [11] Sanjeev Das, Yang Liu, Wei Zhang, and Mahintham Chandramohan. 2015.
Semantics-based online malware detection: Towards efficient real-time protec-
In this research, we propose a framework to generate the first mal- tion against malware. IEEE transactions on information forensics and security 11,
ware threat intelligence knowledge graph, MalKG, that can produce 2 (2015), 289–302.
[12] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018.
structured and contextual threat intelligence from unstructured Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI
threat reports. We propose an architecture that takes as input an Conference on Artificial Intelligence, Vol. 32.
unstructured text corpus, malware threat ontology, entity extraction [13] Christiane Fellbaum. 2010. WordNet. In Theory and applications of ontology:
computer applications. Springer, 231–243.
models, and relation extraction models. It creates vector embed- [14] Shu Guo, Quan Wang, Bin Wang, Lihong Wang, and Li Guo. 2015. Semantically
dings of the triples that can extract latent information from graphs smooth knowledge graph embedding. In Proceedings of the 53rd Annual Meeting
and find hidden information. The framework outputs a MalKG, of the Association for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers). 84–94.
which comprises 27,354 unique entities, 34 relations and approxi- [15] Sonal Gupta and Christopher D Manning. 2014. Improved pattern learning for
mately 40,000 triples in the form of 𝑒𝑛𝑡𝑖𝑡𝑦ℎ𝑒𝑎𝑑 -relation-𝑒𝑛𝑡𝑖𝑡𝑦𝑡𝑎𝑖𝑙 . bootstrapped entity extraction. In Proceedings of the Eighteenth Conference on
Computational Natural Language Learning. 98–108.
Since this is a relatively new line of research, we demonstrate how a [16] Allen D Householder, Jeff Chrabaszcz, Trent Novelly, David Warren, and
knowledge graph can accurately predict missing information from Jonathan M Spring. 2020. Historical Analysis of Exploit Availability Timelines.
the text corpus through the entity prediction application. We moti- In 13th {USENIX } Workshop on Cyber Security Experimentation and Test ( {CSET }
20).
vate and empirically explore different models for encoding triples [17] Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge
and concluded that the tensor factorization based approach resulted graph embedding via dynamic mapping matrix. In Proceedings of the 53rd annual
in the highest overall performance among our dataset. meeting of the association for computational linguistics and the 7th international
joint conference on natural language processing (volume 1: Long papers). 687–696.
Using MT40K as the baseline dataset, we will explore robust [18] Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2016. Knowledge graph com-
models for entity and relation extraction for future work that can pletion with adaptive sparse transfer matrix. In Thirtieth AAAI conference on
artificial intelligence.
generalize to the cybersecurity domain for disparate datasets, in- [19] Guoliang Ji, Kang Liu, Shizhu He, Jun Zhao, et al. 2017. Distant supervision for
cluding blogs, tweets, and multimedia. Another application is to relation extraction with sentence-level attention and entity descriptions.
use vector embeddings to identify possible cybersecurity attacks [20] Corinne L Jones, Robert A Bridges, Kelly MT Huffer, and John R Goodall. 2015.
Towards a relation extraction framework for cyber-security concepts. In Pro-
from the background knowledge encoded in MalKG. The current ceedings of the 10th Annual Cyber and Information Security Research Conference.
prediction model can generate a list of top options for missing infor- 1–4.
mation. It will be interesting to extend this capability to predicting [21] Chanhyun Kang, Noseong Park, B Aditya Prakash, Edoardo Serra, and Venkatra-
man Sivili Subrahmanian. 2016. Ensemble models for data-driven prediction of
threat intelligence about nascent attack vectors and use that infor- malware infections. In Proceedings of the Ninth ACM International Conference on
mation to defend against attacks that are yet to propagate widely. Web Search and Data Mining. 583–592.
[22] Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link predic-
A fundamental issue that we want to address through this research tion in knowledge graphs. In Advances in neural information processing systems.
is inaccurate triples in the knowledge graph. This draws from the 4284–4295.
limitations of entity and relation extraction models and inaccurate [23] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami,
and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv
descriptions in the threat reports. preprint arXiv:1603.01360 (2016).
[24] Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep
learning for named entity recognition. IEEE Transactions on Knowledge and Data
Engineering (2020).
REFERENCES [25] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning
[1] Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and entity and relation embeddings for knowledge graph completion. In Proceedings
Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art of the AAAI Conference on Artificial Intelligence, Vol. 29.
NLP. In Proceedings of the 2019 Conference of the North American Chapter of the [26] Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015.
Association for Computational Linguistics (Demonstrations). 54–59. A review of relational machine learning for knowledge graphs. Proc. IEEE 104, 1
[2] Eslam Amer and Ivan Zelinka. 2020. A dynamic Windows malware detection (2015), 11–33.
and prediction method based on contextual understanding of API call sequence. [27] Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2015. Holographic
Computers & Security 92 (2020), 101760. embeddings of knowledge graphs. arXiv preprint arXiv:1510.04935 (2015).
[3] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, [28] Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way
and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The model for collective learning on multi-relational data.. In Icml, Vol. 11. 809–816.
semantic web. Springer, 722–735. [29] Alessandro Oltramari, Lorrie Faith Cranor, Robert J Walls, and Patrick D Mc-
[4] Şerif Bahtiyar, Mehmet Barış Yaman, and Can Yılmaz Altıniğne. 2019. A multi- Daniel. 2014. Building an Ontology of Cyber Security.. In Semantic Technology
dimensional machine learning approach to predict advanced malware. Computer for Intelligence, Defense and Security. Citeseer, 54–61.
networks 160 (2019), 118–129. [30] Ankur Padia, Konstantinos Kalpakis, Francis Ferraro, and Tim Finin. 2019. Knowl-
[5] Ivana Balažević, Carl Allen, and Timothy M Hospedales. 2019. Tucker: Tensor edge graph fact prediction via knowledge-enriched tensor factorization. Journal
factorization for knowledge graph completion. arXiv preprint arXiv:1901.09590 of Web Semantics 59 (2019), 100497.
(2019). [31] Patrick Pantel, Eric Crestan, Arkady Borkovsky, Ana-Maria Popescu, and Vishnu
[6] Sean Barnum. 2012. Standardizing cyber threat intelligence information with the Vyas. 2009. Web-scale distributional similarity and entity set expansion. In
Structured Threat Information eXpression (STIX). MITRE Corporation 11 (2012), Proceedings of the 2009 Conference on Empirical Methods in Natural Language
1–22. Processing. 938–947.
[7] Florian Bauer and Martin Kaltenböck. 2011. Linked open data: The essentials. [32] Shirley Radack and Rick Kuhn. 2011. Managing security: The security content
Edition mono/monochrom, Vienna 710 (2011). automation protocol. IT professional 13, 1 (2011), 9–11.
[8] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. [33] Nidhi Rastogi, Sharmishtha Dutta, Mohammed J Zaki, Alex Gittens, and Charu
2008. Freebase: a collaboratively created graph database for structuring human Aggarwal. 2020. MALOnt: An Ontology for Malware Threat Intelligence. arXiv
knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on preprint arXiv:2006.11446 (2020).
Management of data. 1247–1250. [34] Jiaming Shen, Zeqiu Wu, Dongming Lei, Jingbo Shang, Xiang Ren, and Jiawei Han.
[9] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Ok- 2017. Setexpan: Corpus-based set expansion via context feature selection and
sana Yakhnenko. 2013. Translating embeddings for modeling multi-relational rank ensemble. In Joint European Conference on Machine Learning and Knowledge
data. In Advances in neural information processing systems. 2787–2795. Discovery in Databases. Springer, 288–304.
[10] Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional [35] Bei Shi, Zhengzhong Zhang, Le Sun, and Xianpei Han. 2014. A probabilistic
LSTM-CNNs. Transactions of the Association for Computational Linguistics 4 co-bootstrapping method for entity set expansion. (2014).
(2016), 357–370.
Predicting malware threat intelligence using KGs Seoul’21, November 14–19, 2021, Seoul, South Korea

[36] Shuming Shi, Huibin Zhang, Xiaojie Yuan, and Ji-Rong Wen. 2010. Corpus-based Conference on Machine Learning (ICML).
semantic class mining: distributional vs. pattern-based approaches. In Proceedings [43] Common Vulnerabilities. 2005. Common vulnerabilities and exposures.
of the 23rd International Conference on Computational Linguistics (Coling 2010). [44] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph
993–1001. embedding: A survey of approaches and applications. IEEE Transactions on
[37] Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, Knowledge and Data Engineering 29, 12 (2017), 2724–2743.
and Jun’ichi Tsujii. 2012. BRAT: a web-based tool for NLP-assisted text annotation. [45] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge
In Proceedings of the Demonstrations at the 13th Conference of the European Chapter graph embedding by translating on hyperplanes.. In AAAI, Vol. 14. Citeseer,
of the Association for Computational Linguistics. 102–107. 1112–1119.
[38] Pero Subasic, Hongfeng Yin, and Xiao Lin. 2019. Building Knowledge Base [46] Congying Xia, Chenwei Zhang, Tao Yang, Yaliang Li, Nan Du, Xian Wu, Wei Fan,
through Deep Learning Relation Extraction and Wikidata.. In AAAI Spring Sym- Fenglong Ma, and S Yu Philip. 2019. Multi-grained Named Entity Recognition.
posium: Combining Machine Learning with Knowledge Engineering. In Proceedings of the 57th Annual Meeting of the Association for Computational
[39] Morton Swimmer. 2008. Towards an ontology of malware classes. Online] January Linguistics. 1430–1440.
27 (2008). [47] Han Xiao, Minlie Huang, Yu Hao, and Xiaoyan Zhu. 2015. TransA: An adaptive
[40] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choud- approach for knowledge graph embedding. arXiv preprint arXiv:1509.05490
hury, and Michael Gamon. 2015. Representing text for joint embedding of text (2015).
and knowledge bases. In Proceedings of the 2015 conference on empirical methods [48] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Em-
in natural language processing. 1499–1509. bedding entities and relations for learning and inference in knowledge bases.
[41] Hung Nghiep Tran and Atsuhiro Takasu. 2019. Analyzing knowledge graph arXiv preprint arXiv:1412.6575 (2014).
embedding methods from a multi-embedding interaction perspective. arXiv [49] Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan
preprint arXiv:1903.11406 (2019). Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A Large-Scale
[42] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Document-Level Relation Extraction Dataset. In Proceedings of the 57th Annual
Bouchard. 2016. Complex embeddings for simple link prediction. International Meeting of the Association for Computational Linguistics. 764–777.