Future Generation Computer Systems 135 (2022) 30–43

Contents lists available at ScienceDirect

Future Generation Computer Systems

journal homepage: www.elsevier.com/locate/fgcs

Boosting Cyber-Threat Intelligence via Collaborative Intrusion

Detection
∗
Massimo Guarascio , Nunziato Cassavia, Francesco Sergio Pisani, Giuseppe Manco
Institute for High Performance Computing and Networking, Italian National Research Council, ICAR-CNR, Via P. Bucci, 87036, Rende (CS), Italy

article info a b s t r a c t

Article history: Sharing threat events and Indicators of Compromise (IoCs) enables quick and crucial decision making
Received 4 November 2021 relative to effective countermeasures against cyberattacks. However, the current threat information
Received in revised form 20 April 2022 sharing solutions do not allow easy communication and knowledge sharing among threat detection
Accepted 23 April 2022
systems (in particular Intrusion Detection Systems (IDS)) exploiting Machine Learning (ML) techniques.
Available online 30 April 2022
Moreover, the interaction with the expert, which represents an important component to gather verified
Keywords: and reliable input data for the ML algorithms, is weakly supported. To address all these issues,
Cyber Threat Intelligence architecture ORISHA, a platform for ORchestrated Information SHaring and Awareness enabling the cooperation
Security data enrichment among threat detection systems and other information awareness components, is proposed here.
Active Learning ORISHA is backed by a distributed Threat Intelligence Platform based on a network of interconnected
Intrusion Detection System Malware Information Sharing Platform instances, which enables the communication with several
Threat analytics Threat Detection layers belonging to different organizations. Within this ecosystem, Threat Detection
SIEM
Systems mutually benefit by sharing knowledge that allows them to refine the underlying predictive
accuracy. Uncertain cases, i.e. examples with low anomaly scores, are proposed to the expert, who acts
with the role of oracle in an Active Learning scheme. By interfacing with a honeynet, ORISHA allows for
enriching the knowledge base with further positive attack instances and then yielding robust detection
models. An experimentation conducted on a well-known Intrusion Detection benchmark demonstrates
the validity of the proposed architecture.
© 2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction to make quick decisions and set up effective countermeasures.

In particular, coupling proactive threat information sharing and
Nowadays, organizations and users face an enormous amount defensive mitigation strategies allow for strengthening the re-
of sophisticated, targeted and widespread cyberattacks. Malicious silience of the entities belonging to the trusted community by
actors were proven able to compromise government computer yielding a herd immunity against new (possibly unknown) at-
systems as well user devices causing various types of damages. tacks and malware. Hence, the need to design platforms, tools and
Phishing, identity theft, information leakage, DDOS and botnet methodologies for accessing and sharing threat events in order
represent some examples of popular threat occurred in 2020 [1]. to react promptly and prevent further damages. Cyber Threat
The outbreak of COVID-19 has further exacerbated this situa- Intelligence (CTI) platforms are considered valuable tools for eas-
tion. As the virus spread during the early part of the 2020, the ing the management of threat information [4]. These solutions
number of cyberattacks against organizations grew exponentially, allow organizations to easily handle the whole process of gather-
reaching a peak in April [2,3]. The pandemic unveiled different ing, preprocessing, enriching, correlating, analyzing and sharing
vulnerabilities of well-known platforms, applications and sys- threat events and associated data [5]. According to Dandurand
tems, and simultaneously stimulated the interest for promoting and Serrano [6], the main requirements for a Threat Intelligence
the usage of information sharing technologies to increase the Platform (TIP) can be summarized into (i) providing services for
degree of security for enterprises and organizations. information sharing, (ii) automatizing the process, (iii) enabling
In this context, timely sharing threat events and Indicators of functionalities for collaborative threat data analysis. Although TIP
Compromise (IoCs) among organizations can be crucial in order technologies can potentially bring important benefits to the orga-
nizations, developing a comprehensive Cyber Threat Intelligence
∗ Corresponding author. platform able to handle information from different sources is
E-mail addresses: [email protected] (M. Guarascio),
difficult to achieve [7]. Standardization, privacy and reliability of
[email protected] (N. Cassavia), [email protected] the shared information are just some examples of the open chal-
(F.S. Pisani), [email protected] (G. Manco). lenges in defining a fully operational platform. Some recent works

https://doi.org/10.1016/j.future.2022.04.028
0167-739X/© 2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

proposed interesting solutions for easing threat data sharing but both, in terms of attack detection capability and quality of the
they only focused on addressing some of the above-mentioned shared information. In Section 4 we describe a suite of experi-
issues [8,9]. Wagner et al. [10] provide an extensive overview ments which demonstrate the benefits of adopting ORISHA within
of the current state-of-the-art and highlight open technical and an intrusion detection scenario. Finally, Section 5 concludes the
non-technical challenges for the cyber threat intelligence. paper and outlines future research directions.
When we consider also Machine Learning (ML) based threat
detection systems, the situation is further exacerbated by some 2. Background and related works
specific challenges:
Threat Intelligence refers to the task of gathering data concern-
• The quality of threat feeds and events is not guaranteed ing attacks or breaches (e.g. context, methods, indicators, devices,
and there is a need for a reliable and automated threat etc.) for enabling the organizations to set up countermeasures
analysis and mitigation. This is particularly problematic for on the basis of a wide range of information [11]. In order to
Threat Detection Systems (TDS, such as Intrusion Detection enhance prevention and detection of new threats, organizations
Systems) based on AI/ML techniques, which are typically can collaborate by sharing information about recent discovered
affected by high false positive rates thus nullifying both their threats. This information is usually made available under the form
detection capabilities and the informative content of their of Indicator of Compromises (IoCs). Basically, an IoC is a piece
detection. of forensic data identifying potentially malicious activity on a
• Combining Threat Intelligence Platforms with AI based system or network. Typical examples of IoCs are the source ip
Threat Intelligence solutions in a single comprehensive address of an attack, the hash of a malicious executable file or
framework is a challenging task since the former ones usu- the URL of a phishing web site.
ally work in an event-based fashion while the latter are typ- Threat Intelligence represents an emerging and relatively new
ically data-driven. In particular, there is no simple way for research line in the field of cybersecurity and, as highlighted
two machine learning algorithms to share, analyze and com- in [12], there is a growing interest in this topic by both academic
pare their findings within a standardized framework that and industrial entities. Cooperation and data sharing allow for
can boost their attack detection and mitigation capabilities. improving the security of the computer networks and reducing
the risk of compromising. The recent research has mainly focused
Contribution and organization. In this work, we propose ORISHA, on devising tools for the threat information sharing, so that in
a platform for ORchestrated Information SHaring and Awareness the last years we have observed a proliferation of threat intelli-
particularly focused on the two above mentioned issues. ORISHA gence platforms [13]. The lack of standards and solid approaches
has the twofold objective of (i) improving the accuracy of TDS resulted in different combinations of products and methodologies
in detecting incoming attacks, and (ii) enabling the sharing of frequently erroneously labeled as threat intelligence. Johnson
reliable and relevant threat information among organizations and et al. [4] provided some tentative guidelines, by devising informa-
threat detection algorithms. The guiding principle is that TDSs can tion sharing goal for organizations, identifying threat information
benefit each other mutually by sharing knowledge, since a threat sources and proposing rules for managing the publication and
feed shared by a TDS can be exploited by another one to improve distribution of the threat information of an organization. How-
its threat modeling strategies. At the same time, the same feed ever, there is no consensus among researchers and practitioners
can be enriched by enabling communication with other layers, on the usage of a specific methodology or technology since there
which can be exploited to further characterize the underlying is no comprehensive solution to the standardization, privacy and
threats and provide important insights about them. reliability issues related to the sharing process.
The framework is based on an exchange protocol where in- In the following, we provide an overview of the main stan-
formation concerning threats is published on a distributed TIP dards and technologies for the threat intelligence, we survey
(implemented as a network of several Malware Information Shar- some relevant works exploiting the collaboration among different
ing Platform (MISP) instances) and made accessible to other ac- network security tools and analyze some emerging approaches in
tors, which can then exploit their capabilities in order to further this research field.
characterize the feed and acquire it in their knowledge base. Standards and solutions for threat information sharing. While in
This collaboration framework has several advantages. On one the past, channels such as mail messages, phone calls, ticket
side, it allows for improving the performances of the threat systems, or face-to-face meetings were the main ways to quickly
prevention and detection systems, yielding more robust and ef- share threat information, with the exponential growth cyberat-
fective machine/deep learning models, reducing the false alarm tacks these channels have been replaced with advanced solutions
rate and the average time required for identifying a successful able to automate the whole process. Different standards, such as
intrusion. On the other side, it enables more robust threat in- Structured Threat Information CybereXpression (STIX) [14], Cyber
telligence by allowing to better contextualize threat data and Observable eXpression (CybOX) [15], Incident Object Description
devise flexible strategies, methodologies and data formats for col- Exchange Format (IODEF) [16] and Trusted Automated eXchange
laborative threat intelligence, also by improving the notification of Indicator Information (TAXII) [17], have been proposed to
mechanisms to appropriately notify relevant stakeholders having simplify the sharing of these indicators. As above mentioned, a
different needs for a contextualized interpretation of threat data. large number of platforms based on these standards have been
The rest of the paper is structured as follows. Section 2 pro- proposed [5,12] and here we review some of those dedicated to
vides an overview on the main platforms and solutions for threat threat information sharing [18]:
information sharing and awareness. We also review some rele-
vant works which focus on the exploitation of external sources to • MISP (Malware Information Sharing Platform)1 is an open
improve the performance of TDS and the role of Active Learning in source software solution for collecting, storing, distributing
cybersecurity solutions. Section 3 describes the devised ORISHA and sharing cybersecurity indicators and threat informa-
platform for threat event sharing. We detail the data exchange tion [19]. The platform encompasses several public MISP
format and discuss how it integrates data-driven intrusion de- communities, available and interconnected via MISP API,
tection system. In particular, Section 3.3 exemplifies the whole sharing threats or cybersecurity indicators worldwide.
information flow and illustrates the benefits in adopting the pro-
posed protocol for the overall threat information sharing process 1 https://misp-project.org/.

31
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Table 1
Threat intelligence solution comparison.
Platform Supported standards Support for Extensibility and SIEM and IDS Documentation Licence
cooperation support for custom integration
solutions
MISP STIX, CybOX, TAXII High ✓ ✓ Advanced Open source (GNU
General Public License)
MITRE CRITs STIX, TAXII, OpenIOC, High × × Medium Open source (GNU
Send/receive information General Public License)
through Facebook’s
ThreatExchange
CIF STIX, CybOX Medium × × Basic Open source (GNU
General Public License)
EclecticIQ platform STIX, TAXII Medium Partial × Medium Commercial
LookingGlass cyber STIX and TAXII via High × × Basic Commercial
scoutPRIME

• MITRE CRITs (Collaborative Research Threats)2 is an open be easily identified by a human attacker therefore they are not
source malware and threat repository that leverages differ- totally reliable and the gathered information can be limited.
ent open source software to create a unified tool for analysts High interaction honeypots can include high level functional-
and security experts engaged in threat defense [20]. This ities such as, e.g., access to the operating system [22], usage of
tool employs a hierarchy to structure cyber threat informa- real devices, etc. Moreover, there exist advanced implementation
tion, thus enabling the analysts to perform complex queries based on complex virtual environments that allow for emulating
and discover previously unknown related content. both services and devices.
• CIF (Collective Intelligence Framework) is an open source By contrast, medium interaction honeypots represent an inter-
cyber threat intelligence platform that allows gathering data mediate solution, since they provide access to the (possibly em-
from different sources and exploiting them for threat iden- ulated) operating system and allow for emulating more services
tification, detection, and mitigation. IP addresses, domains, and functionalities than low interaction honeypots.
and URLs can be stored and preprocessed as threat data Although the interaction level represents a key feature, there
within the platform. It permits to ingest many different exist other relevant aspects that characterize honeypots families,
sources of data sets such as feeds of malicious domains. such as purpose, role, resource level. A more detailed overview can
• EclecticIQ Platform3 is a commercial platform gathering be found in [23].
and interpreting intelligence data from open sources, com-
mercial suppliers and industry partnerships. This platform Improving threat detection via information sharing and data en-
is based on STIX and TAXII standards and provides analyst- richment. The idea to support the collaboration among different
friendly workflows. Threat Detection Systems is not new. In Intrusion Detection sce-
narios, Collaborative Intrusion Detection Systems were proposed
• LookingGlass Cyber4 provides two commercial solutions,
to improve the effectiveness of (local) IDSs. The idea is that
scoutPRIME and scoutSHIELD. These tools provide utilities
for both collecting threat information and handling threat several monitors cooperate in either a hierarchical or peer-to-
responses, by making available collaboration and sharing peer way to discover malicious behaviors, playing both the role
tools for the threat analysis. of sensors and data collectors. Vasilomanolakis et al. [24] provide
a comprehensive survey of the main techniques and solutions
Table 1 compares the TIP solutions illustrated above and high- proposed in this setting. Although this collaboration allows for
light the main features that characterize them. Among the di- improving the performances of singleton IDSs, data trust and
mensions of comparison, we mention support for cooperation, privacy represent two relevant issues. Moreover, the missing of
extensibility and integration with advanced tools. a standard data exchange format limits the capabilities of these
approaches.
Honeypot technology. Honeypots are well-consolidated technolo- A different approach to improve the accuracy of TDSs consists
gies in the cybersecurity domain and allow for logging precious in integrating information from honeypots. In [25], a hybrid and
information on the behavior of attacks and malware. Basically, adaptable honeypot-based approach is proposed that improves
a honeypot is a computer system devised as a bait for hackers, the IDSs for protecting networks from intruders. The main idea
with the purpose to gather information about prospective attacks. consists in recording and analyzing the intruder’s activities and
The main idea consists in monitoring the actions performed by using the results to take administrative actions for protecting the
a malicious user to probe, attack and compromise the decoy network. The authors provide an overview of the main compo-
system. From this simple concept, different implementations of nents of the systems and illustrate some performance and load
honeypots can be derived and when two or more honeypots are testing scenarios.
deployed in a single environment they form a honeynet [21]. A Baykara and Das [26] introduce a honeypot-based approach
relevant and widely adopted classification of honeypots is based aiming at reducing the false positive rate. A honeypot based IDS is
on the level of interaction: low, medium and high. implemented able to monitor in real-time the network traffic on
Low interaction honeypots [22] allow for simulating one or specific servers. The system represents a hybrid honeypot com-
more services exposing simple functions without providing ac- bining the capabilities of low and high interaction honeypots in
cess to the operating system. The main benefits in using these a single structure. The resulting log files are analyzed to identity
solutions are low risk, cost and maintenance. However, they can new zero-day attack and to increase IDS knowledge.
Khosravifar and Bentahar [27] propose a new architecture
2 https://crits.github.io/. combining distributed agents and honeypots. The approach aims
3 https://www.eclecticiq.com/platform. at reducing the false alarm rate in attack detection. Anomalous
4 https://www.lookingglasscyber.com/. connections, initially detected by the IDSs, will be re-routed to a
32
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

honeypot network for a more accurate analysis. If, as a result of (e.g., STIX, TAXI, etc.); (iv) availability of a rich and very detailed
this investigation, a misclassification of the IDS is discovered, the documentation; (v) a number of active communities, and (vi)
connection will be redirected to the original destination so that distributed with open source license.
the interaction can be completed. Different types of Threat Detection Systems (e.g., Intrusion
Sibi Chakkaravarthy et al. [28] illustrate Intrusion Detection detection and prevention systems) can interface with the TIP
Honeypot, an approach combining Honeypots and IDS in a sin- by providing information concerning incoming attacks and feed-
gle system. It includes three main components: Honeyfolder, ing it with new intrusion events/statistics. In addition, they can
Audit Watch, and Complex Event Processing. The former is a de- exploit information stored in the TIP to improve their mod-
coy folder exploiting the Social Leopard Algorithm (SoLA), Audit els/rules/signatures. We consider a TDS layer here, which can
Watch is a module in charge to verify the entropy of the files and include several tools and techniques for real-time and off-line
folders, and the last allows for aggregating data from different detection. Honeypots are deployed with the aim to collect addi-
security systems to confirm the ransomware behavior. tional information concerning new attacks. The deployment can
be enabled by TIP, in order to enrich existing events coming either
Active learning solutions for threat detection. Active Learning (AL)
from the IDS or other sources.
refers to a family approaches and algorithms where new in-
The general concept relies essentially on two main compo-
stances to be labeled are interactively chosen by means of specific
nents in order to actually enable information sharing: a data
queries [29]. Basically, the idea consists in providing unknown
(unlabeled) examples, which can be extracted by adopting differ- exchange format and the layers that orchestrate the communi-
ent strategies, to an oracle that will correctly label them. Active cation between TDSs and TIP. In the following, we instantiate
Learning can be used in any scenario where there is a scarcity of such components to work with ML-Based IDSs aimed at detecting
labeled data or the latter is highly skewed [30]. In cybersecurity anomalous flow connections. We next illustrate how ORISHA
for example, gathering, labeling and sharing data to train any kind enables an Active Learning framework and finally we describe
of engine/classifier represents a hard and expensive task. As a the interaction between the honeynet and the other components.
consequence, Active Learning strategies were recently explored, We remark here that the platform can be extended to embed
aimed at improving the effectiveness of the underlying IDS. A other threat detection systems, provided that suitable adaptation
comprehensive overview on Active Learning based methods and to both the data exchange format and the communication layers
techniques for detecting anomalous behaviors (e.g., intrusion de- is guaranteed.
tection, fault detection, etc.) can be found in [31]. Dang [32] Notably, in order to make the platform scalable with the
proposes the use of active learning in online configuration to number of threats to process, ORISHA can exploit a distributed
reduce the labeling cost but maintaining the classification perfor- approach in which several MISP instances (possibly belonging to
mance. Different from other existing active learning algorithms, the same organization) are used together to handle high loads.
the author focuses on the labeling of rare events deemed more Moreover, the requests can be replicated on multiple nodes to
important for the learning phase. The resulting sampling strategy efficiently process them.
shows substantial improvements. The adoption of active learning
strategies for wireless intrusion detection is studied in [33]. The 3.1. Sharing threat events: ORISHA data exchange format
authors survey the main sampling strategies for AL and then
evaluate the improvements due to the usage of the human in the The ORISHA components exploit a common interface imple-
loop approach on an intrusion detection benchmark. mented as a custom MISP Object in JSON format. A MISP object
Active learning can be considered as a special case of Rein- represents the data structure adopted by MISP to store shared
forcement Learning. Here, the oracle is replaced by the interaction threat events. The general template can be easily extended to
with the environment, where the model is rewarded subject to include further relevant information on specific threat events. Our
its ability provide the best action to be performed. Lopez-Martin objective is to devise a custom ORISHA MISP template, which can
et al. [34] propose a Deep Reinforcement Learning (DRL) frame- embed the whole set of data of relevance for a TDS. These include:
work able to provide more accurate predictions in comparison
with standard machine learning techniques.
• Access to reliable threats, that is, threats for which a general
consensus and interpretation can be achieved. The available
While AL is widely considered a consolidated approach, com-
information can then be used to improve the detection
bining AL strategies with DL architectures represents a new rele-
capabilities. For traditional IDS like Suricata or Snort this
vant research line (i.e., Deep Active Learning). In [29] the authors
is achieved by devising new security policy rules. However
survey some relevant approaches proposed in this direction.
data-aware IDS should be able to exploit such informa-
3. The ORISHA Platform tion for extending their training sets and rebuilding the
underlying ML models with improved accuracy.
In this section, we provide an overview of the proposed OR- • Export of the discovered threats to the MISP network, for
ISHA architecture. Fig. 1 illustrates the idea behind the platform. further investigation and confirmation analysis. In partic-
There are essentially three actors that cooperate: Distributed TIP, ular, information concerning threats should allow further
TDS Layer and Honeynet. The Threat Intelligence Platform rep- labeling/categorization by other actors/TDS accessing it.
resents the core component of the whole architecture. It plays The components of the Custom MISP object are illustrated in
a two-folds role: (i) storing data coming from heterogeneous Table 2. Some of them are particularly relevant for collaborative
sources in an encrypted and distributed way; and (ii) delivering ML-based Network Intrusion Detection:
the gathered information to the other components. In practice,
several MISP instances can cooperate and share data about up- • pcap_file: This file contains the (anomalous) network
coming threat events from different modules integrated in the activity between two devices in ‘‘.pcap’’ Wireshark format. It
platform. The choice of MISP is due to different benefits and is essentially the information exploited by the IDS to detect
capabilities provided by this platform [35,36], as illustrated in the threat and as such it is fundamental for the purposes of
Table 1: (i) integration with SIEMs and IDSs capability; (ii) eas- the sharing philosophy of the platform. In fact, it is the main
ily extensible and flexible architecture enabling the possibility piece of information to analyze for confirming the threat.
to define custom solutions; (iii) support for different standards Also, in case a confirmation is available, it can be exploited
33
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Table 2
ORISHA MISP object.
Name Type Description Multiple
creation-date Datetime Threat event date ×
ip_dst ip-dst Destination IP ✓
ip_dst_port Port Destination port ✓
ip_src ip-src Source IP ✓
pcap_file Attachment PCAP file ×
verified Boolean True if the occurrence of the attack has been verified (e.g., ×
operator check, consensus of a TDS committee, etc.)
signature_type Text Type of signature (e.g. md5, sha1, etc.) ×
signature Text Optional detected file signature ✓
attack_type Text A JSON containing information on IDS classification ×
anomaly_details Attachment Optional JSON file containing anomaly flow statistics ×

Fig. 1. ORISHA platform.

for enriching the knowledge-base of all the TDS accessing

to it. Since it can potentially contain sensitive information, it
can be encrypted for secure sharing with trusted-only peers.
• attack_type: This field stores, in JSON format, relevant in-
formation about the threat. It includes a list of classification
events relative to the pcap associated with the MISP object.
Each event can be relative to a specific IDS and Multiple
IDS can contribute with entries within attack_type. For
example, if algorithm X detected a DOS attack within the
pcap file, then attack_type includes an entry containing
the attack label (DOS), a confidence value/anomaly score,
and the signature of the detection algorithm.
• anomaly_details: An optional JSON fragment containing
aggregate statistics and further information exploited by the
IDS for characterizing the threat. For example, it can include
some summary statistics extracted from the pcap file, or the
signatures used to characterize it.
Fig. 2 shows an example instance. Here, attack_type con-
tains JSON instantiated with a specific entry, in lines 34–38. It
refers to a specific ML-Based IDS, which relatively to the attached
pcap file (anomaly_pcap, as declared in the pcap_file at-
tribute on lines 39–43), detects an anomalous traffic flow (labeled
as ANOMALY) with a 98% confidence level.
The protocol only allows representing single events and does
not model correlations of diverse events that are possibly related
to the same attack. Nevertheless, in Section 4, we show how these
data can be used to capture different up-to-date types of attack
e.g., DDOS, Bot, Infiltration, etc.

3.2. Integrating TDSs with the TIP

As shown in Fig. 1, the distributed TIP interfaces with a pool

of Threat Detection Systems (grouped within the TDS Layer in
Fig. 2. An example MISP security event object.
figure). The latter provide in-the-wild information (e.g. traffic
flow anomalies, attack classification, malware signatures, etc.) about
anomalies and/or attacks on the monitored system, to be shared
34
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Fig. 3. TDS Integration and honeypot interface.

through the distributed TIP. More in general, single TDSs or TDS MISP. The Threat Importer collects data (pcap files) from the
layers distributed on several machines and belonging to different security events and prepares them for the learning phase of
organization cooperate by exploiting the TIP to share relevant EBIDS. Finally, the IDS Event Exporter produces security events
data on detected anomalies or attacks. Particularly relevant here discovered by EBIDS and shares them with the MISP. All commu-
are the interactions among ML-based threat detection systems, nications within the platform occur via Rest Server. The values
which are detailed as follows. included within the attribute ‘‘attack_type’’ are the responsi-
For the purpose of illustration, we assume that the TDS layer bility of the modules who deliver and/or enrich it, as illustrated
includes EBIDS (Ensemble Based IDS) [37], a ML-based Intrusion in Fig. 2. For example, EBIDS labels each flow (gathered from the
Detection technique adopting specialized ensembles of classifi- network traffic into pcap files via TCPDump and preprocessed)
cation models to identify undetected attacks by analyzing traffic as either normal or attack. Then, a sample of (probably) benign
flow statistics extracted from network logs. Here, we use pcap flows (i.e., flows exhibiting an anomaly score less than a threshold
format to share network flow information, but the proposed secu- value) is used to update the local knowledge base, while abnor-
rity event object, described in the previous section, is flexible and mal flows are shared with the Distributed TIP. This means that
allows for supporting data shared in other formats. With respect for each abnormal flow (associated with a pcap file), a security
to the proposed architecture, EBIDS requires as input one or more event is created with the ‘‘attack_type’’ attribute instantiated.
pcap files to compute statistics concerning the network traffic We model ‘‘attack_type’’ as a list, to ensure flexibility. Thus,
flows. In practice, following a widely adopted approach in the lit- each event (and the associated pcap file) can be associated with
erature [38], EBIDS addresses the (binary) classification problem multiple labels (and the respective degree of confidence). In the
of deciding whether a given network connection (or network flow) example shown in Fig. 2, EBIDS can classify the pcap flow as
is associated with an intrusion attack or not. The decision is based ‘‘ANOMALY’’. The event is shared with MISP and made available to
on a fixed-length representation of the connection itself, which other IDSs which can enrich it with further information. Benign
summarizes and aggregates its features. This representation is flows do not trigger events, but events already labeled as attacks
exploited during the training phase of the classification ensemble. can also be further labeled as normal by other TDSs: in such
The learned model allows for discovering anomalous behaviors cases, the label added to ‘‘attack_type’’ will be ‘‘NONE’’, with
(potentially related with different types of cyberattacks) that are the associated confidence value. This interaction is crucial for
shared with the other software modules and organizations under ensuring feedback on false positives. A sophisticated example is
the form of MISP security events. given by the fragment in Fig. 4. The same pcap file has been
Each TDS can act both as a data provider and consumer. The analyzed by three different TDSs: two instances of EBIDS (trained
key for enabling this behavior is the information stored within with proprietary data from organizations XXX and YYY), and a
the security events. As a consumer, a TDS can retrieve custom different IDS (named OTHERIDS). The labels associated with the
security events from the MISP, and exploit the enclosed infor- event are: ‘‘ANOMALY’’ (with confidence 90%) by the first EBIDS
mation to improve the detection capabilities of the underlying instance, ‘‘DDOS’’ (with confidence 70%) and ‘‘SMURF’’ (with con-
classification model. This is achieved by feeding it with new fidence 100%) by ‘‘OTHERIDS’’; and ‘‘NONE’’ (i.e., no anomaly, with
labeled data concerning the attacks represented by the security confidence 66%) by the second EBIDS instance. In the same vein,
events. As a producer, a TDS can identify/classify new attack intrusions (i.e., unauthorized accesses to the deployed services)
instances from which to build and deliver the related security logged by honeypots trigger events describing the network flow
events. Such events can hence be consumed by other TDSs or and the most suitable label (encoded with the ‘‘attack_type’’
exploited to deploy specific honeypots tailored for the underlying attribute) triggered by the forensic analysis.
attacks. Furthermore, it can analyze security events already stored Since EBIDS will be used in Section 4 to demonstrate the
within the MISP, and enrich them with additional attack_type capabilities of ORISHA, it is worth looking at the details of its
labeling. detection model. We next describe the components used to in-
In Fig. 3, we explicit the interactions among the main compo- terface the TDSs with the TIP. EBIDS can be considered an overall
nents. Basically, three macro-modules realize the import/export neural network integrating two components: a number of (weak)
facilities: (i) Log Extraction and Transformation (Honeypot Log deep models sharing the same neural architecture (but trained
ETL) module, (ii) Threat Importer and (iii) IDS Event Exporter. against different data chunks), and a combiner sub-net employed
The former is aimed at gathering data (e.g. pcap files) from the for yielding the overall anomaly score from the predictions of
honeynet and at delivering the associated security events to the the base classifiers. In more detail, the base model architecture
35
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

3.3. Exploiting TDS cooperation and active learning

In this section, we illustrate how different TDSs interfacing

with ORISHA can cooperate to improve their detection capa-
bilities and improve digital evidence on threats useful for the
decision making process. Fig. 6 exemplifies the whole information
flow.
The starting point is represented by the monitored system.
Here, traffic flow from the computer network is periodically
analyzed by the components of the TDS Layer. In this scenario, a
specific anomaly detector (TDS1 in figure), analyzes the pcap files
including a specific logged traffic flow and detects an anomaly.
A MISP security event is then produced and delivered. The event
is distributed in the TIP, where a different IDS (TDS2 in figure)
accesses the event, analyzes the embedded pcap files and pro-
vides an additional labeling (in agreement with the labeling from
TDS1 ) within the attack_type field. The updated MISP object,
Fig. 4. Cooperation example: the same pcap, shared with the TIP, is analyzed
now with two consensus labels, is analyzed by an expert who
by different TDSs.
validates the threat classification. The validated event can now be
retrieved from other IDS (e.g., TDSn in the figure) and included in
their training set for improving the robustness of the underlying
consists of a stack of different kinds of sub-nets: (i) an input layer
classification model.
devoted to handling the input i.e. the relational representation of
Now let us consider the situation where the event released
the network flow obtained by extracting the features introduced
by TDS1 is classified differently (non-anomalous) by TDS2 . Again,
above; (ii) a feature-engineering block, consisting in two layers
the domain expert inspects the event with dissimilar scores and
respectively providing an extended view of the initial features
(including several non-linear functions) and a compressed rep- realizes that the event represents a false alarm. The event is then
resentation of the extended view; (iii) a number m > 1 of returned to TDS1 which can then include the validated event in its
Residual Block sub-net instances which is devoted to yielding a training set and refine the underlying model for better accuracy
flexible hierarchy of abstract data features; (iv) a final Decision and improve its false positive rate.
Layer, equipped with a sigmoid activation function which maps The above described approach resembles the well-known
any given data instance to an anomaly score. The outputs of Query-By-Committee (QbC) strategy [40], with the difference that
these models are combined by a further sub-net which can adopt here, we foresee that the expert validates both the agreement
different strategies based on trainable or non-trainable functions. (in the first situation) and the disagreement (in the second one).
EBIDS works in a continuous stream within two phases: learn- More sophisticated validation criteria can be adopted, to imple-
ing and deployment. In the former the classification model is ment different optimization objectives. For example, in order
trained based on the data collected from the MISP. Fig. 5(a) to reduce the human intervention, automatic validation can be
shows the logical flow of the learning phase: a set of classi- introduced, that confirms the agreements based on confidence
fiers are trained against Threat and Regular Traffic flows stored values and reduces human analysis to the most uncertain cases
in a local database (named PCAP DB in figure). The software based, e.g., on label entropy.
component, named DB Manager, is devoted to filling the PCAP
DB with relevant threat events extracted from the TIP and nor- Honeypot-based data enrichment. In addition to the information
mal traffic flow examples gathered from the computer network. provided by the TDSs, the framework allows for gathering fur-
Moreover, it updates the DB by sampling and discarding out-of- ther attack examples by means of a honeynet so to enrich the
date ‘‘normal traffic’’ and possibly also obsolete ‘‘threat’’ data. The knowledge base used for training of the ML-Based TDSs. In more
examples stored in the PCAP DB are first processed through a details, each deployed honeypot gathers data concerning new in-
data extraction tool (CICFlowMeter [39] in the figure) that extracts trusions in a log file (in JSON or XML format). Notably, a honeypot
statistical summaries from the pcap network information. The re- can be considered an environment where some vulnerabilities
sulting relational representation then feeds a training set, which are deliberately introduced in order to monitor and gather data
is exploited to update EBIDS. The learning phase is periodically on attacks and intrusions. Specifically, they simulate legitimate
performed to keep the model up-to-date. services which should be normally never called, therefore if a
The deployment phase exploits the learned model to detect connection with these services is established then it can be la-
anomalous behaviors and store them as MISP security events. We beled as a positive example (i.e. an attack) in the knowledge
consider a classical Network Intrusion detection scenario: the IDS
base.
analyzes network traffic extracted from a network exploiting spe-
Honeypot LOG ETL, introduced in Section 3.2, is the component
cific tools (e.g. packet capture) with the aim to monitor the devices
devoted to parsing and analyzing the honeypot log files. Basically,
connected to a network. Fig. 5(b) illustrates the components of
it allows for extracting new threats and interfacing with a MISP
the underlying architecture and the typical process: the network
flow is stored into a PCAP file and its preprocessed version is instance to make available these new attack data in the TIP. The
provided to EBIDS for the threat detection. If an anomaly (i.e. a log files directory of each honeypot is synchronized by means
possible new threat) is detected, then it is shared with the TIP of rsync with another one available on the Gateway Router. A
via the IDS Event Exporter. Both the pcap and the preprocessed TCP Dump Script periodically stores the gathered networks flows
data are stored within the event, through the pcap_file and as PCAP files. The honeypot’s PCAP shared with the distributed
anomaly_details attributes. Upon request, pcap files imported TIP can hence be used to enrich the set of positive examples in
from MISP security events can be analyzed and an updated MISP the learning phase of IDS, as shown in Fig. 3. In ORISHA medium
Security Event object is created and stored into the TIP with the to high interaction honeypots are used to capture the blackhat
additional labeling provided by EBIDS. attack behavior.
36
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Fig. 5. TDS Integration example: Learning, deployment and usage of EBIDS.

Fig. 6. An example of execution flow.

Fig. 7. Evaluation methodology.

4. Experimental analysis computer networks. Also, given this context, we do not experi-
ment with honeypots whose main role in the architecture is to
enrich the knowledge base with new (unknown) attacks.
This section illustrates a series of experiments that we con-
ducted to evaluate the effectiveness of our approach. The purpose
4.1. Methodology
is to demonstrate that the sharing features of the collaborative
platform can be fruitfully exploited to improve the accuracy of
The main idea consists in simulating a cooperative environ-
each connected peer. ORISHA is a general platform, devised to ment where different entities share anomalies detected by their
ensure flexibility and extensibility and address different types of own TDSs. This is illustrated in Fig. 7. Each TDS is trained on a
problems. Notwithstanding, in our experimentation we consider a specific subset Di of a global dataset D. The subset Di relative
specific supervised intrusion detection scenario where a number to entity i is further split in train Dtri , validation Dv ai and test
of binary classifiers are exploited for discovering intrusions in Dtei . The train is the partition that the TDS uses for training and
37
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Table 3 Specifically, on Monday, only normal flows are recorded, while

Distribution of the attacks for CICIDS dataset. the other days contain instances of different types of attacks [42],
Day Attack families Number of Percentage summarized in Table 3. In our experiments we only consid-
flows of attacks
ered samples containing attacks. Moreover, each data sample
Monday Only legitimate traffic 529,918 0.000%
is further split in training, validation and test set respectively
Tuesday FTP-Patator, SSH-Patator 445,909 3.103%
Wednesday Dos attacks, Heartbleed 692,703 36.476%
with 50%/25%/25% percentages. Each model is trained, validated
Thursday Web attacks, Infiltration 458,968 0.483% and evaluated against data samples respectively with size of
Friday Bot, DDos, Port Scan 704,245 41.025% 230k, 115k and 115k instances. For each iteration a data subset
(∼15%) of the validation set is sampled uniformly according to
the anomaly confidence, and shared through the MISP. The sam-
ples thus include both events with a high degree of uncertainty
the test is the partition where the evaluation of the TDS takes
and events which, according to the model, represent attacks with
place. The validation set has a specific role here, representing the
high probability. The automated verification, through a pool of
‘‘operational data’’ that the TDS has to inspect and that can in
IDSs in charge to make decisions, is particularly well-suited given
principle be shared through ORISHA. In practice, TDSi classifies
the large number of examples. However, this strategy can be
the whole Dv ai .
easily tuned in an operational scenario, by selecting an ad-hoc
In this scenario, Active Learning represents a natural strategy
number of the most uncertain events for the human expert.
to improve the overall accuracy of the underlying TDSs: the data
We provide an instance of EBIDS to each entity belonging to
gathered from specific TDSs are made available to domain experts
the network. In the following we will refer these detectors as
via the MISP Network, and they are responsible for verifying
TDS0 , . . . , TDSN . These instances are initialized by using the same
uncertain cases and can eventually resolve them. In fact, in Sec-
parameters described in [37]. For the architecture of the base
tion 3.3 we describe how the interaction takes place. Notice that
model,
√ 2 (i) the Extended Input layer produces the transformations
we are considering an operational scenario, where the incoming
x, x , log(x + 1) and sin(x) for every original data feature X ;
flows are unlabeled and hence uncertainty in the response can
(ii) the Embedding Layer maps its input onto a 96-dimensional
play a prominent role in deciding which flow to be analyzed by
vector space; (iii) 3 Residual Blocks are used, for a total of 6
the expert. In fact, as also described in Section 3.3, the approach
Building Blocks; (iv) in every Building Block, the Dense layer
resembles a Query-By-Committee (QbC) strategy, with the only
consists of 32 neurons, and the dropout rate is set to 0.01. Finally,
difference that here we omit to describe the criteria to select and
a Mixture-Of-Expert (MOE) [43] is used to combine the different
submit the most uncertain tuples.
scores provided by the base models. To assess the quality of
Models interact with the system in three ways: by submitting
ORISHA approach (no matter what detection model adopted) we
events, by labeling events submitted by other models, and by
conducted a suite of experiments by varying the ML-techniques
collecting validated events. The validation process is external and
used by each TDS.
delegated to a sovereign expert, and it can be accomplished either
automatically (when possible) or manually by human interven-
4.3. Evaluation metrics
tion. Of course, human intervention can include forensic analysis
and it should be minimized, by devising specific policies based
Recall and Precision are typically used as metrics to estimate
on the degree of disagreement of the responses of the various
the detection capability of Threat Detection Systems since they
models.
provide a measurement on their accuracy in identifying up-
Thus, for a subset of the malignant flows (i.e., those flows
coming attacks and to avoiding false alarms. Specifically, the
classified as anomalous with confidence > .5) some MISP events
former one represents the percentage of real attacks classified
are created. These events are inspected by the oracle, with the
as such whereas the second one is the fraction of real attacks
objective of revising their classification. Based on this revision,
correctly discovered by the model among the number of is-
the newly labeled sample can hence be shared to all the peers
sued alarms. These metrics are frequently combined into the
and added to Dtr1 , . . . , DtrN and the underlying ML models are
F-Measure, computed as their harmonic mean and summariz-
retrained.
ing the overall performances of the model. Although F-Measure
We exploit ORISHA to implement a simple oracle strategy that
represents a state-of-the-art measure for classification tasks, it
exploits a majority voting. In practice, each event shared through
is not well suited when the classes are strongly unbalanced.
the MISP is classified by all the remaining entities and a final class
Therefore, we reported two further metrics, respectively named
label (stored within the attack_type SEO field) is obtained by
AUC and AUC-PR, are widely used to estimate the performances
combining their scores. This approach departs from the standard
of a classification model in presence of skewness on the data
QbC strategy, but it allows us to evaluate the overall performance
(e.g., NIDSs logs).
even in a totally automated situation, where a certain degree
The AUC measures the area under the ROC curve, computed
of uncertainty is still present. In our experiments we tested a
comparing the False Positive Rate (i.e., the percentage of risen
number of data sharing iterations equal to the number of entities,
false alarms) and the True Positive Rate (i.e., the Recall). The AUC-
which was set to 5 models instantiated.
PR (i.e. the area under the Precision–Recall curve) is computed by
plotting precision and recall for different class probability values.
4.2. Dataset and parameters
AUC-PR is typically used in each case where the classification of
the positive class is more relevant than the negative one, since
The tests were conducted on the CICIDS2017 dataset,5 [41]
AUC could overestimate the quality of the model.
a well-known IDS benchmark provided by Canadian Institute of
Cybersecurity, containing several up-to-date cyberattacks.
4.4. Experimental results
The dataset is partitioned into five samples (one for day),
from Monday to Friday and includes six attack profiles based on
A first suite of experiments aims at comparing the behavior
the last updated list (up to 2017) of common attack families.
of different TDSs when they are feed with further information
shared by other entities. After an initialization phase where each
5 https://www.unb.ca/cic/datasets/ids-2017.html. TDS has available only a limited sample of data (locally stored),
38
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Table 4 Table 5
Cooperative learning performances (by choosing the better model at each Cooperative learning performances (by choosing the better model at each
iteration). The values in bold represent an improvement of the model at the iteration). The values in bold represent an improvement of the model at the
correspondent iteration. EBIDS is used as TDS. correspondent iteration. A Bagging-based detector is used as TDS.
Iteration Model AUC AUC-PR F-Measure Iteration Model AUC AUC-PR F-Measure
TDS0 0.999 0.997 0.982 TDS0 0.540 0.469 0.520
TDS1 0.832 0.740 0.689 TDS1 0.879 0.894 0.910
Initialization TDS2 0.884 0.821 0.784 Initialization TDS2 0.568 0.616 0.560
TDS3 1.000 1.000 0.996 TDS3 0.640 0.723 0.670
TDS4 0.932 0.928 0.901 TDS4 0.703 0.773 0.750
Av erage 0.929 0.897 0.870 Av erage 0.666 0.695 0.679
TDS0 0.999 0.997 0.982 TDS0 0.728 0.778 0.772
TDS1 0.990 0.973 0.910 TDS1 0.879 0.894 0.910
0 TDS2 0.884 0.821 0.784 0 TDS2 0.568 0.616 0.560
TDS3 1.000 1.000 0.996 TDS3 0.694 0.768 0.740
TDS4 0.932 0.928 0.901 TDS4 0.703 0.773 0.750
Av erage 0.961 0.944 0.915 Av erage 0.715 0.766 0.744
TDS0 0.999 0.997 0.982 TDS0 0.728 0.778 0.772
TDS1 0.990 0.973 0.910 TDS1 0.879 0.894 0.910
1 TDS2 0.884 0.821 0.784 1 TDS2 0.573 0.640 0.567
TDS3 1.000 1.000 0.996 TDS3 0.694 0.768 0.740
TDS4 0.932 0.928 0.901 TDS4 0.703 0.773 0.750
Av erage 0.961 0.944 0.915 Av erage 0.715 0.771 0.745
TDS0 0.999 0.997 0.982 TDS0 0.792 0.808 0.829
TDS1 0.990 0.973 0.910 TDS1 0.879 0.894 0.910
2 TDS2 0.977 0.940 0.936 2 TDS2 0.648 0.721 0.677
TDS3 1.000 1.000 0.996 TDS3 0.712 0.781 0.756
TDS4 0.932 0.928 0.901 TDS4 0.703 0.773 0.750
Av erage 0.980 0.968 0.945 Av erage 0.747 0.795 0.783
TDS0 0.999 0.997 0.982 TDS0 0.792 0.808 0.829
TDS1 0.999 0.998 0.990 TDS1 0.879 0.894 0.910
3 TDS2 0.999 0.998 0.988 3 TDS2 0.651 0.733 0.682
TDS3 1.000 1.000 0.996 TDS3 0.712 0.781 0.756
TDS4 0.932 0.928 0.901 TDS4 0.703 0.773 0.750
Av erage 0.986 0.984 0.971 Av erage 0.748 0.798 0.784
TDS0 0.999 0.997 0.982 TDS0 0.792 0.808 0.829
TDS1 0.999 0.998 0.990 TDS1 0.879 0.894 0.910
4 TDS2 0.999 0.998 0.988 4 TDS2 0.694 0.770 0.735
TDS3 1.000 1.000 0.996 TDS3 0.714 0.782 0.758
TDS4 0.932 0.928 0.901 TDS4 0.703 0.773 0.750
Av erage 0.986 0.984 0.971 Av erage 0.757 0.805 0.795

the first entity E0 begins to share data sampled from the valida- In a further set of experiments, we modify the underlying
tion set on the basis of strategy described in Section 4.1. Then, active learning strategy with a twofold aim. First, we aim at
this process is iteratively performed by each entity Ei belonging measuring whether a minimal amount of manual labeling can still
to the MISP network. provide benefits. Second, we would like to compare such benefits
In Table 4, the results of this analysis are reported. First, we to those of the automated approach based on majority voting. In
can see that after 5 iterations the averaged performances of the these experiments, a QbC protocol is actually implemented for
TDSs belonging to the network are considerably increased in each iteration, as follows. First of all, we compute the entropy
terms of all evaluation metrics, in particular the initial value of score for each of the events submitted to the MISP and classified
the averaged F-Measure 0.870 increases to 0.971. Notably, some by the pool of IDS. Then, we select the 100 events with the
underperforming TDSs (TDS1 and TDS2 , which exhibit poor predic- highest entropy and add them (with the actual label) to all IDS.
tive performances in the initialization step) benefit substantially This simulates the response by an external expert. The Table 7
from the data sharing and enrichment. In fact, the collaborative summarizes the results of the evaluation. As we can see, the QbC
framework allows for a substantial improvement of their detec- approach still guarantees substantial performance gain for each
tion capabilities, enabling the achievement of results comparable TDS. Once again, we can see the benefits in terms of predictive
to the ones of the other TDSs. accuracy due to the adoption of the ORISHA approach, although
As mentioned above, the same analysis has been conducted the improvements are lesser than the ones obtained with the fully
by changing the underlying TDS instances. In particular, the ad- automated solution shown in Table 4.
ditional experiments consider two further ML-based IDSs relying In the experiments showed so far, each TDS is only aware
respectively on bagging and boosting ensembles. In both mod- of the local training set. During the iterations, the newly added
els, a decision tree is used as a weak learner classifier and all examples are labeled according to the majority vote (and hence
the models are set up with default parameters. The results are no new labeled dataset is disclosed). We compared this situation
summarized in Tables 5 and 6. Once again, we can observe an with a flat situation where, for each model, the training set is ex-
overall improvement of the TDS belonging to the network after 5 tended with all the tuples from the local validation set. The results
iterations and, in particular, the weaker models (i.e., the models are shown in Fig. 8 . We can see that the accuracy is comparable,
with poor performances in the initialization stage) benefit more even though the models trained through ORISHA only exploit the
from threat sharing. training set and (unlabeled) portions of the validations sets.
39
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Table 6
Cooperative learning performances (by choosing the better model at each
iteration). The values in bold represent an improvement of the model at the
correspondent iteration. A Boosting-based detector is used as TDS.
Iteration Model AUC AUC-PR F-Measure
TDS0 0.690 0.742 0.734
TDS1 0.797 0.850 0.842
Initialization TDS2 0.730 0.791 0.800
TDS3 0.680 0.756 0.718
TDS4 0.702 0.768 0.744
Av erage 0.721 0.781 0.763
TDS0 0.690 0.742 0.734
TDS1 0.843 0.880 0.883
Fig. 8. Comparison between the model performances trained by using ORISHA 0 TDS2 0.730 0.791 0.800
AL-Based approach and the performance of the ones trained on the union of the TDS3 0.731 0.795 0.776
training and validation sets (full view). TDS4 0.702 0.768 0.744
Av erage 0.740 0.795 0.782
TDS0 0.776 0.829 0.822
TDS1 0.843 0.880 0.883
1 TDS2 0.730 0.791 0.800
TDS3 0.736 0.799 0.782
TDS4 0.702 0.768 0.744
Av erage 0.758 0.814 0.801
TDS0 0.849 0.885 0.888
TDS1 0.843 0.880 0.883
2 TDS2 0.730 0.791 0.800
TDS3 0.737 0.801 0.783
TDS4 0.702 0.768 0.744
Av erage 0.772 0.825 0.815
TDS0 0.849 0.885 0.888
TDS1 0.843 0.880 0.883
3 TDS2 0.730 0.791 0.800
TDS3 0.737 0.801 0.783
Fig. 9. Average learning time for each iteration. TDS4 0.702 0.768 0.744
Av erage 0.772 0.825 0.815
TDS0 0.852 0.887 0.890
As regards the computation times of ORISHA for each iteration, TDS1 0.843 0.880 0.883
4 TDS2 0.730 0.791 0.800
they mainly depends on the learning times of the single models
TDS3 0.737 0.801 0.783
and the threat sharing process. The training times computed TDS4 0.702 0.768 0.744
for Table 4 are essentially constant, as shown in Fig. 9. The Av erage 0.773 0.826 0.816
overhead for storing or retrieving a single event depends on
the specific network latency and thus it does not influence the
overall performance, which instead depends on the number of
entities and the maximum number of events shared by each An experimental evaluation, conducted on a well-known IDS
entity. As mentioned above, a distributed architecture based on benchmark, demonstrates how merging data sharing and active
connected MISP instances can scale up the overall process by learning strategies can improve the detection capabilities of the
enabling parallel processing. MISP network allowing to discover undetected attacks.
Novelty and limitations. Although the design principles of ORISHA
5. Conclusions are general, the current implementation focuses on sharing digital
evidence among Intrusion Detection Systems. Moreover, the idea
of sharing data to fit different models in a decentralized fashion
Security intelligence and data analytics techniques can be used
is adopted by other emerging frameworks. As an example, Fed-
to strengthen the capabilities of cybersecurity applications in
erated Learning [44] (FL) is a family of methods and techniques
various vertical domains and use cases. These techniques can
enabling the learning of ML models in decentralized way, usually
largely benefit from mechanisms to share digital evidence and en-
performed on edge devices or servers storing data. By fostering
sure interoperability. The current Threat Intelligence platforms do the cooperation of different actors, FL allows for learning robust
not provide native mechanisms to incorporate such mechanisms, and effective predictive models without the need to share data in
especially when data-driven and AI powered threat detection a centralized way. However, many FL frameworks employ mul-
systems are involved. ORISHA is a first attempt to enable a shar- tiple rounds of communication between devices and the central
ing and interoperability protocol among such components, based server, which increases the communication overheads. In partic-
solely on a data-oriented approach. This simple, flexible strategy ular, while FL is mainly devised to distribute the computation on
and data formats for collaborative threat intelligence can trigger more devices so to make scalable and feasible the learning even
specific advantages: Improving the alert effectiveness by reduc- when the computational resources are not sufficient to handle the
ing the amount of false positive alerts; better contextualizing massive amount of yielded data, the main objective of ORISHA
threat data with the contribution of multiple actors; boosting is to share IoCs that can be used by domain experts to take
trust among producers and consumers of threat intelligence in- aware decisions and enable effective countermeasures, and also
formation; and strengthening the robustness of machine learning by automatized ML-Based systems to improve their predictive
and deep learning models adopted by security applications. performances.
40
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Table 7 sharing process of ORISHA in terms of privacy, trust and relia-

Full active learning approach. A Query By Committee approach exploiting bility. All these complementary services (e.g., privacy preserving,
entropy-based criterion is used to select (∼100) uncertain tuples for the expert
verification. EBIDS is used as TDS.
risk assessment, etc.), which can be included and make ORISHA
Iteration Model AUC AUC-PR F-Measure
fully operational, are already represented in Fig. 1 with the name
‘‘Security Service Providers/Consumers’’.
TDS0 0.999 0.997 0.982
TDS1 0.832 0.740 0.689 Future works. We plan to investigate the integration of additional
Initialization TDS2 0.884 0.821 0.784
TDS3 1.000 1.000 0.996
data-aware threat analysis tools (e.g., Malware detection systems
TDS4 0.932 0.928 0.901 which do not necessarily rely on the analysis of network flow).
Av erage 0.929 0.897 0.870 Also, mechanisms for incorporating and sharing higher-level in-
formation, such as machine learning models, can further boost
TDS0 0.999 0.997 0.988
TDS1 0.883 0.782 0.719 security intelligence capabilities. Some initial attempts have been
0 TDS2 0.972 0.944 0.911 proposed [45]. However, the lack of a common representation
TDS3 1.000 1.000 0.996 format, make this task challenging and worth further studies.
TDS4 0.934 0.921 0.902 Finally, the adoption of Deep Active and Deep Reinforcement
Av erage 0.958 0.929 0.902 Learning requires further investigation since can improve the per-
TDS0 0.999 0.997 0.988 formances by simultaneously reducing the expensive interaction
TDS1 0.883 0.782 0.719 with the domain expert.
1 TDS2 0.972 0.944 0.911
TDS3 1.000 1.000 0.996
TDS4 0.934 0.921 0.902 CRediT authorship contribution statement
Av erage 0.958 0.929 0.902
Massimo Guarascio: Conceptualization, Methodology, In-
TDS0 0.999 0.997 0.988
TDS1 0.856 0.769 0.765 vestigation, Writing – original draft, Writing – review &
2 TDS2 0.972 0.944 0.911 editing. Nunziato Cassavia: Conceptualization, Methodology,
TDS3 1.000 1.000 0.996 Investigation, Writing – original draft, Software. Francesco Ser-
TDS4 0.934 0.921 0.902
gio Pisani: Conceptualization, Investigation, Writing – review
Av erage 0.952 0.926 0.912 & editing. Giuseppe Manco: Conceptualization, Methodology,
TDS0 0.999 0.997 0.988 Investigation, Writing - original draft, Writing – review &
TDS1 0.944 0.908 0.847 editing, Supervision.
3 TDS2 0.972 0.944 0.911
TDS3 1.000 1.000 0.996
TDS4 0.934 0.921 0.902 Declaration of competing interest
Av erage 0.970 0.954 0.928
TDS0 0.999 0.997 0.988
The authors declare that they have no known competing finan-
TDS1 0.944 0.908 0.847 cial interests or personal relationships that could have appeared
4 TDS2 0.972 0.944 0.911 to influence the work reported in this paper.
TDS3 1.000 1.000 0.996 Acknowledgments
TDS4 0.934 0.921 0.902
Av erage 0.970 0.954 0.929 This work has been partially supported by EU H2020-SU-ICT-
03-2018 Project No. 830929 CyberSec4Europe (cybersec4europe.
eu).
A different approach to exploit gathered knowledge is adopted
by Transfer Learning (TL). In this case the idea consists in re- References
using deep models, learned on a different domain or for tackling
[1] ENISA, Enisa threat landscape 2020 - list of top 15 threats, 2020,
a different task, by fine-tuning them for addressing the own https://www.enisa.europa.eu/publications/enisa-threat-landscape-2020-
learning problem. This approach perfectly marries the ORISHA list-of-top-15-threats.
philosophy since the models learned by the organizations be- [2] Interpol, Covid-19 cybercrime analysis report, 2020, https://tinyurl.com/
longing to the MISP network are themselves relevant IoCs. As an 6wek2rk.
[3] M.D.T.I. Team, Exploiting a crisis: How cybercriminals behaved during the
example, in [45], an interesting study illustrates a solution based
outbreak, 2020, https://tinyurl.com/cybercrime-during-outbreak.
on the integration of MISP, TheHIVE and Cortex devised to enable [4] C.S. Johnson, M.L. Badger, D. Waltermire, J. Snyder, C. Skorupka, Guide to
the sharing of ML Models in a MISP network. cyber threat information sharing, NIST Special Publ. 80 (2016) 0–150.
[5] S. Brown, J. Gommers, O. Serrano, From cyber security information sharing
Privacy aspects. Trust and privacy are two major issues in Threat to threat management, in: Proceedings of the 2nd ACM Workshop on
Information Sharing. Due to the sensitive nature of the involved Information Sharing and Collaborative Security, 2015, pp. 43–49, http:
data, targets of cyberthreats are not always inclined to share this //dx.doi.org/10.1145/2808128.2808133.
[6] L. Dandurand, O.S. Serrano, Towards improved cyber security information
information unless they are obliged to be compliant with manda-
sharing, in: 2013 5th International Conference on Cyber Conflict (CYCON
tory incident reporting regulations. Although these aspects are 2013), 2013, pp. 1–16.
not the main focus of the work, these issues can be addressed by [7] A. Zibak, A. Simpson, Cyber threat information sharing: Perceived benefits
adopting recent approaches proposed in literature, which can be and barriers, in: Proceedings of the 14th International Conference on
Availability, Reliability and Security, Association for Computing Machinery,
easily integrated with MISP (and hence be exploited by ORISHA).
2019, http://dx.doi.org/10.1145/3339252.3340528.
For example, in [46,47] the authors illustrate TATIS (Trustworthy [8] D.W. Chadwick, W. Fan, G. Costantino, R. de Lemos, F. Di Cerbo, I.
APIs for Threat Intelligence Sharing with User Managed Access Herwono, M. Manea, P. Mori, A. Sajjad, X.S. Wang, A cloud–edge based data
(UMA) and CP-ABE), a solution for fine-grained access control to security architecture for sharing and analysing cyber threat information,
protect threat intelligence APIs using UMA and Ciphertext-Policy Future Gener. Comput. Syst. 102 (2020) 710–722, http://dx.doi.org/10.
1016/j.future.2019.06.026.
Attribute-Based Encryption (CP-ABE). TATIS services are made [9] S. Qamar, Z. Anwar, M.A. Rahman, E. Al-Shaer, B.T. Chu, Data-driven
available under the form of REST API and can be invoked by each analytics for cyber-threat intelligence and information sharing, Comput.
organization belonging to the sharing network to strengthen the Secur. 67 (2017) 35–58, http://dx.doi.org/10.1016/j.cose.2017.02.005.

41
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

[10] T.D. Wagner, K. Mahbub, E. Palomar, A.E. Abdallah, Cyber threat intelli- [35] G. González-Granadillo, M. Faiella, I. Medeiros, R. Azevedo, S. González-
gence sharing: Survey and research directions, Comput. Secur. 87 (2019) Zarzosa, Etip: An enriched threat intelligence platform for improving osint
101589, http://dx.doi.org/10.1016/j.cose.2019.101589. correlation, analysis, visualization and sharing capabilities, J. Inform. Secur.
[11] V. Mavroeidis, S. Bromander, Cyber threat intelligence model: An eval- Appl. 58 (2021) 102715, http://dx.doi.org/10.1016/j.jisa.2020.102715.
uation of taxonomies, sharing standards, and ontologies within cyber [36] W. Tounsi, H. Rais, A survey on technical threat intelligence in the age
threat intelligence, in: 2017 European Intelligence and Security Informatics of sophisticated cyber attacks, Comput. Secur. 72 (2018) 212–233, http:
Conference (EISIC), 2017, pp. 91–98, http://dx.doi.org/10.1109/EISIC.2017. //dx.doi.org/10.1016/j.cose.2017.09.001.
20. [37] F. Folino, G. Folino, M. Guarascio, F. Pisani, L. Pontieri, On learning effective
[12] C. Sauerwein, C. Sillaber, A. Mussmann, R. Breu, Threat intelligence shar- ensembles of deep neural networks for intrusion detection, Inf. Fusion 72
ing platforms: An exploratory study of software vendors and research (2021) 48–69, http://dx.doi.org/10.1016/j.inffus.2021.02.007.
perspectives, Wirtschaftsinformatik Angew. Inform. (2017).
[38] M.H. Bhuyan, D.K. Bhattacharyya, J.K. Kalita, Network anomaly detec-
[13] M.C. Libicki, Sharing Information about Threats is not a Cybersecurity
tion: methods, systems and tools, IEEE Commun. Surv. Tutor. 16 (2013)
Panacea, RAND Corporation, Santa Monica, CA, 2015.
303–336.
[14] B. Jordan, R. Piazza, T. Darley, Stix™version 2.1 committee specification 01,
2020. [39] A.H. Lashkari, G.D. Gil, M.S.I. Mamun, A.A. Ghorbani, Characterization
of tor traffic using time based features, in: Proceedings of the 3rd
[15] T. Darley, I. Kirillov, R. Piazza, D. Beck, Cybox™version 2.1.1. part 01:
International Conference on Information Systems Security and Privacy
Overview - committee specification draft 01/ public review draft 01, 2016.
- Volume 1: ICISSP, SciTePress., 2017, pp. 253–262, http://dx.doi.org/10.
[16] R. Danyliw, J. Meijer, Y. Demchenko, The incident object description
5220/0006105602530262.
exchange format, in: RFC 5070 (Proposed Standard), 2007.
[17] T. Darley, I. Kirillov, R. Piazza, D. Beck, Taxii™version 2.1 committee [40] D.A. Cohn, L.E. Atlas, R.E. Ladner, Improving generalization with ac-
specification 01, 2020. tive learning, Mach. Learn. 15 (1994) 201–221, http://dx.doi.org/10.1007/
[18] ENISA, Exploring the opportunities and limitations of current threat BF00993277.
intelligence platforms, 2017. [41] I. Sharafaldin, A. Habibi Lashkari, A.A. Ghorbani, Toward generating a
[19] C. Wagner, A. Dulaunoy, G. Wagener, A. Iklody, Misp: The design and new intrusion detection dataset and intrusion traffic characterization, in:
implementation of a collaborative threat intelligence sharing platform, in: Proceedings of the 4th International Conference on Information Systems
Proceedings of the 2016 ACM on Workshop on Information Sharing and Security and Privacy - ICISSP, INSTICC, SciTePress, 2018, pp. 108–116,
Collaborative Security, Association for Computing Machinery, 2016, pp. http://dx.doi.org/10.5220/0006639801080116.
49–56, http://dx.doi.org/10.1145/2994539.2994542. [42] R. Panigrahi, S. Borah, A detailed analysis of CICIDS2017 dataset for
[20] M. Goffin, Crits: Collaborative research into threats, 2014, https://crits. designing intrusion detection systems, Int. J. Eng. Technol. 7 (2018)
github.io/. [Online]. 479–482.
[21] M. Bringer, C. Chelmecki, H. Fujinoki, A survey: Recent advances and future [43] S. Masoudnia, R. Ebrahimpour, Mixture of experts: a literature survey, Artif.
trends in honeypot research, Int. J. Comput. Netw. Inform. Secur. (4) (2012) Intell. Rev. 42 (2014) 275–293.
http://dx.doi.org/10.5815/ijcnis.2012.10.07.
[44] T. Li, A.K. Sahu, A. Talwalkar, V. Smith, Federated learning: Challenges,
[22] M.F. Razali, M.N. Razali, F.Z. Mansor, G. Muruti, N. Jamil, Iot honeypot:
methods, and future directions, IEEE Signal Process. Mag. 37 (2020) 50–60,
A review from researcher’s perspective, in: 2018 IEEE Conference on
http://dx.doi.org/10.1109/MSP.2020.2975749.
Application, Information and Network Security (AINS), 2018, pp. 93–98,
[45] D. Preuveneers, W. Joosen, Sharing machine learning models as indicators
http://dx.doi.org/10.1109/AINS.2018.8631494.
of compromise for cyber threat intelligence, J. Cybersecur. Privacy 1
[23] J. Franco, A. Aris, B. Canberk, A. Selcuk Uluagac, A survey of honeypots
(2021) 140–163, http://dx.doi.org/10.3390/jcp1010008, https://www.mdpi.
and honeynets for internet of things, industrial internet of things, and
com/2624-800X/1/1/8.
cyber–physical systems, 2021, CoRR abs/2108.02287.
[24] E. Vasilomanolakis, S. Karuppayah, M. Mühlhäuser, M. Fischer, Taxonomy [46] D. Preuveneers, W. Joosen, Tatis: Trustworthy apis for threat intelligence
and survey of collaborative intrusion detection, ACM Comput. Surv. (47) sharing with uma and cp-abe, in: A. Benzekri, M. Barbeau, G. Gong,
(2015) http://dx.doi.org/10.1145/2716260. R. Laborde, J. Garcia-Alfaro (Eds.), Foundations and Practice of Security,
[25] H. Artail, H. Safa, M. Sraj, I. Kuwatly, Z. Al-Masri, A hybrid honeypot Springer International Publishing, Cham, 2020, pp. 172–188.
framework for improving intrusion detection systems in protecting orga- [47] D. Preuveneers, W. Joosen, J.B. Bernabé, A.F. Skarmeta, Distributed security
nizational networks, Comput. Secur. 25 (2006) 274–288, http://dx.doi.org/ framework for reliable threat intelligence sharing, Secur. Commun. Netw.
10.1016/j.cose.2006.02.009. (2020) 8833765:1–8833765:15, http://dx.doi.org/10.1155/2020/8833765.
[26] M. Baykara, R. Das, A novel honeypot based security approach for real-
time intrusion detection and prevention systems, J. Inform. Secur. Appl. Massimo Guarascio holds a Ph.D. in Systems and
41 (2018) 103–116, http://dx.doi.org/10.1016/j.jisa.2018.06.004. Computer Science Engineering and a master’s degree
[27] B. Khosravifar, J. Bentahar, An experience improving intrusion detection in Computer Science Engineering, both from the Uni-
systems false alarm ratio by using honeypot, in: 22nd International versity of Calabria. He is currently researcher at the
Conference on Advanced Information Networking and Applications (Aina Institute for High Performance Computing and Net-
2008), 2008, pp. 997–1004, http://dx.doi.org/10.1109/AINA.2008.44. working of the National Research Council (ICAR-CNR)
[28] S. Sibi Chakkaravarthy, D. Sangeetha, M.V. Cruz, V. Vaidehi, B. Raman, and shareholder of OKT s.r.l., a spin-off of University
Design of intrusion detection honeypot using social leopard algorithm of Calabria. He co-authored over 50 papers published
to detect iot ransomware attacks, IEEE Access 8 (2020) 169944–169956, in international conference proceedings, chapters and
http://dx.doi.org/10.1109/ACCESS.2020.3023764. journals. His research mainly focuses on machine
[29] P. Ren, Y. Xiao, X. Chang, P. Huang, Z. Li, B. Gupta, X. Chen, X. Wang, learning, anomaly detection and explanation, process
A survey of deep active learning, ACM Comput. Surv. (2021) 54, http: mining, data analytics methods for geosciences and remote sensing, knowledge
//dx.doi.org/10.1145/3472291. discovery and data mining for cybersecurity and fraud detection. He has
[30] B. Settles, Active Learning. Synthesis Lectures on Artificial Intelligence and participated in European and national research projects concerning machine
learning and cybersecurity.
Machine Learning, Morgan & Claypool Publishers, 2012, http://dx.doi.org/
10.2200/S00429ED1V01Y201207AIM018.
[31] A. Shahraki, M. Abbasi, A. Taherkordi, A.D. Jurcut, Active learning for
Nunziato Cassavia holds a Ph.D. and an M.Sc. in
network traffic classification: A technical survey, 2021, arXiv:2106.06933.
Computer Engineering from the University of Calabria,
[32] Q.V. Dang, Active learning for intrusion detection systems, in: 2020 RIVF
Italy. Currently, he is a research fellow at the Insti-
International Conference on Computing and Communication Technologies
tute for High Performance Computing and Networking
(RIVF), 2020, pp. 1–3, http://dx.doi.org/10.1109/RIVF48685.2020.9140751.
(ICAR-CNR) of the National Research Council (CNR) in
[33] K. Yang, J. Ren, Y. Zhu, W. Zhang, Active learning for wireless iot intrusion Italy. His research interests include Cybersecurity, Big
detection, IEEE Wirel. Commun. 25 (2018) 19–25, http://dx.doi.org/10. Data, Distributed Computing System, Data Warehouse
1109/MWC.2017.1800079. and Relational and NoSQL Databases. He co-authored
[34] M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, Application of deep several papers published in international conference
reinforcement learning to intrusion detection for supervised problems, proceedings, chapters and journals and participated in
Expert Syst. Appl. 141 (2020) 112963, http://dx.doi.org/10.1016/j.eswa. European and national research projects concerning Big
2019.112963. Data and Cybersecurity.

42
M. Guarascio, N. Cassavia, F.S. Pisani et al. Future Generation Computer Systems 135 (2022) 30–43

Francesco Sergio Pisani holds a Ph.D. and an M.Sc. in Giuseppe Manco graduated summa cum laude in com-
Computer Engineering from the University of Calabria, puter science and received a Ph.D. degree in computer
Italy. Currently, he is a research fellow at the Insti- science from the University of Pisa. He is currently
tute for High Performance Computing and Networking a Director of Research at the Institute for High Per-
(ICAR-CNR) of the Italian National Research Council. His formance Computing and Networking (ICAR-CNR) of
research mainly focuses on machine and deep learning, the National Research Council of Italy and a contract
recommender system, ensemble learning, knowledge professor at University of Calabria, Italy. His current re-
discovery and data mining for cybersecurity, computer search interests include knowledge discovery and data
vision, anomaly detection. Currently he is involved mining, Recommender Systems and Social Network
in national projects concerning machine learning and Analysis, Deep Learning. He has been the coordinator
cybersecurity, and on an industrial project to build an of several national and international research projects.
advanced platform for customer engagement e satisfaction. He has been serving in the program committee of several international/national
conferences, including: IJCAI, AAAI, IEEE ICDM, ECMLPKDD, SIAM SDM, PAKDD.
He served as a program co-chair of ECML-PKDD 2016. He is serving as an
associate editor for the Journal of Intelligent Information Systems, Knowledge
and Information Systems and Machine Learning Journal.

43