Computer Networks 193 (2021) 108008

Contents lists available at ScienceDirect

Computer Networks
journal homepage: www.elsevier.com/locate/comnet

SPEAR SIEM: A Security Information and Event Management system for the
Smart Grid
Panagiotis Radoglou-Grammatikis a , Panagiotis Sarigiannidis a ,∗, Eider Iturbe b , Erkuden Rios b ,
Saturnino Martinez b , Antonios Sarigiannidis c , Georgios Eftathopoulos d , Yannis Spyridis d ,
Achilleas Sesis d , Nikolaos Vakakis e , Dimitrios Tzovaras e , Emmanouil Kafetzakis f ,
Ioannis Giannoulakis f , Michalis Tzifas f , Alkiviadis Giannakoulias g , Michail Angelopoulos h,i ,
Francisco Ramos j
a
Department of Electrical and Computer Engineering, University of Western Macedonia, Kozani, Greece
b
TECNALIA, Basque Research and Technology Alliance (BRTA), Derio, Spain
c
Sidroco Holdings Ltd, Nicosia, Petraki Giallourou 22, Office 11 1077, Cyprus
d 0INF, Imperial Offices, London, E6 2JG, UK
e Center for Research and Technology Hellas/Information Technologies Institute, 6th km Charilaou-Thermi Road, Thessaloniki, Greece
f Eight Bells Ltd, Agias paraskevis 23, P.C. 2002, Strovolos, Nicosia, Cyprus
g
European Dynamics, 12, Jean Engling str., L-1466, Luxembourg
h
University of Piraeus, Greece
i
Testing Research & Standards Center/Public Power Corporation SA, Leontariou 9, Kantza, Athens, Attica, 15351, Greece
j
Schneider Electric, Charles Darwin s/n, Edificio Bogaris, Sevilla, 41092, Spain

ARTICLE INFO ABSTRACT

Keywords: The technological leap of smart technologies has brought the conventional electrical grid in a new digital
Anomaly detection era called Smart Grid (SG), providing multiple benefits, such as two-way communication, pervasive control
Cybersecurity and self-healing. However, this new reality generates significant cybersecurity risks due to the heterogeneous
Deep learning
and insecure nature of SG. In particular, SG relies on legacy communication protocols that have not been
Intrusion detection
implemented having cybersecurity in mind. Moreover, the advent of the Internet of Things (IoT) creates
Machine learning
SCADA
severe cybersecurity challenges. The Security Information and Event Management (SIEM) systems constitute
Security Information and Event Management an emerging technology in the cybersecurity area, having the capability to detect, normalise and correlate
Smart Grid a vast amount of security events. They can orchestrate the entire security of a smart ecosystem, such as SG.
Nevertheless, the current SIEM systems do not take into account the unique SG peculiarities and characteristics
like the legacy communication protocols. In this paper, we present the Secure and PrivatE smArt gRid (SPEAR)
SIEM, which focuses on SG. The main contribution of our work is the design and implementation of a SIEM
system capable of detecting, normalising and correlating cyberattacks and anomalies against a plethora of SG
application-layer protocols. It is noteworthy that the detection performance of the SPEAR SIEM is demonstrated
with real data originating from four real SG use case (a) hydropower plant, (b) substation, (c) power plant
and (d) smart home.

1. Introduction such as generation diversification, demand response and the optimal

management of the existing resources. In particular, the point of con-
The next-generation electrical grid, also known as Smart Grid (SG), vergence between electrical engineering and the Internet of Things
intends to address multiple challenges of the conventional model, (IoT) creates an intelligent layer over the current model, which allows

∗ Corresponding author.
E-mail addresses: [email protected] (P. Radoglou-Grammatikis), [email protected] (P. Sarigiannidis), [email protected] (E. Iturbe),
[email protected] (E. Rios), [email protected] (S. Martinez), [email protected] (A. Sarigiannidis), [email protected]
(G. Eftathopoulos), [email protected] (Y. Spyridis), [email protected] (A. Sesis), [email protected] (N. Vakakis), [email protected] (D. Tzovaras),
[email protected] (E. Kafetzakis), [email protected] (I. Giannoulakis), [email protected] (M. Tzifas),
[email protected] (A. Giannakoulias), [email protected] (M. Angelopoulos), [email protected] (F. Ramos).

https://doi.org/10.1016/j.comnet.2021.108008
Received 15 September 2020; Received in revised form 25 January 2021; Accepted 11 March 2021
Available online 5 April 2021
1389-1286/© 2021 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

the development of appropriate business applications offering pervasive • Implementing a visual-based detection mechanism through
control, self-monitoring and self-healing [1]. However, this transition ML/DL dimensionality reduction techniques: Through VIDS,
to the SG encloses significant cybersecurity risks that can lead to disas- the security administrator can identify potential, undetected se-
trous consequences [2]. Characteristic examples are the BlackEnergy3 curity issues.
(2015) and Crashoverride (2016) Advanced Persistent Threats (APTs) • Implementing a reputation mechanism reflecting the trust
that caused extensive blackouts in Ukraine [3]. The necessary presence value of each SG asset: GTM can calculate the reputation value
of legacy systems, such as Supervisory Control and Data Acquisition of each SG asset based on the security events and alerts received.
(SCADA)/Industrial Control Systems (ICS) and the advent of IoT in- • Developing two novel Deep Neural Networks (DNNs), namely
crease the attack surface of SG [4]. On the one side, SCADA/ICS SPEAR Stacked Denoising Autoencoder (SDAE) and Payload
use insecure communication protocols, like Modbus and IEC 60870-5- Text CNN Classifier: The proposed DNNs are part of BDAC,
104 [5] that allow the cyberattackers to perform various cyberattacks. detecting particular cyberattacks and anomalies, respectively.
On the other side, IoT generates new cybersecurity concerns [6]. First,
• Evaluating a plethora of ML/DL methods for detecting vari-
IoT relies on the Internet model, which is vulnerable by itself. Second,
ous cyberattacks in four SG use cases: The various ML and DL
the vast amount of the IoT data, such as the smart metering data
methods of BDAC and VIDS are evaluated in four SG use cases: (a)
constitutes an attractive target for potential cyberattackers. Finally, the
hydropower plant, (b) substation, (c) power plant and (d) smart
capability of the various objects to interact with each other without any
home.
human intervention increases the privacy concerns.
Taking into account the critical cybersecurity issues of SG, both The rest of this paper is organised as follows. Section 2 presents
academia and industry have investigated possible countermeasures.
relevant works. Section 3 is devoted to the architecture of SPEAR SIEM.
First, the IEC 62351 standard has defined a set of security controls
Section 4 presents the evaluation analysis. Finally, Section 5 concludes
and guidelines based mainly on existing authentication and autho-
this paper. It is noteworthy that SPEAR SIEM was implemented under
risation technologies [7–9]. Moreover, the Security Information and
the H2020 SPEAR project [14].
Event Management (SIEM) systems constitute an emerging technology
organsisng the monitoring, detection and prevention measures of a
2. Related work
smart ecosystem, such as SG [10]. In particular, a SIEM can aggre-
gate, normalise and correlate various security events, thus identifying
potential security violations [10]. A security event is considered a Many papers have studied the security and privacy issues of SG.
normalised message related to the security status of the monitored in- Some of them are listed in [2,11,15–20]. In particular, in [11], the
frastructure [10]. However, the continuous progression of cyberattacks authors provide a comprehensive survey regarding the intrusion detec-
and malware requires the simultaneous evolution and adoption of the tion in the SG sector. After providing the necessary background about
necessary countermeasures. First, the guidelines of IEC 62351 cannot the SG and IDS, the authors investigate 37 cases related to detecting
be adopted quickly by the vendors and manufacturers, especially when cyberattacks and anomalies against (a) the entire SG ecosystem, (b) the
the corresponding SCADA/ICS operate in real-time since safety issues Advanced Metering Infrastructure (AMI), (c) SCADA systems, (d) sub-
can arise. On the other side, the current SIEM systems include a limited stations and (e) synchrophasors. The DiSIEM project in [15] evaluates
set of intrusion and anomaly detection mechanisms regarding the SG the efficiency of seven SIEM systems: (a) HP ArcSight, (b) IBM QRadar,
application-layer protocols [11]. In addition, they are characterised by (c) Intel McAfee Enterprise Security Manager, (d) Alienvault OSSIM,
a lack of understanding between the complicated relations of the real (e) XL-SIEM, (f) Splunk and (g) Elastic Stack based on various criteria
intrusion instances and fake alerts [12]. Therefore, the difficult goal of like data sources, data storage, User and Entity Behaviour Analytics
ensuring intelligent, safe, viable and efficient SG becomes a major need (UEBA), risk analysis, exposed APIs, resilience, event management and
filled with significant and far-reaching challenges. visualisation. Similarly, in [16], L. Cui et al. examine the detection
Based on the aforementioned remarks, this paper presents a SIEM of False Data Injection (FDI) attacks in SG, utilising ML methods. In
system called Secure and PrivatE smArt gRid (SPEAR) SIEM, which is particular, the authors focus on FDI attacks against (a) energy con-
exclusively focused on the SG ecosystem. The proposed SIEM is focused sumption data, (b) state estimation and (c) load forecasting. In [17],
on detecting, normalising and correlating security events against SG S. Quincozes et al. provide a survey about the intrusion detection and
environments and calculating the reputation value of each SG asset prevention mechanisms concerning the digital substations. M. Gunduz
(hardware or virtual device), which reflects how secure and trustworthy and R. Das in [18] investigate the various threats in SG, providing the
the functionality of each asset is. To this end, SPEAR SIEM is capable corresponding solutions and directions for future work. In a similar
of detecting, normalising and correlating cyberattacks and anomalies manner, in [2], P. Kumar et al. present a detailed study about the smart
against a plethora of SG communication protocols. Moreover, it in- metering networks, paying special attention to the security, privacy and
cludes anomaly detection models that process time-series operational open research issues. Accordingly, in [19], M. Hassan et al. present
data (i.e., raw electricity measurements) of four SG environments, a compilation about the differential privacy techniques for Cyber–
namely (a) hydropower plant, (b) substation, (c) power plant and (d) Physical Systems (CPS). Finally, in [20], I. Stellios et al. study IoT-based
smart home. The architectural model of SPEAR SIEM consists of five
cyberattacks against Critical Infrastructures (CIs), including SG, SCADA
primary components, namely (a) AlienVault OSSIM SIEM [13], (b)
and smart home environments. Subsequently, we pay our attention to
SPEAR SIEM Basis, (c) Message Bus, (d) Big Data Analytics Compo-
some specific cases, highlighting the differences with our work. Each
nent (BDAC), (e) the Visual-based Intrusion Detection System (VIDS)
paragraph focuses on a dedicated case.
and (f) Grid Trusted Module (GTM). Section 3 analyses the architec-
In [21], R. Leszczyna and M. Wróbel review three open-source
ture of SPEAR, detailing the functionality of each component. The
SIEM systems based on the SG conditions. In particular, the SIEM
contributions of this paper are summarised in the following points.
systems investigated are (a) AlienVault OSSIM [13], (b) Cyberoam
• Providing a SIEM system specially designed for SG: The pro- iView [22] and (c) Prelude [23]. For the evaluation procedure, the
posed SIEM can detect, normalise and correlate the security authors adopt the Solution Merit Index (SMI) by B. Sahay and K.
events related to multiple SG application-layer cyberattacks. Gupta [24]. The proposed methodology relies on (a) primary criteria
• Providing a set of operational data-based anomaly detection and (b) secondary criteria. The primary criteria are (a) number of
models: The specific models can detect anomalies based on the available and compatible sensors, (b) number of the out-of-the-box
operational data (i.e., time series electricity data) of four SG use sensors, (c) diversity of available sensors, (d) real-time performance,
cases: (a) hydropower plant, (b) substation, (c) power plant and (e) range and flexibility of reporting, (f) alert correlation, (g) auto-
(d) smart home. response capabilities. On the other hand, the secondary criteria are (a)

2
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

documentation comprehensiveness, (b) complexity of the installation that intrusion sessions presenting a binary relation can compose the
process, (c) complexity of the system configuration, (d) portability and intrusion scenario. Finally, a correlation graph is generated, consisting
(e) hardware requirements. Based on the primary criteria, the OSSIM of the intrusion sessions and their binary relations. The evaluation
performance reaches 97% while the performance of Cyberoam iView analysis shows the efficacy of IACF in terms of (a) the recognition of
and CS Prelude reach 76% and 24.3%, respectively. Concerning the multi-step cyberattacks, (b) the performance of the proposed algorithms
secondary criteria, the Prelude performance approaches 86.8% while and (c) accuracy.
OSSIM and Cyberoam iView reach 59.4% and 56.6%. The complete In our previous work in [29], we present an IDS called ARIES
SMI for OSSIM is 81.96% while the SMI of Prelude and Cyberoam (smArt gRid Intrusion dEtection System), which focuses on SG. The
iView is calculated at 80.68% and 37.16%. Therefore, according to architecture of the proposed IDS consists of three main modules: (a)
the authors, OSSIM is a complete SIEM system appropriate for the Data Collection Module, (b) ARIES Analysis Engine and (c) Response
situational awareness of an SG environment. Module. The Data Collection Module is responsible for collecting (a)
In [25], K. Zhang et al. introduce the Backward Influence Factor network flow statistics, (b) Modbus/TCP payload information and (c)
(BIF) algorithm capable of processing and mining intrusion patterns operational data. Next, the ARIES Analysis Engine consists of three
originating from a sequence of IDS alerts. The proposed algorithm detection layers related to the aforementioned data types. The first
handles efficiently the sequence data analysis issues like random noise, layer focuses on detecting cyberattacks, utilising network flow statis-
disordering and element missing. In particular, it consists of five phases: tics. In particular, it consists of two complementary detection models:
(a) normalisation, (b) intrusion action extraction, (c) intrusion session (a) Intrusion Detection Model and (b) Anomaly Detection Model. First,
pruning, (d) correlation discovery and (e) dynamic correlation graph IDM takes place, adopting a decision tree classifier capable of de-
construction. During the first phase, the IDS alerts are normalised into tecting five cyberattacks: (a) File Transfer Protocol (FTP) brute-force
a common format. Next, the intrusion action extraction phase follows attacks, (b) Secure Shell (SSH) brute-force attacks, (c) DoS, (d) bot
by discriminating the alerts based on two elements: (a) the source IP and (e) port scanning. If the detection outcome of IDM is normal,
address and (b) the destination IP address. Subsequently, the intrusion then ADM is activated, trying to identify a potential anomaly. To
actions are specified, considering the type and the destination port this end, an autoencoder is used. Next, the second layer is devoted
fields. Next, the intrusion session pruning phase undertakes to separate to detecting potential Modbus anomalies by analysing the Modbus
long intrusion actions into smaller sequences called intrusion sessions. payload through the isolated forest algorithm. Finally, the third layer
Then, the pruning process starts, removing the sub-patterns from the focuses on electricity-related operational data and adopts the ARIES
initial sequence. Next, the correlation discovery phase aggregates all Generative Adversarial Network (GAN) to recognise relevant anoma-
pruned sessions, based on their starting time. The BIF algorithm is lies. Finally, the Response Module notifies the security administrator
responsible for computing the attraction score between two sessions. and can generate some automated firewall rules to mitigate the impact
The attraction score is expressed by the Influence Factor (IF). Finally, of the potential cyberattacks/anomalies. The main novelty of this work
the last phase generates a dynamic correlation graph based on the lies in the development of the ARIES GAN at the third detection layer.
higher IF values. The evaluation results demonstrate the efficacy of ARIES, including a
In [26], M. Albanese et al. provide a probability-based framework, comparison study with multiple ML/DL methods.
which assesses and quantifies whether the sequence of events is un- In [30], the authors introduce an anomaly-based IDS for the electri-
explained, considering models of previously learned behaviours. Based cal grid, based on operational data of a real power plant. The proposed
on the authors, such events can originate from (a) intrusion detection IDS consists of two primary stages (a) the training stage and (b) the
and (b) alert correlation processes. Although their work can be applied testing stage. In the first stage, the ML training process is carried out,
to both processes, it does not aim to overcome or replace them. In while the testing stage allows real-time anomaly detection, predicting
contrast, the proposed framework runs on top of them, analysing whether an anomaly exists or not. In particular, the training stage
whether their output is adequately explained. The authors consider that includes four modules: (a) Data Collection Module, (b) Pre-Processing
the available intrusion detection models and alert correlation models Module, (c) Feature Selection module and (d) Training Module. Ac-
are ineffective for explaining a sequence of events identified in data cordingly, the testing stage comprises four modules: (a) Data Collection
streams. The input for the intrusion detection decision is a vector of Module, (b) Pre-processing Module, (c) Anomaly Detection Module and
network packets, while the alert correlation procedure relies on a set (d) Response Module. The main innovation of this work lies in the
of alerts. The proposed framework is actually based on their previous fact that the Pre-Processing Module (in both stages) adopts a complex
work in [27] related to the cybersecurity settings. In particular, the data representation, which results in better detection performance. The
authors adapt the algorithms of [27] appropriately in order to estimate evaluation analysis demonstrates the efficiency of the complex data
the probability that a sequence of events is unexplained. The evaluation representation, comprising a plethora of ML and DL methods, such as
results demonstrate the efficacy of the proposed framework in terms of Principal Component Analysis (PCA), One-Class Support Vector Ma-
accuracy and scalability. chine (SVM), isolation forest, Angle-Based Outlier Detection (ABOD),
K. Zhang et al. in [28] provide an alert correlation framework SOS and autoencoder.
called Intrusion Action Based Correlation Framework (IACF) presenting In [31], M. Ali et al. present MALGRA, which constitutes a combined
a similar architecture as in their previous work in [25]. The pro- ML and N-Gram malware feature extraction and detection system. The
posed framework enhances the aggregation of cybersecurity alerts, methodology behind MALGRA includes six steps: (a) dynamic analysis,
the intrusion actions association, the extraction of intrusion sessions (b) Application Programming Interface (API) call feature extraction, (c)
and finally the intrusion scenarios identification. IACF is composed of N-Gram creation, (d) feature reduction, (e) N-Gram model preparation
three phases: (a) normalisation, (b) intrusion session construction and and (f) testing using samples. First, the authors follow a dynamic analy-
(c) intrusion scenario construction. First, the cybersecurity alerts are sis in order to investigate the behaviour of various malware, utilising an
aggregated and divided into two groups based on the source IP and Artificial Intelligence (AI) sandbox, called SNDBOX. In particular, the
the destination IP address. Thus, the intrusion actions are extracted authors investigate two scenarios. The first one focuses on the API calls
based on the sequence of alerts displaying an intrinsic correlation. and their arguments’ memory location to construct N-Grams. An N-
Next, the extraction of intrusion sessions follows, aiming to split long Gram is a subset of a given data sample with a length of n. In the second
sequences of intrusion actions into smaller intrusion sessions. To this scenario, the N-Grams are implemented based on the function calls and
end, two algorithms are used, namely (a) Time-lag based Sequence their arguments’ address. Next, the Term Frequency–Inverse Document
Splitting (TSS) and (b) Sequence Pruning Algorithm (SPA). Finally, Frequency (TF–IDF) method is adopted in order to reduce the feature
the intrusion scenario construction starts, following the assumption space. TF–IDF is a statistical method assessing how relevant a word is in

3
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 1. SPEAR SIEM architecture.

a document. Finally, the N-Grams are transformed into binary vectors specify the compromised PMUs or PDCs and recover their proper func-
introduced to the ML methods. The evaluation analysis demonstrates tionality. The authors evaluate their system with three use cases: (a)
the effectiveness of MALGRA. To this end, the authors used four ML 7-bus transmission power system, (b) 39-bus New England system and
methods and 60 malicious samples from the virus share website. The IEEE 118-bus system. The experimental result confirms the efficiency
ML methods used are (a) Naive Bayes, (b) Decision Tree, (c) Random of the proposed detection and mitigation system.
Forest and (e) Logistic Regression. Based on the experimental results, Undoubtedly, the previous works introduce significant contribu-
the Logistic Regression accomplishes the best detection accuracy. tions. Based on [21], we use the AlienVault OSSIM as a basis for the
M. Ghafouri et al. [32] provide a detection and mitigation system proposed SPEAR SIEM. However, AlienVault OSSIM focuses mainly on
against cyber–physical attacks related to a Wide Area Management signature-based techniques without considering the special peculiarities
System (WAM) and its components (i.e., Phasor Measurement Unit and characteristics of SG. It is noteworthy that the commercial version
(PMU) and Phasor Data Concentrator (PDC)). A voltage stability prob- of AlienVault OSSIM called AlienVault Unified Security Management
lem refers to the instability of the power system to maintain and control (USM) [13] includes some correlation rules and directives about SCADA
the appropriate voltage values at all buses during the regular operation systems. However, both AlienVault OSSIM and AlienVault USM do
or after an electrical disturbance. This situation can lead to various con- not utilise ML and DL solutions targeted to the SG application layer
sequences, such as load curtailment, brownouts or even power outage. protocols. Furthermore, although several research efforts use ML and
First, the authors study the cyberattacks against WAM, discriminating DL for detecting cyberattacks or anomalies against SG application-
two main categories: (a) cyberattacks against communication links layer protocols, they cannot discriminate the exact cyberattack type.
and (b) cyberattacks related to the WAM devices and data. Based on For instance, they may detect a DoS attack without describing specif-
this study, an attack generation algorithm is implemented, targeting ically how this attack is related to the respective application-layer
the voltage stability. The proposed attack generation algorithm relies protocol. Moreover, a few papers pay attention to industrial protocols
on the power flow equations, addressing traditional anomaly detec- like BACnet and IEC 60870-5-104, without again specifying the exact
tion techniques. Next, the authors introduce a detection mechanism cyberattack type. Also, it is worth mentioning that the existing works
adopting the Thevenin Equivalent (TE) parameters. It is worth noting do not correlate the various SG-related security events.
that the proposed detection scheme does not rely on historical data Therefore, based on the aforementioned remarks, we provide a
and is capable of detecting the aforementioned cyberattacks. Next, a comprehensive SIEM system dedicated to SG, aiming to address the
mitigation framework is presented, allowing the system operator to current shortcomings. First, SPEAR SIEM includes a variety of ML and

4
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 2. SPEAR SIEM operation flow.

DL detectors capable of discriminating the exact cyberattack type. Next, named ‘security events’. Through the Message Bus, these security events
it introduces visual-based detection mechanisms that allow the secu- are sent to GTM and VIDS. Finally, the security events originating from
rity administrator to identify undetected security issues. Next, SPEAR BDAC and VIDS, the GTM updated reputation values and the security
SIEM correlates the security events related to Modbus, thus composing alerts are visualised by VIDS. The following subsections analyse each
security alerts reflecting actual attack scenarios. Finally, SPEAR SIEM component in detail.
introduces an extra protection level that quantifies the trust value of
each SG asset based on the security events received by the various 3.1. AlienVault OSSIM
detectors.
AlienVault OSSIM is an open-source SIEM system capable of pro-
3. SPEAR SIEM architecture viding several security capabilities. Its architecture is composed of two
main components: (a) OSSIM Server and (b) OSSIM Sensors. The OSSIM
The SPEAR SIEM architecture relies on the ARCADE framework [14] Sensors are deployed throughout the SG infrastructure, collecting and
and consists of three layers as illustrated in Fig. 1. First, at the Data normalising security-related information from any asset (hardware or
Capturing Layer, the SPEAR SIEM Basis collects the necessary data for virtual devices). A wide range of OSSIM sensors is available, including
the intrusion detection processes. Three types of data are captured: firewalls, Host-based Intrusion Detection Systems (HIDS) and Network-
(a) network flow statistics, (b) packet payload information and (c) based Intrusion Detection Systems (NIDS). Next, the OSSIM Server
operational data (i.e., time-series electricity data). Then, the Detection aggregates and correlates the security information gathered by the
Layer follows, where the intrusion and anomaly detection processes OSSIM Sensors, thus composing security alerts. A security alert is
take place, generating the corresponding security events. There are four defined as a set of security events associated with each other [13].
intrusion detection processes: (a) network flow-based detection, (b) It is noteworthy that AlienVault OSSIM is already implemented and
packet-based detection, (c) operational data data-based detection and provided by AT&T. In the context of this paper, we use the AlienVault
(d) visual-based detection. The first three are implemented by BDAC OSSIM as a signature-based detection, producing the corresponding
while VIDS carries out the last. Finally, the correlation layer follows security events and alerts.
where the security events are correlated. There are two kinds of corre-
lation. The first one is implemented by VIDS through correlation rules 3.2. SPEAR SIEM basis
for the Modbus/TCP protocol, thus producing alerts reflecting multi-
step Modbus-related attack scenarios. The second kind is conducted by SPEAR SIEM Basis follows a server-sensor architecture consisting of
GTM, which receives the various security events and calculates each two components: (a) SPEAR Sensors and (b) DAPS. Fig. 3 illustrates
SG asset’s reputation value. Fig. 2 illustrates the interactions among the the SPEAR SIEM Basis architecture, showing the relationship between
SPEAR SIEM components. First, the OSSIM Sensors (part of AlienVault the SPEAR Sensors and DAPS. In particular, a SPEAR Sensor consists of
OSSIM) and the SPEAR Sensors (part of SPEAR SIEM Basis) are dis- two main functional elements (a) Network Capturer and Parser (NCP)
tributed throughout the SG infrastructure, thus monitoring, collecting and (b) Asset Discovery (AD). NCP uses a runtime network analyser to
and parsing various data. This information is transmitted then to the continuously capture, parse and forward network traffic data to DAPS.
OSSIM Server (part of AlienVault OSSIM) and Data Acquisition, Parsing More detailed, NCP analyses a plethora of SG application-layer pro-
and Storage (DAPS) (part of SPEAR SIEM Basis), respectively. The OS- tocols by isolating specific payload information and relevant network
SIM Server normalises this information and uses a MySQL database for flow statistics used by BDAC and VIDS to detect intrusions/anomalies.
the storage, while DAPS uses an Elasticsearch database and distributes To this end, Tshark [33] and CICFlowMeter [34] are adopted. The
this information to BDAC and VIDS. The normalised information stored format of the network flow statistics is defined by CICFlowMeter [34].
in the OSSIM server and the detection results of BDAC and VIDS are Finally, AD utilises periodically Nmap [35] to discover which assets

5
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 3. SPEAR SIEM basis architecture.

(hardware and virtual devices) are active, thus collecting and delivering cyberattacks and anomalies. Then, the BDAC Analysis Engine analyses
relevant information to DAPS. this data, identifying potential cyberattacks and anomalies. The BDAC
DAPS is a centralised server consisting of five functional elements: Analysis Engine includes 24 intrusion and anomaly detection models
(a) Streaming Bus, (b) Data Capturing and Parser (DCP), (c) Storage that analyse appropriately the various data types. The intrusion and
Infrastructure, (d) Representational State Transfer (REST) Server and anomaly detection models of the BDAC Analysis Engine are updated
(e) OSSIM Event Manager. First, the Streaming Bus is in charge of periodically via the Training Module. In particular, the Training Module
providing a near real-time streaming data to BDAC and VIDS in order to is fed by the Data Receiving Module with new normal and malicious
detect intrusions/anomalies during the prediction phase. In particular, data, thereby re-training the current intrusion/anomaly detection mod-
the Streaming Bus relies on Apache Kafka and transmits (a) specific els of the BDAC Analysis Engine only whether their accuracy and the
packet payload information, (b) network flow statistics (c) operational F1 score are better compared to the previous ones. Finally, based on the
data and (e) honeypot data. The operational data is retrieved directly BDAC Analysis Engine’s response, the SPEAR Event Extraction Module
by DAPS from the corresponding SG use case, while the honeypot data extracts the corresponding security events. The following subsections
is given by the Honeypot Manager, which is an external component provide more details about the architectural components of BDAC. It is
analysed in [14]. The SPEAR honeypots and how the honeypot data noteworthy that all BDAC modules are located in a common place so
is introduced into DAPS is out of the scope of this paper. More details that the communication interfaces among them are not necessary.
about this content are provided by [36,37] and [14], respectively. Next,
DCP is responsible for importing the data published in the Streaming 3.3.1. Data receiving module
Bus and storing them in the Storage Infrastructure. In turn, the Storage The Data Receiving Module communicates with the DAPS subcom-
Infrastructure persists all captured data originating either from the ponent of the SPEAR SIEM Basis in order to receive (a) network flow
SPEAR Sensors or DAPS. More precisely, the payload information re- statistics, (b) payload information of the SG application layer protocols,
lated to the SG application-layer protocols, the network flow statistics, (c) operational data, (d) honeypots’ logs and (e) asset-related data.
the operational data (i.e., time series electricity data) and the honeypot In particular, the Data Receiving Module utilises the DAPS Streaming
data are stored into an Elasticsearch database. On the other hand, the Bus to monitor the network flow statistics and honeypots’ logs, while
asset-related data originating from AD is stored in an SQLite database. the payload of the SG application-layer protocols and the operational
Next, the REST server transmits the asset-related to BDAC, VIDS and data are received periodically via the DAPS Storage Infrastructure of
GTM. Finally, the OSSIM Event Manager is in charge of retrieving OS- DAPS, utilising specific threshold values. According to the network
SIM security events from the OSSIM Server periodically and forwarding characteristics of each SG use case, these threshold values are defined
them to the Message Bus. The OSSIM security events are retrieved with appropriately. Finally, the asset-related data is received from the DAPS
all the attributes as defined by AlienVault [13] and then they are parsed REST Server.
to match with the SPEAR SIEM security event format (Table A.8).
3.3.2. Big data analysis engine
3.3. BDAC: Big data analytics component The BDAC Analysis Engine is the core architectural component of
BDAC responsible for detecting possible cyberattacks and anomalies.
BDAC is a backend component consisting of four main modules: It focuses mainly on detecting cyberattacks and anomalies against the
(a) Data Receiving Module, (b) Training Module, (c) BDAC Analysis SG application-layer protocols, including Modbus, DNP3, IEC 60870-
Module and (d) Security Event Extraction Module. First, the Data 5-104, IEC 61850 (MMS), BACnet, MQTT, HTTP and SSH. There-
Receiving Module is responsible for communicating with the SPEAR fore, the corresponding detection models are formed (e.g., Modbus
SIEM Basis to receive the appropriate data for detecting potential Intrusion/Anomaly Detection Models).

6
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 4. Flowchart of the Network Flow based Detection Models.

For each of these protocols, two detection categories are identified: the corresponding security event is produced. Otherwise, the TCP/UDP
(a) Network Flow-Based Detection Models and (b) Packet-Based De- Network Flow-Based Intrusion/Anomaly detection models are used in
tection Models. The first category (i.e., Network Flow-Based Detection a similar manner. It should be noted that the last models have been
Models) is devoted to identifying cyberattacks and anomalies based on presented in our previous work in [29] and focus on the TCP and UDP
network flow statistics. It is divided into two subcategories: (a) Network protocols of the transport-layer. Hence, if the TCP/UDP Network Flow-
Flow-Based Intrusion Detection Models and (b) Network Flow-Based Based Intrusion Detection Model detects a specific attack, the respective
Anomaly Detection Models. In particular, the Network Flow-Based security event is generated. Otherwise, the TCP/UDP Network Flow-
Intrusion Detection Models rely on multiclass classification ML/DL Based Anomaly Detection Model undertakes to discover whether a
methods in order to identify specific cyberattack types. In contrast, the possible anomaly exists, generating a suitable security event or not.
Network Flow-Based Anomaly Detection Models use outlier/novelty de- Finally, it should be noted that this process is carried out continuously,
tection to detect potential anomalies. The difference between a cyberat- always monitoring new network flow statistics.
tack and anomaly lies in the fact that a cyberattack specifies a particular The second category (i.e., Packet-Based Anomaly Detection Models)
intrusion type like a Denial of Service Attack (DoS) or a port scan, identifies potential anomalies based on the payload information of each
while an anomaly can originate from an intrusion or another reason like packet. Fig. 5 illustrates the relevant flowchart of the Packet-based
an electrical disturbance. Hence, the second subcategory (i.e., Network Anomaly Detection Models. First, the information of each packet is
Flow-Based Anomaly Detection Models) operates as complementary to received through the Data Receiving Module. Next, the correspond-
the first one (i.e., Network Flow-Based Intrusion Detection Models) ing application layer protocol is identified to execute the appropriate
based on the flowchart presented in Fig. 4. In particular, by checking packet-based anomaly detection model. Finally, if an anomaly is de-
the TCP/User Datagram Protocol(UDP) source and destination port tected, the corresponding security event is produced via the Security
of a network flow received by the Data Receiving Module, the cor- Event Extraction Module.
responding SG application layer protocol is identified. Therefore, the Apart from the application-layer protocols, the BDAC Analysis En-
appropriate Network Flow-Based Intrusion Detection Model related to gine uses operational data (i.e., raw electricity measurements) and
this protocol is activated (e.g., Modbus Network Flow-Based Intrusion honeypots’ logs in order to identify additional anomalies. Thus, the cor-
Detection Model). Then, if this model detects a specific attack, the cor- responding models are identified, i.e., Operational Data-Based Anomaly
responding security event is generated via the Security Event Extraction Detection Models and Honeypot-Based Anomaly Detection Models. The
Module. Otherwise, the relevant Network Flow-Based Anomaly Detec- operational data originate from the local environment of each SG use
tion Model is activated (e.g., Modbus Network Flow-based Anomaly case and is captured through the SPEAR SIEM Basis. In particular, four
Detection Model). Similarly, if the specific model identifies an anomaly, kinds of operational data were considered based on four individual SG

7
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 5. Flowchart of the Packet based Detection Models.

Fig. 6. Flow Diagram of the Operational Data-Based Detection Models.

Fig. 7. Flow Diagram of the Honeypot-Based Detection Model.

use cases, i.e., (a) hydropower plant, (b) substation, (c) power plant and 3.3.2.1. Modbus/TCP intrusion/anomaly detection models. Three
(d) smart home. On the other side, any interaction with a honeypot Modbus/TCP-related intrusion/anomaly detection models were im-
is considered an anomalous activity since a legitimate user will not plemented: (a) Modbus/TCP Network Flow-Based Intrusion Detec-
interact with it. Figs. 6 and 7 show the flowcharts related to the Op- tion Model, (b) Modbus/TCP Network Flow-Based Anomaly Detection
erational Data-Based Anomaly Detection Models and Honeypot-Based Model and (c) Modbus Packet-Based Anomaly Detection Model. The
Anomaly Detection Models, respectively. Regarding the Operational first two rely on Modbus/TCP related network flow statistics. In partic-
Data-Based Anomaly Detection Models, initially, a series of operational ular, the Modbus/TCP Network Flow-Based Intrusion Detection Model
data (i.e., electricity measurements) is collected through the Data Re- utilises a Decision Tree Classifier [38] aiming to identify malicious
ceiving Module. Next, the respective Operational Data-Based Anomaly network flows indicating the following Modbus/TCP cyberattacks:
detection model is applied. If an anomaly is recognised, a relevant
security event is generated by the Security Event Extraction Module. • modbus/function/readInputRegister (DoS): This DoS attack
On the other side, the honeypots’ logs are received via the Data Receiv- floods the target system with Modbus/TCP Read Input Register
ing Module and are transformed into security events by the Security packets (Function Code 04).
Event Extraction Module. Therefore, based on the previous remarks, • modbus/function/writeSingleCoils: This unauthorised access
the following subsections analyse the respective intrusion/anomaly attack sends a Modbus/TCP packet (Function Code 05), which
detection models per SG application-layer protocol and those related changes the status of a single coil either to ON or OFF. Since
to the operational data and honeypots’ logs. the Modbus/TCP protocol does not include any authentication

8
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

or authorisation mechanism, a cyberattacker can send malicious • Flooding: This DoS attack floods continuously the target system
Modbus/TCP commands against the target system. with DNP3 packets.
• modbus/scanner/getfunc: This reconnaissance attack enumer- • DNP3 Reconnaissance: This reconnaissance attack identifies
ates all Modbus/TCP function codes used and supported by the whether the DNP3 protocol is used by the target system or not.
target system. • Replay: This attack replays DNP3 packets originating from a
• modbus/dos/writeSingleRegister: This DoS attack floods the legitimate party to the other endpoint.
target system with Modbus/TCP Write Single Register packets • Masquerading: In this attack, the cyberattacker imitates the
(Function Code 06). behaviour of a legitimate asset, sending the appropriate DNP3
• modbus/function/readDiscreteInputs (DoS): This DoS attack packets.
floods the target system with Modbus/TCP Read Discrete Inputs The DNP3 Network Flow-Based Anomaly Detection Model uses the
packets (Function Code 02). ABOD method [44,45], thus identifying anomalous DNP3 network
• modbus/function/readHoldingRegister (DoS): This DoS attack flows. Both models were trained, utilising normal DNP3 network flow
floods the target system with Modbus/TCP Read Holding Register statistics coming from a real substation environment as well as from
packets (Function Code 03). the DNP3 intrusion detection dataset of N.Rodofile et al. [46]. The
• modbus/function/readCoils (DoS): This DoS attack floods the evaluation analysis of these DNP3 intrusion/anomaly detection models
target system with Modbus/TCP Read Coils packets (Function is presented in our previous work in [41].
Code 01).
3.3.2.3. IEC 60870-5-104 intrusion/anomaly detection models. Three
• modbus/function/readInputRegister: This unauthorised attack
IEC 60870-5-104-related detection models are incorporated into the
sends a Modbus/TCP packet (Function Code 04) used to read the
BDAC Analysis Engine: (a) IEC 60870-5-104 Network Flow-Based In-
values of specific input registers. trusion Detection Model, (b) IEC 60870-5-104 Network Flow-Based
• modbus/function/writeSingleRegister: This unauthorised ac- Anomaly Detection Model and (c) IEC 60870-5-104 Packet-Based
cess attack sends a Modbus/TCP packet (Function Code 06) in Anomaly Detection Model. The first two rely on IEC 60870-5-104
order to write a value to a specific holding register. related network flow statistics specified by the 2404 TCP port. In par-
• modbus/dos/writeSingleCoils: This DoS attack floods the target ticular, the IEC 60870-5-104 Network Flow-Based Intrusion Detection
system with Modbus/TCP Write Single Register packets (Function Model utilises a Decision Tree Classifier [38], detecting the following
Code 06). cyberattacks.
• modbus/function/readDiscreteInput: This unauthorised access
attack sends a Modbus/TCP packet (Function Code 02) to read • c_ci_na_1_DoS: This DoS attack floods the target system with
c_ci_na_1 IEC 60870-5-104 packets.
the status of specific discrete inputs.
• c_sc_na_1: This unauthorised access attack injects a c_sc_na_1 IEC
• modbus/scanner/uid: This reconnaissance attack enumerates
60870-5-104 packet to the target. Since IEC 60870-5-104 does not
which slave IDs are activated.
comprise sufficient authentication and authorisation mechanisms,
• modbus/function/readCoils: This unauthorised access attack
a potential cyberattacker can perform malevolent IEC 60870-
sends a Modbus/TCP packet (Function Code 01) to read the status 5-104 commands to manipulate the target system.
of specific coils.
• c_ci_na_1: This unauthorised access attack injects a c_ci_na_1 IEC
• modbus/function/readHoldingRegister: This unauthorised ac- 60870-5-104 packet to the target.
cess attack sends a Modbus/TCP packet (Function Code 03) to • c_se_na_1: This unauthorised access attack injects a c_se_na_1 IEC
read the values of specific holding registers. 60870-5-104 packet to the target system.
• c_sc_na_1_DoS: This DoS attack floods the target system with
The aforementioned cyberattacks are implemented by Smod, a
c_sc_na_1 IEC 60870-5- 104 packets.
widely known pen-testing tool related to Modbus [39,40]. The Modbus
• c_se_na_1_DoS: This DoS attack floods the target system with
Network Flow-Based Anomaly Detection Model adopts the DIDEROT
c_se_na_1 IEC 60870-5- 104 packets.
Autoencoder [41], identifying anomalous Modbus/TCP network flows.
• m_sp_na_1_DoS: This DoS attack floods the target system with
The DIDEROT autoencoder is analysed in our previous work in [41].
m_sp_na_1 IEC 60870-5- 104 packets.
Finally, the last model focuses on the payload of the Modbus/TCP pack-
ets, recognising Modbus/TCP anomalous packets based on the Isolation The IEC 60870-5-104 Network Flow-Based Anomaly Detection
Forest method [42]. Since there are no sufficient intrusion/anomaly Model adopts the Isolation Forest method [42], detecting anomalous
detection datasets related to the Modbus/TCP, it is worth mentioning IEC 60870-5-104 network flows. Finally, the last model focuses on the
that relevant Modbus/TCP intrusion/anomaly detection datasets were IEC 60870-5-104 packets’ payload information, identifying IEC 60870-
constructed, by implementing Modbus/TCP cyberattacks against a real 5-104 anomalous packets. To this end, it applies the Local Outlier
smart home as well as an emulated SG environment. To this end, the Factor (LOF) method [47,48]. For the training process, a suitable IEC
directions provided by A. Gharib et al. [43] were followed. The evalu- 60870-5-104 intrusion detection dataset was constructed, utilising an
emulated substation environment. For this purpose, the directions of A.
ation analysis related to the Modbus/TCP intrusion/anomaly detection
Gharib et al. [43] were used. The evaluation results related to the IEC
models is analysed in Section 4.
60870-5-104 detection models are presented in Section 4.
3.3.2.2. DNP3 intrusion/anomaly detection models. The BDAC Analysis
3.3.2.4. IEC 61850 (MMS) anomaly detection model. The BDAC Anal-
Engine encloses two detection models related to DNP3: (a) DNP3 ysis Engine includes a single model related to the IEC 61850 (MMS)
Network Flow-Based Intrusion Detection Model and (b) DNP3 Network protocol. The proposed model is named IEC 61850 (MMS) Network
Flow-Based Anomaly Detection Model. Both of them rely on DNP3- Flow-Based Anomaly Detection Model and relies on outlier/novelty
related network flow statistics. In particular, the DNP3 Network Flow- detection and network flow statistics defined by the TCP port 102.
Based Intrusion Detection Model utilises a Decision Tree Classifier [38], In particular, it utilises the Minimum Covariance Determinant (MCD)
which recognises the following five DNP3-related cyberattacks. method [45,49]. Since there are no sufficient intrusion/anomaly de-
tection datasets related to IEC 61850 (MMS), an IEC 61850 (MMS)
• Injection: Since the DNP3 protocol does not include sufficient au- anomaly detection dataset was constructed, by combining normal IEC
thorisation mechanisms, this attack injects malicious DNP3 pack- 61850 (MMS) network flows from an emulated substation environment
ets in a communication established between a DNP3 outstation and abnormal IEC 61850 (MMS) network flows generated statistically.
and master. The evaluation analysis of the specific model is detailed in Section 4.

9
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

3.3.2.5. Bacnet intrusion/anomaly detection models. The BDAC Analysis 3.3.2.9. Operational data based anomaly detection models. The BDAC
Engine includes two detection models related to BACnet. The first one Analysis Engine includes four detection models that analyse operational
called BACnet Network Flow-based Intrusion Detection Model utilises data (i.e., time series electricity measurements), detecting anomalies
the Random Forest method [50], thus detecting three BACnet cyberat- related to four SG use cases: (a) hydropower plant, (b) substation,
tacks: (a) fuzzing, (b) flooding and (c) tampering. The second model (c) power plant and (d) smart home. In particular, the first model
focuses on the BACnet packets’ payload and is named BACnet Packet- related to the hydropower plant environment adopts a GAN [52], which
Based Anomaly Detection Model. It uses a custom text Convolutional was presented in our previous work in [29]. Next, the second model
Neural Network (CNN) [51,52], which detects abnormal BACnet pack- (i.e., related to the substation environment) applies LOF. The remaining
ets. This method is named Payload Text CNN Classifier. Due to the lack models related to the anomalies of the power plant and the smart home
of publicly available intrusion/anomaly detection datasets for BACnet, use also the GAN presented in [29]. For the training process, real data
an appropriate dataset was implemented utilising the equipment of a was used for each SG use case. As in the previous cases, the evaluation
real smart home environment. The evaluation analysis of the aforemen- of the particular models is detailed in Section 4.
tioned models and more details about the Payload Text CNN Classifier 3.3.2.10. Honeypots-based detection model. The Honeypot-Based Detec-
are included in Section 4. tion Model relies on SG honeypots coming from our previous works
3.3.2.6. MQTT intrusion/anomaly detection models. Two detection mod- in [55] and [36,37]. In particular, the honeypots’ logs are collected
els are integrated into the BDAC Analysis Engine regarding the MQTT by the Honeypot Manager that forwards them to the Honeypots-Based
protocol: (a) MQTT Network Flow-Based Intrusion Detection Model Detection Model. The latter undertakes to normalise and transform
and (b) MQTT Packet-Based Intrusion Detection Model. On the one them into security events based on the format of Table A.8. The
Honeypot Manager is analysed in our previous work in [14].
hand, the first model applies the Random Forest method [50] with
MQTT network flow statistics and detects three kinds of MQTT-related
3.3.3. Training module
cyberattacks: (a) unauthorised subscribe, (b) large payload DoS attack
The Training Module is responsible for providing the BDAC Analysis
and (c) connection flooding attack. On the other hand, the second
Engine with the various ML/DL based intrusion/anomaly detection
model uses the SPEAR Payload Text CNN [51,52] with the payload
models. In particular, the main goal behind this module is to train
attributes of the MQTT packets in order to recognise the anomalous
the initial intrusion/anomaly detection models of the BDAC Analysis
MQTT packets. For the training process, an appropriate MQTT intrusion
Engine and re-train them periodically with more and updated data.
detection dataset was constructed, following the directions of [43].
The previous intrusion/anomaly detection models of the BDAC Analysis
As in the previous cases, the evaluation results of the aforementioned
Engine are replaced whether the performance of the new ones is better
models are documented in Section 4.
in terms of the accuracy and the F1 score metrics.
3.3.2.7. HTTP intrusion/anomaly detection models. The BDAC Analysis
Engine integrates two detection models associated with the HTTP 3.3.4. Security event extraction module
protocol: (a) HTTP Network Flow-Based Intrusion Detection Model, (b) The Security Event Extraction Module undertakes to generate nor-
HTTP Network Flow-Based Anomaly Detection Model. The first model malised security events based on the outcome of the BDAC Analysis
adopts a Decision Tree Classifier [38] capable of discriminating the Engine intrusion/anomaly detection models. The format of the SPEAR
following HTTP-related cyberattacks. security events is given in Table A.8. The Security Event Extraction
Module utilises the information given by the Data Receiving Module
• DoS: This DoS attack floods the target system with HTTP packets. concerning (a) the network flow statistics, (b) packet payload infor-
• SQL-Injection: This attack aims to exploit vulnerabilities of web mation of the SG application-layer protocols, (c) operational data and
applications in order to access unauthorised information. (d) honeypots’ logs to fill in the necessary fields of the SPEAR security
• Bruteforce-Web: This attack attempts to access a password- event format. Moreover, it communicates with DAPS in order to receive
protected web application by using multiple passwords’ combi- more information for the assets related to a security event, such as
nations. the asset ID, the asset name and the network ID. Finally, it pushes the
• XSS: Cross-Site Scripting (XSS) is a type of injection attack, where BDAC security events to Message Bus. It is noteworthy that based on
malicious scripts are injected into web applications. the security event information, this module can also indicate and form
automatic firewall rules that are introduced in the Userdata fields of
The HTTP Network Flow-Based Anomaly Detection Model relies the security event format (Table A.8). These firewall rules rely on the
on LOF [47,48]. Both models mentioned above take as input HTTP syntax of the Linux firewall, i.e., iptables [56].
network flow statistics specified by the 80 TCP port. For the train-
ing process, a combined dataset was utilised, including normal HTTP 3.4. VIDS: Visual-based intrusion detection system
network flows originating from an emulated substation environment
and malicious HTTP network flow statistics of the CSE-CIC-IDS2018 VIDS has been designed to receive, store, present, manipulate
dataset [34]. Section 4 details the evaluation results for both HTTP and visualise data (security events, network packets, operational data
detection models. (i.e., time-series electricity measurements) and network assets data)
from the other SPEAR SIEM components on a simple and easy-to-
3.3.2.8. SSH intrusion/anomaly detection models. Two SSH-related de- use visual environment. Moreover, VIDS correlates the Modbus-related
tection models are involved in the BDAC Analysis Engine. The first security events of BDAC, thus composing Modbus security alerts based
one is named SSH Network Flow-Based Intrusion Detection Model on correlation rules. First, VIDS communicates with the Message Bus,
and uses Adaboost [53,54] to recognise SSH bruteforce attacks. The thus consuming and visualising the security events generated only by
second model, called SSH Network Flow-Based Anomaly Detection BDAC and the VIDS itself. The security events of AlienVault OSSIM are
Model applies the MCD method [45,49] to identify anomalous SSH correlated and illustrated by AlienVault OSSIM itself. This communica-
network flows. Both models take as input SSH network flow statistics. tion between VIDS and Message Bus relies on Apache Kafka. Moreover,
The training process relies on a combined dataset, which includes VIDS communicates with DAPS of SPEAR SIEM Basis in order to
normal SSH network flows from an emulated substation environment receive the appropriate data for the visual-based anomaly detection
and malicious SSH network flows of the CSE-CIC-IDS2018 dataset [34]. mechanisms. As in the case of BDAC, VIDS receives from DAPS the
Section 4 details the relevant evaluation results. payload of the SG application-layer protocols, network flow statistics

10
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 8. Anomaly score line chart over time. The values below the red line correspond to normal data and the ones above indicate potential anomalies.. (For interpretation of the
references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 9. Scatter plot of the 2D data representation of the recorded features. In this case, points having a blue tint and located to the left correspond to normal data, while red points
located to the right side indicate potential anomalies. X Dim and Y Dim denote the dimensions after the dimensionality reduction process.. (For interpretation of the references to
colour in this figure legend, the reader is referred to the web version of this article.)

and operational data (i.e., time series electricity measurements). Both produces a latent space in the form of a manifold in two or three
Apache Kafka (Streaming Bus) and the Elasticsearch API (Storage dimensions. The produced output includes a colour indication at each
Infrastructure) are utilised for the communication between VIDS and point, which is automatically adjusted based on the distance from the
the SPEAR SIEM Basis. The role of VIDS is complementary to that statistical centre of the expected data. This distance value corresponds
of BDAC and AlienVault OSSIM, allowing the system operator or the to the measured distance from the centroid of normal values in the
security administrator to observe potential anomalies through appropri- reduced dimensionality space and indicates how close to normal the
ate visualisations. Finally, VIDS communicates with GTM to configure observed data is. The methods also produce a covariance matrix, show-
it and visualise its reputation values of each asset (i.e., hardware or ing the correlation between the recorded features over time, indicating
virtual devices). This communication is based on a REST API. how each parameter influences the rest. The outputs of each algorithm
By focusing on the visual-based detection mechanisms with oper- are saved into a PostgreSQL database of VIDS and are used to plot the
ational data (i.e., time series electricity measurements), several ML visualisation diagrams (Fig. 8, Fig. 9, Fig. 10 and Fig. 11).
and DL-based dimensionality reduction methods are adopted to detect Fig. 8 presents a line-chart displaying the anomaly score of the
anomalies. All of them are available in the VIDS dashboard, thereby operational data (i.e., time series electricity measurements) over time.
giving the user the capability to show different visualisations. It is The red horizontal line represents the threshold of normal values,
inherently arduous to visualise the incoming network and operational calculated as the statistical centre of the normal data. The black line
data in a manner straightforward to understand by humans since in represents the distance from this threshold, indicating how close to
most cases, they comprise a large number of features. The role of normal the incoming data is at each time instant. There are two such
dimensionality reduction in this context is to reduce these features in diagrams, one for the live operational data and one for the historical
a lower-dimensional space and represent all of them with a single 2D operational data stored in the VIDS database. In the latter, the user
or 3D point in space, which is easy to understand by the system oper- can select a time window (i.e., 3 h) and scroll through the diagram,
ator. Towards this goal, each ML/DL dimensionality reduction method observing the anomaly score over this time window.

11
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 10. Scatter plot of the 3D data representation of the recorded features. In this
case, points having a blue tint, located towards the middle correspond to the normal
data, while points with a red tint indicate potential anomalies. X Dim, Y Dim and Z Dim
denote the dimensions after the dimensionality reduction process.. (For interpretation
of the references to colour in this figure legend, the reader is referred to the web
version of this article.)

Fig. 11. Dependency wheel diagram showing the recorded features correlation. Higher
line width indicates a stronger relation between the features.
Figs. 9 and 10 depict the reduced dimensionality space of the
operational data. The user can choose between representations in either
two or three dimensions, with the live and historical data. At each
secure each asset is. To this end, GTM communicates with the Message
time instant, the live scatter plot displays the network’s current status,
Bus to receive the various security events produced by AlienVault
after executing the ML and DL-based dimensionality reduction meth-
OSSIM, BDAC and VIDS. Fig. 12 shows the architecture of GTM. In
ods, using the most recent operational data received from the Storage
particular, since GTM is a backend component, VIDS is utilised for its
Infrastructure of the SPEAR SIEM Basis. In the case of the historical
configuration, defining a specific threshold value for each asset. If an
data, the scatter plot represents the status of the grid throughout the
asset’s reputation value exceeds the particular threshold, then a GTM
whole selected date. The visual patterns formed in these diagrams alert is generated for the specific asset. This communication between
allow the operator to observe the network’s status and determine VIDS and GTM is implemented via a REST API. Then, all security events
anomalies by looking at the projected points’ position and tint. The are received from the Message Bus, and the GTM Functional Process
potential anomalies are showcased by grouped points having a red Unit undertakes to calculate a reputation value for each asset. These
tint. By rendering these charts, VIDS offers an overview of the network reputation values are sent to the VIDS, which undertakes to visualise
status with respect to anomalies in the operational data and provides them. Finally, the reputation values of GTM are stored into the GTM
a comprehensive visualisation through several methods. The security database as historical data.
administrator can deduce whether an anomaly occurs at any given time The operation core of GTM is the GTM Functional Process Unit,
instant by observing the respective patterns. which consists of four elements: (a) the GTM queue, (b) the Fuzzy
Fig. 11 illustrates the correlation among the recorded features of Logic Core, (c) the Fuzzy Reputation Reduction System and (d) the
the operational data. A higher line width indicates a more substantial Fuzzy Reputation Recovery System. First, GTM receives continually
influence between the corresponding features. The user can hover at security events stored into the GTM queue, which applies a First In
each line and observe the actual value of the connection. Values close to First Out (FIFO) model. Next, the Logic Core undertakes to quantify the
0.05 indicate no correlation, while values close to 1 recommend strong severity of each security event based on fuzzy logic rules, considering
relation. The live dependency diagram shows the status corresponding the asset value, the subcategory, the event risk, the priority and the
to the most recent operational data at each time instant. Finally, the reliability of the security events based on Table A.8. The Fuzzy Logic
historical diagram displays the average value throughout the selected Core utilises the fuzzy theory to map the value of each aforementioned
date for each connection. variable into a quantified value without strict rules. Table 1 shows
The VIDS correlation capability relies on correlation rules that focus indicative fuzzy logic rules used by the Fuzzy Logic Core. These rules
on the security events generated by the Modbus Network Flow-Based are derived by forming the fuzzy universe. The fuzzy universe is unique
Intrusion Detection Model. However, similar rules can be identified for and mandatory for each variable used to calculate the quantified value
of the security event.
the other industrial protocols. This kind of correlation aims to identify
The purpose of the Fuzzy Reputation Reduction System is to produce
relationships among the Modbus security events, composing alerts re-
the reputation value of any asset related to the corresponding security
flecting multi-step attack scenarios related to Modbus. The correlation
event. The reputation value of each asset is computed, taking into
rules are constructed by combining the information of the security
account the time difference between the previous reputation value and
events (Table A.8) as well as additional fields, such as time information
the current security event as well as the outcome of the Fuzzy Logic
(e.g., a sequence of events appearing in a specific period time) or Core. Table 2 includes indicative fuzzy logic rules used by the Fuzzy
the number of continuous security events. Event Processing Language Reputation Reduction System in order to calculate the reputation value
(EPL) statements are utilised for the syntax of these correlation rules. of each asset.
Table B.9 in Appendix B summarises these rules. Finally, the Fuzzy Reputation Recovery System undertakes to in-
crease the reputation value based on the time difference between the
3.5. GTM: Grid trusted module last reduction of an asset’s reputation value and the current time. A
threshold in the VIDS determines the frequency, which is utilised to
The goal of GTM is to correlate the various security events and check a possible increment of the reputation value. The functionality
calculate a reputation value for each SG asset (hardware or virtual). of the Fuzzy Reputation Recovery System is also based on fuzzy rules.
This kind of correlation intends to reflect how trustworthy, safe and Table 3 shows a sample of them.

12
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 12. GTM architecture.

Table 1
Indicative rules of the fuzzy logic core.
No Input Output
Rule #1 Asset Value: high, Priority: high, Risk: high, Quantified Value:
Reliability: high, Subcategory: modbus/function/readCoils Low
Rule #2 Asset Value: low, Priority: low, Risk: low, Quantified Value:
Reliability: medium, Subcategory: SQL Injection High
Rule #3 Asset Value: high, Priority: high, Risk: high, Quantified Value:
Reliability: medium, Subcategory: HTTP DoS Low
... ... ...
Rule #20 Asset Value: high, Priority: medium, Risk: high, Quantified Value:
Reliability: medium, Subcategory: DNP3 Reconnaissance Low
Rule #21 Asset Value: high, Priority: medium, Risk: medium, Quantified Value:
Reliability: high, Subcategory: Masquerading Low
Rule #22 Asset Value: high, Priority: medium, Risk: low, Quantified Value:
Reliability: high, Subcategory: Port Scanning Low

Table 2
Indicative rules of the fuzzy reputation reduction system.
No Input Output
Rule #1 Time: Low, Quantified Value: Low Reputation Value: Low
Rule #2 Time: Low, Quantified Value: Low Reputation Value: Medium
Rule #3 Time: Low, Quantified Value: High Reputation Value: Medium
Rule #4 Time: Medium, Quantified Value: Low Reputation Value: Low
Rule #5 Time: Medium, Quantified Value: Medium Reputation Value: Medium
Rule #6 Time: Medium, Quantified Value: High Reputation Value: High
Rule #7 Time: High, Quantified Value: Low Reputation Value: Low
Rule #8 Time: Medium, Quantified Value: Medium Reputation Value: High
Rule #9 Time: Medium, Quantified Value: High Reputation Value: High

13
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table 3 anomaly detection models of the BDAC Analysis Engine and VIDS using
Indicative rules of the fuzzy reputation recovery system.
operational data (i.e., time-series electricity measurements), suitable
No Input Output datasets were produced from scratch based on the indications of se-
Rule #1 Time: Low, Reputation Value: Low Reputation Value: Medium curity and safety experts from each SG infrastructure (i.e., hydropower
Rule #2 Time: High, Reputation Value: Low Reputation Value: Medium
plant, substation, power plant and smart home). Due to the sensitive
Rule #3 Time: Low, Reputation Value: Medium Reputation Value: Medium
Rule #4 Time: High, Quantified Value: Medium Reputation Value: High
nature of these datasets, they cannot be published in the current work.
Rule #5 Time: Low, Quantified Value: High Reputation Value: High
Rule #6 Time: High, Quantified Value: High Reputation Value: High 4.3. Comparative methods

This subsection is devoted to the comparative methods used to

evaluate BDAC and VIDS. In particular, Section 4.3.1 is focused on the
3.6. Message bus
ML and DL comparative methods related to the BDAC Analysis Engine,
while Section 4.3.2 describes the ML and DL dimensionality reduction
Message Bus plays the role of a gateway providing a communication
methods of VIDS.
system among all SPEAR SIEM components that generate and handle
security events. It applies a publish–subscribe pattern based on Apache
4.3.1. BDAC comparative methods
Kafka. In particular, BDAC and VIDS (via the system operator or the se-
Multiple ML and DL methods were investigated and evaluated for
curity administrator) produce security events to the appropriate Apache
each detection model of the BDAC Analysis Engine. In particular,
Kafka topic of the Message Bus. In contrast, VIDS and GTM consume
regarding the detection models adopting a multiclass classification,
them in order to visualise them and compute the assets’ reputation
the following ML methods were used: Logistic Regression [57], Linear
value, respectively.
Discriminant Analysis (LDA) [58], Decision Tree Classifier [38], Naive
Bayes [59], SVM Linear [60], SVM RBF [60], Random Forest [61],
4. Evaluation analysis
Adaboost [53], Multi-Layer Perceptron (MLP) [62], Quadratic Discrim-
inant Analysis [63], K Nearest Neighbour (KNN) [64]. Moreover, three
This section focuses on evaluating the detection performance of custom DNNs were also used in our evaluation analysis. The first
SPEAR SIEM. First, Section 4.1 describes the evaluation environment. two called Dense DNN Relu and Dense DNN Tanh are originating
Next, Section 4.2 and Section 4.3 present the datasets and the compara- from our previous work in [29]. The remaining one called SDAE was
tive methods used in the evaluation analysis. Finally, Sections 4.4.1 and implemented during this work.
4.4.2 presents the evaluation results of BDAC and VIDS, respectively. The SPEAR SDAE is a DNN consisting of consequent encoding layers
of individual Denoising Autoencoders (DAEs), which can be considered
4.1. Evaluation environment a type of MLP. In the beginning, the original input data is used to
generate higher representation. Afterwards, the output of the first
The detection mechanisms of BDAC and VIDS were implemented trained DAE’s hidden layer is used as the next autoencoder’s input to
and evaluated, utilising real data originating from four SG use cases, extract higher representations. The training process of the SPEAR SDAE
namely (a) hydropower plant, (b) substation, (c) power plant and (e) consists of two phases. The first phase is the unsupervised layer-wise
smart home. The first three cases (i.e., hydropower plant, substation pre-training and the second phase is the supervised fine-tuning phase.
and power plant) use logic controllers, such as Programmable Logic During the first phase, each layer is trained separately. For the first
Controllers (PLCs) and Remote Terminal (RTUs) that monitor and phase, the labels are not required since the goal is to extract the feature
control the operation of the entire infrastructure and mainly that of representations from the input data. Then, after the training of all
industrial devices, such as turbines, transformers and generators. These layers, the fine-tuning phase starts, which is a backpropagation phase,
controllers communicate with a centralised server called Master Ter- using supervised training algorithms. This greedy layer-wise procedure
minal Unit (MTU) managed by the system operator through a Human has been shown to yield significantly better local minima than random
Machine Interface (HMI). In particular, through HMI, the system oper- initialisation of deep networks, achieving better generalisation on a
ator can monitor and handle the operation of PLCs and RTUs, sending number of tasks [65]. The SPEAR SDAE was tested to detect possible
the appropriate commands via the corresponding SG application-layer cyberattacks against MQTT and BACnet based on the corresponding
protocols (e.g., Modbus, DNP3 and IEC 61850). Finally, the smart home network flow statistics. In particular, during the training phase, it
environment involves smart metres that measure energy consumption receives as input 83 network flow-related statistic features and the label
and relevant statistics. This information is also stored in an MTU, for each MQTT or BACnet network flow. These features pass through
using the corresponding SG application-layer protocols. The SPEAR two or three encoder layers depending on the specific architecture of
Sensors were deployed in each SG infrastructure, using a Switched each protocol. Then, the representative features are extracted, passing
Port Analyser (SPAN). Therefore, the overall network traffic is directed through a final softmax classification layer with an equal number of
to the SPEAR sensors. In addition, the operational data of each SG nodes as the number of classes.
infrastructure is stored in MTU, which transmits them to DAPS. Regarding the models using outlier/novelty detection mechanisms
to identify whether there is an anomaly or not, the following ML
4.2. Datasets methods were evaluated: ABOD [44], Isolation Forest [42], PCA [66],
MCD [45,49] and LOF [47,48]. Furthermore, two DNNs were also
For each SG application-layer protocol mentioned above in Sec- adopted and evaluated. The first one is called DIDEROT Autoencoder
tion 3.3, appropriate datasets were formed and utilised to train and test and originates from our previous work in [41]. The second one was
the ML and DL models of the BDAC Analysis Engine. These datasets developed during this work. It relies on text CNN, which is a slight
were composed either by creating them from scratch with the emu- variant of a typical CNN. The difference between them is that in the
lation of the respective cyberattacks/anomalies or combining existing conventional CNNs, the sizes of filters in a single layer are usually the
intrusion datasets with the normal records coming from the aforemen- same. In contrast, in text-CNNs, the filters have a fixed width equal to
tioned SG use cases (i.e., hydropower plant, substation, power plant the embedding size of the input sentences but different heights. The
and smart home). New datasets were formed for the Modbus/TCP, IEC sentences are formed by parsing the SG application-layer payload of
60870-5-104, IEC 61850, BACnet and MQTT. In addition, the CSE- each packet and decomposing it into tokens. Each token is usually
CIC-IDS2018 dataset [34] was used for the HTTP and SSH. For the either a payload field or its value. The Payload Text CNN Classifier

14
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table 4 Table 7
Overview of the DeepDense Autoencoder. Overview of the classic dense sequence autoencoder.
Layer (type) Output Shape Param # Layer (type) Output Shape Param #
input_1(Input Layer) (None, inputDim) 0 dense_1 (Dense) (None,128) 57728
dense_1 (Dense) (None,128) 57728 dense_2 (Dense) (None,64) 8256
dense_2 (Dense) (None,64) 8256 dense_3 (Dense) (None,lowDim) 195
dense_3 (Dense) (None,32) 2080 dense_4 (Dense) (None,64) 256
dense_4 (Dense) (None,lowDim) 99 dense_5 (Dense) (None,128) 8320
dense_5 (Dense) (None,32) 128 dense_6 (Dense) (None,inputDim) 58050
dense_6 (Dense) (None,64) 2112
dense_7 (Dense) (None,128) 8320
dense_8 (Dense) (None,inputDim) 58050
the number of the correct classifications that recognised the normal
Table 5
behaviour activities as normal. On the other side, False Negatives (FN)
Overview of the FF autoencoder. denote the number of the wrong classifications that identified malicious
Layer (type) Output Shape Param # activities as normal. Finally, False Positives (FP) define the number
input_1(Input Layer) (None, inputDim) 0 of the incorrect classifications that detected the normal activities as
dense_1 (Dense) (None,lowDim) 1353 malicious or anomalous. Therefore, the following metrics are defined
dense_2 (Dense) (None,inputDim) 1800 (Eqs. (1)–(4)).
Accuracy (ACC) (equation (1)) indicates the ratio between the
Table 6 correct classifications and the total number of data samples. ACC can
Overview of the LSTM autoencoder. be utilised as an unbiased evaluation metric when the training dataset
Layer (type) Output Shape Param # comprises an equivalent quantity of data samples for all classes. For ex-
input_1(Input Layer) (None, inputDim,1) 0 ample, if the training dataset contains 90% data samples characterised
lstm_1 (LSTM) (None,lowDim) 60 as normal and 10% data samples as anomalous, then the ACC can reach
repeat_vector_1 (ReepatVector) (None,inputDim,3) 0 90% by classifying every case as normal.
lstm_2 (LSTM) (None,inputDim,1) 20
𝑇𝑃 + 𝑇𝑁
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦(𝐴𝐶𝐶) = (1)
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁
The False Positive Rate (FPR) (equation (2)) denotes the proportion of
consists of 3 layers. The first layer is an embedding layer, which normal behaviours recognised as malicious/anomalous. FPR is calcu-
transforms the words of each payload/sentence in word embeddings. lated by dividing FP with the sum of FP and TN.
Word embeddings are dense vectors representing the projection of the
𝐹𝑃
word into a continuous vector space. During the convolution process, 𝐹𝑃𝑅 = (2)
𝐹𝑃 + 𝑇𝑁
a filter 𝑤 of size ℎ𝑥𝑑 is applied in a window of ℎ words of the sentence
to extract a new feature. In particular, ℎ represents the height and 𝑑 The True Positive Rate (TPR) (equation (3)) determines what propor-
denotes the width of the token embeddings that form a sentence. This tion of actual malicious/anomalous activities was identified as mali-
filter is applied to each possible window generating a feature map. cious/anomalous. TPR is focused essentially on FN and is calculated by
After this procedure, a global max-pooling layer follows, extracting dividing TP with the sum of FN and TP.
the most important feature of each feature map. Filters of 3 different 𝑇𝑃
𝑇𝑃𝑅 = (3)
window sizes (4, 6, 8) are used in the different channels to extract more 𝑇𝑃 + 𝐹𝑁
features by processing 4-grams, 6-grams and 8-grams. Consequently, Finally, the F1 score (equation (4)) expresses the golden ratio between
the features from the global max-pooling layers are concatenated and the TPR and Precision, taking into account both FN and FP. Precision
passed through a dense feature layer and a final output layer. is another evaluation metric, which computes the proportion of those
data samples classified as malicious/anomalous.
4.3.2. VIDS comparative methods 2 × 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑇 𝑃 𝑅 𝑇𝑃
Concerning the visual-based detection mechanisms of VIDS, four 𝐹1 = where 𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = (4)
𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑇 𝑃 𝑅 𝑇𝑃 + 𝐹𝑃
ML dimensionality reduction methods are investigated and compared
with each other, including PCA [66], Singular Value Decomposition 4.4.1. BDAC evaluation results
(SVD) [67], Independent Component Analysis (ICA) [68] and Semi- This subsection summarises the evaluation results of the various in-
Random Projection (SRP) [69]. In addition, four DNNs, namely (a) trusion and anomaly detection models that compose the BDAC Analysis
DeepDense Autoencoder, (b) Feed Forward (FF) Autoencoder, (c) Long Engine. The comprehensive ML/DL comparative analysis of the BDAC
Short-Term Memory (LSTM) Autoencoder and (d) Classic Dense Se- evaluation results is provided by Tables C.10–C.28 in Appendix C.
quence Autoencoder were constructed during this work. Adam is It is noteworthy that all ML and DL methods were fine-tuned after
utilised as the optimisation method, the Mean Squared Error (MSE) several experiments. Fig. 13 summarises the detection performance of
is used for the loss, while the Rectified Linear Unit (ReLU) and sig- the BDAC network flow-based intrusion detection models. The Mod-
moid are used for the activation functions. Tables 4–7 summarise bus/TCP Network Flow-Based Intrusion Detection Model adopts a de-
these DNNs. For each ML/DL dimensionality reduction method, a cision tree, where 𝐴𝐶𝐶 = 0.964, 𝑇 𝑃 𝑅 = 0.749, 𝐹 𝑃 𝑅 = 0.019 and
time window of 30 instances (the most recent operational data of the 𝐹 1 = 0.749. Decision trees are efficient ML methods used for both
corresponding evaluation environment) is used as input. classification and regression problems. Their architecture consists of
internal nodes and leaves. The internal nodes and their edges separate
4.4. Evaluation results the whole space into smaller sub-spaces based on the training features.
In contrast, the leaves symbolise the various classes. Consequently,
Before proceeding to the analysis of the BDAC and VIDS detec- different paths are formed that can be translated into logical rules
tion performance, we need to introduce first the necessary back- leading to particular classes. In this paper, we use the Classification
ground terms. True Positives (TP) define the number of the correct and Regression Tree (CART) method with the Information Gain (IG)
classifications that detected the cyberattacks and anomalies as mali- criterion. More details about the decision trees are given in [38].
cious/anomalous behaviours. Accordingly, True Negatives (TN) denote The IEC 60870-5-104 Network Flow-Based Intrusion Detection Model

15
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 13. BDAC network flow-based intrusion detection models evaluation results.

Fig. 14. BDAC network flow-based & packet-based anomaly detection models evaluation results.

Fig. 15. BDAC operational data-based anomaly detection models evaluation results.

adopts also a CART decision tree whose ACC, TPR, FPR and the F1 score Detection Model and SSH Network Flow-Based Intrusion Detection
reach 0.953, 0.815, 0.026 and 0.815. On the other side, the BACnet Model use a CART decision tree classifier where their performance is
and the MQTT Network Flow-Based Intrusion Detection Models apply defined by the following metrics, respectively: 𝐴𝐶𝐶 = 0.964, 0.911,
the SPEAR SDAE method, which is analysed previously in Section 4.3. 0.026 and 0.911 and 𝐴𝐶𝐶 = 0.960, 𝑇 𝑃 𝑅 = 0.958, 𝐹 𝑃 𝑅 = 0.038 and
In the first case, the ACC, TPR, FPR and the F1 score reach 0.909, 𝐹 1 = 0.955.
0.991, 0.090 and 0.979, respectively. On the contrary, the efficiency of Fig. 14 illustrates the detection performance of those BDAC Anal-
the MQTT Network Flow-Based Intrusion Detection Model is reflected ysis Engine models detecting anomalies based on outlier or novelty
by the following metrics 𝐴𝐶𝐶 = 0.992, 𝑇 𝑃 𝑅 = 0.984, 𝐹 𝑃 𝑅 = 0.005 detection techniques. First, the Modbus Network Flow-Based Anomaly
and 𝐹 1 = 0.984. Finally, both HTTP Network Flow-Based Intrusion Detection Model utilises the DIDEROT autoencoder, where its detection

16
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Fig. 16. VIDS evaluation results.

performance is defined by 𝐴𝐶𝐶 = 0.950, 𝑇 𝑃 𝑅 = 0.999, 𝐹 𝑃 𝑅 = 𝑇 𝑃 𝑅 = 0.954, 𝐹 𝑃 𝑅 = 0.058 and 𝐹 1 = 0.938. In contrast, the evaluation
0.099 and 𝐹 1 = 0.952. The DIDEROT autoencoder is described by metrics of the SSH Network Flow-Based Anomaly Detection Model are
our previous work in [42]. In particular, it is a DNN composed of six 𝐴𝐶𝐶 = 0.957, 𝑇 𝑃 𝑅 = 0.970, 𝐹 𝑃 𝑅 = 0.050 and 𝐹 1 = 0.944.
fully connected layers that represent the encoder and decoder, evenly. Fig. 15 depicts the detection performance of the BDAC Opera-
Both the encoder and decoder map the input data 𝑥 to an output 𝑦. tional Data-Based Anomaly Detection Models. In particular, the ARIES
Based on the dimensionality reduction property, the training process GAN [29] is applied in the three of the four SG use cases: (a) hy-
intends to reduce the reconstruction error 𝐿(𝑥, 𝑦), which typically is dropower plant, (b) power plant and (c) smart home. As mentioned
the Euclidean distance in space 𝑋. The anomaly detection process in Section 2, the ARIES GAN is discussed in our previous work in [29].
is conducted by calculating and comparing the reconstruction error In contrast, in the substation use case, the LOF [47] method is used,
𝐿(𝑥, 𝑦) with a threshold 𝑇 , which is defined heuristically. In contrast, where 𝐴𝐶𝐶 = 0.873, 𝑇 𝑃 𝑅 = 0.993, 𝐹 𝑃 𝑅 = 0.157 and 𝐹 1 = 0.759.
the Modbus Packet-Based Anomaly Detection Model applies the isola- Regarding the ARIES GAN, the evaluation metrics in the hydropower
tion forest method [42], where 𝐴𝐶𝐶, 𝑇 𝑃 𝑅, 𝐹 𝑃 𝑅 and the 𝐹 1 score plant use case equal with 𝐴𝐶𝐶 = 0.746, 𝑇 𝑃 𝑅 = 0.978, 𝐹 𝑃 𝑅 = 0.311
are calculated at 0.943, 0.952, 0.062 and 0.930. The isolation forest and 𝐹 1 = 0.607. Similarly, the efficacy of the ARIES GAN in the power
method detects outliers or differently anomalies by intentionally ‘‘over- plant use case is reflected by 𝐴𝐶𝐶 = 0.851, 𝑇 𝑃 𝑅 = 0.982, 𝐹 𝑃 𝑅 = 0.188
fitting’’ a function memorising each data point. Since the data space and 𝐹 1 = 0.755.
is relatively empty around outliers/anomalies, the function requires
fewer memorisation steps. To this end, full decision trees are used, 4.4.2. VIDS evaluation results
calculating the path length between the root and each leaf (data point).
This subsection is devoted to evaluating the detection performance
The final measure for each data point is the average path length, which
of VIDS. The detailed ML/DL comparative analysis is provided by
is relatively short. Similarly, the IEC 60870-5-104 Network Flow-Based
Tables D.29–D.32 in Appendix D. As illustrated by Fig. 16, almost
Anomaly Detection Model adopts the isolation forest, where 𝐴𝐶𝐶 =
in all SG use cases the LSTM-Autoencoder presents the best efficacy
0.948, 𝑇 𝑃 𝑅 = 0.967, 𝐹 𝑃 𝑅 = 0.074, 𝐴𝐶𝐶 = 0.952. However, on
in terms of ACC and the F1 score. Only, in the smart home envi-
the other side, the IEC 60870-5-104 Packet-Based Anomaly Detection
ronment, the FF-Autoencoder overcomes the LSTM-Autoencoder. Both
Model utilises the LOF method [47]. The evaluation metrics for this
LSTM-Autoencoder and FF-Autoencoder are detailed in Section 4.3.
model are 𝐴𝐶𝐶 = 0.926, 𝑇 𝑃 𝑅 = 0.859, 𝐹 𝑃 𝑅 = 0.005, 𝐴𝐶𝐶 = 0.921.
The LOF functionality relies on the local density. An outlier/anomaly
5. Conclusions
is detected by comparing the local density of the point investigated with
the local density of its neighbours. The locality is provided by KNN [64]
through which the density is estimated by measuring their distance. Although the modern electrical grid provides several benefits, such
When the density of the point investigated is significantly lower than as pervasive control and self-healing, it involves crucial cybersecurity
its neighbours’ density, it is considered an outlier/anomaly. The IEC risks. In particular, the combination of the insecure SG communi-
61850 (MMS) Network Flow-Based Anomaly Detection Model applies cation protocols, the IoT security issues and the rapid evolution of
the MCD method [49] with 𝐴𝐶𝐶 = 0.981, 𝑇 𝑃 𝑅 = 0.986, 𝐹 𝑃 𝑅 = 0.22 cyberattacks and malware can lead to disastrous consequences, such
and 𝐹 1 = 0.977. The MCD is a robust estimator of multivariate scatter as extensive blackouts and brownouts. The SIEM systems constitute
and location. Its resiliency to the masking effect, makes it efficient to a state-of-the-art cybersecurity technology, which can organise and
detect outliers/anomalies. M. Hubert and D. Michiel in [49] provide manage the monitoring, detection and prevention measures.
a detailed description about MCD, using simplified examples. Next, In this work, we presented the SPEAR SIEM, which focuses on the
both the BACnet Packet-Based Intrusion Detection Model and the MQTT peculiarities of SG. In particular, SPEAR SIEM is composed of four
Packet-Based Anomaly Detection Model adopt the Payload Text CNN main components, namely (a) SPEAR SIEM Basis, (b) BDAC, (c) VIDS
described earlier in Section 4.3. The detection performance of the first and (d) GTM. SPEAR SIEM Basis undertakes to monitor the infras-
is reflected by the following metrics 𝐴𝐶𝐶 = 0.967, 𝑇 𝑃 𝑅 = 0.967, tructure, thus providing the necessary data to the other components.
𝐹 𝑃 𝑅 = 0.032 and 𝐹 1 = 0.982. Similarly, the performance of the MQTT Next, BDAC integrates a set of ML/DL-based intrusion and anomaly
Packet-Based Anomaly Detection Model is defined by 𝐴𝐶𝐶 = 0.985, detection models related to the SG communication protocols and SG
𝑇 𝑃 𝑅 = 0.985, 𝐹 𝑃 𝑅 = 0.014 and 𝐹 1 = 0.985. Finally, the HTTP operational data (i.e., time-series electricity measurements). Next, VIDS
Network Flow-Based Anomaly Detection Model and the SSH Network is a parallel detection and correlation mechanism, which relies on
Flow-Based Anomaly Detection Model use LOF and MCD, respectively. visual analytics. Finally, GTM correlates the various security events and
The detection performance of the first is reflected by 𝐴𝐶𝐶 = 0.946, computes the reputation value of each SG asset. The evaluation analysis

17
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table A.8
SPEAR SIEM security event format.
Security event field name Security event field description
SPEAR Component Identifier of the SPEAR SIEM component, which generates the security event. Three options are available: AlienVault
OSSIM, BDAC and VIDS.
Date Date and time of the security event.
Sensor The sensor, which processed the security event.
Device IP The IP address of the sensor, which processed the security event.
Event Type ID Identifier assigned by the component, which generates the security event.
Unique Event ID Unique identifier assigned by the component, which generates the security event.
Protocol Protocol related to the security event.
Category Event taxonomy for the security event. In the context of BDAC and VIDS, it is ‘‘Cyberattack’’ or ‘‘Anomaly’’.
Subcategory Subcategory of the security event taxonomy type listed under Category. In the context of BDAC and VIDS, it is a
specific cyberattack or anomaly.
Data Source Name Name of the external application or device that produced the security event. In the context of BDAC and VIDS, it
related to VIDS itself or the internal modules of BDAC.
Data Source ID Identifier related to the external application or device which generated the security event. In the context of BDAC and
VIDS, it is related to the internal modules of BDAC or VIDS itself.
Product Type Product type related to the security event.
Additional Info Uniform Resource Locator (URL) including more details about the security event.
Priority It reflects the significance of the security event in the range between 0–5.
Reliability It reflects the detection reliability in the range between 0–10.
Risk Risk calculation relies on the formula: Asset Value * Event Reliability * Event Priority / 25
OTX Indicators Number of indicators related to an OTX IP reputation or OTX pulse event. In the context of BDAC and VIDS, it is null.
Source/Destination ID Identifier of the source/destination related to the security event.
Source/Destination IP IP addresses of source/destination, respectively related to security event.
Source/Destination Hostname Hostname of source/destination.
Source/Destination MAC Address Media Access Control (MAC) of source/destination.
Source/Destination Port Port of source/destination.
Source/Destination Latest Update The last time when the component, which generated the security event updated the source/destination properties.
Source/Destination Username and Domain Username and domain related to source/destination.
Source/Destination Asset Value Asset value of source/destination. It reflects the significance of source/destination.
Source/Destination Location If the origin of source/destination is known, it reflects the host country.
Source/Destination Context If the asset belongs to a user-defined group of entities, AlienVault OSSIM shows the relevant contexts. In the context of
BDAC and VIDS, it is null.
Source/Destination Asset Groups When the source/destination belongs to one or more asset groups, this field lists the asset group name or names.
Source/Destination Networks When the source/destination belongs to one or more networks, this field lists the networks.
Source/Destination Logged Users A list of users and their information related to source/destination.
Source/Destination OTX IP Reputation (Yes or No) Whether or not the OTX IP Reputation identifies the IP address as suspicious.
Source/Destination Service List of services or applications related to the source/destination ports.
Service Port Port utilised by the service or application.
Service Protocol Protocol utilised by the service or application.
Raw Log Raw log details of the security event.
Filename Name of a file related to the security event.
Username Usernames related to the security event.
Password Passwords related to the security event.
Userdata 1-9 User-generated log fields.
Rule Detection AlienVault OSSIM NIDS rule used to detect the security event. In the context of BDAC and VIDS, BDAC internal
modules and VIDS itself are used, respectively.

Table B.9
VIDS correlation rules for modbus.
No Description
Rule #1 If there are 𝑋 or more consecutive events denoting a modbus/function/readInputRegister (DoS) attack, then an alert called
‘modbus/function/readInputRegister (DoS)’ is raised. 𝑋 is defined by the user.
Rule #2 If there are 𝑋 or more consecutive events denoting a modbus/dos/writeSingleRegister attack, then an alert called
‘modbus/dos/writeSingleRegister’ is raised. 𝑋 is defined by the user.
Rule #3 If there are 𝑋 or more consecutive events denoting a modbus/function/readDiscreteInputs (DoS) attack, then an alert called
‘modbus/function/readDiscreteInputs (DoS)’ is raised. 𝑋 is defined by the user.
Rule #4 If there are 𝑋 or more consecutive events denoting a modbus/function/readHoldingRegister (DoS) attack, then an alert called
‘modbus/function/readHoldingRegister (DoS)’ is raised. 𝑋 is defined by the user.
Rule #5 If there are 𝑋 or more consecutive events denoting a modbus/function/readCoils (DoS) attack, then an alert called
‘modbus/function/readCoils (DoS))’ is raised. 𝑋 is defined by the user.
Rule #6 If there are 𝑋 or more consecutive events denoting a modbus/dos/writeSingleCoils attack, then an alert called
‘modbus/dos/writeSingleCoils’ is raised. 𝑋 is defined by the user.
Rule #7 If there are 𝑋 events denoting a modbus/scanner/uid attack and right after 𝑋 events denoting a modbus/scanner/getfunc, then an alert
called ‘Modbus Reconnaissance’. 𝑋 is defined by the user.
Rule #8 If there are 𝑋 or more consecutive events denoting a modbus/scanner/getfunc attack, then an alert called ‘Modbus Reconnaissance’ is
raised. 𝑋 is defined by the user.
Rule #9 If there are 𝑋 or more consecutive events denoting a modbus/scanner/uid attack, then an alert called ‘Modbus Reconnaissance’ is
raised. 𝑋 is defined by the user.
Rule #10 If there are 𝑋 events denoting a modbus/scanner/uid attack and right after 𝑋 events denoting a modbus/function/writeSingleCoils,
then an alert called ‘modbus/function/writeSingleCoils’ is raised. 𝑋 is defined by the user.
Rule #11 If there are 𝑋 events denoting a modbus/scanner/getfunc attack and right after 𝑋 events denoting a modbus/function/writeSingleCoils,
then an alert called ‘modbus/function/writeSingleCoils’ is raised. 𝑋 is defined by the user.
(continued on next page)

18
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table B.9 (continued).

No Description
Rule #12 If there are 𝑋 or more consecutive events denoting a modbus/function/writeSingleCoils, then an alert called
‘modbus/function/writeSingleCoils’ is raised. 𝑋 is defined by the user.
Rule #13 If there are 𝑋 events denoting a modbus/scanner/uid attack and right after 𝑋 events denoting a modbus/function/readInputRegister,
then an alert called ‘modbus/function/readInputRegister’ is raised. 𝑋 is defined by the user.
Rule #14 If there are 𝑋 events denoting a modbus/scanner/getfunc attack and right after 𝑋 events denoting a
modbus/function/readInputRegister, then an alert called ‘modbus/function/readInputRegister’ is raised. 𝑋 is defined by the user.
Rule #15 If there are 𝑋 or more consecutive events denoting a modbus/function/readInputRegister, then an alert called
‘modbus/function/readInputRegister’ is raised. 𝑋 is defined by the user.
Rule #16 If there are 𝑋 events denoting a modbus/scanner/uid attack and right after 𝑋 events denoting a modbus/function/writeSingleRegister,
then an alert called ‘modbus/function/writeSingleRegister’ is raised. 𝑋 is defined by the user.
Rule #17 If there are 𝑋 events denoting a modbus/scanner/getfunc attack and right after 𝑋 events denoting a
modbus/function/writeSingleRegister, then an alert called ‘modbus/function/writeSingleRegister’ is raised. 𝑋 is defined by the user.
Rule #18 If there are 𝑋 or more consecutive events denoting a modbus/function/writeSingleRegister, then an alert called
‘modbus/function/writeSingleRegister’ is raised. 𝑋 is defined by the user.
Rule #19 If there are 𝑋 events denoting a modbus/scanner/uid attack and right after 𝑋 events denoting a modbus/function/readDiscreteInput,
then an alert called ‘modbus/function/readDiscreteInput’ is raised. 𝑋 is defined by the user.
Rule #20 If there are 𝑋 events denoting a modbus/scanner/getfunc attack and right after 𝑋 events denoting a
modbus/function/readDiscreteInput, then an alert called ‘modbus/function/readDiscreteInput’ is raised. 𝑋 is defined by the user.
Rule #21 If there are 𝑋 or more consecutive events denoting a modbus/function/readDiscreteInput, then an alert called
‘modbus/function/readDiscreteInput’ is raised. 𝑋 is defined by the user.
Rule #22 If there are 𝑋 events denoting a modbus/scanner/uid attack and right after 𝑋 events denoting a modbus/function/readHoldingRegister,
then an alert called ‘modbus/function/readHoldingRegister’ is raised. 𝑋 is defined by the user.
Rule #23 If there are 𝑋 events denoting a modbus/scanner/getfunc attack and right after 𝑋 events denoting a
modbus/function/readHoldingRegister, then an alert called ‘modbus/function/readHoldingRegister’ is raised. 𝑋 is defined by the user.
Rule #24 If there are 𝑋 or more consecutive events denoting a modbus/function/readHoldingRegister, then an alert called
‘modbus/function/readHoldingRegister’ is raised. 𝑋 is defined by the user.

Table C.10 CRediT authorship contribution statement

Modbus/TCP network flow-based intrusion detection model evaluation results.
Classification problem Multi-class classification
Panagiotis Radoglou-Grammatikis: Conceptualization, Methodol-
Data Type Network flow statistics (related only to
ogy, Software, Validation, Formal analysis, Writing - original draft.
Modbus/TCP network flows specified by the
502 TCP port) Panagiotis Sarigiannidis: Supervision, Project administration, Fund-
Features All features exported by CICFlowMeter [54],
ing acquisition, Writing - review & editing, Resources. Eider Iturbe:
excluding FlowID, SrcIP, DstIP and Timestamp Conceptualization, Methodology, Software, Investigation, Writing -
Total Dataset Size 255000 Modbus flows original draft. Erkuden Rios: Methodology, Resources, Investigation,
Writing - review & editing. Saturnino Martinez: Software, Validation,
Training Dataset Size 70%
Data curation. Antonios Sarigiannidis: Formal analysis, Investigation,
Tesing Dataset Size 30%
Writing - review & editing, Visualization. Georgios Eftathopoulos:
ML/DL Method ACC TPR FPR F1
Software, Investigation, Visualization. Yannis Spyridis: Software, In-
Logistic Regression 0.943 0.603 0.030 0.603 vestigation, Visualization. Achilleas Sesis: Software, Investigation, Vi-
LDA 0.943 0.604 0.030 0.604
Decision Tree Classifier 0.964 0.749 0.019 0.749
sualization. Nikolaos Vakakis: Software, Investigation, Visualization,
Naïve Bayes 0.928 0.497 0.038 0.497 Resources. Dimitrios Tzovaras: Formal analysis, Investigation, Writing
SVM RBF 0.918 0.426 0.044 0.426 - review & editing, Supervision, Resources. Emmanouil Kafetzakis:
SVM Linear 0.921 0.453 0.042 0.453 Formal analysis, Investigation, Writing - review & editing. Ioannis Gi-
Random Forest 0.947 0.633 0.028 0.633
MLP 0.938 0.570 0.033 0.570
annoulakis: Investigation, Writing - review & editing. Michalis Tzifas:
Adaboost 0.887 0.214 0.060 0.214 Investigation, Writing - review & editing. Alkiviadis Giannakou-
Quadratic Discriminant Analysis 0.941 0.593 0.031 0.593 lias: Formal analysis, Investigation, Resources, Validation. Michail
Dense DNN Relu 0.945 0.619 0.029 0.619 Angelopoulos: Resources, Validation. Francisco Ramos: Resources,
Dense DNN Tanh 0.945 0.619 0.029 0.619
Validation.

Declaration of competing interest

demonstrates the efficiency and applicability of SPEAR SIEM in four SG
environments, namely (a) hydropower plant, (b) substation, (c) power The authors declare that they have no known competing finan-
plant and (d) smart home. cial interests or personal relationships that could have appeared to
Our future plans related to this work include the incorporation of influence the work reported in this paper.
more intrusion and anomaly detection models in the BDAC Analysis
Engine that will focus on Profinet and EtherCAT. Moreover, appropriate Acknowledgement
association rules will be investigated in order to correlate security
events related to other industrial protocols. To this end the Apriori and This project has received funding from the European Union’s Hori-
Eclat ML methods will be investigated. Finally, appropriate self-healing zon 2020 Research and Innovation Programme under grant agreement
mechanisms will be examined to be integrated into SPEAR SIEM, taking No. 787011 (SPEAR).
full advantage of the network automation capabilities offered by the
Software-Defined Networking (SDN) technology. In particular, the SDN Appendix A. SPEAR SIEM security event format
controller will be able to mitigate the potential malicious flows or
re-arrange them, thus ensuring the stability of the SG infrastructure. Table A.8 summarises the format of the SPEAR security events.

19
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table C.11 Table C.14

Modbus/TCP network flow-based anomaly detection model evaluation results. IEC 60870-5-104 network flow-based anomaly detection model evaluation results.
Classification problem Outlier/novelty detection Classification problem Outlier/Novelty detection
Data type Network flow statistics (related only to Modbus/TCP Data Type
network flows specified by the 502 TCP port)
Features Flow Duration, TotLen Fwd Pkts, Fwd Pkt Len
Features Flow Duration, TotLen Fwd Pkts, Fwd Pkt Len Mean, Mean, Bwd Pkt Len Std, Flow IAT Std, Bwd Pkts/s,
Bwd Pkt Len Std, Flow IAT Std, Bwd Pkts/s, Subflow Subflow Fwd Byts, Init Fwd Win Byts, Active Mean
Fwd Byts, Init Fwd Win Byts, Active Mean
Total Dataset Size 100000 IEC 60870-5-104 flows
Total Dataset Size 255000 Modbus flows
Training Dataset Size 70%
Training Dataset Size 70%
Tesing Dataset Size 30%
Tesing Dataset Size 30%
ML/DL Method ACC TPR FPR F1
ML/DL Method ACC TPR FPR F1
ABOD 0.937 0.947 0.074 0.942
ABOD 0.949 0.999 0.100 0.951 Isolation Forest 0.948 0.967 0.074 0.952
Isolation Forest 0.950 0.999 0.099 0.952 PCA 0.699 0.892 0.347 0.537
PCA 0.540 0.846 0.567 0.488 MCD 0.948 0.999 0.102 0.950
MCD 0.948 0.999 0.102 0.950 LOF 0.953 0.941 0.038 0.941
LOF 0.947 0.999 0.104 0.950 DIDEROT Autoencoder 0.881 0.852 0.089 0.877
DIDEROT Autoencoder 0.950 0.999 0.099 0.952

Table C.15
Table C.12
IEC 60870-5-104 packet-based anomaly detection model evaluation results.
Modbus/TCP packet-based anomaly detection model evaluation results.
Classification problem Outlier/Novelty detection
Classification problem Outlier/Novelty Detection
Data type Attributes of IEC 60870-5-104 Payload
Data Type Attributes of Modbus/TCP Payload
Features frame_length, testfr_con, testfr_act, stopdt_con,
Features TCP-LEN, TRANSACTION-ID, PROTOCOL-ID,
stopdt_act, startdt_con, startdt_act
UNIT-ID, FCODE, LEN, START-ADDR,
BYTE-COUNT Total Dataset Size 100000 IEC 60870-5-104 packets
Total Dataset Size 255000 Modbus packets Training Dataset Size 75%
Training Dataset Size 70% Tesing Dataset Size 25%
Tesing Dataset Size 30% ML/DL Method ACC TPR FPR F1
ML/DL Method ACC TPR FPR F1 ABOD 0.508 0.452 0.440 0.466
Isolation Forest 0.893 0.860 0.074 0.889
ABOD 0.819 0.800 0.166 0.784
PCA 0.535 0.500 0.431 0.513
Isolation Forest 0.943 0.952 0.062 0.930
MCD 0.734 0.594 0.125 0.691
PCA 0.909 0.869 0.062 0.888
LOF 0.926 0.859 0.005 0.921
MCD 0.905 0.857 0.062 0.878
DIDEROT Autoencoder 0.748 0.568 0.072 0.692
LOF 0.943 0.952 0.062 0.930
DIDEROT Autoencoder 0.888 0.074 0.898 0.968

Table C.16
Table C.13
IEC 61850 (MMS) network flow-based anomaly detection model evaluation results.
IEC 60870-5-104 network flow-based intrusion detection model evaluation results.
Classification problem Outlier/Novelty Detection
Classification Problem Multi-class classification
Data Type Network flow statistics (related only to MMS
Data type Network flow statistics (related only to IEC
network flows identified by the 102 TCP port)
60870-5-104 network flows specified by the
2404 TCP port) Features Flow Duration, TotLen Fwd Pkts, Fwd Pkt Len
Mean, Bwd Pkt Len Std, Flow IAT Std, Bwd Pkts/s,
Features Flow Duration, TotLen Fwd Pkts, Fwd Pkt
Subflow Fwd Byts, Init Fwd Win Byts, Active Mean
Len Mean, Bwd Pkt Len Std, Flow IAT Std,
Bwd Pkts/s, Subflow Fwd Byts, Init Fwd Total Dataset Size 80000 IEC 61850 (MMS) flows
Win Byts, Active Mean Training Dataset Size 75%
Total Dataset Size 100000 IEC 60870-5-104 flows Testing Dataset Size 25%
Training Dataset Size 70% ML/DL Method ACC TPR FPR F1
Tesing Dataset Size 30% ABOD 0.973 0.970 0.024 0.966
ML/DL Method ACC TPR FPR F1 Isolation Forest 0.977 0.971 0.019 0.971
PCA 0.506 0.524 0.511 0.514
Logistic Regression 0.900 0.602 0.056 0.602
MCD 0.981 0.986 0.022 0.977
LDA 0.904 0.619 0.054 0.619
LOF 0.954 0.924 0.022 0.945
Decision Tree Classifier 0.953 0.815 0.026 0.815
DIDEROT Autoencoder 0.960 0.982 0.115 0.9743
Naïve Bayes 0.855 0.421 0.082 0.421
SVM RBF 0.853 0.413 0.083 0.413
SVM Linear 0.843 0.375 0.089 0.375
Random Forest 0.918 0.672 0.046 0.672
MLP 0.904 0.619 0.054 0.619
Adaboost 0.843 0.375 0.089 0.375 Appendix C. BDAC evaluation results — comprehensive ML/dl
Quadratic Discriminant Analysis 0.899 0.598 0.057 0.598 comparative analysis
Dense DNN Relu 0.909 0.636 0.051 0.636
Dense DNN Tanh 0.916 0.664 0.047 0.664
The Appendix C presents the ML/DL comparative analysis related
to the intrusion and anomaly detection models of the BDAC Analysis
Appendix B. VIDS correlation rules for modbus Engine. In particular, Tables C.10–C.28 reflect this evaluation process.
It is worth noting that all ML and DL methods were fine-tuned after
Table B.9 summarises the VIDS correlation rules for Modbus. several experiments.

20
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table C.17
BACnet network flow-based intrusion detection model evaluation results. Table C.20
Classification Problem Multi-class Classification MQTT packet-based anomaly detection model evaluation results.
Classification Problem Outlier/Novelty Detection
Data Type Attributes of BACnet Payload
Data Type Attributes of MQTT Pyload
Features All features exported by CICFlowMeter [54],
excluding FlowID, SrcIP, SrcPort, DstIP DstPort, Features MQTT payload text is parsed and split into tokens
Protocol and Timestamp using ntlk regular expression tokenizer. The result
is a sentence with tokens for each packet.
Total Dataset Size 100000 BACnet flows
Total Dataset Size 90000 MQTT packets
Training Dataset Size 70%
Training Dataset Size 75%
Tesing Dataset Size 30%
Tesing Dataset Size 25%
ML/DL Method ACC TPR FPR F1
ML/DL Method ACC TPR FPR F1
Logistic Regression 0.960 0.982 0.115 0.974
Naïve Bayes 0.902 0.925 0.115 0.894 Multinomial Naïve Bayes 0.728 0.728 0.271 0.667
KNN 0.934 0.968 0.090 0.928 Logistic Regression 0.890 0.890 0.109 0.880
SVM RBF 0.924 0.952 0.090 0.897 SVM RBF 0.890 0.890 0.109 0.880
SPEAR SDAE 0.909 0.991 0.090 0.979 Payload text CNN 0.985 0.985 0.014 0.985
Random Forest 0.959 0.969 0.090 0.972

Table C.18
BACnet packet-based intrusion detection model evaluation results. Table C.21
Classification problem Outlier/Novelty Detection HTTP network flow-based intrusion detection model evaluation results.

Data Type Attributes of BACnet Payload Classification Problem Multi-Class Classification

Features BACnet payload text is parsed and split into Data Type Network flow statistics (related only to HTTP(S)
tokens, using the ntlk regular expression tokenizer. network flows identified by the 80 TCP port)
The result is a sentence composed of tokens for Features All features exported by CICFlowMeter [37],
each packet. excluding FlowID, SrcIP, DstIP and Timestamp
Total Dataset Size 100000 BACnet packets Total Dataset Size 150000 HTTP flows
Training Dataset Size 75% Training Dataset Size 70%
Tesing Dataset Size 25% Tesing Dataset Size 30%
ML/DL Method ACC TPR FPR F1 ML/DL Method ACC TPR FPR F1
Multinomial Naïve Bayes 0.771 0.661 0.339 0.761 Logistic Regression 0.937 0.844 0.038 0.844
Logistic Regression 0.808 0.850 0.330 0.872 LDA 0.946 0.866 0.033 0.866
SVM RBF 0.962 0.961 0.032 0.978 Decision Tree Classifier 0.964 0.911 0.026 0.911
Payload Text CNN 0.967 0.967 0.032 Naïve Bayes 0.878 0.696 0.075 0.696
SVM RBF 0.908 0.770 0.057 0.770
SVM Linear 0.928 0.822 0.044 0.822
Random Forest 0.922 0.807 0.048 0.807
Table C.19 MLP 0.940 0.851 0.037 0.851
MQTT network flow-based intrusion detection model evaluation results. Adaboost 0.760 0.400 0.150 0.400
Classification Problem Multi-Class Classification Quadratic Discriminant Analysis 0.911 0.777 0.055 0.777
Dense DNN Relu 0.940 0.851 0.037 0.851
Data Type Network flow statistics (related only to MQTT Dense DNN Tanh 0.940 0.851 0.0370 0.851
network traffic identified by the 1883/8883 TCP
ports)
Features All features exported by CICFlowMeter [54],
excluding FlowID, SrcIP, SrcPort, DstIP DstPort,
Protocol and Timestamp
Total Dataset Size 90000 MQTT flows Table C.22
HTTP network flow-based anomaly detection model evaluation results.
Training Dataset Size 75%
Classification Problem Out-
Tesing Dataset Size 25% lier/Novelty
ML/DL Method ACC TPR FPR F1 Detection

Logistic Regression 0.939 0.878 0.040 0.863 Data Type Network flow statistics (related only to HTTP(S)
Naïve Bayes 0.869 0.739 0.086 0.761 network flows identified by the 80 TCP port)
KNN 0.941 0.950 0.065 0.926 Features All features exported by CICFlowMeter [37],
SVM RBF 0.956 0.913 0.028 0.907 excluding FlowID, SrcIP, DstIP and Timestamp
Random Forest 0.970 0.967 0.017 0.982
Total Dataset Size 150000 HTTP flows
SPEAR SDAE 0.992 0.984 0.005 0.984
Training Dataset Size 70%
Tesing Dataset Size 30%
ML/DL Method ACC TPR FPR F1

Appendix D. VIDS evaluation results — comprehensive ML/dl ABOD 0.577 0.571 0.416 0.558
Isolation Forest 0.833 0.948 0.281 0.850
comparative analysis
PCA 0.596 0.592 0.400 0.581
MCD 0.719 0.545 0.106 0.660
The Appendix D shows the ML/DL comparative analysis related to LOF 0.946 0.954 0.058 0.938
the intrusion and anomaly detection models of VIDS. In particular, DIDEROT Autoencoder 0.934 0.927 0.061 0.902
Tables D.29–D.32 reflect this evaluation process. It is worth noting that
all ML and DL methods were fine-tuned after several experiments.

21
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table C.23
Table C.26
SSH network flow-based intrusion detection model evaluation results.
Operational data based anomaly detection model — substation use case.
Classification Problem Multi-Class Classification
Classification Problem Outlier/Novelty Detection
Data Type Network flow statistics (related only to SSH
Data Type Operational Data — Substation Use Case
network flows identified by the 22 TCP port)
Features ACTIVE_POWER_SOE, APPARENT_POWER_SOE,
Features Dst Port, Flow Duration, TotLen Fwd Pkts,
CURRENT_SOE, FRECUENCY_SOE
Fwd Pkt Len Mean, Bwd Pkt Len Std,
REACTIVE_POWER_SOE, TEMPERATURE_SOE,
Flow IAT Std, Bwd Pkts/s, Subflow Fwd Byts,
TRAFOS_POSITION_SOE, VOLTAGE_SOE
Init Fwd Win Byts, Active Mean
Total Dataset Size 10000 time-series
Total Dataset Size 10000 SSH flows
Training Dataset Size 70%
Training Dataset Size 70%
Tesing Dataset Size 30%
Tesing Dataset Size 30%
ML/DL Method ACC TPR FPR F1
ML/DL Method ACC TPR FPR F1
ABOD 0.839 0.995 0.200 0.713
Logistic Regression 0.859 0.750 0.058 0.821
Isolation Forest 0.850 0.951 0.175 0.718
LDA 0.945 0.920 0.038 0.928
PCA 0.847 0.961 0.181 0.716
Decision Tree Classifier 0.960 0.958 0.038 0.955
MCD 0.822 0.991 0.220 0.691
Naïve Bayes 0.823 0.741 0.154 0.640
LOF 0.873 0.993 0.157 0.759
SVM RBF 0.837 0.660 0.339 0.788
ARIES GAN 0.840 0.961 0.189 0.708
SVM Linear 0.799 0.845 0.307 0.307
Random Forest 0.955 0.903 0.009 0.942
MLP 0.903 0.841 0.010 0.910
Adaboost 0.950 0.890 0.010 0.934
Quadratic Discriminant Analysis 0.500 0.500 0.250 0.666
Dense DNN Relu 0.916 0.985 0.014 0.906
Dense DNN Tanh 0.916 0.836 0.011 0.904 Table C.27
Operational data based anomaly detection model — power plant use case.
Classification Problem Outlier/Novelty Detection
Data Type Operational Data — Power Plant Use Case
Features v24_batteries, v60_batteries, generator_speed,
Table C.24
gen_motor_voltage, gen_motor_current,
SSH network flow-based anomaly detection model evaluation results.
exc_motor_voltage, exc_motor_current,
Classification Problem Outlier/Novelty Detection incom_cooling_water, gen_status_winding2,
Data Type Network flow statistics (related only to SSH gen_outlet_air, exc_set_bearing2, grid_phase_r,
network flows identified by the 22 TCP port) grid_phase_s, grid_phase_t, main_mg_nn, exc_mg_nn,
overvolt_main_gen, overcur_main_gen,
Features Flow Duration, TotLen Fwd Pkts, Fwd Pkt Len
rem_command, com_fault
Mean, Bwd Pkt Len Std, Flow IAT Std, Bwd Pkts/s,
Subflow Fwd Byts, Init Fwd Win Byts, Active Mean Total Dataset Size 10000 time-series

Total Dataset Size 10000 SSH flows Training Dataset Size 70%

Training Dataset Size 70% Tesing Dataset Size 30%

Tesing Dataset Size 30% ML/DL Method ACC TPR FPR F1

ML/DL Method ACC TPR FPR F1 ABOD 0.692 0.989 0.397 0.600
Isolation Forest 0.813 0.960 0.231 0.705
ABOD 0.935 0.870 0.013 0.922
PCA 0.851 0.982 0.187 0.755
Isolation Forest 0.943 0.901 0.013 0.941
MCD 0.715 0.299 0.158 0.329
PCA 0.701 0.596 0.247 0.564
LOF 0.829 0.992 0.220 0.730
MCD 0.957 0.970 0.050 0.944
ARIES GAN 0.851 0.982 0.188 0.755
LOF 0.925 0.913 0.066 0.909
DIDEROT Autoencoder 0.946 0.954 0.058 0.938

Table C.28
Table C.25
Operational data based anomaly detection model — smart home use case.
Operational data based anomaly detection model – hydropower plant use case.
Classification Problem Outlier/Novelty Detection
Classification Problem Outlier/Novelty Detection
Data Type Operational Data — Smart Home Use Case
Data Type Oprational Data — Hydropower Plant Use Case
Features AoutPhL1, AoutPhL2, AoutPhL3, BattAmp,
Features ’DE’, ’power’, ’waterlevel’, ’NDE’, ’nozzles’
BattTemp, BattVolt, PinPhL1, PinPhL2, PinPhL3,
Total Dataset Size 10000 time-series PoutPhL1, PoutPhL2, PoutPhL3, VoutPhL1,
Training Dataset Size 70% VoutPhL2, VoutPhL3

Tesing Dataset Size 30% Total Dataset Size 10000 time-series

ML/DL Method ACC TPR FPR F1 Training Dataset Size 70%

ABOD 0.581 0.993 0.522 0.487 Tesing Dataset Size 30%

Isolation Forest 0.716 0.948 0.341 0.572 ML/DL Method ACC TPR FPR F1
PCA 0.745 0.978 0.312 0.606
ABOD 0.649 0.668 0.362 0.597
MCD 0.733 0.210 0.135 0.240
Isolation Forest 0.769 0.976 0.279 0.615
LOF 0.579 0.996 0.525 0.486
PCA 0.859 0.976 0.167 0.724
ARIES GAN 0.746 0.978 0.311 0.607
MCD 0.729 0.992 0.332 0.581
LOF 0.690 0.735 0.344 0.676
ARIES GAN 0.859 0.976 0.167 0.725

22
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Table D.29 Table D.32

VIDS evaluation results — hydropower plant use case. VIDS evaluation results — smart home use case.
Classification Problem Visual-based Anomaly Detection Classification Problem Visual-based Anomaly Detection
Data Type Oprational Data — Hydropower Plant Use Case Data Type Oprational Data — Power Plant Use Case
Features ’DE’, ’power’, ’waterlevel’, ’NDE’, ’nozzles’ Features AoutPhL1, AoutPhL2, AoutPhL3, BattAmp,
BattTemp, BattVolt, PinPhL1, PinPhL2, PinPhL3,
Total Dataset Size 10000 time-series
PoutPhL1, PoutPhL2, PoutPhL3, VoutPhL1,
Training Dataset Size 70% VoutPhL2, VoutPhL3
Tesing Dataset Size 30% Total Dataset Size 10000 time-series
ML/DL Method 2D-ACC 3D-ACC 2D-F1 3D-F1 Training Dataset Size 70%
PCA 0.8626 0.3568 0.7618 0.4205 Tesing Dataset Size 30%
SVD 0.8912 0.3881 0.7954 0.4172
ML/DL Method 2D-ACC 3D-ACC 2D-F1 3D-F1
ICA 0.8357 0.6738 0.7291 0.5635
SRP 0.7806 0.2344 0.6442 0.3797 PCA 0.9220 0.9229 0.8351 0.8286
Deep Dense Autoencoder 0.7354 0.4207 0.0782 0.4089 SVD 0.910 0.940 0.105 0.881
FF Autoencoder 0.2344 0.2344 0.4780 0.3797 ICA 0.8814 0.8551 0.7799 0.7341
LSTM Autoencoder 0.9830 0.9500 0.9637 0.8819 SRP 0.8822 0.7178 0.7818 0.5801
Classic Dense Sequence 0.2344 0.3548 0.3797 0.4111 Deep Dense Autoencoder 0.9212 0.8864 0.8318 0.7528
Autoencoder FF Autoencoder 0.9280 0.9254 0.8468 0.8358
LSTM Autoencoder 0.7627 0.6890 0.6577 0.5954
Classic Dense Sequence 0.9017 0.8881 0.7943 0.7740
Table D.30 Autoencoder
VIDS evaluation results — substation use case.
Classification Problem Visual-based Anomaly Detection
Data Type Oprational Data — Substation Use Case
Features ACTIVE_POWER_SOE, APPARENT_POWER_SOE,
CURRENT_SOE, FRECUENCY_SOE
REACTIVE_POWER_SOE, TEMPERATURE_SOE, References
TRAFOS_POSITION_SOE, VOLTAGE_SOE
Total Dataset Size 10000 time-series [1] H. Farhangi, The path of the smart grid, IEEE Power Energy Mag. 8 (1) (2009)
18–28.
Training Dataset Size 70% [2] P. Kumar, Y. Lin, G. Bai, A. Paverd, J.S. Dong, A. Martin, Smart grid metering
Tesing Dataset Size 30% networks: A survey on security, privacy and open research issues, IEEE Commun.
Surv. Tutor. 21 (3) (2019) 2886–2927.
ML/DL Method 2D-ACC 3D-ACC 2D-F1 3D-F1
[3] S. Kwon, H. Yoo, T. Shon, IEEE 1815.1-based power system security with
PCA 0.6210 0.2754 0.5637 0.4318 bidirectional RNN-based network anomalous attack detection for cyber-physical
SVD 0.7184 0.3999 0.6361 0.1597 system, IEEE Access 8 (2020) 77572–77586.
ICA 0.7615 0.5376 0.6573 0.0647 [4] S. Tan, D. De, W.-Z. Song, J. Yang, S.K. Das, Survey of security advances in
SRP 0.8053 0.7316 0.7260 0.2431 smart grid: A data driven approach, IEEE Commun. Surv. Tutor. 19 (1) (2017)
Deep Dense Autoencoder 0.7573 0.2754 0.6654 0.4318 397–422.
FF Autoencoder 0.6412 0.2754 0.5575 0.4318 [5] P. Radoglou-Grammatikis, P. Sarigiannidis, I. Giannoulakis, E. Kafetzakis, E.
LSTM Autoencoder 0.8122 0.8491 0.8443 0.7815 Panaousis, Attacking IEC-60870-5-104 SCADA systems, in: 2019 IEEE World
Classic Dense Sequence 0.7643 0.2754 0.5285 0.4318 Congress on Services (SERVICES), vol. 2642-939X, 2019, pp. 41–46.
Autoencoder [6] P.I. Radoglou Grammatikis, P.G. Sarigiannidis, I.D. Moscholios, Securing the
internet of things: Challenges, threats and solutions, Internet of Things 5 (2019)
41–70, http://dx.doi.org/10.1016/j.iot.2018.11.003, http://www.sciencedirect.
Table D.31
com/science/article/pii/S2542660518301161.
VIDS evaluation results — power plant use case.
[7] S.S. Hussain, T.S. Ustun, A. Kalam, A review of IEC 62351 security mechanisms
Classification Problem Visual-based Anomaly Detection for IEC 61850 message exchanges, IEEE Trans. Ind. Inf. (2019).
Data Type Oprational Data — Power Plant Use Case [8] T.S. Ustun, S.S. Hussain, IEC 62351-4 security implementations for IEC 61850
MMS messages, IEEE Access 8 (2020) 123979–123985.
Features v24_batteries, v60_batteries, generator_speed,
[9] R. Schlegel, S. Obermeier, J. Schneider, Assessing the security of IEC 62351, in:
gen_motor_voltage, gen_motor_current,
3rd International Symposium for ICS & SCADA Cyber Security Research 2015
exc_motor_voltage, exc_motor_current,
(ICS-CSR 2015) 3, 2015, pp. 11–19.
incom_cooling_water, gen_status_winding2,
[10] S. Bhatt, P.K. Manadhata, L. Zomlot, The operational role of security information
gen_outlet_air, exc_set_bearing2, grid_phase_r,
and event management systems, IEEE Secur. Priv. 12 (5) (2014) 35–41.
grid_phase_s, grid_phase_t, main_mg_nn, exc_mg_nn,
[11] P. Radoglou-Grammatikis, P.G. Sarigiannidis, Securing the smart grid: A compre-
overvolt_main_gen, overcur_main_gen,
hensive compilation of intrusion detection and prevention systems, IEEE Access
rem_command, com_fault
7 (2019) 46595–46620.
Total Dataset Size 10000 time-series [12] S.M. Milajerdi, R. Gjomemo, B. Eshete, R. Sekar, V. Venkatakrishnan, Holmes:
Training Dataset Size 70% real-time apt detection through correlation of suspicious information flows, in:
2019 IEEE Symposium on Security and Privacy (SP), IEEE, 2019, pp. 1137–1152.
Tesing Dataset Size 30% [13] AlienVault, AlienVault OSSIM Documentation, 2020, https://www.alienvault.
ML/DL Method 2D-ACC 3D-ACC 2D-F1 3D-F1 com/documentation/.
[14] P. Radoglou-Grammatikis, P. Sarigiannidis, E. Iturbe, E. Rios, A. Sarigiannidis,
PCA 0.8571 0.6466 0.7879 0.0371
O. Nikolis, D. Ioannidis, V. Machamint, M. Tzifas, A. Giannakoulias, M. An-
SVD 0.8187 0.8701 0.7494 0.7873
gelopoulos, A. Papadopoulos, F. Ramos, Secure and private smart grid: The
ICA 0.8037 0.7007 0.7274 0.6396
SPEAR architecture, in: 2020 6th IEEE Conference on Network Softwarization
SRP 0.8126 0.2973 0.7360 0.4583
(NetSoft), 2010, pp. 450–456.
Deep Dense Autoencoder 0.7969 0.6116 0.7084 0.5723
[15] S.G. Zarzosa, D2.1 in-depth analysis of SIEMs extensibility, Tech. Rep. 1, DiSIEM
FF Autoencoder 0.7861 0.9143 0.7257 0.8657
Project, 2017.
LSTM Autoencoder 0.7027 0.9776 0.8298 0.9631
Classic Dense Sequence 0.7782 0.9085 0.7009 0.8501 [16] L. Cui, Y. Qu, L. Gao, G. Xie, S. Yu, Detecting false data attacks using machine
Autoencoder learning techniques in smart grid: A survey, J. Netw. Comput. Appl. (2020)
102808.
[17] S.E. Quincozes, C. Albuquerque, D. Passos, D. Mossé, A survey on intrusion
detection and prevention systems in digital substations, Comput. Netw. 184
(2020) 107679.

23
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

[18] M.Z. Gunduz, R. Das, Cyber-security on smart grid: Threats and potential [46] N.R. Rodofile, K. Radke, E. Foo, Framework for SCADA cyber-attack
solutions, Comput. Netw. 169 (2020) 107094. dataset creation, in: Proceedings of the Australasian Computer Science Week
[19] M. Hassan, M.H. Rehmani, J. Chen, Differential privacy techniques for cyber Multiconference, 2017, pp. 1–10.
physical systems: A survey, IEEE Commun. Surv. Tutor. 22 (1) (2020) 746–789, [47] L. You, Q. Peng, Z. Xiong, D. He, M. Qiu, X. Zhang, Integrating aspect analysis
http://dx.doi.org/10.1109/COMST.2019.2944748. and local outlier factor for intelligent review spam detection, Future Gener.
[20] I. Stellios, P. Kotzanikolaou, M. Psarakis, C. Alcaraz, J. Lopez, A survey of Comput. Syst. 102 (2020) 163–172.
IoT-enabled cyberattacks: Assessing attack paths to critical infrastructures and [48] S. Mishra, M. Chawla, A comparative study of local outlier factor algorithms for
services, IEEE Commun. Surv. Tutor. 20 (4) (2018) 3453–3495, http://dx.doi. outliers detection in data streams, in: Emerging Technologies in Data Mining and
org/10.1109/COMST.2018.2855563. Information Security, Springer, 2019, pp. 347–356.
[21] R. Leszczyna, M.R. Wróbel, Evaluation of open source SIEM for situation aware- [49] M. Hubert, M. Debruyne, Minimum covariance determinant, Wiley Interdiscip.
ness platform in the smart grid environment, in: 2015 IEEE World Conference Rev. Comput. Stat. 2 (1) (2010) 36–43.
on Factory Communication Systems (WFCS), IEEE, 2015, pp. 1–4. [50] G. Apruzzese, M. Andreolini, M. Colajanni, M. Marchetti, Hardening random
[22] C. iView, Cyberoam iview centralized logging & reporting for distributed forest cyber detectors against adversarial attacks, IEEE Trans. Emerg. Top.
enterprises and MSSPs, Tech. Rep. 1, Cyberoam iView, 2020. Comput. Intell. (2020).
[23] T. Andrejak, Prelude-SIEM documentation release 4.0, Tech. Rep. 4, CS, 2017. [51] R.U. Khan, X. Zhang, M. Alazab, R. Kumar, An improved convolutional neural
[24] B. Sahay, A. Gupta, Development of software selection criteria for supply chain network model for intrusion detection in networks, in: 2019 Cybersecurity and
solutions, Ind. Manage. Data Syst. (2003). Cyberforensics Conference (CCC), IEEE, 2019, pp. 74–77.
[52] D.S. Berman, A.L. Buczak, J.S. Chavis, C.L. Corbett, A survey of deep learning
[25] K. Zhang, S. Luo, Y. Xin, H. Zhu, Y. Chen, Online mining intrusion patterns from
methods for cyber security, Information 10 (4) (2019) 122.
IDS alerts, Appl. Sci. 10 (8) (2020) 2983.
[53] D. Tang, L. Tang, R. Dai, J. Chen, X. Li, J.J. Rodrigues, Mf-adaboost: Ldos attack
[26] M. Albanese, R.F. Erbacher, S. Jajodia, C. Molinaro, F. Persia, A. Picariello, G.
detection based on multi-features and improved adaboost, Future Gener. Comput.
Sperlì, V. Subrahmanian, Recognizing unexplained behavior in network traffic,
Syst. 106 (2020) 347–359.
in: Network Science and Cybersecurity, Springer, 2014, pp. 39–62.
[54] A. Subasi, E. Kremic, Comparison of adaboost with multiboosting for phishing
[27] M. Albanese, C. Molinaro, F. Persia, A. Picariello, V. Subrahmanian, Finding"
website detection, Procedia Comput. Sci. 168 (2020) 272–278.
unexplained" activities in video., in: IJCAI, 2011, pp. 1628–1634.
[55] D. Pliatsios, P. Sarigiannidis, T. Liatifis, K. Rompolos, I. Siniosoglou, A novel and
[28] K. Zhang, F. Zhao, S. Luo, Y. Xin, H. Zhu, An intrusion action-based IDS
interactive industrial control system honeypot for critical smart grid infrastruc-
alert correlation analysis and prediction framework, IEEE Access 7 (2019)
ture, in: 2019 IEEE 24th International Workshop on Computer Aided Modeling
150540–150551.
and Design of Communication Links and Networks (CAMAD), 2019, pp. 1–6.
[29] P. Radoglou-Grammatikis, P. Sarigiannidis, E. Georgios, P. Emmanouil, ARIES: [56] P. Radoglou-Grammatikis, P. Sarigiannidis, T. Liatifis, T. Apostolakos, S.
A novel multivariate intrusion detectionsystem for smart grid, Sensors (2020). Oikonomou, An overview of the firewall systems in the smart grid paradigm,
[30] G. Efstathopoulos, P.R. Grammatikis, P. Sarigiannidis, V. Argyriou, A. Sarigian- in: 2018 Global Information Infrastructure and Networking Symposium (GIIS),
nidis, K. Stamatakis, M.K. Angelopoulos, S.K. Athanasopoulos, Operational data IEEE, 2018, pp. 1–4.
based intrusion detection system for smart grid, in: 2019 IEEE 24th International [57] Y. Wang, A multinomial logistic regression modeling approach for anomaly
Workshop on Computer Aided Modeling and Design of Communication Links and intrusion detection, Comput. Secur. 24 (8) (2005) 662–674.
Networks (CAMAD), IEEE, 2019, pp. 1–6. [58] H. Li, L. Zhang, B. Huang, X. Zhou, Cost-sensitive dual-bidirectional linear
[31] M. Ali, S. Shiaeles, G. Bendiab, B. Ghita, MALGRA: Machine learning and N-gram discriminant analysis, Inform. Sci. 510 (2020) 283–303.
malware feature extraction and detection system, Electronics 9 (11) (2020) 1777. [59] S. Mukherjee, N. Sharma, Intrusion detection using naive Bayes classifier with
[32] M. Ghafouri, M. Au, M. Kassouf, M. Debbabi, C. Assi, J. Yan, Detection and feature reduction, Proc. Technol. 4 (2012) 119–128.
mitigation of cyber attacks on voltage stability monitoring of smart grids, IEEE [60] M. Awad, R. Khanna, Support vector machines for classification, in: Efficient
Trans. Smart Grid 11 (6) (2020) 5227–5238. Learning Machines: Theories, Concepts, and Applications for Engineers and
[33] M. Tsoukalos, Using tshark to watch and inspect network traffic, Linux J. 2015 System Designers, Apress, Berkeley, CA, 2015, pp. 39–66, http://dx.doi.org/10.
(254) (2015) 1. 1007/978-1-4302-5990-9_3, https://doi.org/10.1007/978-1-4302-5990-9_3.
[34] I. Sharafaldin, A.H. Lashkari, A.A. Ghorbani, Toward generating a new intrusion [61] S. Liu, L. Liu, Y. Fan, L. Zhang, Y. Huang, T. Zhang, J. Cheng, L. Wang, M. Zhang,
detection dataset and intrusion traffic characterization, in: ICISSP, 2018, pp. R. Shi, D. Mao, An integrated scheme for online dynamic security assessment
108–116. based on partial mutual information and iterated random forest, IEEE Trans.
[35] G.F. Lyon, Nmap Network Scanning: The Official Nmap Project Guide to Network Smart Grid 11 (4) (2020) 3606–3619.
Discovery and Security Scanning, Insecure, Sunnyvale, CA, USA, 2009. [62] P. Radoglou-Grammatikis, P.G. Sarigiannidis, Flow anomaly based intrusion de-
[36] I. Siniosoglou, G. Efstathopoulos, D. Pliatsios, I.D. Moscholios, A. Sarigiannidis, tection system for Android mobile devices, in: 2017 6th International Conference
G. Sakellari, G. Loukas, P. Sarigiannidis, Neuralpot: An industrial honeypot on Modern Circuits and Systems Technologies (MOCAST), 2017, pp. 1–4.
implementation based on deep neural networks, in: 2020 IEEE Symposium on [63] B. Ghojogh, M. Crowley, Linear and quadratic discriminant analysis: Tutorial,
Computers and Communications (ISCC), IEEE, 2020, pp. 1–7. 2019, arXiv preprint arXiv:1906.02590.
[37] P. Diamantoulakis, C. Dalamagkas, P. Radoglou-Grammatikis, P. Sarigiannidis, [64] H. Yang, S. Liang, J. Ni, H. Li, X. Shen, Secure and efficient kNN classification
G. Karagiannidis, Game theoretic honeypot deployment in smart grid, Sensors for industrial internet of things, IEEE Internet Things J. (2020) 1.
20 (15) (2020) 4199. [65] Y. Kim, Convolutional neural networks for sentence classification, 2014, arXiv
[38] P. Radoglou-Grammatikis, P.G. Sarigiannidis, An anomaly-based intrusion detec- preprint arXiv:1408.5882.
tion system for the smart grid based on CART decision tree, in: 2018 Global [66] A.A. Imayakumar, A. Dubey, A. Bose, Anomaly detection for primary distribution
Information Infrastructure and Networking Symposium (GIIS), 2018, pp. 1–5. system measurements using principal component analysis, in: 2020 IEEE Texas
[39] J. Luswata, P. Zavarsky, B. Swar, D. Zvabva, Analysis of SCADA security using Power and Energy Conference (TPEC), IEEE, 2020, pp. 1–6.
penetration testing: A case study on modbus TCP protocol, in: 2018 29th Biennial [67] O. Edfors, M. Sandell, J.-J. Van de Beek, S.K. Wilson, P.O. Borjesson, OFDM
Symposium on Communications (BSC), IEEE, 2018, pp. 1–5. Channel estimation by singular value decomposition, IEEE Trans. Commun. 46
(7) (1998) 931–939.
[40] P. Radoglou-Grammatikis, I. Sinisioglou, T. Liatifis, A. Kourouniadis, K. Rompo-
[68] A. Tharwat, Independent component analysis: An introduction, Appl. Comput.
los, P. Sarigiannidis, Implementation and detection of modbus cyberattacks, in:
Inform. (2020).
2020 9th International Conference on Modern Circuits and Systems Technologies
[69] R. Zhao, K. Mao, Semi-random projection for dimensionality reduction and
(MOCAST), 2020, pp. 1–4.
extreme learning machine in high-dimensional space, IEEE Comput. Intell. Mag.
[41] P. Radoglou-Grammatikis, P. Sarigiannidis, G. Efstathopoulos, P.-A. Karypidis, A.
10 (3) (2015) 30–41.
Sarigiannidis, DIDEROT: An intrusion detection and prevention system for DNP3-
based SCADA systems, in: Proceedings of the 15th International Conference on
Availability, Reliability and Security, in: ARES ’20, Association for Computing
Machinery, New York, NY, USA, 2020, pp. 1–8, http://dx.doi.org/10.1145/ Panagiotis Radoglou Grammatikis received the Diploma
3407023.3409314, https://doi.org/10.1145/3407023.3409314. degree (MEng, 5 years) from the Dept. of Informatics
[42] S. Hariri, M.C. Kind, R.J. Brunner, Extended isolation forest, 2018, arXiv preprint and Telecommunications Eng. (now Dept. of Electrical and
arXiv:1811.02141. Computer Eng.), Faculty of Eng., University of Western
[43] A. Gharib, I. Sharafaldin, A.H. Lashkari, A.A. Ghorbani, An evaluation framework Macedonia, Greece, in 2016. He is now a PhD candidate
for intrusion detection dataset, in: 2016 International Conference on Information in the same department. His main research interests are
Science and Security (ICISS), IEEE, 2016, pp. 1–6. in the area of cybersecurity and mainly focus on intrusion
[44] H.-P. Kriegel, M. Schubert, A. Zimek, Angle-based outlier detection in high- detection, vulnerability research and applied cryptography.
dimensional data, in: Proceedings of the 14th ACM SIGKDD International He has published 17 research papers in international scien-
Conference on Knowledge Discovery and Data Mining, 2008, pp. 444–452. tific journals, conferences and book chapters, including IEEE
[45] Y. Zhao, Z. Nasrullah, Z. Li, Pyod: A python toolbox for scalable outlier detection, Access, Computer Networks (ELSEVIER), Internet of Things
2019, arXiv preprint arXiv:1901.01588. (ELSEVIER) and Sensors (MDPI). Moreover, he received the

24
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

Best Paper award in 2019 IEEE International Workshop on project on Secure and Privacy-aware Smart IoT Systems.
Computer Aided Modeling and Design of Communication Previously, she was the coordinator of the H2020 MUSA
Links and Networks (IEEE CAMAD). He has served as a re- project on Multicloud Security, successfully ended in 2017,
viewer for several scientific journals and possesses working as well as the chair of the Data Protection, Security and
experience as a security engineer and software developer. Privacy in Cloud Cluster of EU-funded research projects,
Currently, he is working as a research associate at the launched by DG-CNECT in April 2015. Furthermore, she
University of Western Macedonia in national and European has worked in multiple large European and Spanish projects
funded research projects, including (a) H2020-DS-SC7-2017 on cybersecurity and trust such as POSEIDON, PDP4E,
(DS-07-2017), SPEAR: Secure and PrivatE smArt gRid, TACIT, RISC, ANIKETOS, SWEPT, CIPHER and SHIELDS.
(b) H2020-SU-DS-2018 (SU-DS04-2018), SDN-microSENSE: Her main research interests include Trust and Security,
SDN-microgrid reSilient Electrical eNergy SystEm, (c) MARS: Risk Management, and AI for Cybersecurity. Mrs. Erkuden
sMart fArming with dRoneS (Competitiveness, Entrepreneur- collaborates with Technology Platforms and Forums such
ship, and Innovation), (d) H2020-ICT-2020-1 (ICT-56-2020) as Cybersecurity PPP ECSO, ETSI Secure Artificial Intelli-
TERMINET: nexT gEneRation sMart INterconnectEd ioT and gence Working Group, AIOTI WG4 Policy and Privacy and
(e) H2020-LC-SC3-EE-2020-1 (LC-SC3-EC-4-2020) EVIDENT: the Spanish National Network on Cybersecurity. She has
bEhaVioral Insgihts anD Effective eNergy policy acTions. been member of Programme Committees of Journals and
Finally, he is a member of the IEEE and the Technical Conferences including the Journal of Computer Networks
Chamber of Greece. and Communications, the International Workshop on Model-
Prof. Panagiotis Sarigiannidis is an Associate Professor in Driven Engineering for the Internet-of-Things, the ARES
the Department of Electrical and Computer Engineering in Conference on Availability, Reliability and Security, the
the University of Western Macedonia, Kozani, Greece since IEEE International Conference on Cloud Engineering (IC2E),
2016. He received the B.Sc. and Ph.D. degrees in computer The International Workshop on Security Testing And Mon-
science from the Aristotle University of Thessaloniki, Thes- itoring, The Workshop on Security, Privacy, and Identity
saloniki, Greece, in 2001 and 2007, respectively. He has Management in the Cloud, the Cybersecurity and Privacy
published over 180 papers in international journals, confer- Innovation Forum, CSP Forum and SECSE.
ences and book chapters, including IEEE Communications
Surveys and Tutorials, IEEE Transactions on Communi- Saturnino Martinez is computer engineer from the Uni-
cations, IEEE Internet of Things, IEEE Transactions on versity of Deusto, Bilbao, Spain, in 1999, and in 2013 he
Broadcasting, IEEE Systems Journal, IEEE Wireless Com- obtained the M.S. degree in Software Architecture from
munications Magazine, IEEE/OSA Journal of Lightwave the same University. Satur Martinez works as cybersecurity
Technology, IEEE Access, and Computer Networks. He has researcher in the TRUSTECH unit of Fundación Tecnalia
been involved in several national, European and inter- Research & Innovation, Derio, Spain. His research interests
national projects. He is currently the project coordinator are focused on Cybersecurity in all its aspects, machine
of three H2020 projects, namely (a) H2020-DS-SC7-2017 learning, Artificial Intelligence and Big Data, where he
(DS-07-2017), SPEAR: Secure and PrivatE smArt gRid, accumulates more than 20 years of experience. Before
(b) H2020-LC-SC3-EE2020-1 (LC-SC3-EC-4-2020), EVIDENT: joining Tecnalia, he worked for more than 15 years at Panda
bEhaVioral Insgihts anD Effective eNergy policy acTions, Security as a cybersecurity researcher and threat hunter.
and (c) H2020-ICT-2020-1 (ICT-56-2020), TERMINET: nexT During this period, I worked on many projects applying
gEneRation sMart INterconnectEd ioT, while he coordi- ML and DL in the IT Cybersecurity environment, such as
nates the Operational Program MARS: sMart fArming with classification and categorisation of Malware, prioritisation
dRoneS (Competitiveness, Entrepreneurship, and Innova- of incidents, graphical analysis of incidents, etc. He also
tion) and the Erasmus+ KA2 ARRANGE-ICT: SmartROOT: defined and designed the Panda EDR as well as the Threat
Smart faRming innOvatiOn Training. He also serves as a Hunting platform that exploited said data. In Tecnalia, he
principal investigator in the H2020-SU-DS-2018 (SU-DS04- has participated in multiple European H2020 cybersecurity
2018), SDN-microSENSE: SDN-microgrid reSilient Electrical projects in the industrial environment, mainly in electricity
eNergy SystEm and in three Erasmus+ KA2: (a) ARRANGE- sector and in the IoT, creating SIEM tools oriented to
ICT: pArtneRship foR AddressiNG mEgatrends in ICT, (b) OT environments and applying ML and DL techniques to
JAUNTY: Joint undergAduate coUrses for smart eNergy anomaly detection. He has also been professor of malware
managemenT sYstems, and (c) STRONG: advanced firST analysis and wireless network security in the Master of
RespONders traininG (Cooperation for Innovation and the Security at the University of Deusto.
Exchange of Good Practices). His research interests include
Dr. Antonios Sarigiannidis received the B.Sc. degree in
telecommunication networks, internet of things and network
Information Technology from the Aristotle University of
security. He is an IEEE member and participates in the
Thessaloniki in 2007 and the M.Sc. degree in Communi-
Editorial Boards of various journals, including International
cation Systems and Technologies, specialising in advanced
Journal of Communication Systems and EURASIP Journal
optical and wireless technologies, from the Aristotle Uni-
on Wireless Communications and Networking.
versity of Thessaloniki in 2009. He obtained his Ph.D. in
Eider Iturbe received the M.S. degree in telecommunication Information Technology from the Aristotle University of
engineering from University of Basque Country, Bilbao, Thessaloniki in 2016. His Ph.D. thesis includes the develop-
Spain, in 2003 and graduated in the European Master of ment of bandwidth allocation algorithms in Communication
Project Management in the same university, in 2013. From Networks. His research interests include machine learning
2003 to 2009, she worked for software consultancy firms mechanism and optimisation techniques as well as visuali-
where she acquired management skills and a great technical sation techniques regarding analytics, big data and security
expertise in the cybersecurity field. She is currently senior analysis. Recently, he has been involved in IoT and M2M
scientist of Cybersecurity research team of TRUSTECH unit research towards in coverage analysis and security services.
in Fundación Tecnalia Research & Innovation, Derio, Spain. He actively participated in both national and EU funded
Moreover, she has worked in multiple large European and projects. He is the author of more than 30 publications in
Spanish research projects on cybersecurity, privacy and leading international journals and conferences.
trust such as SPEAR, ENACT, POSEIDON, PDP4E, TACIT,
Georgios Efstathopoulos studied in National Technical
MUSA, SWEPT and CIPHER. Her main research interests
University of Athens, where he received the Diploma of
include Cyber Security, Privacy, Risk Management, and AI
Electrical and Computer Engineer with distinction. He re-
for Cybersecurity.
ceived his Ph.D. degree under the supervision of Professor
Erkuden Rios received the Ph.D., M.S. and B.S. degree in A. Manikas in the Communications and Signal Processing
telecommunication engineering from University of Basque Group, Department of Electrical and Electronic Engineering,
Country, Bilbao, Spain, in 1997. After working six years for Imperial College London. Also, he worked as software
Ericsson Spain, currently she is senior scientist of Cybersecu- developer and quantitative analyst at the Investment Bank
rity research team of TRUSTECH unit in Fundación Tecnalia sector. He has been working as a quantitative analyst in
Research & Innovation, Derio, Spain. She is currently the the financial sector for the last 9 years. Over the last 3
coordinator of the Security WP in the H2020 SPEAR project years, Georgios has been actively involved in a number of
on Secure Smart Grids as well as in the H2020 ENACT data analytics, machine learning and AI projects in various

25
P. Radoglou-Grammatikis et al. Computer Networks 193 (2021) 108008

industries, which includes autonomous vehicles, finance, Dr. Ioannis Giannoulakis received his Ph.D. from the
smart grid, insurance and healthcare sectors. National Technical University of Athens, Greece in the
area of applied mathematics. He is author/co-author of
several books, scientific papers and reports for technical,
business and societal aspects of the modern telecommunica-
Yannis Spyridis is a part-time lecturer in the Computer
tion systems and in the area of cybersecurity, privacy, and
Science department of the University of Sheffield Interna-
data protection. Furthermore, he has been involved in the
tional Faculty - CITY College. He received a B.Sc. (Hons)
study of the business perspectives and the commercialisation
in Computer Science in 2018 and in the same year, he
roadmaps of several European research projects that are
started studying towards a Ph.D. degree at the Department
directly related to cybersecurity.
of Electronic and Electrical Engineering of The University
of Sheffield. He is also working as a research associate
in several European projects, funded by the H2020 and
Mr. Michalis Tzifas holds a B.Sc. in Physics from University
Erasmus+ programmes. His main research interest evolves
of Athens and a M.Sc. degree in Technical Applications from
around the areas of IoT distributed architectures and wire-
the National Technical University of Athens. He possesses
less communications in unmanned aerial vehicle (UAV)
expertise in a range of modern wireless communications
networks, as well as security and privacy in smart grids.
systems, including LTE, 5G and Next Generation networks,
and he is mostly experienced in the experimental evaluation
and assessment of performance as well as in networking
Dr. Achilleas Sesis has a background in aerospace engi- cybersecurity issues.
neering, materials science and nanotechnology, and after
experiencing the relevant academic and industrial re-
search sectors for over 12 years, in 2015 he moved into
entrepreneurship focusing on AI and machine learning ap-
plications related to NLP, anomaly detection and visual
analytics. He has work on several projects in the field Dr. Alkiviadis Giannakoulias holds a Ph.D. in ‘‘Wireless
education, Industry 4.0, Internet of Things and artificial Communication and Computer Systems’’ from the National
intelligence. As well as an avid knowledge seeker, he also Technical University of Athens (NTUA) and a BEng. and
is a keen promoter of collaborative approaches in business M.Sc. from the University of KENT at Canterbury. He is
and life. an experienced Information Security Analyst/Manager with
expert knowledge and experience in cybersecurity on a
technical, operational and strategic level, acquired through
relevant research activities at various positions, including
Mr. Vakakis Nikolaos has been a research associate in the
those of manager, expert and analyst. Demonstrated his-
Information Technologies Institute of Center for Research
tory in the area of ICT security auditing, risk assessment
and Technology Hellas, since July 2018. He received the
and management, ICT security policy specifications, de-
5 years diploma in Electrical and Computer Engineering
velopment, best practices and implementation, cyber crisis
from Aristotle University of Thessaloniki, Greece (2017),
management, incident handling and response, penetration
with specialisation in electronics and computer engineering.
testing and network forensics, cybersecurity capacity build-
Currently he is finalising his studies in the M.Sc. Inter-
ing, education and training. ICT technical knowledge and
departmental/Interdisciplinary Postgraduate Programme on
experience, and proven professional understanding of cy-
Advanced Computer and Communication Systems from Aris-
bersecurity issues on Healthcare, Transport, Energy and
totle University of Thessaloniki, specialising in IoT and web
utilities vertical sectors. Background in economics, laws
technologies, by conducting his thesis titled ‘‘Towards a
and communication resulting in proven insight and proven
verifiable, smart contract-based e-auction framework’’. He
understanding and interest into cybersecurity, acquired
is involved in several H2020 R&D projects and his main
through relevant EU-funded research projects. Expertise,
research interests include blockchain applications in the
knowledge and experience in the areas of: Economics of
energy domain, cybersecurity and anomaly detection in IoT,
cybersecurity, Knowledge management on cybersecurity,
artificial intelligence, data analytics and data visualisation
Legal aspects of cybersecurity, Cybercrime investigations,
techniques and finally semantic web technologies in web
Communication, dissemination and/or awareness raising on
accessibility domain.
cybersecurity, Quantitative and qualitative methods with
application on cybersecurity. He speaks fluent English and
has basic knowledge of German. Finally, he has published
Dr. Dimitrios Tzovaras received the Diploma and the Ph.D. several articles in magazines and international conferences.
degree in electrical and computer engineering from Aristotle
University of Thessaloniki, Thessaloniki, Greece, in 1992
and 1997, respectively. He is the Director at the Infor-
mation Technologies Institute of the Centre for Research Dr. Francisco Ramos received his Ph.D. in 2018, his M.S.
and Technology Hellas. His main research interests include in Electrical Engineering and M.S. in Physics (majoring in
visual analytics, 3D object recognition, search and retrieval, Electronics) in 1997 and his M.Sc. Degree in Computer
behavioural biometrics, assistive technologies, information & Network Engineering in 2012, all from the University
and knowledge management, computer graphics, and virtual of Granada. He received an Executive MBA from Instituto
reality. Internacional San Telmo-IESE in 2007. He started working
as Designer Engineer in Alcatel in Brussels in 1997. In
1998 he joined Eben Technologies in Paris as Consulting
Dr. Emmanouil Kafetzakis received his B.Sc. degree in Engineer and in 1999 he moved to Switzerland joining
Informatics and Telecommunications from the University of EMMicroelectronics-Swatch Group as Project Manager. In
Athens, Greece, in 2002, and his M.Sc. and Ph.D. degrees 2002 he was appointed as Product Manager and Head of
from the same university, in 2004 and 2011. In the past Product Line. In 2005 he joined Telvent part of Schneider
he has collaborated with the Institute of Informatics and Electric in Sevilla where he was RTU Product Manager for
Telecommunications of the National Centre for Scientific the Electric Sector. In 2007 he was appointed as R&D Area
Research ‘‘Demokritos’’, having an active role in the ICT Director. He is currently working as Innovation Manager at
and cybersecurity national and European projects. He has Schneider Electric in charge of R&D collaborative projects
served also as a Research Assistant in the Department of In- in the domains of Smart Grids, Cybersecurity, Renewables,
formatics and Telecommunications of University of Athens. Smart Cities, Internet of the Things and Embedded Systems.
His research interests include network security protocols and He has participated in more than 100 R&D projects in
information security governance and risk management. He programs such H2020, FP7, ECSEL and Artemis among
holds Ericsson Award of Excellence in Telecommunications. others.

26