2023 International Conference on Machine Learning and Applications (ICMLA)

Improving Transparency and Explainability of Deep

Learning based IoT Botnet Detection using
Explainable Artificial Intelligence (XAI)
Rajesh Kalakoti Sven Nõmm Hayretdin Bahsi
Department of Software Science Department of Software Science Department of Software Science
School of Information Technology, School of Information Technology School of Information Technology,
Taltech, Tallinn, Estonia. Taltech, Tallinn, Estonia Taltech, Tallinn, Estonia
2023 International Conference on Machine Learning and Applications (ICMLA) | 979-8-3503-4534-6/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICMLA58977.2023.00088

[email protected] [email protected] [email protected]

Abstract—Ensuring the utmost security of IoT systems is ing higher accuracy even at the expense of interpretability,
imperative, and robust botnet detection plays a pivotal role creating complex and challenging-to-understand models [4].
in achieving this goal. Deep learning-based approaches have The European Union’s General Data Protection Regulation
been widely employed for botnet detection. However, the lack of
interpretability and transparency in these models can limit these (GDPR) has addressed the issue of explainability, emphasizing
models’ effectiveness. In this research, we present a Deep Neural the importance of understanding the underlying logic behind
Network (DNN) model specifically designed for the detection Artificial Intelligence algorithmic decisions that may adversely
of IoT botnet attack types. Our model performs exceptionally, affect individuals [5]. Consequently, to instil trust in cyber
demonstrating outstanding performance of classification metrics security systems, Artificial Intelligence must be transparent
with 99% accuracy, F1 score, recall, and precision. To gain
deeper insights into our DNN model’s behaviour, we employ seven and interpretable. Various strategies have been proposed to
different post hoc explanation techniques to provide local expla- enhance the intelligibility of Artificial Intelligence decisions
nations. We evaluate the quality of Explainable AI (XAI) methods for human understanding to meet these requirements. These
using metrics such as high faithfulness, monotonicity, complexity, techniques, commonly called "XAI" (Explainable Artificial
and sensitivity. Our findings highlight the effectiveness of XAI Intelligence), have already been implemented in numerous
techniques in enhancing the interpretability and transparency of
the DNN model for IoT botnet detection. Specifically, our results domains, including healthcare, Natural Language Processing,
indicate that DeepLIFT yields high faithfulness, high consistency, and financial services [6].
low complexity, and low sensitivity among all the explainers. As the complexity and volume of cyber attacks, such as
Index Terms—XAI, Deep learning, IoT Botnet, Post-hoc expla- malware, intrusion, and spam, continue to rise, effectively
nation, explainable artificial intelligence managing them is becoming increasingly challenging [7]. In
the cybersecurity domain, conventional approaches such as
I. I NTRODUCTION rule-based, statistics-based, and signature-based have tradi-
Cybersecurity incorporates extensive practices and measures tionally been employed for intrusion detection [8]. However,
to protect networks, devices, and data against unauthorized the surge in data transmission over the Internet and the
access or illicit usage. It entails the critical task of striking emergence of new networking paradigms like the Internet
a delicate balance to uphold the security properties of confi- of Things (IoT), cloud computing, and fog/edge computing
dentiality, integrity, and availability (CIA) for information pro- [9, 10] have rendered these traditional approaches inadequate
tection [1]. Meanwhile, cyber defensive mechanisms emerge in processing large volumes of data and have resulted in high
at multiple levels, including application, network, host, and computing costs [9]. Furthermore, Artificial Intelligence (AI)
data, to fortify the overall security infrastructure [2]. Artificial has emerged as a fundamental technology in Industry 4.0 [11].
Intelligence techniques, particularly machine learning (ML) Consequently, AI techniques, such as machine learning (ML)
and deep learning (DL) algorithms have demonstrated re- algorithms and deep learning (DL) algorithms, have become
markable capabilities in various cybersecurity domains. These increasingly essential in providing intelligent cybersecurity
techniques have exhibited impressive performance levels when services and management. For example, Ucci et al. [7] high-
applied to benchmark datasets for tasks such as intrusion de- lighted the successful utilization of ML methods for various
tection, spam email filtering, botnet detection, fraud detection, aspects of malware analysis, including malware detection,
and identification of malicious applications [3]. It is essential similarity analysis, and category analysis. Additionally, Kwon
to acknowledge that these techniques are not error-free. They et al. [12] employed DL-based and feature selection [13]
can occasionally produce errors that may be more costly approaches for network anomaly detection and network traffic
than traditional cyber defensive approaches. On the contrary, analysis, demonstrating these techniques’ efficacy in cyber
cyber security developers have sometimes prioritized achiev- security.

1946-0759/23/$31.00 ©2023 IEEE 595

DOI 10.1109/ICMLA58977.2023.00088
Authorized licensed use limited to: Tallinn University of Technology. Downloaded on March 31,2024 at 18:52:30 UTC from IEEE Xplore. Restrictions apply.
 However, despite the potential benefits, AI-based ap- Our work is concluded in Section VI.
proaches in cyber security also face significant challenges.
Limitations such as the availability of cybersecurity-related II. R ELATED W ORK
data [14], the vulnerability of AI models to adversarial at-
tacks [15], and concerns regarding ethics and privacy [16] Several studies in the literature have applied explainabil-
are inherent issues encountered by AI-based cybersecurity ity methods to network intrusion detection problems. For
systems. Among these challenges, the black-box nature of AI instance, Szczepanski et al. introduced a hybrid Oracle Ex-
models is a particularly noteworthy limitation that requires plainer Intrusion Detection System that combines Artificial
careful consideration when integrating AI models into the Neural Networks (ANNs) and Decision Trees, utilizing micro
cybersecurity domain [17]. The opaque nature of AI models aggregation techniques to enhance performance [20]. In this
leads to cybersecurity decisions being generated without clear paper [21], the authors employed the Explainable Boosting
rationale or justifiability, making it difficult for humans to Machine (EBM) as a glass-box classifier for detecting network
comprehend how these decisions are reached. Consequently, intrusions. The performance of EBM was compared with
this lack of explainability renders the cyber defensive mecha- other AI classifiers, including decision tree, logistic regression,
nisms as black-box systems, rendering them highly susceptible Support Vector Machine (SVM), and Deep Neural Network
to information breaches and AI-based cyber threats [18]. (DNN). The ASNM-NPBO dataset [22] (Advanced Security
Explainable AI (XAI) has emerged as a response to the Network Metrics & Non-Payload-Based Obfuscations) con-
growing black box problem addressing the limitations of using sisted of ASNM features extracted from tcpdump capture
AI. XAI enables users and experts to better understand the of obscured malicious and legitimate TCP transmissions on
underlying logic and key data evidence behind the results specified vulnerable network services.
generated by AI-based statistical models [19]. In conclusion, Zolanvari et al. proposed a model-agnostic XAI framework
there are several motivations for applying Explainable AI named TRUST (Transparency Based on Statistical Theory)
(XAI) in cyber security, and they are like Building trust: designed for numerical applications. This framework utilizes
Integrating XAI in cyber security aims to enhance trust by factor analysis to transform input features into latent features,
promoting transparency and understanding of decision models ranks features using mutual information and employs a multi-
related to cybersecurity. Justice, social responsibility, and risk modal Gaussian distribution to generate new samples for each
mitigation: Applying XAI in cyber security is driven by class label. The TRUST XAI model underwent evaluation
addressing severe social problems, potential risks to human through a case study using three datasets: NSL-KDD [24],
lives, and the ethical considerations involved. It goes beyond UNSW [25], and a custom dataset named "WUSTL-IIoT,"
simple cost-benefit calculations and ensures fairness and social derived from their testbed experiment involving intrusion
responsibility. network traffic data. The evaluation showed that the TRUST
This research comprehensively evaluates seven post hoc XAI model achieved a 98% success rate in explaining random
local explainability methods, including Integrated Gradients, test samples, outperforming LIME. Hariharan et al. employ
Gradient * Input, DeepLIFT, Saliency, SHAP, feature ablation a comprehensive set of interpretability algorithms, including
and LIME. The evaluation is based on quantitative metrics Permutation Importance, SHapley Additive exPlanation, Local
such as faithfulness, monotonicity, complexity, and sensitivity. Interpretable Model-Agnostic Explanation, Contextual Impor-
Specifically, we assess and compare the effectiveness of these tance, and Utility algorithms [26] over NSL-KDD [24] dataset.
local explanation methods when applied to multiclass attack These algorithms are applied to Intrusion Detection Systems
types for IoT botnet detection in a detailed benchmarking (IDSs) implemented on three machine learning models: Ran-
setting. The experiments are performed on a deep learning- dom Forest, eXtreme Gradient Boosting, and Light Gradient
based model over the N-BaIoT dataset, carefully selected from Boosting. The study thoroughly compares explanations based
the problem domain to ensure the robustness and relevance on accuracy, consistency, and stability to enhance the under-
of our findings. To the best of our knowledge, very few standing of cyber-attack predictions in network traffic for cyber
empirical studies have evaluated the quality of post hoc local security personnel. Additionally, the paper presents a case
explainability methods for network intrusion detection tasks, study focusing on DoS attack variants, which offers valuable
particularly in IoT botnet detection. Our research aims to insights into the impact of features on prediction performance.
bridge this gap by systematically assessing and comparing A Bidirectional Long Short-Term Memory based Explainable
multiple explainability methods, providing valuable insights Artificial Intelligence framework (BiLSTM-XAI) was devel-
into the effectiveness of these methods for enhancing the oped, utilizing the Krill herd optimization (KHO) algorithm
interpretability and trustworthiness of deep learning-based to select significant database features [27]. SHAP and LIME
models in IoT security applications. explainable AI algorithms were employed for analysis, and
The content of our paper is presented as follows: Section the proposed BiLSTM-XAI model achieved an accuracy of
II provides related information about the addressed topic and 98.2% when evaluated using the NSL-KDD dataset [24]. By
summarizes the literature. Section III describes data utilized implementing XAI models, the complexities of the BiLSTM
in this study. Section IV explains the methodology of our re- framework are reduced, leading to improved detection accu-
search. Section V presents the results and discussions obtained. racy with explanations provided for each learning prediction.

596

Authorized licensed use limited to: Tallinn University of Technology. Downloaded on March 31,2024 at 18:52:30 UTC from IEEE Xplore. Restrictions apply.
 III. DATA SET D ESCRIPTION hidden layers, each comprising 9 hidden units, as specified in
In this work, N-BaIoT [28] is used. In this dataset, 115 hyperparameters (see Tab. III). The Rectified Adam (RAdam)
network traffic features were extracted based mainly on de- optimizer with a learning rate of 0.0129692 was employed
scriptive statistics measures from the network traffic generated to train the model. The SELU (Scaled Exponential Linear
by bots deployed in a collected testing environment settings of Unit) activation function is applied after each hidden layer to
9 different IoT devices infected by Mirai and Gafgyt malware introduce non-linearity, allowing the DNN to learn complex
types. More significantly, the features that are specified for patterns and relationships within the data effectively.
each instance reflect the aggregated descriptive statistics of
TABLE III: DNN Model Hyperparameters (Random Search)
the raw traffic of the network in five-time windows (100 ms,
500 ms, 306 1.5 s, 10 s, and 1 min), which are coded L5, L3, Hyperparameter Value
Hidden Layers 3
L1, L0.1 and L0.01, respectively as shown in Table I. There Hidden Units 9
are five main feature categories: host-IP (traffic emanating Learning Rate 0.01296
from a specific IP address, coded as H), host-MAC and IP Optimizer RAdam
Activation SELU
(traffic emanating from the same MAC and IP, coded MI), Batch Size 256
channel (traffic between specific hosts, coded HH), socket Epochs 21
(traffic between specific hosts, including ports, coded HpHp),
and network jitter (time interval between packets in channel The input layer receives a 115-dimensional feature vector,
communication, coded as HH_jit). The packet count, mean representing aggregated statistics from five different time
and variance packet sizes are calculated for each significant windows. This input is passed through the first fully connected
category. There have been extra statistical values like the hidden layer with 9 units, applying the SELU activation
correlation coefficient (PCC) of packet size, radius, covariance, function. The output from the first hidden layer is then fed to
and magnitude derived for network Channel and Socket along the second hidden layer, which consists of 9 units and employs
with packet count, mean, and variance. In this study, From the SELU activation function. Finally, the output layer is
the N-BaIoT dataset, attack type classification is developed. implemented with another fully connected layer, mapping the
For this classification, the data points have been classified into activations from the last hidden layer to the output size. This
eight attack types and legitimate network traffic: ACK, Benign, output size corresponds to the number of classes (ack, benign,
COMBO, JUNK, SCAN, SYN, TCP, UDP, and PLAIN. A combo, junk, scan, syn, tcp, udp, udpplain) used for botnet
description of these attacks can be found in Table II. attack type classification. During the training phase, the model
undergoes 21 epochs with a batch size 256. To compute the
TABLE I: Details of Features of N-BaIoT dataset. loss during training, we utilize the CrossEntropyLoss function,
Feature Category Category Code No.Of features Statistical Value Feature Time Frame Windows which is well-suited for multi-class classification tasks
Hos Mac & IP MI 15 100 Micrso Sconds
Host IP H 15
Packet Count, Mean
Variance
500 micro seconds To generate prediction labels, we rely on the softmax activa-
Network Jitter HH_jit 15 1.5 Seconds
Channel HH 35 Packet Count, Mean 10 Seconds tion function. This function is crucial in providing prediction
Socket HpHp 35 Variance, Magnitude,Radius, CoVariance, Correlation 1 Minute
probabilities for each class, enabling a deeper understanding
of the model’s confidence in its predictions and the proba-
TABLE II: Botnet Attack types used in this Study for DNN bility distribution across different classes( attacks and begin).
model botnet attack detection Utilizing the softmax activation function enhances the inter-
pretability and significance of our classifier’s outputs, which
Class Name Description
ACK Gafgyt malware Sending Spam data
is essential for evaluating explainers regarding Faithfulness,
Beign Legitimate Network Traffic Monotonicity, and Sensitivity metrics mentioned in the section
COMBO Gafgyt malware Sending spam data and opening a connection IV-C
JUNK Mirai Malware ACK-Flooding
SCAN Scans The network devices for vulnerabilities,(Mirai & Gafgyt B. Explaination Methods
SYN Mirai Malware SYN-Flooding
Our research focused on enhancing the interpretability
TCP Gafgyt malware TCP Flooding
UDP UDP flooding (Mirai & Gafgyt) of deep learning-based intrusion detection systems in IoT
UDPPLAIN Mirai malware UDP flooding with Less of an option for higher packet per second networks by applying various Explainable AI (XAI) feature
attribution methods. In this study, we employed seven dis-
tinct explanation methodologies, which were carefully selected
IV. M ETHODOLOGY
based on their widespread usage in the literature. Below, we
A. DNN for Botnet attack prediction present a concise summary of each method. However, please
This research paper proposes a fully connected Deep Neural note that this is not an exhaustive description of each method.
Network (DNN) model for detecting IoT botnet attacks in the For data, the input x ∈ Rd , where d is the dimensionality
N-BaIoT dataset through network traffic analysis. The dataset of the feature set, and the black box model M maps the
was split into training and testing sets using an 80:20 ratio. input to an output M(x) ∈ Y. Denote D = (xi , y i ) as the
As a preprocessing step, we applied min-max normalization, dataset consisting of all input-output pairs. g as an explanation
scaling the feature values between 0 and 1. Model consists of 3 mapping that for predictor M and point of interest x returns an

597

Authorized licensed use limited to: Tallinn University of Technology. Downloaded on March 31,2024 at 18:52:30 UTC from IEEE Xplore. Restrictions apply.
importance score g(M, x) = ϕx ∈ Rd for all features. Denote instance x within the neighborhood N (x) using a simpler
D : Rd × Rd 7→ R≥0 as a metric in the space of explanations interpretable model the prediction of the interpretable model
and SRd × Rd 7→ R≥0 a metric in the space of inputs. The around x is denoted as The approximation is obtained by min-
evaluation criterion µ is the mapping that takes predictor M, imizing the object function defined as argminL(M, g, πx ) +
g∈G
explainer g and the point of interest x as arguments and
Ω(g)
computes a scalar.
1) Integrated Gradients: Integrated attributes [29] feature C. XAI evaluation
importance to inputs by integrating the model’s gradients along Various metrics assess XAI explanation quality [35],
a path from a baseline to the actual input, The authors define grouped into three categories: user-focused, application-
x̄ as a baseline input representing the absence of a feature focused, and functionality-focused evaluations. The primary
in the input x. IntegratedR 1 Gradients is defined as g(M, x) = metrics employed to assess local explanations in this work
IG(x) = (x − x̄) × α=0 ∂M(x̄+α·(x−x̄))∂x dα where x̄ is the include high faithfulness, monotonicity, low complexity, and
baseline input. max sensitivity for the explainers utilized.
2) Gradient * Input: Gradient * Input computes [30] the 1) High Faithfulness: Faithfulness measures how well the
importance of each feature by multiplying the model’s partial explanation function g aligns feature importance scores with
derivative with respect to that feature by the input feature the black-box model M
value. The formula for Gradient * Input, denoted as g(M, x),
is given by: g(M, x) = ∂M(x) ∂x ×x X
!
3) DeepLIFT: DeepLIFT (Deep Learning Important Fea- µF (M, g; x) = corr g(M, x)i , M(x) − M(xB )
|d|
Tures) [31] attributes each input x is assigned value C∆xi ∆y B∈(|B| ) i∈B
that signifies a deviation from its original value towards (1)
a reference where xB = xi |i ∈ B}
Pnvalue. DeepLIFT utilizes a summation-to-delta The faithfulness metric iteratively replaces a random subset
property: i=1 C∆xi ∆o = ∆o where o = M(x) and ∆o
is the difference between the model output of the input and of given attributions with a baseline value. Then it measures
the reference value. the correlation between the sum of this attribution and the
4) Saliency: The saliency-based [32] approach focuses on difference in the model’s output.
the gradient of the output of a neural network with respect to 2) Monotonocity: monotonicity checks if the predictor’s
the input x. In our case, the output corresponds to the attack output changes consistently with changes in the sum of feature
score a(x). The basic idea behind the saliency is that small attributions when moving from one input point to another.
perturbations to important features will cause large changes in Monotonicity ensures that the explanation provided by the
the model output; the saliency score measures the sensitivity of explainer is consistent with the behavior of the predictor.
the model output against the input perturbation. Concretely, the 3) Low Complexity: The complexity metric computes the
saliency score sℓ of the ℓ-th dimension of input x is defined as entropy of the fractional contribution of each feature to the
total magnitude of the attribution individually.
sℓ (x) = ∂a(x)
∂xℓ where xℓ is the ℓ-th dimension of x. For each
malicious packet x, we calculate the saliency scores sℓ (x) for d
X
ℓ = 1, 2, . . . , L, and report the dimensions with high saliency µC (M, g; x) = − Pg (i) log Pg (i) (2)
scores as suspected parts. i=1
5) Shap: SHAP [33] is a game-theoretic explanation where
method based on PMShapley values. SHAP is defined as |g(M, x)i|
g(M, x) = ϕ0 + j=1 ϕj , where ϕj is the feature attribution Pg (i) = P ; Pg = Pg (1), ....Pg (d) (3)
j ∈ |d||g(M, x)j |
of feature j.
6) Feature Ablation: Feature ablation assesses the impor- 4) Maximum Sensitivity: The maximum sensitivity metric
tance of individual features in a model’s predictions by system- measures the maximum sensitivity of an explanation using a
atically removing or ablating each feature and observing the Monte Carlo sampling-based approximation. It assesses how
change in the model’s output. The process involves evaluating sensitive the explanation function is to small changes in the
the model’s performance on the modified input with the ab- input data in the neighbourhood of the point of interest. By
lated feature(s) and comparing it to the original performance. sampling nearby points and comparing their explanations, the
The formula for feature ablation can be represented as follows: metric aims to ensure that similar inputs with similar model
outputs receive consistent explanations
Ablation(x, i) = M(x) − M(xablatedi )

By comparing the model’s output before and after ablation, µM (M(x), g, r; x) = max Distance(g(M(x), x), g(M(x), z))
z∈Nr
feature ablation allows for understanding the influence of each (4)
individual feature on the model’s decision-making process. where Nr = z ∈ Distancex |p(x, z) ≤ r, M(x) = M(x)(z)
7) LIME: LIME [34] is an explanation method that ap- In the above equation, Nr represents the set of points within
proximates the model’s behavior locally around a specific data a certain radius r around the point of interest x.

598

Authorized licensed use limited to: Tallinn University of Technology. Downloaded on March 31,2024 at 18:52:30 UTC from IEEE Xplore. Restrictions apply.
 The experiments were carried out on a computer running
Pop!_OS 22.04 LTS x86_64 operating system with the fol-
lowing hardware configuration: 32 GB of DDR4-2666R ECC
RAM, AMD Ryzen 5 5600G with Radeon Graphics (12) @
3.900GHz processor. The scripts were developed using the
Python 3 programming language and Pytorch library.
V. R ESULTS & D ISCUSSION
In this section, we will present the results of our research,
which encompass an analysis of the model’s performance,
explanations provided by the explainers, and an evaluation of
these explainers using four different metrics.
To evaluate the performance of our DNN model for botnet
(a) TCP attack (b) UDP attack
attack type detection, we utilized metrics such as classification
accuracy, precision, recall, F1 score, and the confusion matrix
(see results in Fig. 2). We present Fig. 1, which illustrates Fig. 3: Example of Top 10 influential features from each
the graph of testing and training loss over epochs and the explainer for ACK and UDP attack
training and testing accuracies throughout the training process.
In the classification report, the model achieved more than 99%
accuracy for all attacks. However, the recall for TCP attacks is
97% , and the precision for UDP attacks is 98%, both of which
are slightly lower than those for other attacks, respectively.

(a) ACK attack (b) SYN Attack

(a) Accuracy (b) Loss

Fig. 4: Example of Top 10 influential features from each

Fig. 1: Training and Testing Accuracies and Loss Values of explainer for ACK and SYN attacks
DNN Model for IoT Botnet Attack Detection

Feature Ablation, SHAP, Saliency, and LIME, to explain the

predictions of our DNN model on the test data. Each explana-
tion method provided feature importance scores based on its
specific approach (e.g., gradient-based or SHAP values). We
aggregated the feature importance scores across all instances
in the test data using absolute sum for each class of prediction
label. Subsequently, we ranked the aggregated features based
on their importance scores.
Fig. 3 shows the example of top 10 feature influences from
every explainer in TCP and UDP attack predictions by the
DNN model. Based on the explanation results provided by
the different explainers, it is evident that the most influential
features in the IoT botnet attack prediction by our DNN model
are derived from the Host-based (MI & H) category. On
Fig. 2: Confusion matrix and classification report of DNN the other hand, features belonging to the Channel (HH) and
model for IoT botnet attack type detection Socket-based (HpHp) network categories exhibit relatively less
influence. Furthermore, the Network Jitter (HH_jit) features
In our research, we employed seven explanation methodolo- show minimal impact on the predictions of the model. Notably,
gies, namely Integrated Gradients, InputXGradient, DeepLIFT, we have observed similar trends in the influence of network

599

Authorized licensed use limited to: Tallinn University of Technology. Downloaded on March 31,2024 at 18:52:30 UTC from IEEE Xplore. Restrictions apply.
categories for other types of attack class predictions, for a relatively higher sensitivity value, suggesting less stable
Example, for ACK& SYN attacks in Fig 4. explanations for neighbouring data points.

VI. C ONCLUSION
TABLE IV: comparison results of the evaluation of XAI
metrics This research paper highlights the importance of using XAI
methods to enhance the interpretability and transparency of
Explainer/metric High Faithfulness Sensitivity Complexity Monotonocity
Integrated Gradients 0.44 ± 0.32 0.61 ± 1.44 3.32 ± 0.38 52.00%
deep learning-based model for detecting IoT botnet attacks of
Gradient * Input 0.42 ± 0.36 0.61 ± 1.44 3.75 ± 0.29 60.00% multiclass classication. We used DNN model for classifying
DeepLIFT 0.71 ± 0.12 0.48 ± 0.76 2.82 ± 0.12 69.19% various IoT botnet attack types, achieving remarkable perfor-
SHAP 0.55 ± 0.18 8.17 ± 8.11 3.30 ± 0.40 44.00% mance with 99% accuracy, F1 score, recall, and precision.
Feature Ablation 0.48 ± 0.31 0.38 ± 0.76 4.29 ± 0.41 39.00% However, the black-box nature of deep learning models posed
Saliency 0.27 ± 0.26 9.79 ± 21.36 4.33 ± 0.06 46.00%
challenges in understanding their decision-making process. We
LIME 0.39 ± 0.21 10.62 ± 5.68 4.30 ± 0.31 28.00%
explored seven post hoc explanation techniques to address
this issue to provide local explanations for our DNN model’s
In our research, we evaluated the faithfulness of explanation behaviour. Through rigorous quantitative evaluation using high
methods by analyzing the correlation between the importance faithfulness, monotonicity, complexity, and sensitivity metrics,
assigned to attributes by these methods and their impact on we thoroughly assessed the quality of these explanations.
the predictive model’s probabilities. Additionally, we measure Notably, among all the explainers, DeepLIFT emerged as
monotonicity to assess the effect of individual features on a standout performer, demonstrating high faithfulness, high
model probability by incrementally adding each attribute in consistency, low complexity, and low sensitivity. Its ability to
order of increasing importance and evaluating its impact on the furnish accurate and stable explanations signifies the reliability
model’s probability. The DNN model prediction probabilities of our DNN model’s decision-making process.
are obtained using the Softmax activation function. A high
faithfulness correlation value indicates that the explanations R EFERENCES
accurately capture the original model’s behaviour and can be [1] “What is cybersecurity? | cisa,” https://www.cisa.
considered trustworthy. Similarly, a high monotonicity score gov/news-events/news/what-cybersecurity, (Accessed on
indicates that the explanations are consistent with the model’s 07/17/2023).
predictions for the given input. Furthermore, we compute the [2] D. S. Berman, A. L. Buczak, J. S. Chavis, and C. L.
low complexity metric by calculating the entropy of feature Corbett, “A survey of deep learning methods for cyber
attribution derived from the explanations. Furthermore, the security,” Information, vol. 10, no. 4, p. 122, 2019.
sensitivity metric assesses the robustness of the explainer [3] S. M. Mathews, “Explainable artificial intelligence appli-
output, ensuring that nearby inputs in the feature space have cations in nlp, biomedical, and malware classification: A
similar explanations when the sensitivity value is low. In this literature review,” in Intelligent Computing: Proceedings
metric, for obtaining the nearest neighbour points related to the of the 2019 Computing Conference, Volume 2. Springer,
prediction label of the explanation, we utilized the Euclidean 2019, pp. 1269–1292.
distance with a radius value of 0.1, which helps identify data [4] M. Sahakyan, Z. Aung, and T. Rahwan, “Explainable
points in the feature space closest to the instance and have artificial intelligence for tabular data: A survey,” IEEE
similar explanations for the predicted label. access, vol. 9, pp. 135 392–135 422, 2021.
Table IV shows comparison results of the evaluation of [5] B. Goodman and S. Flaxman, “European union regu-
XAI metrics. Among the explainers, DeepLIFT emerged as lations on algorithmic decision-making and a “right to
the most promising, achieving the highest mean faithfulness explanation”,” AI magazine, vol. 38, no. 3, pp. 50–57,
correlation of 0.71 ± 0.12. This indicates that the expla- 2017.
nations provided by DeepLIFT are highly consistent with [6] M. Nazar, M. M. Alam, E. Yafi, and M. M. Su’ud,
the DNN model’s behaviour, making them more trustworthy “A systematic review of human–computer interaction
and accurate. Additionally, DeepLIFT demonstrated a lower and explainable artificial intelligence in healthcare with
sensitivity value of 0.48 ± 0.76, implying robust and consistent artificial intelligence techniques,” IEEE Access, vol. 9,
explanations for nearby data points in the feature space. More- pp. 153 316–153 348, 2021.
over, DeepLIFT achieved an impressive Monotonicity score of [7] D. Ucci, L. Aniello, and R. Baldoni, “Survey of machine
69.19%, indicating a strong consistency in how each feature learning techniques for malware analysis,” Computers &
influences the model’s predictions. This consistent relationship Security, vol. 81, pp. 123–147, 2019.
between feature importance and model performance enhances [8] S. Han, M. Xie, H.-H. Chen, and Y. Ling, “Intrusion
the credibility of DeepLIFT’s explanations. Furthermore, Gra- detection in cyber-physical systems: Techniques and
dient * Input exhibited the highest Monotonicity score of challenges,” IEEE systems journal, vol. 8, no. 4, pp.
60%, showing a strong trend of monotonically increasing 1052–1062, 2014.
model performance with adding features. SHAP achieved a [9] D. Gümüşbaş, T. Yıldırım, A. Genovese, and F. Scotti,
respectable faithfulness correlation of 0.55 ± 0.18 but had “A comprehensive survey of databases and deep learning

600

Authorized licensed use limited to: Tallinn University of Technology. Downloaded on March 31,2024 at 18:52:30 UTC from IEEE Xplore. Restrictions apply.
 methods for cybersecurity and intrusion detection sys- study on iiot security,” IEEE Internet of Things Journal,
tems,” IEEE Systems Journal, vol. 15, no. 2, pp. 1717– 2021.
1731, 2020. [24] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani,
[10] R. Donida Labati, A. Genovese, V. Piuri, F. Scotti, and “A detailed analysis of the kdd cup 99 data set,” in
S. Vishwakarma, “Computational intelligence in cloud 2009 IEEE symposium on computational intelligence for
computing,” Recent Advances in Intelligent Engineering: security and defense applications. Ieee, 2009, pp. 1–6.
Volume Dedicated to Imre J. Rudas’ Seventieth Birthday, [25] N. Moustafa and J. Slay, “Unsw-nb15: a comprehensive
pp. 111–127, 2020. data set for network intrusion detection systems (unsw-
[11] I. H. Sarker, M. H. Furhad, and R. Nowrozy, “Ai- nb15 network data set),” in 2015 military communi-
driven cybersecurity: an overview, security intelligence cations and information systems conference (MilCIS).
modeling and research directions,” SN Computer Science, IEEE, 2015, pp. 1–6.
vol. 2, pp. 1–18, 2021. [26] S. Hariharan, R. Rejimol Robinson, R. R. Prasad,
[12] D. Kwon, H. Kim, J. Kim, S. C. Suh, I. Kim, and K. J. C. Thomas, and N. Balakrishnan, “Xai for intrusion de-
Kim, “A survey of deep learning-based network anomaly tection system: comparing explanations based on global
detection,” Cluster Computing, vol. 22, pp. 949–961, and local scope,” Journal of Computer Virology and
2019. Hacking Techniques, vol. 19, no. 2, pp. 217–239, 2023.
[13] R. Kalakoti, S. Nõmm, and H. Bahsi, “In-depth feature [27] S. Sivamohan and S. Sridhar, “An optimized model for
selection for the statistical machine learning-based botnet network intrusion detection systems in industry 4.0 using
detection in iot networks,” IEEE Access, vol. 10, pp. xai based bi-lstm framework,” Neural Computing and
94 518–94 535, 2022. Applications, vol. 35, no. 15, pp. 11 459–11 475, 2023.
[14] R. Al Nafea and M. A. Almaiah, “Cyber security threats [28] Y. Meidan, M. Bohadana, Y. Mathov, Y. Mirsky, A. Shab-
in cloud: Literature review,” in 2021 international con- tai, D. Breitenbacher, and Y. Elovici, “N-baiot: Network-
ference on information technology (ICIT). IEEE, 2021, based detection of iot botnet attacks using deep autoen-
pp. 779–786. coders,” IEEE Pervasive Computing, vol. 17, no. 3, pp.
[15] A. Kuppa and N.-A. Le-Khac, “Black box attacks on 12–22, 2018.
explainable artificial intelligence (xai) methods in cyber [29] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic at-
security,” in 2020 International Joint Conference on tribution for deep networks,” in International conference
neural networks (IJCNN). IEEE, 2020, pp. 1–8. on machine learning. PMLR, 2017, pp. 3319–3328.
[16] M. Roopak, G. Y. Tian, and J. Chambers, “Deep learning [30] A. Shrikumar, P. Greenside, A. Shcherbina, and A. Kun-
models for cyber security in iot networks,” in 2019 IEEE daje, “Not just a black box: Learning important fea-
9th annual computing and communication workshop and tures through propagating activation differences,” arXiv
conference (CCWC). IEEE, 2019, pp. 0452–0457. preprint arXiv:1605.01713, 2016.
[17] J. Gerlings, A. Shollo, and I. Constantiou, “Reviewing [31] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning
the need for explainable artificial intelligence (xai),” important features through propagating activation differ-
arXiv preprint arXiv:2012.01007, 2020. ences,” in International conference on machine learning.
[18] G. Jaswal, V. Kanhangad, and R. Ramachandra, AI and PMLR, 2017, pp. 3145–3153.
deep learning in biometric security: trends, potential, and [32] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep
challenges. CRC Press, 2021. inside convolutional networks: Visualising image clas-
[19] C. Rudin, “Stop explaining black box machine learning sification models and saliency maps,” arXiv preprint
models for high stakes decisions and use interpretable arXiv:1312.6034, 2013.
models instead,” Nature machine intelligence, vol. 1, [33] S. M. Lundberg and S.-I. Lee, “A unified approach
no. 5, pp. 206–215, 2019. to interpreting model predictions,” Advances in neural
[20] M. Szczepański, M. Choraś, M. Pawlicki, and R. Kozik, information processing systems, vol. 30, 2017.
“Achieving explainability of intrusion detection system [34] M. T. Ribeiro, S. Singh, and C. Guestrin, “" why should
by hybrid oracle-explainer approach,” in 2020 Interna- i trust you?" explaining the predictions of any classifier,”
tional Joint Conference on Neural Networks (IJCNN). in Proceedings of the 22nd ACM SIGKDD international
IEEE, 2020, pp. 1–8. conference on knowledge discovery and data mining,
[21] T. A. El-Mihoub, L. Nolle, and F. Stahl, “Explainable 2016, pp. 1135–1144.
boosting machines for network intrusion detection with [35] L. Coroama and A. Groza, “Evaluation metrics in ex-
features reduction,” in International Conference on In- plainable artificial intelligence (xai),” in Advanced Re-
novative Techniques and Applications of Artificial Intel- search in Technologies, Information, Innovation and
ligence. Springer, 2022, pp. 280–294. Sustainability: Second International Conference, ARTIIS
[22] “Asnm datasets,” ASNM Datasets, (Accessed on 2022, Santiago de Compostela, Spain, September 12–15,
07/29/2023). 2022, Revised Selected Papers, Part I. Springer, 2022,
[23] M. Zolanvari, Z. Yang, K. Khan, R. Jain, and N. Meskin, pp. 401–413.
“Trust xai: Model-agnostic explanations for ai with a case

601

Authorized licensed use limited to: Tallinn University of Technology. Downloaded on March 31,2024 at 18:52:30 UTC from IEEE Xplore. Restrictions apply.