Received 8 August 2022, accepted 30 August 2022, date of publication 5 September 2022, date of current version 12 September 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3204171

Explainable Artificial Intelligence in

CyberSecurity: A Survey
NICOLA CAPUANO1 , GIUSEPPE FENZA2 , (Member, IEEE),
VINCENZO LOIA 2 , (Senior Member, IEEE),
AND CLAUDIO STANZIONE3 , (Member, IEEE)
1 Schoolof Engineering, University of Basilicata, 85100 Potenza, Italy
2 Departmentof Management and Innovation Systems, University of Salerno, 84084 Fisciano, Italy
3 Defence Analysis & Research Institute, Center for Higher Defence Studies, 00165 Rome, Italy

Corresponding author: Vincenzo Loia ([email protected])

1 ABSTRACT Nowadays, Artificial Intelligence (AI) is widely applied in every area of human being’s daily
2 life. Despite the AI benefits, its application suffers from the opacity of complex internal mechanisms and
3 doesn’t satisfy by design the principles of Explainable Artificial Intelligence (XAI). The lack of transparency
4 further exacerbates the problem in the field of CyberSecurity because entrusting crucial decisions to a system
5 that cannot explain itself presents obvious dangers. There are several methods in the literature capable of
6 providing explainability of AI results. Anyway, the application of XAI in CyberSecurity can be a double-
7 edged sword. It substantially improves the CyberSecurity practices but simultaneously leaves the system
8 vulnerable to adversary attacks. Therefore, there is a need to analyze the state-of-the-art of XAI methods in
9 CyberSecurity to provide a clear vision for future research. This study presents an in-depth examination of
10 the application of XAI in CyberSecurity. It considers more than 300 papers to comprehensively analyze the
11 main CyberSecurity application fields, like Intrusion Detection Systems, Malware detection, Phishing and
12 Spam detection, BotNets detection, Fraud detection, Zero-Day vulnerabilities, Digital Forensics and Crypto-
13 Jacking. Specifically, this study focuses on the explainability methods adopted or proposed in these fields,
14 pointing out promising works and new challenges.

15 INDEX TERMS Artificial intelligence, cybersecurity, explainable artificial intelligence, security paradigm,
16 trust.

17 I. INTRODUCTION 14.5%, by 2026.2 These numbers help convey the potential 28

18 Context. Artificial Intelligence (AI) is becoming more and of these two fields together and the need to find the proper 29

19 more prevalent in our daily lives. To quantify this phe- cohesion. Even if AI algorithms appear effective in outcomes 30

20 nomenon numerically, Grand View Research valued the and predictions, they suffer from opacity, making it diffi- 31

21 global AI market size at USD 93.5 billion in 2021 and cult to gain insight into their internal working mechanisms. 32

22 forecasts a compound annual growth rate (CAGR) of 38.1% This aspect further exacerbates the problem in a field like 33

23 from 2022 to 2030.1 Recently, AI finds widely application in CyberSecurity because entrusting important decisions to a 34

24 many areas as well as in the CyberSecurity domain. system that cannot explain itself presents obvious dangers. 35

25 Likewise, Mordor Intelligence valued the global CyberSe- On the light of this scenario, Explainable Artificial Intelli- 36

26 curity market at $156.24 billion in 2020 with an expectation gence (XAI) suggests a transition toward more interpretable 37

27 to be worth $352.25 billion, with an annual growth rate of AI to overcome this issue. XAI principles intend to develop 38

strategies that will result in better explainable models while 39

The associate editor coordinating the review of this manuscript and keeping high-performance levels. 40
approving it for publication was Ilsun You .
1 https://www.grandviewresearch.com/industry-analysis/artificial- 2 https://www.mordorintelligence.com/industry-reports/cyber-security-
intelligence-ai-market market

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
VOLUME 10, 2022 93575
 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

41 Problems and Motivations. Identifying gaps in the litera- TABLE 1. List of key acronyms.
42 ture to solve the critical issue of CyberSecurity for future ICT
43 systems is critical. The absence of transparency undermines
44 confidence. Security practitioners may hesitate to trust the
45 systems if they do not understand how crucial decisions are
46 made. However, the application of XAI in CyberSecurity
47 can be a double-edged sword: it can substantially improve
48 CyberSecurity practices but it may also facilitate new attacks
49 on the AI applications since it will also be Explainable to the
50 attacker, which may pose severe security threats [1]. As with
51 all innovations, there are pros and cons, but in this case,
52 it seems that the pros outweigh the cons mitigating the risks of
53 AI adoption in analogy to other application domains, like in
54 the Open Source context. Furthermore, the definition of AI
55 models compliant with XAI principles, or the development
56 of model agnostic XAI frameworks, will allow large-scale
57 AI usage in industrial and human scenarios, increasing the
58 capabilities to timely recognize vulnerabilities.
59 This study aims to compensate for the lack of investigation
60 in this area by focusing on the proposed techniques and
61 how they achieve explainability in order to design a path of
62 promising and appropriate future research directions, hoping
63 that interested researchers will be able to quickly and effec-
64 tively grasp the key features of the methods analyzed.
65 Contribution. This paper collects and analizes the results
66 of an in-depth survey on XAI in CyberSecurity. It aims to take
67 a step back to get a complete picture of the current state of
68 the art in this field of research, considering XAI applications
69 in several areas of CyberSecurity. This work stands out from
70 other works because it focuses on understanding explain-
71 ability and on comparing explainable and non-explainable
72 procedures used in the most studied areas of CyberSecurity.
73 One of the main points is to provide a solid foundation for
74 further discussion using the lens of the literature.
75 The main contributions of this paper are:
76 • A detailed discussion on the main concepts, objectives,
77 and consequences of enabling Explainability in various
78 CyberSecurity applications.
79 • An organized overview of existing XAI approaches in
80 CyberSecurity, based on a literature review of over II. BACKGROUND ON EXPLAINABLE ARTIFICIAL 97

81 300 papers (an outlook of surveys on XAI, AI in Cyber- INTELLIGENCE 98

82 Security, and XAI in CyberSecurity is also included). DARPA, the Defense Advanced Research Projects Agency, 99

83 • A summary tables of the explainable methods analyzed financed the ‘‘Explainable AI (XAI) Program’’ at the begin- 100

84 and the most frequently used datasets for each field of ning of 2017 [2]. XAI aims to develop more understandable 101

85 application. models while maintaining a high degree of learning perfor- 102

86 • A discussion on past efforts, current trends and future mance (prediction accuracy); and enable human users to com- 103

87 challenges. prehend, adequately trust, and manage the future generation 104

88 Organization. Table 1 presents acronyms used in the of artificially intelligent partners. 105

89 document for clarity to be provided to the reader. The rest After the launch of the program, the scientific contribution 106

90 of the survey is structured as follows. Section II presents in the Explainable Artificial Intelligence field has grown 107

91 an Explainable Artificial Intelligence overview. Section III significantly, as shown in Figure 1. 108

92 explores CyberSecurity Threats Foundations and AI applica-

93 tions. Section IV analyzes related surveys, while Section V A. XAI TAXONOMY 109

94 discusses XAI works in CyberSecurity. Section VI dis- Throughout the presented literature, various terms have been 110

95 cuss the findings and finally Section VII concludes this adopted, trying to cover all possible fields of application. 111

96 survey. Following are just a few of the wide variety used: 112

93576 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

FIGURE 1. Evolution of the number of total publications whose title,

abstract and/or keywords refer to the field of XAI until 2021. Data
retrieved from Scopus using as search key [TITLE-ABS-KEY (Explainable
AND Artificial AND Intelligence)].
FIGURE 2. XAI Principles presented by NIST in [3].

113 Trasparency: Do users grasp the format and language

114 choices made by the model? shed transparency on complex black-box models). These 148

115 Fairness: Can it be proven that model judgments are fair techniques include producing local explanations for spe- 149

116 to protected groups? cific inputs or the entire model globally. Following a quick 150

117 Trust: How comfortable are human users with using the overview: 151

118 system? • Model Specific or Model Agnostic: This determines 152

119 Usability: How well-equipped is the system to give users a whether or not the interpretation method is restricted 153

120 secure and productive environment in which to complete their to a specific model. Model-specific methods and tools 154

121 tasks? are those that are specific to a model. Model agnos- 155

122 Reliability: How resistant is the system to changes in tic methods can be applied to any ML model to gain 156

123 parameters and inputs? interpretability. Internal model data such as weights and 157

124 Causality: Do the predicted changes in the output, result- structural details are not accessible to these models. 158

125 ing from input perturbation, occur in the actual system? • Intrinsic or Extrinsic (post-hoc): This indicates 159

126 In the middle of 2020, the National Institute of Standards whether the model is interpretable on its own or whether 160

127 and Technology (NIST) presented four fundamental princi- interpretability requires using methods that examine 161

128 ples for explainable AI systems [3] as shown in Figure 2. models after training. Simple, comprehensible models, 162

129 The Explanation principle obligates AI systems to supply like decision trees, are intrinsic. Utilizing an interpreta- 163

130 evidence, support, or reasoning for each output. A system tion strategy after training to achieve interpretability is 164

131 fulfils the Meaningful principle if the recipient understands extrinsic. 165

132 the system’s explanations. The Explanation Accuracy princi- • Local or Global: Whether the interpretation method 166

133 ple imposes accuracy on a system’s explanations and in the describes a single data record or all of a model’s 167

134 end Knowledge Limits principle states that systems identify behaviour depends on whether it is local or global. 168

135 cases they were not designed or approved to operate, or their Global methods and tools interpret the entire model, 169

136 answers are not reliable [3]. whereas Local methods and tools only explain a single 170

137 Over the years, a vast taxonomy has been developed on prediction. 171

138 the various ways and methods that can make an AI model
139 explainable. The first distinction needed is between Inter- B. XAI FRAMEWORKS 172

140 pretability and Explainability. Interpretability is all about An XAI framework is a tool that creates reports on model 173

141 understanding the cause and effect within an AI system. activity and tries to explain how it works. The following are 174

142 On the other hand, Explainability goes beyond interpretabil- the main ones encountered during the Survey. 175

143 ity in that it helps us understand how and why a model came LIME. Local Interpretable Model-agnostic Explana- 176

144 up with a prediction in a human-readable form. Figure 3 tions (LIME) is a framework that seeks to provide 177

145 presents the current taxonomy and makes a crucial dis- an individual-level explanation of individual predictions 178

146 tinction between true transparency (interpretable models) (Local) in an extrinsic (Post-hoc) manner and is able 179

147 and post-hoc interpretations (additional techniques used to to explain any model without needing to ‘peak’ into it 180

VOLUME 10, 2022 93577

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

GRAD-CAM. Gradient-weighted Class Activation Map- 220

ping (GRAD-CAM) [8] is a technique for producing a 221

class-specific heat map from a single image. Grad-CAM 222

produces a class discriminative localization map as a result. 223

The framework makes use of the feature maps generated by 224

a CNN’s final convolutional layer. This is Local-based, Post- 225

hoc but Model-Specific. 226

CEM. Contrastive Explanation Method (CEM) [9] pro- 227

vides explanations for classification models. More in detail, 228

it retrieves the features that should be sufficiently present to 229

predict the same class for the input instance. It also iden- 230

tifies minimal features to change for associating the input 231

instance to a different class. This is Local-based, Post-hoc but 232

Model-Agnostic. 233

FIGURE 3. A visual representation of XAI taxonomy. III. CYBERSECURITY THREATS FOUNDATIONS AND AI 234

APPLICATIONS 235

181 (Model-Agnostic) [4]. In order to figure out what parts of the If it were measured as a country, Cybercrime, which inflicted 236

182 interpretable input are contributing to the prediction, it per- damages around $6 trillion globally in 2021, would be the 237

183 turbs the input around its neighbourhood and see how the world’s third-largest economy after the U. S. and China. 238

184 model’s predictions behave. Then it generates a new dataset CyberSecurity Ventures expects global cybercrime costs to 239

185 consisting of perturbed samples and the corresponding pre- grow by 15% per year over the next five years, reaching 240

186 dictions of the black box model. On this new dataset, LIME $10.5 trillion annually by 2025, up from $3 trillion in 2015. 241

187 then trains an interpretable model, which is weighted by the In addition to being exponentially more considerable than 242

188 proximity of the sampled instances to the instance of interest. the damage caused by natural disasters in a year, this rep- 243

189 SHAP. SHapley Additive exPlanations (SHAP) [5] is a resents the most significant transfer of economic wealth in 244

190 framework with a clear goal, explaining the prediction of an history and poses a threat to the incentives for innovation and 245

191 instance x by computing the contribution of each feature to investment [10]. 246

192 the prediction. Like LIME, it is a Local-based, Post-hoc, and CyberSecurity is the process of defending ICT systems 247

193 Model-Agnostic paradigm. The SHAP explanation technique against various cyber threats or attacks. A ‘‘cyber-attack’’ 248

194 uses coalitional game theory to compute Shapley values. is any criminal activity that preys on electronic information 249

195 A data instance’s feature values operate as coalition members. systems, networks, or infrastructure. Information is primarily 250

196 Shapley values inform how fairly distributed the prediction is intended to be stolen, altered, or destroyed. In the current 251

197 across the characteristics. A player might be a single feature cyber-attack situation, attack vectors that take advantage of a 252

198 value or a collection of feature values. It is not necessary lack of readiness and (system as well as human) preparedness 253

199 to establish a local model in SHAP (as opposed to LIME), to access sensitive data or compromise systems are frequent. 254

200 but rather the same function is used to calculate the Shapley The main problems of CyberSecurity are the knowledge of 255

201 values for each dimension. various cyber-attacks and the development of complementary 256

202 Anchors. The Anchors approach [6] locates a decision protection mechanisms. 257

203 rule that ‘‘anchors’’ the prediction adequately and uses it to The risks usually connected to any attack take into account 258

204 explain specific predictions of any black box classification three security variables: threats, who is attacking; vulnerabil- 259

205 model. If changes in other feature values do not affect the ities, or the holes they are attacking; and impacts, or what 260

206 prediction, a rule anchors it. Anchors reduces the number the assault does. A security incident is an act that threatens 261

207 of model calls by combining reinforcement learning tech- the confidentiality, integrity, or availability of information 262

208 niques with a graph search algorithm. The ensuing expla- assets and systems. Obtaining illegal access, destruction, and 263

209 nations are expressed as simple IF-THEN rules known as alteration of information to harm possibly are just a few 264

210 anchors. This framework is Local-based, Post-hoc and then examples of potential breaches and security violations on 265

211 Model-Agnostic. a computer system or mobile device. Threats describe all 266

212 LORE. LOcal Rule-based Explanations (LORE) [7] cre- of the security mentioned above infractions’ potential risk 267

213 ates an interpretable predictor for a given black box instance. and hazard, and attacks describe any attempts to commit 268

214 A decision tree is used to train the local interpretable predictor a violation. 269

215 on a dense set of artificial cases. The decision tree allows Measures to safeguard information and communication 270

216 for the extraction of a local explanation, which consists of a technology, the unprocessed data and information it con- 271

217 single choice rule and a collection of counterfactual rules for tains, as well as their processing and transmission, associated 272

218 the reversed decision. This framework is Local-based, Post- virtual and physical elements of the systems, the degree of 273

219 hoc and then Model-Agnostic. protection attained as a result of the application of those 274

93578 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

275 measures, and ultimately the associated field of professional

276 endeavour, are all associated with CyberSecurity.
277 Cyber-attacks or intrusions require defence techniques to
278 protect data or information, information systems, and net-
279 works. They are in charge of preventing data breaches and
280 security incidents, as well as monitoring and responding to
281 intrusions, defined as any unauthorized action that causes
282 damage to an information system.
283 ENISA, the European Union Agency for CyberSecurity,
284 provided a report with an analysis of the top 15 cyber threats,
285 showed in Figure 4, that dominated the period between Jan-
286 uary 2019 and April 2020 [11].
287 Only some of these threats were addressed in this sur-
288 vey, focusing on those application areas where Explainable
289 Artificial Intelligence has been most explored. In particular,
290 the world of Intrusion Detection Systems, Malware detec-
291 tors, prevention against Spam and Phishing, and detection
292 of BotNets was extensively explored. In addition, a shorter
293 analysis was conducted on Fraud Detection, Zero-Day Vul-
294 nerabilities, Digital Forensics, Cyber-Physical Systems and
295 Crypto-Jacking.

296 IV. RELATED WORKS

297 The following sub-sections analyze the existing surveys
298 related to this work. First, there is an analysis of existing
299 surveys in the general field of Explainable Artificial Intel-
300 ligence. Subsequently, attention will be focused on surveys
301 about AI applications in CyberSecurity. To conclude, there
302 is an investigation of the few existing works that attempt to
303 clarify the applications of Explainable Artificial Intelligence
304 in CyberSecurity.

305 A. SURVEYS ON EXPLAINABLE ARTIFICIAL INTELLIGENCE

306 High-performance AI systems, particularly those based
307 on DL, behave similarly to black boxes that provide
308 good results but can hardly justify a given output in a
309 human-understandable way [12], [13]. It is essential to min-
310 imize potential biases (e.g., algorithmic, racial, ideological
311 and gender biases) during the ethical AI solution development
312 stage [14], [15].
313 Adadi and Berrada [16] conducted an exhaustive literature
314 analysis, collecting and analyzing 381 different scientific
315 papers between 2004 and 2018. They organized all of the
316 scientific work in explainable AI along four primary axes and
FIGURE 4. Top 15 Cyber Threats presented by ENISA in [11].
317 emphasized the importance of introducing more formalism
318 in the field of XAI and more interaction between people and
319 machines.
320 Abdul et al. [17] evaluated a large corpus of explainable of recent studies on visual interpretability of neural net- 329

321 research based on 289 core papers and 12412 citing publica- works, covering visualization and diagnosis of CNN (Con- 330

322 tions and created a citation network to set an HCI (Human volutional Neural Network) representations, techniques for 331

323 Computer Interaction) research agenda in Explainability. disentangling CNN representations into graphs or trees, and 332

324 This work focused primarily on developing an HCI research learning of CNNs with disentangled and interpretable rep- 333

325 agenda in Explainability and investigating how HCI research resentations ending with a middle-to-end learning based on 334

326 might aid in the development of existing explainable systems model interpretability. 335

327 that are effective for end-users. Staying on the subject of visu- The authors of [19] employed a loss for each filter in 336

328 alization for XAI, [18] provides a comprehensive assessment high-level convolutional layers to force each filter to learn 337

VOLUME 10, 2022 93579

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

338 extremely particular object components to improve the inter- An external or surrogate model and the base model are used 394

339 pretability of traditional CNNs. Also, Angelov et al. [20] in post hoc approaches. The base model remains unmodified, 395

340 cover the visualization technique; in particular, they proposed while the external model generates an explanation for the 396

341 a broader taxonomy, considering whether the explanation users by mimicking the behavior of the base model. In addi- 397

342 is local or not, if the models are transparent or opaque, tion, post hoc approaches are classified into two groups: 398

343 if the techniques are model-specific or model-agnostic, and model-agnostic and model-specific. Model-agnostic meth- 399

344 whether explanations are created by simplification, conveyed ods can be used with any AI/ML model, but model-specific 400

345 through visualizations or based on feature relevance. In the approaches only apply to certain models. 401

346 same line, one of the works worth mentioning is that edited by Carvalho et al. [33] add a criterion on the stage of model 402

347 Arrieta et al. [21], which developed a new style of organiza- development, in-model interpretability that concerns ML 403

348 tion that first distinguishes between transparent and post-hoc models that have inherent interpretability in it (through con- 404

349 approaches and then creates sub-categories. straints or not). The need to consider the perspectives of 405

350 A methodological approach for evaluating the inter- diverse stakeholders is highlighted in [34]. As a result, expla- 406

351 pretability of ML models is proposed in [22], based nations should be adapted to the particular audience for which 407

352 on a taxonomy that separates three forms of Explain- they are intended to deliver the relevant information. In [35] 408

353 ability: imitate the processing, explain the representation, a survey of XAI methods in deployment is made, and [36] 409

354 and explain-producing networks. Methods for describing which considers the XAI for tabular data. To end this review 410

355 black-box models on a wide scale, such as data mining and of works in Explainable Artificial Intelligence it is worth 411

356 ML, were reviewed in [23]. They provided a full taxonomy considering also [37] where are identified future research 412

357 of Explainability strategies based on the problem they were directions with Explainability as the starting component of 413

358 dealing with. any AI system. 414

359 In [24] are examined and presented several XAI In this section, only works published in the last 5 years, 415

360 approaches, validation measures, and the types of explana- i.e., from 2018 to 2022, have been analysed. However, these 416

361 tions that can be generated to improve the acceptance of works are focused only on the survey of XAI methods empha- 417

362 expert systems among general users. sizing the most common ones and the general requirements 418

363 The authors in [25] focus on machine interpretation in the of explainability that are different in CyberSecurity context. 419

364 medical industry and reveal the difficulty of reading a black

365 box model’s choice. B. SURVEYS ON ARTIFICIAL INTELLIGENCE 420

366 In philosophy and sociology, Mittelstadt et al. [26] pay APPLICATIONS IN CYBERSECURITY 421

367 attention to the differences between these models and This section presents works that survey the existing literature 422

368 explanations. on AI applications in the world of CyberSecurity. AI and 423

369 Miller’s work [27] is likely the most important attempt to ML play a substantial role in the protection of computer 424

370 articulate the connection between human science and XAI. systems [13], [38], [39], [40], [41]. 425

371 Miller gave an in-depth assessment of studies on the expla- The interaction of AI and CyberSecurity was discussed by 426

372 nation problem in philosophy, psychology, and cognitive sci- the author in [42]. The study looked, in particular, at ML, and 427

373 ence in his paper. According to the author, the latter could be DL approaches to countering Cyber threats [43]. 428

374 a vital resource for the advancement of the field of XAI. There are various advantages and disadvantages to the use 429

375 In [28], the attention is focused on the fidelity of work of AI in this field, as briefly analyzed in [44] and [45], and 430

376 closely related to the explanation accuracy. The authors sur- work like that done in [46], where all the existing literature 431

377 veyed several studies that have evaluated explanation fidelity. on the last decade is analyzed, can be of help to those who are 432

378 Predictive accuracy, descriptive accuracy, and relevancy entering into the specific sector. 433

379 are three types of metrics presented by the Predictive, Sarker et al. [47] proposed a broad definition of Cyber- 434

380 Descriptive, and Relevant (PDR) framework for evaluating Security that takes into account all relevant definitions. 435

381 interpretability methodologies [29]. They discussed trans- Information Security, Network security, operational secu- 436

382 parent models and post-hoc interpretation, believing that rity, application security, Internet of Things (IoT) Security, 437

383 post-hoc interpretability could improve a model’s predictive Cloud security, and infrastructure Security are all covered by 438

384 accuracy and that transparent models could expand their use CyberSecurity [48]. 439

385 cases by increasing predictive accuracy, demonstrating that In [46], more than 770 papers were analyzed, and an 440

386 the combination of the two methods is ideal in some cases. overview of the challenges that ML techniques face in pro- 441

387 As presented in [30], an alternative perspective on hybrid tecting Cyberspace against attacks was provided by present- 442

388 XAI models entails augmenting black-box model expertise ing literature on ML techniques for CyberSecurity, including 443

389 with that of transparent model. intrusion detection, spam detection, and malware detection 444

390 The stages are ante-hoc and post-hoc, according to Vilone on computer and mobile networks. 445

391 and Longo [31], [32]. In general, ante-hoc methods consider Related to this, Gupta et al. [49] provide a thorough exam- 446

392 generating the rationale for the decision from the very begin- ination of the various ML and DL models used in mobile 447

393 ning of the data training to achieve optimal performance. network electronic information Security. 448

93580 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

449 The main distinction that came up when analyzing the In [66] Vigano et al. presented Explainable Security (XSec), 504

450 literature on this subject is the use of ML or DL techniques. a new security paradigm that involves several different stake- 505

451 In [50] and [51], both cases are analyzed with an in-depth holders and is multifaceted by nature. In [67] the authors 506

452 analysis of the various techniques used. Furthermore, both carried out a comprehensive literature review of various DL 507

453 papers specify that only the last three years of literature have architectures applied in CyberSecurity, including state-of- 508

454 been considered, showing that it is a field that has been the-art studies conducted with explainable AI. Indeed, [68] 509

455 receiving attention for not very long. focuses on Android Malware Defenses and XAI applications 510

456 Shaukat et al. [52] examined the performance of various in this field; they point out that nine out of ten primary sources 511

457 ML algorithms in terms of time complexity for identifying are proposed after 2019, indicating that Explainable Deep 512

458 Cyber-attacks. The authors focused on fraud detection, intru- Learning approaches for malware defenses are a current hot 513

459 sion detection, spam detection, and virus detection during research topic. 514

460 their investigation. Works analysed in this section are in the last 3 years, 515

461 Alabadi and Celik in [53] presented a comprehensive sur- i.e., from 2020 to 2022. Although all of these publications 516

462 vey about using CNN as a key solution for anomaly detection. are outstanding, none demonstrate how explainability occurs 517

463 Kim and Park [54] focus the attention on ML in in key sectors of AI in CyberSecurity, which is the primary 518

464 Cyber-Physical Systems (CPS), which is the integration of focus of this survey. 519

465 a physical system into the real world and control applications
466 in a computing system, interacting through a communica- V. LITERATURE REVIEW 520

467 tions network. They suggest a CPS structure that divides In the following subsections, the works that seek to achieve 521

468 the system’s functions into three layers: physical, network, explainability in the field of CyberSecurity were reviewed. 522

469 and software applications. In the sphere of CyberSecurity, In particular, the discussion focuses on the following appli- 523

470 researchers apply DL techniques for a variety of applications cation fields: 524

471 such as detecting network intrusions, malware traffic detec- • Intrusion Detection Systems 525
472 tion and classification, and so on, as analyzed extensively • Malware Detection 526
473 in [55], [56], [57], and [58]. • Phishing and Spam Detection 527
474 The performance of seven DL models on the CSE-CIC- • BotNet Detection 528
475 IDS2018 and Bot-IoT datasets is examined in [59]. The
The template used for describing the results of the analysis 529
476 models are evaluated on two datasets in this benchmark, and
of the works falling in the above application fields is this: 530
477 three evaluation metrics are reported. The whole execution
• Brief Introduction, a small analysis of the specific topic; 531
478 of the study is made public in order to facilitate objective
• Why XAI, a motivation based mostly on data, for why 532
479 comparisons and transparency in [60]. For the specific field
480 of phishing interesting approach is defined in [61] and for Explainable Artificial Intelligence is needed in that par- 533

481 ransomware attacks in [62]. ticular domain; 534

• State of art of AI methods, a quick look at applied AI 535

482 Also in this section, only works published in the last
483 5 years, i.e., from 2018 to 2022, have been analysed. How- methods; 536

• State of the art of Explainable Artificial Intelligence, 537

484 ever, these works are focused only on the survey of CyberSe-
485 curity threats and methods. an exhaustive analysis of existing XAI methods with a 538

specific focus on the explainability method; 539

486 C. XAI SURVEYS IN CYBERSECURITY • Consideration, a brief discussion of the analysis carried 540

487 Compared to the previous two sections, few works focus on out and an overview of the main directions explainable 541

488 and survey XAI methods in CyberSecurity. Currently, only methods are moving. 542

489 two work focus exclusively on this area, which are [63], [64]. In addition to the CyberSecurity applications aforemen- 543

490 However, it must be pointed out that in [63], the authors tioned above, other fields will be treated with lesser level of 544

491 provide a quick overview and, above all, do not pay attention detail, due to the availability of a fewer number of works, 545

492 on the different applications within CyberSecurity. In [64] focusing only on the review of works using XAI, that are: 546

493 the authors focus on application of XAI in CyberSecurity for Fraud Detection, Zero-Day Vulnerabilities, Digital Foren- 547

494 specific vertical industry sectors, namely in smart healthcare, sics, and Crypto-Jacking. 548

495 smart banking, smart agriculture, smart cities, smart gover- All application fields were selected according to the rele- 549

496 nance, etc.. vance and volume of literature to the current state of the art. 550

497 Exciting work is [65] where the authors made three con-
498 tributions: a proposal and discussion of desiderata for the A. INTRUSION DETECTION SYSTEMS 551

499 explanation of outputs generated by AI-based CyberSecu- Intrusion Detection Systems enable continuous security mon- 552

500 rity systems; a comparative analysis of approaches in the itoring of a cyber perimeter in order to timely identify attacks 553

501 literature on Explainable Artificial Intelligence (XAI), and a on computers and computer networks. 554

502 general architecture that can serve as a roadmap for guiding IDSs can be implemented with hardware appliances or with 555

503 research efforts towards AI-based CyberSecurity systems. special software; sometimes, they combine both systems [69]. 556

VOLUME 10, 2022 93581

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

557 They do not replace firewalls but integrate them to provide

558 more comprehensive protection. The purpose of the firewall
559 is to selectively (and ‘‘mechanically’’) intercept data packets
560 (according to a set of predefined rules that packets must fol-
561 low in order to enter or leave the local network). Traditional
562 firewalls operate on the lowest layers of network communi-
563 cation, thus with filtering rules limited to IP addresses, ports,
564 time of day and a few other criteria [70].
565 IDSs, on the other hand, are placed ‘‘downstream’’ of the
566 firewall and analyze data packets and the behaviour they
567 generate. Therefore, if an attack originates within the local
568 network, the firewall will not be able to block it. At the same
569 time, the IDS can detect anomalous situations.
570 IDS systems can be divided into two categories depending
571 on where the intrusion-detection sensors are placed (on the
572 network or a host/endpoint).
573 Network-based IDS systems (NIDS) analyze IP packets, FIGURE 5. IDS Publications from 2000 to 2021, retrieved from Scopus
574 policing the entire network data traffic. This way, they can using as search key [TITLE-ABS-KEY (intrusion AND detection AND
systems)].
575 complement the firewall where it does not block packets
576 due to misconfiguration or unrestrictive rules; they can also
577 monitor the behaviour of users inside the network. increasing trend of publications in this area. Most of these 609

578 Host-based intrusion detection systems (HIDS) are typ- developed in recent years are based on Machine and Deep 610

579 ically tools that are installed on a machine (host) and Learning algorithms. 611

580 are intended to protect a specific PC (a kind of ‘‘super- The approach based on ML and DL automates the analyti- 612

581 antivirus’’). They can also integrate firewall functions, sand- cal process to find intrusions. High performance, adaptability, 613

582 boxing, and so on. flexibility, and the capacity to identify zero-day assaults are 614

583 Another distinction can be made in detecting and alert- the significant benefits of the ML technique. However, there 615

584 ing approaches, which are Signature-based and Anomaly- are some drawbacks to ML-based IDS, including high bias 616

585 based. While Signature-based detection is used to detect propensity, inability to manage outliers, difficulties handling 617

586 known threats, Anomaly-based detection detects changes in huge datasets, and complex data preprocessing. 618

587 behaviour. Signature-based detection is based on a predefined The DL-based approach can handle dynamic data 619

588 set of known Indicators Of Compromise (IOCs). Malicious that changes over time, recognize large-scale and multi- 620

589 network attack behaviour, email subject line content, file dimensional data and identify anomalies in the data. Never- 621

590 hashes, known byte sequences, or malicious domains are theless, DL-based approaches have many drawbacks, such as 622

591 all examples of IOCs. Signatures may also include network a lack of flow information, vulnerability to evasion attempts, 623

592 traffic alerts, such as known malicious IP addresses attempt- poor data knowledge required to design relevant features, and 624

593 ing to access a system. Unlike Signature-based detection, a lack of qualified domain experts to review the implemen- 625

594 Anomaly-based detection can discover unknown suspicious tation. These very latter two points lead back to the need for 626

595 behaviour. Anomaly detection begins by training the system explainability, a need shared by any agent attempting to give 627

596 with a normalized baseline and comparing activity to that an explanation for the model result and be able to improve it 628

597 baseline. consequently. 629

598 1) WHY XAI IN IDSs? 2) ARTIFICIAL INTELLIGENCE IN IDSs 630

599 In BakerHostetler’s 2021 Data Security Incident Report,3 Chawla et al. [71] propose a Host-based IDS that uses 631

600 some interesting numbers help to understand why the col- sequences of system calls to identify the expected behaviour 632

601 laboration of AI and humans is needed to combat an already of a system. The work describes an efficient Anomaly-based 633

602 huge problem. 58 % of detected incidents are attributable intrusion detection system based on CNN layers to capture 634

603 to Network Intrusion, the most significant cause among the local correlations of structures in the sequences and Gated 635

604 top 5. Recurrent Units layer to learn sequential correlations from 636

605 On average, in 2020 were needed 92 days to discover the the higher level features. 637

606 presence of an intrusion, 6 days to contain it, 42 days for By examining Linux kernel 5.7.0-rc1, the authors of [72] 638

607 forensic efforts to complete, and 90 days total from the date bridge the gap between theoretical models and application 639

608 of discovery to notification to end-user. Figure 5 shows the settings. This environment investigates the viability of HIDS 640

in modern operating systems and the constraints placed 641

3 https://www.bakerlaw.com/webfiles/Privacy/2021/Alerts/2021-DSIR- on HIDS developers. Keeping the focus on HIDS in [73], 642

Report.pdf Gassais et al. propose a framework for intrusion detection in 643

93582 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

644 IoT which combines user and kernel space using AI tech- modern Cyber threats. The strategy adopted makes the final 699

645 niques to automatically get devices behavior, process the decision after cross-validation of the local explanation of the 700

646 data into numeric arrays to train several ML algorithms, and predicted outcome with the global explanation of SHAP. 701

647 raise alerts whenever an intrusion is found. In [74] and [75] The general idea proposed in [95] against adversarial 702

648 the authors focus the attention on Cloud Environment by attacks is divided into two parts, initialization and detection. 703

649 detecting Anomalies while [76] propose a Siamese-CNN to During initialization, the model is trained with an SVM and 704

650 determine the attack type converting it to an image. features and characteristics that make a Normal classification 705

651 Analyzing the Network-based approaches, in [77], the are deduced via LIME. During detection, the Intrusion Detec- 706

652 authors present a NIDS model that employs a non-symmetric tion System goes to compare. If it does not find the data as 707

653 deep AutoEncoder and a Random Forest classifier. Using a Normal, it classifies as an attack. On the other hand, if it is 708

654 non-symmetric deep Auto Encoder for efficient feature selec- classified as Normal, there is a risk of an adversarial attack 709

655 tion reduces the model’s complexity, similar to [78] and [79] that is fooling the model. So a further check is done by reusing 710

656 where the classifier is the Support Vector Machine. LIME. After that, the final result is reached. 711

657 Ali et al. in [80] use a Fast Learning Network with a FAIXID [96] is a new proposed framework that uses data 712

658 Swarm optimization algorithm, similar to the works in [81] cleaning techniques. They used four algorithms in the exper- 713

659 and [82]. The most recent work brings the spotlight on the use iment to make the results explainable. They use the Boolean 714

660 of Neural Networks [83], [84] and Adversarial Methods [85], Rule Column Generation (BRCG) algorithm [97], which pro- 715

661 [86], [87]. vides a directly interpretable supervised learning method for 716

binary classification. Logistic Rule Regression (LogRR) [98] 717

662 3) EXPLAINABLE ARTIFICIAL INTELLIGENCE IN IDSs is a directly interpretable supervised learning method that can 718

663 In [88], a system is proposed that is based on rules dictated perform logistic regression on rule-based functions. The Pro- 719

664 by experts. It is Hybrid in the sense that it is a combination of toDash algorithm [99] provides example-based explanations 720

665 human work and ML. The Explainability comes from Rule- to summarize datasets and explain the predictions of an AI 721

666 based; the model behind it is a Decision Tree, a white-box model. Finally, the Contrastive Explanations Method (CEM) 722

667 model. is used to compute explanations that highlight both relevant 723

668 Szczepanski et al. in [89] propose a combination of oracle positives (PP) and relevant negatives (NP). Their proposal is 724

669 (ML model, in this case, tested ANN with a PCA) and an not static but involves the use of algorithms depending on the 725

670 explainer module that would explain why a given classifica- specific case. 726

671 tion is made. In the explainer module, one compares the dis- The work proposed in [100] defines a method to make 727

672 tance from the clusters created on the training data. Then, the rules for accessing the network dynamically and not statically 728

673 cluster closest to the test set instance is used for explanation. as, for example, the rules set in a firewall may be. Thus, 729

674 In [90], the idea is to use an adversarial approach in order Explainability is the focus of the proposal. The explanation 730

675 to be able to account for the minimal changes necessary for a of the results consists of two main steps: i) training a model to 731

676 classifier to arrive at an incorrect classification. The method approximate the local decision boundary of the target predic- 732

677 thus makes it possible to visualize the features responsible for tive model, and ii) reasoning about the trained model and the 733

678 misclassification. For example, regular connections with low given input based on an explanation logic. The explanation is 734

679 duration and low login success are misclassified as attacks. Local-based. They are inspired by LEMNA [101]. 735

680 In contrast, attack connections with a low error rate and The aim in [102] is to increase transparency in an IDS 736

681 higher login success are misclassified as regular, demonstrat- based on a Deep Neural Network. Feedback is presented by 737

682 ing that relevant features significantly affect the final result. computing the input features most relevant to the predictions 738

683 A new way of interpreting an Intrusion Detection System made by the system. The model adopted is an MLP. Two 739

684 is presented in [91]. The authors propose the use of SHAP for forms of feedback are generated: 1) offline feedback (after 740

685 both local and global explanations. SHAP, by its nature, is a training, before deployment) and 2) online feedback (during 741

686 local method; they propose combining all local explanations deployment). In offline feedback, the user is given the most 742

687 to obtain a global explanation of the model. Almost equal relevant input features for each concept learned from the sys- 743

688 work, with some less experimentation, is proposed in [92]. tem. This information allows the user to evaluate whether the 744

689 Le et al. [93] propose similar work through SHAP with an input characteristics that guide the IDS’s decision toward a 745

690 ensemble Tree model given a Decision Tree and a Random particular class (i.e., the type of attack) align with the domain 746

691 Forest model. Specifically, at the global level, they use a experts’ knowledge. On the other hand, the user is given the 747

692 Heatmap for visualizing the impact of individual features on most relevant input characteristics for each prediction in the 748

693 the classification of the overall model. At the local level, online feedback. 749

694 they use a Decision Plot to explain decisions on individual In [103], the authors focus on the possibilities of analyzing 750

695 instances of the datasets. Another similar work is the frame- encrypted traffic, particularly for accurate detection of DoH 751

696 work proposed by [94], consisting of a Random Forest model (DNS Over HTTPS) attacks. They implement an explainable 752

697 using SHAP. The model can assess the credibility of the pre- AI through the use of SHAP that allows visualizing the 753

698 dicted results and ensure a high level of accuracy in detecting contribution of individual features to the model classification 754

VOLUME 10, 2022 93583

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

755 decision. Similarly, EXPLAIN-IT [104] is applied to the It would be good to consider frameworks with intrinsic 806

756 YouTube video quality classification problem in encrypted interpretability and not the application of methods for a 807

757 traffic scenarios. The work is based on a methodology that post-hoc explanation. Furthermore, the final output should 808

758 deals with unlabeled data, create meaningful clusters and be aimed at precise figures and not just any user, such as 809

759 proposes an explanation of the clustering results to the end- analysts and defenders. To be explored for future research 810

760 user. They use LIME interpreting clusters that are associated is the topic of adversarial attacks where the collaboration 811

761 with a Local-based strategy then. Alike, ROULETTE [105] between humans and machines is necessary and explanations 812

762 focuses on Network traffic. Specifically, attention is coupled are fundamental to combat this type of intrusion. 813

763 with a multi-output DL strategy that helps better discriminate

764 between network intrusions categories. As Post-hoc expla- B. MALWARE DETECTION 814

765 nations, they consider visual explanation maps produced The term malware refers to programs potentially harmful to 815

766 through Grad-CAM. the user, which are aimed at stealing sensitive data, control- 816

767 A two-stage ML-based Wireless Network IDS (WNIDS) is ling the PC, or stealing user identity. The term malware origi- 817

768 implemented in [106] to improve the detection of imperson- nates from the contraction of the words ‘‘malicious software’’ 818

769 ation and injection attacks in a Wi-Fi network. The XAI was and stands for a program (an executable, a dynamic library, 819

770 implemented to gain insight into the decisions made by the a script, an HTML page, a document with macros, etc.) 820

771 first-stage ML model, especially for cases where records were having unwanted and potentially dangerous effects on the 821

772 predicted as impersonation or injection. The features that user such as stealing sensitive data, controlling activity at the 822

773 contribute significantly to their prediction were determined. PC, identity theft, encrypting the hard disk with subsequent 823

774 This set of features almost corresponds to those identified by ransom demands, and so on. 824

775 the feature selection method for the second-stage ML model. Malware is usually classified according to its behaviour as 825

776 They use SHAP. Botnet, Backdoor, Information Stealer, Downloaders, Scare- 826

777 In [107], the authors create a framework with a Deep ware, Rootkit, Worm, Virus, Ransomware or Trojans. 827

778 Neural Network at its base and apply an XAI method depend- Some of the most common methods an attacker uses 828

779 ing on who benefits from it. For data scientists, SHAP are Spam, Phishing, Hacking, Banner advertising, Search 829

780 and BRCG [97] are proposed, while for analysts Protodash page rank, Expired domains or Domain Name Server (DNS) 830

781 is used. For end-users where an explanation on the sin- hijacking. 831

782 gle instance is required, they suggest SHAP, LIME, and Malware detection techniques can be classified into 832

783 CEM. Saran et al. [108] propose a comparison between the three main categories (although other classifications 833

784 NetFlow-based feature set4 and the feature set designed by exist): (i) Signature-based, (ii) Anomaly-based, and 834

785 the CICFlowMeter tool.5 This reliable comparison demon- (iii) Heuristic-based. 835

786 strates the importance and need for standard feature sets When using a Signature-based approach, programmers 836

787 among NIDS datasets, such as evaluating the generalizability scan a file for malware, compare the information with a 837

788 of ML model performance in different network environments database of virus signatures, and then verify the results. If the 838

789 and attack scenarios. The SHAP method is used to explain the information matches the information in the database, the file 839

790 prediction results of ML models by measuring the importance is infected with viruses. This approach limits the detection 840

791 of features. For each dataset, key features that influence of unknown malware, but its main advantage is that it works 841

792 model predictions were identified. well for known malware. 842

793 In conclusion, this work mentions [109], where an explain- Anomaly-based methods mitigate the limitations of 843

794 able automotive intrusion detection system is proposed, signature-based techniques, allowing detection of any known 844

795 and [110] where a new general method is presented and tested or unknown malware by applying classification techniques 845

796 on an IDS dataset. In [111] instead, the authors emphasize the to the actions of a system for malware detection. Detec- 846

797 importance of trust but do not use XAI methods. tion of malware activity is improved by moving from 847

pattern-based to classification-based detection to identify 848

798 4) CONSIDERATIONS ABOUT IDS AND XAI normal or anomalous behaviour. Applying AI to Signature- 849

799 It is interesting to note that most of the methods analyzed use based and Anomaly-based detection systems improves the 850

800 already developed methods to make the results explainable, efficiency of malware detection. Heuristic-based method use 851

801 so the explanation is post-hoc. In particular, in the case of data mining and ML techniques to learn the behavior of an 852

802 methods already in the research landscape, SHAP is the most executable file. 853

803 adopted method. LIME, on the other hand, has been adopted
804 in only one case. Some frameworks are white-box in nature; 1) WHY XAI IN MALWARE DETECTION? 854

805 most are based on a decision tree. According to AV-Test Institute,6 more than 1 billion malware 855

programs are out there, and 560, 000 new pieces of mal- 856

ware are detected every day. Statista detected that 68.5% of 857
4 https://en.wikipedia.org/wiki/NetFlow
5 https://github.com/CanadianInstituteForCyberSecurity/CICFlowMeter 6 https://www.av-test.org/en/statistics/malware/

93584 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

Staying on the use of graphs but moving to Signature- 890

based systems, HLES-MMI [121] is a method that iden- 891

tifies metamorphic malware families based on computing 892

the similarities among the higher-level engine signatures. 893

Khan et al. [122] analyzed ResNet and GoogleNet models 894

while [123], [124] focus the attention on private cloud envi- 895

ronments and detection for non-domain experts. 896

A Hybrid-based approach method is proposed in [125] 897

where the framework use more than one complementary filter 898

and a wrapper feature selection approach to identify the most 899

significant runtime behavioural characteristics of malware. 900

An approach where AI is proliferating is the detection 901

by image visualization. For example, Baptista et al. [126] 902

designed an image-based malware detection tool based on 903

unsupervised learning testing to determine if malicious files 904

could be differentiated from benign ones by focusing on 905

FIGURE 6. Malware Detection Publications from 2001 to 2021, retrieved features extracted from their visual representation. In [127], 906
from Scopus using as search key [TITLE-ABS-KEY (malware AND the defined architecture consists of three main components: 907
detection)].
image generation from malware samples, image augmenta- 908

tion, and classification in a malware family using CNN mod- 909

els. Other similar works are [128], [129], [130], [131]. In the 910
858 businesses were victimized by ransomware in 2021, a con- Android world it is worth considering DL-DROID, an auto- 911
859 siderable increase from the previous three years. Overall, the mated dynamic analysis framework for Android malware 912
860 number of detected malware types stood at 28.84 million detection. In [132] and [133] satisfying results are obtained 913
861 in 2010; by 2020, this had reached nearly 678 million.7 using ML and DL techniques. However, the main problem 914
862 Figure 6 shows the increasing trend of publications in this remains the non-Explainability and the subsequent lack of 915
863 area, reflecting its considerable attention. XAI can assist trust in model outcomes, so the next section will explore 916
864 with risk identification and prioritization, incident response works that somehow attempt to solve this problem. 917
865 coordination, and malware threat detection. XAI appears to
866 be a good answer in situations demanding explainability, 3) EXPLAINABLE ARTIFICIAL INTELLIGENCE IN MALWARE 918
867 interpretability, and accountability, where humans require DETECTION 919
868 assistance in fighting a massive number of attacks. One of the main works in this area is Drebin [134]; how- 920

ever, for consistency, it will not be analyzed in-depth as 921

869 2) ARTIFICIAL INTELLIGENCE IN MALWARE DETECTION it is a pre-2018 work. Drebin explains his decisions by 922

870 In [112], the authors propose an Anomaly-based approach reporting, for each application, the most influential features, 923

871 where the system employs significant features of activ- i.e., those present in the application and to which the clas- 924

872 ity to model normal and malicious behaviour of users in sifier assigns the highest absolute weights. Melis et al. [135] 925

873 Cloud-based environments. Similar are the works in [113] provide an approach for the Explainability of malware detec- 926

874 and [114] where extreme surveillance through malware hunt- tion in Android systems with an extension of the conceptual 927

875 ing is delivered. Keeping with Anomaly-based approaches, approach provided by Drebin on non linear models. Stay- 928

876 Alaeiyan et al. introduce [115] VECG, a tool for exploring ing focused on Mobile, the authors of [136] use LIME in 929

877 and supplying required environmental conditions at runtime, a method to identify locations deemed important by CNN 930

878 while in [116] Stiborek et al. propose a novel tool that detects in the opcode sequence of an Android application to help 931

879 malware observing the interactions between the operating detect malware, while Kumar et al. [137] propose a static 932

880 systems and network resources. methodology for malware detection in Android where Fea- 933

881 ASSCA [117] is a system architecture that combines the ture Extraction provides transparency. 934

882 DL model based on sequence data and the ML model based XMal [138] is an MLP-based approach with an attention 935

883 on API statistical features, similar to what happens in [118] mechanism to detect when an Android App is malware. The 936

884 where the API call relation is extracted, the ordered cycle interpretation phase aims to automatically produce neural 937

885 graph is constructed based on Markov chain and then the language descriptions to interpret key malicious behaviours 938

886 graph convolution neural network (GCN) detects malware. within apps. Although the method is not so clear, the authors 939

887 Other exciting works based on DL of Behavior Graphs say they achieve better performance in interpretation than 940

888 are [119], [120] where for the detection are used file content LIME and DREBIN. 941

889 and file relations. The authors in [139] propose a backtracking method to pro- 942

vide a high-fidelity explanation of the DL detection method. 943

7 https://www.statista.com/topics/8338/malware/dossierKeyfigures The backtracking method selects the most important features 944

VOLUME 10, 2022 93585

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

945 contributing to the classification decision, thus resulting in a binary, they try to use this framework to demystify the work- 1001

946 transparent and multimodal framework. ings of the MalConv architecture. As a result, they can better 1002

947 Feichtner et al. [140] designed a Convolutional Neu- explain the workings of ML algorithms and the decisions 1003

948 ral Network (CNN) to identify sample-based correlations they make using the proposed framework. Additionally, the 1004

949 between parts of the description text and the permission analysis will enable network inspection without starting from 1005

950 groups an app requests. They employ LIME to calculate a scratch. 1006

951 score for each word that shows the output’s significance and Hsupeng et al. [148] introduce an explainable flow-data 1007

952 visualize it as a heatmap. classification model for hacker attacks and malware detec- 1008

953 As analyzed in the previous section, several methods focus tion. The flow data used for training the model is converted 1009

954 on malware detection as an image; in [141], the authors from packets by CICFlowMeter. This process significantly 1010

955 propose a method relying on application representation in shrank the data size, reducing the requirement for data stor- 1011

956 terms of images used to input an Explainable Deep Learning age. For Explainability, they utilize SHAP further to inves- 1012

957 model. They represent a mobile application in terms of image tigate the relation between cyberattacks and network flow 1013

958 and localize the salient parts useful to the model to output features. 1014

959 a certain precision by exploiting the Grad-CAM algorithm. MalDAE [149] is a framework that explores the dif- 1015

960 In this way, the analyst can acquire knowledge about the areas ference and relation between the dynamic and static API 1016

961 of the image symptomatic of a specific prediction. call sequences, which are correlated and fused by seman- 1017

962 Shifting the focus from mobile applications to more gen- tics mapping. MalDAE provides a practical and explainable 1018

963 eral ones, LEMNA [101] is one of the main methods in framework for detecting and understanding malware based on 1019

964 the landscape of Explainability techniques. It was developed correlation and fusion of the static and dynamic characteris- 1020

965 specifically for DL-Based Security Applications and is, there- tics. The explainable theoretical framework divides all API 1021

966 fore, one of the references in the general field of CyberSe- calls into several types of malicious behaviours according to 1022

967 curity. It was included in this section because the authors’ their impact on security and builds a hierarchical malware 1023

968 primary experimentation is conducted on a Malware Detec- explanation architecture. 1024

969 tion Dataset. Given a sample of input data, LEMNA generates Several works in the literature attempt to interpret malware 1025

970 a small set of interpretable features to explain how the input detection by generating Adversarial attacks. The authors 1026

971 sample is classified. The central idea is to approximate a local in [150] discovered that MalConv neural network does not 1027

972 area of the complex DL decision boundary using a simple learn any useful characteristics for malware detection from 1028

973 interpretable model. LEMNA uses a fused lasso-enhanced the data and text sections of executable files but instead has 1029

974 mixed regression model to generate high-fidelity explanation a tendency to learn to distinguish between benign and mali- 1030

975 results for a range of DL models, including RNN. cious samples based on the characteristics found in the file 1031

976 DENAS [142] is a rule generation approach that extracts header. Based on this discovery, they devised a novel attack 1032

977 knowledge from software-based DNNs. It approximates the method that creates adversarial malware binaries by altering 1033

978 nonlinear decision boundary of DNNs, iteratively superim- a small number of file header bytes. For the explanation, 1034

979 posing a linearized optimization function. they use Feature Attribution to identify the most influential 1035

980 CADE [143] is designed to detect drifting samples that input features contributing to each decision and adapt it 1036

981 deviate from the original training distribution and provide the to provide meaningful explanations for classifying malware 1037

982 corresponding explanations to reason the meaning of the drift. binaries. Other such works are [151], [152] employing SHAP 1038

983 The authors derive explanations based on distance changes, and [153] proposing a new explanation algorithm to identify 1039

984 i.e., features that cause the most significant changes to the the root cause of evasive samples. It identifies the minimum 1040

985 distance between the drifting sample and its nearest class. number of features that must be modified to alter the decision 1041

986 It was included in this paragraph because it is tested on a of a malware detector, using Action Sequence Minimizer and 1042

987 Malware detection dataset. Feature Interpreter. 1043

988 Pan et al. [144], [145] in two related works propose a To conclude the section, it is necessary to analyze the 1044

989 hardware-assisted malware detection framework developing work of Fan et al. [154]. They designed principled guide- 1045

990 a regression-based Explainable Machine Learning algorithm. lines to assess the quality of five explanation approaches by 1046

991 They apply a Decision Tree or Linear Regression to interpret designing three critical quantitative metrics to measure their 1047

992 the final result. Stability, Robustness, and Effectiveness. The five explanation 1048

993 In order to understand how a Deep Network architecture approaches are SHAP, LIME, Anchors, LEMNA and LORE. 1049

994 generalizes to samples that are not in the training set and Based on the generated explanation results, they conducted a 1050

995 explains the outcomes of deep networks in real-world test- sanity check of such explanation approaches in terms of the 1051

996 ing, the authors of [146] propose a framework that interpo- three metrics mentioned. Based on their analysis, the ranking 1052

997 lates between samples of different classes at different layers. of the five explaining approaches in terms of the Stability 1053

998 By examining the weights and gradients of various levels metric is LIME ≥ SHAP > Anchors > LORE > LEMNA. The 1054

999 in the MalConv architecture [147] and figuring out what ranking of the five explaining approaches in the Robustness 1055

1000 the architecture discovers by examining raw bytes from the metric is LIME > SHAP > Anchors > LORE > LEMNA. 1056

93586 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

Spam is also called junk mail. It has existed almost as 1090

long as the internet as a means of selling products or services 1091

to a larger market of buyers than have ever expressed inter- 1092

est in those products or services. After obtaining the email 1093

addresses of a considerable number of individuals, spammers 1094

bulk send their offers hundreds or thousands at a time. Spam 1095

can be very dangerous if it is part of a phishing attempt. 1096

1) WHY XAI IN PHISHING AND SPAM DETECTION ? 1097

According to the IC3 report,8 Phishing (including vishing, 1098

SMiShing, and pharming) was the most common threat in the 1099

United States in 2020, with 241, 342 victims. Following that 1100

were nonpayment/non-delivery (108, 869 victims), extortion 1101

(76, 741 victims), personal data breach (45, 330 victims), and 1102

identity theft (43, 330 victims). These data show how huge 1103

this problem directly affects the population, which, if not well 1104

educated, can easily fall into the trap. The Figure 8 proves the 1105
FIGURE 7. Phishing attacks grouped per Quarter9 .
dizzying amount of attention that Phishing attack detection 1106

is attracting from academics in recent years. Explaining to a 1107

user why a particular email is a phishing attempt or why it 1108

1057 In the Effectiveness metric is LIME > LORE > Anchors ≥ has been classified as Spam is no slight advantage. XAI in 1109

1058 SHAP > LEMNA. this field is directly connected to the population that could 1110

benefit from it to prevent a threat that is now constant. 1111

1059 4) CONSIDERATIONS ABOUT MALWARE DETECTION

1060 AND XAI 2) ARTIFICIAL INTELLIGENCE IN PHISHING AND SPAM 1112

1061 Several recent publications attempting to explain the results DETECTION 1113

1062 of a malware detector have been reviewed. The signifi- Phishing. State of the art on the application of AI in Phishing 1114

1063 cantly smaller number of algorithms that perform detection Detection is substantial, so only recent works with the most 1115

1064 using images stands out compared to DL, and black-box significant impact in terms of citations have been analyzed. 1116

1065 ML approaches. Another factor to note is the significant Hybrid Ensemble Feature Selection (HEFS) is an 1117

1066 effort put into developing Explainable methods in Mobile interesting approach proposed in [155] with a new feature 1118

1067 environments, particularly on Android platforms. Comparing selection framework. In the first phase of HEFS, a novel 1119

1068 the Black-box and Explainable methods, it is surprising how Cumulative Distribution Function gradient (CDF-g) algo- 1120

1069 fewer graph-based methods are used in the latter than in the rithm is exploited to produce primary feature subsets, which 1121

1070 former; using these for greater transparency might be a good are then fed into a data perturbation ensemble to yield sec- 1122

1071 starting point. Several articles use established techniques with ondary feature subsets. The second phase derives a set of 1123

1072 Post-hoc Explainability that can help the analyst understand baseline features from the secondary feature subsets using 1124

1073 the basis on which the model is categorized, particularly a function perturbation ensemble. The best performance is 1125

1074 SHAP and LIME. Another widely used technique is Feature achieved with Random Forest. The latter is one of the seven 1126

1075 Attribution, which works similarly to the above approaches. implemented and compared models for the real-time detec- 1127

1076 What appears to be obvious is the necessity for applications tion of phishing web pages by investigating the URL of the 1128

1077 created with Intrinsic Explanation rather than Post-hoc, as is web page explored in [156]. In [157], Yerima et al. propose 1129

1078 usually the case. The Explanation in these cases is built during an approach based on a Convolutional Neural Network tested 1130

1079 data training. The model should be a Hybrid of Signature- and on a dataset obtained from 6, 157 genuine and 4, 898 phishing 1131

1080 Anomaly-based methodologies that, when applied together, websites; a small dataset instead is used in [158] where the 1132

1081 can give significant benefits. However, it should be recog- authors introduce a Deep Belief Network (DBN). Jain et al. 1133

1082 nized that significant progress is being made in this area. propose a ML-based novel Anti-Phishing approach that 1134

extracts the features from the client-side only. They examined 1135

1083 C. PHISHING AND SPAM DETECTION

the various attributes of Phishing and legitimate websites 1136

in-depth. As a result, they identified nineteen outstanding 1137

1084 Phishing refers to a particular type of Internet fraud; the
features to distinguish Phishing websites from legitimate 1138
1085 purpose of the malicious attackers, in this circumstance, is to
ones. DTOF-ANN (Decision Tree and Optimal Features 1139
1086 get hold of users’ personal and confidential data. More specif-
based Artificial Neural Network) [159] is a Neural-Network 1140
1087 ically, phishers practice the theft of logins and passwords,
1088 credit card and bank account numbers, and additional con-
1089 fidential data. 8 https://www.ic3.gov/Media/PDF/AnnualReport/2020_IC3Report.pdf

VOLUME 10, 2022 93587

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

FIGURE 8. Phishing Detection Publications from 2004 to 2021, retrieved from Scopus using as search key [TITLE-ABS-KEY (phishing AND
detection)].

1141 Phishing detection model based on a Decision Tree and detection task. The evaluation is done on available datasets 1173

1142 Optimal feature selection. comprising 6, 000 spam and 2, 313 non-spam image samples. 1174

1143 The authors of [160] propose Jail-Phish, a Heuristic tech- Other interesting works are [167], [168]. 1175

1144 nique which uses Search Engine results and Similarity-based These works are mostly based on Deep Neural Networks 1176

1145 features to detect Phishing sites. in which Interpretability and Explainability of the final detec- 1177

1146 The last work to be highlighted for Phishing Detection is tion are challenging, so the next section will analyze the 1178

1147 PhishBench [161], a benchmarking framework that can help state of the art of explainable models in Phishing and Spam 1179

1148 researchers by providing a template to develop new methods Detection. 1180

1149 and features as well as a platform to compare their proposed

1150 techniques with previous works.
1151 Spam. Shifting to Spam Detection, an intelligent system 3) EXPLAINABLE ARTIFICIAL INTELLIGENCE IN PHISHING 1181

1152 that is based on Genetic Algorithm (GA) and Random Weight AND SPAM DETECTION 1182

1153 Network (RWN) is proposed in [162]. A similar proposal The current state of the art for Phishing and Spam detection 1183

1154 is given by [163] where the authors propose a combina- with explainable methodologies is relatively poor. Therefore, 1184

1155 tion of the Word Embedding technique and Neural Network techniques that are not created on-demand for Phishing and 1185

1156 algorithm. Spam Detection but use datasets targeted at these application 1186

1157 Barushka et al. [164] propose a Spam filter integrating an domains were also considered. 1187

1158 N-gram tf-idf feature selection, a modified distribution-based Phishing. Phishpedia [169] is a Hybrid DL system that 1188

1159 balancing algorithm and a regularized Deep multi-layer per- addresses two prominent technical challenges in phishing 1189

1160 ceptron NN model with rectified linear units (DBB-RDNN- identification, (i) accurate recognition of identity logos on 1190

1161 ReL). In the same wake Douzi et al. [165] present a Hybrid webpage screenshots and (ii) matching logo variants of the 1191

1162 approach based on the Neural Network model Paragraph same brand. The authors compare the identity logo and input 1192

1163 Vector-Distributed Memory (PV-DM). box providing Explainable annotations on webpage screen- 1193

1164 In [166], the authors propose Spam detection in social shots for the Phishing report. 1194

1165 media with a DL architecture based on Convolutional Neu- Two works where the goal is not Phishing detection, but 1195

1166 ral Network (CNN) and Long Short Term Neural Network a dataset of this type is used for tests are [170], [171]. 1196

1167 (LSTM). The first is based on a Deep embedded Neural Network 1197

1168 DeepCapture is an image spam email detection tool based expert system (DeNNeS) with a rule extraction algorithm 1198

1169 on a Convolutional Neural Network (CNN). The key idea for Explainability. The second is based on the Multi-Modal 1199

1170 is built on a CNN-XGBoost framework consisting of eight Hierarchical Attention mechanism (MMHAM) that permits 1200

1171 layers only with a large number of training samples using data the Explainability thanks to the hierarchical system. 1201

1172 augmentation techniques tailored towards the image Spam Kluge et al. [172] propose a framework to convey to the 1202

user which words and phrases in an e-mail influenced a 1203

9 Source: https://apwg.org/trendsreports/ Phishing detector’s classification of the e-mail as suspicious. 1204

93588 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

1205 They do it by locally perturbing inspiring to Anchors. The the decision-making process from becoming less effective for 1260

1206 last analyzed work is [173], where the authors use LIME and the business and the individual user. As seen in the analysis 1261

1207 Explainable Boosting Machine (EBM) [174]. conducted in [175], the user accepts AI makes mistakes, 1262

1208 Spam. The authors of [175] looked into how different as long as it is explained how and why so that it can improve in 1263

1209 ML explanations, ML model’s accuracy, and user confidence the case of a false negative above all. A consideration beyond 1264

1210 in the ML model affect user performance in a simulated XAI in CyberSecurity is the education that must be provided 1265

1211 Spam detection task. According to their findings, a user’s to everyone with a technological device which happens to be 1266

1212 confidence level in the model significantly influences the surfing the internet where Phishing and Spam are continually 1267

1213 decision process. Users performed better when using an accu- around the corner. Similar to how one trains models, one 1268

1214 rate model. Participants were more likely to spot false alarms might devise strategies to teach individuals to avoid falling 1269

1215 generated by the more accurate model and more willing to victim to these scams. These strategies need to be Explainable 1270

1216 follow through on a model ‘‘miss’’ when an additional model so that anyone can comprehend why certain decisions are 1271

1217 explanation was given. taken. 1272

1218 FreshGraph [176] is a two-step system for recommending

1219 new products to target people that is Spam-aware. First, use D. BOT (Net) DETECTION 1273

1220 item-user Meta-Path similarity and then entropy encoding A ‘‘Bot’’ or Robot, is a software program that performs 1274

1221 measurements on a heterogeneous information network struc- automatic, repetitive, preset operations. Bots often mimic 1275

1222 ture to identify false positives from candidate lists and avoid or replace the behaviour of human users. Since they are 1276

1223 potential Spam. The suggested approach takes advantage of automated, they work considerably more quickly than actual 1277

1224 the semantic data stored within the graph structure, which individuals [182]. 1278

1225 considers user activity in addition to item content aspects Malware and Internet bots can be programmed/hacked 1279

1226 for more precise audience targeting. Graph structure provides to access users’ accounts, search the Internet for contact 1280

1227 Explainability. information, transmit Spam, and execute other dangerous 1281

1228 Gu et al. [177] examine the use of DL models to predict operations. Attackers may use malicious Bots in a Botnet, 1282

1229 the effectiveness of outbound telemarketing for insurance or network of Bots, to launch these attacks and conceal their 1283

1230 policy loans to decrease Spam problems created by phon- source. A Botnet is a collection of online-connected devices 1284

1231 ing non-potential customers. They propose an Explainable running one or more Bots, frequently without the owners’ 1285

1232 multiple-filter Convolutional Neural Network (XmCNN) to knowledge. Since each device has a unique IP address, Botnet 1286

1233 reduce overfitting. Explainability is calculated using feature activity comprises many IP addresses, making it more chal- 1287

1234 importance by including a CancelOut layer after the input lenging to locate and stop the source of malicious Bot traf- 1288

1235 layer. fic. When used to infect additional computers, Spam e-mail 1289

1236 These two methods avoid getting into spam and are not recipients’ devices can help Botnets grow larger. They are 1290

1237 spam detector methods. However, they still use Explainable commanded by hackers known as Botmasters or Bot herders. 1291

1238 methods of AI to avoid spam; that is why they were analyzed Botnets are hard to spot since they consume very few 1292

1239 in this section. computer resources. This keeps them from interfering with 1293

1240 The following analysis will focus on techniques that were applications’ regular operation and does not make the user 1294

1241 not created to avoid Spam but instead use Spam datasets suspicious. However, the most sophisticated Botnets can also 1295

1242 as testing. GRACE [178] generates contrastive samples that alter their behaviour by the CyberSecurity systems of the PCs 1296

1243 are concise, informative and faithful to the neural network to evade detection. Most of the time, users are unaware that 1297

1244 model’s specific prediction. SLISEMAP [179] finds local their devices are part of a Botnet and are under the control of 1298

1245 Explanations for all data items and builds a (typically) two- online criminals [183]. 1299

1246 dimensional global visualization of the black box model such

1247 that data items with similar Local Explanations are projected 1) WHY XAI IN BOT (Net) DETECTION? 1300
1248 nearby. [180], [181] are two works focused on text classifica- Spamhaus monitors both IP addresses and domain names 1301
1249 tion that use Spam datasets. used by threat actors to run botnet Command & Con- 1302

trol (C&C) servers. As a result, Spamhaus Malware Labs 1303

1250 4) CONSIDERATIONS ABOUT PHISHING AND SPAM
researchers found and blacklisted 17, 602 botnet C&C servers 1304
DETECTION AND XAI
1251
hosted on 1, 210 distinct networks.10 This represents a mas- 1305
1252 As anticipated earlier, state of the art of Explainable Artificial sive 71.5% increase over the number of botnet C&Cs wit- 1306
1253 Intelligence in Phishing and Spam detection is very meagre. nessed in 2018. Since 2017, the number of newly discovered 1307
1254 From the analysis, very few methods are built Ad-hoc for botnet C&Cs has nearly doubled, rising from 9, 500 to 17, 1308
1255 detecting these two types of Cyber-attacks. Phishing and 602 in 2019. The figure 9 shows the increasing attention of 1309
1256 Spam are the main threats affecting anyone using a technolog- researchers in this area. 1310
1257 ical device, so using AI for prevention and detection is nec-
1258 essary. AI that simultaneously conveys assurance about the 10 https://www.spamhaus.org/news/article/793/spamhaus-botnet-threat-
1259 decision made and provides awareness is required to prevent report-2019

VOLUME 10, 2022 93589

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

3) EXPLAINABLE ARTIFICIAL INTELLIGENCE IN BOT (Net) 1346

DETECTION 1347

BotStop [192] is a Packet-based Botnet detection system 1348

that examines incoming and outgoing network traffic in an 1349

IoT device to prevent infections from Botnets. The proposed 1350

system is founded on Explainable ML algorithms thanks to 1351

SHAP use with features extracted from network packets. 1352

Once an attack is detected, the source is blocked. Always 1353

SHAP is used in [193] to determine the relevant traffic fea- 1354

tures in a framework to detect traffic generated by a Bot and 1355

then determine the type of Bots using a Convolutional Neural 1356

Network. 1357

Suryotrisongko et al. [194] propose the XAI and OSINT 1358

combination for Cyber Threat Intelligence Sharing in pre- 1359

venting Botnet DGA. This research applied four existing XAI 1360

techniques: Anchors, SHAP, Counterfactual Explanation and 1361

FIGURE 9. BotNet Detection Publications from 2005 to 2021, retrieved LIME. This latter is also used in [195] and [196] where the 1362
from Scopus using as search key [TITLE-ABS-KEY (botnet AND detection)]. final goal is the detection in IoT Networks. 1363

BD-GNNExplainer [197] is a Botnet Detection Model 1364

based on Graph Neural Network. The explanation is 1365

1311 AI, applied with Explainable methods, is certainly among attributable to subgraph decomposition theory [198], where 1366
1312 the best methods to counter this phenomenon in which a huge it is feasible to determine whether the learned model is inter- 1367
1313 number of resources have to be vanquished. pretable by identifying the subgraph with the most significant 1368

influence on prediction and judging whether the subgraph is 1369

1314 2) ARTIFICIAL INTELLIGENCE IN BOT (Net) DETECTION faithful to general knowledge. 1370

1315 This section quickly reviews the newest and most cited meth- Reference [199], [200], [201], three explainable studies 1371

1316 ods in BotNet Detection. For Bot Detection, refer to the focused on DGA-based botnet detection, are also worth 1372

1317 comprehensive survey by Cresci et al. [182]. mentioning, as is [202], in which the authors created a 1373

1318 Fast-flux hunter (FFH) [184] is a framework that can Gradient-based Explainable Variational Autoencoder for Net- 1374

1319 improve the performance level in detecting and predicting work Anomaly Detection utilizing a BotNet dataset as a test. 1375

1320 unknown and Zero-day fast-flux Botnets. FFH distinguishes Bot-Detective [203] is an explainable Twitter bot detection 1376

1321 the fast-flux Botnets domain from legitimate domains in service with crowdsourcing functionalities that uses LIME. 1377

1322 an online mode based on new rules, features, or classes to LIME is also used in JITBot [204], An Explainable Just-In- 1378

1323 enhance learning using the EFuNN algorithm. Time Defect Prediction Bot, and in [205], a bot-type classifi- 1379

1324 TS-ASRCAPS [185] is a framework based on double- cation schema. 1380

1325 stream networks, which uses multimodal information to SHAP and LIME are used in [206] for game BOT detec- 1381

1326 reflect the characteristics of Domain Generation Algorithms, tion, while in [207], the authors used a Decision Tree model, 1382

1327 and an attention-sliced recurrent neural network to automati- Explainable by definition, for automatic detection on Twitter 1383

1328 cally mine the underlying semantics. with a particular case study on posts about COVID-19. 1384

1329 The authors of [186] propose a memory-efficient DL

1330 method, named LS-DRNN, for Botnets attack detection
1331 in IoT networks. S-DRNN method employs SMOTE and 4) CONSIDERATIONS ABOUT BOT (Net) DETECTION 1385

1332 DRNN algorithms only. However, LS-DRNN combines Long AND XAI 1386

1333 Short-Term Memory Autoencoder (LAE), SMOTE, and As noted in the previous sections, almost all of the frame- 1387

1334 DRNN algorithms. works declared Explainable use existing methods for Post- 1388

1335 The framework proposed in [187] uses ML combined with hoc Explanation, SHAP and LIME above all. In BotNet 1389

1336 a honeynet-based detection method for predicting if an IoT Detection, the almost total focus on IoT networks and devices 1390

1337 device can be a part of a Botnet. should be especially noted, demonstrating that these occupy 1391

1338 In [188], the authors use a CNN to perceive subtle differ- a very important slice of the Net. As in the case of Spam and 1392

1339 ences in power consumption and detect Anomalies. Phishing, it is critical to alert if you have entered a BotNet 1393

1340 In [189], the authors point out one of their proposal’s main and are feeding it unknowingly, and even more important 1394

1341 cons, the framework’s non-Explainability. They emphasize to Explain what you have inferred and how you got into 1395

1342 that this is a problem with DL models and that this implies it, so that you can avoid falling into it again in the future. 1396

1343 a lack of confidence. The following section will analyze It is moving in this direction, as evidenced by the increasing 1397

1344 frameworks that try to explain why a particular classification number of publications on the subject, however, one must 1398

1345 is made. Other interesting works are [190], [191]. consider that also improving is the malicious part of the 1399

93590 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

1400 fight. That is why it is increasingly important that supporting Furthermore, the admissibility of digital/network analysis 1455

1401 human decisions is AI, which can counter a considerable part performed by XAI in court is still debatable as it would 1456

1402 of these attacks in an automated way. For there to be the necessitate a review of applicable laws (e.g., evidence law). 1457

1403 right cooperation between human and AI, Explainability of However, XAI can be used efficiently and legally in the 1458

1404 the latter is necessary to build trust in the former. future to support the digital/network forensic profession if it 1459

is not viewed as a replacement for a digital/network forensic 1460

1405 E. OTHER CYBERSECURITY TREATS examiner but rather as a reliable tool to aid in investigations. 1461

1406 The Macro Categories considered up to this point are those in ATLE2FC [216] is a model for IoT Forensics using Ensem- 1462

1407 which the greatest effort has been spent with the purpose of ble Classification with an Explainable layer consisting of 1463

1408 applying Explaining Artificial Intelligence in CyberSecurity. FPGrowth with GRU-based RNN classifier for rule estima- 1464

1409 Fraud Detection. The financial sector is one of the ones tion and severity classification. 1465

1410 most frequently targeted by cyberattacks. Frauds are frequent For media forensic investigations focusing on media foren- 1466

1411 Cyber-attacks linked to money and reputation issues in this sic object modification detection, such as DeepFake detec- 1467

1412 field. Data leaks and illegal credit losses may be the root of tion, a domain-adapted forensic data model is introduced 1468

1413 such attacks. in [217] and [218]. 1469

1414 xFraud, an Explainable fraud transaction detection frame- Cyber Physical Systems. When an adversary gains access 1470

1415 work based on Graph Neural Networks (GNN), is presented to a computer system that controls equipment in a manufac- 1471

1416 in [208]. The authors designed a Learnable Hybrid Explainer turing facility, oil pipeline, refinery, electric generating plant, 1472

1417 that leverages GNNExplainer and centrality measures to learn or other similar infrastructure, they can control the operations 1473

1418 node- and edge-level Explanations simultaneously. of that equipment to harm those assets or other property. This 1474

1419 Srinath et al. [209] present an Explainable Machine Learn- is known as a Cyber-Physical attack on critical infrastructure. 1475

1420 ing framework for identifying credit card defaulters using Cyber-Physical attacks pose a risk not only to the owners and 1476

1421 DALEX [210]. operators of those assets but also to their suppliers, clients, 1477

1422 Zero-Day Vulnerabilities. The term ‘‘Zero-day’’ refers enterprises, and people nearby the targeted asset, as well as 1478

1423 to recently identified security flaws that hackers utilize to to any individual or entity they could negatively impact. For 1479

1424 attack systems. The expression ‘‘Zero-day’’ alludes to the example, a Cyber-Physical attacker may take down cameras, 1480

1425 notion that the vendor or developer has ‘‘Zero days’’ to repair switch off the lights in a building, cause a car to wander off 1481

1426 the defect because they have just become aware of it. When the road, or make a drone land in the hands of adversaries. 1482

1427 hackers use a vulnerability before developers have a chance Wickramasinghe et al. [219] propose a Desiderata on 1483

1428 to fix it, a Zero-day assault is launched. Explainability of unsupervised approaches in Cyber-Physical 1484

1429 The authors of [211] propose a new visualization technique Systems since they generate a large amount of unlabeled 1485

1430 using similarity matrices of features depicting behaviour data. These are potential solutions for meaningfully mining 1486

1431 patterns of malware and displaying them in image form these data, maintaining and improving desired functions, and 1487

1432 for faster analysis for detection of Zero-day malware. improving the safety of these systems. 1488

1433 Kumar et al. [212] use Shapley Ensemble Boosting and Bag- An Explainable Cyber-Physical Systems based on Knowl- 1489

1434 ging Approach instead for the same goal. edge Graph is proposed in [220] for Energy Systems while 1490

1435 The authors in [213] propose a method for Zero-Day Web in [221] the authors propose a framework to build Self- 1491

1436 Attacks delivering outlier explanations. The method shows Explainable Cyber-Physical System. 1492

1437 that Explanations can be backwards transformed through Crypto-Jacking. Crypto-jacking, a new Malware that 1493

1438 n-gram encoding and dimensionality reduction. resides on a computer or mobile device and uses its resources 1494

1439 In [214], Zhou et al. define a Zero-day artificial immune to ‘‘mine’’ Cryptocurrencies, is a severe online threat. In addi- 1495

1440 system driven by XAI for intrusion detection in telecommu- tion to compromising various devices, including PCs, laptops, 1496

1441 nications. The central part of the artificial immune system is cellphones, and even network servers, Crypto-Jacking can 1497

1442 extracting strict rules for benign traffic. It uses a Decision take control of web browsers. Using Crypto-Jacking, crim- 1498

1443 Tree that is, by definition, a white-box model. inals compete with sophisticated Crypto mining operations 1499

1444 Digital Forensics. Digital Forensics or Computer Foren- without the high overhead costs by stealing computational 1500

1445 sics finds its place in Forensic Science or Criminalistics. It is, power from victims’ devices. 1501

1446 therefore, that branch of Forensic science that deals with It is a threat comparable to BotNets, where unknowingly 1502

1447 investigating the contents of digital devices, during investi- the user feeds activities with malicious purposes through their 1503

1448 gation and trial, for evidentiary purposes. The collected data device. 1504

1449 are identified, acquired, analyzed, and a technical report is There are no works that make Explainable Artificial 1505

1450 written. Intelligence methods in the detection of Cryptojacking, 1506

1451 Hall et al. [215] assert that the application of AI in dig- one that goes in this direction in the detection of Cryp- 1507

1452 ital/network forensics is still a ‘‘Black box’’ at this time, tomining is that of Karn et al. [222]. They designed and 1508

1453 requiring verification by digital/network Forensic investi- implementated an automated cryptomining pod (manage- 1509

1454 gators, and is therefore unlikely to be justified in court. ment of applications inside containers) detection in a 1510

VOLUME 10, 2022 93591

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

TABLE 2. Summary of methods.

1511 Kubernetes cluster. Explainability is provided using SHAP, better orient the reader. It is also unthinkable to include all 1517

1512 LIME, and a novel auto-encoding-based scheme for LSTM studied papers; hence only a selection of works was discussed 1518

1513 models. in this survey for synthesis and relevancy considerations, 1519

prioritizing all work that proposed XAI methods with appli- 1520

1514 VI. DISCUSSION AND CHALLENGES cation in CyberSecurity. 1521

1515 Due to the broad spectrum of XAI approaches, analyzing Table 2 summarizes the principal works of XAI for each 1522

1516 the different surveys involving these works were preferred to CyberSecurity application analyzed with a focus on the 1523

93592 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

TABLE 3. Summary of most used cyber dataset in main cyber application fields.

1524 ML/DL model, the type of explanation and a summary con- users are likely to ask. Thus, explainability can only occur 1561

1525 cerning the taxonomy presented in section II-A. through human-machine interaction. In [242], the authors 1562

1526 Table 3 presents the main datasets for each application field present an example and approach for creating a concept for 1563

1527 encountered during the survey, highlighting the use of aged an XAI-driven junior cyber analyst based on understanding 1564

1528 datasets. Methods and datasets are ordered by year for each the information needs of both humans and AI components in 1565

1529 application field. terms of the work context and workflow. This method may 1566

1530 The selection criteria were based mainly on a backward be required to design future systems that people can use, par- 1567

1531 and forward snowballing strategy that consists of using the ticularly for critical systems where human stakeholders can- 1568

1532 reference list of the selected papers and the citations to these not interact with black-box outputs from intelligent agents, 1569

1533 papers to identify additional papers [241]. The proposed as is the case in many CyberSecurity applications. Therefore, 1570

1534 review was founded on a solid foundation that included the idea and proposal are to think about and build frame- 1571

1535 the most critical areas of XAI and CyberSecurity subjects. works that have human-machine interaction at their core for 1572

1536 Because of the investigated domains’ importance and rapid CyberSecurity applications, which is vital in many cases. The 1573

1537 growth, it has been determined that non-traditional sources only way to get there is to build models understandable to 1574

1538 are also necessary to analyze since they are essential and humans. 1575

1539 impactful in the field. In the following the main challenges How to achieve Explainability. In the current state of 1576

1540 emerged after the review conducted. the art, as shown in the Table 2, the proposed methods use 1577

1541 More formalism is needed. XAI is a multidimensional post-hoc explanation in most cases. Developing models that 1578

1542 target that a single theoretical approach cannot achieve. How- provide an intrinsic explanation is a priority; an explana- 1579

1543 ever, the synergistic employment of techniques from diverse tion method developed ad-hoc for that particular type of 1580

1544 study horizons must be done in a well-integrated manner. application is necessary for a field such as CyberSecurity, 1581

1545 In other words, for the area to advance, it needs to be where one risks providing an assist to the attacker. Moreover, 1582

1546 supported by a separate research community, which, at this the problem may be precisely in terms of explanation, and 1583

1547 point of development, should primarily focus on increased the risk is to provide an untruthful output. As pointed out 1584

1548 formalism. The reference is mainly to works that apply several times in [101], LIME, one of the most widely used 1585

1549 Explainable Artificial Intelligence methods in CyberSecurity methods, assumes that the decision boundary is locally linear. 1586

1550 without specifying in what and how, at what level, with output However, when the local decision boundary is non-linear, 1587

1551 reported to whom (whether users, analysts or developers) as it is in the majority of complex networks, those expla- 1588

1552 and especially with what techniques. In the same field of nation approaches cause significant inaccuracies. In some 1589

1553 application (e.g., Malware Detection), it would be good to cases, the linear portion is severely constrained to a relatively 1590

1554 unify the work in terms of Explainability so that those in tiny region. The artificial data points beyond the linear zone 1591

1555 charge of analyzing and preventing cyber-attacks can have a are easily struck by standard sampling methods, making it 1592

1556 unified and more understandable view. hard for a linear model to estimate the decision boundary 1593

1557 Human in the loop. It is not enough to explain the near x. The challenge then is not easy, the inverse correlation 1594

1558 model; the user must comprehend it. Furthermore, even with between model opacity and performance is well known, but 1595

1559 an appropriate explanation, establishing such an understand- an effort is needed to develop increasingly high-performing 1596

1560 ing may necessitate supplementary responses to queries that but transparent models. 1597

VOLUME 10, 2022 93593

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

1598 Adversial Attacks. An in-depth investigation of how pat- [7] R. Guidotti, A. Monreale, S. Ruggieri, D. Pedreschi, F. Turini, and 1655

1599 tern explanations can provide new attack surfaces for the F. Giannotti, ‘‘Local rule-based explanations of black box decision sys- 1656
tems,’’ 2018, arXiv:1805.10820. 1657
1600 underlying systems is needed. A motivated attacker can use [8] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and 1658
1601 the information offered by the explanations to perform mem- D. Batra, ‘‘Grad-CAM: Visual explanations from deep networks via 1659

1602 bership inference and pattern mining attacks, damaging over- gradient-based localization,’’ in Proc. IEEE Int. Conf. Comput. Vis. 1660
(ICCV), Oct. 2017, pp. 618–626. 1661
1603 all system privacy. Regular adversarial attacks are predicated
[9] A. Dhurandhar, P.-Y. Chen, R. Luss, C.-C. Tu, P. Ting, K. Shanmugam, 1662
1604 on the assumption that an adversary may inject a perturbation and P. Das, ‘‘Explanations based on the missing: Towards contrastive 1663
1605 into an input sample that is undetectable to humans, and, as a explanations with pertinent negatives,’’ in Proc. Adv. Neural Inf. Process. 1664
Syst., vol. 31, 2018, pp. 1–12. 1665
1606 result, the ground-truth class of the perturbed input does not
[10] S. Morgan. (2020). Special report: Cyberwarfare in the C-suite, 1666
1607 change. The second issue is that a ML model’s projected online. Cybercrime Magazine. [Online]. Available: https://cybersecurity 1667
1608 class changes. Attackers have developed several techniques ventures.com/cybercrime-damages-6-trillion-by-2021/ 1668

1609 to exploit weaknesses in XAI-enabled CyberSecurity frame- [11] Enisa Threat Landscape 2020—List of Top 15 Threats, ENISA, Athens, 1669
Greece, 2020. 1670
1610 works. Adversary attacks circumvent authentication systems,
[12] P. Linardatos, V. Papastefanopoulos, and S. Kotsiantis, ‘‘Explainable AI: 1671
1611 such as the XAI-enabled facial authentication system, while A review of machine learning interpretability methods,’’ Entropy, vol. 23, 1672

1612 poisoning attacks were used to alter or damage training no. 1, p. 18, Dec. 2020. 1673

1613 data [243]. To combat these attacks, a solution could be to [13] A. Rawal, J. Mccoy, D. B. Rawat, B. Sadler, and R. Amant, ‘‘Recent 1674
advances in trustworthy explainable artificial intelligence: Status, chal- 1675
1614 analyze ‘‘Desiderata for adversarial attacks in different sce- lenges and perspectives,’’ IEEE Trans. Artif. Intell., no. 4, Aug. 2021, doi: 1676
1615 narios involving explainable ML models’’ presented in [244]. 10.1109/TAI.2021.3133846. 1677

[14] A. Rai, ‘‘Explainable AI: From black box to glass box,’’ J. Acad. Market- 1678
ing Sci., vol. 48, no. 1, pp. 137–141, Jan. 2020. 1679
1616 VII. CONCLUSION [15] A. Kale, T. Nguyen, F. C. Harris, Jr., C. Li, J. Zhang, and X. Ma, 1680

1617 XAI is a framework to help understand and interpret the ‘‘Provenance documentation to enable explainable and trustworthy 1681
AI: A literature review,’’ Data Intell., pp. 1–41, Feb. 2022, doi: 1682
1618 predictions of AI algorithms. CyberSecurity is an area where 10.1162/dint_a_00119. 1683
1619 AI can analyze datasets and track a wide range of security [16] A. Adadi and M. Berrada, ‘‘Peeking inside the black-box: A sur- 1684
1620 threats and malicious behaviors. The only way to address the vey on explainable artificial intelligence (XAI),’’ IEEE Access, vol. 6, 1685

1621 many CyberSecurity challenges, with an increasing number pp. 52138–52160, 2018. 1686

[17] A. Abdul, J. Vermeulen, D. Wang, B. Y. Lim, and M. Kankanhalli, 1687

1622 of attacks, is through the integration of human and AI. This ‘‘Trends and trajectories for explainable, accountable and intelligible 1688
1623 paper reviews work proposed in the past five years that seeks systems: An HCI research agenda,’’ in Proc. CHI Conf. Hum. Factors 1689

1624 to bridge human and machine through explainability. After Comput. Syst., 2018, pp. 1–18. 1690

[18] Q.-S. Zhang and S.-C. Zhu, ‘‘Visual interpretability for deep learn- 1691
1625 a careful analysis of the two ecosystems, XAI and Cyber- ing: A survey,’’ Frontiers Inf. Technol. Electron. Eng., vol. 19, no. 1, 1692
1626 Security, an analysis was conducted of the areas of Cyber- pp. 27–39, 2018. 1693

1627 Security most affected by the use of AI. What distinguishes [19] Q. Zhang, Y. N. Wu, and S.-C. Zhu, ‘‘Interpretable convolutional neural 1694

1628 this work is the exploration of how each method provides networks,’’ in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 1695
Jun. 2018, pp. 8827–8836. 1696
1629 explainability for different application areas, highlighting the [20] P. P. Angelov, E. A. Soares, R. Jiang, N. I. Arnold, and P. M. Atkinson, 1697
1630 lack of formalism and the need to move toward a standard. ‘‘Explainable artificial intelligence: An analytical review,’’ Wiley Inter- 1698

1631 The final analysis explored the most relevant problems and discipl. Rev., Data Mining Knowl. Discovery, vol. 11, no. 5, p. e1424, 1699
2021. 1700
1632 open challenges. Considerable effort is needed to ensure that [21] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, 1701
1633 ad hoc frameworks and models are built for safety and not the A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, 1702

1634 application of general models for post-hoc explanation. and F. Herrera, ‘‘Explainable artificial intelligence (XAI): Concepts, 1703
taxonomies, opportunities and challenges toward responsible AI,’’ Inf. 1704
Fusion, vol. 58, pp. 82–115, Jun. 2020. 1705

1635 REFERENCES [22] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal, 1706
‘‘Explaining explanations: An overview of interpretability of machine 1707
1636 [1] M. Taddeo, T. McCutcheon, and L. Floridi, ‘‘Trusting artificial intelli- learning,’’ in Proc. IEEE 5th Int. Conf. Data Sci. Adv. Analytics (DSAA), 1708
1637 gence in cybersecurity is a double-edged sword,’’ Nature Mach. Intell., Oct. 2018, pp. 80–89. 1709
1638 vol. 1, no. 12, pp. 557–560, Dec. 2019. [23] G. Riccardo, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and 1710
1639 [2] D. Gunning and D. Aha, ‘‘Darpa’s explainable artificial intelligence D. Pedreschi, ‘‘A survey of methods for explaining black box models,’’ 1711
1640 (XAI) program,’’ AI Mag., vol. 40, no. 2, pp. 44–58, 2019. ACM Comput. Surv., vol. 51, no. 5, pp. 1–42, 2018. 1712
1641 [3] P. J. Phillips, C. A. Hahn, P. C. Fontana, D. A. Broniatowski, and [24] M. R. Islam, M. U. Ahmed, S. Barua, and S. Begum, ‘‘A systematic review 1713
1642 M. A. Przybocki, ‘‘Four principles of explainable artificial intelligence,’’ of explainable artificial intelligence in terms of different application 1714
1643 NIST Interagency, Gaithersburg, MD, USA, Internal Rep. NISTIR-8312, domains and tasks,’’ Appl. Sci., vol. 12, no. 3, p. 1353, Jan. 2022. 1715
1644 Aug. 2020, doi: 10.6028/NIST.IR.8312. [25] E. Tjoa and C. Guan, ‘‘A survey on explainable artificial intelligence 1716
1645 [4] M. T. Ribeiro, S. Singh, and C. Guestrin, ‘‘‘Why should i trust (XAI): Toward medical XAI,’’ IEEE Trans. Neural Netw. Learn. Syst., 1717
1646 you?’ Explaining the predictions of any classifier,’’ in Proc. 22nd vol. 32, no. 11, pp. 4793–4813, Oct. 2021. 1718
1647 ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2016, [26] B. Mittelstadt, C. Russell, and S. Wachter, ‘‘Explaining explanations in 1719
1648 pp. 1135–1144. AI,’’ in Proc. Conf. Fairness, Accountability, Transparency, Jan. 2019, 1720
1649 [5] S. M. Lundberg and S.-I. Lee, ‘‘A unified approach to interpreting model pp. 279–288. 1721
1650 predictions,’’ in Proc. Adv. Neural Inf. Process. Syst., vol. 30, 2017, [27] T. Miller, ‘‘Explanation in artificial intelligence: Insights from the social 1722
1651 pp. 1–10. sciences,’’ Artif. Intell., vol. 267, pp. 1–38, Feb. 2018. 1723

1652 [6] M. T. Ribeiro, S. Singh, and C. Guestrin, ‘‘Anchors: High-precision [28] S. Mohseni, N. Zarei, and E. D. Ragan, ‘‘A multidisciplinary survey and 1724
1653 model-agnostic explanations,’’ in Proc. AAAI Conf. Artif. Intell., vol. 32, framework for design and evaluation of explainable AI systems,’’ ACM 1725
1654 no. 1, Apr. 2018, pp. 1–9. Trans. Interact. Intell. Syst., vol. 11, nos. 3–4, pp. 1–45, Dec. 2021. 1726

93594 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

1727 [29] W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu, ‘‘Defini- [52] K. Shaukat, S. Luo, V. Varadharajan, I. Hameed, S. Chen, D. Liu, and 1800
1728 tions, methods, and applications in interpretable machine learning,’’ Proc. J. Li, ‘‘Performance comparison and current challenges of using machine 1801
1729 Nat. Acad. Sci. USA, vol. 116, no. 44, pp. 22071–22080, 2019. learning techniques in cybersecurity,’’ Energies, vol. 13, no. 10, p. 2509, 1802
1730 [30] O. Loyola-Gonzalez, ‘‘Black-box vs. White-box: Understanding their May 2020. 1803
1731 advantages and weaknesses from a practical point of view,’’ IEEE Access, [53] M. Alabadi and Y. Celik, ‘‘Anomaly detection for cyber-security based 1804
1732 vol. 7, pp. 154096–154113, 2019. on convolution neural network : A survey,’’ in Proc. Int. Congr. Hum.- 1805
1733 [31] G. Vilone and L. Longo, ‘‘Explainable artificial intelligence: A systematic Comput. Interact., Optim. Robotic Appl. (HORA), Jun. 2020, pp. 1–14. 1806
1734 review,’’ 2020, arXiv:2006.00093. [54] S. Kim and K.-J. Park, ‘‘A survey on machine-learning based security 1807
1735 [32] G. Vilone and L. Longo, ‘‘Classification of explainable artificial intel- design for cyber-physical systems,’’ Appl. Sci., vol. 11, no. 12, p. 5458, 1808
1736 ligence methods through their output formats,’’ Mach. Learn. Knowl. Jun. 2021. 1809
1737 Extraction, vol. 3, no. 3, pp. 615–661, Aug. 2021. [55] D. S. Berman, A. L. Buczak, J. S. Chavis, and C. L. Corbett, ‘‘A survey 1810
1738 [33] D. V. Carvalho, M. E. Pereira, and J. S. Cardoso, ‘‘Machine learning of deep learning methods for cyber security,’’ Information, vol. 10, no. 4, 1811
1739 interpretability: A survey on methods and metrics,’’ Electronics, vol. 8, p. 122, 2019. 1812
1740 no. 8, p. 832, Jul. 2019. [56] D. Gumusbas, T. Yldrm, A. Genovese, and F. Scotti, ‘‘A comprehensive 1813
1741 [34] M. Langer, D. Oster, T. Speith, H. Hermanns, L. Kästner, E. Schmidt, survey of databases and deep learning methods for cybersecurity and 1814
1742 A. Sesing, and K. Baum, ‘‘What do we want from explainable artificial intrusion detection systems,’’ IEEE Syst. J., vol. 15, no. 2, pp. 1717–1731, 1815
1743 intelligence (XAI)?—A stakeholder perspective on XAI and a conceptual Jun. 2021. 1816
1744 model guiding interdisciplinary XAI research,’’ Artif. Intell., vol. 296, [57] O. Lifandali and N. Abghour, ‘‘Deep learning methods applied to intru- 1817
1745 Jul. 2021, Art. no. 103473. sion detection: Survey, taxonomy and challenges,’’ in Proc. Int. Conf. 1818
1746 [35] U. Bhatt, A. Xiang, S. Sharma, A. Weller, A. Taly, Y. Jia, J. Ghosh, Decis. Aid Sci. Appl. (DASA), Dec. 2021, pp. 1035–1044. 1819
1747 R. Puri, J. M. F. Moura, and P. Eckersley, ‘‘Explainable machine learning [58] J. Zhang, L. Pan, Q.-L. Han, C. Chen, S. Wen, and Y. Xiang, ‘‘Deep 1820
1748 in deployment,’’ in Proc. Conf. Fairness, Accountability, Transparency, learning based attack detection for cyber-physical system cybersecurity: 1821
1749 Jan. 2020, pp. 648–657. A survey,’’ IEEE/CAA J. Autom. Sinica, vol. 9, no. 3, pp. 377–391, 1822
1750 [36] M. Sahakyan, Z. Aung, and T. Rahwan, ‘‘Explainable artificial Mar. 2022. 1823
1751 intelligence for tabular data: A survey,’’ IEEE Access, vol. 9, [59] M. A. Ferrag, L. Maglaras, S. Moschoyiannis, and H. Janicke, ‘‘Deep 1824
1752 pp. 135392–135422, 2021. learning for cyber security intrusion detection: Approaches, datasets, 1825
1753 [37] G. Vilone and L. Longo, ‘‘Notions of explainability and evaluation and comparative study,’’ J. Inf. Secur. Appl., vol. 50, Feb. 2020, 1826
1754 approaches for explainable artificial intelligence,’’ Inf. Fusion, vol. 76, Art. no. 102419. 1827
1755 pp. 89–106, Dec. 2021.
[60] S. Gamage and J. Samarabandu, ‘‘Deep learning methods in network 1828
1756 [38] M. Z. Siddiqui, S. Yadav, and M. S. Husain, ‘‘Application of artificial intrusion detection: A survey and an objective comparison,’’ J. Netw. 1829
1757 intelligence in fighting against cyber crimes: A review,’’ Int. J. Adv. Res. Comput. Appl., vol. 169, Nov. 2020, Art. no. 102767. 1830
1758 Comput. Sci., vol. 9, no. 2, pp. 118–122, 2018.
[61] A. Basit, M. Zafar, X. Liu, A. R. Javed, Z. Jalil, and K. Kifayat, ‘‘A com- 1831
1759 [39] Z. I. Khisamova, I. R. Begishev, and E. L. Sidorenko, ‘‘Artificial intelli-
prehensive survey of AI-enabled phishing attacks detection techniques,’’ 1832
1760 gence and problems of ensuring cyber security,’’ Int. J. Cyber Criminol.,
Telecommun. Syst., vol. 76, no. 1, pp. 139–154, Jan. 2021. 1833
1761 vol. 13, no. 2, pp. 564–577, 2019.
[62] T. R. Reshmi, ‘‘Information security breaches due to ransomware 1834
1762 [40] I. A. Mohammed, ‘‘Artificial intelligence for cybersecurity: A systematic
attacks—A systematic literature review,’’ Int. J. Inf. Manage. Data 1835
1763 mapping of literature,’’ Artif. Intell., vol. 7, no. 9, pp. 1–5, 2020.
Insights, vol. 1, no. 2, Nov. 2021, Art. no. 100013. 1836
1764 [41] H. Suryotrisongko and Y. Musashi, ‘‘Review of cybersecurity research
[63] S. Hariharan, A. Velicheti, A. S. Anagha, C. Thomas, and N. Balakrish- 1837
1765 topics, taxonomy and challenges: Interdisciplinary perspective,’’ in Proc.
nan, ‘‘Explainable artificial intelligence in cybersecurity: A brief review,’’ 1838
1766 IEEE 12th Conf. Service-Oriented Comput. Appl. (SOCA), Nov. 2019,
in Proc. 4th Int. Conf. Secur. Privacy (ISEA-ISAP), Oct. 2021, pp. 1–12. 1839
1767 pp. 162–167.
1768 [42] J.-H. Li, ‘‘Cyber security meets artificial intelligence: A survey,’’ Fron- [64] G. Srivastava, R. H. Jhaveri, S. Bhattacharya, S. Pandya, 1840

1769 tiers Inf. Technol. Electron. Eng., vol. 19, no. 12, pp. 1462–1474, P. K. R. Maddikunta, G. Yenduri, J. G. Hall, M. Alazab, and 1841

1770 Dec. 2018. T. R. Gadekallu, ‘‘XAI for cybersecurity: State of the art, challenges, 1842
open issues and future directions,’’ 2022, arXiv:2206.03585. 1843
1771 [43] T. C. Truong, Q. B. Diep, and I. Zelinka, ‘‘Artificial intelligence in the
1772 cyber domain: Offense and defense,’’ Symmetry, vol. 12, no. 3, p. 410, [65] J. N. Paredes, J. Carlos, L. Teze, G. I. Simari, and M. V. Martinez, ‘‘On the 1844

1773 Mar. 2020. importance of domain-specific explanations in AI-based cybersecurity 1845

1774 [44] C. V. Dalave and T. Dalave, ‘‘A review on artificial intelligence in cyber systems (technical report),’’ 2021, arXiv:2108.02006. 1846

1775 security,’’ in Proc. 6th Int. Conf. Comput. Sci. Eng. (UBMK), 2022, [66] L. Vigano and D. Magazzeni, ‘‘Explainable security,’’ in Proc. IEEE Eur. 1847

1776 pp. 304–309. Symp. Secur. Privacy Workshops (EuroS PW), Sep. 2020, pp. 293–300. 1848

1777 [45] M. Akhtar and T. Feng, ‘‘An overview of the applications of artificial [67] V. Ravi et al., ‘‘Deep learning for cyber security applications: A compre- 1849

1778 intelligence in cybersecurity,’’ EAI Endorsed Trans. Creative Technol., hensive survey,’’ TechRxiv, 2021, doi: 10.36227/techrxiv.16748161.v1. 1850

1779 vol. 8, no. 29, Dec. 2021, Art. no. 172218. [68] Y. Liu, C. Tantithamthavorn, L. Li, and Y. Liu, ‘‘Deep learning for 1851

1780 [46] K. Shaukat, S. Luo, V. Varadharajan, I. A. Hameed, and M. Xu, ‘‘A survey Android malware defenses: A systematic literature review,’’ 2021, 1852

1781 on machine learning techniques for cyber security in the last decade,’’ arXiv:2103.05292. 1853

1782 IEEE Access, vol. 8, pp. 222310–222354, 2020. [69] Z. Ahmad, A. Shahid Khan, C. Wai Shiang, J. Abdullah, and F. Ahmad, 1854
1783 [47] I. H. Sarker, M. H. Furhad, and R. Nowrozy, ‘‘AI-driven cybersecurity: An ‘‘Network intrusion detection system: A systematic study of machine 1855
1784 overview, security intelligence modeling and research directions,’’ Social learning and deep learning approaches,’’ Trans. Emerg. Telecommun. 1856
1785 Netw. Comput. Sci., vol. 2, no. 3, pp. 1–18, May 2021. Technol., vol. 32, no. 1, p. e4150, Jan. 2021. 1857

1786 [48] I. H. Sarker, A. S. M. Kayes, S. Badsha, H. Alqahtani, [70] S. K. Biswas, ‘‘Intrusion detection using machine learning: A comparison 1858
1787 P. Watters, and A. Ng, ‘‘Cybersecurity data science: An overview study,’’ Int. J. pure Appl. Math., vol. 118, no. 19, pp. 101–114, 2018. 1859
1788 from machine learning perspective,’’ J. Big Data, vol. 7, no. 1, pp. 1–29, [71] A. Chawla, B. Lee, S. Fallon, and P. Jacob, ‘‘Host based intrusion detec- 1860
1789 Dec. 2020. tion system with combined CNN/RNN model,’’ in ECML PKDD 2018 1861
1790 [49] C. Gupta, I. Johri, K. Srinivasan, Y.-C. Hu, S. M. Qaisar, and K.-Y. Huang, Workshops. ECML PKDD 2018 (Lecture Notes in Computer Science), 1862
1791 ‘‘A systematic review on machine learning and deep learning models for vol. 11329, C. Alzate, A. Monreale, H. Assem, A. Bifet, T. S. Buda, 1863
1792 electronic information security in mobile networks,’’ Sensors, vol. 22, B. Caglayan, B. Drury, E. García-Martín, R. Gavaldà, I. Koprinska, S. 1864
1793 no. 5, p. 2017, Mar. 2022. Kramer, N. Lavesson, M. Madden, I. Molloy, M.-I. Nicolae, and M. Sinn, 1865
1794 [50] A. F. Jahwar and S. Y. Ameen, ‘‘A review on cybersecurity based on Eds. Cham, Switzerland: Springer, 2019, doi: 10.1007/978-3-030-13453- 1866
1795 machine learning and deep learning algorithms,’’ J. Soft Comput. Data 2_12. 1867
1796 Mining, vol. 2, no. 2, pp. 14–25, Oct. 2021. [72] J. Byrnes, T. Hoang, N. N. Mehta, and Y. Cheng, ‘‘A modern imple- 1868
1797 [51] Y. Xin, L. Kong, Z. Liu, Y. Chen, Y. Li, H. Zhu, M. Gao, H. Hou, and mentation of system call sequence based host-based intrusion detection 1869
1798 C. Wang, ‘‘Machine learning and deep learning methods for cybersecu- systems,’’ in Proc. 2nd IEEE Int. Conf. Trust, Privacy Secur. Intell. Syst. 1870
1799 rity,’’ IEEE Access, vol. 6, pp. 35365–35381, 2018. Appl. (TPS-ISA), Oct. 2020, pp. 218–225. 1871

VOLUME 10, 2022 93595

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

1872 [73] R. Gassais, N. Ezzati-Jivan, J. M. Fernandez, D. Aloise, and [95] E. Tcydenova, T. W. Kim, C. Lee, and J. H. Park, ‘‘Detection of adversar- 1944
1873 M. R. Dagenais, ‘‘Multi-level host-based intrusion detection system for ial attacks in ai-based intrusion detection systems using explainable AI,’’ 1945
1874 Internet of Things,’’ J. Cloud Comput., vol. 9, no. 1, pp. 1–16, Dec. 2020. Hum.-Centric Comput. Inf. Sci., vol. 11, pp. 1–14, Sep. 2021. 1946

1875 [74] E. Besharati, M. Naderan, and E. Namjoo, ‘‘LR-HIDS: Logistic regres- [96] H. Liu, C. Zhong, A. Alnusair, and S. R. Islam, ‘‘FAIXID: A framework 1947
1876 sion host-based intrusion detection system for cloud environments,’’ for enhancing AI explainability of intrusion detection results using data 1948
1877 J. Ambient Intell. Humanized Comput., vol. 10, no. 9, pp. 3669–3692, cleaning techniques,’’ J. Netw. Syst. Manage., vol. 29, no. 4, pp. 1–30, 1949
1878 Sep. 2019. Oct. 2021. 1950

1879 [75] M. Liu, Z. Xue, X. He, and J. Chen, ‘‘SCADS: A scalable approach using [97] S. Dash, O. Gunluk, and D. Wei, ‘‘Boolean decision rules via column 1951

1880 spark in cloud for host-based intrusion detection system with system generation,’’ in Proc. Adv. Neural Inf. Process. Syst., vol. 31, 2018, 1952

1881 calls,’’ 2021, arXiv:2109.11821. pp. 1–11. 1953

1882 [76] D. Park, S. Kim, H. Kwon, D. Shin, and D. Shin, ‘‘Host-based intru- [98] D. Wei, S. Dash, T. Gao, and O. Gunluk, ‘‘Generalized 1954

1883 sion detection model using Siamese network,’’ IEEE Access, vol. 9, linear rule models,’’ in Proc. Int. Conf. Mach. Learn., 2019, 1955

1884 pp. 76614–76623, 2021. pp. 6687–6696. 1956

1885 [77] N. Shone, T. N. Ngoc, V. D. Phai, and Q. Shi, ‘‘A deep learning approach [99] K. S. Gurumoorthy, A. Dhurandhar, G. Cecchi, and C. Aggarwal, 1957

1886 to network intrusion detection,’’ IEEE Trans. Emerg. Topics Comput. ‘‘Efficient data representation by selecting prototypes with importance 1958

1887 Intell., vol. 2, no. 1, pp. 41–50, Feb. 2018. weights,’’ in Proc. IEEE Int. Conf. Data Mining (ICDM), Nov. 2019, 1959
pp. 260–269. 1960
1888 [78] Y. Jia, M. Wang, and Y. Wang, ‘‘Network intrusion detection algorithm
[100] H. Li, F. Wei, and H. Hu, ‘‘Enabling dynamic network access control 1961
1889 based on deep neural network,’’ IET Inf. Secur., vol. 13, no. 1, pp. 48–53,
with anomaly-based IDS and SDN,’’ in Proc. ACM Int. Workshop Secur. 1962
1890 Jan. 2019.
Softw. Defined Netw. Netw. Function Virtualization (SDN-NFVSec), 2019, 1963
1891 [79] M. Al-Qatf, Y. Lasheng, M. Al-Habib, and K. Al-Sabahi, ‘‘Deep learning pp. 13–16. 1964
1892 approach combining sparse autoencoder with SVM for network intrusion
[101] W. Guo, D. Mu, J. Xu, P. Su, G. Wang, and X. Xing, ‘‘LEMNA: Explain- 1965
1893 detection,’’ IEEE Access, vol. 6, pp. 52843–52856, 2018.
ing deep learning based security applications,’’ in Proc. ACM SIGSAC 1966
1894 [80] M. H. Ali, B. A. D. Al Mohammed, A. Ismail, and M. F. Zolkipli, ‘‘A new Conf. Comput. Commun. Secur., Oct. 2018, pp. 364–379. 1967
1895 intrusion detection system based on fast learning network and particle [102] K. Amarasinghe and M. Manic, ‘‘Improving user trust on deep neural 1968
1896 swarm optimization,’’ IEEE Access, vol. 6, pp. 20255–20261, 2018. networks based intrusion detection systems,’’ in Proc. 44th Annu. Conf. 1969
1897 [81] Z. Wang, ‘‘Deep learning-based intrusion detection with adversaries,’’ IEEE Ind. Electron. Soc. (IECON), Oct. 2018, pp. 3262–3268. 1970
1898 IEEE Access, vol. 6, pp. 38367–38384, 2018. [103] T. Zebin, S. Rezvy, and Y. Luo, ‘‘An explainable AI-based intrusion 1971
1899 [82] B. Yan and G. Han, ‘‘Effective feature extraction via stacked sparse detection system for DNS over HTTPS (DoH) attacks,’’ IEEE Trans. Inf. 1972
1900 autoencoder to improve intrusion detection system,’’ IEEE Access, vol. 6, Forensics Security, vol. 17, pp. 2339–2349, 2022. 1973
1901 pp. 41238–41248, 2018. [104] A. Morichetta, P. Casas, and M. Mellia, ‘‘EXPLAIN-IT: Towards explain- 1974
1902 [83] K. Jiang, W. Wang, A. Wang, and H. Wu, ‘‘Network intrusion detec- able AI for unsupervised network traffic analysis,’’ in Proc. 3rd ACM 1975
1903 tion combined hybrid sampling with deep hierarchical network,’’ IEEE CoNEXT Workshop Big DAta, Mach. Learn. Artif. Intell. Data Commun. 1976
1904 Access, vol. 8, pp. 32464–32476, 2022. Netw., Dec. 2019, pp. 22–28. 1977
1905 [84] Y. Yu and N. Bian, ‘‘An intrusion detection method using few-shot [105] G. Andresini, A. Appice, F. P. Caforio, D. Malerba, and G. Vessio, 1978
1906 learning,’’ IEEE Access, vol. 8, pp. 49730–49740, 2020. ‘‘ROULETTE: A neural attention multi-output model for explainable 1979

1907 [85] Y. Yang, K. Zheng, B. Wu, Y. Yang, and X. Wang, ‘‘Network intrusion network intrusion detection,’’ Exp. Syst. Appl., vol. 201, Sep. 2022, 1980

1908 detection based on supervised adversarial variational auto-encoder with Art. no. 117144. 1981

1909 regularization,’’ IEEE Access, vol. 8, pp. 42169–42184, 2020. [106] A. A. Reyes, F. D. Vaca, G. A. Castro Aguayo, Q. Niyaz, and V. Devab- 1982

1910 [86] J. Clements, Y. Yang, A. A. Sharma, H. Hu, and Y. Lao, ‘‘Rallying haktuni, ‘‘A machine learning based two-stage Wi-Fi network intrusion 1983

1911 adversarial techniques against deep learning for network security,’’ in detection system,’’ Electronics, vol. 9, no. 10, p. 1689, Oct. 2020. 1984

1912 Proc. IEEE Symp. Ser. Comput. Intell. (SSCI), Dec. 2021, pp. 01–08. [107] S. Mane and D. Rao, ‘‘Explaining network intrusion detection system 1985

1913 [87] G. Andresini, A. Appice, N. Di Mauro, C. Loglisci, and D. Malerba, using explainable AI framework,’’ 2021, arXiv:2103.07110. 1986

1914 ‘‘Multi-channel deep feature learning for intrusion detection,’’ IEEE [108] M. Sarhan, S. Layeghy, and M. Portmann, ‘‘Evaluating standard feature 1987

1915 Access, vol. 8, pp. 53346–53359, 2020. sets towards increased generalisability and explainability of ML-based 1988
network intrusion detection,’’ 2021, arXiv:2104.07183. 1989
1916 [88] T. Dias, N. Oliveira, N. Sousa, I. Praça, and O. Sousa, ‘‘A hybrid approach
[109] N. I. Mowla, J. Rosell, and A. Vahidi, ‘‘Dynamic voting based explainable 1990
1917 for an interpretable and explainable intrusion detection system,’’ in Intel-
intrusion detection system for in-vehicle network,’’ in Proc. 24th Int. 1991
1918 ligent Systems Design and Applications. ISDA 2021 (Lecture Notes in
Conf. Adv. Commun. Technol. (ICACT), Feb. 2022, pp. 406–411. 1992
1919 Networks and Systems), vol. 418, A. Abraham, N. Gandhi, T. Hanne,
[110] M. Zolanvari, Z. Yang, K. Khan, R. Jain, and N. Meskin, ‘‘TRUST 1993
1920 T. P. Hong, T. N. Rios, and W. Ding, Eds. Cham, Switzerland: Springer,
XAI: Model-agnostic explanations for AI with a case study on IIoT 1994
1921 2022, doi: 10.1007/978-3-030-96308-8_96.
security,’’ IEEE Internet Things J., early access, Oct. 21, 2022, doi: 1995
1922 [89] M. Szczepanski, M. Choras, M. Pawlicki, and R. Kozik, ‘‘Achieving
10.1109/JIOT.2021.3122019. 1996
1923 explainability of intrusion detection system by hybrid oracle-explainer
[111] B. Mahbooba, R. Sahal, W. Alosaimi, and M. Serrano, ‘‘Trust in intrusion 1997
1924 approach,’’ in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2020,
detection systems: An investigation of performance analysis for machine 1998
1925 pp. 1–8.
learning and deep learning models,’’ Complexity, vol. 2021, pp. 1–23, 1999
1926 [90] D. L. Marino, C. S. Wickramasinghe, and M. Manic, ‘‘An adversar- Mar. 2021. 2000
1927 ial approach for explainable AI in intrusion detection systems,’’ in [112] M. Rabbani, Y. L. Wang, R. Khoshkangini, H. Jelodar, R. Zhao, and P. Hu, 2001
1928 Proc. 44th Annu. Conf. IEEE Ind. Electron. Soc. (IECON), Oct. 2018, ‘‘A hybrid machine learning approach for malicious behaviour detection 2002
1929 pp. 3237–3243. and recognition in cloud computing,’’ J. Netw. Comput. Appl., vol. 151, 2003
1930 [91] M. Wang, K. Zheng, Y. Yang, and X. Wang, ‘‘An explainable machine Feb. 2020, Art. no. 102507. 2004
1931 learning framework for intrusion detection systems,’’ IEEE Access, vol. 8, [113] D. Arivudainambi, V. K. Ka, and P. Visu, ‘‘Malware traffic classi- 2005
1932 pp. 73127–73141, 2020. fication using principal component analysis and artificial neural net- 2006
1933 [92] Y. Wang, P. Wang, Z. Wang, and M. Cao, ‘‘An explainable intrusion work for extreme surveillance,’’ Comput. Commun., vol. 147, pp. 50–57, 2007
1934 detection system,’’ in Proc. IEEE 23rd Int. Conf. High Perform. Com- Nov. 2019. 2008
1935 put. Commun., 7th Int. Conf. Data Sci. Syst., 19th Int. Conf. Smart [114] A. Namavar Jahromi, S. Hashemi, A. Dehghantanha, K.-K.-R. Choo, 2009
1936 City, 7th Int. Conf. Dependability Sensor, Cloud Big Data Syst. Appl. H. Karimipour, D. E. Newton, and R. M. Parizi, ‘‘An improved two- 2010
1937 (HPCC/DSS/SmartCity/DependSys), Dec. 2021, pp. 1657–1662. hidden-layer extreme learning machine for malware hunting,’’ Comput. 2011
1938 [93] T.-T.-H. Le, H. Kim, H. Kang, and H. Kim, ‘‘Classification and expla- Secur., vol. 89, Feb. 2020, Art. no. 101655. 2012
1939 nation for intrusion detection system based on ensemble trees and SHAP [115] M. Alaeiyan, S. Parsa, and M. Conti, ‘‘Analysis and classification 2013
1940 method,’’ Sensors, vol. 22, no. 3, p. 1154, Feb. 2022. of context-based malware behavior,’’ Comput. Commun., vol. 136, 2014
1941 [94] S. Wali and I. Khan, ‘‘Explainable AI and random forest pp. 76–90, Feb. 2019. 2015
1942 based reliable intrusion detection system,’’ TechRxiv, 2021, doi: [116] J. Stiborek, T. Pevný, and M. Rehák, ‘‘Multiple instance learning for mal- 2016
1943 10.36227/techrxiv.17169080.v1. ware classification,’’ Exp. Syst. Appl., vol. 93, pp. 346–357, Mar. 2018. 2017

93596 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

2018 [117] L. Xiaofeng, J. Fangshuo, Z. Xiao, Y. Shengwei, S. Jing, and P. Lio, [138] B. Wu, S. Chen, C. Gao, L. Fan, Y. Liu, W. Wen, and M. R. Lyu, ‘‘Why 2091
2019 ‘‘ASSCA: API sequence and statistics features combined architecture for an Android APP is classified as malware: Toward malware classifica- 2092
2020 malware detection,’’ Comput. Netw., vol. 157, pp. 99–111, Jul. 2019. tion interpretation,’’ ACM Trans. Softw. Eng. Methodol., vol. 30, no. 2, 2093
2021 [118] S. Li, Q. Zhou, R. Zhou, and Q. Lv, ‘‘Intelligent malware detection pp. 1–29, Apr. 2021. 2094
2022 based on graph convolutional network,’’ J. Supercomput., vol. 78, no. 3, [139] D. Zhu, T. Xi, P. Jing, D. Wu, Q. Xia, and Y. Zhang, ‘‘A transparent and 2095
2023 pp. 4182–4198, Feb. 2022. multimodal malware detection method for Android apps,’’ in Proc. 22nd 2096

2024 [119] Y. Fan, S. Hou, Y. Zhang, Y. Ye, and M. Abdulhayoglu, ‘‘Gotcha–Sly Int. ACM Conf. Model., Anal. Simul. Wireless Mobile Syst. (MSWIM), 2097

2025 malware!: Scorpion a Metagraph2vec based malware detection system,’’ 2019, pp. 51–60. 2098

2026 in Proc. 24th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, [140] J. Feichtner and S. Gruber, ‘‘Understanding privacy awareness in Android 2099
2027 Jul. 2018, pp. 253–262. APP descriptions using deep learning,’’ in Proc. 10th ACM Conf. Data 2100

2028 [120] F. Xiao, Z. Lin, Y. Sun, and Y. Ma, ‘‘Malware detection based on deep Appl. Secur. Privacy, 2020, pp. 203–214. 2101

2029 learning of behavior graphs,’’ Math. Problems Eng., vol. 2019, pp. 1–10, [141] G. Iadarola, F. Martinelli, F. Mercaldo, and A. Santone, ‘‘Towards 2102

2030 Feb. 2019. an interpretable deep learning model for mobile malware detec- 2103

2031 [121] A. G. Kakisim, M. Nar, and I. Sogukpinar, ‘‘Metamorphic malware tion and family identification,’’ Comput. Secur., vol. 105, Jun. 2021, 2104

2032 identification using engine-specific patterns based on co-opcode graphs,’’ Art. no. 102198. 2105

2033 Comput. Standards Interfaces, vol. 71, Aug. 2020, Art. no. 103443. [142] S. Chen, S. Bateni, S. Grandhi, X. Li, C. Liu, and W. Yang, ‘‘DENAS: 2106

2034 [122] R. U. Khan, X. Zhang, and R. Kumar, ‘‘Analysis of ResNet and Automated rule generation by knowledge extraction from neural net- 2107

2035 GoogleNet models for malware detection,’’ J. Comput. Virol. Hacking works,’’ in Proc. 28th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. 2108

2036 Techn., vol. 15, no. 1, pp. 29–37, 2019. Found. Softw. Eng., Nov. 2020, pp. 813–825. 2109

2037 [123] D. Nahmias, A. Cohen, N. Nissim, and Y. Elovici, ‘‘Deep feature trans- [143] L. Yang, W. Guo, Q. Hao, A. Ciptadi, A. Ahmadzadeh, X. Xing, and 2110

2038 fer learning for trusted and automated malware signature generation G. Wang, ‘‘CADE: Detecting and explaining concept drift samples for 2111

2039 in private cloud environments,’’ Neural Netw., vol. 124, pp. 243–257, security applications,’’ in Proc. 30th USENIX Secur. Symp. (USENIX 2112

2040 Apr. 2020. Security), 2021, pp. 2327–2344. 2113

2041 [124] Q. Le, O. Boydell, B. Mac Namee, and M. Scanlon, ‘‘Deep learning at [144] Z. Pan, J. Sheldon, and P. Mishra, ‘‘Hardware-assisted malware detection 2114

2042 the shallow end: Malware classification for non-domain experts,’’ Digit. using explainable machine learning,’’ in Proc. IEEE 38th Int. Conf. 2115

2043 Invest., vol. 26, pp. S118–S126, Jul. 2018. Comput. Design (ICCD), Oct. 2020, pp. 663–666. 2116

2044 [125] S. Huda, R. Islam, J. Abawajy, J. Yearwood, M. M. Hassan, and [145] Z. Pan, J. Sheldon, and P. Mishra, ‘‘Hardware-assisted malware detec- 2117

2045 G. Fortino, ‘‘A hybrid-multi filter-wrapper framework to identify run- tion and localization using explainable machine learning,’’ IEEE Trans. 2118

2046 time behaviour for fast malware detection,’’ Future Gener. Comput. Syst., Comput., early access, Feb. 11, 2022, doi: 10.1109/TC.2022.3150573. 2119

2047 vol. 83, pp. 193–207, Jun. 2018. [146] S. Bose, T. Barao, and X. Liu, ‘‘Explaining AI for malware detection: 2120
Analysis of mechanisms of MalConv,’’ in Proc. Int. Joint Conf. Neural 2121
2048 [126] I. Baptista, S. Shiaeles, and N. Kolokotronis, ‘‘A novel malware detec-
Netw. (IJCNN), Jul. 2020, pp. 1–8. 2122
2049 tion system based on machine learning and binary visualization,’’ in
[147] M. Al-Fawa’reh, A. Saif, M. T. Jafar, and A. Elhassan, ‘‘Malware detec- 2123
2050 Proc. IEEE Int. Conf. Commun. Workshops (ICC Workshops), May 2019,
tion by eating a whole APK,’’ in Proc. 32nd Int. Conf. for Internet 2124
2051 pp. 1–6.
Technol. Secured Trans. (ICITST), Dec. 2020, pp. 1–7. 2125
2052 [127] F. O. Catak, J. Ahmed, K. Sahinbas, and Z. H. Khand, ‘‘Data augmen-
[148] B. Hsupeng, K.-W. Lee, T.-E. Wei, and S.-H. Wang, ‘‘Explainable mal- 2126
2053 tation based malware detection using convolutional neural networks,’’
ware detection using predefined network flow,’’ in Proc. 24th Int. Conf. 2127
2054 PeerJ Comput. Sci., vol. 7, p. e346, Jan. 2021.
Adv. Commun. Technol. (ICACT), Feb. 2022, pp. 27–33. 2128
2055 [128] Q. Qian and M. Tang, ‘‘Dynamic API call sequence visualisation for
[149] W. Han, J. Xue, Y. Wang, L. Huang, Z. Kong, and L. Mao, ‘‘MalDAE: 2129
2056 malware classification,’’ IET Inf. Secur., vol. 13, no. 4, pp. 367–377,
Detecting and explaining malware based on correlation and fusion of 2130
2057 Jul. 2019.
static and dynamic characteristics,’’ Comput. Secur., vol. 83, pp. 208–233, 2131
2058 [129] M. Jain, W. Andreopoulos, and M. Stamp, ‘‘Convolutional neu- Jun. 2019. 2132
2059 ral networks and extreme learning machines for malware classifica- [150] L. Demetrio, B. Biggio, G. Lagorio, F. Roli, and A. Armando, ‘‘Explain- 2133
2060 tion,’’ J. Comput. Virol. Hacking Techn., vol. 16, no. 3, pp. 229–244, ing vulnerabilities of deep learning to adversarial malware binaries,’’ 2134
2061 Sep. 2020. 2019, arXiv:1901.03583. 2135
2062 [130] G. Bendiab, S. Shiaeles, A. Alruban, and N. Kolokotronis, ‘‘IoT malware [151] I. Rosenberg, S. Meir, J. Berrebi, I. Gordon, G. Sicard, and E. O. David, 2136
2063 network traffic classification using visual representation and deep learn- ‘‘Generating end-to-end adversarial examples for malware classifiers 2137
2064 ing,’’ in Proc. 6th IEEE Conf. Netw. Softwarization (NetSoft), Jun. 2020, using explainability,’’ in Proc. Int. Joint Conf. Neural Netw. (IJCNN), 2138
2065 pp. 444–449. Jul. 2020, pp. 1–10. 2139
2066 [131] D. Gibert, C. Mateu, J. Planes, and R. Vicens, ‘‘Using convolutional [152] G. Severi, J. Meyer, S. Coull, and A. Oprea, ‘‘Explanation-Guided 2140
2067 neural networks for classification of malware represented as images,’’ backdoor poisoning attacks against malware classifiers,’’ in Proc. 30th 2141
2068 J. Comput. Virol. Hacking Techn., vol. 15, no. 1, pp. 15–28, Mar. 2019. USENIX Secur. Symp. (USENIX Security), 2021, pp. 1487–1504. 2142
2069 [132] Y. Ye, L. Chen, S. Hou, W. Hardy, and X. Li, ‘‘DeepAM: A heterogeneous [153] W. Song, X. Li, S. Afroz, D. Garg, D. Kuznetsov, and H. Yin, ‘‘Automatic 2143
2070 deep learning framework for intelligent malware detection,’’ Knowl. Inf. generation of adversarial examples for interpreting malware classifiers,’’ 2144
2071 Syst., vol. 54, no. 2, pp. 265–285, Feb. 2018. 2020, arXiv:2003.03100. 2145
2072 [133] S. Sharma, C. R. Krishna, and S. K. Sahay, ‘‘Detection of advanced mal- [154] M. Fan, W. Wei, X. Xie, Y. Liu, X. Guan, and T. Liu, ‘‘Can we trust 2146
2073 ware by machine learning techniques,’’ in Soft Computing: Theories and your explanations? Sanity checks for interpreters in Android malware 2147
2074 Applications (Advances in Intelligent Systems and Computing), vol. 742, analysis,’’ IEEE Trans. Inf. Forensics Security, vol. 16, pp. 838–853, 2148
2075 K. Ray, T. Sharma, S. Rawat, R. Saini, and A. Bandyopadhyay, Eds. 2021. 2149
2076 Singapore: Springer, 2019, doi: 10.1007/978-981-13-0589-4_31. [155] K. L. Chiew, C. L. Tan, K. Wong, K. S. C. Yong, and W. K. Tiong, ‘‘A new 2150
2077 [134] D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon, K. Rieck, hybrid ensemble feature selection framework for machine learning-based 2151
2078 and C. Siemens, ‘‘DREBIN: Effective and explainable detection of phishing detection system,’’ Inf. Sci., vol. 484, pp. 153–166, May 2019. 2152
2079 Android malware in your pocket,’’ in Proc. NDSS, vol. 14, 2014, [156] O. K. Sahingoz, E. Buber, O. Demir, and B. Diri, ‘‘Machine learn- 2153
2080 pp. 23–26. ing based phishing detection from URLs,’’ Exp. Syst. Appl., vol. 117, 2154
2081 [135] M. Melis, D. Maiorca, B. Biggio, G. Giacinto, and F. Roli, ‘‘Explaining pp. 345–357, Mar. 2019. 2155
2082 black-box Android malware detection,’’ in Proc. 26th Eur. Signal Process. [157] S. Y. Yerima and M. K. Alzaylaee, ‘‘High accuracy phishing detection 2156
2083 Conf. (EUSIPCO), Sep. 2018, pp. 524–528. based on convolutional neural networks,’’ in Proc. 3rd Int. Conf. Comput. 2157
2084 [136] M. Kinkead, S. Millar, N. McLaughlin, and P. O’Kane, ‘‘Towards explain- Appl. Inf. Secur. (ICCAIS), Mar. 2020, pp. 1–6. 2158
2085 able CNNs for Android malware detection,’’ Proc. Comput. Sci., vol. 184, [158] P. Yi, Y. Guan, F. Zou, Y. Yao, W. Wang, and T. Zhu, ‘‘Web phishing 2159
2086 pp. 959–965, Jan. 2021. detection using a deep learning framework,’’ Wireless Commun. Mobile 2160
2087 [137] R. Kumar, Z. Xiaosong, R. U. Khan, J. Kumar, and I. Ahad, ‘‘Effective Comput., vol. 2018, pp. 1–9, Sep. 2018. 2161
2088 and explainable detection of Android malware based on machine learning [159] E. Zhu, Y. Ju, Z. Chen, F. Liu, and X. Fang, ‘‘DTOF-ANN: An artificial 2162
2089 algorithms,’’ in Proc. Int. Conf. Comput. Artif. Intell. (ICCAI), 2018, neural network phishing detection model based on decision tree and opti- 2163
2090 pp. 35–40. mal features,’’ Appl. Soft Comput., vol. 95, Oct. 2020, Art. no. 106505. 2164

VOLUME 10, 2022 93597

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

2165 [160] R. S. Rao and A. R. Pais, ‘‘Jail-phish: An improved search engine [181] A. Capillo, E. de Santis, F. Mascioli, and A. Rizzi, ‘‘Mining M-grams by 2239
2166 based phishing detection system,’’ Comput. Secur., vol. 83, pp. 246–267, a granular computing approach for text classification,’’ in Proc. 12th Int. 2240
2167 Jun. 2019. Joint Conf. Comput. Intell., 2020, pp. 350–360. 2241
2168 [161] A. El Aassal, S. Baki, A. Das, and R. M. Verma, ‘‘An in-depth benchmark- [182] S. Cresci, ‘‘A decade of social bot detection,’’ Commun. ACM, vol. 63, 2242
2169 ing and evaluation of phishing detection research for security needs,’’ no. 10, pp. 72–83, Sep. 2020. 2243
2170 IEEE Access, vol. 8, pp. 22170–22192, 2020. [183] H. Owen, J. Zarrin, and S. M. Pour, ‘‘A survey on botnets, issues, threats, 2244
2171 [162] H. Faris, H. Faris, A.-Z. Ala’M, A. A. Heidari, I. Aljarah, M. Mafarja, methods, detection and prevention,’’ J. Cybersecurity Privacy, vol. 2, 2245
2172 M. A. Hassonah, and H. Fujita, ‘‘An intelligent system for spam detec- no. 1, pp. 74–88, Feb. 2022. 2246
2173 tion and identification of the most relevant features based on evo- [184] A. Almomani, ‘‘Fast-flux hunter: A system for filtering online fast-flux 2247
2174 lutionary random weight networks,’’ Inf. Fusion, vol. 48, pp. 67–83, botnet,’’ Neural Comput. Appl., vol. 29, no. 7, pp. 483–493, Apr. 2018. 2248
2175 Aug. 2019. [185] X. Pei, S. Tian, L. Yu, H. Wang, and Y. Peng, ‘‘A two-stream network 2249
2176 [163] G. Chetty, H. Bui, and M. White, ‘‘Deep learning based spam detec- based on capsule networks and sliced recurrent neural networks for DGA 2250
2177 tion system,’’ in Proc. Int. Conf. Mach. Learn. Data Eng. (iCMLDE), botnet detection,’’ J. Netw. Syst. Manage., vol. 28, no. 4, pp. 1694–1721, 2251
2178 Dec. 2019, pp. 91–96. Oct. 2020. 2252
2179 [164] A. Barushka and P. Hajek, ‘‘Spam filtering using integrated distribution-
[186] S. I. Popoola, B. Adebisi, R. Ande, M. Hammoudeh, and A. A. Atayero, 2253
2180 based balancing approach and regularized deep neural networks,’’ Appl.
‘‘Memory-efficient deep learning for botnet attack detection in IoT net- 2254
2181 Intell., vol. 48, no. 10, pp. 3538–3556, Oct. 2018.
works,’’ Electronics, vol. 10, no. 9, p. 1104, May 2021. 2255
2182 [165] S. Douzi, F. A. AlShahwan, M. Lemoudden, and B. Ouahidi, ‘‘Hybrid
2183 email spam detection model using artificial intelligence,’’ Int. J. Mach. [187] V. A. Memos and K. E. Psannis, ‘‘AI-powered honeypots for enhanced 2256

2184 Learn. Comput., vol. 10, no. 2, pp. 316–322, Feb. 2020. IoT botnet detection,’’ in Proc. 3rd World Symp. Commun. Eng. (WSCE), 2257
Oct. 2020, pp. 64–68. 2258
2185 [166] G. Jain, M. Sharma, and B. Agarwal, ‘‘Spam detection in social media
2186 using convolutional and long short term memory neural network,’’ Ann. [188] W. Jung, H. Zhao, M. Sun, and G. Zhou, ‘‘IoT botnet detection via 2259

2187 Math. Artif. Intell., vol. 85, no. 1, pp. 21–44, Jan. 2019. power consumption modeling,’’ Smart Health, vol. 15, Mar. 2020, 2260

2188 [167] S. Magdy, Y. Abouelseoud, and M. Mikhail, ‘‘Efficient spam and phish- Art. no. 100103. 2261

2189 ing emails filtering based on deep learning,’’ Comput. Netw., vol. 206, [189] M. Mazza, S. Cresci, M. Avvenuti, W. Quattrociocchi, and M. Tesconi, 2262

2190 Apr. 2022, Art. no. 108826. ‘‘RTbust: Exploiting temporal patterns for botnet detection on Twitter,’’ 2263

2191 [168] S. Bosaeed, I. Katib, and R. Mehmood, ‘‘A fog-augmented machine learn- in Proc. 10th ACM Conf. Web Sci., Jun. 2019, pp. 183–192. 2264

2192 ing based SMS spam detection and classification system,’’ in Proc. 5th [190] C. Joshi, R. Ranjan, and V. Bharti, ‘‘A fuzzy logic based feature engi- 2265

2193 Int. Conf. Fog Mobile Edge Comput. (FMEC), Apr. 2020, pp. 325–330. neering approach for botnet detection using ANN,’’ J. King Saud Univ.- 2266

2194 [169] Y. Lin, R. Liu, D. M. Divakaran, J. Y. Ng, Q. Z. Chan, Y. Lu, Y. Si, Comput. Inf. Sci., pp. 1–11, Jul. 2021, doi: 10.1016/j.jksuci.2021.06.018. 2267

2195 F. Zhang, and J. S. Dong, ‘‘Phishpedia: A hybrid deep learning based [191] H.-T. Nguyen, Q.-D. Ngo, D.-H. Nguyen, and V.-H. Le, ‘‘PSI-rooted 2268
2196 approach to visually identify phishing webpages,’’ in Proc. 30th USENIX subgraph: A novel feature for IoT botnet detection using classifier algo- 2269
2197 Secur. Symp. (USENIX Security), 2021, pp. 3793–3810. rithms,’’ ICT Exp., vol. 6, no. 2, pp. 128–138, Jun. 2020. 2270
2198 [170] S. Mahdavifar and A. A. Ghorbani, ‘‘Dennes: Deep embedded neural [192] M. M. Alani, ‘‘BotStop : Packet-based efficient and explainable IoT 2271
2199 network expert system for detecting cyber attacks,’’ Neural Comput. botnet detection using machine learning,’’ Comput. Commun., vol. 193, 2272
2200 Appl., vol. 32, no. 18, pp. 14753–14780, 2020. pp. 53–62, Sep. 2022. 2273
2201 [171] Y. Chai, Y. Zhou, W. Li, and Y. Jiang, ‘‘An explainable multi-modal [193] P. P. Kundu, T. Truong-Huu, L. Chen, L. Zhou, and S. G. Teo, ‘‘Detec- 2274
2202 hierarchical attention model for developing phishing threat intelligence,’’ tion and classification of botnet traffic using deep learning with model 2275
2203 IEEE Trans. Dependable Secure Comput., vol. 19, no. 2, pp. 790–803, explanation,’’ IEEE Trans. Dependable Secure Comput., early access, 2276
2204 Apr. 2022. Jun. 15, 2022, doi: 10.1109/TDSC.2022.3183361. 2277
2205 [172] K. Kluge and R. Eckhardt, ‘‘Explaining the suspicion: Design of an [194] H. Suryotrisongko, Y. Musashi, A. Tsuneda, and K. Sugitani, ‘‘Robust 2278
2206 XAI-based user-focused anti-phishing measure,’’ in Innovation Through botnet DGA detection: Blending XAI and OSINT for cyber threat intel- 2279
2207 Information Systems. WI 2021 (Lecture Notes in Information Systems ligence sharing,’’ IEEE Access, vol. 10, pp. 34613–34624, 2022. 2280
2208 and Organisation), vol. 47, F. Ahlemann, R. Schütte, and S. Stieglitz, [195] N. Ben Rabah, B. Le Grand, and M. K. Pinheiro, ‘‘IoT botnet detection 2281
2209 Eds. Cham, Switzerland: Springer, 2021, doi: 10.1007/978-3-030-86797- using black-box machine learning models: The trade-off between per- 2282
2210 3_17. formance and interpretability,’’ in Proc. IEEE 30th Int. Conf. Enabling 2283
2211 [173] P. R. G. Hernandes, C. P. Floret, K. F. C. De Almeida, V. C. Da Silva, Technol., Infrastruct. Collaborative Enterprises (WETICE), Oct. 2021, 2284
2212 J. P. Papa, and K. A. P. Da Costa, ‘‘Phishing detection using URL- pp. 101–106. 2285
2213 based XAI techniques,’’ in Proc. IEEE Symp. Ser. Comput. Intell. (SSCI),
[196] A. Guerra-Manzanares, S. Nomm, and H. Bahsi, ‘‘Towards the integration 2286
2214 Dec. 2021, pp. 01–06.
of a post-hoc interpretation step into the machine learning workflow for 2287
2215 [174] H. Nori, S. Jenkins, P. Koch, and R. Caruana, ‘‘InterpretML:
IoT botnet detection,’’ in Proc. 18th IEEE Int. Conf. Mach. Learn. Appl. 2288
2216 A unified framework for machine learning interpretability,’’ 2019,
(ICMLA), Dec. 2019, pp. 1162–1169. 2289
2217 arXiv:1909.09223.
[197] X. Zhu, Y. Zhang, Z. Zhang, D. Guo, Q. Li, and Z. Li, ‘‘Interpretability 2290
2218 [175] M. Stites, M. Nyre-Yu, B. Moss, C. Smutz, and M. Smith, ‘‘Sage advice?
evaluation of botnet detection model based on graph neural network,’’ in 2291
2219 The impacts of explanations for machine learning models on human
Proc. IEEE Conf. Comput. Commun. Workshops (INFOCOM WKSHPS), 2292
2220 decision-making in spam detection,’’ in Proc. Int. Conf. Hum.-Comput.
May 2022, pp. 1–6. 2293
2221 Interact., Jul. 2021, pp. 269–284.
2222 [176] D. Zhang, Q. Zhang, G. Zhang, and J. Lu, ‘‘FreshGraph: A spam-aware [198] D. Luo, W. Cheng, D. Xu, W. Yu, B. Zong, H. Chen, and X. Zhang, 2294

2223 recommender system for cold start problem,’’ in Proc. IEEE 14th Int. ‘‘Parameterized explainer for graph neural network,’’ in Proc. Adv. Neural 2295

2224 Conf. Intell. Syst. Knowl. Eng. (ISKE), Nov. 2019, pp. 1211–1218. Inf. Process. Syst., vol. 33, 2020, pp. 19620–19631. 2296

2225 [177] J. Gu, J. Na, J. Park, and H. Kim, ‘‘Predicting success of outbound [199] M. Zago, M. G. Pérez, and G. M. Pérez, ‘‘Early DGA-based botnet 2297

2226 telemarketing in insurance policy loans using an explainable multiple- identification: Pushing detection to the edges,’’ Cluster Comput., vol. 24, 2298

2227 filter convolutional neural network,’’ Appl. Sci., vol. 11, no. 15, p. 7147, no. 3, pp. 1695–1710, Sep. 2021. 2299

2228 Aug. 2021. [200] A. Drichel, N. Faerber, and U. Meyer, ‘‘First step towards EXPLAINable 2300

2229 [178] T. Le, S. Wang, and D. Lee, ‘‘GRACE: Generating concise and informa- DGA multiclass classification,’’ in Proc. 16th Int. Conf. Availability, Rel. 2301

2230 tive contrastive sample to explain neural network model’s prediction,’’ Secur., Aug. 2021, pp. 1–13. 2302

2231 in Proc. 26th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, [201] F. Becker, A. Drichel, C. Müller, and T. Ertl, ‘‘Interpretable visualiza- 2303
2232 Aug. 2020, pp. 238–248. tions of deep neural networks for domain generation algorithm detec- 2304
2233 [179] A. Björklund, J. Mäkelä, and K. Puolamäki, ‘‘SLISEMAP: Super- tion,’’ in Proc. IEEE Symp. Visualizat. Cyber Secur. (VizSec), Oct. 2020, 2305
2234 vised dimensionality reduction through local explanations,’’ 2022, pp. 25–29. 2306
2235 arXiv:2201.04455. [202] Q. P. Nguyen, K. W. Lim, D. M. Divakaran, K. H. Low, and M. C. Chan, 2307
2236 [180] A. Occhipinti, L. Rogers, and C. Angione, ‘‘A pipeline and comparative ‘‘GEE: A gradient-based explainable variational autoencoder for network 2308
2237 study of 12 machine learning models for text classification,’’ Exp. Syst. anomaly detection,’’ in Proc. IEEE Conf. Commun. Netw. Secur. (CNS), 2309
2238 Appl., vol. 201, Sep. 2022, Art. no. 117193. Jun. 2019, pp. 91–99. 2310

93598 VOLUME 10, 2022

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

2311 [203] M. Kouvela, I. Dimitriadis, and A. Vakali, ‘‘Bot-detective: An explainable [225] G. Creech and J. Hu, ‘‘A semantic approach to host-based intrusion detec- 2382
2312 Twitter bot detection service with crowdsourcing functionalities,’’ in tion systems using contiguousand discontiguous system call patterns,’’ 2383
2313 Proc. 12th Int. Conf. Manage. Digit. EcoSystems, Nov. 2020, pp. 55–63. IEEE Trans. Comput., vol. 63, no. 4, pp. 807–819, Apr. 2014. 2384
2314 [204] C. Khanan, W. Luewichana, K. Pruktharathikoon, J. Jiarpakdee, [226] G. Creech, ‘‘Developing a high-accuracy cross platform host-based intru- 2385
2315 C. Tantithamthavorn, M. Choetkiertikul, C. Ragkhitwetsagul, and sion detection system capable of reliably detecting zero-day attacks,’’ 2386
2316 T. Sunetnanta, ‘‘JITBOT: An explainable just-in-time defect predic- Ph.D. dissertation, School Eng. Inf. Technol., Univ. College, Univ. New 2387
2317 tion bot,’’ in Proc. 35th IEEE/ACM Int. Conf. Automated Softw. Eng., South Wales, Austral. Defence Force Acad., Sydney, NSW, Australia, 2388
2318 Sep. 2020, pp. 1336–1339. 2014. [Online]. Available: http://handle.unsw.edu.au/1959.4/53218 2389
2319 [205] I. Dimitriadis, K. Georgiou, and A. Vakali, ‘‘Social botomics: A system- [227] N. Moustafa and J. Slay, ‘‘UNSW-NB15: A comprehensive data set for 2390
2320 atic ensemble ML approach for explainable and multi-class bot detec- network intrusion detection systems (UNSW-NB15 network data set),’’ 2391
2321 tion,’’ Appl. Sci., vol. 11, no. 21, p. 9857, Oct. 2021. in Proc. Mil. Commun. Inf. Syst. Conf. (MilCIS), Nov. 2015, pp. 1–6. 2392
2322 [206] E. Park, K. Ho Park, and H. Kang Kim, ‘‘Understand watchdogs: Discover [228] C. Kolias, G. Kambourakis, A. Stavrou, and S. Gritzalis, ‘‘Intrusion 2393
2323 how game bot get discovered,’’ 2020, arXiv:2011.13374. detection in 802.11 networks: Empirical evaluation of threats and a public 2394
2324 [207] D. B. Lira, F. Xavier, and L. A. Digiampietri, ‘‘Combining clustering dataset,’’ IEEE Commun. Surveys Tuts., vol. 18, no. 1, pp. 184–208, 2395
2325 and classification algorithms for automatic bot detection: A case study 1st. Quart., 2016. 2396
2326 on posts about COVID-19,’’ in Proc. 17th Brazilian Symp. Inf. Syst., [229] R. Panigrahi and S. Borah, ‘‘A detailed analysis of CICIDS2017 dataset 2397
2327 Jun. 2021, pp. 1–7. for designing intrusion detection systems,’’ Int. J. Eng. Technol., vol. 7, 2398
2328 [208] S. X. Rao, S. Zhang, Z. Han, Z. Zhang, W. Min, Z. Chen, Y. Shan, Y. Zhao, pp. 479–482, Dec. 2018. 2399
2329 and C. Zhang, ‘‘xFraud: Explainable fraud transaction detection,’’ Proc. [230] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, ‘‘Toward generating 2400
2330 VLDB Endowment, no. 3, pp. 427–436, Nov. 2021. a new intrusion detection dataset and intrusion traffic characterization,’’ 2401
2331 [209] T. Srinath and H. Gururaja, ‘‘Explainable machine learning in identi- in Proc. 4th Int. Conf. Inf. Syst. Secur. Privacy, vol. 1, Jan. 2018, 2402
2332 fying credit card defaulters,’’ Global Transitions Proc., vol. 3, no. 1, pp. 108–116. 2403
2333 pp. 119–126, Jun. 2022. [231] T. M. Kebede, O. Djaneye-Boundjou, B. N. Narayanan, A. Ralescu, and 2404
2334 [210] P. Biecek, ‘‘DALEX: Explainers for complex predictive models in R,’’ D. Kapp, ‘‘Classification of malware programs using autoencoders based 2405
2335 J. Mach. Learn. Res., vol. 19, no. 1, pp. 3245–3249, 2018. deep learning architecture and its application to the Microsoft malware 2406
2336 [211] S. Venkatraman and M. Alazab, ‘‘Use of data visualisation for zero- classification challenge (BIG 2015) dataset,’’ in Proc. IEEE Nat. Aerosp. 2407
2337 day malware detection,’’ Secur. Commun. Netw., vol. 2018, pp. 1–13, Electron. Conf. (NAECON), Jun. 2017, pp. 70–75. 2408
2338 Dec. 2018. [232] H. S. Anderson and P. Roth, ‘‘EMBER: An open dataset for training static 2409
2339 [212] R. Kumar and G. Subbiah, ‘‘Zero-day malware detection and effec- PE malware machine learning models,’’ 2018, arXiv:1804.04637. 2410
2340 tive malware analysis using Shapley ensemble boosting and bagging [233] G. Severi, T. Leek, and B. Dolan-Gavitt, ‘‘MALREC: Compact full- 2411
2341 approach,’’ Sensors, vol. 22, no. 7, p. 2798, Apr. 2022. trace malware recording for retrospective deep analysis,’’ in Detection 2412
2342 [213] J. H. Sejr, A. Zimek, and P. Schneider-Kamp, ‘‘Explainable detection of of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2018 2413
2343 zero day web attacks,’’ in Proc. 3rd Int. Conf. Data Intell. Secur. (ICDIS), (Lecture Notes in Computer Science), vol. 10885, C. Giuffrida, S. Bardin, 2414
2344 Jun. 2020, pp. 71–78. and G. Blanc, Eds. Cham, Switzerland: Springer, 2018, doi: 10.1007/978- 2415
2345 [214] Q. Zhou, R. Li, L. Xu, A. Nallanathan, J. Yang, and A. Fu, 3-319-93411-2_1. 2416
2346 ‘‘Towards explainable meta-learning for DDoS detection,’’ 2022, [234] R. Ronen, M. Radu, C. Feuerstein, E. Yom-Tov, and M. Ahmadi, 2417
2347 arXiv:2204.02255. ‘‘Microsoft malware classification challenge,’’ 2018, arXiv:1802.10135. 2418
2348 [215] S. W. Hall, A. Sakzad, and K. R. Choo, ‘‘Explainable artificial intelli- [235] L. Taheri, A. F. A. Kadir, and A. H. Lashkari, ‘‘Extensible Android mal- 2419
2349 gence for digital forensics,’’ WIREs Forensic Sci., vol. 4, no. 2, Mar. 2022. ware detection and family classification using network-flows and API- 2420
2350 [216] Y. S. Pethe and P. R. Dandekar, ‘‘ATLE2FC: Design of an augmented calls,’’ in Proc. Int. Carnahan Conf. Secur. Technol. (ICCST), Oct. 2019, 2421
2351 transfer learning model for explainable IoT forensics using ensemble pp. 1–8. 2422
2352 classification,’’ in Proc. Int. Conf. Appl. Artif. Intell. Comput. (ICAAIC), [236] G. Sakkis, I. Androutsopoulos, G. Paliouras, V. Karkaletsis, 2423
2353 May 2022, pp. 131–137. C. D. Spyropoulos, and P. Stamatopoulos, ‘‘A memory-based approach 2424
2354 [217] C. Kraetzer, D. Siegel, S. Seidlitz, and J. Dittmann, ‘‘Process-driven to anti-spam filtering for mailing lists,’’ Inf. Retr., vol. 6, no. 1, pp. 49–73, 2425
2355 modelling of media forensic investigations-considerations on the example 2003. 2426
2356 of DeepFake detection,’’ Sensors, vol. 22, no. 9, p. 3137, Apr. 2022. [237] B. Klimt and Y. Yang, ‘‘The enron corpus: A new dataset for email 2427
2357 [218] S. Dennis, K. Christian, S. Stefan, and D. Jana, ‘‘Forensic data model for classification research,’’ in Machine Learning: ECML 2004 (Lecture 2428
2358 artificial intelligence based media forensics—Illustrated on the example Notes in Computer Science), vol. 3201, J. F. Boulicaut, F. Esposito, F. 2429
2359 of DeepFake detection,’’ Electron. Imag., vol. 34, pp. 1–6, Jan. 2022. Giannotti, and D. Pedreschi, Eds. Berlin, Germany: Springer, 2004, doi: 2430
2360 [219] C. S. Wickramasinghe, K. Amarasinghe, D. L. Marino, C. Rieger, 10.1007/978-3-540-30115-8_22. 2431
2361 and M. Manic, ‘‘Explainable unsupervised machine learning for cyber- [238] R. Shams and R. E. Mercer, ‘‘Classifying spam emails using text and read- 2432
2362 physical systems,’’ IEEE Access, vol. 9, pp. 131824–131843, 2021. ability features,’’ in Proc. IEEE 13th Int. Conf. Data Mining, Dec. 2013, 2433
2363 [220] P. R. Aryan, F. J. Ekaputra, M. Sabou, D. Hauer, R. Mosshammer, pp. 657–666. 2434
2364 A. Einfalt, T. Miksa, and A. Rauber, ‘‘Explainable cyber-physical energy [239] D. Zhao, I. Traore, B. Sayed, W. Lu, S. Saad, A. Ghorbani, and D. Garant, 2435
2365 systems based on knowledge graph,’’ in Proc. 9th Workshop Model. ‘‘Botnet detection based on traffic behavior analysis and flow intervals,’’ 2436
2366 Simul. Cyber-Phys. Energy Syst., May 2021, pp. 1–6. Comput. Secur., vol. 39, pp. 2–16, Nov. 2013. 2437
2367 [221] M. Blumreiter, J. Greenyer, F. J. C. Garcia, V. Klos, [240] M. Zago, M. G. Pérez, and G. M. Pérez, ‘‘UMUDGA: A dataset for 2438
2368 M. Schwammberger, C. Sommer, A. Vogelsang, and A. Wortmann, profiling algorithmically generated domain names in botnet detection,’’ 2439
2369 ‘‘Towards self-explainable cyber-physical systems,’’ in Proc. ACM/IEEE Data Brief, vol. 30, Jun. 2020, Art. no. 105400. 2440
2370 22nd Int. Conf. Model Driven Eng. Lang. Syst. Companion (MODELS-C), [241] C. Wohlin, ‘‘Guidelines for snowballing in systematic literature studies 2441
2371 Sep. 2019, pp. 543–548. and a replication in software engineering,’’ in Proc. 18th Int. Conf. Eval. 2442
2372 [222] R. R. Karn, P. Kudva, H. Huang, S. Suneja, and I. M. Elfadel, ‘‘Crypto- Assessment Softw. Eng. (EASE), 2014, pp. 1–10. 2443
2373 mining detection in container clouds using system calls and explainable [242] E. Holder and N. Wang, ‘‘Explainable artificial intelligence (XAI) inter- 2444
2374 machine learning,’’ IEEE Trans. Parallel Distrib. Syst., vol. 32, no. 3, actively working with humans as a junior cyber analyst,’’ Hum.-Intell. 2445
2375 pp. 674–691, Mar. 2021. Syst. Integr., vol. 3, no. 2, pp. 139–153, Jun. 2021. 2446
2376 [223] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, ‘‘A detailed analysis [243] A. Kuppa and N.-A. Le-Khac, ‘‘Adversarial XAI methods in cyberse- 2447
2377 of the KDD CUP 99 data set,’’ in Proc. IEEE Symp. Comput. Intell. Secur. curity,’’ IEEE Trans. Inf. Forensics Security, vol. 16, pp. 4924–4938, 2448
2378 Defense Appl., Jul. 2009, pp. 1–6. 2021. 2449
2379 [224] L. Dhanabal and S. P. Shantharajah, ‘‘A study on NSL-KDD dataset for [244] J. Vadillo, R. Santana, and J. A. Lozano, ‘‘When and how to fool 2450
2380 intrusion detection system based on classification algorithms,’’ Int. J. Adv. explainable models (and humans) with adversarial examples,’’ 2021, 2451
2381 Res. Comput. Commun. Eng., vol. 4, no. 6, pp. 446–452, 2015. arXiv:2107.01943. 2452

VOLUME 10, 2022 93599

 N. Capuano et al.: Explainable Artificial Intelligence in CyberSecurity: A Survey

2453 NICOLA CAPUANO received the degree in com- VINCENZO LOIA (Senior Member, IEEE) 2481
2454 puter science and the Ph.D. degree in computer sci- received the degree in computer science from the 2482
2455 ence and information engineering. He is currently University of Salerno, Italy, in 1985, and the Ph.D. 2483
2456 an Assistant Professor at the School of Engineer- degree in computer science from the Université 2484
2457 ing, University of Basilicata, Italy. He is the author Pierre & Marie Curie Paris VI, France, in 1989. 2485
2458 of about 120 publications in scientific journals, He is currently a Computer Science Full Professor 2486
2459 conference proceedings, and books. His research at the University of Salerno, where he worked 2487
2460 interests include computational intelligence, AI in as a Researcher, from 1989 to 2000, and as an 2488
2461 education, knowledge-based systems, and cogni- Associate Professor, from 2000 to 2004. He is 2489
2462 tive robotics. He is an Executive Committee Mem- the Co-Editor-in-Chief of Soft Computing and 2490
2463 ber of the Learning Ideas Conference, as well as a scientific referee and a the Editor-in-Chief of Journal of Ambient Intelligence and Humanized 2491
2464 member of the editorial board for several other international journals and Computing. He serves as an editor for 14 other international journals. 2492
2465 conferences. He is an Associate Editor of the Journal of Ambient Intelligence
2466 and Humanized Computing and Frontiers in Artificial Intelligence.

2467 GIUSEPPE FENZA (Member, IEEE) received the CLAUDIO STANZIONE (Member, IEEE) recei- 2493
2468 degree and Ph.D. degrees in computer sciences ved the bachelor’s degree in economics and 2494
2469 from the University of Salerno, Italy, in 2004 and business management and the master’s degree in 2495
2470 2009, respectively. He is currently an Associate economics from the University of Salerno, Italy, in 2496
2471 Professor of computer science at the Univer- 2019 and 2021, respectively. He is currently pur- 2497
2472 sity of Salerno. The research activity concerns suing the Ph.D. degree in innovation sciences for 2498
2473 computational intelligence methods to support defence and security–digital transformation and 2499
2474 semantic-enabled solutions and decision-making. cybersecurity with the Center for Higher Defence 2500
2475 He has over 60 publications in fuzzy decision mak- Studies (CASD). His research interests include 2501
2476 ing, knowledge extraction and management, situa- explainable artificial intelligence, with a view in 2502
2477 tion and context awareness, semantic information retrieval, service oriented cyber security applications to analyze the existing methods and literature 2503
2478 architecture, and ontology learning. More recently, he worked in automating in order to achieve a greater transparency in military and cyber security 2504
2479 open source intelligence and big data analytics for counterfeiting extremism fields. 2505
2480 and supporting information disorder awareness. 2506

93600 VOLUME 10, 2022