Survey On Explainable AI - From Approaches, Limitations and Applications Aspects

Human-Centric Intelligent Systems (2023) 3:161–188
https://doi.org/10.1007/s44230-023-00038-y
REVIEW ARTICLE
Survey on Explainable AI: From Approaches, Limitations

and Applications Aspects
Wenli Yang1 · Yuchen Wei1 · Hanyu Wei1 · Yanyu Chen1 · Guan Huang1 · Xiang Li1 · Renjie Li1 · Naimeng Yao1 ·
Xinyi Wang1 · Xiaotong Gu1 · Muhammad Bilal Amin1 · Byeong Kang1
Received: 16 March 2023 / Accepted: 24 July 2023 / Published online: 10 August 2023
© The Author(s) 2023
Abstract
In recent years, artificial intelligence (AI) technology has been used in most if not all domains and has greatly benefited our
lives. While AI can accurately extract critical features and valuable information from large amounts of data to help people
complete tasks faster, there are growing concerns about the non-transparency of AI in the decision-making process. The
emergence of explainable AI (XAI) has allowed humans to better understand and control AI systems, which is motivated
to provide transparent explanations for the decisions made by AI. This article aims to present a comprehensive overview
of recent research on XAI approaches from three well-defined taxonomies. We offer an in-depth analysis and summary of
the status and prospects of XAI applications in several key areas where reliable explanations are urgently needed to avoid
mistakes in decision-making. We conclude by discussing XAI’s limitations and future research directions.
Keywords Explainable AI · Machine learning · Human-centered · Survey
Abbreviations KG Knowledge graph

AI Artificial intelligence LO Logic-oriented
XAI Explainable artificial intelligence CT Computed tomography
GDPR General data protection regulation MRI Magnetic resonance imaging
SO Source-oriented US Ultrasound
RO Representation-oriented Grad-CAM Gradient-weighted class activation mapping
CNN Convolutional neural network LIME Locally-interpretable model-agnostic
NNKX Neural network knowledge extraction explanations
REFNE Rule extraction from neural network SHAP Shapley additive explanations
ensemble CMGE Counterfactual multi-granularity graph sup-
ERE Electric rule extraction porting fact extraction
* Yuchen Wei Naimeng Yao

[email protected] [email protected]
Wenli Yang Xinyi Wang
Hanyu Wei Xiaotong Gu
Yanyu Chen Muhammad Bilal Amin
Guan Huang Byeong Kang
Xiang Li 1
School of ICT, University of Tasmania, Hobart, TAS 7005,
[email protected]
Australia
Renjie Li
[email protected]
13
Vol.:(0123456789)
162 Human-Centric Intelligent Systems (2023) 3:161–188
EHR Electronic health records engineers in different domains, they need to access and opti-
LEMNA Local explanation method using nonlinear mise explanations of the decision process for the deployment
approximation of AI systems, especially in real-world environments.
In a more detailed manner, XAI should consider differ-
ent cultural and contextual factors. For example, in different
1 Introduction contexts and cultural backgrounds, XAI may need to provide
different interpretations for the same objects and phenomena.
Deep learning has been contributing to artificial intel- To address this, scholars have proposed the Contextual Util-
ligence (AI) systems to speed up and improve numerous ity Theory [9], which explains the final decision outcome by
tasks, including decision-making, predictions, identifying assessing the importance and influence of different factors.
anomalies and patterns, and even recommendations and so Additionally, tailoring explanations based on user expertise
on. Although the accuracy of deep learning models has dra- is another crucial aspect of designing effective explainable
matically improved during the last decade, this improved artificial intelligence (XAI) systems. By considering the
accuracy has often been achieved through increased model varying levels of technical knowledge and expertise among
complexity, which may induce common sense mistakes in users, XAI systems can provide explanations that are better
practice without providing any reasons for the mistakes, suited to their individual needs. For example, in healthcare,
making it impossible to fully trust its decisions. It’s also patients often have varying levels of medical knowledge and
challenging to achieve targeted model improvement and technical understanding. When presenting AI-driven diag-
optimisation [1]. Without reliable explanations that accu- noses or treatment recommendations to patients, explana-
rately represent the current AI system processes, humans tions should be tailored to their level of health literacy. For
still consider AI untrustworthy due to a variety of dynam- patients with limited medical expertise, explanations should
ics and uncertainties [2] when deploying AI applications in use plain language and visual aids to help them comprehend
real-world environments. This motivates the inherent need the reasoning behind the AI-generated recommendations.
and expectation from human users that AI systems should On the other hand, for healthcare professionals who have a
be explainable to help confirm decisions. deeper understanding of medical concepts, explanations can
Explainable AI (explainable artificial intelligence (XAI)) delve into more technical details and provide insights into
is often considered a set of processes and methods that are the model’s decision-making process [10].
used to describe deep learning models, by characterizing From a long-term perspective, more focus should be done
model accuracy, transparency, and outcomes in AI systems on usability and maintainability, which requires improved
[3]. XAI methods aim to provide human-readable explana- personalisation, evolution with time, and data management
tions to help users comprehend and trust the outputs created [11]. These aspects are essential for ensuring the continued
by deep learning algorithms. Additionally, some regula- effectiveness and relevance of XAI approaches. One area
tions such as European General Data Protection Regulation that requires attention is improved personalization, where
(general data protection regulation (GDPR))[4] have been XAI systems can be tailored to individual users or specific
introduced to drive further XAI research, demanding the application domains [12]. Also, as AI models and data
important ethics [5], justifications [6], trust [7] and bias [8] evolve over time, XAI systems need to adapt and evolve
to explore reliable XAI solutions. as well. Data management is another critical aspect for the
The need for XAI is multi-factorial and depends on the long-term usability and maintainability of XAI systems. As
concerned people (Table 1), whether they are end users, AI data volumes increase and data distributions change, XAI
developers or product engineers. End-users need to trust the methods should be able to handle shifting data characteris-
decisions and be reassured based on the explainable pro- tics and adapt accordingly [13].
cess and feedback. On the other hand, AI developers need At present, XAI has gained a great deal of attention across
to understand the limitations of current models to validate different application domains. Accordingly, an increasing
and improve future versions. Besides, regarding product number of XAI tools and approaches are being introduced
Table 1 Explainable AI: who Who? Why? For what?

need it? Why? For what?
End user Understand decisions Persuasive explanation
AI developer Understand limitations; improve future visions; Intrinsic explanation;
debug algorithms training and valida-
tion
Product engineer System design, integration, and deployment Complete explanation
13
Human-Centric Intelligent Systems (2023) 3:161–188 163
both in industry and academia. These advancements aim such as different requirements, suitable XAI approaches, and
to address the ongoing trade-off between interpretability domain-specific limitations. Conducting such an analysis is
and predictive power.There is a growing recognition that crucial as it allows us to gain a deeper understanding of
highly interpretable models might encounter limitations in the performance of XAI techniques in real-world scenarios.
capturing complex relationships, which can lead to reduced Additionally, it helps us identify the challenges and opportu-
accuracy. On the other hand, complex models often achieve nities that arise when applying these approaches to different
superior accuracy but at the expense of interpretability. application domains. By bridging this gap, we can make
Balancing these considerations becomes crucial and is con- significant strides towards developing more effective and
tingent upon the specific requirements of the application reliable XAI systems tailored to specific domains and their
domain. In certain contexts, such as healthcare or finance, unique characteristics.
interpretability and transparency play pivotal roles in ensur- In this survey, our primary objective is to provide a com-
ing regulatory compliance and addressing ethical considera- prehensive overview of explainable artificial intelligence
tions [14, 15]. In other domains, such as medical image or (XAI) approaches across various application domains by
signal recognition, where accuracy is paramount, the focus exploring and analysing the different methods and tech-
may be more on predictive power than interpretability [16]. niques employed in XAI and their application-specific con-
The current XAI methods exhibit various dimensions and siderations. We achieve this by utilizing three well-defined
descriptions to understand deep learning models and some taxonomies, as depicted in Fig. 1. Unlike many existing sur-
survey papers [3, 17, 18] have summarized the methods and veys that solely focus on reviewing and comparing meth-
basic differences among different XAI approaches. How- ods, we go beyond that by providing domain mapping. This
ever, the state-of-the-art analysis with respect to existing mapping provides insights into how XAI methods are inter-
approaches and limitations for different XAI-enabled appli- connected and utilized across various application domains,
cation domains still lacks investigation. and even in cases where domains intersect. Additionally,
The field of explainable artificial intelligence (XAI) has we delve into a detailed discussion on the limitations of the
witnessed the emergence of numerous methods and tech- existing methods, acknowledging the areas where further
niques aimed at comprehending the intricate workings of improvements are necessary. Lastly, we summarize the
deep learning models. Currently, some survey papers have future directions in XAI research, highlighting potential
made efforts to summarize these methods and offer a fun- avenues for advancements and breakthroughs. Our contri-
damental understanding of the distinctions among various butions in this survey can be summarized as follows:
XAI approaches [3, 17, 18]. However, while certain sur-
vey papers have focused on specific domains like health- • Develop a new taxonomy for the description of XAI
care [19] or medical applications [20], there still exists a approaches based on three well-defined orientations with
substantial gap in the state-of-the-art analysis pertaining a wider range of explanation options;
to the existing approaches and their limitations across all • Investigate and examine various XAI-enabled applica-
XAI-enabled application domains. This gap necessitates a tions to identify the available XAI techniques and domain
comprehensive investigation encompassing various aspects insights through case studies;
Fig. 1 The proposed organiza-

tion to discuss the approaches,
limitations and future directions
in XAI
13
• Discuss the limitations and gaps in the design of XAI articles, survey papers and scholarly publications that
methods for the future directions of research and devel- contribute to the understanding of XAI approaches and
opment. their application in the selected domains through case
studies.
In order to comprehensively analyze XAI approaches, limi- • Limitations and challenges: The scope also encompass
tations, and future directions from application perspectives, examining the limitations and challenges of existing XAI
our survey is structured around two main themes, as depicted methods and approaches. This could involve identify-
in Fig. 1. The first theme focuses on general approaches and ing common issues, gaps in the literature, and areas that
limitations in XAI, while the second theme aims to analyze require further research or improvement.
the available XAI approaches and domain-specific insights.
Under each domain, we explore four main sub-themes: Having the scope of review established, the selected data-
problem definition, available XAI approaches, case studies, bases and a search engine include Scopus, Web of Science
and domain insights. Before delving into each application and Google Scholar (Search engine) and arXiv between
domain, it is important to review the general taxonomies 2013 and 2023. The search terms based on the scopes are:
of XAI approaches. This provides a foundation for under-
standing and categorizing the various XAI techniques. In • XAI keywords: explainable, XAI, interpretable.
each domain, we discuss the available and suitable XAI • Review keywords: survey, review, overview, literature,
approaches that align with the proposed general taxonomies bibliometric, challenge, prospect, trend, insight, oppor-
of XAI approaches. Additionally, we examine the domain- tunity, future direction.
specific limitations and considerations, taking into account • Domain keywords: medical, biomedical, healthcare,
the unique challenges and requirements of each application wellness, civil, urban, transportation, cyber security,
area. We also explore cross-disciplinary techniques that con- information security, education, training, learning and
tribute to XAI innovations. The findings from these discus- teaching, coaching, finance, economics, law, legal sys-
sions are summarized as limitations and future directions, tem.
providing valuable insights into current research trends and
guiding future studies in the field of XAI. With the selected search terms, the two-round search strings
were designed to effectively retrieve relevant information
and narrow down the search results.
2 Taxonomies of XAI Approaches The first round, focusing on general research papers, con-
sisted of the following search string: (explainable OR XAI
2.1 Review Scope and Execution OR interpretable) AND (survey OR review OR overview
OR literature OR bibliometric OR challenge OR prospect
This work is mainly based on a scope of review refers to OR trend OR opportunity OR "future direction").
the specific boundaries and focus of the research being con- The second round, aimed at selecting specific applica-
ducted. In the context of an XAI survey, the scope typically tion domains, utilized the following search string: (explain-
includes the following aspects: able OR XAI OR interpretable) AND (medical, biomedical
OR healthcare OR wellness OR civil OR urban OR trans-
• XAI approaches: The review will focus on examining portation OR “cyber security” OR “information security”
and analyzing different XAI approaches and methods that OR education OR training OR “learning and teaching” OR
have been proposed in the literature. This include visu- coaching OR finance OR economics OR law OR “legal
alization techniques, symbolic explanations, ante-hoc system”).
explanations, post-hoc explanations, local explanations, Publications that did not clearly align with the scopes
global explanations and any other relevant techniques. based on their title or abstract were excluded from this
• Application domains: The review may consider various review. While not all literature explicitly stated this infor-
application domains where XAI techniques have been mation, the extracted data was organized and served as the
applied, including medical and biomedical, healthcare, foundation for our analysis.
finance, law, cyber security, education and training, civil
engineering. The scope involve exploring the usage of 2.2 XAI Approaches
XAI techniques in these domains and analyzing their
effectiveness and limitations across multiple domains. The taxonomies in the existing survey papers generally cat-
• Research papers: The review will involve studying and egorised XAI approaches based on scope (local or global)
synthesizing research papers that are relevant to the cho- [21], stage (ante-hoc or post-hoc) [17] and output for-
sen scope. These papers may include original research mat (numerical, visual, textual or mixed) [22]. The main
13
difference between the existing study and our survey is every layer of a neural network model and measures their
that this paper focuses on the human perspective involving suitability for classification, which is beneficial to provide
source, representation, and logic reasoning. We summarise a better understanding of the roles and dynamics of the
the taxonomies categorised in this survey in Fig. 2: intermediate layers. Zhang et al. [24] used an explanatory
Source-oriented (source-oriented (SO)) the sources that graph to reveal the knowledge hierarchy hidden inside a
support building explanations can be either subjective (S) or pre-trained convolutional neural network (CNN) model.
objective (O) cognition, depending on whether the explana- In the graph, each node represents a part pattern, and
tions are provided based on the fact or human experience. each edge encodes co-activation relationships and spatial
For example, in the medical field, if the explanation of a relationships between patterns. A phase-critic model is
diagnosis is provided based on the patient’s clinical symp- developed to generate a candidate textual explanation for
toms and explains the cause and pathology in detail dur- the input image [25].
ing the AI learning process, this is from objective cognitive By contrast, qualifying symbolic explanations, such as
concern. In contrast, explanations with subjective cognitive rules and graphical explanations, are provided under human
consider patients’ current physical conditions and doctors’ knowledge. Rules explanations are usually in the form of
medical knowledge. “If-Then" rules to obtain the inferential process of neural
Representation-oriented (representation-oriented networks. Rule extraction techniques include neural network
(RO)) core representation among the XAI approaches can knowledge extraction (neural network knowledge extrac-
generally be classified into visualisation-based (V), sym- tion (NNKX)) [26], rule extraction from neural network
bolic-based (S) or even hybrid (H) methods. Visual-based ensemble (rule extraction from neural network ensemble
methods are the most common representation ways includ- (REFNE)) [27], and Electric Rule Extraction (electric rule
ing input visualisation and model visualisation. Input visu- extraction (ERE)) [28]. Moreover, knowledge graphs can
alisation methods provide an accessible way to view and also help AI models be more explainable and interpretable
understand how input data affect model outputs, while model and are widely used in explainable methods. Andriy and
visualisation methods provide analysis based on the aspect Mathieu [29] applied rule mining using knowledge graph
of layers or features inside the model. (KG)s to reveal semantic bias in neural network models.
Besides visualization-based methods, other formats of Visualisation-based methods and coding symbolic explana-
explanations, including numerical, graphical, rules, and tions generally belong to objective cognitive, while qualify-
textual explanations, are covered in symbolic-based meth- ing symbolic explanations are always subjective cognitive
ods. Symbolic-based methods tend to describe the process by adding human knowledge and opinions.
of deep learning models by extracting insightful informa- Hybrid methods can generate mixed explanations, con-
tion, such as meaning and context, and representing them sisting of visualization explanations and symbolic informa-
in different formats. The coding symbolic-based explanation, which can be either subjective or objective cognitive.
tion is provided directly from the factual features, includ- In [30], visual and textual explanations are employed in the
ing numerical, graphical and textual explanations. For visual question answering task. Both subjective and objec-
instance, a numerical method [23] monitors the features at tive cognitive explanations are provided in this work. We
Fig. 2 Taxonomies of XAI

approaches in this survey
XAI principles
Source- Representation- Logic-

oriented oriented oriented
End-end
Subjective cognitive Visualization-based
relationship
Middle-end
Objective cognitive Symbolic-based
relationship
Correlation
Hybrid-based
relationship
13
summarise the above-mentioned XAI representations with Deep learning methods now can automatically extract
surveyed related work in Table 2. abstract features by end-to-end prediction processing, which
Logic-oriented (logic-oriented (LO)) a well-developed can obtain direct results, but this is insufficient to provide
XAI method with a specific representation can be integrated diagnostic evidence and pathology, and the features can-
logic reasoning into deep learning models, including end- not be completely trusted or accepted. For example, for
end (E-E) relationship, middle-end (M-E) relationship, and glaucoma screening, doctors can use intraocular pressure
correlation (Corr) relationship shown in Table 3. The end- testing, visual field testing, and manual inspection of the
end explanations focus on providing how the AI system optic disc to diagnose the disease and give the cause and
processes from the first input stage to the final output result. pathology based on the patient’s symptoms and pathological
The middle-end performs explanations by considering the reports [154]. However, the deep learning model is difficult
internal AI system structure. The correlation is used to rep- to explain the correlation or causality between its input and
resent the correlations among sequence inputs or consecu- output in different contexts, lacking an explanation of the
tive outputs of deep learning models. Most XAI approaches process, which is difficult to support reasoning in medical
target to clarify the relationship between input features and diagnosis and research.
output results, and very few research relates to middle-end Additionally, due to the data-driven nature of deep learn-
and correlation relationships. ing, models can easily learn the deviations in the training
[155]. This phenomenon is common in medical image pro-
cessing. For example, a deep learning model identifies cer-
tain diseases from images during training while the actual
3 Applications Using XAI Approaches
diagnosis could be another disease, and this should not occur
at all. If users intend to to improve a model, the explanation
Nowadays, applications using XAI approaches have been
of the model is a prerequisite, because before being able
covered in various domains. This section provides details for
to solve a problem, its existence and causality need to be
different XAI techniques used for each application. Some of
identified.
the main applications are summarised in Table 4.
3.1.2 XAI Based Proposals
3.1 Medical and Biomedical
The research on explainable deep learning can enhance the
3.1.1 Problem Definition capabilities for AI-assisted diagnosis by integrating with
large-scale medical systems, providing an effective and
The use of AI systems in medical and biomedical research interactive way to promote medical intelligence. Different
has increasing influences on medical advice and therapeutic from common explainable deep learning methods, the deep
decision-making processes, especially in one of the most learning explanation research of medical image analysis is
common areas that apply deep learning techniques, the med- not only affected by the data, but also related to medical
ical and biomedical image analysis, such as image registra- experts’ knowledge.
tion and localisation, detection of anatomical and cellular In terms of source-oriented, the objective cogitative
structures, tissue segmentation, and computer-aided disease explanation is provided based on visible, measurable find-
diagnosis. ings obtained by medical examinations, tests, or images,
Medical and biomedical image analysis refers to the while the subjective cogitative explanation needs to consider
extraction of meaningful information from digital images, the medical experts’ knowledge and patient situations. The
which utilised a variety of medical imaging techniques, existing XAI proposals cover both objective and subjective
including computed tomography (computed tomogra- cognitive aspects. For example, the gradient-weighted class
phy (CT)), magnetic resonance imaging (Magnetic reso- activation mapping [gradient-weighted class activation map-
nance imaging (MRI)), ultrasound (ultrasound (US)), and ping (Grad-CAM)] method proposed in [36] performs expla-
X-rays, covering important body parts such as the brain, nation by highlighting the important regions in the image,
heart, breast, lung and kidney [153]. Due to the advantages which refers to objective cogitative. Some researchers also
of deep learning in the field of medical image analysis, in consider using subjective sources, such as in [85], authors
recent years, more and more researchers have adopted deep presented the explanation by combining time series, histo-
learning to solve problems of medical image analysis and pathological images, knowledge databases as well as patient
achieved good performances. Although medical image anal- histories.
ysis based on deep learning has made great progress, it still In terms of representation-oriented, visualisation methods
faces some urgent problems in medical practice. emphasise the visualisation of training data rules and the vis-
ualisation inside the model, which is the most popular XAI
13
Table 2 Summarising the XAI approaches from representation-oriented
Type Sub-type Typical examples Pros Cons References
Visualization-based Model-agnostic visualization Attribute-based; attention-based, The outputs are simple to understand It only offers a user-friendly interface [31–35]
Human-Centric Intelligent Systems (2023) 3:161–188
perturbation-based; CAM-based; because there is no requirement for seeing and comprehending input
counterfactual interpretation, etc. to comprehend the internal model data and does not provide informa-
structure tion on the context of the model
learning process
Model-specific visualization Concept attribution; dimensionality The explanation can uncover the The features used to determine inter- [36, 37]
reduction; graph neural network information that is hidden inside the pretability might not be repeatable
explanation, etc. model among different models
Symbolic-based Coding symbolic Knowledge graph The explanation is more objective and The explanation may be incomplete [23–25, 38, 39]
understandable for its close connec- with some hidden information
tion with factual features
Qualifying symbolic Rule-extraction techniques; semantic The explanation can incorporate prior The explanation aims at describing [29, 40–45]
web; knowledge graph knowledge the inferential process of a model
trained on categorical inputs. It is
highly dependent on reliable rules or
knowledge graphs
Hybrid-based Rivelo; explainer; TensorBoard Users can interactively explore a set If the mixed explanation is not well [46–51]
of visual and textual instance-level designed and structured, the explana-
explanations tion retrieved by humans might be
confusing
13
167
Table 3 Summarising the XAI approaches from logic-oriented

Type Typical examples Pros Cons References
E-E relationship Regression-based partitioned Better applicability with no limita- Still has challenging to understand [52–54]
method; Importance estimation tions for various models the internal logic that how to gen-
network, etc. erate outputs and how to forecast
the model behaviour in a variety
of situations
M-E relationship Feature-wise relevance; Layer-wise The procedure is quite straightfor- The explanation may be partial [55–57]
relevance; etc. ward, and it is easy to identify just covering particularly part of
internal structure or any bias in the interesting areas
middle
Corr relationship Input-level correlation (feature, It has high benefits to understand The explanation may be complex by [58, 59]
label, image, etc.) why the model produces a given considering different dimensions
result during the entire learning of relationships
approaches used in medical image analysis. Some typical with shapley additive explanations (shapley additive expla-
examples include attributed-based and perturbation-based nations (SHAP)) and correlation analysis to achieve the opti-
methods for model-agnostic explanations as well as CAM- mal feature vector for classification.
based and concept attribution for model-specific explana-
tions. Locally-interpretable model-agnostic explanations
(locally-interpretable model-agnostic explanations (LIME)) 3.1.3 Cases Studies
[86] is utilised to generate explanations for the classification
of medical image patches. Zhu et al. [87] used rule-based Explainable AI applications in the medical field assist
segmentation and perturbation-based analysis to generate the healthcare providers in making accurate diagnoses, treat-
explanation for visualising the importance of each feature ment decisions, risk assessments, and recommendations.
in the image. The concept attribution [37] is introduced by The transparency and interpretability of these AI models
quantifying the contribution of features of interest to the ensure that clinicians can trust and validate the outputs, lead-
CNN network’s decision-making. Symbolic methods focus ing to improved patient care and outcomes [92].
on the symbolic information representations that simulate Lesion classification: The visualisation of lesion areas
the doctor’s decision-making process with natural language, mainly refers to heat maps [88], attention mechanisms [89,
along with the generated decision results, such as primary 90], and associated with other diagnostic means such as
diagnosis reports, etc. For example, Kim et al.[66] intro- structural remodeling [91] and language models to repre-
duced concept activation vectors (CAVs), which provided sent report text [156] to find out lesion areas. These meth-
textual interpretation of neural network internal state with ods provide visual evidence to explore the basis for medi-
user-friendly concepts. Lee et al. [73] provided explainable cal decision-making. For example, Biffi et al. [91] used a
computer-aided diagnoses by combining a visual point- visualisation method on the original images to measure the
ing map and diagnostic sentences based on the predefined specificity of pathology, using interpretable characteristics
knowledge base. of specific tasks to distinguish clinical conditions and make
In terms of logical-oriented, explanations focused on end- the decision-making process transparent. Garcia-Peraza-
end logic reasoning, such as the above-mentioned LIME, Herrera et al. [92] used embedded activation charts to detect
perturbation-based methods are utilised to explain the rela- early squamous cell tumors, showing the focus on the inter-
tionship between input medical images and predicted results. pretability of the results and use it as a constraint to provide
For example, a linear regression model is embedded into a more detailed attention map. Paschali et al. [88] used a
LIME [86] to identify relevant regions by plotting heat maps model to activate the fine-grained Logit heat map to explain
with varying color scales. Zhang et al. [56] provided a diag- the medical imaging decision-making process. Lee et al.
nostic reasoning process and translate gigapixels directly to [90] used head CT scan images to detect acute intracranial
a series of interpretable predictions. Shen et al. [58] used a hemorrhage, and proposed an interpretable deep learning
hierarchical architecture to present a concept learning-based framework. Liao et al. [89] provided a visual explanation
correlation model. The prediction’s interpretability is aided basis for the automatic detection of glaucoma based on the
by the model’s intermediate outputs, which anticipate diag- attention mechanism, and in the process of automatic glau-
nostic elements connected to the final classification. A cor- coma detection, the system provides three types of output:
relation-XAI approach proposed by [59] is used for feature prediction result, attention map, and prediction basis, which
selection merging generalised feature importance obtained enhances the result interpretability.
13
Disease diagnosis and treatment: Research on XAI in
Pain detection; quality of life analysis; surgery complication

disease diagnosis and treatment has recently gained much
Financial-related figure forecasting; credit risk manage-

attention. Amoroso et al.[93] applied clustering and dimen-
Lesion classification; disease diagnosis and treatment
Decision Vehicle actions; power system management

sion reduction to outline the most important clinical feature
Feedback providing; intelligent tutoring systems

for patients and designed oncological therapies in the pro-
Intrusion detection system; malware detection

posed XAI framework. In the context of high-risk diagno-
ses, explainable AI techniques have been applied to provide
visual cues and explanations to clinicians. For example,
Sarp et al. [94] utilized LIME (local interpretable model-
ment; legal text classification

agnostic explanations) to generate visual explanations for
a CNN-based chronic wound classification model. These
visual cues help clinicians understand the model’s decision-
making process and provide transparency in the diagnosis.
Moreover, Wu et al. [95] proposed a counterfactual multi-
Case studies
granularity graph supporting fact extraction (counterfactual

multi-granularity graph supporting fact extraction (CMGE))
for lymphedema diagnosis. CMGE is a graph-based neural
network that extracts facts from electronic medical records,
providing explanations and causal relationships among fea-
Corr
tures. These explanations assist clinicians in comprehending

✓
✓
✓
x
x
the reasoning behind the diagnosis and identifying relevant

factors contributing to the condition.
M-E
In the domain of relatively low-risk screenings, explain-

✓
✓
✓
✓
✓
x
able AI research has explored the integration of medical

records and natural language processing methods to provide
E-E
LO
interpretable diagnostic evidence. For instance, Wang et al.

✓
✓
✓
✓
✓
✓
[96] and Lucieri et al. [97] have integrated medical records

into map and image processing, creating diagnostic reports
H
✓
✓
✓
✓
✓
directly from medical images using multi-modal medical

information. This integration enables the generation of inter-
M
✓
✓
✓
✓
✓
pretable evidence and explanations for clinicians during the

screening process.
RO
Additionally, hybrid XAI approaches have been explored,

✓
✓
✓
✓
✓
✓
I
such as Mimir proposed by Hicks et al. [84]. Mimir learns

intermediate analysis steps in deep learning models and
O
✓
✓
✓
✓
✓
incorporates these explanations to produce structured and

semantically correct reports that include both textual and
SO
✓
✓
✓
✓
✓
S
visual elements. These explanations aid in understanding

the screening results and provide insights into the features
[36, 37, 40, 60–88, 88,
contributing to the risk assessment, assisting clinicians in

[98–101] [102–112]
89, 89, 90, 90–97]
[36, 63, 144–152]
making informed decisions and recommendations for further

[113, 113–126]
[17, 127–134]
screenings or preventive measures.

References
[135–143]
3.1.4 Domain‑Specific Insights
Table 4 Applications of XAI
In terms of biomedical field, the end-users of XAI are mostly

pharmaceutical companies and biomedical researchers.
Medical and biomedical
Education and training
With the use of AI and XAI, they can understand the rea-
soning behind predictions made for disease diagnosis and
Application field
Civil engineering
Finance and law
diagnostic evidence. This insight into the decision-making

Cybersecurity
Healthcare
process can enhance the transparency and trustworthiness

of AI-based predictions, leading to more accurate, reliable
and efficient disease diagnosis. Nonetheless, XAI techniques
13
have their limitations. For example, due to the inherent com- 3.2.2 XAI Based Proposals
plexity of disease diagnosis and treatment, XAI may strug-
gle to provide full and concise explanations of all factors Adding to the numerous research of SARS-CoV-2 mutations,
influencing a prediction. Further, if a situation is novel or Garvin et al. [160] used a method called iRF-LOOP [161],
significantly different from past scenarios used in training an XAI algorithm, in combination with Random Intersection
the AI, the explanations offered by XAI may be insufficient Trees (RIT) [162] for the matrix of variable site mutations
or not entirely accurate. Moreover, given the dynamic and analysis. The network as the output of the iRF-LOOP model
multifaceted nature of public health and disease spread, XAI includes a score of the ability to predict the absence or pres-
might struggle to provide real-time explanations or take into ence of another variable site mutation for each variable site
account every single factor influencing disease spread. This mutation. Ikemura et al. [163] used an unsupervised XAI
includes factors like socioeconomic conditions, behavior method BLSOM (batch-learning self-organizing map) for
changes, and environmental changes. Additionally, if the oligonucleotide compositions of SARS-Cov-2 genomes that
situation changes rapidly, as in the case of a new disease out- reveals new noval characteristics of genome sequences and
break, the explanations provided by XAI might be outdated drivers of species-specific clustering. This BLSOM method
or not entirely accurate. Therefore, XAI provides significant presented the contribution levels of the variables at each
advantages in the biomedical field by enhancing transpar- node by visualising the explanation with a heat map.
ency and trust in AI predictions, it also faces challenges In terms of source-oriented, XAI in healthcare is mainly
related to the complexity and dynamic nature of biomedi- visible, measurable findings, medical histories, numeri-
cal data and scenarios. More research and advancements cal records and reports. This information conducts from
are needed to improve the capability of XAI in handling a series of medical examinations, which is not subjective
these challenges and providing clear, concise, and real-time to the explanation methods or requirements. For example,
explanations. SHAP is an explainer that helps to visualise the output of
the machine learning model and compute the contribution
3.2 Healthcare of each feature to the prediction [164].
In terms of representation-oriented, in the healthcare
3.2.1 Problem Definition field, explanations of AI models are more realistically appli-
cable to overall AI processes, but individual decisions need
AI-assisted exploration has broad applications in healthcare to be carefully considered. XAI in the healthcare field mainly
including drug discovery [104, 105] and disease diagnosis includes causality, which is the capacity to identify haphaz-
[103, 106]. Deep learning techniques achieved high accu- ard connections among the system’s many components, and
racy on classification problems—e.g., using MRI images transferability, which is the capacity to apply the information
to identify different stages of dementia [157]. The Health- the XAI provides to different issue domains[107].
care industry can now examine data at extraordinary rates In terms of logical-oriented, explanations in healthcare
without sacrificing accuracy due to the development of deep mainly focus on correlation analysis. For example, SHAP
learning. Healthcare offers unique challenges, with typically has been widely used in the healthcare industry to provide
much higher requirements for interpretability, model fidel- explanations for hospital admission [108], quality of life
ity, and performance than most other fields. More interpret- [109], surgery complication [110], Oncology [111] and risk
able solutions are in demand apart from the binary clas- factor analysis of in-hospital mortality [112].
sification of positive-negative testing, and they can benefit
clinicians, allowing them to have an understanding of the 3.2.3 Cases Studies
results. In healthcare, critical applications like predicting a
patient’s end-of-life may have more stringent conditions on Pain detection based on facial expressions: Understand-
the fidelity of interpretation than just predicting the cost of ing the choices and restrictions of various pain recognition
a procedure[158]. Researchers have analysed the interpret- models is essential for the technology’s acceptability in
ability of deep-learning algorithm for the detection of acute high-risk industries like healthcare. Researchers have pro-
intracranial haemorrhage from small datasets [90]. More- vided a method for examining the variances in learned rep-
over, some studies focus on explaining how AI improves resentations of models trained on experimental pain (BioVid
health-associated records of individual patients’ electronic heat pain dataset) and clinical pain (UNBC shoulder pain
health records (electronic health records (EHR))[159]. This dataset). To do this, they first train two convolutional neural
is because AI models for treating and diagnosing patients networks, one for each dataset, to recognise pain and the
might be complex as these tools are not always sufficient on absence of pain automatically. The performance of the heat
their own to support the medical community and medical pain model is then assessed using pictures from the shoulder
staff may not be exposed to these technologies before[159]. pain dataset, and vice versa. This is known as a cross-dataset
13
evaluation. Then, to determine which areas of the photo- security are significant concerns when dealing with sensitive
graphs in the database are most relevant, they employed a health data. Additionally, the ability of XAI to provide clear,
Layer-wise Relevance Propagation [165]. In this study, they comprehensible explanations in cases where the underlying
showed that the experimental pain model is paying more AI model is extremely complex remains a challenge. Moreo-
attention to facial expression. ver, the healthcare field is inherently dynamic and complex,
Prediction of caregiver quality of life: The caregiver’s with countless interacting variables, so the explainability
quality of life may be impacted by the patient’s depression provided by current XAI methods might not be complete or
and employment status before the onset of symptoms. Some fully accurate. Finally, there are also significant regulatory
researchers analysed the quality of life of caregivers, using and ethical considerations that come with the application of
SHAP to visualize the overall impact of all features and then AI and XAI in healthcare. Regulators will need to establish
selecting a subset of the seven most predictive features to clear guidelines for the use of these technologies to ensure
establish a simpler model (M7) for a global explanation. that they are used responsibly and ethically.
Simpler models may be easier to comprehend, less likely
to be influenced by unimportant or noisy aspects, more 3.3 Cybersecurity
accurate, and more useful. In this study, SHAP was used to
provide post-hoc explanations. Studies explored the most 3.3.1 Problem Definition
predictive features that impact the quality of caregivers’ life,
including the weekly caregiving duties, age and health of the Cybersecurity is the use of procedures, protections, and
caregiver, as well as the patient’s physical functioning and technologies to defend against potential online threats to
age of onset [109]. data, applications, networks, and systems [166]. Maintain-
ing cybersecurity is becoming more and more challenging
3.2.4 Domain‑Specific Insights because of the complexity and huge amount of cyber threats,
including viruses, intrusions, and spam [167].
In healthcare, the end-users of XAI systems range from cli- In recent years, intelligent network security services and
nicians and healthcare professionals to patients and their management have benefited from the use of AI technology,
families. Given the high-stakes nature of many medical deci- such as ML and DL algorithms.
sions, explainability is often crucial to ensuring these stake- There has been a variety of AI methods and tools devel-
holders understand and trust AI-assisted decisions or diag- oped to defend against threats to network systems and appli-
noses. One of the primary benefits of XAI in the healthcare cations that may be present inside an organisation or outside
domain is the potential to make complex medical decisions of it. However, it is challenging for humans to comprehend
more transparent and interpretable, leading to improved how these outcomes are produced since the developed net-
patient care. By providing clear explanations for AI-driven work security-related decision-making model based on arti-
predictions, such as identifying risk factors for a particu- ficial intelligence lacks reasons and rational explanations
lar disease, XAI can help clinicians make more informed [168]. This is due to the black-box nature of AI models. As
decisions about a patient’s treatment plan. Patients, too, can a result of the network vulnerability, the network defence
benefit from clearer explanations about their health status mechanism in this situation transforms into a black box sys-
and prognosis, which can lead to better communication with tem that is susceptible to information leakage and the effects
their healthcare providers and a greater sense of agency in of AI [169]. In order to combat cyber security that takes
their care. For instance, in the context of disease diagnosis, advantage of AI’s flaws, XAI is a solution to the growing
AI models equipped with XAI can interpret complex medi- issue of the black box in AI. Due to XAI’s logical interpreta-
cal imaging data, such as MRI scans, to accurately diagnose tion and key data proof interoperability, experts and general
a disease like dementia or cancer. Not only can these models users can comprehend AI-based models [170].
highlight which features are most important in reaching a Zhang et al. [171] divided these applications into three
diagnosis, but they can also provide a visual explanation categories: defensive network threats applications using
that assists clinicians in understanding the model’s reason- XAI, network security XAI applications in different indus-
ing. Similarly, in drug discovery, XAI can assist in identify- tries, and defence methods against network threats in XAI
ing novel therapeutic targets and predicting the efficacy of applications. This section mainly analyses the defensive
potential drugs, improving the speed and accuracy of drug applications of XAI against network attacks.
development. The transparency provided by XAI in this pro-
cess can improve trust and confidence in the AI model’s sug- 3.3.2 XAI Based Proposals
gestions, and potentially speed up the regulatory approval
process. However, the implementation of XAI in the health- In terms of source-oriented, the objective explanation is
care domain is not without challenges. Privacy and data based on the detected system or network data, while the
13
subjective explanation depends on the analysts’ expertise the relevance features between the input layer and the last
and backgrounds. The existing XAI proposals cover both layer to explain what input feature contributes to making
objective and subjective aspects. For example, Hong et al. that decision.
[113] proposed a framework to provide explanations from
the model and also combine with the analysts’ knowledge
to eliminate the false positive errors of the decision made by 3.3.3 Cases Studies
the AI Models. Amarasinghe and Manic [114] proposed the
method to give the most relevant input features align with Intrusion detection systems XAI studies in intrusion
the domain experts’ knowledge for each concept learned detection systems are normally used to provide expla-
from the system. nations regarding different user perspectives. Most
In terms of representation-oriented, embedding with approaches use already-developed methods to make the
human understandable texts to interpret the results of the results interpretable, with SHAP being the most adopted.
decision-making process is a common way. For example, LIME, on the other hand, has been adopted in only a few
Amarasinghe et al.[115] used text summary to interpret the cases. Shraddha et al. [123] proposed a framework with
reason to the end user of an explainable DNNs-based DoS a DNN at its base and apply an XAI method to add trans-
anomaly detection in process monitoring. Besides, to present parency at every stage of the deep learning pipeline in
the explanation logic, DENAS [116] is a rule-generation intrusion detection system (IDS). Explanations give users
approach that extracts knowledge from software-based measurable factors as to what features influence the pre-
DNNs. It approximates the nonlinear decision boundary diction of a cyber-attack and to what degree. Research-
of DNNs, iteratively superimposing a linearized optimiza- ers create XAI mechanisms depending on who benefits
tion function. Moreover, an image visualisation method was from them. For data scientists, SHAP and BRCG [124] are
also used to explain, as Gulmezoglu [117] generated LIME proposed, while for analysts Protodash is used. For end-
and used saliency maps to examine the most dominant fea- users where an explanation of a single instance is required,
tures of the website fingerprinting attack after the trained researchers suggest SHAP, LIME, and CEM. Hong et al.
DL model. Feichtner et al. [118] used LIME to calculate a [113] proposed a network intrusion detection framework
score for each word, showing the importance of output, and called FAIXID making use of XAI and data cleaning tech-
used the heat map visualisation method to interpret the text niques to enhance the explainability and understandability
description and application request permissions based on of intrusion detection alerts. The proposed XAI algorithms
samples of the correlation between groups. Another interpre- included exploratory data analysis (EDA), Boolean Rule
tation proposal [119] used the image to represent a mobile column generation (BRCG), and contrastive explanations
application and localise the useful salient parts of the model method (CEM) that deployed in different explainability
by the Grad-CAM algorithm. In this way, the analyst can modules respectively to provide cybersecurity analysts
gain knowledge by using the image symptom region for a with comprehensive and high-quality explanations about
specific prediction. the detection decisions made by the framework. On the
In terms of logic-oriented, explanations focus on other hand, collecting analysts’ feedback through the eval-
end-end logic reasoning similar to LIME used in other uation module to enhance the explanation models by data
areas, while local explanation method using nonlinear cleaning also proved effective in this work as well.
approximation (LEMNA) (Local explanation method Malware detection The effectiveness of malware detec-
using nonlinear approximation) is optimised for security tion increases when AI models are applied to signature-
applications based on deep learning, such as PDF mal- based and anomaly-based detection systems. Heuristic-
ware recognition and binary reverse [120]. In contrast to based [172] methods were proposed to understand the
the strong assumption that the model made by LIME is behavior of an executable file using data mining and deep
locally linear, the LEMNA scheme can deal with local learning approaches. The development of interpretable
nonlinearities and takes into account the dependencies techniques for malware detection in mobile environments-
between features. LEMNA can be used for the interpreta- particularly on Android platforms-has received a lot of
tion of a function starting position detection model in a attention. The adversarial attack method is also used to
scene in binary reverse. Yan et al. [121] proposed a tech- improve interpretability. Bose et al. [125] provided a
nique for extracting a rule tree merged from generated framework for interpolating between various classes of
rule tree of the hidden layer and the output layer of the samples at various levels to look at many levels of weights
DNN, which shows the more important input feature in and gradients as well as the raw binary bytes to understand
the prediction task. Also, the middle-to-end explanation is how the MalConv architecture [126] learns in an effort to
used in this area. Amarasinghe et al. [115] used the layer- understand the mechanisms at work.
wiser relevance propagation (LRP) method [122] to find
13
3.3.4 Domain‑Specific Insights 3.4.2 XAI Based Proposals
AI-based cybersecurity systems carry two different kinds XAI in finance is mainly in the form of explaining which
of secondary risks. The possibility of generating mislead- features are more important and which features are less
ing negative results that result in erroneous conclusions is important in the AI model. Under this form, different sta-
present in the first category. The second is the chance of tistical-based techniques can be applied to find the features’
receiving inaccurate notifications or erroneous warnings importance. In the legal domain, information is represented
due to false positive results [173]. In such circumstances, as natural language texts, so the objective of XAI models is
it is imperative to take the required mitigating action to to identify the important words that contribute to the final
ensure that violations or unique events are handled more prediction.
accurately, keeping the decision-making process’s ability In terms of source-oriented, most XAI models in the
to be understood and supported [174]. Additionally, the finance domain are objective and cognitive-based, as they
use of AI by hackers makes it possible for them to cir- are based on fact, not on the human experience. In the legal
cumvent security measures so that data tampering goes domain, those XAI models are objective cognitive since the
unnoticed. This makes it challenging for businesses to highlighted words are from the input data. As CNNs have
correct the data supplied into AI-based security systems. been proven useful in text input data, general XAI methods
As a result, compared to conventional model-based algo- (e.g. Grad-CAM, LIME, SHAP) for CNNs have been used to
rithms, the current difficulty with AI-based systems is explain the AI model trained for legal text [134]. To demon-
that they make decisions that lack reason [167]. Hence, strate the contribution of each word in the given sentence on
XAI is required to enhance trust and confidence in AI- the final prediction, XAI models, like LIME, can indicate the
based security systems. contribution of the word “awesome" on the positive senti-
In cybersecurity, XAI requires a more structured ment prediction result.
approach, utilizing various integrated techniques from In terms of representation-oriented, XAI models in the
diverse fields, guided by a dedicated research community finance domain vary among visualization-based, symbolic-
focusing on increasing formalism. In particular, for areas based and hybrid. For visualisation-based, some tree XAI
like malware detection, there is a need for unified and models [177] can give tree-shape visualisations. For sym-
clearly explained methods, ensuring a coherent under- bolic-based, SHAP and LIME models can give numerical
standing for all stakeholders, including users, analysts, forms. In this case, XAI models try to give the explanation
and developers, to enhance the analysis and prevention of or contribution of different features in making the predic-
cyber-attacks. Moreover, currently, there’s no recognized tion. Symbolic-based XAI in the financial domain empha-
system for gauging whether one XAI system is more user- sises giving the explanation in a quantitative manner. This
intelligent than another. A well-defined evaluation system is more applied in credit risk management. For hybrid, it is
to select the most effective explainability technique is symbolic-based and basically a combination of visualisation.
needed. XAI also needs to grapple with substantial secu- Visualisation in the representation-oriented category has
rity and privacy issues. Current XAI models are still vul- been adopted to the legal domain to visualize the explain-
nerable to adversarial attacks, leading to public concerns ability [132].
about XAI security.
3.4.3 Cases Studies
3.4 Finance Financial data forecasting AI models are being used to pre-

dict financial-related figures, such as stocking price, profit
3.4.1 Problem Definition of the company, etc. However, sophisticated AI models can
only give predictions rather than how these predictions are
In the finance domain, XAI is mainly applied in finan- being made. In a typical application of XAI in financial
cial forecasting [175] and credit risk management [176]. forecasting cases, Local interpretable model-agnostic expla-
Financial forecasting is the problem of predicting dif- nations (LIME) are used to explain the model predictions
ferent financial-related metrics, such as profit, financial locally around each example. LIME is an agnostic-based
healthy ratios, etc. Credit risk management is the problem XAI model which is to simplify the complex model by using
of how to manage credit risks from different subjects. For a linear regression model locally. For instance, Agarwal
instance, from a bank perspective, when a model predicts et al. [178] applied LIME on a stock price AI prediction
a client to be at high risk of default, it should give a rea- model (the AdaBoost model) to explain the prediction. The
son for different stakeholders to understand. AI (AdaBoost model) model used 20 previous days’ stock
prices as the feature to predict the next day’s stock price. The
13
output of LIME is a 2-dimensional bar chart, with the Y-axis In terms of financial data forecasting, the end-users of
showing features and the X-axis showing the importance of XAI are mostly financial institutions like fund manage-
features. In this way, the bar chart clearly shows how the AI ment companies and asset management companies. With
prediction model makes the prediction (i.e., which features explanations provided by AI models, financial professionals
play dominant roles in the prediction model). Similarly, the gain insights into the reasoning behind investment recom-
SHAP XAI model can also be used to explain the financial mendations, risk assessments, and other financial decisions.
forecasting model. The principle of the SHAP model is to This helps them validate the suggestions and communicate
compute the contribution level of each feature to the predic- the rationale more effectively to their clients. One limita-
tion based on game theory. In the same work of Agarwal tion of XAI techniques is that they cannot explain events
et al. [178], SHAP is used to explain the AdaBoost and ran- that never happened in the past. When faced with new and
dom forest-based stock price prediction model. The output unprecedented circumstances, the explanations provided by
of the SHAP model is a 2-dimensional chart, with the X-axis XAI may not adequately account for these events, leading
showing the contribution of features to the model and the to potentially inaccurate forecasts. There is limited current
Y-axis showing each feature used in the prediction model. research analyzing the specific computational cost in XAI
Credit risk management In addition to the financial models in finance. The computational cost of XAI used in
foresting case, XAI has also been applied in the case of the finance domain depends on the complexity of the under-
credit risk management. Credit risk may happen to a finan- lying AI model, feature dimensions, and hardware level. If
cial institution due to an expected financial loss. In [177], a the underlying AI model is a regression or tree-based model
SHAP model and a minimal spanning tree model were used and does not include many factors, the computational cost
to explain an extreme gradient boosting (XGBoost) based will be relatively low. However, if the underlying AI model
credit default risk default probability estimation model. is based on complex neural networks and includes tons of
Features used in the estimation model include financial factors in the model, the computational cost will be high.
information from the balance sheet. The minimal spanning
tree model presents a clustering tree based on financial data, 3.5 Law
and the SHAP model presents the contribution level of each
feature. The principle of the minimal spanning tree model 3.5.1 Problem Definition
is to present the credit risk prediction in a clustered-based
visualisation, while the principle of the SHAP model is to In the legal domain, one of the major concerns of XAI is
present the explainability at a contribution level. whether the legal decisions are made fairly towards cer-
tain individuals or groups [133], since it is the high-stake
3.4.4 Domain‑Specific Insights domain, and decisions have real significant impacts on real
human beings. The explainability of AI models provides
In terms of credit risk management, the end-users of XAI transparency on how the decisions are made. It is the desired
are mostly financial institutions like banks and insurance feature for decisions, commendations, and predictions made
companies. XAI provides transparency and explanations for in the legal domain by AI algorithms. However, there are
AI-driven decisions, allowing these institutions to under- few works that have been done in the legal domain apart
stand and validate the factors influencing risk assessments from general XAI methods [17].
and fraud detection. This empowers financial institutions to
make more informed and accountable decisions. XAI tech- 3.5.2 XAI Based Proposals
niques may struggle to provide clear and concise explana-
tions for every aspect of the decision, potentially leading In the legal domain, information is represented as natural
to incomplete or partial explanations. In addition, credit language texts, so the objective of XAI models is to identify
risk is a dynamic and evolving field, influenced by vari- the important words that contribute to the final prediction.
ous economic, regulatory, and market factors, so the XAI In the legal domain, those XAI models are objective cog-
may not be able to provide real-time explanations of the nitive since the highlighted words are from the input data.
risk management decisions. XAI techniques may struggle As CNNs have been proven useful in text input data, general
to provide clear and concise explanations for every aspect XAI methods (e.g. Grad-CAM, LIME, SHAP) for CNNs
of the decision, potentially leading to incomplete or partial have been used to explain the AI model trained for legal
explanations. In addition, credit risk is a dynamic and evolv- text [134]. To demonstrate the contribution of each word in
ing field, influenced by various economic, regulatory, and the given sentence on the final prediction, XAI models, like
market factors, so the XAI may not be able to provide real- LIME, can indicate the contribution of the word “awesome"
time explanations of the risk management decisions. on the positive sentiment prediction result.
13
In terms of logic-oriented, there are few XAI models in explainability requires certain level of reasoning capabilities
the finance domain falling into this end. As in the finance to have the explain make sense to the users.
domain, the cases are about the prediction of a probability Another challenge is the linking of the evidences. Many
or a numeric value, and the demand for the explanation of legal decisions made by AI systems involve multiple parts of
AI models tends to be about which factors contribute more the input text or documents. Explains using only one piece
to the final prediction. For the legal domain, the XAI models of the information are incomplete. The advent of the large
are end-end relationships, since the explanation has been language models (LLMs), such as GPT series models, may
expressed as the relationship between input words and out- facilitate the reasoning and explaining of decisions made in
put predictions. AI applications in law. The LLMs can be instructed to give
reference for the outputs generated. This provides opportuni-
3.5.3 Cases Studies ties for explainable use of AI in law, but the models involve
extra resources.
Legal text classification Authors in [132] provided a case
study from a lawyer’s perspective to utilise Grad-CAM, 3.6 Education and Training
LIME, and SHAP to explain the legal text classification out-
comes. The AI model consists of DistiBERT for embedding 3.6.1 Problem Definition
and CNN for classification. The method used to represent
the explainability is the heatmap visualisation, i.e., high- As one of the essential methods to improve and optimise
lighted words with different intensities corresponding to dif- learning, AI is now widely applied in the field of educational
ferent contributions to the final prediction. Apart from two research [180, 181]. The applications of AI in education
evaluation metrics, the responses of lawyers on the given (AIED) have shown great benefits in several ways, includ-
explanation have also been collected. The scores on visuali- ing support instructions, personalised learning systems,
sations for the selected six correctly classified sentences are and automated assessment systems [182]. At the same time,
from 4.2 to 6.66 with 0 for worst and 10 for best. The key there are some risks associated with the use of AI, given the
point made by lawyers is that the explanations made by XAI specific nature of education. For example, bias is a promi-
should be understandable for users who have no professional nent issue in discussions on AI ethics in education. When
knowledge of the legal domain. using AI techniques for student performance prediction or
risk identification, they are likely to produce more biased
3.5.4 Domain‑Specific Insights results for a particular group of students based only on the
different demographic variables (such as gender) of the stu-
Explainability not only is necessary for AI applications in dents [183]. Consequently, concerns about AI in relation
Law, but also is required by law (e.g., GDPR) for AI applica- to fairness, accountability, transparency, and ethics (FATE)
tions. In this subsection, we address the necessity of explain- have sparked a growing debate [135].
ability in AI applications in law. The decisions made in law XAI contributes to making the decision-making process
require explainability by nature [179] as it forms the impor- of an AI model more transparent and fair by explaining the
tant part of outputs. All judgements need reasons. Lawyers internal processes and providing a logical chain for generat-
need to explain to the clients, judges need reference to rel- ing outcomes. This is essential for human users to build trust
evant articles or cases to support the decision[132]. For and confidence in AIED. Recently, there has been a growing
the AI-empowered legal consultation or recommender sys- body of research showing the opportunities and demands of
tems, the more important information is why this is relevant applying advanced XAI for education [135].
instead of just listing the relevant articles or similar cases.
For judge results prediction, it is only helpful to profession- 3.6.2 XAI Based Proposals
als like lawyers when explainability is provided.
Although the necessity of explainability in AI appli- In terms of source-oriented, there are different objective fac-
cations in Law, its adoption is faced with challenges and tors that can be used to analyse and evaluate the performance
difficulties. The explainability can be the relevant articles of students in the education domain. Also, since each student
or similar case, but more importantly, the analysis to link comes from a different family environment and has a very
them to the target case. The heatmap, mentioned above as distinct personality, subjective perceptions and feelings in
an example, may provide certain extend of explainability by learning have significant impacts on educational outcomes
highlighting the key words used to make decisions. How- due to the different circumstances of each student. Alonso
ever, the explainability in law applications requires more and Casalino [136] proposed a method that uses an XAI
descriptions in natural language as the most inputs of AI tool, named ExpliClas, to analyse objective attributes of
systems in law are texts written in natural language. This students’ learning processes. Their method provides both
13
global and local explanations. There are also many studies tutoring instructions based on the student’s learning
that use objective characteristics, such as age, gender, and activity performance. Conati et al. [143] attempted to
study hours, to predict and explain students’ performance in integrate multiple machine learning algorithms into
teaching activities [137]. an interactive simulation environment so as to provide
In terms of representation-oriented, the XAI approaches hints to students regarding their learning behaviours and
in education mainly include visualization-based and sym- needs. The proposed XAI mechanism consists of behav-
bolic-based explanations. In [138], a deep learning-based iour discovery, user classification, and hint selection. In
method was proposed to achieve the automatic classifica- the behaviour discovery phase, the authors first apply
tion of online discussion texts in the educational context. an unsupervised clustering algorithm to group students
Particularly, the authors used gradient-based sensitivity and then use association rule mining to analyse student
analysis (SA) to visualise the significance of words in con- behaviour further. In the user classification phase, they
versation texts for recognising the phases of cognitive pres- build a supervised classifier to predict students’ learning.
ence, thus providing the explanation for the deep learning In the hint selection phase, the previous classification
model. Recently, some researchers have also applied the result and association rules will be used to trigger the
symbolic approach in education, expecting to adopt sym- corresponding hints.
bolic knowledge extraction to provide logical reasoning for
the AI interpretation. Hooshyar and Yang [139] provided
a framework that integrates neural-symbolic computing to 3.6.4 Domain‑Specific Insights
address the interpretability of AI models. The framework
considered using prior knowledge, such as logic rules and The end-users of XAI in education and training mainly
knowledge graphs. include students and educators. XAI can help students
In terms of logical-oriented, XAI in education is primar- understand and interpret the outcomes of AI-driven sys-
ily required to provide explanations for machine learning tems, such as automated grading or recommendation algo-
black-box and rule-based models. SHAP was employed in rithms, providing them with transparency and insights into
[140] to explain the black-box student dropout classifica- the feedback. Additionally, XAI provides educators with
tion model in relation to Shapley values. Explanation from a great opportunity to gain a deeper understanding of the
the rule-based algorithms specialises in showing clear logic AI-powered educational tools they employ in their class-
from the input data to the output results. For example, in rooms. By using XAI, educators can acquire insights into
[141], global explanations were provided to train nursing the underlying reasons behind specific recommendations
students by analysing the temporal logic between actions. or suggestions generated by these systems. Consequently,
they can adapt and customize their teaching strategies
3.6.3 Cases Studies based on this understanding.
While XAI holds great potential and prospects for
Feedback providing For students, getting timely and forma- application in the field of education, there are currently
tive feedback from educators about performance and assign- challenges and limitations that need to be addressed.
ments is an important way of improving the efficiency and Many AI algorithms, such as deep learning neural net-
quality of learning. Feedback should include not only the stu- works, can be complex and difficult to interpret. XAI
dent’s marks and evaluation but also, and more importantly, techniques still have limitations to provide clear and com-
an explanation of the problems with the assignment and learn- prehensive explanations for the decisions made by these
ing. XAI has been applied in this area, the relevant techniques complex models, which can hinder their adoption in edu-
include sequence mining, natural language processing, logic cational settings. Also, there is a Lack of standardization
analysis and machine learning approaches. Take writing ana- for XAI. We need standardized metrics and a framework
lytics as an example, which aims to provide feedback for stu- to evaluate and assess the explanations provided by XAI,
dents to improve their writing. Knight et al. [142] introduced particularly when comparing different XAI techniques
an open-source tool, AcaWriter, which provides informative and approaches. The absence of standardized practices
feedback to fit different learning contexts. AcaWriter employs and guidelines can lead to inconsistency and confusion in
the Stanford CoreNLP to process each input sentence, and implementing XAI solutions. Addressing trade-offs is an
then it uses a rule-based model to extract the matched patterns. essential step in developing machine learning models, and
Their research demonstrates the great application of XAI in XAI is no exception. Finding the right balance between
education by explaining to students about their writing. explainability and performance is crucial, especially in
Intelligent tutoring systems In addition to providing educational contexts where accurate feedback and predic-
feedback to students, XAI can also help give personalised tions are necessary.
13
3.7 Civil Engineering cognition method, is used to explain the highlight of impor-

tant regions in 2D or 3D images. Time series, 2D images,
3.7.1 Problem Definition 3D images, Lidar images, knowledge databases and ethical
criteria are utilised as subject sources to explain the model
AI systems used in civil engineering research have a sig- [147, 185, 187].
nificant impact on the decision-making processes in road In terms of representation-oriented, visual interpretation
transport and power systems. In particular, autonomous driv- is the highest level semantics to understand which parts of
ing techniques in road transport and power system analysis the image impact the model, emphasing on visual structure
and power systems are the common areas used deep learn- of data and model, which is the primary XAI method used in
ing techniques, such as navigation and path planning, scene autonomous driving. These XAI methods can be divided into
recognition, lane and obstacle detection, as well as planning, gradient-based and back propagation-based. Gradient-based
monitoring, and controlling the power system [150, 184]. interpretation methods include CAM, and its enhanced vari-
In the field of autonomous driving, deep learning tech- ants such as Guided Grad-CAM, Grad-CAM, Grad-CAM++
niques are normally utilised to recognize scenes for digi- and Smooth Grad CAM++. CAM can highlight the discrim-
tal images [184, 185]. While in the field of power system inative regions of a scene image used for scene detection
analysis, deep learning techniques are used to extract fea- [147]. Backpropagation-based methods contain guided back-
tures from the underlying data for power system manage- propagation, layered relevance propagation, visual backprop
ment, such as power grid synthesis, state estimation, and and deep lift. Visual Backprop shows which input pixels
photovoltaic (PV) power prediction [150, 186]. Deep learn- set contributes to steering self-driving cars [144]. Symbolic
ing explainable techniques are used to automatically extract interpretation uses understandable language to provide evi-
abstract features of images or depth non-linear features of dence for result recommendations in autonomous driving
underlying data through end-to-end predictive processing and power system management. In autonomous driving,
to obtain results, which is not sufficient to provide the evi- proposed AI methods make decisions according to traffic
dence to trust and accept the result of autonomous driving rules. For example, “the traffic light ahead turned red,” thus
and power system management. For example, one can use “the car stopped” [185]. In power system management, it
traffic lights and signal recognition for driving planning, in uses the data gathered from occupant actions for resources
which the traffic lights at crosswalks and intersections are such as room lighting to forecast patterns of energy resource
an essential function in following traffic rules and prevent- usage [188]. Hybrid interpretation combines visual inter-
ing traffic accidents. Deep learning methods have achieved pretation and symbolic interpretation to provide steering
prominence in traffic sign and light recognition, but they determination in autonomous driving. For example, Berke-
are hard to explain the correlation between inputs and out- ley Deep Drive-X (BDD-X) is introduced in autonomous
puts and lack an explanation to support reasoning in driving driving which includes the description of driving pictures
planning studies [187]. In power system management, deep and annotations for textual interpretation [49].
learning methods may mislead the output explanations of In terms of logical-oriented, the end-end explanations are
power stability to provide unreliable recommendations, so used to explain the relationship between input images includ-
explanations can increase user trust [150]. ing obstacle and scene images and the prediction. For exam-
ple, LIME is utilised to explain the relationship between
3.7.2 XAI Based Proposals input radar image and prediction results [189]. Middle-end
explanations reveal reasons behind the autoencoder-based
XAI can improve the management of autonomous driving assessment model and how they can help drivers reach a bet-
and power system, providing an effective interaction to pro- ter understanding and trust in the model and its results. For
mote smart civil engineering. Deep learning interpretation example, a rule-based local surrogate interpretable method
research in autonomous driving and power systems is a com- is proposed, namely MuRLoS, which focuses on the interac-
mon interpretable deep learning method because it is not tion between features [149]. Correlation expatriation is used
only influenced by data, but also relates to expert knowledge in the risk management of self-driving and power systems.
and ethical principles. For example, SHAP is used to assess and explain collision
In terms of source-oriented, objective interpretability risk using real-world driving data for self-driving [190].
obtains visible or measurable results from 2D and 3D images
or underlying datasets, while subjective interpretability 3.7.3 Cases Studies
requires consideration of the knowledge from automotive
or electrical experts and the ethical standards of their fields. Decisive vehicle actions Decisive vehicle actions in autono-
Currently, XAI proposals include objective and subjec- mous driving are based on multiple tasks, such as scene
tive cognitive aspects. For example, CAM, as an objective recognition, obstacle detection, lane recognition, and path
13
planning. It can use attention mechanisms, heat maps, diag- resources because of real-time explanations based on timely
nostic models and texture descriptions to recognise obsta- response. Moreover, deceive vehicle actions require high
cles, scenes and lanes and steer the car operation [147, 185, dimensional sensor data, such as the inputs from LiDAR
187]. As mentioned before, CAM is used to highlight the and stereo cameras, which lead the methods, like LIME and
main area for recognition [63]. Visual Backprop, unlike SHAP, which adopts approximate local decision bounda-
CAM-based, emphases highlighting pixel-level to filter ries, are expensive for computation and especially for
features of scene images [144]. Grad-CAM is combined high-dimensional inputs. The requirements in XAI that can
with existing fine-grained visualisations to provide a high- generate real-time, informative explanations without over-
resolution class-discriminative visualisation [36]. Visual burdening the computational resources of the system.
attention heat maps are used to explain the vehicle control- In terms of infrastructure system management, such as
ler behaviour through segmenting and filtering simpler and power or water system management, general public, includ-
more accurate maps while not degrading control accuracy ing governments and residents, are the key end-users in
[145]. A neural motion planner uses 3D detection instances power system management. Government bodies want to
with descriptive information for safe driving [146]. An oversee the safe and fair use of AI in power system man-
interpretable tree-based representation as hybrid presenta- agement. Meanwhile, residents may be curious about the
tions combines rules, actions, and observation to generate mechanics of AI used to manage power systems in the city.
multiple explanations for self-driving [147]. An architecture XAI can be used to evaluate AI systems for safety, fairness,
is used for joint scene prediction to explain object-induced transparency, and adherence to regulatory requirements.
actions [149]. An auto-discern system utilises surroundings Interpretation complexity is a primary challenge for XAI in
observations and common-sense reasoning with answers for infrastructure system management due to the multidimen-
driving decisions [148]. sional nature of the data, which includes factors from power
Power system management Power system manage- generators, transmission lines, and power consumers. More-
ment normally consists of stability assessment, emergency over, unlike the case of autonomous driving, power system
control, power quality disturbance, and energy forecasting. operations demand more technical expertise and need to
CNN classifier, combined with non-intrusive load monitor- adhere to various regulatory requirements. Consequently,
ing (NILM), is utilised to estimate the activation state and XAI is not only to provide coherent and insightful interpre-
provide feedback for the consumer-user [150]. The shape tations of the system’s operations but also to demonstrate
method is firstly used in emergency control for reinforce- that these operations comply with all relevant regulations.
ment learning for grid control (RLGC) under three differ- The entire process in infrastructure system management is
ent outputs analysis [151]. Deep-SHAP is proposed for the starting from generation and distribution to monitor con-
under-voltage load shedding of power systems, and it adds sumer usage patterns. The complexity is future amplified by
feature classification of the inputs and probabilistic analysis the demands for load balancing and power outages, which
of the outputs to increase clarity [152]. influences the public life and the city operation. Moreover,
it also need to fix the various regulations and standers. To
3.7.4 Domain‑Specific Insights evidence such compliance, XAI may need to generate more
complex or detailed explanations, thus increasing the com-
In terms of transportation systems, operators, such as driv- putational cost.
ers and passengers, are the primary end-users in scenarios
involving decisive vehicle actions, because they may want
to comprehend the reasoning behind the decisions made 3.8 Cross‑Disciplinary Techniques for XAI
by the autonomous system. It is very important in high- Innovations
stake domains which human lives are risk. XAI can provide
explanations for AI decisions to enhance the system more XAI innovations for cross-disciplinary refers to the advance-
transparent and fostering trust. Real-time explanations pose ments and developments in explainable AI (XAI) that span
a significant challenge for XAI in decisive vehicle actions, multiple domains and disciplines. It involves the integra-
because decisions need to be made with fractions of a section and adaptation of XAI techniques and methodologies
ond. Rapidly changing environments such as weather con- to address complex problems and challenges that arise in
ditions, pedestrian movement and other vehicles actions diverse fields.
promote XAI should ideally make quick and accurate deci- One aspect of XAI Innovations for cross-disciplinary
sions. Moreover, every driving situation can be unique. XAI is the exploration and utilization of common XAI tech-
needs suitable for diversity situation and adapt its expla- niques across different domains. These techniques, such as
nations which based on context-aware interoperability. As attention-based models, model-agnostic methods, and rule-
previously mentioned, XAI demands more computational based methods, can be applied to various fields to provide
13
transparent and interpretable explanations for AI models. Another aspect of XAI innovations for cross-disci-
Below are some examples of common XAI techniques: plinary involves the development of domain-specific
XAI approaches. In Table 5, we summarize some typi-
1. Regression-based partitioned methods:can be applied cal suitable XAI approaches for different domains. These
to any black-box model. For example, LIME approxi- approaches can be tailored to the unique characteristics
mates the decision boundaries of the model locally and and requirements of specific domains, taking into account
generates explanations by highlighting the features that the specific challenges and complexities of each field.
contribute most to the prediction for a specific instance. Domain-specific XAI approaches consider various fac-
LIME can be used in domains such as healthcare, cyber tors, including domain knowledge, regulations, and ethi-
security, finance, or education to provide instance-level cal considerations, to create an XAI framework that is
interpretability and explainability. SHAP is another specifically designed for a particular domain. By incor-
common technique based on cooperative game theory, porating domain expertise and contextual information,
which can be applied to different domains to explain the these approaches provide explanations that are not only
importance of features in the decision-making process. interpretable but also relevant and meaningful within their
For example, in medical diagnostics, SHAP can help respective domains.
understand which medical parameters or biomarkers By tailoring XAI approaches to specific domains, prac-
have the most impact on a particular diagnosis. titioners can gain deeper insights into the behavior of AI
2. Feature importance: Feature importance techniques models within the context of their field. This not only
assess the relevance and contribution of each feature enhances transparency and trust in AI systems but also ena-
in the model’s predictions. Methods like permutation bles domain-specific considerations to be incorporated into
importance, Gini importance, or gain-based importance the decision-making process, ensuring the explanations are
are commonly used. Feature importance can be useful in relevant and aligned with the requirements and constraints
various domains to identify the factors that drive specific of each domain.
outcomes or decisions. For instance, in finance, feature Furthermore, XAI innovations for cross-disciplinary
importance can help understand which financial indica- emphasize the importance of collaboration and the integra-
tors or market factors play a crucial role in investment tion of expertise from different fields. This approach recog-
decisions. nizes that the challenges and complexities of XAI extend
3. Partial dependence plots: Partial dependence plots visu- beyond individual domains and require a multidisciplinary
alize the relationship between a feature and the model’s perspective. Collaboration and integration of expertise ena-
output while holding other features constant. These plots ble a holistic approach to XAI, where insights from different
show how changing the value of a specific feature affects disciplines can inform the development of innovative and
the model’s predictions. Partial dependence plots can effective solutions. For example, in the field of healthcare,
be employed in domains such as healthcare, where they collaboration between medical practitioners, data scientists,
can provide insights into the impact of certain medical and AI researchers can lead to the development of XAI tech-
treatments or interventions on patient outcomes. niques that not only provide interpretable explanations but
4. Rule-based models: Rule-based models provide trans- also align with medical guidelines and regulations. This
parent and interpretable decision-making processes by integration of expertise ensures that the explanations gener-
expressing decision rules in the form of “if-then” state- ated by XAI systems are not only technically sound but also
ments. These models can be used in various domains to relevant and meaningful in the specific healthcare context.
generate explanations that are easily understandable by Similarly, in the domain of cybersecurity, collaboration
humans. In legal applications, rule-based models can between cybersecurity experts, AI specialists, and legal pro-
help explain legal reasoning by mapping legal principles fessionals can lead to the development of XAI techniques
and regulations to decision rules. that address the unique challenges of cybersecurity threats.
By combining knowledge from these different fields, XAI
These are just a few examples of common XAI techniques systems can provide interpretable explanations that enhance
that can be applied across different domains. The choice of the understanding of AI-based security measures, assist in
technique depends on the specific requirements and charac- identifying vulnerabilities, and facilitate decision-making
teristics of each domain. We summarise some typeical suit- processes for cybersecurity professionals.
able XAI approaches for each domain shown in Table 5. By The collaboration and integration of expertise from differ-
leveraging these techniques, domain experts and practition- ent fields also foster a cross-pollination of ideas and perspec-
ers can gain insights into the inner workings of AI models tives, driving innovation and the development of novel XAI
and make informed decisions based on understandable and techniques. By leveraging the diverse knowledge and experi-
interpretable explanations. ences of experts from various domains, XAI can evolve and
13
Table 5 XAI suitability analysis for application domains

Domain Typical approaches Explanation importance
Medical and biomedical Attention-based explanations It is suitable for medical image analysis tasks, such as identifying regions of interest
or explaining the predictions of deep learning models by highlighting the important
areas in an image that contribute to the model’s decision
Model-Agnostic explanations It is suitable for medical applications where feature importance and individual instance
explanations are required. It quantifies the contribution of each feature to a model’s
prediction and provides explanations at the individual patient level
Rule extraction It is suitable for medical domains where interpretable decision rules are desired, which
is used to generate human-readable rules that align with medical guidelines, making
it suitable for decision support systems and improving trust in AI-driven medical
applications
Healthcare Counterfactual explanations It is suitable for healthcare applications where personalized treatment recommenda-
tions or interventions are required which can enable personalized care plans and
enhancing patient engagement
Attention-based explanations It is suitable to analyse patient records, clinical notes, or time-series data. They enable
the model to dynamically attend to important features, leading to improve interpret-
ability and the identification of critical factors influencing healthcare outcomes
Case-based reasoning It is suitable to evolving knowledge, integrates expert knowledge, supports decision-
making, and facilitates learning from past experiences. By leveraging historical data
and real-world examples, it supports healthcare professionals in decision-making,
diagnosis, treatment planning, and improving patient outcomes
Cyber security Model-Agnostic explanations It is suitable to identify the key factors or variables that contribute to the likelihood of
a security breach or attack. By understanding the importance of different features,
security analysts can prioritize their efforts and focus on mitigating the most critical
vulnerabilities
Graph-based explanations It is suitable to aid in detecting complex cyber threats and visualizing attack relation-
ships. It enables the identification of patterns, anomalies, and influential factors,
providing explanations that enhance situational awareness, threat detection, and
decision-making in cyber security operations
Finance Model-Agnostic explanations It is suitable to interpret complex deep learning models used for risk assessment, fraud
detection, or portfolio management. It provides individual feature importance scores
that help understand the factors contributing to predictions, enabling better decision-
making and risk management
Rule extraction It is suitable for finance to help identify specific conditions or criteria that drive finan-
cial outcomes or decisions, such as loan approvals or investment recommendations
Law Case-based reasoning It is suitable in the legal domain as it allows for the retrieval and adaptation of past
legal cases to support current decision-making, which can enhance legal decision-
making by providing context-specific explanations based on past legal experiences
Rule extraction It is suitable in law to uncover the underlying rules and criteria used by legal systems.
These techniques can help in understanding the decision-making process of legal
systems and provide explanations for legal outcomes
Education and training Model-Agnostic explanations It is suitable to identify the key features or factors that contribute to student perfor-
mance, engagement, or learning outcomes by providing localized explanations that
highlight the importance and influence of specific features, such as demographic
information, learning activities, or socio-economic factors, on educational outcomes
Rule extraction It is suitable to explain the reasoning behind educational decisions or outcomes by
explicitly stating the conditions and criteria that influence educational decisions,
such as grading rubrics or admission criteria
Interactive explanations It is suitable to present complex educational data in an understandable manner, facili-
tating a deeper understanding of student performance, engagement, and progress
13
Table 5 (continued)
Domain Typical approaches Explanation importance
Civil engineering Attention-based explanations It is suitable for image-based tasks such as defect detection, structural damage
assessment, or material characterization. This can aid in the interpretation of model
outputs, improve trust in the predictions, and guide subsequent inspection or repair
actions
Model-Agnostic explanations It is suitable to explain the factors influencing the structural integrity or performance
of a building or infrastructure. This information can guide engineers in making
informed decisions regarding maintenance, repairs, or design modifications
Rule extraction It is suitable for models used for tasks such as traffic signal control, behaviour analysis,
or demand forecasting. By extracting interpretable rules, such as IF-THEN state-
ments transportation planners and policymakers can gain insights into the decision-
making process of the models
adapt to meet the evolving needs and challenges of different Interactive explanation: in the current landscape of XAI
industries and societal contexts. research, there is recognition that a single explanation may
not be sufficient to address all user concerns and questions in
decision-making scenarios. As a result, the focus has shifted
4 Discussion towards developing interactive explanations that allow for a
dynamic and iterative process. However, there are challenges
As the concerns on explainability and the attentions for XAI, that need to be addressed in order to effectively implement
regulations such as GDPR set out the transparency rules interactive explanation systems. One of the key challenges is
about the data processing. As most modern AI systems are the ability to handle a wide range of user queries and adapt
data-driven AI, these requirements are actually applicable to the explanations accordingly. Users may have diverse infor-
all application domains. Not only the explainability is neces- mation needs and may require explanations that go beyond
sary, but also the way of explaining is required. superficial or generic responses. In particular, addressing
In this section, we will summarize the limitations of queries that involve deep domain knowledge or intricate rea-
existing XAI approaches based on the above review in each soning processes can be complex and requires sophisticated
application domain, and identify future research directions. techniques. Another challenge is striking a balance between
providing timely responses to user queries and maintaining
4.1 Limitations computational efficiency. Interactive explanation systems
need to respond quickly to user interactions to facilitate a
Adaptive integration and explanation: many existing smooth and engaging user experience. However, generat-
approaches provide explanations in a generic manner, with- ing accurate and informative explanations within a short
out considering the diverse backgrounds (culture, context, response time can be demanding, and trade-offs may need
etc.) and knowledge levels of users. This one-size-fits-all to be made depending on the specific domain and computa-
approach can lead to challenges in effective comprehension tional resources available. Moreover, the design and imple-
for both novice and expert users. Novice users may strug- mentation of interactive explanation systems should also
gle to understand complex technical explanations, while consider the context and domain-specific requirements. Dif-
expert users may find oversimplified explanations lacking ferent domains may have unique challenges and constraints
in depth. These limitations hinder the ability of XAI tech- that need to be taken into account when developing interac-
niques to cater to users with different levels of expertise and tive explanations. It is important to ensure that the interac-
may impact the overall trust and usability of the system. tive explanation systems are tailored to the specific domain
Furthermore, the evaluation and assessment of XAI tech- and can effectively address the needs of users in that context.
niques often prioritize objective metrics, such as fidelity or Connection and consistency in hybrid explanation: in
faithfulness, which measure how well the explanations align the context of hybrid explanations in XAI, it is crucial to
with the model’s internal workings. While these metrics are ensure connection and consistency among different sources
important for evaluating the accuracy of the explanations, of explanations. Hybrid approaches aim to leverage multiple
they may not capture the subjective aspects of user under- techniques to provide users in various domains with different
standing and interpretation. The perceived quality of expla- application purposes, achieving robustness and interpretabil-
nations can vary among users with different expertise levels, ity. However, it is necessary to address potential conflicts
as well as under different situations or conditions. and ensure coordinated integration of different components
13
within these hybrid systems. Currently, many works focus AI models or data evolve, the explanations may become
on combining various explanation techniques to complement outdated or less accurate. To ensure relevance and use-
each other and enhance overall system performance. While fulness, XAI systems should be designed to incorporate
this integration is valuable, it is important to acknowledge mechanisms for updating explanations to reflect the latest
that different techniques may have inherent differences in model updates or data changes. Another limitation is the
their assumptions, methodologies, and outputs. These dif- assumption of stationary data distributions. XAI methods
ferences can result in conflicts or inconsistencies when com- are typically trained on historical data, assuming that the
bined within a hybrid explanation system. Therefore, careful future data will follow a similar distribution. However, if
attention should be given to the design of complex hybrid the data distribution changes over time, the performance of
explanation systems. The structure and architecture need the XAI system may deteriorate. Adapting XAI methods to
to be thoughtfully planned to ensure seamless connections handle shifting data distributions is essential for maintain-
between components. This involves identifying potential ing their effectiveness and ensuring reliable explanations
conflicts early on and developing strategies to resolve them. in dynamic environments. Scalability is another crucial
Additionally, efforts should be made to establish a unified consideration, particularly for large-scale AI systems. XAI
framework that allows for effective coordination and integra- techniques that work well on small-scale or controlled
tion of the different techniques used in the hybrid system. datasets may face challenges when applied to large-scale
Furthermore, the evaluation and validation of hybrid expla- AI systems with complex models and massive amounts
nation systems should include assessing the consistency of of data. Efficient algorithms and sufficient computational
explanations provided by different sources. This evaluation resources are necessary to handle the increased computa-
process helps identify any discrepancies or inconsistencies tional demands of explaining large-scale AI systems with-
and guides the refinement of the system to ensure a coherent out sacrificing performance or usability.
and unified user experience.
Balancing model interpretability with predictive
accuracy: currently, researchers are developing hybrid 4.2 Future Directions
approaches that aim to strike a better balance between inter-
pretability and accuracy, such as using post-hoc interpret- To address the first limitation, building the context-aware-
ability techniques with complex models or designing new ness XAI is important, we need to explore how to generate
model architectures that inherently provide both interpret- explanations by considering mission contexts (surrounding
ability and high accuracy.However, they also come with their environment, situations, time-series datasets.), mapping user
own limitations. Post-hoc interpretability techniques gener- roles (end-user, domain expert, business manager, AI devel-
ate explanations after the model has made its predictions, oper, etc.) and targeted goals (refine the model, debugging
which means they do not directly influence the model’s system errors, detecting bias, understand AI learning pro-
decision-making process. As a result, the explanations may cess, etc.) regardless of the type of AI system. So far, most
not capture the full complexity and nuances of the model’s of these studies were still conceptual with limited considera-
internal workings. Furthermore, post-hoc techniques can be tion, the more general context-driven systems and practical
computationally expensive and may not scale well to large implementations will be an important direction for future
datasets or complex models with high-dimensional inputs. research.
Designing new model architectures such as rule-based mod- Secondly, interactive explanations (e.g., conversation
els or attention mechanisms in neural networks may struggle system Interfaces, games, using audio, visuals, video, etc.)
to capture complex interactions and may require a significant should be explored further. This is a promising approach to
amount of manual rule engineering. It is crucial to recog- building truly human-centred explanations by identifying
nize that there is no universal solution to the interpretability- users’ requirements and providing better human-AI collabo-
accuracy trade-off. The choice of approach depends on the ration. These incorporating theories and frameworks allow
specific requirements of the application, available resources, an iterative process from humans, which is a crucial aspect
and acceptable trade-offs in the given context. Researchers of building successful XAI systems.
and practitioners must carefully consider the limitations and Finally, the hybrid explanation should be applied by
benefits of different techniques to strike an appropriate bal- concerning fusing heterogeneous knowledge from different
ance based on their specific use cases. sources, managing time-sensitive data, inconsistency, uncer-
Long-term usability and maintainability: the current tainty, etc. Among these conditions, hybrid explanation has
XAI methods face several limitations when deployed in been an interesting and increasing topic in recent years. This
real-world scenarios. One significant limitation is the need will also involve a wide range of criteria and strategies that
for continuous explanation updates. XAI systems generate target a clear structure and consensus on what constitutes
explanations based on training data, and as the underlying success and trustworthy explanations.
13
5 Conclusion References
This paper addresses a wide range of explainable AI top- 1. Zhang Y, Tiňo P, Leonardis A, Tang K. A survey on neural
network interpretability. IEEE Trans Emerg Top Comput Intell.
ics. XAI is a rapidly growing field of research, as it fills 2021;20:20.
a gap in current AI approaches, allowing people to better 2. Tomsett R, Preece A, Braines D, Cerutti F, Chakraborty S,
understand AI models and therefore trust their outputs. Srivastava M, Pearson G, Kaplan L. Rapid trust calibration
By summarising the current literature, we have proposed through interpretable and uncertainty-aware AI. Patterns.
2020;1(4):100049.
a new taxonomy for XAI from the human perspective. 3. Arrieta AB, Díaz-Rodríguez N, Del Ser J, Bennetot A, Tabik
The taxonomy considers source-oriented, representation- S, Barbado A, García S, Gil-López S, Molina D, Benjamins
oriented aspect, and logic-oriented perspectives. R, et al. Explainable artificial intelligence (XAI): concepts,
It is very important that we have elaborated on the taxonomies, opportunities and challenges toward responsible
AI. Inf Fusion. 2020;58:82–115.
applications of XAI in multiple areas in the paper, includ- 4. Regulation GDP. General data protection regulation (GDPR).
ing medical, healthcare, cybersecurity, finance and law, Intersoft Consult. 2018;24:1.
education and training, and civil engineering. We provide 5. Bostrom N, Yudkowsky E. The ethics of artificial intelligence.
a comprehensive review of different XAI approaches and In: Artificial intelligence safety and security. New York: Chap-
man and Hall; 2018. p. 57–69.
identify the key techniques for case studies. Finally, we 6. Weld DS, Bansal G. The challenge of crafting intelligible intel-
discuss the limitations of existing XAI methods and pre- ligence. Commun ACM. 2019;62(6):70–9.
sent several corresponding areas for further research: (1) 7. Das A, Rad P. Opportunities and challenges in explainable arti-
context-awareness XAI, (2) interactive explanations, and ficial intelligence (XAI): a survey. arXiv:2006.11371 (arXiv
preprint) (2020).
(3) hybrid explanations. 8. Challen R, Denny J, Pitt M, Gompels L, Edwards T, Tsaneva-
Overall, this paper provides a clear survey of the current Atanasova K. Artificial intelligence, bias and clinical safety.
XAI research and application status from the human per- BMJ Qual Saf. 2019;28(3):231–7.
spective. We hope this article will provide a valuable ref- 9. Patil MS, Främling K. Context, utility and influence of an
explanation. arXiv:2303.13552 (arXiv preprint); 2023.
erence for XAI-related researchers and practitioners. We 10. Ooge J, Verbert K. Explaining artificial intelligence with tai-
believe XAI will build a bridge of trust between humans lored interactive visualisations. In: 27th international confer-
and AI. ence on intelligent user interfaces; 2022. p. 120–3.
11. Saeed W, Omlin C. Explainable AI (XAI): a systematic meta-
Acknowledgements This work was supported by the Institute of survey of current challenges and future opportunities. Knowl
Information and Communications Technology Planning and Evalu- Based Syst. 2023;11:0273.
ation (IITP) grant funded by the Korean Government (MSIT) (2022- 12. Förster M, Klier M, Kluge K, Sigler I. Fostering human
0-00078, Explainable Logical Reasoning for Medical Knowledge agency: a process for the design of user-centric XAI systems;
Generation). 2020.
13. Kotriwala A, Klöpper B, Dix M, Gopalakrishnan G, Ziobro
Author Contributions The authors confirm their contribution to the D, Potschka A. Xai for operations in the process industry-
paper as follows: study conception and design: WY; draft manuscript applications, theses, and research directions. In: AAAI spring
preparation: YW, HW, YC, GH, XL, RL, NY, XW, and XG. Supervi- symposium: combining machine learning with knowledge
sion: MBA and BK. All authors reviewed the results and approved the engineering; 2021.
final version of the manuscript. 14. Albahri A, Duhaim AM, Fadhel MA, Alnoor A, Baqer NS,
Alzubaidi L, Albahri O, Alamoodi A, Bai J, Salhi A, et al.
Data availability Not applicable. A systematic review of trustworthy and explainable artificial
intelligence in healthcare: assessment of quality, bias risk, and
Declarations data fusion. Inf Fusion. 2023;20:20.
15. Kurshan E, Chen J, Storchan V, Shen H. On the current and
Conflict of interest The authors declare that they have no known com- emerging challenges of developing fair and ethical AI solu-
peting financial interests or personal relationships that could have ap- tions in financial services. In: Proceedings of the second ACM
peared to influence the work reported in this paper. international conference on AI in finance; 2021. p. 1–8.
16. Komorowski P, Baniecki H, Biecek P. Towards evaluating
Open Access This article is licensed under a Creative Commons Attri- explanations of vision transformers for medical imaging. In:
bution 4.0 International License, which permits use, sharing, adapta- Proceedings of the IEEE/CVF conference on computer vision
tion, distribution and reproduction in any medium or format, as long and pattern recognition; 2023. p. 3725–3731.
as you give appropriate credit to the original author(s) and the source, 17. Adadi A, Berrada M. Peeking inside the black-box: a survey
provide a link to the Creative Commons licence, and indicate if changes on explainable artificial intelligence (XAI). IEEE Access.
were made. The images or other third party material in this article are 2018;6:52138–60.
included in the article's Creative Commons licence, unless indicated 18. Minh D, Wang HX, Li YF, Nguyen TN. Explainable artifi-
otherwise in a credit line to the material. If material is not included in cial intelligence: a comprehensive review. Artif Intell Rev.
the article's Creative Commons licence and your intended use is not 2021;20:1–66.
permitted by statutory regulation or exceeds the permitted use, you will 19. Chaddad A, Peng J, Xu J, Bouridane A. Survey of explainable
need to obtain permission directly from the copyright holder. To view a AI techniques in healthcare. Sensors. 2023;23(2):634.
copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
13
20. Tjoa E, Guan C. A survey on explainable artificial intelligence 41. Ribeiro MT, Singh S, Guestrin C. Anchors: High-precision
(XAI): toward medical XAI. IEEE Trans Neural Netw Learn model-agnostic explanations. In: Proceedings of the AAAI
Syst. 2020;32(11):4793–813. conference on artificial intelligence; 2018. vol. 32.
21. Angelov PP, Soares EA, Jiang R, Arnold NI, Atkinson PM. 42. Teng F, Yang W, Chen L, Huang L, Xu Q. Explainable predic-
Explainable artificial intelligence: an analytical review. Wiley tion of medical codes with knowledge graphs. Front Bioeng
Interdiscip Rev Data Min Knowl Discov. 2021;11(5):1424. Biotechnol. 2020;8:867.
22. Vilone G, Longo L. Classification of explainable artificial 43. Sun P, Gu L. Fuzzy knowledge graph system for artificial
intelligence methods through their output formats. Mach Learn intelligence-based smart education. J Intell Fuzzy Syst.
Knowl Extract. 2021;3(3):615–61. 2021;40(2):2929–40.
23. Alain G, Bengio Y. Understanding intermediate layers using 44. Panchenko A, Ruppert E, Faralli S, Ponzetto SP, Biemann
linear classifier probes. arXiv:1610.01644 (arXiv preprint); C. Unsupervised does not mean uninterpretable: the case for
2016. word sense induction and disambiguatio; 2017. Association for
24. Zhang Q, Cao R, Shi F, Wu YN, Zhu S-C. Interpreting CNN Computational Linguistics.
knowledge via an explanatory graph. In: Proceedings of the 45. Bennetot A, Laurent J-L, Chatila R, Díaz-Rodríguez N.
AAAI conference on artificial intelligence, vol. 32; 2018. Towards explainable neural-symbolic visual reasoning. arXiv:
25. Hendricks LA, Hu R, Darrell T, Akata Z. Grounding visual 1909.09065 (arXiv preprint); 2019.
explanations. In: Proceedings of the European conference on 46. Tamagnini P, Krause J, Dasgupta A, Bertini E. Interpreting
computer vision (ECCV); 2018. p. 264–79. black-box classifiers using instance-level visual explanations.
26. Bondarenko A, Aleksejeva L, Jumutc V, Borisov A. Classifica- In: Proceedings of the 2nd workshop on human-in-the-loop
tion tree extraction from trained artificial neural networks. Proced data analytics; 2017. p. 1–6.
Comput Sci. 2017;104:556–63. 47. Spinner T, Schlegel U, Schäfer H, El-Assady M. Explainer:
27. Zhou Z-H, Jiang Y, Chen S-F. Extracting symbolic rules a visual analytics framework for interactive and explain-
from trained neural network ensembles. AI Commun. able machine learning. IEEE Trans Visual Comput Graph.
2003;16(1):3–15. 2019;26(1):1064–74.
28. Barakat N, Diederich J. Eclectic rule-extraction from support 48. Hendricks LA, Akata Z, Rohrbach M, Donahue J, Schiele B,
vector machines. Int J Comput Intell. 2005;2(1):59–62. Darrell T. Generating visual explanations. In: European confer-
29. Nikolov A, d’Aquin M. Uncovering semantic bias in neural net- ence on computer vision. Springer; 2016. p. 3–19.
work models using a knowledge graph. In: Proceedings of the 49. Kim J, Rohrbach A, Darrell T, Canny J, Akata Z. Textual
29th ACM international conference on information and knowl- explanations for self-driving vehicles. In: Proceedings of the
edge management; 2020, p. 1175–84. European conference on computer vision (ECCV); 2018. p.
30. Riquelme F, De Goyeneche A, Zhang Y, Niebles JC, Soto A. 563–78.
Explaining VQA predictions using visual grounding and a 50. Park DH, Hendricks LA, Akata Z, Rohrbach A, Schiele B,
knowledge base. Image Vis Comput. 2020;101:103968. Darrell T, Rohrbach M. Multimodal explanations: justifying
31. Erion G, Janizek JD, Sturmfels P, Lundberg SM, Lee S-I. Learn- decisions and pointing to the evidence. In: Proceedings of the
ing explainable models using attribution priors; 2019. IEEE conference on computer vision and pattern recognition;
32. Robnik-Šikonja M, Bohanec M. Perturbation-based explanations 2018. p. 8779–8788.
of prediction models. In: Human and machine learning. Berlin: 51. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R,
Springer; 2018. p. 159–75. Zemel R, Bengio Y. Show, attend and tell: neural image cap-
33. Laugel T, Lesot M-J, Marsala C, Renard X, Detyniecki M. The tion generation with visual attention. In: International confer-
dangers of post-hoc interpretability: Unjustified counterfactual ence on machine learning; 2015. p. 2048–2057.
explanations. arXiv:1907.09294 (arXiv preprint); 2019. 52. Gu D, Li Y, Jiang F, Wen Z, Liu S, Shi W, Lu G, Zhou C.
34. Chefer H, Gur S, Wolf L. Transformer interpretability beyond Vinet: a visually interpretable image diagnosis network. IEEE
attention visualization. In: Proceedings of the IEEE/CVF con- Trans Multimed. 2020;22(7):1720–9.
ference on computer vision and pattern recognition; 2021. p. 53. Slack D, Hilgard S, Jia E, Singh S, Lakkaraju H. Fooling lime
782–91. and shap: adversarial attacks on post hoc explanation methods.
35. Jalaboi R, Faye F, Orbes-Arteaga M, Jørgensen D, Winther O, In: Proceedings of the AAAI/ACM conference on AI, ethics,
Galimzianova A. Dermx: an end-to-end framework for explain- and society; 2020. p. 180–86.
able automated dermatological diagnosis. Med Image Anal. 54. Zhang Z, Rudra K, Anand A. Explain and predict, and then
2023;83:102647. predict again. In: Proceedings of the 14th ACM international
36. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra conference on web search and data mining; 2021. p. 418–26.
D. Grad-cam: visual explanations from deep networks via gradi- 55. Montavon G, Binder A, Lapuschkin S, Samek W, Müller K-R.
ent-based localization. In: Proceedings of the IEEE international Layer-wise relevance propagation: an overview. Explainable
conference on computer vision; 2017. p. 618–26. AI: interpreting, explaining and visualizing deep learning;
37. Graziani M, Andrearczyk V, Marchand-Maillet S, Müller H. 2019. 193–209.
Concept attribution: explaining CNN decisions to physicians. 56. Zhang Z, Chen P, McGough M, Xing F, Wang C, Bui M, Xie
Comput Biol Med. 2020;123:103865. Y, Sapkota M, Cui L, Dhillon J, et al. Pathologist-level inter-
38. Zhang Q, Wu YN, Zhu S-C. Interpretable convolutional neural pretable whole-slide cancer diagnosis with deep learning. Nat
networks. In: Proceedings of the IEEE conference on computer Mach Intell. 2019;1(5):236–45.
vision and pattern recognition; 2018. p. 8827–836. 57. Sarlin P-E, DeTone D, Malisiewicz T, Rabinovich A. Super-
39. Liang X, Hu Z, Zhang H, Lin L, Xing EP. Symbolic graph glue: learning feature matching with graph neural networks. In:
reasoning meets convolutions. Adv Neural Inf Process Syst. Proceedings of the IEEE/CVF conference on computer vision
2018;31:25. and pattern recognition; 2020. p. 4938–47.
40. Li CY, Liang X, Hu Z, Xing EP. Knowledge-driven encode, 58. Shen S, Han SX, Aberle DR, Bui AA, Hsu W. Explainable
retrieve, paraphrase for medical image report generation. In: hierarchical semantic convolutional neural network for lung
Proceedings of the AAAI conference on artificial intelligence; cancer diagnosis. In: CVPR workshops; 2019. p. 63–6.
2019. vol. 33, p. 6666–73.
13
59. Gozzi N, Malandri L, Mercorio F, Pedrocchi A. Xai for myo- 77. Yang S, Niu J, Wu J, Liu X. Automatic medical image report gen-
controlled prosthesis: explaining EMG data for hand gesture eration with multi-view and multi-modal attention mechanism.
classification. Knowl-Based Syst. 2022;240:108053. In: International conference on algorithms and architectures for
60. Simonyan K, Vedaldi A, Zisserman A. Deep inside convolutional parallel processing. Springer; 2020. p. 687–99.
networks: visualising image classification models and saliency 78. Barnett AJ, Schwartz FR, Tao C, Chen C, Ren Y, Lo JY, Rudin
maps. arXiv:1312.6034 (arXiv preprint); 2013. C. A case-based interpretable deep learning model for classifica-
61. Zeiler MD, Fergus R. Visualizing and understanding convolu- tion of mass lesions in digital mammography. Nat Mach Intell.
tional networks. In: European conference on computer vision. 2021;3(12):1061–70.
Springer; 2014. p. 818–33. 79. Saleem H, Shahid AR, Raza B. Visual interpretability in
62. Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M. Striving 3d brain tumor segmentation network. Comput Biol Med.
for simplicity: the all convolutional net. arXiv:1412.6806 (arXiv 2021;133:104410.
preprint); 2014. 80. Wang S, Yin Y, Wang D, Wang Y, Jin Y. Interpretability-based
63. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning multimodal convolutional neural networks for skin lesion diag-
deep features for discriminative localization. In: Proceedings of nosis. IEEE Trans Cybern. 2021;20:20.
the IEEE conference on computer vision and pattern recognition; 81. Ahmed U, Jhaveri RH, Srivastava G, Lin JC-W. Explainable deep
2016. p. 2921–9. attention active learning for sentimental analytics of mental dis-
64. Olah C, Mordvintsev A, Schubert L. Feature visualization. Dis- order. Trans Asian Low-Resour Lang Inf Proces. 2022;20:22.
till. 2017;2(11):7. 82. Lu Y, Perer A. An interactive interpretability system for breast
65. Zhang Z, Xie Y, Xing F, McGough M, Yang L. Mdnet: a semanti- cancer screening with deep learning. arXiv:2210.08979 (arXiv
cally and visually interpretable medical image diagnosis network. preprint); 2022.
In: Proceedings of the IEEE conference on computer vision and 83. Figueroa KC, Song B, Sunny S, Li S, Gurushanth K, Men-
pattern recognition; 2017. p. 6428–36. donca P, Mukhia N, Patrick S, Gurudath S, Raghavan S, et al.
66. Kim B, Wattenberg M, Gilmer J, Cai C, Wexler J, Viegas F, et al. Interpretable deep learning approach for oral cancer classifica-
Interpretability beyond feature attribution: quantitative testing tion using guided attention inference network. J Biomed Opt.
with concept activation vectors (TCAV). In: International confer- 2022;27(1):015001.
ence on machine learning; 2018. p. 2668–77. 84. Hicks SA, Eskeland S, Lux M, de Lange T, Randel KR, Jeppsson
67. Wu B, Zhou Z, Wang J, Wang Y. Joint learning for pulmonary M, Pogorelov K, Halvorsen P, Riegler M. Mimir: an automatic
nodule segmentation, attributes and malignancy prediction. In: reporting and reasoning system for deep learning based analysis
2018 IEEE 15th international symposium on biomedical imaging in the medical domain. In: Proceedings of the 9th ACM multi-
(ISBI 2018); 2018. p. 1109–13. IEEE. media systems conference; 2018. p. 369–74.
68. de Vos BD, Wolterink JM, Leiner T, de Jong PA, Lessmann N, 85. Holzinger A, Malle B, Saranti A, Pfeifer B. Towards multi-modal
Išgum I. Direct automatic coronary calcium scoring in cardiac causability with graph neural networks enabling information
and chest CT. IEEE Trans Med Imaging. 2019;38(9):2127–38. fusion for explainable AI. Inf Fusion. 2021;71:28–37.
69. Eitel F, Ritter K, ADNI. Testing the robustness of attribution 86. Palatnik de Sousa I, Maria Bernardes Rebuzzi Vellasco M,
methods for convolutional neural networks in MRI-based Alz- Costa da Silva E. Local interpretable model-agnostic expla-
heimer’s disease classification. In: Interpretability of machine nations for classification of lymph node metastases. Sensors.
intelligence in medical image computing and multimodal learn- 2019;19(13):2969.
ing for clinical decision support. Berlin: Springer; 2019. p. 3–11. 87. Zhu P, Ogino M. Guideline-based additive explanation for
70. Clough JR, Oksuz I, Puyol-Antón E, Ruijsink B, King AP, Schna- computer-aided diagnosis of lung nodules. In: Interpretability
bel J.A. Global and local interpretability for cardiac MRI classifi- of machine intelligence in medical image computing and multi-
cation. In: International conference on medical image computing modal learning for clinical decision support. Springer; 2019; p.
and computer-assisted intervention; Springer. 2019. p. 656–4. 39–47.
71. Gasimova A. Automated enriched medical concept generation for 88. Paschali M, Ferjadnaeem M, Simson W, et al. Improving the
chest X-ray images. In: Interpretability of machine intelligence in interpretability of medical imaging neural networks. In: Com-
medical image computing and multimodal learning for clinical puter vision and pattern recognition; 2019.
decision support. Springer; 2019. p. 83–92. 89. Liao W, Zou B, Zhao R, Chen Y, He Z, Zhou M. Clinical inter-
72. Kim ST, Lee J-H, Ro YM. Visual evidence for interpreting diag- pretable deep learning model for glaucoma diagnosis. IEEE J
nostic decision of deep neural network in computer-aided diagno- Biomed Health Inform. 2019;24(5):1405–12.
sis. In: Medical imaging 2019: computer-aided diagnosis. 2019; 90. Lee H, Yune S, Mansouri M, Kim M, Tajmir SH, Guerrier CE,
vol. 10950, p. 139–47. SPIE. Ebert SA, Pomerantz SR, Romero JM, Kamalian S, et al. An
73. Lee H, Kim ST, Ro YM. Generation of multimodal justification explainable deep-learning algorithm for the detection of acute
using visual word constraint model for explainable computer- intracranial haemorrhage from small datasets. Nat Biomed Eng.
aided diagnosis. In: Interpretability of machine intelligence in 2019;3(3):173–82.
medical image computing and multimodal learning for clinical 91. Biffi C, Oktay O, Tarroni G, Bai W, De Marvao A, Doumou
decision support. Springer; 2019. p. 21–9. G, Rajchl M, Bedair R, Prasad S, Cook S, et al. Learning inter-
74. Shen S, Han SX, Aberle DR, Bui AA, Hsu W. An interpret- pretable anatomical features through deep generative models:
able deep hierarchical semantic convolutional neural network application to cardiac remodeling. In: International conference
for lung nodule malignancy classification. Expert Syst Appl. on medical image computing and computer-assisted intervention.
2019;128:84–95. Springer; 2018. p. 464–71.
75. Arun N, Gaw N, Singh P, Chang K, Aggarwal M, Chen B, et al. 92. Garcia-Peraza-Herrera LC, Everson M, Li W, Luengo I, Berger
Assessing the (un) trustworthiness of saliency maps for local- L, Ahmad O, Lovat L, Wang H-P, Wang W-L, Haidry R, et al.
izing abnormalities in medical imaging (arXiv preprint); 2020. Interpretable fully convolutional classification of intrapapillary
76. Zeng X, Wen L, Xu Y, Ji C. Generating diagnostic report for capillary loops for real-time detection of early squamous neopla-
medical image by high-middle-level visual information incor- sia. arXiv:1805.00632 (arXiv preprint); 2018.
poration on double deep learning models. Comput Methods Pro- 93. Amoroso N, Pomarico D, Fanizzi A, Didonna V, Giotta F, La
grams Biomed. 2020;197:105700. Forgia D, Latorre A, Monaco A, Pantaleo E, Petruzzellis N, et al.
13
A roadmap towards breast cancer therapies supported by explain- 111. Farhadloo M, Molnar C, Luo G, Li Y, Shekhar S, Maus RL,
able artificial intelligence. Appl Sci. 2021;11(11):4881. Markovic S, Leontovich A, Moore R. Samcnet: towards a spa-
94. Sarp S, Kuzlu M, Wilson E, Cali U, Guler O. The enlightening tially explainable AI approach for classifying MXIF oncology
role of explainable artificial intelligence in chronic wound clas- data. In: Proceedings of the 28th ACM SIGKDD conference
sification. Electronics. 2021;10(12):1406. on knowledge discovery and data mining; 2022. p. 2860–70.
95. Wu H, Chen W, Xu S, Xu B. Counterfactual supporting facts 112. Jiang Z, Bo L, Xu Z, Song Y, Wang J, Wen P, Wan X, Yang T,
extraction for explainable medical record based diagnosis with Deng X, Bian J. An explainable machine learning algorithm for
graph network. In: Proceedings of the 2021 conference of the risk factor analysis of in-hospital mortality in sepsis survivors
north American chapter of the association for computational with ICU readmission. Comput Methods Programs Biomed.
linguistics: human language technologies; 2021. p. 1942–55. 2021;204:106040.
96. Wang X, Peng Y, Lu L, Lu Z, Summers RM. Tienet: Text-image 113. Liu H, Zhong C, Alnusair A, Islam SR. Faixid: a frame-
embedding network for common thorax disease classification and work for enhancing AI explainability of intrusion detection
reporting in chest X-rays. In: Proceedings of the IEEE conference results using data cleaning techniques. J Netw Syst Manage.
on computer vision and pattern recognition; 2018. p. 9049–58. 2021;29(4):1–30.
97. Lucieri A, Bajwa MN, Braun SA, Malik MI, Dengel A, Ahmed 114. Amarasinghe K, Manic M. Improving user trust on deep neural
S. On interpretability of deep learning based skin lesion classi- networks based intrusion detection systems. In: IECON 2018-
fiers using concept activation vectors. In: 2020 international joint 44th annual conference of the IEEE Industrial electronics soci-
conference on neural networks (IJCNN); 2020. p. 1–10. IEEE. ety; 2018. p. 3262–68. IEEE.
98. Senior AW, Evans R, Jumper J, Kirkpatrick J, Sifre L, Green 115. Amarasinghe K, Kenney K, Manic M. Toward explainable deep
T, Qin C, Žídek A, Nelson AW, Bridgland A, et al. Improved neural network based anomaly detection. In: 2018 11th inter-
protein structure prediction using potentials from deep learning. national conference on human system interaction (HSI); 2018.
Nature. 2020;577(7792):706–10. IEEE. p. 311–7.
99. Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger 116. Chen S, Bateni S, Grandhi S, Li X, Liu C, Yang W. Denas:
O, Tunyasuvunakool K, Bates R, Žídek A, Potapenko A, et al. automated rule generation by knowledge extraction from neural
Highly accurate protein structure prediction with alphafold. networks. In: Proceedings of the 28th ACM joint meeting on
Nature. 2021;596(7873):583–9. European software engineering conference and symposium on
100. Merk D, Friedrich L, Grisoni F, Schneider G. De novo design the foundations of software engineering, 2020; p. 813–25.
of bioactive small molecules by artificial intelligence. Mol Inf. 117. Gulmezoglu B. Xai-based microarchitectural side-channel analy-
2018;37(1–2):1700153. sis for website fingerprinting attacks and defenses. IEEE Trans
101. Zhavoronkov A, Ivanenkov YA, Aliper A, Veselov MS, Ala- Depend Sec Comput. 2021;20:10.
dinskiy VA, Aladinskaya AV, Terentiev VA, Polykovskiy DA, 118. Feichtner J, Gruber S. Understanding privacy awareness in
Kuznetsov MD, Asadulaev A, et al. Deep learning enables rapid android app descriptions using deep learning. In: Proceedings
identification of potent ddr1 kinase inhibitors. Nat Biotechnol. of the tenth ACM conference on data and application security
2019;37(9):1038–40. and privacy; 2020. p. 203–14.
102. Karimi M, Wu D, Wang Z, Shen Y. Explainable deep relational 119. Iadarola G, Martinelli F, Mercaldo F, Santone A. Towards an
networks for predicting compound-protein affinities and contacts. interpretable deep learning model for mobile malware detection
J Chem Inf Model. 2020;61(1):46–66. and family identification. Comput Secur. 2021;105:102198.
103. Ezzat D, Hassanien AE, Ella HA. An optimized deep learn- 120. Guo W, Mu D, Xu J, Su P, Wang G, Xing X. Lemna: explaining
ing architecture for the diagnosis of covid-19 disease based deep learning based security applications. In: Proceedings of the
on gravitational search optimization. Appl Soft Comput. 2018 ACM SIGSAC conference on computer and communica-
2021;98:106742. tions security; 2018. p. 364–79.
104. Segler MH, Kogej T, Tyrchan C, Waller MP. Generating focused 121. Yan A, Chen Z, Zhang H, Peng L, Yan Q, Hassan MU, Zhao
molecule libraries for drug discovery with recurrent neural net- C, Yang B. Effective detection of mobile malware behavior
works. ACS Cent Sci. 2018;4(1):120–31. based on explainable deep neural network. Neurocomputing.
105. Preuer K, Renz P, Unterthiner T, Hochreiter S, Klambauer G. Fré- 2021;453:482–92.
chet chemnet distance: a metric for generative models for mole- 122. Bach S, Binder A, Montavon G, Klauschen F, Müller K-R,
cules in drug discovery. J Chem Inf Model. 2018;58(9):1736–41. Samek W. On pixel-wise explanations for non-linear classi-
106. Wan Y, Zhou H, Zhang X. An interpretation architecture for fier decisions by layer-wise relevance propagation. PLoS One.
deep learning models with the application of covid-19 diagnosis. 2015;10(7):0130140.
Entropy. 2021;23(2):204. 123. Mane S, Rao D. Explaining network intrusion detection system
107. Loh HW, Ooi CP, Seoni S, Barua PD, Molinari F, Acharya UR. using explainable AI framework. arXiv:2103.07110 (arXiv pre-
Application of explainable artificial intelligence for healthcare: print); 2021.
a systematic review of the last decade (2011–2022). Comput 124. Dash S, Gunluk O, Wei D. Boolean decision rules via column
Methods Programs Biomed. 2022;20: 107161. generation. Adv Neural Inf Process Syst. 2018;31:25.
108. Duckworth C, Chmiel FP, Burns DK, Zlatev ZD, White NM, 125. Bose S, Barao T, Liu X. Explaining AI for malware detection:
Daniels TW, Kiuber M, Boniface MJ. Using explainable machine analysis of mechanisms of malconv. In: 2020 international joint
learning to characterise data drift and detect emergent health conference on neural networks (IJCNN); 2020. IEEE. p. 1–8.
risks for emergency department admissions during covid-19. Sci 126. Al-Fawa’reh M, Saif A, Jafar MT, Elhassan A. Malware detection
Rep. 2021;11(1):1–10. by eating a whole APK. In: 2020 15th international conference
109. Antoniadi AM, Galvin M, Heverin M, Hardiman O, Mooney for internet technology and secured transactions (ICITST); 2020.
C. Prediction of caregiver quality of life in amyotrophic lat- IEEE. p. 1–7.
eral sclerosis using explainable machine learning. Sci Rep. 127. Ohana JJ, Ohana S, Benhamou E, Saltiel D, Guez B. Explain-
2021;11(1):1–13. able AI (XAI) models applied to the multi-agent environment
110. Zeng X, Hu Y, Shu L, Li J, Duan H, Shu Q, Li H. Explainable of financial markets. In: International workshop on explain-
machine-learning predictions for complications after pediatric able, transparent autonomous agents and multi-agent systems.
congenital heart surgery. Sci Rep. 2021;11(1):1–11. Springer; 2021. p. 189–207.
13
128. Gramegna A, Giudici P. Shap and lime: an evaluation of discrimi- 147. Omeiza D, Webb H, Jirotka M, Kunze L. Explanations in auton-
native power in credit risk. Front Artif Intell. 2021;140:25. omous driving: a survey. IEEE Trans Intell Transport Syst.
129. Wijnands M. Explaining black box decision-making: adopting 2021;20:20.
explainable artificial intelligence in credit risk prediction for p2p 148. Kothawade S, Khandelwal V, Basu K, Wang H, Gupta G. Auto-
lending. Master’s thesis, University of Twente; 2021. discern: autonomous driving using common sense reasoning.
130. El Qadi A, Trocan M, Diaz-Rodriguez N, Frossard T. Feature arXiv:2110.13606 (arXiv preprint); 2021.
contribution alignment with expert knowledge for artificial 149. Gao Y, Zhang S, Sun J, Yu S, Yamamoto T, Li Z, Li X. A joint
intelligence credit scoring. Signal, Image and Video Process- framework based on accountable AI for driving behavior assess-
ing; 2022. 1–8. ment and backtracking. In: 2022 IEEE 25th international confer-
131. de Lange PE, Melsom B, Vennerød CB, Westgaard S. Explain- ence on intelligent transportation systems (ITSC); 2022. IEEE.
able AI for credit assessment in banks. J Risk Financ Manage. p. 268–74.
2022;15(12):556. 150. Machlev R, Heistrene L, Perl M, Levy K, Belikov J, Mannor S,
132. Górski Ł, Ramakrishna S. Explainable artificial intelligence, law- Levron Y. Explainable artificial intelligence (XAI) techniques for
yer’s perspective. In: Proceedings of the eighteenth international energy and power systems: review, challenges and opportunities.
conference on artificial intelligence and law; 2021. p. 60–8. Energy AI. 2022; 20:100169.
133. Berk RA, Bleich J. Statistical procedures for forecasting crimi- 151. Zhang K, Xu P, Zhang J. Explainable AI in deep reinforcement
nal behavior: a comparative assessment. Criminol Pub Pol’y. learning models: a shap method applied in power system emer-
2013;12:513. gency control. In: 2020 IEEE 4th conference on energy internet
134. Mardaoui D, Garreau D. An analysis of lime for text data. In: and energy system integration (EI2); 2020. IEEE. p. 711–6.
International conference on artificial intelligence and statistics; 152. Zhang K, Zhang J, Xu P-D, Gao T, Gao DW. Explainable AI in
2021. p. 3493–501. PMLR. deep reinforcement learning models for power system emergency
135. Khosravi H, Shum SB, Chen G, Conati C, Tsai Y-S, Kay J, control. IEEE Trans Comput Soc Syst. 2021;9(2):419–27.
Knight S, Martinez-Maldonado R, Sadiq S, Gašević D. Explain- 153. Shen D, Wu G, Suk H-I. Deep learning in medical image analy-
able artificial intelligence in education. Comput Educ Artif Intell. sis. Annu Rev Biomed Eng. 2017;19:221–48.
2022;3:100074. 154. Thompson AC, Jammal AA, Medeiros FA. A review of deep
136. Alonso JM, Casalino G. Explainable artificial intelligence for learning for screening, diagnosis, and detection of glaucoma
human-centric data analysis in virtual learning environments. In: progression. Transl Vis Sci Technol. 2020;9(2):42–42.
International workshop on higher education learning methodolo- 155. Moolayil J. An introduction to deep learning and Keras. In: Learn
gies and technologies online. Springer; 2019. p. 125–38. Keras for deep neural networks. Berlin: Springer; 2019. p. 1–16.
137. Ghai B, Liao QV, Zhang Y, Bellamy R, Mueller K. Explain- 156. Zhang Z, Chen P, Sapkota M, Yang L. Tandemnet: Distilling
able active learning (XAL) toward AI explanations as inter- knowledge from medical images using diagnostic reports as
faces for machine teachers. Proc ACM Human Comput Interact. optional semantic references. In: International conference on
2021;4(CSCW3):1–28. medical image computing and computer-assisted intervention.
138. Hu Y, Mello RF, Gašević D. Automatic analysis of cognitive Springer; 2017. p. 320–8.
presence in online discussions: an approach using deep learning 157. Altinkaya E, Polat K, Barakli B. Detection of Alzheimer’s disease
and explainable artificial intelligence. Comput Educ Artif Intell. and dementia states based on deep learning from MRI images: a
2021;2:100037. comprehensive review. J Inst Electron Comput. 2020;1(1):39–53.
139. Hooshyar D, Yang Y. Neural-symbolic computing: a step toward 158. Mathews SM. Explainable artificial intelligence applications in
interpretable AI in education. Bull Tech Committee Learn Tech- nlp, biomedical, and malware classification: a literature review.
nol (ISSN: 2306-0212) 2021;21(4), 2–6. In: Intelligent computing-proceedings of the computing confer-
140. Melo E, Silva I, Costa DG, Viegas CM, Barros TM. On the use ence; Springer. 2019. p. 1269–92.
of explainable artificial intelligence to evaluate school dropout. 159. Madanu R, Abbod MF, Hsiao F-J, Chen W-T, Shieh J-S. Explain-
Educ Sci. 2022;12(12):845. able AI (XAI) applied in machine learning for pain modeling: a
141. Fernandez-Nieto GM, Echeverria V, Shum SB, Mangaroska review. Technologies. 2022;10(3):74.
K, Kitto K, Palominos E, Axisa C, Martinez-Maldonado 160. Garvin MR, Prates ET, Pavicic M, Jones P, Amos BK, Geiger
R. Storytelling with learner data: guiding student reflec- A, Shah MB, Streich J, Gazolla JGFM, Kainer D, et al. Poten-
tion on multimodal team data. IEEE Trans Learn Technol. tially adaptive SARS-COV-2 mutations discovered with novel
2021;14(5):695–708. spatiotemporal and explainable AI models. Genome Biol.
142. Knight S, Shibani A, Abel S, Gibson A, Ryan P. Acawriter: a 2020;21(1):1–26.
learning analytics tool for formative feedback on academic writ- 161. Cliff A, Romero J, Kainer D, Walker A, Furches A, Jacobson
ing. J Writing Res. 2020;20:20. D. A high-performance computing implementation of iterative
143. Conati C, Barral O, Putnam V, Rieger L. Toward personalized random forest for the creation of predictive expression networks.
XAI: a case study in intelligent tutoring systems. Artif Intell. Genes. 2019;10(12):996.
2021;298:103503. 162. Shah RD, Meinshausen N. Random intersection trees. J Mach
144. Bojarski M, Choromanska A, Choromanski K, Firner B, Jackel L, Learn Res. 2014;15(1):629–54.
Muller U, Zieba K. Visualbackprop: visualizing cnns for autono- 163. Ikemura T, Wada K, Wada Y, Iwasaki Y, Abe T. Unsupervised
mous driving. arXiv:1611.05418 (arXiv preprint); 2016. explainable AI for simultaneous molecular evolutionary study of
145. Kim J, Canny J. Interpretable learning for self-driving cars by forty thousand sars-cov-2 genomes. bioRxiv; 2020.
visualizing causal attention. In: Proceedings of the IEEE inter- 164. Lundberg SM, Lee S-I. A unified approach to interpreting model
national conference on computer vision; 2017. p. 2942–50. predictions. Adv Neural Inf Process Syst. 2017;30:25.
146. Zeng W, Luo W, Suo S, Sadat A, Yang B, Casas S, Urtasun R. 165. Prajod P, Huber T, André E. Using explainable AI to identify
End-to-end interpretable neural motion planner. In: Proceedings differences between clinical and experimental pain detection
of the IEEE/CVF conference on computer vision and pattern models based on facial expressions. In: International conference
recognition; 2019. p. 8660–9. on multimedia modeling. Springer; 2022. p. 311–22.
13
166. Dasgupta D, Akhtar Z, Sen S. Machine learning in cyber- 180. Williamson B. Digital policy sociology: software and sci-
security: a comprehensive survey. J Defense Model Simul. ence in data-intensive precision education. Crit Stud Educ.
2022;19(1):57–106. 2019;20:1–17.
167. Ucci D, Aniello L, Baldoni R. Survey of machine learning tech- 181. Luan H, Tsai C-C. A review of using machine learning
niques for malware analysis. Comput Secur. 2019;81:123–47. approaches for precision education. Educ Technol Soc.
168. Perarasi T, Vidhya S, Ramya P, et al. Malicious vehicles identify- 2021;24(1):250–66.
ing and trust management algorithm for enhance the security in 182. Akgun S, Greenhow C. Artificial intelligence in education:
5g-vanet. In: 2020 second international conference on inventive addressing ethical challenges in k-12 settings. AI Ethics.
research in computing applications (ICIRCA); 2020. p. 269–75. 2021;20:1–10.
IEEE. 183. Gardner J, Brooks C, Baker R. Evaluating the fairness of predic-
169. Jaswal G, Kanhangad V, Ramachandra R. AI and deep learning in tive student models through slicing analysis. In: Proceedings of
biometric security: trends, potential, and challenges. Boca Raton: the 9th international conference on learning analytics and knowl-
CRC Press; 2021. edge. p. 225–234; 2019.
170. Rudin C. Stop explaining black box machine learning models for 184. Atakishiyev S, Salameh M, Yao H, Goebel R. Explainable artifi-
high stakes decisions and use interpretable models instead. Nat cial intelligence for autonomous driving: a comprehensive over-
Mach Intell. 2019;1(5):206–15. view and field guide for future research directions. arXiv:2112.
171. Zhang Z, Hamadi HA, Damiani E, Yeun CY, Taher F. Explain- 11561 (arXiv preprint); 2021.
able artificial intelligence applications in cyber security: state- 185. Ni J, Chen Y, Chen Y, Zhu J, Ali D, Cao W. A survey on theo-
of-the-art in research. arXiv:2208.14937 (arXiv preprint); 2022. ries and applications for self-driving cars based on deep learning
172. Capuano N, Fenza G, Loia V, Stanzione C. Explainable arti- methods. Appl Sci. 2020;10(8):2749.
ficial intelligence in cybersecurity: a survey. IEEE Access. 186. Yousuf H, Zainal AY, Alshurideh M, Salloum SA. Artificial
2022;10:93575–600. intelligence models in power system analysis. In: Artificial intel-
173. Buczak AL, Guven E. A survey of data mining and machine ligence for sustainable development: theory, practice and future
learning methods for cyber security intrusion detection. IEEE applications; Springer. 2021. p. 231–42.
Commun Surv Tutor. 2015;18(2):1153–76. 187. Lorente MPS, Lopez EM, Florez LA, Espino AL, Martínez JAI,
174. Chalapathy R, Chawla S. Deep learning for anomaly detection: de Miguel AS. Explaining deep learning-based driver models.
a survey. arXiv:1901.03407 (arXiv preprint); 2019. Appl Sci. 2021;11(8):3321.
175. Carta S, Podda AS, Reforgiato Recupero D, Stanciu MM. 188. Konstantakopoulos IC, Das HP, Barkan AR, He S, Veeravalli T,
Explainable AI for financial forecasting. In: International con- Liu H, Manasawala AB, Lin Y-W, Spanos CJ. Design, bench-
ference on machine learning, optimization, and data science; marking and explainability analysis of a game-theoretic frame-
Springer. 2021. p. 51–69. work towards energy efficiency in smart infrastructure. arXiv:
176. Chromik M, Eiband M, Buchner F., Krüger A, Butz A. I think i 1910.07899 (arXiv preprint); 2019.
get your point, AI! the illusion of explanatory depth in explain- 189. Pannu HS, Malhi A, et al. Deep learning-based explainable target
able AI. In: 26th international conference on intelligent user classification for synthetic aperture radar images. In: 2020 13th
interfaces; 2021. p. 307–17. international conference on human system interaction (HSI);
177. Bussmann N, Giudici P, Marinelli D, Papenbrock J. Explain- 2020. p. 34–9. IEEE.
able machine learning in credit risk management. Comput Econ. 190. Nahata R, Omeiza D, Howard R, Kunze L. Assessing and
2021;57(1):203–16. explaining collision risk in dynamic environments for autono-
178. Agarwal A, Bhatia A, Malhi A, Kaler P, Pannu HS, et al. Machine mous driving safety. In: 2021 IEEE international intelligent
learning based explainable financial forecasting. In: 2022 4th transportation systems conference (ITSC); 2021. p. 223–30.
international conference on computer communication and the IEEE.
internet (ICCCI); 2022. p. 34–8. IEEE.
179. Eliot DLB. The need for explainable AI (XAI) is especially cru-
cial in the law. Available at SSRN 3975778; 2021.
13

Survey On Explainable AI - From Approaches, Limitations and Applications Aspects

Uploaded by

Copyright:

Available Formats

Survey On Explainable AI - From Approaches, Limitations and Applications Aspects

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Survey On Explainable AI - From Approaches, Limitations and Applications Aspects

Uploaded by

Copyright:

Available Formats

Human-Centric Intelligent Systems (2023) 3:161–188

Survey on Explainable AI: From Approaches, Limitations

Keywords Explainable AI · Machine learning · Human-centered · Survey

Abbreviations KG Knowledge graph

* Yuchen Wei Naimeng Yao

Table 1 Explainable AI: who Who? Why? For what?

Fig. 1 The proposed organiza-

Fig. 2 Taxonomies of XAI

Source- Representation- Logic-

Table 3 Summarising the XAI approaches from logic-oriented

Disease diagnosis and treatment: Research on XAI in

Pain detection; quality of life analysis; surgery complication

Financial-related figure forecasting; credit risk manage-

Lesion classification; disease diagnosis and treatment

Decision Vehicle actions; power system management

Feedback providing; intelligent tutoring systems

Intrusion detection system; malware detection

ment; legal text classification

granularity graph supporting fact extraction (counterfactual

tures. These explanations assist clinicians in comprehending

the reasoning behind the diagnosis and identifying relevant

In the domain of relatively low-risk screenings, explain-

able AI research has explored the integration of medical

interpretable diagnostic evidence. For instance, Wang et al.

[96] and Lucieri et al. [97] have integrated medical records

directly from medical images using multi-modal medical

pretable evidence and explanations for clinicians during the

Additionally, hybrid XAI approaches have been explored,

such as Mimir proposed by Hicks et al. [84]. Mimir learns

incorporates these explanations to produce structured and

visual elements. These explanations aid in understanding

contributing to the risk assessment, assisting clinicians in

[36, 63, 144–152]

making informed decisions and recommendations for further

screenings or preventive measures.

In terms of biomedical field, the end-users of XAI are mostly

Education and training

diagnostic evidence. This insight into the decision-making

process can enhance the transparency and trustworthiness

3.3.4 Domain‑Specific Insights 3.4.2 XAI Based Proposals

3.4 Finance Financial data forecasting AI models are being used to pre-

3.7 Civil Engineering cognition method, is used to explain the highlight of impor-

Table 5 XAI suitability analysis for application domains

You might also like

Abbreviations KG Knowledge graph

3.3.4 Domain‑Specific Insights 3.4.2 XAI Based Proposals

3.4 Finance Financial data forecasting AI models are being used to pre-

3.7 Civil Engineering cognition method, is used to explain the highlight of impor-