Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

Digital Object Identifier 10.1109/ACCESS.2017.DOI

Explainable AI: current status and future

directions
PRASHANT GOHEL1 , PRIYANKA SINGH 1 , AND MANORANJAN MOHANTY 2
1
DA-IICT, Gandhinagar, Gujarat, India
2
Centre for Forensic Science, University of Technology Sydney, Australia
Corresponding author: Prashant Gohel (e-mail: [email protected]).
arXiv:2107.07045v1 [cs.LG] 12 Jul 2021

ABSTRACT Explainable Artificial Intelligence (XAI) is an emerging area of research in the field of
Artificial Intelligence (AI). XAI can explain how AI obtained a particular solution (e.g., classification or
object detection) and can also answer other "wh" questions. This explainability is not possible in traditional
AI. Explainablity is essential for critical applications, such as defence, health care, law and order, and
autonomous driving vehicles, etc, where the know how is required for trust and transparency. A number
of XAI techniques so far have been purposed for such applications. This paper provides an overview of
these techniques from a multimedia (i.e., text, image, audio, and video) point of view. Advantages and
shortcomings of these techniques have been discussed, and pointers to some future directions have also
been provided.

INDEX TERMS Explainable Artificial Intelligence (XAI), Explainability, Interpretable Artificial Intelli-
gence.

I. INTRODUCTION
In recent years, Artificial Intelligence (AI)-based applica-
tions have been used in various aspects of human life, such
as science, business, finance and social networking, etc. AI-
based algorithms have been successfully applied to all types
of data (text, image, audio, video) in various applications,
such as healthcare, defence, law and order, governance, au-
tonomous industry, etc. An AI algorithm can now efficiently
solve a classification, regression, clustering, transfer learning
or optimizations problem [16]. This current day AI is mainly
limited to a sub-branch known as machine learning (ML).
Machine learning provides a computer with a set of examples
(aka training data set), and let the computer learn from the
example set. Once well trained, the computer can then answer
questions related to what it was taught previously. Typically, FIGURE 1: AI vs XAI
this traditional AI is a blackbox that can answer “yes” and
“no” type questions without elaborating how that answer is
obtained. insight of how a result was obtained will therefore not only
In many applications, an explanation of how an answer can induce trustfulness but also can avoid life-threatening
was obtained is crucial for ensuring trust and transparency. errors. In some other applications (e.g., law and order),
An example of one such application is a medical application, answers to other "wh" questions (such as "why", "when",
where the doctors should be damn sure about a conclusion. "where", etc.) could be required. The traditional AI is unable
They, for example, would like to know how AI decided to answer these "wh" questions.
whether someone is suffering from a disease by analyzing a This explainability requirement lead a new area of AI
CT scan image. AI-based systems are not 100% perfect. An research, know as Explainable AI (XAI). Figure 1 shows

VOLUME 4, 2016 1
 Gohel et al.: Explainable AI: current status and future directions

how XAI can add new dimensions to AI by answering the develop rationale approach to implementing action driven by
"wh" questions that were missing in traditional AI. The XAI, AI and also helpful for end users to understand [70].
therefore, has drawn a great interest from critical applica- In some basic level applications of AI, such as symptoms
tions, such as health care, defence, law and order, etc., where based health diagnosis, explainability is straightforward. But
explaining how an answer was obtained (i.e., answers to "wh" in the race of achieving more human level accuracy, re-
questions) is as important as obtaining the answer. In both searchers and scientists developed more complex algorithms.
academia and industry, XAI research, therefore, has become Neural network and deep learning based applications for
a priority. Although a number of work have been already decision making is quite elusive and less interpretable [10]
proposed, more and more work is required to realize the full [12].
potential of XAI.
In this paper, we survey existing research on XAI from 1) Case Study 1: Oil Refinery Assets Reliability: Furnace
multimedia (text, image, audio, and video) point of view. Flooding Predictions
Since each media is different than the other (i.e., image The Situation: Stable combustion is critical for uninter-
is different than video) in some sense, an XAI method rupted operation of furnace. Due to unidentified factors,
applicable to one media may not be effective for another. if stable combustion is interrupted, it leads to disastrous
We group the proposed XAI methods for each media, point incident. For consideration of safety, it is highly required
out their advantages and disadvantages, and provide pointers that such conditions need to be identified and acted upon.
to some future works. We believe that our classification of If such preventive measures are not taken, furnace may flood
XAI methods will provide a guide and inspirations to future and eventually occurs in explosion. Such incident turns into
research in multi-modal applications (for example, XAI for shut down of plant which causes production delay and huge
defence where AI-based solutions are required for image, maintenance cost. Unplanned shutdown of industrial plant
text, audio, etc.). results in to huge financial loss.
The rest of the paper is organized as follows. Section II The ML/AI Solution: A reliable prediction of furnace
discusses why classical blackbox AI. In Section III, we intro- flooding is required to alert maintenance staff at least 30 min-
duce XAI by discussing its scopes, objectives, and various utes. To develop such kind of predictive maintenance model,
tools proposed to realize explainability. This section also it requires to collect calibration from all sensor data including
provides a classification tree outlining various XAI methods weather and humidity etc. This model should predict flooding
proposed for multimedia data. These methods are elaborated prediction at least 20 minutes prior.
in the remaining sections. Section IV explains classification Why Explainable AI: There could be n number of dif-
tree for XAI techniques, where transparent and post hoc ferent factors which are responsible for flooding or shut
texhniques are explained. Section V discusses XAI methods down of combustion chamber or furnace. After shutting down
applied to image data. Section VI discusses XAI methods the furnace maintenance staff needs to investigate cause of
applied to natural language processing (text data). Section failure. This investigation helps to identify which sensor
VII and VIII gives understanding about how XAI works with causes unstability of continuous operation in furnace. Mak-
video and audio type of data. Section IX explains about multi ing prediction in such a way that helps to identify cause of
modal data like caliberated data from sensors in CSV format failure with respect to different parts of combustion chamber.
and also explains prevalence of XAI in defence and industrial This explainability makes easier to address snag in industrial
applications for providing predictive maintenance to reduce operation with out wasting much time in investigation.
cost of production and maintenance.
2) Case Study 2: Video Threat Detection
II. THE BLACKBOX AI The Situation: In today’s age, it is required to secure physi-
A. OVERVIEW cal asset and human. A typical solution is to combine security
Blackbox models of machine learning are rapidly being used personnel and digital camera for video analytics. Due to
with a tag of AI-enabled technology for various critical human eye limitation. it is not possible to watch every entry
domains of human life. The list of domains varies from socio- point and every video feed at all times. Absolute human
economic justice, cyber forensics, criminal justice, etc. But supported surveliance may have certain error and threats of
these AI-powered models are lagging to win the trust of naive miss identification.
people because these models are less transparent and less The ML/AI Solution: AI and deep learning based models
accountable [68]. For example, there are cases in criminal evaluate video feeds to detect threats. These threats are later
justice where the AI-enabled justice model release criminals flagged for the security personnel. AI based object or face
on parole and grants bail. This leads to serious consequences recognition models evaluate video feeds at air port to identify
among people and the government [69]. visitors who carry weapon or those are known criminals.
These AI model should ignore normal employee or air port
B. EXPLAINABILITY REQUIREMENT staff.
“Explainability” is a need and expectation that makes "deci- Why Explainable AI: Due to skewness or bias of model, it
sion" of intrinsic AI model more transparent. This need will may possible that trained AI model detects innocents visitors
2 VOLUME 4, 2016
Gohel et al.: Explainable AI: current status and future directions

or employees due to certain weighted features in training predictions as original models then there is no need for
samples. Such kind of incidents raises legality aspects for an original model. Despite original and XAI models are
genuinity of such surveliance systems. Transparency in such computing the same predictions there are pretty good chances
systems are one of the crucial factor before framing one that both approaches are using a different set of features for
person as a criminal or suspect. Company of AI enabled making the same predictions. Hence, it is not faithful towards
surveliance system are required to provide justification in the computation of the black box. It also happens sometimes
the court. An individual humiliated and searched publically that the XAI model provides too much extra information
by security forces leads to several legal consequences on which is not relevant to the original inferences of a black-box
government as well as airport authority. model.
For image processing domain saliency maps are consid-
C. USAGE ered the best tool for image classification. These maps are
AI covers the entire ecosystem of computer-enabled interdis- being useful to determine which part of an image is con-
ciplinary technologies. AI enables a group of technologies to sidered and which is omitted by the model for prediction.
behave more cognitively and context-oriented like human or But saliency maps are not explaining how different parts of
animal rather than rule-oriented. AI is all about mimicking images are contributing to the given prediction. As shown
the complex cognitive behavior of all living entities on the in Fig. 2 Saliency maps are not able to demonstrate except
earth [71]. where neural network model is focusing.
There are many day-to-day usages of AI-enabled appli-
cations from object recognition, product recommendation in
online shopping portals, chatbots for customer service and
document processing, etc. AI is also one of the important
tools for medical imaging and diagnosis. CT scan-based
tumor diagnoses are effective and more accurate for certain
conditions [71].
AI will have much more applications in the future. AI will
FIGURE 2: Saliency does not explain anything except where
be a reliable helping hand for doctors to do surgery and di-
the network is looking. We have no idea why this image
agnoses. Autonomous vehicle driving is one of the upcoming
is labelled as either a dog or a musical instrument when
areas where AI will be an important tool. AI can decide while
considering only saliency. The explanations look essentially
on-road driving for elder people either by taking complete
the same for both classes. Credit: Chaofen Chen, Duke Uni-
control or by assisting a human driver. For the criminal justice
versity [11]
system, AI can make an important decision to declare a
.
person guilty or non guilty [69]. AI-enabled decision-making
systems provide better support for professionals. Because of Consider, for instance, a case where the explanations for
the surge in applications of AI in corporate and industry, it multiple (or all) of the classes are identical. This situation
becomes a hot topic for ethical concerns. AI usage policies would happen often when saliency maps are the explanations,
will be decided by government agencies on aspects like because they tend to highlight edges, and thus provide similar
privacy, optimization, etc. To make AI more reliable we explanations for each class. These explanations could be
need to make it more transparent and interpretable. This identical even if the model is always wrong. Then, showing
motivation makes upcoming AI development with aspects only the explanations for the image’s correct class misleads
of explainability. This area of explainability oriented AI is the user into thinking that the explanation is useful and that
known as XAI. the black box is useful, even if neither one of them is.

D. KEY ISSUES WITH EXPLAINABLE ML

The main reason behind the difficulty to understand and inter-
pret the Black box ML model is either black-box function is
quite complicated for a human to understand. Because deep
learning-based models are recursive with non-differentiable
recursive functions as active functions. Another reason is
some functions are proprietary so it is not allowed to expose
publicly. There is a belief in XAI researchers that inter-
pretable ML models may reduce the accuracy of prediction
and conclusion. Due to this belief, many researchers are now FIGURE 3: Panda image was tampered by adding some
having good expertise in deep learning but not in XAI [11]. adversarial noise [72]
Many times explainable AI methods provide justifications
that are not aligned with what the original method computes. Fig. 3 illustrates such an example where an image of a
If explainable methods are computing the same results and Panda is predicted as a Gibbon with high confidence after
VOLUME 4, 2016 3
 Gohel et al.: Explainable AI: current status and future directions

the original Panda image was tampered by adding some B. SCOPE

adversarial noise.

III. XAI AS A TOOL TO OPEN BLACK BOX

In section III-A objectives like transparency, trust, bias
and fairness of XAI are discussed. Section III-B provides
overview about different scopes of XAI.

A. OBJECTIVES

FIGURE 5: Scopes of XAI

Ideally, scope of XAI can be as broad as scope of AI. Major

scopes are NLP (natural language processing), health care,
FIGURE 4: Objectives of XAI engineering, and defense. NLP and engineering comprise
banking, finance, digitization, and automation. These scopes
The main objective of XAI is to answer the "wh" questions of XAI are depicted in Figure 5.
related to an obtained answer. For example, XAI should be
1. Data Protection: European Union and its regulatory
able to answer "why a particular answer was obtained?",
body have a ’right to explanation’ clause. That makes to
"how a particular answer was obtained", "when a particular
enables explanation from XAI algorithms.
AI-based system can fail?" [13] [14] [15] . By doing this,
2. Medical: XAI can diagnose a patient by observing
XAI can provide trustworthiness, transparency, confidence,
his/her past medical records. Using AI/ML algorithms in the
and informativeness (Figure 4).
medical image processing domain it is easier for medical
1) Transparency and Informativeness
experts to diagnose patients with malignant cancer tumors
and other lung diseases.
XAI can enhance transparency as well as fairness by provid-
ing a justification that can be understood by a layman. The 3. Defense: XAI in defense practices becomes crucial
minimum criteria for a transparent AI model are it should be because of automated weapon and surveillance systems. XAI
expressive enough to be human-understandable. also provides good second-hand support during combat mode
Transparency is important to assess the performance of the training and real-time combat tactics.
XAI model and its justification. Transparency can assure any 4. Banking: The banking system is one of the biggest
false training to model that causes vulnerabilities in the pre- financial sectors which affects human life the most. In day-
diction that makes a huge loss in person to the end consumer. to-day life, there are many fraud transactions and cones by
False training is possible to tweak the generalization of any cheaters. Well-trained XAI models can help to investigate
AI/ML model that leads to providing unethical benefits to any fraudulent transactions and help to reduce false positives
party unless it is not made transparent. cases.

2) Trust and confidence IV. CLASSIFICATION TREE

Trust is one of the important factors that makes humans rely XAI techniques are classified in two categories of transparent
on any specific technology. A logical and scientific justifica- and post-hoc methods. Transparent methods are such meth-
tion for any prediction and conclusion makes humans favor ods where the inner working and decision-making process
the prediction or conclusion made by AI/ML algorithms. of the model is simple to interpret and represent. Bayesian
model, decision trees, linear regression, and fuzzy inference
3) Bias Understanding and Fairness systems are examples of transparent models. Transparent
Bias and variance trade-off in AI/ML model makes XAI pro- methods are useful where internal feature correlations are
mote fairness and helps to mitigate bias ( bias-variance trade not that much complex or linear in nature. Figure 6 depicts
off) of prediction at the time of justification or interpretation detailed classification of various XAI techniques and ap-
[59]. proaches with respect to various types of data [28].
4 VOLUME 4, 2016
Gohel et al.: Explainable AI: current status and future directions

FIGURE 6: XAI classification with respect to type of data

A. POSTHOC METHODS for all data sets while local methods are limited to specific
When there is a nonlinear relationship or higher data com- kind of data sets. In contrast,model-agnostic tools can be
plexity exists, posthoc methods are useful to interpret model used for any AI/ML model. Here pairwise analysis of input
complexity. In this case, the posthoc approach is a useful and results plays a key role behind interpretability. In next
tool to explain what the model has learned when it is not sections, we have discussed model specific techniques like
following a simple relationship among data and features. feature relevance, condition based explanations,rule based
Result-oriented interpretability methods are based on fea- learning and saliency map.
ture summary’s statistical and visualization-based presenta-
tion. statistical presentation denotes statistics for each feature B. TRANSPARENT METHODS
where the feature’s importance is quantified based on its Transparent method like logistic regression, support vector
weight in prediction. machine, Bayesian classifier, K nearest neighbour provides
A post-hoc XAI method receives a trained and/or tested justification with local weights of features. Models falls
AI model as input, then generates useful approximations of under this category satisfies three properties named as algo-
the model’s inner working and decision logic by producing rithmic transparency, decomposability and simulatability.
understandable representations in the form of feature impor- Simulatability stands for simulation of model must be
tance scores, rule sets, heat maps, or natural language. Many executed by a human. For human enabled simulation com-
posthoc methods try to disclose relationships between feature plexity of model plays an important role. For an example
values and outputs of a prediction model, regardless of its in- sparse matrix model is easy to interpret compared dense
ternals. This helps users identify the most important features matrix because sparse matrix model ia easy to justify and
in an ML task, quantify the importance of features, reproduce visualize by humans.
decisions made by the black-box model, and identify biases Decomposability stands for explainability of each aspect
in the model or data. of model from input of data to hyper parameters as well as
Some post-hoc methods, such as Local Interpretable inherent calculations. This characteristics defines behavior of
Model-agnostic Explanations, extract feature importance a model and its performance constraints. Complex input fea-
scores by perturbing real samples, observing the change in tures are not readily interpretable. Due to this contraints such
the ML model’s output given the perturbed instances, and models are not belongs to category of transparent model.
building a local simple model that approximates the original Algorithmic transparency defines algorithm level inter-
model’s behavior in the neighborhood of the original sam- pretability from input of given data to final decision or
ples. Posthoc methods are further classified in model agnos- classification. Decision making process should be understood
tic and model specific. Model-specific techniques supports by users with transparency. For an example linear model is
explainability constraints with respect to learning algorithm deemed transparent because error plot is easy to visualize and
and internal structure of given deep learning model. model- interpret. With help of visualization user can understand how
agnostic techniques applies pair wise analysis of model model is reacting in different situation.
inputs and predictions to understand learning mechanism and The transparent model is realized with the following XAI
to generate explanations. techniques.
It is observed that global methods are capable to explain
VOLUME 4, 2016 5
 Gohel et al.: Explainable AI: current status and future directions

1) Linear/Logistic Regression decomposable, algorithmic transparency and human simula-

Logistic Regression (LR) is a transparent model to predict de- tion. Complex variable dependency may affect transparency
pendent variable which follows property of binary variable. and human simulation for bayesian model.
This method assumes there is a flexible fit between predictors
and predicted variables. C. MODEL SPECIFIC
For understanding of logistic regression, model it requires Model specific XAI models are realized using following
audience to have knowledge of regression techniques and techniques.
its working methodology. Due to this constraints, depending
upon type of audience logistic regression falls either in trans- 1) Feature Relevance
parent or posthoc methods.Even though logistic regression It is always important to figure out the most impactful
is the simplest form of supervised classification techniques, features which are crucial for decision makings. For this
its mathematical and statistical concepts are need to be taken feature, importance is introduced. Feature importance shows
care off. the impact factor of each feature in derived decisions [26].
Along with feature importance, correlation among features is
2) Decision Trees also useful for explainability. In AI-based medical diagnosis
Decision trees is a transparent tool which satisfies trans- model, feature correlation in training data is one of the
parency in a large context. It is a hierarchical decision making driving forces for diagnosis.
tool. Smaller scale decision trees are easily simulatable.
Increment in number of levels in trees make it more algo- 2) Condition based Explanation
rithmically transparent but less simulatable. Due to its poor Condition based explanation is required on the basis of
generalization property, ensembling of trained decision trees "why", "why despite" and "why given". Some specific ob-
are useful to overcome poor generalization property. This served inputs plays key role to justify prediction. By asking
modification makes decision tree tool less transparent. “Why?” oriented questions, model will provide all possible
explanations with set of conditions. This condition set is gen-
3) K-Nearest Neighbors erated with completeness phenomena. "what if" provides hy-
KNN (K-Nearest Neighbors) is a voting based tool that pothetical reasoning for counterfactual justification. A simple
predicts class of test sample with help of voting the classes of logical model converts user inputs in to the form of con-
its k nearest neighbors. Voting in KNN depends on distance straints based inputs and provide justification that whether
and similarity between examples. Simple KNN supports constraints are being satisfied in the form of conditions.
tansparency, algorithmic transparency and human centric
simulation. KNN’s transparency depends on the features, 3) Rule based learning
parameter N and distance function used to measure similarity. Explainability is required because ML model output is nu-
Higher value of K impacts simulation of model by human merical and neural network is too much complex that normal
user. Complex distance function restricts decomposability of user can not understand the complexity of hyperparameters
the model and transparency of algorithmic operation. and its effect on final prediction.
After getting some insightful understanding of trained
4) Rule based learning model and interpretability of results, suitable approach is
Rule based model defines rule to train model. Rule can to explain derived results to customers and naive users is
be defined in the simple conditional if-else form or first translation of those insights into rules such that it can provide
order predictive logic. Format of rules depends on type of full transparency for XAI [25]. Once rules are framed for all
knowledge base. Rules provides two advantages to this type possible predictions, It makes even the most complex neural
of model. First, since format of rules are in linguistic terms network model transparent.
it is transparent for user to understand. Second, it can han-
dle uncertainty better than classical rule based model [25]. 4) Feature based saliency map
The number of rules in model improves the performance of Saliency maps are generally used with image processing
model with compromising interpretability and transparency areas of applications to show what parts of video frames or
of model. Model with less number of rules can be easily images are the most significant for derived decision of CNN.
simulated by human. XAI saliency map is a tool which is useful showcase
inner working of DNNs. Gradinet computation using back
5) Bayesian Model propagation algorithm are used as quantified measures to
Bayesian model are probabilistic model with notion of con- project intensity of colours on plane.
ditional dependencies between set of dependent and indepen-
dent variables. Bayesian model is transparent enough for end D. MODEL AGNOSTIC
users who are having knowledge of conditional probability. Model agnostic techniques are also applied for text, image,
Bayesian model are enough suitable for all three properties audio and video. various techniques like LIME, perturbation,
6 VOLUME 4, 2016
Gohel et al.: Explainable AI: current status and future directions

LRP, SHAP, provenance and taxonomy inductions, counter 3) LRP: Layer-wise Relevance Propagation
factual explanations are applicable on different type of data LRP is useful to unbox complex neural networks. It pro-
like text, image, audio and video. pogates predictions backward in the neural network. For
backward propogations specific rules are designed.
1) LIME- Local Interpretable Model-agnostic Explanations
4) Provenance and taxonomy induction
Model agnosticism specifies the property that LIME is able
Provenance and taxonomy induction are logical inference
to provide justification for any type of supervised learning
based techniques to justify result based on partially derived
model’s prediction. This technique is applicable for any sort
results. In section VI-A it is discussed with detail [80]. Com-
of data like image, text and video. This means that LIME
prehensive analysis of important XAI techniques is presented
is able to handle any supervised learning model and provide
in Table 1.
justification.
LIME provides local optimum explanations which com- V. XAI AND IMAGE
putes important features around the vicinity of given partic- Explanations in XAI are often categorized into two main
ular instance to be explained. By default it generates 5000 aspects. The first category is whether the given explanation
samples of the feature vector which are following normal is limited to the given conclusion of a model or it describes
distributions. After producing normally distributed samples the entire prediction process which includes training aspects
it finds the target variables for samples whose decisions are also. The second category differentiates between whether
explained by LIME. explanation comes directly from the prediction process or it
After obtaining local generated dataset and their predic- requires posthoc analysis [1] [23].
tions it assigns weights to each of the rows how close they Popular instance-level explanation methods for image
are from original samples. Then it uses a feature selection classification such as LIME [2], SHAP [3] and LRP [4],
technique like lasso or PCA (Principle Component Analysis) typically create feature importance rankings. Although in-
to get significant features. Detailed discussion about LIME is sightful, these methods have clear drawbacks: they do not
referred in section V-A. determine the optimal explanation size, they do not account
LIME has found much success and support in the field for feature dependence, and they are related to only one
of XAI and is implemented for text, image, and tabular prediction class.
data. One noiticeable observation about LIME is that it is
applicable and extendable to all significant machine learning A. LIME
domains. In the domain of text processing, embeddings and Local interpretable model-agnostic explanations (LIME), as
vectorization of given word or sentence can be considered the name suggests it interprets the model locally and explains
as a basic unit for sampling. For Image, segmented parts of the classification of the model in a faithful manner. In LIME,
Image are considered as samples for input. the prediction of the model is used as labels for supervised
training to train the XAI model.
2) Perturbation
Perturbation helps to generate desired explanation drivers
and analyze impact of perturbed features on the given target.
It provides summary of all features for given pertubed results.
In perturbation mechanism local changes are observed
on target results and perturbation scores are assigned to all
features using LIME or SHAP methods.
Perturbation method is easy to implement and it is not FIGURE 7: Explaining an image classification prediction
applicable to specific architecture of model. This method made by Google’s Inception neural network. The top3 classes
can be applied to type of AI/ML model. Disadvantage of predicted are “Electric Guitar” (p= 0.32), “Acoustic guitar”
perturbation method is, it is computationally expensive if (p= 0.24) and “Labrador” (p= 0.21) [73]
number of features are relatively greater than normal average.
Sparse linear models are a useful tool to explain LIME-
As there are more number of features it takes more time to
based justification [5]. Using a sparse linear model it is
evaluate combination of all features.
possible to highlight important pixels with their weights for
This scenario occurs specifically when dimensions of input a particular respective class as shown in Fig. 7. This set
are more because number of combinations of all features of important pixel areas give intuition as to why the model
grows rapidly. Moreover, this mechanism can underestimate would think that class may be present. As described in figure
the selected feature’s contribution because respective feature 5 important pixel-based explanation is given. It interprets
reaches saturation level in perturbation such that perturbing the original image as electric guitar, acoustic guitar, and
them do not have any impact on derived results. Labrador with respect to the confidence score of 0.32, 0.24,
and 0.21.
VOLUME 4, 2016 7
 Gohel et al.: Explainable AI: current status and future directions

Approach Advantages Drawbacks Future Directions

The resulting explanations are found To reduce local fidelity of justific
Plug and play
LIME [1][4] to be unstable. The ranking does not , Non redundant instance based ju
based
account for feature dependence. cation can be tried.
Small perturbations with no change
Optimized for SHAP can be used to define cont
SHAP [5][8] in prediction leads to different
speed up of each feature.
explanation.
Suitable for Low abstract level explanation With layer wise attribution of neu
LRP [0]
neural network with relevance map class discriminavity can be increa
Individual pixels are typically It is a self explainable approach f
Feature importance
Heatmap not meaningful for classification but future work req
based presentation
humans less interpretable text-based presentation
More human
SEDC and SEDC-T [6][7] More than one irreducible explanation counter factual analysis for text b
centric explanation
Explanation is limited with respect
to only local features.
Feature weight based
Feature Importance [77][31][32] May drive attention of user Reduction of local fidelity on pre
explanation
towards from important global
dependency.
Such techniques assume that end
users can understand
More convenient
Induction [77] specific representations, Generalization is required for div
for programming
such as first-order logic rules and
reasoning trees.
It is more accessible for lay users
Natural language but not much compatible validation of justification is requi
Provenance [77][33]
based explanation for validation because it is more reliability.
based on natural language.
TABLE 1: Analysis of various XAI appraoches

B. SHAP (SHAPLEY ADDITIVE EXPLANATIONS) For explainability, we generally try to provide the ex-
The main objective of SHAP is to understand the prediction planation on the basis of the selection and rejection of the
of an input A by computing the decision-making contribution specific alternatives or outcomes. For given scenario, why
of each feature for the classification. SHAP computes Shap- only outcome A selected not B. A useful tool to provide
ley values using coalitional game theory. It is a technique such a discriminative explanation is using counterfactuals.
described by Shapley (1953) [9] as an approach for assigning We can use counterfactuals to provide reasonably valid ar-
a reward to game players according to their contribution to guments at the end of the conclusion by machine learning
the game. SHAP assigns each feature an importance value model which is supported by either deep learning or classical
for a particular prediction [6]. statistical modeling. With the nature of counterfactuals, a
The key difference between LIME and SHAP is the pro- certain set of features are defined that can change the decision
cess to assign weights to the regression linear model. LIME of the model. If those features are not available then the
uses cosine measure between the original and the perturbed final conclusion of the model will be changed. It is argued
image. SHAP, the weights are determined using the Shapley that they are more likely to comply with recent regulatory
formula. LIME and SHAP methods have their drawbacks: developments such as GDPR. The counterfactual approach
they do not determine the optimal explanation size, they do helps to understand and satisfy three important needs of
not account for feature dependence, and they are related to interpretability: how an interpretation of a model was made,
only one prediction class. it provides the scope to tweak with adverse decisions, and
gives clues to receive intended results in prediction.
C. COUNTERFACTUAL VISUAL EXPLANATIONS There is a scenario of classification in classes A, B, and
For human psychology, it is convenient to explain by giving C. let’s say there is a feature set (a1, a2,......, a10) which are
contrastive explanations rather than giving direct explana- relatively required to get prediction A. For Given input result
tions to the conclusion or prediction of the machine learning of prediction is class B because feature set (a11,a12,.....a20)
model. We can explain by providing reasons, why only a is present and feature set (a1, a2,......,a10) is absent. The
certain class is selected and why others are rejected. approach with the feature set based interpretation leads to
8 VOLUME 4, 2016
Gohel et al.: Explainable AI: current status and future directions

smooth convincing in human-critical domains like Crime, This step-by-step analysis provides a very effective expla-
Forensic. [7]. nation for external symptoms. To cover a broad spectrum of
XAI it is required to make the justification that is independent
of the AI model. Such methods are known as model agnostic
XAI methods. LIME (Local Interpretable Model-Agnostic
Explanation) [2] is an example of a model agnostic method.
LIME is a framework to quantify weights of all factors
which are there to make conclusion or prediction. There are
other model agnostic XAI techniques also like SHAPLEY
[6]. Deep learning is a very important tool for accurate
medical diagnosis but its black-box approach for prediction
FIGURE 8: Figure a depicts an image which predicted as and conclusion makes it restricted for the certain critical area
warplane by model. Figure b and c shows that by removing of human medical science.
body and main wings from plane model predicts tail portion
as wing rather than Plane. So figure b is an critically mini- 1) Explainability methods for XAI-Healthcare
mum portion in an image to get classified as Warplane. [8] There are two types of methods for an explanation of medical
imaging. One method is based on attribution based and
This evidence-based approach of counterfactual is known another method is based on perturbation.
as Search for EviDence Counterfactual for Image Classifica- Attribution LIME is an attribution-based approach for
tion(SEDC). Fig. 8 explains how body of plane i s minimally medical image diagnosis. In the attribution-based methods,
critically portion to classify as an Image. one needs to determine the contribution and weight of each
As per the research of dhurandhar et al. [7] there is a feature. The success of attribution-based explanation is based
notion of pertinent positive(PP) and pertinent negative(PN). on the generality of assigned weights for a given prediction or
A pertinent positive (PP) is a factor that is minimally required conclusion at the end of the model. Heat maps are an example
for the justification of the final decision of the model. the of attribution maps. Fig. 9 explains how various feature set
pertinent negative is a factor whose absence is minimally with respect to different kernels in VGG 16 demonstrates heat
required for justifying the conclusion. Figure 6 denotes that map feature weights.
the plane body is critical minimum evidence for getting DeepTaylor [41] has provided an approach to generat-
classified as Warplane. ing specific positive evidence for a given prediction. The
The advanced approach is also under research which is deepTaylor approach of XAI is useful for justifying CNN-
known as SEDC-T. Where T stands for predefined target based classification. It explains without changing underlying
class, not just another class. In SEDC segments images architecture, this property makes it an effective XAI tool.
are removed until the predicted class is not changed but in DeepExplain provides a unified framework using gradient
SEDC-T segments are removed from Image until pre-defined and perturbation-based attribution methods [44] [52].
class is reached.SEDC-T gives a more detailed explanation DeepLIFT (Deep Learning Important FeaTures) is a tech-
of why the image is not predicted as a correct class rather nique based on decomposing the prediction of a neural net-
than just explaining the reason behind the prediction of the work for specific input. The entire backpropagation process
incorrect class. is observed along with observation of weight and bias on each
neuron on every layer of the entire architecture. Based on a
D. XAI AND HEALTHCARE variety of weights on neurons specific scores are assigned to
XAI and healthcare is an effective combo of digital tech- each feature of input [45].
nology. In the trend of AI-based diagnosis systems, trust Perturbation In this approach, input features are getting
over AI-based conclusions is a matter of serious concern. changed to observe the impact on final prediction at the end
Trust is an important factor for the perseverance of AI in the of the last layer in the neural network. Perturbation can be
medical and healthcare segment of the digital industry [42]. achieved by masking or editing certain input features and
If-else diagnosis models are inherently explainable because it observations are recorded as model start training using for-
consists of feature value sets and will assign a score based on ward pass and backward pass. This is similar to the sensitivity
a feature value of an instance case of health diagnosis. If-else- analysis performed in parametric control system models [50]
based explainable medical diagnosis systems are well suited [51].
for external symptomatic disease diagnosis [43]. Whether The sensitivity of each feature based on input variation is
a given deceased person is having asthma or not can be recorded. This continuous observation makes the XAI prac-
detected by checking whether the symptoms list of the person titioner justify different predictions at the end of the neural
having what amount of matching criteria with If-else based network. Rank assignment to various features is similar to
feature values. For example, if the patient is already having deep explain. one drawback of deeplift approach is computa-
a past history of respiratory illness and cough then there are tionally expensive. After each forward and backward pass of
higher chances of having asthma. a number of iterations, observation of sensitivity with respect
VOLUME 4, 2016 9
 Gohel et al.: Explainable AI: current status and future directions

FIGURE 9: Attributions of VGG-16 with images from Imagenet using the methods implemented in [42]

to features is recorded. Occlusion is an important technique

for extracting important features from given Image [46] [47].
It is a straightforward model to perform a model agnostic
approach that explores the latent feature ranking of a model.
All pixels occlusion is computationally expensive, hence 3
x 3 and 10 x 10 tiles are generally useful for occlusion [48]
[49]. Trade-off is there with respect to the size of tiles and
accuracy.

2) XAI for Health care applications

Brain Imaging CNN is a tool for accurate Image classifica-
tion. Features-based classification of Alzheimer’s using CNN FIGURE 10: The proposed segmentation architecture, con-
gives robust [64] classification and accuracy. sisting of a down sampling path (top)and an up sampling
Using post hoc analysis we can understand there is a path (bottom), interconnected by skip connections and by the
certain amount of overfitting is available due to certain fea- bottleneck layer. [66]
tures. In post model analysis it is possible to tweak certain
hyperparameters so that more accurate results are possible to
extract. Methods like Guided backpropagation (GBP), LRP, lungs CT scan, which are fed in pipeline individually and
and DeepShap are useful for brain imaging and classification combined through a convolutional LSTM layer. The architec-
[65]. ture of convolutional LSTM is described in figure 9. The re-
During surge of COVID 19 pandemic AI along with ex- sulting feature maps are then processed with downsampling.
plainability plays an important role in covid 19 diagnosis. Downsampling generates five sequences of dense blocks and
Major steps are depicted in Fig. 11. then squeezing-excitation is performed. At last, max-pooling
• Extraction of lung information from chest CT scan. operation is performed. At the end, six channel segmentation
• Classification of CT scans in the category of covid is generated for lobs and nonlungs area of CT scan as shown
positive and negative using convolution. in Fig. 11.
• Localization of lung symptoms like ground glass and .
crazy paving in CT scans.
• Provide well-documented justification [66]. VI. XAI AND TEXT
Fig. 10 explains neural network architecture of covid 19 In general, Natural language processing (NLP) systems are
detection model to test presence of covid 19 by analysing CT having inherent explainability. NLP major applications using
scan of lung. The sequence of 3 consecutive slices (224×224) machine learning are sentiment analysis, hate speech detec-
10 VOLUME 4, 2016
Gohel et al.: Explainable AI: current status and future directions

of the neighborhood or clustering-based approach.

Provenance: This approach is based on the reasoning
steps. It is thoroughly validated approach for reasoning based
derived justification. In this approach, final result is deriva-
tion from series of reasoning steps. This is best technique for
automatic question answer explanation [35].

B. VISUALIZATION TECHNIQUES FOR NLP

Presentation is a crucial segment of XAI as far as justification
is concerned for a naive person. There are many ways for a vi-
sualization based on chosen XAI approach or technique. For
an effective attention-based mechanism that gives weightage
to different features, the saliency map-based technique is an
FIGURE 11: Overview of the COVID-19 detection approach important tool that demonstrates scores of individual features
for CT scan classification as either COVID-19 positive or [36] [29]. Here, we have provided a detailed description of
negative. [66] different visualization techniques in fig.6.
Saliency: There is a strong correlation between feature
score-based justification and saliency-based visual presen-
tion, text summary generation [17] [18]. For all these appli- tation. There is many research-based demonstrations where
cations machine learning models are used like decision trees, saliency-based visualization technique is chosen. Saliency-
sequential modeling, logistic regression, a bag of words, based visualizations are popular because they present visu-
skip grams [19] [61] [62]. Due to the recent advancement ally perceptive explanations and can be easily understood by
in word embeddings, It gives more efficiency to black-box- different types of end-users [24].
based inferences and conclusions [20]. One drawback of this Raw declarative representations: This technique is
increased efficiency is these models are less interpretative based on the presentation of logic rules, trees, and programs.
and less explainable. Digital ethics are a big concern as far It contains sequential derivation based on logic rules [40].
as reliability is concerned over black-box systems. Hence Natural language explanation: In this explainable tech-
Explainable AI makes more sense for NLP-based applica- nique, the explanation is provided in more comprehensive
tions of AI and Deep Learning [21] [22]. Figure 7 consists of natural language [38]. It is an application of the generative
various explanation techniques for NLP like a saliency heat neural network model where natural language sentences are
map, saliency highlights, declarative explanation, and natural generated by the NN (Neural network) model [30]. For this
language inference. purpose, sophisticated and dedicated models of particular
domains eg. pharma, medical, crime, etc [39]. are trained and
A. EXPLAINABILITY TECHNIQUES FOR NLP deployed in production. This model is usually known as the
There are five different techniques that are useful for provid- generative model. Fig. 12 and Fig. 13 are describing saliency
ing mathematical justification for the conclusion and classifi- based highlighting and POS based tags for visualization.
cation of AI-based NLP system [27] [55].
Feature importance: This technique is based on the C. XAI AND HATE SPEECH DETECTION
weight of various features based on feature engineering con- In this section, we are explaining hate speech detection using
cepts. Specific scores are going to be assigned to individual XAI. This section demonstrates explainability techniques for
features based on their contribution to the final prediction. hate speech detection using XAI [60] [63]. There are text
This technique is based on various features of NLP. Some classification based techniques which are useful for providing
features are handcrafted or annotated, which are extracted more insights about trained model for hate speech detection
manually by feature engineering [31], lexical features based and used data set for training [54] [57] [58]. This projected
on tokens and n-gram [33], or latent features using LDA [34] insights are useful to make trained model more accurate
and Attention mechanism [32]. Text-based features are more for hate speech detection. Fig. 14 shows saliency map for
convenient for the human to understand just because they classification of hate vs offensive speech. Fig. 14 shows how
in form of a readable text. There are certain disadvantages certain key words draws classification towards specific class
also with hand-crafted features due to their local optimum of hate speech with higher weight.
derivations. Shapley values are useful for feature importance based
Example driven: In an example-driven approach, exam- analysis of hate speech. Feature importance map like saliency
ples of text being provided are in favor of the final conclusion maps are useful for visualization. Gradient explainer based
and some are against the final conclusion. This approach approach is also useful but generate feature independence
leads to an instance and label base justification for given based justification.
prediction and conclusion. They are similar in the essence
VOLUME 4, 2016 11
 Gohel et al.: Explainable AI: current status and future directions

FIGURE 12: Saliency highlighting [37]

FIGURE 13: Visualizaton of POS tags [81]

activation function for backward propagation. LRP is struc-

tured as a tool for pixel-wise decomposition for relevance to
a decision. LRP satisfies certain conditions for developing
justifications as below.
• The relevance difference of each layer must converge
must sum at the final layer of the model.
• The relevance difference at any neuron of a given layer
apart from the final layer is the sum of incoming rele-
vance differences to that layer.
FIGURE 14: Directed hate misclassified as offensive lan-
Other techniques like LIME are useful to provide justifi-
guage with very high confidence(94%). The two words
cations to develop explanations for models without accessing
“bitch” and “faggot” are the two main positive contributors to
internal structure and weights. LIME approximates decisions
the score. Although the two words are indeed offensive, they
by many sampled inputs. LIME is effectively applicable to
misdirect the classifier which misses the clear hate emerging
data like text, image, and tabular data.
from the tweet [53].
The most popular of these techniques, known as Local
Interpretable Model Explanations or LIME, seeks to ap-
proximate the decision function by many closely sampled
VII. XAI AND VIDEO input points, which all center around the input point to be
Local optimum justifications are an effective technique for explained. It can then attribute positive or negative influence
the image domain. A model agnostic technique like LIME on the decision function to the differences in the sampled in-
shows a good success rate using local explanations. . For puts, and overlay this on the original input.
video analysis and explainability, frame-wise decomposition
of video is applied. VIII. XAI AND AUDIO
LRP is another popular technique. LRP assumes that it In this area, much research is yet to be done. For linguistic
can access the internal architecture of a given complex neu- applications, audio waves are converted into text form. After
ral network. LRP access model’s internal weight, bias, and converting into text, NLP-based XAI techniques are applica-
12 VOLUME 4, 2016
Gohel et al.: Explainable AI: current status and future directions

ble. Such techniques comprise lime, perturbation, SHAP and In the aviation sector, aircraft maintenance is handled by
taxonomy-based inductions [79] [56]. scheduled or event-based triggered maintenance. Such sort of
Now a days, auto speech recognition powered voice as- maintenance is unreliable because that causes serious disaster
sistant like alexa and siri are being used more frequently by when aircraft is in the air. Such disasters can be prevented if
users [78]. Audio waveform based key word classification for predictive maintenance is applied. In any giant mechanical
virtual agents is more convincing along with visual presence system, there is a gradual degradation in the reading of
of agent rather than only voice or text based output [74]. various tools or sensors, these tools are making a cumulative
It is observed that the visual presence of virtual agents in effect for final break down [77]. Such calibration of readings
graphical 2d or 3d forms develops trust in the XAI systems. can be used as a feature set and model training is possible for
To evaluate this observation, a user study is conducted in failure diagnosis and remaining useful life (RUL) prediction
which a virtual agent demonstrates XAI visualization of a of aircraft or any critical electro-mechanical system.
neural network-based speech recognition model. This model The prediction of failure with explanation makes the jus-
classifies audio keywords with respect to their spectrograms. tification for derived diagnosis. Hence it improves reliability
In this study, users are classified into three groups. First, and saves cost [67].
interact with the text. Second, interact with voice, and Third, Maintenance with interpretability for failure diagnosis can
interact with virtual agents [74]. The results show that the add useful insights about the disposal of certain parts which
visual appearance of an agent gains more trust rather than might not be available with crew members of maintenance
only text or voice-based interactions. teams. The pipeline of predictive maintenance with XAI
LIME framework is applied to generate XAI visualization insights is mentioned in Fig. 16. of sequential steps like
to understand voice classification. Model agnostic character- data collection, data cleaning, feature selection, diagnosis,
istics of LIME make it applicable to any sort of input data. and explanation with validation. One advantage of predictive
Fig 15 shows XAI visualization of the keyword "House". maintenance it helps to mandatory understanding of different
physical components and their physical properties.

X. CONCLUSION
It is emphasized here that XAI is an important and manda-
tory aspect of AI/ML based application to use in real time.
Our study has started discussion from conventional AI and
limitations. The need for XAI is well explained in the case of
studies along with key issues of explainable AI.
Objectives and scopes of XAI are discussed in length and
breadth. We discussed major objectives like transparency,
fairness, bias, and confidence. Scope of XAI is discussed
in detail for its application in the major domain like NLP,
medical, defense and Engineering.
Different methodologies (post hoc and transparent) for
FIGURE 15: A spectrogram of an audio sample (left), its
explainability are discussed to get preliminary hands on to
segmentation into superpixels (center) and the output for the
dive into this field. Conceptual and detailed explanations with
user containing LIME visualisations and additional phoneme
the example for all methodologies are also discussed. After
information (right) [74]
providing a conceptual understanding of XAI approaches, we
have provided XAI as a tool to be applied to specific kinds of
IX. XAI AND MULTIMODAL DATA data like image, text, video, audio, and multimodal data.
Some time input data may differ from conventional input data This survey elaborates a conceptual understanding of XAI
like audio, video, text, and image. If data is in CSV format, it along with the importance of explainability that motivates
requires different pre-processing of data and normalization. researchers for diversified aspects of XAI. This purpose mo-
For example, in industry sensor data are calibrated in CSV tivates researchers for interpretable AI/ML methods. These
datasheets. detailed highlights make a baseline for the understanding of
In industrial automation, maintenance is one of the crucial the current literature of XAI, which can be approached in
aspects for the continuity of industry. Due to various physical two ways. 1) Transparent ML models which are interpretable
parameters like temperature, vibration, pressure, RPM, etc., to an extent by themselves only. 2) Post hoc methods for
there are significant impacts on various parts of the assembly explainability which makes the model more interpretable. We
line or mechanical system which leads to failure [75]. XAI presented XAI as a tool for responsible AI, a paradigm that
along with failure diagnosis makes the ML model more can enable series of algorithms that will work in synergy to
transparent and interpretable towards the provided diagnosis achieve the goal of responsible AI. Responsible AI stands for
of the failed component [76]. trust, confidence, fairness, and transparency.
VOLUME 4, 2016 13
 Gohel et al.: Explainable AI: current status and future directions

FIGURE 16: Functional diagram of an automated failure diagnosis model [67]

REFERENCES tational Linguistics (Volume 1: Long Papers),pages 950–962, Vancouver,

[1] A. Adadi and M. Berrada, "Peeking Inside the Black-Box: A Survey on Canada. Association for Computational Linguistics
Explainable Artificial Intelligence (XAI)," in IEEE Access, vol. 6, pp. [17] Nikos Voskarides, Edgar Meij, Manos Tsagkias,Maarten de Rijke, and
52138-52160, 2018, doi: 10.1109/ACCESS.2018.2870052. Wouter Weerkamp. 2015.Learning to explain entity relationships in knowl-
[2] Marco Tulio Ribeiro,Sameer Singh, Carlos Guestrin,“Why Should edge graphs. In Proceedings of the 53rd Annual Meeting of the Association
I Trust You?”Explaining the Predictions of Any Classifier," for Computational Linguistics and the 7th International Joint Confer-
https://arxiv.org/pdf/1602.04938.pdf enceon Natural Language Processing
[3] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model [18] Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion
predictions. In Advances in Neural Information Processing Systems, pages Jones, Aidan N. Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. At-
4765–4774, 2017. tention is all you need. In NeuralIPS
[4] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick [19] Martin Tutek and JanˇSnajder. 2018. Iterative recursive attention model
Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise for interpretable sequence classification. In Proceedings of the 2018
explanations for non-linear classifier decisions by layer-wise relevance EMNLP Work-shop BlackboxNLP: Analyzing and Interpreting Neural
propagation. PloS one, 10(7):e0130140, 2015. Networks for NLP, Brussels, Belgium. Association for Computational
[5] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang and Y. Gong, "Locality- Linguistics.
constrained Linear Coding for image classification," 2010 IEEE Computer [20] James Thorne,Andreas Vlachos,Christos Christodoulopoulos, and Arpit
Society Conference on Computer Vision and Pattern Recognition, San Mittal. 2019. Gener-ating token-level explanations for natural language
Francisco, CA, 2010, pp. 3360-3367, doi: 10.1109/CVPR.2010.5540018. inference. In Proceedings of the 2019 Conference of the North Amer-
[6] Lundberg, Scott M., Gabriel G. Erion, and Su-In Lee. "Consistent ican Chapter of the Association for Computational Linguistics: Human
individualized feature attribution for tree ensembles." arXiv preprint Language Technologies, Volume 1 (Long and Short Papers), Minneapolis,
arXiv:1802.03888 (2018). Minnesota. Association for Computational Linguistics.
[7] Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun [21] Robert Schwarzenberg, David Harbecke, Vivien Mack-etanz, Eleftherios
Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on Avramidis, and Sebastian Moller.2019. Train, sort, explain: Learning to
the missing: Towards contrastive explanations with pertinent negatives. diagnose translation models. In Proceedings of the 2019 Conference of the
In Advances in Neural Information Processing Systems, pages 592–603, North American Chapter of the Association for Computational Linguistics
2018 (Demonstrations)
[8] Vermeire, Tom and D. Martens. “Explainable Image Classification with [22] Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable ? In Pro-
Evidence Counterfactual.” ArXiv abs/2004.07511 (2020): n. pag. ceedings of the 57th Annual Meeting of the Association for Computational
[9] Lloyd S Shapley. “A value for n-person games”. In:Contributions to the Linguistics, Florence, Italy. Association for Computational Linguistics.
Theory of Games2.28 (1953), pp. 307–317 [23] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep
[10] Vaishak Belle Ioannis Papantonis.,Principles and Practice of Explainable inside convolutional networks: Visualising image classification models
Machine Learning*, Sep 2020 and saliency maps. arXiv preprint arXiv:1312.6034
[11] Cynthia Rudin, Stop explaining black box machine learning models for [24] Nina Poerner, Hinrich Schutze, and Benjamin Roth.2018. Evaluating
high stakes decisions and use interpretable models instead, neural network explanation methods using hybrid documents and morpho
[12] Igami. (2017). “Artificial intelligence as structural estimation: Eco-nomic syntactic agreement. In Proceedings of the 56th Annual Meeting of the
interpretations of deep blue, bonanza, and AlphaGo.” Association for Computational Linguistics (Volume 1: Long Papers),
[13] A. Neerincx, J. van der Waa, F. Kaptein, and J. van Diggelen, “Using per- Melbourne, Australia. Association for Computational Linguistics.
ceptual and cognitive explanations for enhanced human-agent team perfor- [25] Nicolas Prollochs, Stefan Feuerriegel, and Dirk Neumann. 2019. Learning
mance,” in Proc. Int. Conf. Eng. Psychol. Cogn. Ergonom. (EPCE),2018, interpretable negation rules via weak supervision at document level: A
pp. 204–214. reinforcement learning approach. In Proceedings of the 2019Conference
[14] J. C. Garcia, D. A. Robb, X. Liu, A. Laskov, P. Patron, and H. of the North American Chapter of the Association for Computational
Hastie,“Explain yourself: A natural language interface for scrutable au- Linguistics: Human Language Technologies, Volume 1 (Long and Short
tonomous robots,” inProc. Explainable Robot. Syst. Workshop HRI, 2018 Papers), Minneapolis, Minnesota.Association for Computational Linguis-
[15] Mantong Zhou, Minlie Huang, and Xiaoyan Zhu.2018. An interpretable tics.
reasoning network for multi-relation question answering.In Proceedings [26] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard
of the 27th International Conference on Computational Linguistics, Socher. 2019b. Explain your-self! leveraging language models for com-
[16] Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy.2017. An in- mon sense reasoning.arXiv preprint arXiv:1906.02361.
terpretable knowledge transfer model for knowledge base completion.In [27] Reid Pryzant, Sugato Basu, and Kazoo Sone. 2018a. Interpretable neural
Proceedings of the 55 th Annual Meeting of the Association for Compu- architectures for attributing anad’s performance to its writing style. In

14 VOLUME 4, 2016
Gohel et al.: Explainable AI: current status and future directions

Proceedings of the 2018 EMNLP Workshop Blackbox NLP: Analyzing [50] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. Ex-
and Interpreting Neural Networks for NLP, Brussels, Belgium. Association plaining explanations: An overview of interpretability of machine learning.
for Computational Linguistics. In 2018 IEEE 5th International Conference on Data Science and Advanced
[28] Piyawat Lertvittayakumjorn and Francesca Toni. 2019.Human-grounded Analytics (DSAA), pages 80–89, 2018
evaluations of explanation methods for text classification. In Proceedings [51] A. Fernandez, F. Herrera, O. Cordon, M. Jose del Jesus, and F. Marcel-loni.
of the 2019 Conference on Empirical Methods in Natural Language Evolutionary fuzzy systems for explainable artificial intelligence:Why,
Processing and the 9th International Joint Conference on Natural Lan- when, what for, and where to?IEEE Computational Intelligence Magazine,
guage Processing (EMNLP-IJCNLP). Hong Kong,China. Association for 14(1):69–81, Feb 2019
Computational Linguistics. [52] Grgoire Montavon, Wojciech Samek, and Klaus-Robert Mller. Methodsfor
[29] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2015. Visualizing interpreting and understanding deep neural networks.Digital Signal Pro-
and understanding neural models in nlp.arXiv preprint arXiv:1506.01066. cessing, 73:1 – 15, 2018.
[30] Qiuchi Li, Benyou Wang, and Massimo Melucci. 2019.CNM: An inter- [53] Edoardo Mosca,Explainability of Hate Speech Detection Models, Technis-
pretable complex valued network for matching. In Proceedings of the che Universit at Munchen, Department of Mathematics
2019 Conference of the North American Chapter of the Association [54] Arras, L., Montavon, G., Muller, & Samek, W. (2017). Explaining recur-
for Computational Linguistics: Human Language Technologies, Volume rent neural network predictions in sentiment analysis. In Proceedings of
1 (Long and Short Papers),pages 4139–4148, Minneapolis, Minnesota. the 8th workshop on computational approaches to subjectivity, sentiment
Association for Computational Linguistics. and social media analysis
[31] Vijay N. Garla, Cynthia Brandt, Ontology-guided feature engineer- [55] Baziotis, C., Pelekis, N., & Doulkeridis, C. (2017). Datastories at semeval-
ing for clinical text classification, Journal of Biomedical Informat- 2017 task 4:Deep lstm with attention for message-level and topic-based
ics, Volume 45, Issue 5,2012, Pages 992-998, ISSN 1532-0464, sentiment analysis. In Proceedings of the 11th international workshop on
https://doi.org/10.1016/j.jbi.2012.04.010. semantic evaluation. (semeval-2017)(pp. 747–754)
[32] Minh-Thang Luong, Hieu Pham, Christopher D. Manning, Effective Ap- [56] Chen, Y., Zhou, Y., Zhu, S., & Xu, H. (2012). Detecting offensive language
proaches to Attention-based Neural Machine Translation in social media to protect adolescent online safety. In2012 international
[33] Ashish Sureka, Pankaj Jalote ,Detecting Duplicate Bug Report Using conference on privacy, security, risk and trust and 2012 international
Character N-Gram-Based Features conference on social computing
[34] P M Prihatini, I K Suryawan, IN Mandia. Feature extraction for document [57] Chatzakou, D., Kourtellis, N., Blackburn, J., De Cristofaro, E., Stringhini,
text using Latent Dirichlet Allocation G., & Vakali,A. (2017). Mean birds: Detecting aggression and bullying on
[35] Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard twitter. In Proceedings of the 2017 acm on web science conference(pp.
Weikum. 2017. Automated Template Generation for Question Answering 13–22).
over Knowledge Graphs. In Proceedings of the 26th International Confer- [58] Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017). Automated
ence on World Wide Web (WWW ’17). International World Wide Web hate speech detection and the problem of offensive language. In Eleventh
Conferences Steering Committee, Republic and Canton of Geneva, CHE, international aaai conference on web and social media
1191–1200. DOI:https://doi.org/10.1145/3038912.3052583 [59] Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018). Mea-
[36] Mahnaz Koupaee, William Yang Wang, IN Mandia. Analyzing and Inter- suring and mitigating unintended bias in text classification. In Proceedings
preting Convolutional Neural Networks in NLP. of the 2018 aaai/acm conference on ai, ethics, and society(pp. 67–73).
[60] Online harassment 2017. Pew Research Center
[37] Lin, Zhouhan, et al. "A structured self-attentive sentence embedding."
[61] Founta, A. M., Chatzakou, D., Kourtellis, N., Blackburn, J., Vakali, A.,
arXiv preprint arXiv:1703.03130 (2017).
& Leontiadis, I.(2019). A unified deep learning architecture for abuse
[38] REITER, E., & DALE, R. (1997). Building applied natural lan-
detection. In Proceedings of the 10 th acm conference on web science(pp.
guage generation systems. Natural Language Engineering, 3(1), 57-87.
105–114).
doi:10.1017/S1351324997001502
[62] Glorot, X., Bordes, A., & Bengio, Y. (2011). Domain adaptation for large-
[39] Dani Yogatama, Chris Dyer, Wang Ling, and Phil Blunsom, deep mind,
scale sentiment classification: A deep learning approach. In Proceedings of
Generative and Discriminative Text Classificationwith Recurrent Neural
the 28th international conference on international conference on machine
Networks
learning(513–520).
[40] Dani Yogatama, Chris Dyer, Wang Ling, and gren, H. & Nieves, J.C.
[63] Mathew, B., Dutt, R., Goyal, P., & Mukherjee, A. (2019). Spread of hate
A dialogue-based approach for dealing with uncertain and conflicting
speech in onlinesocial media. InProceedings of the 10th acm conference
information in medical diagnosis. Auton Agent Multi-Agent Syst 32,
on web science(pp. 173–182).
861–885 (2018). https://doi.org/10.1007/s10458-018-9396-x
[64] Eitel, F.; Ritter, K.; Alzheimer’s Disease Neuro imaging Initiative (ADNI).
[41] Montavon, G.; Lapuschkin, S.; Binder, A.; Samek, W.; Müller, Phil Blun- Testing the Robustness of Attribution Methods for Convolutional Neural
som, deep mind, Generative and Discriminative Text Classificationwith Networks in MRI-Based Alzheimer’s Disease Classification. In Inter-
Recurrent Neural Networks pretability of Machine Intelligence in Medical Image Computing and
[42] Amitojdeep Singh, Sourya Sengupta,Vasudevan Lakshminarayanan, Multimodal Learning for Clinical Decision Support, ML-CDS 2019, IM-
Explainable Deep Learning Models in Medical Image Analy- IMIC 2019; Lecture Notes in Computer Science; Suzuki, K., et al., Eds.;
sis,https://doi.org/10.3390/jimaging6060052 Springer: Cham, Switzerland, 2019; Volume 11797.
[43] Yan, C., LindK.R. Explaining nonlinear classification decisions with deep [65] Pereira, S.; Meier, R.; Alves, V.; Reyes, M.; Silva, C.A. Automatic brain
taylor decomposition. Pattern Recognit. 2017, 65, 211–222. tumor grading from MRI data using convolutional neural networks and
[44] Ancona, M.; Ceolini, E.; Öztireli, C.; Gross, M. Towards better under- quality assessment. In Understanding and Interpreting Machine Learning
standing of gradient-based attribution methods for deep neural networks. in Medical Image Computing Applications; Springer: Cham, Switzerland,
arXiv 2017, arXiv:1711.06104. 2018; pp. 106–114.
[45] Avanti Shrikumar, Peyton Greenside, Anshul Kundaje,Learning [66] Matteo Pennisi, Isaak Kavasidis, Concetto Spampinato, Vincenzo Schin-
Important Features Through Propagating Activation Differences. inà, Simone Palazzo, Francesco Rundo, Massimo Cristofaro, Paolo Cam-
https://arxiv.org/abs/1704.02685 pioni, Elisa Pianura, Federica Di Stefano, Ada Petrone, Fabrizio Albarello,
[46] Erico Tjoa, and Cuntai Guan,Fellow, IEEE, A Survey on Giuseppe Ippolito, Salvatore Cuzzocrea, Sabrina Conoci, An Explainable
Explainable Artificial Intelligence(XAI): towards Medical XAI. AI System for Automated COVID-19 Assessment and Lesion Categoriza-
https://arxiv.org/pdf/1907.07374.pdf tion from CT-scans, https://arxiv.org/abs/2101.11943
[47] Liam Hiley, Alun Preece, Yulia Hicks, Supriyo Chakraborty, Prudhvi [67] MSophie ten Zeldam, Arjan de Jong, Richard Loenders loot and Tiedo
Gurram, and Richard Tomsett. Explaining motion relevance for activity Tinga, Automated Failure Diagnosis in Aviation Maintenance UsingeX-
recognition in video deep learning models, 2020 plainable Artificial Intelligence (XAI)
[48] Doshi-Velez and Been Kim. Towards a rigorous science of interpretable [68] Mashrur Chowdhury, Adel W. Sadek, Advantages and Limitations of
machine learning, 2017. cite arXiv:1702.0860 Artificial Intelligence.
[49] Sebastian Lapuschkin, Stephan Waldchen, Alexander Binder, Gregoir [69] Christopher Rigano, NIJ, USING ARTIFICIAL INTELLIGENCE TO
eMontavon, Wojciech Samek, and Klaus-Robert Muller. Unmasking ADDRESS CRIMINAL JUSTICE NEEDS.
clever hans predictors and assessing what machines really learn.Nature [70] Robbins, Mark D.. (2019). AI Explainability Regulations and Responsi-
Communications, 10(1):1096, 2019. bilities.

VOLUME 4, 2016 15
 Gohel et al.: Explainable AI: current status and future directions

[71] Y. X. Zhong, "A Cognitive Approach to Artificial Intelligence Research,"

2006 5th IEEE International Conference on Cognitive Informatics, Bei-
jing, China, 2006, pp. 90-100, doi: 10.1109/COGINF.2006.365682.
[72] Arun Das,Graduate Student Member, IEEE,and Paul Rad,Senior Member,
IEEE,Opportunities and Challenges in ExplainableArtificial Intelligence
(XAI): A Survey",doi : https://arxiv.org/pdf/2006.11371.pdf
[73] "Why Should I Trust You?": Explaining the Predictions of Any
Classifier,Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin,
https://arxiv.org/abs/1602.04938
[74] “Let me explain!”: exploring the potential of virtual agents inexplain-
able AI interaction design, Katharina Weitz,Dominik Schiller,Ruben
Schlagowski,Tobias Huber, Elisabeth Andre
[75] Hrnjica, Bahrudin & Softic, Selver. (2020). Explainable AI in Manufactur-
ing: A Predictive Maintenance Case Study.
[76] Shukla, Bibhudhendu & Fan, Ip-Shing & Jennions, I.K.. (2020). Oppor-
tunities for Explainable Artificial Intelligence in Aerospace Predictive
Maintenance.
[77] S. Matzka, "Explainable Artificial Intelligence for Predictive Maintenance
Applications," 2020, (AI4I), doi: 10.1109/AI4I49448.2020.00023
[78] S. J. du Preez, M. Lall and S. Sinha, "An intelligent web-based voice
chat bot," IEEE EUROCON 2009, 2009, pp. 386-391, doi: 10.1109/EUR-
CON.2009.5167660.
[79] Nuobei Shi, Qin Zeng, Raymond Lee,The design and implementation
of Language Learning Chatbot with XAI using Ontology and Transfer
Learning, (NLPD 2020),doi: https://arxiv.org/abs/2009.13984
[80] Alejandro Barredo Arrieta et al.,Explainable Artificial Intelligence (XAI):
Concepts, Taxonomies, Opportunities and Challenges toward Responsible
AI, DOI: 10.1016/j.inffus.2019.12.012.
[81] visualization to output the fine-grained part-of-speech tags,
https://spacy.io/usage/rule-based-matching

16 VOLUME 4, 2016