sahilp read this https://www.researchgate.

net/profile/MennatallahE l−
Assady/publication/332802468T owardsX AIS tructuringt heP rocesseso fE xplanations/links/5ccad56b92851c8d22146613/T owards−
XAI−Structuring−the−P rocesses−of −Explanations.pdf
 XAI eXplainable Artificial Intelligence
[Visual Data Exploration Seminar]
Sahil Pasricha
Department of Computer Science
University of Konstanz
Konstanz, Germany
[email protected]

ABSTRACT impact on the decision model.

With the advent of cheap processing power, Neural networks The analysts know the fact that these Neural Networks are
have acquired the mainstream status. They have proved to not even near to 100 percent accuracy but this seems to have
be useful in noncritical domains like shopping and, enter- no impact on its usage but, what actually has been hurting
tainment but, are yet unreliable for critical applications like its capability is, its missing ability to explain its decision
healthcare and autonomous driving. This is due to the fact neither when machine made a right decision nor when it
that despite attaining considerable accuracy, AI algorithms made a wrong one. This makes the Neural Networks not
often are black boxes missing an capability to justify their just black box but also limits the chances to increase the
decision. accuracy beyond a certain point.
We aim to survey various visualization and interaction Our goal here is to offer analyst, an informative insights to
techniques that enable understanding and diagnosis of the analyze huge information spaces in order to support decision
lower level features of neural networks, namely its weights, making by deep analysis of data models. by compressing
activation or data-flow in yet summarized fashion. Based on some least relevant features and highlighting features which
our understanding of them, we will than define an evaluation have more impact on decision making .
criteria to contrast and compare the selected approaches. There have already been considerable research in this area
with focus on different domain , data-set and models.We
will analyse all the developed techniques for visualization
Keywords and will also identify their limitations and possible alter-
Visualization; Interactive Machine Learning(IML), Deep Neu- nate/resolution through other model/add-on. We are pri-
ral Networks(DNN) marily targeting the visualization and interaction through
the diagnosis of lower level features of an DNN model. These
features include weights , activation and data flow all of
1. INTRODUCTION which have an considerable influence on output and same
At last, Its just about presenting a mathematical model to can be calculated using back propagation method, as rep-
an analyst who is already a domain expert, why it’s such a resented in figure 1. A thicker connection from the input
complex task ? The neural network models are humongous, layer shows the corresponding input node is more important
an average neural network contains hundreds of neurons and to the final result. There is also a threshold that hides the
tens of hidden layers, total possible combinations in this case connections that are less important. Based on this thresh-
may exceed a hundred thousand. Presenting the analyst old, of hidden nodes, four less important hidden nodes are
with these numbers for analysis may mystify the analyst removed from the neural network after it is trained. This
rather then helping him. reduction resulted in 15 percent reduction in cost while im-
The power of AI lies in its capability to make decisions pacting the output by just 0.5 % A more important connec-
based on features adapted during learning process. With tion has higher weights on it, and its connected nodes have
advent of new technology, Neural Networks have witnessed a higher impact to the output result, which is also valuable
considerable improvement in accuracy. A widespread busi- information about the data.
nesses and research domains are now counting on machines In this paper we focus on methods that adds following
to help them in analysis and decision making. Overall, the factors to the NN models
role of AI in decision making as well as, the impact these Transparency - The ability of a model to explain its final
machine made decisions have, both are rising. results, whether it predicted them right or wrong and factors
The accuracy of a model is an important factor to deter- impacting it
mine the precision of a model but at same time, the unex- Reliability - The measure of accuracy of model which helps
plained decision of model leaves it unreliable for fields like user estimate its precision
autonomous driving. In order to build this trust, a model Interaction - Once the model had generated results, user
shall either produce accurate results or explain the reasons shall be able to Interact with model from UI.
behind these results. We will also study the different techniques used by vari-
Currently, due to the missing transparency, an analyst has ous models to represent data, this includes data reduction ,
to manually run, time consuming ”hit and trial” on features grouping nodes, presenting only essential features and show
he think could help improve accuracy. XAI can here play detailed feature on demand, removing representation that
a crucial role in informing analysts about factors and their
 needs of any particular domain or its independent of any
data set. The former one build representation model for
features by highlighting the aspects to data processing stem
of model. Foe example - use of heat maps to represent the
weight of a feature and its ration in final output.
Some data flow systems use visualizations as authoring
tools and allow users to directly edit the graph to modify
the data-flow. Only high level components are represented
in data flow are manually added one-by-one, their graphs
are typically much smaller compared to dataflow graphs of
deep neural networks.

Figure 1: Coloring a model based on relavance.[13]

2.2 Interactive and Visual AI
By adding interactivity to the AI , the information for-
aging process takes shape of a cyclic sense making model
are below threshold (for weight , output ,etc), color and where, and analyst trigger the flow of model , check outputs,
shape encoding and many more. Also these methods vary make necessary changes and re triggers the flow. These it-
largely in terms of their behaviour, while some focus on in- erations are aimed at enhancing the accuracy of model.
put/output other focus on layers or weights. The overall focus is to make the user aware of the features
that are responsible for any given results and at same time
2. VISUALIZING ANN making this visualization concise and relevant so that user
In a nutshell,ANN utilize multiple layers of mathemati- shall not be bombarded with information nor shall any im-
cal processing to make model of the information itâĂŹs fed portant visualising be left hidden from user. This can be
(train data). This model is then used to make sense of test done using three concepts as core of visualization. First,
data. Typically, an ANN [1] has anywhere from dozens to the non important nodes/features shall be identified using
weight, activation and data flow followed by exploration
millions of artificial neuronsâĂŤcalled unitsâĂŤarranged in
techniques provided to user if he want to dive deep into any
a series of layers. The input layer is responsible for receiving
major feature, this enables him to understand the model to
information from the outside world. This is the data that
granular level and thirdly, the user shall be allowed to inter-
the network aims to process. From the input neurons, the
act within this UI framework where he can make use of rich
data goes through one or more hidden units. The hidden
visualization to make calculated guesses.
unitâĂŹs job is to transform the input into something the
output unit can use. In this survey paper , we are target- 2.2.1 XAI Optimizing Factors
ing the visualization techniques that are more concentrated
Even after removing non critical nodes, the data is still
on diagnosis of lower level feature of model which primarily
enough huge to be understood by human brain. To further
includes weights and activation’s [6].
optimize the Visualization model, we being to use some fea-
To understand XAI , lets start with understanding the
tures that may help compress data in a way that its hidden
combination that lays the foundation of XAI. Interactive
at first and can be made available on demand. This helps
machine learning is all about adding human in the loop of
keeping the initial visualization size minimum with compro-
information processing. This includes a loop of building
mising the efficiency of model to present subtle details.
, training and correcting process. Visual Analytic is then
build on top of IML and facilitates the workflow by making • Details On Demand
user more aware of data and at same time giving him more
control over data. Some VA system goes beyond the diag- • Zoom
nosis and lets their visualization helps the user to not just • Overview First
understand but also optimize the model. This refinement is
assisted by VA model by detecting problem is model. Visual Analytics Mantra : Analyse First - Show the Impor-
tant - Zoom, Filter and Analyse Further - Details On De-
2.1 Related Works mand. Both start with an overview/aggregation approach
To design a model framework, we surveyed a number of and end in a refinement of their hypothesis and analysis.[10]
visualization techniques and XAI models along with their In order to achieve these steps, some graph compression
limitations, scope and mainly the abstraction process they mechanics can be considered which are further discussed in
follow to represent the information. we also studied the next section
existing Interactive machine learning tools.In this survey we
focus on subset of prominent explainers for image domain. 2.2.2 Abstraction Level
The whole concept of XAI rests on the balance of 3 factors
2.1.1 Approaches for XAI namely - what to show , what to hide and what to remove.
Local Interpret-able Model Agnostic Explanations (LIME) A too high abstraction level with compress the model to an
[1], it used linear model to explain the complex models. An- extent that it may end up hiding even critical information
other model agnostic approach , ANCHORS used same ap- while, a highly low level abstraction will almost nullify the
proach but additionally focuses on most influential ahead impact of abstraction by overloading the human with data
(Anchors) to explain decision rules.[5] huge enough to fit a mental map. Some model make use
One important factor to consider while evaluating these of threshold, a neuron’s contribution in output to decide
models is to check if the model is specially tailored to meet weather or not to show that entity on model. Abstraction
level also varies in terms of data representation. Information
Flow model clusters data on base of features and represent
low level details at that level (Figure 6) while Image Inter-
preter Model presents the low level details at level (Figure
5) of individual details.

2.2.3 Overlying Details

Once abstraction details are fixed, the question arises, Figure 2: LIME,explaining individual predictions.[9]
how to represent the details on demand. This refers to
that that is not ”essentially critical” but ”possibly critical”.
This may includes some deep level features like weight and model’s composition.While information flow is model depen-
histo trend. This area is a hot topic of research and goal dent but it is independent of domain.
is to present maximum possible information and most un-
derstandable style. Most of models use hover to present ad- 3.1.2 Model Independent Explainers
ditional details and mouse click to expand and compressed This includes XAI techniques that are structured on either
cluster.Information Flow model makes use of color coding data or domain.
to represent a neuron’s relevance to output and LIME, An- LIME [8] is one such method , its core function is to ex-
chors use color coding to represent the sentiments of a given plain predictions of classifiers by approximating it locally
word (Refer to figure 3). with an interpret-able model. The outputs of LIME are
then to be reviewed by domain expert to check accuracy
of model and tweak its features to optimize same. Also as
3. SURVEY: VISUALIZATION TECHNIQUES LIME works only on prediction analysis its independent of
For this survey, we analyzed 10 research papers directly domain and relies solely on data.LIME supports interpret
from the field of XAI to evaluate various visualization mod- able explanations i.e. representation that is understandable
els, their scope and their dependency on factors like data set to humans, regardless of the actual features used by the
and domain The focus here is to survey various visualization model.
techniques that have been recently researched for the field of While LIME has done a great job in presenting the expla-
AI. We used the dependency as primary parameter as the to nations of outputs , its more of a input-output model and
classify them. For this survey the latest works in this filed rather then a complete network exploration model. Also, its
from Machine Learning, HCI and visualization societies. We dependency on data leaves it non scalable for newly added
analyzed 10 papers directly from the filed of XAI. data to the stream.
We have defined dependency as primary criteria to define
standard for evaluation measure and compare various visu- 3.2 Domain Dependency
alization techniques based on this parameter. we recognized This covers relationship of XAI methods with their do-
the following three dimensions of dependency to be most mains. While some methods are tailored to fit needs of one
significant for our purposes of organizing XAI evaluation specific domain , other general models are designed keep-
methods: ing in mind the diversity of domains. These are broadly
classified into 2 categories.
3.1 Model Dependency
Refer to parameter to distinguish if given XAI technique 3.2.1 Domain Dependent
is based on AI model or if it is completely independent of These techniques are specialized models for relevant do-
it. This further has various combinations with other two main and further have subclass based on their dependency
evaluation methods i.e. Data and Domain that will be fur- on data and network model. Hence need a domain expert
ther briefed in later section of this paper. Based on this to tweak the model further. first one amongst them is Deep
parameter, the two possible sub category of models are Taylor [7] this technique is domain, model and data depen-
dent. It’s specifically designed for non linear Neural Network
3.1.1 Model Dependent Explainers models, which in general are hard to visualize due to multi
These models are primarily based on fixed features of a layer nonlinear structure. The models Aims at using input
ANN model that includes weights , biased and activation elements to reduce network classification decision. This is
function. Therefore the primary focus to highlight the de- done using propagation.
ciding factors and their impact in final decision making. One
such model is DeepLIFT [12], (Deep Learning Important 3.2.2 Domain Independent
FeaTures), a model, data and domain dependent model for ANCHORS[2] a domain, model independent method that
decomposing the output prediction of a neural network on uses novel model-agnostic explanations based on if-then rules,
a specific input by analyzing the contributions of all neu- called anchors highest coverage and respecting a user-specified
rons in a reverse pattern (back propagation) in the network precision threshold. An anchor explanation is a decision rule
to every feature of the input. DeepLIFT assigns contribu- that sufficiently tie a prediction locally such that changes to
tion score to each neuron. Information Flow [14] is another the rest of the features values do not matter, i.e., similar
Model dependent method primarily focused on graph visu- instances covered by the same anchor have the same pre-
alization using tensor flow . Non critical nodes are identified diction outcome. Anchor is applied on tabular, images and
and decoupled. An interactive feature is to encourage user textual data sets. It adopts a simulated annealing approach
exploration of model by using cluster expansion. The model that randomly grows, shrinks, or replaces nodes in an ex-
detect repeated structures and highlight same to emphasize pression tree. Refer to figure 3 for Anchors implementation
 Figure 3: ANCHORS,Sentiment Prediction.[8]

in text data. Although both LIME and ANCHORS expla-

nations are computed to be right at local levels, the word “
Not “ would have a positive influence on left but just oppo-
site on right , While such explanations provide insight into
the model, their scope is not clear.

3.3 Data set Dependency

Figure 4: Information Flow Graph.[14]
These model are classified on basis of their contingent on
particular set of data.

3.3.1 Data set Independent

Here we study the models that are adaptable to diverse
range of data sets. Of course, they are best suited for appli-
cations where the feature dependency is not constant.
Image interpreter [3] is domain dependent with data in-
dependent approach as an option. The model aims at new
interactive visualization of neural networks, with the intent
of showing the actual behavior of the network given user-
provided input. It allows user to explore the layer-by-layer
output of a network.This feature is useful for developers who
want to improve their model in IML workflow.The model
typically represents the lower level feature per layer but
there are two limitations that limit its efficacy are firstly
, it lacks an overall summarized description of model * as
shows in figure 5. Second, it clubs the presentation of feature
on layer level and hence no high level abstraction details are
present thereby divest user from Bigger Picture.

3.3.2 Data set Dependent

LIME , the same model we discussed above also satisfies
this sub criteria. While it explains the prediction without
being dependent on domain or model, it has tightly bound
with the data set. This model makes use of representation
that is understandable to humans, regardless of the actual
features used by the model.The method being domain inde-
Figure 5: Image Interpreter Model[3]
pendent is suitable for text as well as image data
This again brings us to already discussed method ,Infor-
mation Flow [14] This model is build on top of Tensor flow
visualizer tool from google . Currently, Tensorflow’s APIs
and their data-flow models simplify the creation of neural
networks for deep learning. Yet developers have to read code
and manually build a mental map of a model to understand
its complicated structure. An average Network typical con-
sists of thousands of neurons and to present them on graphs
typically will not be very useful. .When the Network is run-
ning , we store write its log with relevant information and
then use visualizer to high-level structure of the model, akin
to diagrams that deep learning experts typically draw to ex-
plain their models, and enables users to explore its nested
structure on demand.Figure 4, shows the contracted version
of a graph and the window on top right displays the prop-
erties of selected node i.e. conv2 [10].

Classification of XAI methods

Model Name Data De- Model De- Domain
pendent pendent Depen-
dent
LIME Yes No No
ANCHORS Yes No No
Deep Taylor Yes Yes Yes
Information Flow Yes/No Yes No Figure 6: Graph Clustering[14]
DeepLift Yes Yes Yes
Image Interpreter Yes/No Yes No
the expanded graph while Figure 6. (b) is contacted high-
level diagram showing dependency between train and weights.
A diagram with train and weights can be visualized by hov-
ering over the train namespace (figure 6.c) .
3.4 Edge Contraction
We can use edges by extracting some non critical edges 4. USABILITY EVALUATION
from graph , but it still leaves a visual map huge enough to The above section talks about the parameters to decide
be understood by human brain.[11] To optimize the visual- whether a method fits data, model or domain. Next , we
ization further without removing any more data from graph will explore how it fits to the other entity in information
, we use Edge contraction. This helps in reducing the graph processing domain i.e Human. For this we have found some
size and segmenting the graph into sub graphs. parameters. [4] We also took inspiration form knowledge
The Information Flow model use group edges to simplify Generation Model [10] (figure - 7 )
layout and introduce a more understandable and interactive
structure. Here we group edges or edges between groups. 4.1 Understanding
This resulted graph is a collection of connected subgaphs This steps offer fitting answer to our previous question
which are then more efficient in terms of [14] i.e. how much to present that’s just a balance of essential
information. Once we have decide the XAI model, and data
• Responsiveness - The divided data graph is now signif-
is presented to user. This steps is a measure of how a model
icantly fast, when a node is extended , only the parts
and its functionality is explained to a give user. The con-
of that sub graph are needed to be processed
tent is meant to be given a brief overview about the subject
• Stability Enlarging a node only impacts the layout of and allowing user with an easy transition to deep details on
that sub graph only which helps the user retain the mouse clicks etc.
same metal map of the remaining model.
4.2 Diagnosis
• Legibility - Edge routing declutters the graph as there Diagnosis stands for the identification of nature of a model.
are comparatively lesser edges to present and lesser It enables naive user to visually explain and learn about fea-
connections to understand tures of models without need of external help. For instance
LIME model prompts for an input image , when a user chose
LIME model comprehends the overall output but it don’t one of image in question, now the model sends a request to
make use of any edge compression technique at all. he backend code to process this image through the network,
ANCHOR, Deep Lift model too make use of mathemat- which in return presents the input image with the class label.
ics to decide what to present and what not in information Going through this process once give the user a complete un-
visualization model but don’t use graph optimization. derstanding of working on a model on an external level and
While DeepTaylor works to decompose the network de- hence help the user to further carry on exploration loop and
pending on contributions of its input elements.It also makes come out with interesting facts about data that otherwise
no use of edge correction. will never have been possible with normal ML processes.
Figure 6 represents a graph compression in information
flow model [14] using edge contraction . Part (a)represents 4.3 Refinement
 resented in a spreadsheet.

Figure 7: Knowledge Generation Model[10]

6. REFERENCES
[1] A. Adadi and M. Berrada. Peeking inside the
black-box: A survey on explainable artificial
intelligence (xai). IEEE Access, 6:52138–52160, 2018.
This face marks as the most interesting phase that XAI [2] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini,
shall strive for. This face helps imporve the overall accuracy F. Giannotti, and D. Pedreschi. A survey of methods
of model. The goal is to present the user with information for explaining black box models. ACM computing
that has be possibility of being critical.Some models (Figure surveys (CSUR), 51(5):93, 2018.
1) represents the importance of node by spectrum of colors
[3] A. W. Harley. An interactive node-link visualization of
( with blue being least critical and ping being most critical).
convolutional neural networks. In International
Systems also offer guidance to users by suggesting suitable
Symposium on Visual Computing, pages 867–877.
refinements. Taking input in GUI from user for further op-
Springer, 2015.
timization of model is also a primary part of this model.
[4] D. A. Keim. explainer: A visual analytic framework
for interactive and explainable machine learning.
4.4 Provenance Tracking and Reporting University of Konstanz, 2019.
The model aims and providing the solutions to common [5] S. Liu, X. Wang, M. Liu, and J. Zhu. Towards better
issues of missing justification and provenance tracking.This analysis of machine learning models: A visual
allows user to see feedback by features he worked on, the analytics perspective. Visual Informatics, 1(1):48–56,
decisions he made to refined and improved output . This 2017.
step plays an important step in trust building as now user [6] T. Miller. Explanation in artificial intelligence:
will not just have a model to understand but he will also Insights from the social sciences. Artificial Intelligence,
be curious to know the impacts of his decision on models 2018.
accuracy. [7] G. Montavon, S. Lapuschkin, A. Binder, W. Samek,
and K.-R. Müller. Explaining nonlinear classification
decisions with deep taylor decomposition. Pattern
5. CONCLUSION Recognition, 65:211–222, 2017.
we extensively compared five models and looked at all dif- [8] M. T. Ribeiro, S. Singh, and C. Guestrin. Why should
ferent approaches, dependency, Abstraction Level and, In- i trust you?: Explaining the predictions of any
teraction levels. As core concept of XAI, they all present classifier. In Proceedings of the 22nd ACM SIGKDD
explanation to decisions in some way or other. The differ- international conference on knowledge discovery and
ence majorly on data and domain on which we are applying data mining, pages 1135–1144. ACM, 2016.
the model. An network that deals with image processing [9] M. T. Ribeiro, S. Singh, and C. Guestrin. Anchors:
and has a variable domain but fixed data range and model High-precision model-agnostic explanations. In
may be the best suited with the ”Image Interpreter model” Thirty-Second AAAI Conference on Artificial
while if this network on other hand with order a more local Intelligence, 2018.
view of details i.e at feature/cluster level , An information [10] D. Sacha, A. Stoffel, F. Stoffel, B. C. Kwon, G. Ellis,
flow model may be a better fit. LIME and Anchors make ex- and D. A. Keim. Knowledge generation model for
ceptional use of deep mathematics to express the relevance visual analytics. IEEE transactions on visualization
of any given cluster on output and not only this, they are and computer graphics, 20(12):1604–1613, 2014.
also flexible to be deployed over differnt models and domains [11] W. Samek, T. Wiegand, and K.-R. Müller.
of network can be best used for fields of linguists and related Explainable artificial intelligence: Understanding,
fields where the number of features is just enough to be rep- visualizing and interpreting deep learning models.
 arXiv preprint arXiv:1708.08296, 2017.
[12] A. Shrikumar, P. Greenside, and A. Kundaje. Learning
important features through propagating activation
differences. In Proceedings of the 34th International
Conference on Machine Learning-Volume 70, pages
3145–3153. JMLR. org, 2017.
[13] F.-Y. Tzeng and K.-L. Ma. Opening the black
box-data driven visualization of neural networks. In
VIS 05. IEEE Visualization, 2005., pages 383–390.
IEEE, 2005.
[14] K. Wongsuphasawat, D. Smilkov, J. Wexler, J. Wilson,
D. Mane, D. Fritz, D. Krishnan, F. B. Viégas, and
M. Wattenberg. Visualizing dataflow graphs of deep
learning models in tensorflow. IEEE transactions on
visualization and computer graphics, 24(1):1–12, 2017.