Foundation of

Explainable AI (XAI)
5CS1213

Prof. Ananthanagu
Assistant Professor
Prof. Ananthanagu Dept. of CSE 1
Information presented in the
slides are for educational
purposes. Credit is given to
the respective sources.
Prof. Ananthanagu 2
Course Syllabus Foundation of Explainable AI (XAI) 5CS1213
Module-I Introduction 5 Hrs.
What Is Explainable AI (Black box to White Box), Need for Explainable AI (XAI), Explainability vs. Interpretability,
Explainability Types, Tools for Model Explainability-SHAP, LIME, ELI5, Skater, What-if-tool, AIX360. Methods of XAI
for ML, XAI Compatible Models, XAI Meets Responsible AI, Evaluation of XAI.
Module-II Explainability for Linear Models and Non-Linear Models 6Hrs.
Explainability for Linear Models: Linear Regression, Trust in ML Model: SHAP, LIME Explanation and ML Model, Logistic
Regression, When to Use Which Library.
Explainability for Non-Linear Models: Decision Tree-SHAP, Non-Linear Model Explanation – LIME.
Module-III Explainability for Structured Data 7Hrs.
Permutation Feature Importance, Shapley Values, Explaining Tree-Based Models, Partial Dependence Plots and Related
Plots.
Module-IV Explainability for Unstructured Data 7Hrs.
Image Data: Integrated Gradients (IG), XRAI Grad-Cam, LIME for Images.
Text Data: Overview of Building Models with Text, Tokenization, Word Embeddings and Pretrained Embeddings, Lime for
Text Data, Layer Integrated Gradients, Layer-Wise Relevance Propagation (LRP), Language Interpretability Tool
Module-V Advanced and Emerging Topics 5 Hrs.
Alternative Explainability Techniques, Explainability by Design, Modalities- Time-Series Data, Multimodal Data, Evaluation
of Explainability Techniques
Interacting with Explainable AI, How to Effectively Present Explanations, Common Pitfalls in Using Explainability
AI Ethics, Biasness, and Reliability: AI Ethics, Biasness in AI, Data Bias, Algorithmic Bias, Interpretation Bias, Reliability
in AI Prof. Ananthanagu 3
Course Syllabus Foundation of Explainable AI (XAI) 5CS1213

Text Books:
T1. Pradeepta Mishra, Practical Explainable AI Using Python Artificial Intelligence Model
Explanations Using Python-based Libraries, Extensions, and Frameworks, Apress 2022,
Modules : I, II, V

T2. Michael Munn and David Pitman, Explainable AI for Practitioners Designing and
Implementing Explainable ML Solutions, O’Reilly 2023 Modules : I, III, IV, V

Reference Books:
R1. Denis Rothman, Hands-on Explainable AI (XAI) with Python Interpret, Visualize, Explain,
and Integrate Reliable AI for Fair, Secure, and Trustworthy AI apps, Packt Publishing 2022
R2. Mayuri Mehta, Vasile Palade, Indranath Chatterjee, Explainable AI: Foundations,
Methodologies and Applications, Intelligent Systems Reference Library, Volume 232,
Springer. 2023.

Prof. Ananthanagu 4
Course Syllabus Foundation of Explainable AI (XAI) 5CS1213

Module-IV Explainability for Unstructured Data 7Hrs.

• Image Data: Integrated Gradients (IG),
• XRAI Grad-Cam,
• LIME for Images.
• Text Data: Overview of Building Models with Text,
• Tokenization,
• Word Embeddings and Pretrained Embeddings,
• Lime for Text Data, Layer Integrated Gradients,
• Layer-Wise Relevance Propagation (LRP),
• Language Interpretability Tool

Prof. Ananthanagu 5
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
•Feature Attribution in XAI
• Feature attribution is a core concept in explainable AI.
• It involves identifying which features (or inputs) contributed the most to a model’s prediction.
• Knowing which factors influenced the outcome is critical for trust, transparency, and debiasing.
•Global Feature Attribution
• Global feature attribution provides an overview of which features are important across the entire dataset.
• Techniques like Feature Importance in decision trees or Permutation Feature Importance in random forests provide
insights into which features are most significant.
•Example:
• In a loan approval model, global feature attribution might reveal that credit score and income are the top predictors,
influencing the decision across the dataset.
•Local Feature Attribution
• Local feature attribution, on the other hand, focuses on individual predictions.
• It highlights which features were important in making a particular decision.
• For instance, SHAP values provide a breakdown of how each feature influenced a specific prediction.
•Example:
• If a customer is denied a loan, SHAP values can explain that low credit score and high debt-to-income
ratio contributed most to the decision.
Prof. Ananthanagu 6
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Methods of Feature Attribution:

•LIME (Local Interpretable Model-Agnostic Explanations):

• LIME perturbs input features slightly and measures how the predictions change, providing
interpretable explanations for individual predictions.
•SHAP (Shapley Additive Explanations):
• SHAP values quantify the contribution of each feature to the final prediction using game-theory
principles, making it more mathematically rigorous.
•Integrated Gradients:
• A technique primarily used for deep learning models.
• It approximates the integral of gradients along the path from a baseline input to the actual input.

Prof. Ananthanagu 7
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Attribution:
• It’s the product of importance of a variable x in the equation F(x) and the variable value.
• It should satisfy the sensitivity and implementation Invariance

• Gradient tell us the importance of variable in a function

• Integrated gradients is a method originally proposed in Sundararajan et al., “Axiomatic Attribution for
Deep Networks” that aims to attribute an importance value to each input feature of a machine
learning model based on the gradients of the model output with respect to the input.
• In particular, integrated gradients defines an attribution value for each feature by considering the
integral of the gradients taken along a straight path from a baseline instance x′ to the input instance x

Prof. Ananthanagu 8
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
The attributions must satisfy the following axioms:

1. Sensitivity axiom: if only value of a single variable is changed from its baseline keeping all the other
variables constant. The attribution of the variable is the difference in predication before and after
changing the function.

For example: for a function F(x,y)= ax+by

o If a variable x changed from its baseline value x0 to x1 then the value is changed from ax0+by to
ax1+by and the prediction difference between these two equations ie. a(x1-x0) is our attribution

1. Sensitivity axiom: If two inputs x and x' differ only at dimension i and lead to different predictions,
then the feature associated with that dimension must have non-zero attribution.​
​
M(1,0,1)= POSTIVITE​
M(1,1,1)= NEGATIVE

Prof. Ananthanagu 9
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Baseline: Baseline values are the values of the variable where attribution values are 0 in a function F(x)
prediction is almost 0 because no variable has any importance, and they are contributing 0 towards the
F(x) predictions

Baseline values are generally given as mode values in numerical cases and black or noisy images in
problems dealing with images

For example: in an equation F(x,y). At baseline value the attributions for x and y are 0 and that means
idealiy the value is a*0+b*0 and the prediction is 0

Prof. Ananthanagu 10
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
The attributions must satisfy the following axioms:

2. Implementation invariance axiom: an attribution method should be such that the attributions do not
depend on the particular implementation of the model.

If two models m and m' have identical input/output behavior, then the attributions for m and m' are
identical.

Prof. Ananthanagu 11
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
• IG was one of the first successful approaches to model explainability.
• IG is a local attribution method, meaning it provides an explanation for a model’s prediction for a
single example image.
• It produces an easy-to-interpret saliency mask that highlights the pixels or regions in the image that
contribute most to the model’s prediction.

Prof. Ananthanagu 12
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
• The gradient of a function tells us how the function values change when the inputs are changed
slightly.
• For just one dimension, if the derivative is positive (or negative), that shows the function is increasing
(or decreasing) with respect to the input.
• Since the gradient is a vector of derivatives, the gradient tells us for each input feature if the model
function prediction will increase or decrease when you take a tiny step in some direction of the
feature space.
• The more the model prediction depends on a feature, the higher the attribution value for that feature.
• Gradients only give local information about the model function behavior, but this linear interpretation
is limiting.
• Once the model is confident in its prediction, small changes in the inputs won’t make much difference
• For example, when given an image of a cockatoo, if your model is robust and has a prediction score of 0.88,
modifying a few pixels slightly (even pixels that pertain to the cockatoo itself) likely won’t change the prediction
score for that class.
• For example, once the model fully learns how the value of a specific pixel affects the model’s predicted class, the
gradient of the model prediction for that pixel will become smaller and smaller, and eventually go to zero.
• The gradient for the model’s prediction with respect to that pixel has saturated.
Prof. Ananthanagu 13
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)

Prof. Ananthanagu 14
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline
• A good baseline is one that contains neutral or uninformative pixel feature information
• When working with image models, the most commonly used baseline images are black image, white image, or
noise.

Prof. Ananthanagu 15
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 16
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)

Choosing a Baseline

• A simple baseline that consists of a completely black

image (i.e., no pixel information) and consider the
straight line path from the baseline to the input image,
and then examine the model’s prediction score for its
predicted class, as shown in Figure below.
• A linear interpolation between two points x, y is given
by αy + (1 − α) x
• where the values of α range from 0 to 1.

Prof. Ananthanagu 17
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 18
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 19
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 20
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 21
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 22
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 23
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 24
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline
• As α increases and more information is introduced to our baseline image, the signal sent to the model
and our confidence in what is actually contained in the image increases.
• When α = 0, at the baseline, there is, of course, no way for the model (or anyone, really) to be able to
make an accurate prediction.
• There is no information in the image! However, as we increase α and move along the straight line path,
the content of the image becomes clearer and the model can make a reasonable prediction

Prof. Ananthanagu 25
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline
https://github.com/munnm/XAI-for-practitioners/blob/main/04
-image/integrated_gradients.ipynb

Prof. Ananthanagu 26
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 27
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 28
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 29
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 30
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 31
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 32
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 33
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline
Formally, the importance value of the i-th pixel feature value of an image x for the model f is defined as:

Here x′ denotes the baseline image.

This is precisely the line integral of the gradient with respect to the i-th feature on the straight line path from the
baseline image to the input image

Prof. Ananthanagu 34
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

• Blur Integrated Gradients (Blur-IG) specifically addresses the issues that arise with choosing a specific baseline.
• In short, Blur-IG removes the need to provide a baseline as a parameter and instead advocates to use the
blurred input image as the baseline when implementing Integrated Gradients
This is done by applying a Gaussian blur filter parameterized by its variance α. We then compute
the Integrated Gradients along the straight line path from this blurred image to the true,
unblurred image. As σ increases, the image becomes more and more blurred, as shown in Figure
4-10. The maximum scale σmax should be chosen so that the maximally blurred image is
information-less, meaning the image is so blurred it wouldn’t be possible to classify what is in
the image.

Prof. Ananthanagu 35
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 36
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

https://www.unofficialgoog
ledatascience.com/2017/0
3/attributing-deep-networ
ks-prediction-to.html

Prof. Ananthanagu 37
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline https://colab.research.google.com/drive/19sYttTGTjHrn2Etp99rIG2ttSW6bYxc6?usp=sharing

Prof. Ananthanagu 38
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline https://colab.research.google.com/drive/19sYttTGTjHrn2Etp99rIG2ttSW6bYxc6?usp=sharing

Prof. Ananthanagu 39
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline https://colab.research.google.com/drive/19sYttTGTjHrn2Etp99rIG2ttSW6bYxc6?usp=sharing

Prof. Ananthanagu 40
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline https://colab.research.google.com/drive/19sYttTGTjHrn2Etp99rIG2ttSW6bYxc6?usp=sharing

Prof. Ananthanagu 41
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline https://colab.research.google.com/drive/19sYttTGTjHrn2Etp99rIG2ttSW6bYxc6?usp=sharing

Prof. Ananthanagu 42
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline https://colab.research.google.com/drive/19sYttTGTjHrn2Etp99rIG2ttSW6bYxc6?usp=sharing

Prof. Ananthanagu 43
 Foundation of Explainable AI (XAI) 5CS1213
Unit-IV Explainability for Unstructured Data
Image Data: Integrated Gradients (IG)
Choosing a Baseline

Prof. Ananthanagu 44
 Module-IV

Thank you with regards

Prof. Ananthanagu

Prof. Ananthanagu 45