Explainable

AI (XAI)
CS 3244 A
Machine Learning
Recap from W10a&b

What did we learn?

• Feature Engineering →
Architecture Engineering

• CNN: exploits spatial information

using convolutions
Neurons with Recurrence

• RNN: exploits history information

using recurrence

NUS CS3244: Machine Learning Recurrent Neural Network (RNN) 2

NUS CS3244: Machine Learning 14
Student Learning Outcomes

Week 11A: Learning Outcomes

1. Describe multiple methods to interpret feature importance
2. Appropriately interpret feature attributions from each type of
explanation
3. Describe how LIME explanations are generated
4. Describe how Grad-CAM explanations are generated

NUS CS3244: Machine Learning 3

Student Learning Outcomes

Week 11A: Lecture Outline

1. Introduction
1. Motivation for Explainable AI (XAI)
2. Explaining Why: Feature Importance
2. Explanation techniques
1. Glassbox Models (Linear Regression, Logistic Regression)
2. Model-Agnostic Explanations (LIME)
3. Model-Specific Explanations (Grad-CAM)

NUS CS3244: Machine Learning 4

 Image credit

Case 1:
Does patient have cancer? Why??

Evidence for Evidence for

Prediction: Cancer Cancer No Cancer
Further reading: https://coderzcolumn.com/tutorials/machine-
learning/how-to-use-lime-to-understand-sklearn-models-predictions
Explanation: Feature Attributions
NUS CS3244: Machine Learning 6
 Case 2:
Is this skin cancer? Why??

Prediction: Skin Cancer Explanation: Highlighted Salient Region

Further reading: https://towardsdatascience.com/medical-image-analysis-using-probabilistic-layers-and-grad-cam-42cc0118711f
Image credit: https://news.yale.edu/2019/11/13/yale-study-reveals-hyperhotspots-identifying-skin-cancer-risk

NUS CS3244: Machine Learning 7

Feature Importance
• Explains
• Which features are important for the prediction
• In what way the features influenced the prediction
• Implementation
• Weights in Linear / Logistic Regression
• Surrogate Weights from LIME
• Saliency Maps of CNN

NUS CS3244: Machine Learning 8

 Interpreting
Linear Regression
Discussion E11b.1

How would you interpret?

Linear Regression
𝑛
𝑦ො = 𝑤0 + 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛 = ෍ 𝑤𝑟 𝑥𝑟
𝑟=0
⊤
=𝒘⋅𝒙=𝒘 𝒙

NUS CS3244: Machine Learning 10

Discussion E11b.1 Solution

How would you interpret?

Linear Regression
𝑛
𝑦ො = 𝑤0 + 𝑤1 𝑥1 + 𝑤2 𝑥2 + ⋯ + 𝑤𝑛 𝑥𝑛 = ෍ 𝑤𝑟 𝑥𝑟
𝑟=0
⊤
=𝒘⋅𝒙=𝒘 𝒙

Weighted Sum Interpretation Gradient Interpretation

Bigger 𝑤𝑟 means Bigger 𝑤𝑟 means
• Larger weight • Steeper slope for 𝑥𝑟 axis
• Changes in 𝑥𝑟 lead to bigger changes in 𝑦ො
• More importance for to 𝑥𝑟 • More importance for 𝑥𝑟
• Direction? Supportive (positive) or opposing • Direction indicates increasing or decreasing
(negative) influence influence
NUS CS3244: Machine Learning 11
 Interpreting
Logistic Regression
Discussion E11b.2

How would you interpret?

Logistic Regression
1
𝑦ො = 𝜎 𝑧 = 𝜎 𝒘⋅𝒙
1 + 𝑒 −𝑧
𝑛 𝒘⋅𝒙
𝑧 =𝒘⋅𝒙=෍ 𝑤𝑟 𝑥𝑟
𝑟=0

NUS CS3244: Machine Learning 13

Discussion E11b.2 Solution

How would you interpret?

Logistic Regression
1
𝑦ො = 𝜎 𝑧 = 𝜎 𝒘⋅𝒙
1 + 𝑒 −𝑧
𝑛 𝒘⋅𝒙
𝑧 =𝒘⋅𝒙=෍ 𝑤𝑟 𝑥𝑟
𝑟=0

Weighted Sum Interpretation Gradient Interpretation

Bigger 𝑤𝑟 means Bigger 𝑤𝑟 means?
• Larger importance • Steepness? Sigmoid bounded between 0 and 1
• Direction indicates influence • Direction in 2D (or higher)?
NUS CS3244: Machine Learning 14
 New 3D Demo:
https://www.desmos.com/3d/zcrutuxt2v
Discussion E11b.2 Solution

How would you interpret?

Logistic Regression
1
𝑦ො = 𝜎 𝑧 = 𝜎 𝒘⋅𝒙
1 + 𝑒 −𝑧
𝑛 𝒘⋅𝒙
𝑓 =𝒘⋅𝒙=෍ 𝑤𝑟 𝑥𝑟
𝑟=0

Weighted Sum Interpretation Gradient Interpretation

Bigger 𝑤𝑟 means Bigger 𝑤𝑟 means?
• Larger importance • Steeper slope for 𝑥𝑟 near decision boundary
• Direction indicates influence • Decision boundary more perpendicular to 𝑥𝑟
• Weight sign indicates direction of pos/neg prediction
NUS CS3244: Machine Learning 19
Questions!
Local Interpretable Model-agnostic Explanations

LIME
 How to describe with just 𝑥1 and 𝑥2 ?
Non-Linear Decision Boundary 𝑓 𝒙
𝑥2 Prediction Model
𝑓 𝒙
𝑥1
• Non-linear model of 𝒙 = 𝑥2
FN ⋮
FN FN • Shown as curvy decision boundary

Explanation: Linear Model

𝑛
FP 𝑔 𝒙 =𝒘⋅𝒙=෍ 𝑤𝑟 𝑥𝑟
FP 𝑟=0
FP • Simple to interpret
• But too many errors between 𝑔 and 𝑓
𝐿 =𝑓 𝒙 −𝑔 𝒙
𝑥1
Image Credit: https://santiagof.medium.com/model-interpretability-making-your-
model-confess-lime-89db7f70a72b NUS CS3244: Machine Learning 22
 How to describe with just 𝑥1 and 𝑥2 ?
Non-Linear Decision Boundary
𝑥2 Prediction Model
𝑓 𝒙
𝑥1
• Non-linear model of 𝒙 = 𝑥2
⋮
• Shown as curvy decision boundary

Explanation: Gradients
𝑑𝑓 𝜕𝑓/𝜕𝑥1
𝑔 𝒙 = ∇𝑓 𝒙 = = 𝜕𝑓/𝜕𝑥2
𝑑𝒙
⋮
• Steepness for each feature 𝑥𝑟
• Difficult to remember, since gradients
𝑥1 are different for each instance (point)
Image Credit: https://santiagof.medium.com/model-interpretability-making-your-
model-confess-lime-89db7f70a72b NUS CS3244: Machine Learning 23
 LIME
Local Interpretable Model-agnostic Explanations
𝑥2 Prediction Model
𝑓 𝒙
𝑥1
• Non-linear model of 𝒙 = 𝑥2
⋮
• Shown as curvy decision boundary

Explanation: LIME
1. Starting with instance 𝒙 to explain
2. Focus on Local region
3. Training set as neighbors 𝒙 𝜂 ∈ 𝑋 𝜂
4. Train surrogate model, e.g., linear:
𝑛
𝑥1 𝑔 𝒙 =𝒘⋅𝒙=෍ 𝑤𝑟 𝑥𝑟
𝑟=0
Image Credit: https://santiagof.medium.com/model-interpretability-making-your-
model-confess-lime-89db7f70a72b NUS CS3244: Machine Learning 24
 Python API:
LIME https://github.com/marcotcr/lime
Find “best” explainer 𝑔 that minimizes 𝜉 𝒙
“Faithful” “Simple”

argmin 𝜉 𝒙 = 𝐿 𝑓, 𝑔, 𝜋𝑥 + Ω 𝑔
𝑔∈𝐺

Locally-weighted error loss function Sparsity regularization

𝑛
2
𝐿 𝑓, 𝑔, 𝜋𝑥 = ෍ 𝜋𝑥 𝒙 𝜂 𝑓 𝒙𝜂 −𝑔 𝒙𝜂 Ω 𝑔 = 𝒘 1 =෍ 𝑤𝑟
𝑟=1
𝒙 𝜂 ∈𝑋 𝜂 • Want simpler explanation
• ⇒ fewer weights
Neighbor Predictor Explainer • ⇒ L1 norm (LASSO)
proximity model model
• Penalizes large sum of weights
function e.g.,
2
− 𝑑 𝒙,𝒙 𝜂
𝑒
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). " Why should i trust you?" Explaining the predictions of any classifier. KDD’16.
NUS CS3244: Machine Learning 25
Demo

Case 1: Does patient have cancer? Why do the two set of weights differ?

Instance 𝒙 Logistic Regression LIME

Evidence for Cancer Evidence for No Cancer

Prediction 𝑦ො = Cancer Weights 𝒘 of Weights 𝒘 of

predictor 𝑓 surrogate explanation 𝑔
Further reading: https://coderzcolumn.com/tutorials/machine-learning/how-to-use-lime-to-understand-sklearn-models-predictions
NUS CS3244: Machine Learning 26
Questions!
Gradient-weighted Class Activation Mapping

Grad-CAM
Explaining Image Predictions
• LIME to explain image prediction?
• What are the input features?
• Feature = Pixels?
• Too many features
• Need “super pixels”

• Another way: Attribution → Saliency Map

• Feature = Activation Map
• Grad-CAM
Image credit: https://arxiv.org/pdf/1908.04389.pdf NUS CS3244: Machine Learning 29
Recap from W10A

Convolutional Neural Network Explain 𝑦ො (𝑐)

(prediction towards class 𝑐)
in terms of activation maps 𝑨 𝐿−1

𝑨 𝐿−1 𝑨𝐿
𝑐

Backprop
Forward prop through Conv layers ∇𝑦ො 𝑐 𝑨[𝐿−1]

Key concepts
 Learn Spatial Feature  Flattening  Learn Nonlinear Features  Classification
• Series of multiple convolution + • Convert to • With fully connected layers • Softmax ≔ Multiclass
pooling layers fixed-length (regular neurons) Logistic Regression
• Progressively learn more diverse 1D vector • Learns nonlinear relations • Feature input = image
and higher-level features with multiple layers embedding vector
(typically large vector)
Image credit: https://towardsdatascience.com/a-comprehensive-
guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53 NUS CS3244: Machine Learning 30
Grad-CAM Example: Why did the CNN predict “9”?
𝑘 = 17
× 0.5

𝑦ො = "9"
𝑘 = 22
× 0.4 +

𝒙
𝑘 = 31 ReLU ෍ 𝛼𝑘 𝑨𝑘
𝑐 [𝐿−1]

× 0.1 𝑘
Class Activation Map
(CAM)
𝐿−1 𝑐
𝑨 𝐿−1 𝑨𝑘 × 𝛼𝑘 𝑐 = "9"
Final Conv Layer 𝐿 − 1 Activation Importance Explain saliency
Activation Maps Map Weight for “9”
NUS CS3244: Machine Learning 31
Grad-CAM Example: What is the evidence to predict “7”?
𝑘 = 17
× 0.1

𝑦ො = "9"
𝑘 = 22
× 0.4 +

𝒙
𝑘 = 31 ReLU ෍ 𝛼𝑘 𝑨𝑘
𝑐 [𝐿−1]

× 0.1 𝑘
Class Activation Map
(CAM)
𝐿−1 𝑐
𝑨 𝐿−1 𝑨𝑘 × 𝛼𝑘 𝑐 = "7"
Final Conv Layer 𝐿 − 1 Activation Importance Explain saliency
Activation Maps Map Weight for “7”
NUS CS3244: Machine Learning 32
 𝑐
Importance Weight 𝛼𝑘 of Activation Map
𝑐 𝑐
𝜕 𝑦Ƹ 𝜕 𝑦Ƹ
[𝐿−1] [𝐿−1] 𝐿−1 𝐿−1
𝑎11𝑘 ⋯ 𝑎1𝑤𝑘 𝑐 𝜕𝑎11𝑘 ⋯ 𝜕𝑎1𝑤𝑘
[𝐿−1] 𝜕 𝑦ො
𝑨𝑘 = ⋮ ⋱ ⋮ = ⋮ ⋱ ⋮
[𝐿−1]
[𝐿−1]
𝑎ℎ1𝑘 ⋯ 𝑎 [𝐿−1] 𝜕𝑨𝑘 𝜕 𝑦Ƹ 𝑐 ⋯ 𝜕 𝑦Ƹ 𝑐
ℎ𝑤𝑘 [𝐿−1]
≠ 𝑾𝑘 𝜕𝑎ℎ1𝑘
𝐿−1 𝐿−1
𝜕𝑎ℎ𝑤𝑘
Activation of
𝜕𝑓 𝐿−1 𝑑𝑔 𝐿−1 𝜕𝑓 𝐿 𝑑𝝈 𝑑𝑦ො 𝑐
𝑘th channel
[𝐿−1] 𝑑𝑓 𝐿−1 𝜕𝑔 𝐿−1 𝑑𝑓 𝐿 𝑑𝝈
=
at pixel (ℎ, 𝑤) 𝜕𝑎 ℎ𝑤𝑘
in layer 𝐿 − 1

Sum of all gradients in activation map

𝑐 𝑐 𝑐 𝑐
𝑑𝑦ො 𝜕 𝑦ො 𝜕 𝑦ො 𝜕 𝑦ො 𝑐
𝐿−1
=෍ 𝐿−1
= 𝐿−1
+ ⋯+ 𝐿−1
= 𝛼𝑘
𝑨 𝐿−1 𝑑𝑨𝑘 𝑖𝑗 𝜕𝑎
𝑖𝑗𝑘 𝜕𝑎11𝑘 𝜕𝑎ℎ𝑤𝑘
Final Conv Layer 𝐿 − 1 Gradient Interpretation: Steeper ⇒ More important
Activation Maps
NUS CS3244: Machine Learning 33
Recap from W10A

Convolutional Neural Network Explain 𝑦ො (𝑐)

(prediction towards class 𝑐)
in terms of activation maps 𝑨 𝐿−1

𝑨 𝐿−1 𝑨𝐿
𝑐

Backprop
Forward prop through Conv layers ∇𝑦ො 𝑐 𝑨[𝐿−1]

Key concepts
 Learn Spatial Feature  Flattening  Learn Nonlinear Features  Classification
• Series of multiple convolution + • Convert to • With fully connected layers • Softmax ≔ Multiclass
pooling layers fixed-length (regular neurons) Logistic Regression
• Progressively learn more diverse 1D vector • Learns nonlinear relations • Feature input = image
and higher-level features with multiple layers embedding vector
(typically large vector)
Image credit: https://towardsdatascience.com/a-comprehensive-
guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53 NUS CS3244: Machine Learning 34
Grad-CAM Steps Python API:
https://github.com/jacobgil/pytorch-grad-cam

1. Compute Activation Maps 𝑨[𝐿−1] of last conv layer 𝐿 − 1

1. via Forward Propagation
2. Choose class label 𝑐 to explain about (e.g., predict “9”, “car”)
3. Filter prediction 𝑦ො to be about class 𝑐
1
𝑦ො 0 0
2
𝑦ො 𝑐 0 𝑐 𝑐 0
1. Given: 𝒚
ෝ= , 𝒆 = , then ෝ
𝒚 =𝒚 ෝ∘𝒆 =
𝑦ො𝑐 1 𝑦𝑐
𝑦ො𝑛 0 0
2. To generate explanation only for that class 𝑐
𝑐 [𝐿]
4. Compute importance weight 𝛼𝑘 for each Activation Map 𝑨𝑘
1. ෝ
Backprop from 𝒚 𝑐 to get gradients (relative to activations) at last conv layer
5. Compute weighted sum with ReLU to get Class Activation Map
NUS CS3244: Machine Learning 35
Questions!
Advanced (not in exam)

Understanding Filters Hard to interpret kernels.

They are just matrices used
for convolution.

Types of Dogs Bowtie? Chain?

NUS CS3244: Machine Learning 37

Further Reading
• https://christophm.github.io/
interpretable-ml-book

NUS CS3244: Machine Learning 38

 Brian Y. Lim | [email protected]
AI Applications https://ubiquitous.comp.nus.edu.sg
X
Interpretable Directed Rubric-Informed
XAI Reasoning Framework [CHI’19]
m an-desire Diversity [CHI’22] XAI [IUI’23]
Hu d
Understanding People Informs Explaining AI
Attempt Message Score
How People should Reason and Explain How XAI Generates Explanations
1 Physical activity is good for health. 37%
• Explanation goals • Bayesian probability Rubric-Using
• Similarity modeling Let’s go for some exercise.
How
• Inquiry People
and reasoning should Reason Human Judge
How XAIqueries
• Intelligibility Generates Explanations 2 Physical activity is good for health. 54%
and
• Causal explanation andExplain
causal attribution • XAI elements
• Rational choice decisions • Data structures

d
• Visualizations 3 Physical activity is good for health. 55% Rubric-Informed

Hu
…

Human-
Let’s walk more and reduce sitting XAI Judge (IRIS)

an-informe
How People actually Reason (with Errors) How XAI Support Reasoning and Mitigate Errors time.
• Dual process model • Mitigate representative bias
Philosophy
How People Psychology
actually
Reason How
• Mitigate XAI Supports
availability bias Reasoning Related to dreamlining +1% Non-accountable

man-inspired
• System 1 Heuristic Biases
• Mitigate anchoring bias
(with
• System 2 Weaknesses Errors) (and Mitigates Errors) Is a musical time +2% AI Judge
• Mitigate confirmation bias Has prerequisite playing game +1%
• Moderate trust

XAI Gap and Trends [CHI’18] Centered RePrompt [CHI’23]

XAI
“My best friend will be
going to school in another
country for 4 years.”

m
“My best friend is leaving “best, friend, going, school,
the country for a long time country, years, current,

Hu
to study overseas. So sad!” advance, cold, sad”

Text-to-Image
a b c
Generative AI

Hu d
ma nd e
n-confou
Parsimony vs. Performance [CHI’20] Privacy harms Explanations [CHI’22] Explanations harm Privacy [ICCV’21] XAI Perceptual Process [CHI’22]
Classification: Cycling
Blur Level None Weak Strong
Linear Cognitive-GAM GAM

Image

Unbiased Biased
Regular
Saliency Map
Stimuli 1 Select 2 Organize 3 Interpret

Simple Accurate a Recall Memory b Compare c Categorize

Less Curviness Debiased

Saliency Map 1 Highlight 2 Recognize 3a Synthesize 3b Contrast 3c Classify
Saliency Cue Counterfactual Cues Concept
Research, Advanced (not in exam)

Debiased-CAM
to mitigate image perturbations with
faithful visual explanations of machine learning

Wencan Zhang Mariella Dimiccoli Brian Y. Lim*

[email protected] [email protected] [email protected]
“Fish” 41
“Fish” 42
“Fish” 43
Bias due to Image Corruptions and Perturbations
Blur Color Shift Day-Night Lighting

“Biking” “Kitchen” “Is Snowy”

44
Base Model Unbiased-CAM

CNN0 “Fish”

Biased-CAM0
Bias
for “Fish”

CNN0 “Dog”

45
Fine-tuned Model Unbiased-CAM

CNN0 “Fish”

Biased-CAM0
Bias

CNNf “Fish”

46
Debiased Model Unbiased-CAM

CNN0 “Fish”

Biased-CAM
Bias
Secondary
prediction task
Multi-task (mt)
prediction
Primary
CNNdf “Fish” Prediction task

47
Debiased Model Unbiased-CAM

CNN0 “Fish” Self-

Supervised
Training
Debiased-CAM
Biased-CAM
Bias

CNNdf “Fish”

48
 ForImageForbias
different leadsprediction
different totasks
prediction CAMand distortions
tasks
bias…types …
Blur (𝜎) Blur (𝜎) Color Temperature (Δ𝑇) Day-Night Ratio (𝜌)
Bias Level 0 16 32 0 16 32 0K –1600K –3600K 0 0.4 0.8

Biking “A man on a horse on a street.” Cleaning & Chores Is Snowy

1

Saliency
Regular
CNN
0
FineTuned
CNN

Debiased
CNN

Debiased-CAM improves CAM faithfulness 49

 Stronger bias worsens Performance & CAM distortions

Prediction Explanation
Performance Faithfulness
Precision-Recall AUC Pearson Correlation Coefficient

DebiasedCNN
FineTunedCNN
RegularCNN

Debiased-CAM improves Performance & CAM faithfulness 50

 Stronger bias worsens Performance & CAM distortions
Classification | Blur Captioning | Blur Classification | Color Classification | Day-to-Night

Performance (PR AUC)

0.95

0.90

0.85

-5400 (6600) 5400

1.00 Color Temperature Bias

CAM Faithfulness (PCC)

0.85

0.70
-5400 (6600) 5400
Color Temperature Bias Night Bias Ratio (𝜌)

sb = single-bias
RegularCNN DebiasedCNN (sb) FineTunedCNN (sb) DebiasedCNN (mb) FineTunedCNN (mb)
mb = multi-bias

Debiased-CAM improves Performance & CAM faithfulness 51

Privacy harms Explainability

Debiased-CAM helps
Wrapping Up
[W10b] Student Learning Outcomes

What did we learn?

Feature Importance Explanations
Feature Attribution
Glassbox model 𝑓
Weights 𝒘 of 𝑓 e.g. Linear Regression,
Logistic Regression
Model-agnostic
Blackbox
Weights 𝒘 of 𝑔 explainer model 𝑔
nonlinear model 𝑓
e.g. LIME

Saliency Map

Blackbox
Grad-CAM explanation 𝑔
CNN model 𝑓

NUS CS3244: Machine Learning 54

This Thursday:
Deepavali Holiday

NUS CS3244: Machine Learning 55

Next week:
Unsupervised Learning
Image credit: https://hip2save.com/2019/11/27/lego-classic-
creative-fun-900-piece-set-only-20-at-walmart-regularly-40/

NUS CS3244: Machine Learning

Pre-Lecture Activity for next week

W12 Pre-Lecture Task (due before next Mon)

Read
1. Clustering With More Than Two Features? Try This To Explain Your Findings by
Mauricio Letelier
Task
1. Describe other use cases where you need to apply domain knowledge with
data-driven unsupervised learning to better understand your business or
engineering problem
Tip: you can your own projects too; you don’t have to be correct
2. Post a 1–2 sentence answer to the topic in your tutorial group: #tg-xx

NUS CS3244: Machine Learning 57