Interpretability of Neural

Networks
Kian Katanforoosh

Kian Katanforoosh
 Neural networks are deployed:

• In our phones: to recommend content,

• In banks: to manage investments,
• In hospitals: to help doctors diagnose disease symptoms,
• In insurance agencies: to evaluate risk and underwrite documents,
• In cars: to help avoid accidents.

How can one who will be held accountable for a decision trust a
neural network’s recommendation, and justify its use?

[Xie, Ras, et al. (2020): Explainable Deep Learning: A Field Guide for the Uninitiated]
 Today’s outline

I. Interpreting Neural Networks’ outputs

A. with saliency maps
B. with occlusion sensitivity
C. with class activation maps (Global Average Pooling)
II. Visualizing Neural Networks from the inside
A. with gradient ascent (class model visualization)
B. with dataset search
C. the deconvolution and its applications
III. (Optional: Deep Dream: going deeper in NNs)

Kian Katanforoosh
 I. A. Interpreting and visualizing Neural Networks with saliency maps
Context: You have built an animal classifier for a zoo. They are reluctant to use your model
without human supervision, because they don’t understand the decision process of the
model.
Question: How can you alleviate their concerns?
⎛ 0.04 ⎞ “iguana
⎛ siguana ⎞
softmax ⎜ 0.85 ⎟ ⎛ siguana ⎞ ⎜ ⎟
∑
“dog”
⎜ ⎟
⎜! ⎟ ⎜ ⎟ s
⎜ animals animal ⎟
CNN
⎜
⎜ 0.07 ⎟
⎟ “cat” ⎜ sdog ⎟ ⎜ ⎟
⎜ 0.01 ⎟ “ant” ⎜! ⎟ ⎜ sdog ⎟
⎜ ⎟
⎜! ⎜ ⎟ ⎜ ⎟
= ⎜ ∑ sanimal ⎟
⎟
⎜⎝ 0.02 ⎟⎠ “crab” softmax ⎜ scat ⎟
⎜ ⎟ ⎜ animals
⎟
⎜ sant ⎟ ⎜! ⎟
Can be used for segmentation? ⎜! ⎟ ⎜ ⎟
∂sdog (x) ⎜ ⎟ scrab
= ⎜ ⎟
∂x ⎜⎝ scrab ⎟⎠ ⎜ ∑ sanimal ⎟
⎝ animals ⎠
Yes

indicates which pixels need to be changed

Saliency maps
the least to affect the class score the most.
[Karen Simonyan et al. (2014): Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps] Kian Katanforoosh
 I. B. Interpreting and visualizing Neural Networks with occlusion sensitivity
Context: You have built an animal classifier for a zoo. They are a little reluctant to use your
model without human supervision, because they don’t understand the decision process of
the model.
Question: What can you do to alleviate their concerns?
⎛ 0.04 ⎞ “iguana”

⎜ 0.85 ⎟ “dog”
⎜ ⎟
⎜! ⎟
⎜ ⎟
⎜ 0.07 ⎟
“cat”
CNN
⎜ 0.01 ⎟ “ant”
⎜ ⎟
⎜! ⎟
⎜⎝ 0.02 ⎟⎠ “crab”
Probability map of the true class for
different positions of the grey square
⎛ 0.05 ⎞ “iguana”
⎜ 0.83 ⎟ “dog”
⎜ ⎟ Indicates low confidence on the true class
⎜! ⎟ for the corresponding position of the grey square
CNN ⎜ ⎟
⎜ 0.07 ⎟
“cat”

⎜ 0.01 ⎟ “ant”
⎜ ⎟ Indicates high confidence on the true class
⎜! ⎟ for the corresponding position of the grey square
⎜⎝ 0.02 ⎟⎠ “crab”

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 I. B. Interpreting and visualizing Neural Networks with occlusion sensitivity

Probability map of the true class for

different positions of the grey square

Indicates low confidence on the true class

for the corresponding position of the grey square

Indicates high confidence on the true class

for the corresponding position of the grey square

Occlusion sensitivity

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 III. C. Interpreting NNs using class activation maps
Context: Along with the classification output, the zoo now wants real-time visualization of
the model’s decision process. You have one day. What do you do?
Using a classification network for localization:

⎛ 0.02 ⎞
⎜ 0.93 ⎟
⎜ ⎟
input ⎜ 0.04 ⎟
ZERO CONV ReLU
MAX … CONV ReLU
MAX
Flatten
FC SOFTMAX
output
⎜! ⎟
PAD POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

Why this?
Converted to:
⎛ 0.02 ⎞
⎜ 0.93 ⎟
⎜ ⎟
input ⎜ 0.04 ⎟
ZERO CONV ReLU
MAX … CONV ReLU
MAX
Global
Average FC SOFTMAX
output
⎜! ⎟
PAD POOL POOL
Pooling ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

[Bolei Zhou et al. (2016): Learning Deep Features for Discriminative Localization] Kian Katanforoosh
 III. C. Interpreting NNs using class activation maps
Softmax
Volume Probas
(4,4,6) 83 9 93 11
Volume a [1]
1 a [1]
1
55 33 0 22 (1,1,6) ⎛ 0.02 ⎞ “car”
255 134 123 94
93 18 83 83 a2[1] wn −1,1 a2[1] ⎜ 0.03 ⎟ “grass”
⎜ ⎟
y

255 134 12393 94

22 83 1
255 134 123
202 94
34 44
22 83 29
187 93 GAP 93
a [1]
wn −1,2
y ⎜ 0.04 ⎟ “human”
Last 123 94 34 83 44
2 187 30
⎜ ⎟
=
1 3

⎜!
34 76 232 92 wn −1,3
⎟
CONV 44 123
94 42 34
94 8322 44
4 187 22 124 y
[1]
34 76
30 232 1 22 a
layer
123 231
34 44 187
83 187
34 2 192
76 232 192 134
4
wn −1,4 ⎜ 0.07 ⎟ “bed”
34 44
⎜ ⎟
y

104.7 a5[1] an[1]−1

⎜ 0.91 ⎟
34 76 232 124 “dog”
34 255 232 232
92 34 wn −1,5 y
34 76
⎜⎝ 0.09 ⎟⎠
y

a6[1] wn −1,6 an[1] “sky”

34 76 187 124 y y

Input image

wn −1,1 ∗
y
+ wn −1,2 ∗
y
+ … + wn −1,6 ∗ y
=

Class activation map

for “dog"

[Bolei Zhou et al. (2016): Learning Deep Features for Discriminative Localization] Kian Katanforoosh
 III. C. Interpreting NNs using class activation maps

[Bolei Zhou et al. (2016): Learning Deep Features for Discriminative Localization]
Source video: Kyle McDonald

Kian Katanforoosh
 Today’s outline

I. Interpreting Neural Networks’ outputs

A. with saliency maps
B. with occlusion sensitivity
C. with class activation maps (Global Average Pooling)
II. Visualizing Neural Networks from the inside
A. with gradient ascent (class model visualization)
B. with dataset search
C. The deconvolution and its applications
III. (Optional: Deep Dream: going deeper in NNs)

Kian Katanforoosh
 II. A. Visualizing NNs from the inside using gradient ascent (class model visualization)

Context: The zoo trusts that your model correctly locates animals. They get scared and
they ask you whether the model understands what a dog is.
Given this trained ConvNet, generate an image which is representative of the class “dog” according to the ConvNet

Input image x ⎛ 0.02 ⎞

(256,256,3) ⎜ 0.93 ⎟
⎜ ⎟
input
output
⎜ 0.04 ⎟
ZERO CONV ReLU
MAX CONV ReLU
MAX CONV ReLU
MAX
Flatten
FC SOFTMAX ⎜! ⎟
PAD POOL POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

Keep the weights fixed and Gradient ascent: Repeat this process:
use gradient ascent on the input image
1. Forward propagate image x
∂L
to maximize this loss :
2. Compute the objective L

2 x = x +α 3. Backpropagate to get dL/dx

L = sdog (x) − λ x ∂x
4. Update x’s pixels with gradient
ascent
2

“x should look natural” Kian Katanforoosh

[Karen Simonyan et al. (2014): Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps]
 II. A. Visualizing NNs from the inside using gradient ascent (class model visualization)
We can do this for all classes:

L2
Regularization

Looks better with additional

regularization methods.
Class model visualization

[Jason Yosinski et al. (2015): Understanding Neural Networks Through Deep Visualization]
[Karen Simonyan et al. (2014): Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps]
Kian Katanforoosh
 II. A. Visualizing NNs from the inside using gradient ascent (class model visualization)

This method can be applied to any activation in the network

in order to interpret what a neuron is detecting

Input image x ⎛ 0.02 ⎞

(256,256,3) ⎜ 0.93 ⎟
⎜ ⎟
input
output
⎜ 0.04 ⎟
ZERO CONV ReLU
MAX CONV ReLU
MAX CONV ReLU
MAX
Flatten
FC
SOFTMAX ⎜! ⎟
PAD POOL POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

On the class score: On any activation:

change to
L = S dog (x) − R(x) L = a (x) − R(x)
[l ]
j

[Jason Yosinski et al. (2015): Understanding Neural Networks Through Deep Visualization] Kian Katanforoosh
 II. B. Visualizing NNs from the inside using dataset search

Context: The zoo loved the technique, and asks if there are other alternatives.

⎛ 0.02 ⎞
⎜ 0.93 ⎟
⎜ ⎟
input
output
⎜ 0.04 ⎟
Input image ZERO CONV ReLU
MAX CONV ReLU
MAX CONV ReLU
MAX
Flatten
FC
SOFTMAX ⎜! ⎟
(256,256,3) PAD POOL POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

Only one feature maps (among 256) is displayed here : (5,5,256)

Given a filter, what examples in the dataset lead to a strongly activated feature map?

Top 5 It seems that the filter has

images learned to detect shirts

Top 5 It seems that the filter has

learned to detect edges
images
Kian Katanforoosh
II. B. Visualizing NNs from the inside using dataset search
How did we crop the dataset images on the previous slide?

Input image
(64,64,3)
Encoding volume
(13,13,256)
13

CONV 13
(5 layers)

256 values

Conclusion: the deeper the activation,

the more it “sees” from the image
Kian Katanforoosh
 Today’s outline

I. Interpreting Neural Networks’ outputs

A. with saliency maps
B. with occlusion sensitivity
C. with class activation maps (Global Average Pooling)
II. Visualizing Neural Networks from the inside
A. with gradient ascent (class model visualization)
B. with dataset search
C. The deconvolution and its applications
III. (Optional: Deep Dream: going deeper in NNs)

Kian Katanforoosh
 II. C. The deconvolution and its applications

Motivation behind deconvolution/upsampling layers

100-d (64,64,3)
random code generated image

⎛ 0.47 ⎞
⎜ ⎟ Generator “G”
⎜ ⎟ DECONV
⎜ ⎟ (Neural Network)
⎜⎝ 0.19 ⎟⎠

z Binary classification

⎧ y = 0 if x = G(z)
⎪
x Discriminator “D”
(Neural Network) ⎨
⎪⎩ y = 1 otherwise

Real images
(database)

Kian Katanforoosh
 II. C. The deconvolution and its applications

Motivation behind deconvolution/upsampling layers

Per-Pixel
Input image
Classification
(400, 400, 3)
(400, 400, 1)
Information
Encoded

De-convolutions
Convolutions (increases
(reduces volume Encoding volume height
height and width) and width)

Kian Katanforoosh
Get your pencils ready, we’re about to study the deconvolution.

Kian Katanforoosh
 nx − f + 2p 8−4+2×2
Consider the following CONV1D: ny = ⌊ ⌋+1=⌊ ⌋+1=5
s 2
0
0
x1 y1
x2 CONV 1D y2
x3
x= x4 1 filter of size 4 y = y3
y4
⋮ stride = 2
x8 y5
0
0
The system of equations is:
Let’s define our filter as: y1 = w1 ⋅ 0 + w2 ⋅ 0 + w3 ⋅ x1 + w4 ⋅ x2
y2 = w1 ⋅ x1 + w2 ⋅ x2 + w3 ⋅ x3 + w4 ⋅ x4
f = (w1, w2, w3, w4) ⋮
y5 = w1 ⋅ x7 + w2 ⋅ x8 + w3 ⋅ 0 + w4 ⋅ 0
Kian Katanforoosh
 Let’s rewrite the system of equations:
(5,12)
(5,1)
y1 = w1 ⋅ 0 + w2 ⋅ 0 + w3 ⋅ x1 + w4 ⋅ x2 (12,1)
y2 = w1 ⋅ x1 + w2 ⋅ x2 + w3 ⋅ x3 + w4 ⋅ x4
⋮
y = Wx
y5 = w1 ⋅ x7 + w2 ⋅ x8 + w3 ⋅ 0 + w4 ⋅ 0

0
0
w1 w2 w3 w4 0 . . . . . . . . . . . . 0 x1
y1
x2
y2 0 0 w1 w2 w3 w4 0 . . . . . . . . .
1D CONV can be rewritten as a x3
y3 = 0 0 0 0 w1 w2 w3 w4 0 . . . . . . .
matrix vector multiplication! x4
y4 ⋮
y5 ⋮
0 . . . . . . 0 0 0 0 w1 w2 w3 w4 x8
0
0
Kian Katanforoosh
 (5,12)
(5,1)
(12,1)

1D CONV y = Wx
(12,1) (12,5)
(5,1)
−1 x = Hy
1D DECONV If W is invertible, then ∃H = W such that

−1 T
In practice, we would even assume that W is orthogonal, i.e. W =W
For example

(w1, w2, w3, w4) = (−1,0,0,0,1)

T
Thus: x=W y Deconvolution ~ Transposed convolution

Kian Katanforoosh
 T
Let’s rewrite: x=W y

Transposed convolution with stride 2 Sub-pixel convolution with stride 1/2

Kian Katanforoosh
 What to remember?

Implementing a deconvolution (sub-pixel version) is similar to the

convolution, with:

1. Create a sub-pixel version of the input (i.e., insert zeros and pad)

2. Flip the filters.

3. Divide the stride by 2.

Kian Katanforoosh
 III. A. Interpreting NNs using deconvolutions
Motivation of DeconvNets for visualization: Here is a CNN, trained on ImageNet (1.3m images, 1000 classes), we’re
trying to interpret by reconstructing the activation’s zone of influence in the input space.

Input image ⎛ 0.02 ⎞

(256,256,3) ⎜ 0.93 ⎟
⎜ ⎟
input
output
⎜ 0.04 ⎟
ZERO CONV ReLU
MAX CONV ReLU
MAX CONV ReLU
MAX
Flatten
FC
SOFTMAX ⎜! ⎟
PAD POOL POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

Only one feature maps (among 256) is displayed here : (5,5,256)

• Keep the max activation of

a feature map
• Set all other activations of
the layer to 0
Reconstruction

UNPOOL
ReLU
DECONV
DECONV ReLU UNPOOL DECONV ReLU UNPOOL

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 II. C. The deconvolution and its applications

⎛ 0 0 1 0 ⎞
6 12 ⎜
1 0 0 0
⎟ 6 12
⎜ ⎟
⎜ 1 0 0 0 ⎟
⎜⎝ 0 0 0 1 ⎟⎠ 42 7
42 7
Switches

MAXPOOL UNPOOL

0 0
? 0
? 12
? ?0
1 12 1

6 -1 3 6
? ?0 ?
0 0
?
2

42 3 0 1 42
? ?0 0
? 0
?

1 3 7 0
? ?
0 0
? 7
?
8

Kian Katanforoosh
 II. C. The deconvolution and its applications

We need to pass the filters and switches from the ConvNet to the DeconvNet.

Input image ⎛ 0.02 ⎞

(256,256,3) ⎜ 0.93 ⎟
⎜ ⎟
input
output
⎜ 0.04 ⎟
ZERO CONV ReLU
MAX CONV ReLU
MAX CONV ReLU
MAX
Flatten
FC
SOFTMAX ⎜! ⎟
PAD POOL POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

Filters Switches

(5,5,256)

Switches
Filters Switches Filters • Keep the max activation of a feature map
• Set all other activations of the layer to 0
Reconstruction

UNPOOL
ReLU
DECONV
DECONV ReLU UNPOOL DECONV ReLU UNPOOL

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 II. C. The deconvolution and its applications a = Ι{a [l ] [l+1]
≥ 0}.a [l+1]

1 0 3 0 -2 1 -31 -3 -2 1 -31 -3
4 0 0 0 3 4 2 14 3 4 2 14
30 2 0 1 -2 12 4 1 -2 12 4 1
1 0 0 7 10 10 2 1 10 10 2 1
“ReLU forward” “ReLU backward” “ReLU DeconvNet”
Switches

1 -2 3 -4 -2
? 0? -31
? 0? 0? 1? 0? 0?

4 -1 -1 -2 ⎛ 1 0 1 0 ⎞ 3? 0? 0? 0? 3? 4? 2? 14
?
⎜ ⎟
⎜ 1 0 0 0 ⎟
⎜ 1 1 0 10 ⎟

30 2 0 1
⎜⎝ 1 0 0 1 ⎟⎠
-2
? 12
? 0? 1? 0? 12
? 4? 1?

1 -2 -9 7 10
? 0? 0? 1? 10
? 10
? 2? 1?
[Sringenberg & Dosovitskiy et al. (2015): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 II. C. The deconvolution and its applications

We need to pass the filters and switches from the ConvNet to the DeconvNet.

Input image ⎛ 0.02 ⎞

(256,256,3) ⎜ 0.93 ⎟
⎜ ⎟
input
output
⎜ 0.04 ⎟
ZERO CONV ReLU
MAX CONV ReLU
MAX CONV ReLU
MAX
Flatten
FC
SOFTMAX ⎜! ⎟
PAD POOL POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

Filters Switches

(5,5,256)

Switches
Filters Switches Filters • Keep the max activation of a feature map
• Set all other activations of the layer to 0
Reconstruction

UNPOOL
ReLU
DECONV
DECONV ReLU UNPOOL DECONV ReLU UNPOOL

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 II. C. The deconvolution and its applications

Other CONV layers can be visualized the same way

Input image ⎛ 0.02 ⎞

(256,256,3) ⎜ 0.93 ⎟
⎜ ⎟
input
output
⎜ 0.04 ⎟
ZERO CONV ReLU
MAX CONV ReLU
MAX CONV ReLU
MAX
Flatten
FC
SOFTMAX ⎜! ⎟
PAD POOL POOL POOL (x3) ⎜ ⎟
⎜ 0.07 ⎟
⎜ 0.11 ⎟
⎜ ⎟
⎝ 0.09 ⎠

(5,5,256)
(13,13,128)
(20,20,128)

UNPOOL
ReLU
UNPOOL
Reconstruction DECONV
ReLU
UNPOOL DECONV
ReLU
…
DECONV
…
…

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 Results on a Layer 1
validation set of
50,000 images
Patches
• Top-9 strongest
activations per
filter in the 1st
layer

• Because we know Filters

the position of the
activation and all
the pooling
switches we can
crop the part of
the image that
fired the activation

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 Results on a Layer 2 reconstructions (deconv)
validation set of
50,000 images

• Learning a more complex set of

patterns than 1st layer edges.

• Covers a much larger space of the

image because of the pooling layer
before.

• Top-1 strongest activation per feature

map in the 2nd layer (256 feature Filters
maps.)

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 Results on a
validation set of
50,000 images

• Learning a more
complex set of patterns
than 1st layer edges

• Covers a much larger

space of the image
probably because of the
pooling layer before.

• Features are more

invariant to small Filters
changes. Ex: A dot,
spiral, circle all fire the
same 2nd layer feature
very strongly

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 Results on a
validation set of
50,000 images

• 3rd layer: increased

complexity

• An activated neuron is
seeing ≈80x80 part of a
256x256 image

• Learning objects, faces

etc..
Filters

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
 Results on a
validation set of
50,000 images

• 3rd layer: increased

complexity

• An activated neuron is
seeing ≈80x80 part of a
256x256 image
Faces
• Learning objects, faces
etc..

• Patches: Semantic
grouping, not structural Clouds

[Matthew D. Zeiler and Rob Fergus (2013): Visualizing and Understanding Convolutional Networks]
Kian Katanforoosh
[Link to video: https://www.youtube.com/watch?v=AgkfIQ4IGaM]
[Jason Yosinski et al. (2015): Understanding Neural Networks Through Deep Visualization]
 Today’s outline

I. Interpreting Neural Networks’ outputs

A. with saliency maps
B. with occlusion sensitivity
C. with class activation maps (Global Average Pooling)
II. Visualizing Neural Networks from the inside
A. with gradient ascent (class model visualization)
B. with dataset search
C. The deconvolution and its applications
III. (Optional: Deep Dream: going deeper in NNs)

Kian Katanforoosh
 III. Deep Dream: going deeper in NNs

How to boost the activation of a neuron?

input

ZERO
PAD
CONV ReLU
MAX
POOL
CONV ReLU
MAX
POOL
…

Activations
Dreaming process (to repeat): (5,5,256)
1. Forward propagate image until dreaming layer
2. Set gradients of dreaming layer to be equal to its activations
3. Backpropagate gradients to input image
4. Update Pixels of the image

[Alexander Mordvintsev et al. (2015): Inceptionism: Going Deeper into Neural Networks] Kian Katanforoosh
 III. Deep Dream: going deeper in NNs

[Alexander Mordvintsev et al. (2015): Inceptionism: Going Deeper into Neural Networks] Kian Katanforoosh
 III. Deep Dream: going deeper in NNs

If you dream in lower layers:

[Github repo (deepdream): https://github.com/google/deepdream/]

[Alexander Mordvintsev et al. (2015): Inceptionism: Going Deeper into Neural Networks] Kian Katanforoosh
 III. Deep Dream: going deeper in NNs

[Link to video: https://www.youtube.com/watch?v=DgPaCWJL7XI]

[Github repo (deepdream): https://github.com/google/deepdream/]
[Alexander Mordvintsev et al. (2015): Inceptionism: Going Deeper into Neural Networks]
Kian Katanforoosh
 III. Deep Dream

Did the neural network learned the right features to detect an object?

Dumbbells

the network failed to understand the essence of a dumbbell

[Github repo (deepdream): https://github.com/google/deepdream/]

[Alexander Mordvintsev et al. (2015): Inceptionism: Going Deeper into Neural Networks] Kian Katanforoosh
Questions we are now able to answer:

- What part of the input is responsible for the output?

- Occlusion sensitivity
- Class Activation Maps

- What is the role of a given neuron/filter/layer?

- Deconvolutions can help visualize the role of a neuron
- Search dataset images maximizing the activation
- Gradient ascent (class model visualization)

- Can we check what the network focuses on given an input image?

- Occlusion sensitivity
- Saliency maps (one-time gradient ascent)
- Class Activation Maps

- How does a neural network see our world?

- Gradient ascent (class model visualization)
- Deep Dream

- Do these visualization have use cases?

- Segmentation (saliency maps)
- Art (Deep Dream)
Kian Katanforoosh
 Duties for next week

Project Meeting #2: Meet with your assigned TA between 10/07 and 11/05 to discuss your
milestone report.

Project Milestone due 11/05 Friday 11:59 PM

Completed modules for 11/10:

• C5M1: Recurrent Neural Networks (slides)

Quizzes (due at 9:30am PST):

• Recurrent Neural Networks
Programming Assignments (due at 9:30am PST):
• Building a Recurrent Neural Network - Step by Step
• Dinosaur Land -- Character-level Language Modeling
• Jazz improvisation with LSTM

Kian Katanforoosh, Andrew Ng, Younes Bensouda Mourri