Contents

● Introduction
● Layer-Wise Relevance Propogation (LRP)
● Which LRP Rule for Which Layer?
● Conclusion
● Discussion
Introduction
 Background & Motivation

● Rise of large datasets is a main driver for the success of machine learning techniques in both
industrial and scientific applications
● Large datasets can be plagued by spurious correlations; leads to “Clever Hans” predictors

All classify
correctly, but only
(i) generalizes
Explainable Machine Learning

● Feature selection is one solution: only present the model with “good” input features
○ This can be difficult to apply in practice
○ Consider image recognition, where individual pixels do not have fixed roles
● Explainable machine learning takes opposite approach: train the model, then examine which
features the model actually learned
○ We do not care about feature selection during training
○ “Bad” features can be removed later, and the model can be retrained on cleaned data
● Taylor Decomposition is a foundational explainable ML technique related to LRP
Taylor Decomposition

● Produce explanations by performing a Taylor expansion of the prediction 𝑓(𝑥) at some nearby
reference point
● First-order terms (the elements of the sum) quantify the relevance of each input feature, forming
the explanation

Reference
point
Problems with Taylor Decomposition

● Unstable when applied to DNN

○ Shattered gradients: while 𝑓(𝑥) is generally accurate, the gradient is often very noisy, containing little
meaningful information
○ Adversarial examples: small perturbations of the input can cause dramatic changes to the function value
● It can be difficult to choose a meaningful reference point
○ 0 → may be far away from real input
Alternative Explanation Techniques

● Integrate a large number of local gradient estimates

● Replace the gradient with a coarser estimate of effect, such as model response to patch-like
perturbations
● Optimization techniques involving a local surrogate model or the explanation itself
● Common Problem: these techniques are all computationally expensive, involving multiple
network evaluation
Four Properties of Good Explanation Techniques

● Conservation: if we find explainable evidence in the output, it must show up somewhere in the
input features (no loss of evidence)
● Positivity: either a feature is relevant (positive) or irrelevant (zero)
● Continuity: if two inputs are almost the same, and the prediction is almost the same, then the
explanation should be almost the same
● Selectivity: models must agree with explanation; removing evidence from input should reduce
confidence in the output

LRP satisfies all of these properties; the previous techniques do not

Layer-Wise Relevance
Propogation (LRP)
LRP Explained

● LRP is an explanation technique which propagates the prediction backwards using purposely
designed local propagation rules
● LRP is subject to the conservation property
○ What has been received by a neuron must be redistributed to the lower layer in equal amount
○ (It’s also subject to the other properties, but this one is explicitly mentioned)
LRP Explained (ii)

Referring to the previous equation:

● 𝒛𝒋𝒌: the quantity which models the extent to which neuron j has contributed to make neuron k
relevant
● σ𝒋 𝒛𝒋𝒌 :the denominator which serves to enforce the conservation property
LRP Rules for Deep Rectifier Networks

● Question: How do we determine 𝑧𝑗𝑘 (the contribution of neuron 𝑗 to 𝑅𝑘)?

● Answer: With LRP rules!
● We’ll talk about three in the following slides:
○ Basic (LRP-0)
○ Epsilon (LRP-𝟄)
○ Gamma (LRP-𝞬)
● Note that we’re working in the context of ReLU activations:
LRP Rules: LRP-0

● Redistribute relevance in proportion to the contributions of each input to the neuron activation
● Note 𝑧𝑗𝑘 = 𝑎𝑗𝑤𝑗𝑘
● Properties:
○ If 𝑎𝑗 = 0 or 𝑤𝑗𝑘 = 0, then 𝑅𝑗 = 0, which allows for compatibility with concepts such as zero
weight, deactivation, or absent connections
○ Uniform application produces an explanation equivalent to (𝐺𝑟𝑎𝑑𝑖𝑒𝑛𝑡 ✕ 𝐼𝑛𝑝𝑢𝑡), which is
undesirable since gradients are noisy
LRP Rules: LRP-𝟄

● This rule aims to solve the problem of gradient noise by introducing a small positive term, 𝟄, to the
denominator
● 𝟄 diminishes relevance scores, aiming to absorb some relevance when contributions to neuron 𝑘
are weak, contradictory, etc.
● As 𝟄 becomes larger, only the most salient explanation factors are preserved
● Result: sparser explanations in terms of input features, and less noise
LRP Rules: LRP-𝞬

● This rule aims to reduce noise and improve stability by favoring the effect of positive contributions
over negative ones with the introduction of a 𝞬 parameter applied to 𝑤𝑗𝑘.
● As 𝞬 increases, negative contributions disappear
● Limits how large positive and negative contributions can grow during propagation, improving
stability
Bonus LRP Rule: LRP-𝞪𝞫

● Like LRP-𝞬, this rule aims to treat positive and negative contributions asymmetrically
● Applies two parameters, 𝞪 and 𝞫, to positive and negative contributions, respectively
● Subject to conservation constraint 𝞪 = 𝞫 + 1
● Using LRP-𝞬 where 𝞬 = ∞ causes LRP-𝞬 to become equivalent to LRP-𝞪𝞫 where 𝞪 = 1 and 𝞫
= 0 (among other rules not covered in this paper)
 Apply LRP rule to
weights

Implementing LRP Efficiently

● Consider the generic LRP rule (pictured right)

● For any layer 𝑗, 𝑅𝑗 can be computed in four steps:

● Note the third step is equivalent to a gradient computation, where 𝒂 is the vector of lower-layer
activations:
Implementing LRP Efficiently (in code)
LRP as a Deep Taylor Decomposition

● Deep Taylor Decomposition views LRP as a succession of Taylor expansions performed at each
neuron
● Treat the relevance score 𝑅𝑘 as a function of lower-level activations (𝑎𝑗 )𝑗 denoted by the vector 𝒂,
and then perform a first-order Taylor expansion of 𝑅𝑘(𝒂) at some reference point in the space of
activations:
LRP as a Deep Taylor Decomposition (ii)

● DTD requires a closed-form expression for the terms of the previous equation
● Substitute the true relevance function with a model that is easier to analyze:

● Modulation term 𝑐𝑘 is a constant set in such a way that 𝑅෠𝑘 (𝒂) = 𝑅𝑘(𝒂) at the current data
point
● Then the Taylor expansion becomes:
LRP as a Deep Taylor Decomposition (iii)

● Relation to LRP-0/𝟄/𝞬: LRP rules can be recovered within the DTD framework by changing the
reference point:
○ LRP-0: 0
○ LRP-𝟄: 𝟄 · (𝑎𝑘 + 𝟄) − 1 𝒂
○ LRP-𝞬:
LRP as a Deep Taylor Decomposition (iv)
LRP as Deep Taylor Decomposition (v)

● LRP-0: ෥
𝒂=0

● LRP-𝟄: ෥
𝒂=𝟄 · (𝑎𝑘 + 𝟄) − 1 𝒂

● LRP- 𝞬: ෥
𝒂=
Which LRP Rule for Which Layer?
Properties of Explanations

● LRP is a general framework for propagation, leaving flexibility for different rules at each layer, and
for the parameters ε and γ
○ Optimal selection for parameters requires a measure of explanation quality, which is still being researched

● Focus on 2 main explanation properties: fidelity and understandability

○ Fidelity is the accuracy of the explanation’s representation of the output neuron
■ To visually assess fidelity, we must assume the network properly solved the task (is using correct
visual features and avoiding distracting elements)
○ Understandability is the interpretability of the explanation to a human
Properties of Explanations (ii)

● Explanation is complex
○ Lacks understandability

● Fails to focus on castle

○ Lacks fidelity

red = positively relevance

Model: VGG-16
blue = negatively relevance
Properties of Explanations (ii)

● Explanation is very sparse

○ Lacks understandability

● Much of the noise is removed

○ Has fidelity

red = positive relevance

Model: VGG-16
blue = negative relevance
Properties of Explanations (ii)

● Explanation is very clearly outlined

○ Has understandability

● Too much is highlighted (Ex. lamp post)

○ Lacks fidelity

red = positive relevance

Model: VGG-16
blue = negative relevance
Properties of Explanations (ii)
● Well outlined explanation
○ Has understandability
● Castle is correctly identified
○ Has fidelity

red = positive relevance

Model: VGG-16
blue = negative relevance
Rule Choices with VGG-16

LAYER RULE EXPLANATION

Upper LRP-0 ● Upper layers have about 4000 neurons (4 per class)
● Relatively low neuron count entangles concepts that form classes
● LRP-0 is close to function and gradient, and can ignore entanglements

Middle LRP-ε ● Middle layers are less entangled, but layer stacking and convolution
weight sharing adds spurious variations
● LRP-ε can filter out spurious variations

Lower LRP-γ ● Although similar to middle layers, LRP-γ at these layers uniformly spreads
relevance to whole features instead of individually calculating each pixel
● This helps make the explanation more understandable
Handling the Top Layer

red = positive relevance

Classification Task: Passenger Car blue = negative relevance
Conclusion
Conclusion

● Layer-wise Relevance Propagation (LRP) can explain SOTA predictions in terms of their input
features by propagating the prediction backwards through the model with various rules
● These can be implemented efficiently and modularly (in most modern neural net softwares)
● Through parameter tuning even complex models can have high quality explanations
● With Neuralization-Propagation (NEON), LRP can be applied beyond DNNs to other model types,
increasing its scope to help many other scenarios that require explainable machine learning
solutions
For

● LRP satisfies several properties of good explanatory ML techniques, and produces faithful and
understandable explanations
● LRP can be extended to a broad range of ML models beyond just DNN
● LRP can be implemented efficiently compared to other explanation techniques
● LRP can be easily modified to fit a variety of use cases via different rules
Against

● Little empirical evidence presented, with no comparison to other SOTA methods

● Many types of LRP rules were left out and it is unclear why this is the case
● LRP itself only applies to ReLU activations
● Explanation quality still requires human assessment, which can be time-consuming and error-
prone
● No formal evaluation criteria, only fidelity and understandability, which are subject to bias
○ Fidelity requires the assumption that the model is functioning exactly as intended as well
● Authors offer heuristics for applying LRP rules (i.e., “use epsilon for middle layers”), but only
support these with intuition rather than with more rigorous forms of evidence
● The relationship to the DTD framework is clear, but it is not clear why this relationship is valuable