CS282BR: Topics in Machine Learning

Interpretability and Explainability

Hima Lakkaraju

Assistant Professor
Harvard Business School + Computer Science
 Background

 Strong understanding: Linear algebra, probability,

algorithms, machine learning (cs181 or equivalent),
programming in python, numpy, sklearn;

 Familiarity with statistics, optimization

2
 Motivation

Machine Learning is EVERYWHERE!!

[ Weller 2017 ]
Motivation: Why Model Understanding?

Input This model is

relying on incorrect
features to make
this prediction!! Let
me fix the model
Model understanding facilitates debugging.
Model Understanding

Predictive Prediction = Siberian Husky

Model
Motivation: Why Model Understanding?

This prediction is
Defendant Details biased. Race and
gender are being
used to make the
prediction!!
Model Understanding
Model understanding facilitates bias detection.
Race

Crimes

Gender

Predictive
Prediction = Risky to Release
Model

[ Larson et. al. 2016 ]

 Motivation: Why Model Understanding?
Model Understanding

Increase
Loan Applicant Details
salary by I have some means
50K + pay for recourse. Let me
credit card go and work on my
promotion and pay
bills on time
my bills on time.
Model understanding helps
for next 3 provide recourse to individuals
months
who are adversely to
affected by model predictions.
get a loan

Predictive
Prediction = Denied Loan
Model
Loan Applicant
Motivation: Why Model Understanding?

Model Understanding
Patient Data This model is using
If gender = female, irrelevant features when
if ID_num > 200, then sick predicting on female
subpopulation. I should
If gender = male, not trust its predictions
25, Female, Cold if cold = true and cough = true, then for that group.
32, Male, No
31, Male, Cough
Model understanding helps assess
sick if and when to trust
model predictions when making decisions.
Predictions

Healthy
Sick
Predictive Sick
Model .
.
Healthy
Healthy 7
Sick
Motivation: Why Model Understanding?

Patient Data Model Understanding

If gender = female, This model is using

if ID_num > 200, then sick irrelevant features when
predicting on female
25, Female, Cold If gender = male, subpopulation. This
32, Male, No if cold = true and cough = true, cannot be approved!
31, Male, Cough then sick
.

Predictions

Healthy
Sick
Predictive Sick
Model .
.
Healthy
Healthy 8
Sick
Motivation: Why Model Understanding?

Utility Stakeholders

Debugging End users (e.g., loan applicants)

Bias Detection Decision makers (e.g., doctors, judges)

Recourse Regulatory agencies (e.g., FDA, European

commission)
If and when to trust model predictions
Researchers and engineers
Vet models to assess suitability for
deployment
 Achieving Model Understanding

Take 1: Build inherently interpretable predictive models

[ Letham and Rudin 2015; Lakkaraju et. al. 2016 ]

 Achieving Model Understanding
Take 2: Explain pre-built models in a post-hoc manner

Explainer

[ Ribeiro et. al. 2016, 2018; Lakkaraju et. al. 2019]

 Inherently Interpretable Models vs.
Post hoc Explanations

Example

In certain settings, accuracy-interpretability trade offs may exist.

[ Cireşan et. al. 2012, Caruana et. al. 2006, Frosst et. al. 2017, Stewart 2020]
Inherently Interpretable Models vs.
Post hoc Explanations

can build interpretable + complex models might

accurate models achieve higher accuracy
 Inherently Interpretable Models vs.
Post hoc Explanations
Sometimes, you don’t have enough data to build your model
from scratch.

And, all you have is a (proprietary) black box!

[ Ribeiro et. al. 2016 ]

Inherently Interpretable Models vs.
Post hoc Explanations

If you can build an interpretable model which is also adequately

accurate for your setting, DO IT!

Otherwise, post hoc explanations come to the rescue!

Let’s get into some details!
 Next Up!

 Define and evaluate interpretability

 somewhat! 

 Taxonomy of interpretability evaluation

 Taxonomy of interpretability based on

applications/tasks

 Taxonomy of interpretability based on methods

[Doshi-Velez & Kim, 2017]
17
 Motivation for Interpretability

 ML systems are being deployed in complex high-

stakes settings

 Accuracy alone is no longer enough

 Auxiliary criteria are important:

 Safety
 Nondiscrimination
 Right to explanation

18
 Motivation for Interpretability

 Auxiliary criteria are often hard to quantify

(completely)
 E.g.: Impossible to enumerate all scenarios violating safety
of an autonomous car

 Fallback option: interpretability

 If the system can explain its reasoning, we can verify if
that reasoning is sound w.r.t. auxiliary criteria

19
 Prior Work: Defining and Measuring
Interpretability
 Little consensus on what interpretability is and how
to evaluate it

 Interpretability evaluation typically falls into:

 Evaluate in the context of an application

 Evaluate via a quantifiable proxy

20
 Prior Work: Defining and Measuring
Interpretability
 Evaluate in the context of an application
 If a system is useful in a practical application or a
simplified version, it must be interpretable

 Evaluate via a quantifiable proxy

 Claim some model class is interpretable and present
algorithms to optimize within that class
 E.g. rule lists

You will know it when you see it!

21
 Lack of Rigor?

 Yes and No
 Previous notions are reasonable
Important to formalize these notions!!!

 However,

 Are all models in all “interpretable” model classes equally

interpretable?
 Model sparsity allows for comparison

 How to compare a linear model with a decision tree?

 Do all applications have same interpretability needs?

22
 What is Interpretability?

 Defn: Ability to explain or to present in

understandable terms to a human

 No clear answers in psychology to:

 What constitutes an explanation?
 What makes some explanations better than the others?
 When are explanations sought?

23
 When and Why Interpretability?

 Not all ML systems require interpretability

 E.g., ad servers, postal code sorting
 No human intervention

 No explanation needed because:

 No consequences for unacceptable results
 Problem is well studied and validated well in real-world
applications  trust system’s decision

When do we need explanation then?

24
 When and Why Interpretability?

 Incompleteness in problem formalization

 Hinders optimization and evaluation

 Incompleteness ≠ Uncertainty
 Uncertainty can be quantified
 E.g., trying to learn from a small dataset (uncertainty)

25
Incompleteness: Illustrative Examples

 Scientific Knowledge
 E.g., understanding the characteristics of a large dataset
 Goal is abstract

 Safety
 End to end system is never completely testable
 Not possible to check all possible inputs

 Ethics
 Guard against certain kinds of discrimination which are too
abstract to be encoded
 No idea about the nature of discrimination beforehand

26
Taxonomy of Interpretability Evaluation

Claim of the research should match the type of the evaluation!

27
 Application-grounded evaluation

 Real humans (domain experts), real tasks

 Domain experts experiment with exact application

task

 Domain experts experiment with a simpler or partial

task
 Shorten experiment time
 Increases number of potential subjects

 Typical in HCI and visualization communities

28
 Human-grounded evaluation

 Real humans, simplified tasks

 Can be completed with lay humans
 Larger pool, less expensive

 Potential experiments
 Pairwise comparisons
 Simulate the model output
 What changes should be made to input to change the
output?

29
 Functionally-grounded evaluation

 No humans, just proxies

 Appropriate for a class of models already validated
 E.g., decision trees
 A method is not yet mature
 Human subject experiments are unethical
 What proxies to use?

 Potential experiments
 Complexity (of a decision tree) compared to other other
models of the same (similar) class
 How many levels? How many rules?

30
 Open Problems: Design Issues

 What proxies are best for what real world

applications?

 What factors to consider when designing simpler

tasks in place of real world tasks?

31
 Taxonomy based on applications/tasks

 Global vs. Local

 High level patterns vs. specific decisions

 Degree of Incompleteness
 What part of the problem is incomplete? How incomplete
is it?
 Incomplete inputs or constraints or costs?

 Time Constraints
 How much time can the user spend to understand
explanation?

32
 Taxonomy based on applications/tasks

 Nature of User Expertise

 How experienced is end user?
 Experience affects how users process information
 E.g., domain experts can handle detailed, complex
explanations compared to opaque, smaller ones

 Note: These taxonomies are constructed based on intuition

and are not data or evidence driven. They must be treated as
hypotheses.

33
 Taxonomy based on methods

 Basic units of explanation:

 Raw features? E.g., pixel values
 Semantically meaningful? E.g., objects in an image
 Prototypes?

 Number of basic units of explanation:

 How many does the explanation contain?
 How do various types of basic units interact?
 E.g., prototype vs. feature

34
 Taxonomy based on methods

 Level of compositionality:
 Are the basic units organized in a structured way?
 How do the basic units compose to form higher order
units?

 Interactions between basic units:

 Combined in linear or non-linear ways?
 Are some combinations easier to understand?

 Uncertainty:
 What kind of uncertainty is captured by the methods?
 How easy is it for humans to process uncertainty?
35
Questions??
 Relevant Conferences to Explore

 ICML
 NeurIPS
 ICLR
 UAI
 AISTATS
 KDD
 AAAI
 FAccT
 AIES
 CHI
 CSCW
 HCOMP 37
 Breakout Groups

 Say hi to your neighbors! Introduce yourselves!

 What topics are you most excited about learning as part of

this course?

 Are you convinced that model interpretability/explainability

is important?

 Do you think we can really interpret/explain models

(correctly)?

 What is your take on inherently interpretable models vs. post

hoc explanations? Would you favor one over the other? Why? 38