Jean Moral Theory Meets Cognitive

Nicod Science
Lecture
s
How the Cognitive Sciences Can
2007 Transform Traditional Debates

Stephen Stich
Dept. of Philosophy
& Center for Cognitive Science
Rutgers University
[email protected]

1
 Jean
Nicod
Lecture 4
Lecture Stephen Stich
s Daniel Kelly
2007 Joshua Knobe

Debunking Moral Intuition

A Hodgepodge of Multipurpose Kludges

2
 Jean
Nicod
Lecture 4
Lecture Stephen Stich
s Joshua Knobe
2007 Daniel Kelly

Debunking Moral Intuition

A Hodgepodge of Multipurpose Kludges

3
 Introduction
 Philosophers – and more recently cognitive
scientists – have offered many accounts of the
psychological mechanisms & processes
underlying intuitive moral judgment

 Moral philosophers have always insisted that

sometimes the outputs of those processes –
people’s “moral intuitions” – are not to be
trusted
 though they disagree about when skepticism is
warranted

4
 Introduction
 Our goal in this talk is to sketch a newly
emerging perspective on the mechanisms
underlying moral intuition …

 and to explore its implications for the hotly

debated issue of whether and when intuitions
should be relied on

5
 Introduction
 Philosophers have typically assumed that those
mechanisms were well designed for …
something

 But we now have reasons to think that many of

theses mechanisms are not well designed
for ANYTHING

6
 Introduction

Moral Psychology is a
Kludge

A hodgepodge of multipurpose
kludges!

7
 Introduction
 Before explaining and defending this claim it
will be useful to consider some of the reasons
that philosophers – both classic &
contemporary – have offered for discounting
moral intuitions

8
 Philosophical Background
 When should we be skeptical about
moral intuitions?

 The “Moral Sense”

Sense & “Ideal Observer”
Observer traditions
 Reflective Equilibrium
 Evolutionary arguments debunking intuition

9
 Philosophical Background
 The “Moral Sense”
Sense & “Ideal Observer”
Observer traditions

 Ideal observer theorists maintain that our moral

intuitions are correct (or justified) when made under
ideal conditions

 When conditions are not ideal – e.g. when we have

false beliefs about relevant non-moral matters, or we
are irrational – our intuitions are not to be trusted

10
 Philosophical Background
 The “Moral Sense”
Sense & “Ideal Observer”
Observer traditions

 For Hutcheson – an important precursor of this

tradition – moral judgments are the product of a “moral
sense” implanted in us by “the Author of Nature”
 Thus it can be relied upon when doing its job
properly
 But, like other senses, it can mislead when
conditions are unfavorable

11
 Philosophical Background
 Reflective Equilibrium
 Rawls’ “Decision Procedure for Ethics”
(1951)

 Narrow Reflective Equilibrium

 Bring intuitions about
 particular cases
 moral principles
into accord
 To do this, sometimes an intuition about a particular
case must be rejected
12
 Philosophical Background
 Wide Reflective Equilibrium
 Bring intuitions about
 particular cases
 moral principles
into accord with the rest of our beliefs
 including beliefs about scientific matters, history,
politics – even metaphysics & semantics
 Even more of our intuitions about particular cases
will have to be rejected

13
 Philosophical Background
 Evolutionary arguments debunking intuition
 Perhaps the most influential writer in this
tradition is Peter Singer The
Expanding
Circle
Ethics and Sociobiology
Peter Singer

FARRAR, STRAUS & GIROUX

New York
1981

Updated in “Ethics & Intuition

(2005)
 Philosophical Background
 In The Expanding Circle, Singer focuses on
nepotistic intuitions which maintain that, in various
domains, we ought to value the welfare of our kin and
tribesmen more than the welfare of people outside
these circles
 The psychological processes leading to judgments of
this sort were adaptive in ancestral environments
(and perhaps they still are)
 But once we see why we have these nepotistic &
tribal intuitions,
intuitions Singer suggests, we can also see
that there is no good reason to use them in a
“decision procedure for ethics”
15
 Philosophical Background
 In “Ethics and Intuition” (2005) Singer develops the
argument by focusing on the sort of “trolley problems”
that have loomed large in recent philosophical and
empirical studies

16
 Philosophical Background
 Singer (following Greene) maintains that the
neuroscientific evidence suggests that
intuitions about the “footbridge” case are the
result of our emotional reaction to cases in
which harm is caused by the sort of
interaction that would have occurred in
ancestral environments

17
 Philosophical Background
“The salient feature that explains our different intuitive judgments
concerning the two cases is that the footbridge case is the kind
of situation that was likely to arise during the eons of time over
which we were evolving; whereas the standard trolley case
describes a way of bringing about someone’s death that has
only been possible in the past century or two…. But what is
the moral salience of the fact that I have killed someone in
a way that was possible a million years ago, rather than in
a way that became possible only two hundred years ago?
I would answer: none….

18
 Philosophical Background
“At [a] more general level …this … casts serious doubt on the
method of reflective equilibrium. There is little point in
constructing a moral theory designed to match considered
moral judgments that themselves stem from our evolved
responses to the situations in which we and our ancestors
lived during the period of our evolution as social mammals,
primates, and finally, human beings. We should, with our
current powers of reasoning and our rapidly changing
circumstances, be able to do better than that.” (348)”

What I am saying, in brief, is this. Advances in our understanding

of ethics … undermine some conceptions of doing ethics
…. Those conceptions of ethics tend to be too respectful
of our intuitions. Our better understanding of ethics gives
us grounds for being less respectful of them.”
them. (349)

19
 Philosophical Background

 We agree with Singer’s skepticism about intuition

 But we also think his skepticism is

not radical enough!

20
 Philosophical Background
 Assumptions that Singer and the friends of
intuition share:
share

 The psychological system underlying our moral

intuitions is well designed

 Thus there is some point to – or reason for – the

intuitive moral judgments people make when the
system is working properly
 Though Singer (unlike the friends of intuition) insists that
the function the system is designed for is of dubious
moral importance,
importance and thus that the intuitions are not to
be taken seriously
21
 Philosophical Background
 We believe that the engine of moral intuition is not
well designed at all

 Far from being the sort of “elegant machine”

celebrated in the writings of some evolutionary
psychologists, we think that it is a kludge

 a cluster of mechanisms cobbled together rather

awkwardly from bits of mental machinery most of
which were designed for functions that have
noting to do with morality

22
 Philosophical Background
 To use a term that may be more common in Paris,
we maintain that the engine of moral intuition is the
result of bricolage

François Jacob Claude Lévi-Strauss 23

 Philosophical Background
 This explains many of the quirks of moral
intuition …

 And provides yet another reason to be

skeptical of their use in moral deliberation

24
Overview of the Rest of the Talk

25
Overview of the Rest of the Talk

 Two examples of the “kludginess” of the

mechanisms underlying moral intuition
 Dan Kelly’s work on Moral Disgust
 Joshua Knobe’s work on intentionality
judgments & unconscious moral
judgments

 From kludginess to skepticism

26
 Kelly on Disgust
 Kelly has constructed a rich,
nuanced, empirically supported
account of the psychological
mechanisms underlying the
uniquely human disgust
system and how that system
evolved Daniel
Kelly
 In this talk I’ll only have time to for a
brief sketch of two central themes
27
 Kelly on Disgust
 The Entanglement Thesis
 Disgust is itself a kludge – a uniquely
human emotion produced by the merger of
two distinct systems
 The Co-Optation Thesis
 After the merger, disgust was co-opted by
 the norm system
 the ethnic boundary system
which were central elements in the
emergence of human ultra-sociality
28
 Kelly on Disgust
 Kelly assembles a vast array of evidence for
these theses, drawn from
 neuroscience
 social psychology
 cognitive psychology
 developmental psychology
 evolutionary psychology
 gene-culture co-evolution theory
 As usual, the devil is in the details
 So I join Paul Rozin in urging that you read the
work as it appears in print
29
 Kelly on Disgust
The Entanglement Thesis

 Disgust exhibits a puzzling array of

elicitors

which evoke an equally puzzling cluster of

responses

30
 Kelly on Disgust
The Entanglement Thesis

 Elicitors include
 Foods:
Foods dog meat, grubs, insects

31
 Kelly on Disgust
The Entanglement Thesis

 Elicitors include
 Foods:
Foods dog meat, grubs, insects
 Substances associated with the body:
body feces, vomit,
spit
 Organic decay
 People and objects associated with illness:
illness a shirt
once worn by a person with leprosy
 Sexual practices:
practices necrophilia, incest
 Some moral transgressions & transgressors: ors rape,
torture, child molestation
 Members of low status outgroups:
outgroups untouchables,
Jews
32
 Kelly on Disgust
Some elicitors are pan-cultural
The Entanglement Thesis

 Elicitors include
 Foods:
Foods dog meat, grubs, insects
 Substances associated with the body:
body feces, vomit,
spit
 Organic decay
 People and objects associated with illness:
illness a shirt
once worn by a person with leprosy
 Sexual practices:
practices necrophilia, incest
 Some moral transgressions & transgressors: rape,
torture, child molestation
 Members of low status outgroups:
outgroups untouchables,
Jews
33
 OthersKelly on Disgust
are culturally local
The Entanglement Thesis
(or idiosyncratic)
 Elicitors include
 Foods:
Foods dog meat, grubs, insects
 Substances associated with the body:
body feces, vomit,
spit
 Organic decay
 People and objects associated with illness:
illness a shirt
once worn by a person with leprosy
 Sexual practices:
practices necrophilia, incest
 Some moral transgressions & transgressors: rape,
torture, child molestation
 Members of low status outgroups:
outgroups untouchables,
Jews
34
 Kelly on Disgust
The Entanglement Thesis

 The disgust response includes

 Gape face (occasionally accompanied by retching)
 Feeling of nausea
 Sense oral incorporation

 Quick withdrawal
 A more sustained & cognitive sense of
offensiveness
 A more sustained & cognitive sense of
contamination
35
 Kelly on Disgust
The Entanglement Thesis

 How are all of these connected?

connected

 The Entanglement Thesis maintains that the

human emotion of disgust is the result of the
fusion of two distinct mechanisms
 each of which has homologous counterparts in
other species
 though they have combined only in humans

36
 Kelly on Disgust
The Entanglement Thesis

 One mechanism (“the poison avoidance

mechanism”)
mechanism is directly linked to digestion
 It evolved to regulate food intake and protect the gut
against ingested substances that are poisonous or
otherwise harmful
 It was designed to expel substances entering the
gastro-intestinal system via the mouth
 And to acquire new elicitors very quickly
 As John Garcia famously demonstrated, ingested

substances that induce gut-based distress often

generate acquired aversions

37
 Kelly on Disgust
The Entanglement Thesis

 The other mechanism (“the parasite avoidance

mechanism”)
mechanism
 Evolved to protect against infection from pathogens
and parasites,
parasites by avoiding them
 Not specific to ingestion, but serves to guard against
coming into close physical proximity with infectious
agents
 This involves avoiding not only visible pathogens and
parasites,
parasites but also places, substances and other
organisms that might be harboring them

38
These elements Kelly on Disgust
of the disgust response are
traceable to theThe
poison avoidance
Entanglement Thesissystem

 The disgust response includes

 Gape face (occasionally accompanied by retching)
 Feeling of nausea
 Sense oral incorporation

 Quick withdrawal
 A more sustained & cognitive sense of
offensiveness
 A more sustained & cognitive sense of
contamination
39
 Kelly
and these are on Disgust
traceable to
the parasite avoidance poison
The Entanglement system
Thesis

 The disgust response includes

 Gape face (occasionally accompanied by retching)
 Feeling of nausea
 Sense oral incorporation

 Quick withdrawal
 A more sustained & cognitive sense of
offensiveness
 A more sustained & cognitive sense of
contamination
40
 Kelly are
These elicitors on traceable
Disgust to
the poison
Theavoidance system
Entanglement Thesis

 Elicitors include
 Foods:
Foods dog meat, grubs, insects
 Substances associated with the body:
body feces, vomit,
spit
 Organic decay
 People and objects associated with illness:
illness a shirt
once worn by a person with leprosy
 Sexual practices:
practices necrophilia, incest
 Some moral transgressions & transgressors: rape,
torture, child molestation
 Members of low status outgroups:
outgroups untouchables,
Jews
41
 Kelly
and these are on Disgust
traceable to
the parasite
The avoidance
Entanglementsystem
Thesis

 Elicitors include
 Foods:
Foods dog meat, grubs, insects
 Substances associated with the body:
body feces, vomit,
spit
 Organic decay
 People and objects associated with illness:
illness a shirt
once worn by a person with leprosy
 Sexual practices:
practices necrophilia, incest
 Some moral transgressions & transgressors: rape,
torture, child molestation
 Members of low status outgroups:
outgroups untouchables,
Jews
42
 Kelly on Disgust
The Entanglement Thesis

 One bit of evidence supporting the Entanglement

Thesis is that different components of that response
are on different developmental schedules
 Distaste & gape are present within the first year of life
 Contamination sensitivity emerges significantly later

 Once the full system in in place, the components of

the response are produced together – they form a
nomological cluster
 Any elicitor of disgust will reliably produce all or most of
those clustered components

43
 Kelly on Disgust
The Entanglement Thesis

 A puzzle:
puzzle
 Why should the sight of a festering sore or a person
with leprosy evoke a gape face and a feeling of
nausea?

 The solution:
solution Disgust is a kludge!
kludge

 But it is kludge with features that could be readily co-

opted and put to other uses as humans began living
in larger groups and human ultrasociality emerged

44
Kelly on Disgust
The Co-Optation Thesis

45
 Kelly on Disgust
The Co-Optation Thesis

 The Gape Face as a Signal

 As group size increased, there was an increasing need
for a perspicuous signal warning of dangerous
foods and risk of infectious disease
 In humans, the face and facial expressions provide a
rich source of such social information
 The gape face,
face which clearly has roots in the facial
motions that accompany retching, was co-opted as a
signal,
signal warning others not just against toxic foods,
foods but
also against the presence of parasites and
contagious pathogens

46
 Kelly on Disgust
The Co-Optation Thesis

 Co-Optation by the Norm System

 As group size increased, there was increased need for
complex social coordination
 The norm system – whose structure we considered
briefly in the 2nd Lecture – played an important role in
facilitating this co-ordination
 And the disgust system had features that made it an
obvious candidate to be co-opted by the norm system
as it evolved

47
 Kelly on Disgust
The Co-Optation Thesis

 The S&S model suggests that compliance motivation &

punitive motivation are linked to “the emotion system”

48
 Kelly on Disgust
The Co-Optation Thesis

other
Acquisition emotion
Execution Mechanism
Mechanism triggers

norm data complian belief

base ce s
infer contents of
normative rules
identify norm

r1---------- motivatio
implicating
behavior

r2---------- n
r3---------- emotio judgment
…… n
Rule-
rn---------- system
related
reasoning punitive
capacity motivati
on explicit
reasonin
g
Proximal
Cues in
Environment post-hoc
justificatio
49
n
 Kelly on Disgust
The Co-Optation Thesis

 But psychological & neurological evidence indicates

that there are several separate emotion systems –
the disgust system being one of them

50
 Kelly on Disgust
The Co-Optation Thesis

other
Acquisition emotion
Execution Mechanism
Mechanism triggers

norm data complian belief

base ce s
infer contents of
normative rules
identify norm

r1---------- motivatio
implicating
behavior

r2---------- n
DISGUST
r3---------- judgment
…… other
Rule-
rn---------- emotion
related s
reasoning punitive
capacity motivati
on explicit
reasonin
g
Proximal
Cues in
Environment post-hoc
justificatio 51
n
 Kelly on Disgust
The Co-Optation Thesis
 Disgust is a natural candidate to provide both
compliance & punitive motivation for norms that
involve intrinsically disgusting matters, like the disposal
of corpses & bodily wastes, and other activities that are
antecedently salient to the disgust system, like eating
practices
 Compliance is motivated by making norm violating
behavior disgusting & thus aversive
 Punitive motivation is provided because the violator
is considered dirty and contaminated and is
avoided or shunned

52
 Kelly on Disgust
The Co-Optation Thesis

other
Acquisition emotion
Execution Mechanism
Mechanism triggers

norm data complian belief

base ce s
infer contents of
normative rules
identify norm

r1---------- motivatio
implicating
behavior

r2---------- n
DISGUST
r3---------- judgment
…… other
Rule-
rn---------- emotion
related s
reasoning punitive
capacity motivati
on explicit
reasonin
g
Proximal
Cues in
Environment post-hoc
justificatio 53
n
 Kelly on Disgust
The Co-Optation Thesis

 The norm system is thus a kludge built with kludgy

parts
 Not surprisingly, this can lead to some very quirky and
disturbing behavior
 Several recent studies have focused on the fact that
the disgust system can be triggered by many things
that have nothing to do with norms
 but even when triggered by these non-moral items,
items
the disgust system can have dramatic and persistent
influence on a person’s judgments about moral
issues

54
 Kelly on Disgust
The Co-Optation Thesis

other
Acquisition emotion
Execution Mechanism
Mechanism triggers

norm data complian belief

base ce s
infer contents of
normative rules
identify norm

r1---------- motivatio
implicating
behavior

r2---------- n
DISGUST
r3---------- judgment
…… other
Rule-
rn---------- emotion
related s
reasoning punitive
capacity motivati
on explicit
reasonin
g
Proximal
Cues in
Environment post-hoc
justificatio 55
n
 Kelly on Disgust
The Co-Optation Thesis

 Wheatley & Haidt have shown that when participants

are hypnotically induced to feel a brief pang of
disgust when they encounter the work “often” and
then presented with the following scenario

“Dan is a student council representative at his school.

This semester he is in charge of scheduling
discussions about academic issues. He often picks
topics that appeal to both professors and students in
order to stimulate discussion.”

many judge that Dan is doing something wrong!

wrong
56
 Kelly on Disgust
The Co-Optation Thesis

 Schnall et al. have shown participants make more

severe moral judgments when the judgments are
made in a disgusting office:

 greasy pizza boxes

 sticky chair
 a dried up smoothie
 a chewed up pen

57
 Kelly on Disgust
The Co-Optation Thesis

 Other studies have focused on prima facie irrational

downstream consequences of the disgust system
being triggered in moral deliberation

58
 Kelly on Disgust
The Co-Optation Thesis

Downstream
other consequenc
Acquisition emotion es
Execution Mechanism
Mechanism triggers

norm data complian belief

base ce s
infer contents of
normative rules
identify norm

r1---------- motivatio
implicating
behavior

r2---------- n
DISGUST
r3---------- judgment
…… other
Rule-
rn---------- emotion
related s
reasoning punitive
capacity motivati
on explicit
reasonin
g
Proximal
Cues in
Environment post-hoc
justificatio 59
n
 Kelly on Disgust
The Co-Optation Thesis

 The Lady Macbeth Effect

 Zhong & Liljenquist have shown that recalling an

unethical deed increased the desire for products
related to cleansing, like antiseptic wipes

 And that cleaning one’s hands after describing a past

unethical deed reduced moral emotions like guilt &
shame
 and also reduced the likelihood that participants

would volunteer to help a desperate graduate

student!
60
 Kelly on Disgust
The Co-Optation Thesis

 The Lady Macbeth Effect

 Schnall et al. (unpublished) compared judgments

about moral severity in two groups of participants
 One group had just used an alcohol-based

cleansing gel on their hands

 The other group had just used an ordinary, non-

cleansing hand cream

 The moral judgments of those using the cleansing gel

were significantly less severe!

61
 Kelly on Disgust
The Co-Optation Thesis

 Ethnic Boundary Markers

 Boyd & Richerson & their students have argued that
another crucial step in the development of human ultra-
sociality was the emergence of mechanisms which
allow people to recognize members of their own tribe or
“ethnie”

 This is important because in-group members share

beliefs & norms that facilitate coordination

62
 Kelly on Disgust
The Co-Optation Thesis
 Since different cuisines & eating practices are one of
the more visible correlates of ethnie membership, and
since disgust is heavily involved in regulating food
intake, disgust was a natural candidate to be co-opted
by the emerging system of ethnic identification
 Eating practices of out-groups and other readily
detectable signs of out-group membership came to
evoke disgust
 And disgust came to provided a significant part of the
motivation to avoid out-group members

63
 Kelly on Disgust
The Co-Optation Thesis
 Though the evolutionary function of the ethnic
boundary marker system was to facilitate cooperation
by keeping groups apart, the kludgy solution to this
problem has some unfortunate consequences

 Out-group members are not simply avoided, they are

also considered offensive & contaminating

 People who embrace different norms are often felt to

be disgusting and sub-human!

64
Kludge Meets Kass

65
 Kludge Meets Kass
 Leon Kass, M.D., Ph.D.

 Conservative bio-ethicist

 Chairman of the U. S. A.
President's Council on
Bioethics from 2002 to 2005

66
 Kludge Meets Kass
 In his book, Life, Liberty & the Defense of Dignity (2002),
there is a chapter called “The Wisdom of Repugnance”

 Kass maintains that

 "in crucial cases...repugnance is the emotional

expression of deep wisdom,
wisdom beyond reason's power
fully to articulate it.”

 “In this age in which everything is held to be permissible

so long as it is freely done, and in which our bodies are
regarded as mere instruments of our autonomous
rational will, repugnance may be the only voice left that
speaks up to defend the core of our humanity. Shallow
are the souls that have forgotten how to shudder."
shudder
67
 Kludge Meets Kass
 The claims play a central role in Kass’ critique of human
cloning

 Others have adopted the idea to argue against abortion,

abortion
pornography & same-sex marriage

68
 Kludge Meets Kass
 Some philosophers, most notably
Martha Nussbaum, have
challenged Kass, arguing that
disgust should be discounted in
moral & legal deliberation because
(roughly) it reminds us of our
animal origins

69
 Kludge Meets Kass
I think Kelly’s work offers a far more
plausible &
powerful
critique

70
 Kludge Meets Kass
 There is no reason to think there is

wisdom in repugnance
because

Disgust is a Kludge

and the psychological system that bases moral judgments

on disgust is a

Kludge twice over!

71
 Kludge Meets Kass
 Anti-Jewish Nazi propaganda often invoked the
imagery and language of disgust, purity,
contamination & dehumanization very flagrantly

A poster advertising
the film The
Eternal Jew

Hitler described
“the Jew” as “a
maggot in a
festering abscess,
hidden away inside
the clean and 72
healthy body of the
 Knobe on Norms and
Intentional Action
 My second example draws
some elegant and exciting
work by Joshua Knobe
which demonstrates the
way in which unconscious
moral judgments –
judgments which an agent
may explicitly reject – can
nonetheless have
significant impact on a
range of morally relevant
intuitions
73
 Knobe on Norms and
Intentional Action
 In his new book, Kluge, Gary
Marcus argues that more recently
evolved, computationally slow and
consciously accessible mental
processes – “System 2 Processes”
in the currently fashionable jargon –
were grafted onto older (System 1)
psychological systems designed for
quite different purposes

 The resulting kludgy architecture

accounts for many of the quirks and
shortcomings that contemporary
cognitive science has discovered

74
 Knobe on Norms and
Intentional Action

 I think that Knobe’s work provides an

important & disquieting illustration of this
phenomenon in the moral domain

75
 Knobe on Norms and
Intentional Action
 The story begins with “the side effect effect” (aka
the Knobe effect) – one of best known and most
surprising finding in the emerging field of experimental
philosophy

 Knobe (2003) reports an experiment in which

participants were presented with a pair of almost
identical vignettes

76
 Knobe on Norms and
Intentional Action
The vice-president of a company went to the
chairman of the board and said, ‘We are thinking
of starting a new program. It will help us increase
profits, but it will also harm [help]
help the
environment.’

The chairman of the board answered, ‘I don’t care

at all about harming [helping]
helping the environment.
I just want to make as much profit as I can. Let’s
start the new program.’

They started the new program. Sure enough, the 77

environment was harmed [helped].
helped
 Knobe on Norms and
Intentional Action
 In the harm case, participants were asked how much
blame the chairman deserved (on a scale from 0 – 6)
and whether he intentionally harmed the
environment
 In the help case, participants were asked how much
praise the chairman deserved (on a scale from 0 – 6)
and whether he intentionally helped the environment

 In the harm case, 82% said the chairman brought

about the side-effect intentionally
 In the help case, 77% said the chairman did not bring
about the side-effect intentionally
78
 Knobe on Norms and
Intentional Action
 Knobe’s initial hypothesis was that people’s moral
assessment of the side-effect plays a substantial
role in determining whether they are willing to say that
the side-effect was brought about intentionally
 A judgment that the side-effect is morally bad makes
it more likely that it will be judged to be intentional

 Though this seems incompatible with the widespread

idea that judgments of intentionality are judgments
about a purely factual matter,
matter it does have an
obvious rationale since judgments about whether an
action is intentional play a central role in determining
whether an agent deserves praise or blame 79
 Knobe on Norms and
Intentional Action
 Subsequent research showed that, if the hypothesis is
understood as a claim about the effect of moral
judgments that people consciously make,
make this
hypothesis is mistaken

 The problem emerges clearly in study Knobe ran in

collaboration with David Pizarro & Paul Bloom

80
 Knobe on Norms and
Intentional Action
 Liberal university students were given Knobe-style
vignettes in which an advertising executive approves
an ad campaign which has the side-effect of
encouraging interracial sex
or placing gardenias in one’s office

81
 Knobe on Norms and
Intentional Action
 None of the participants judged that inter-
racial sex (or placing gardenias) is morally
wrong
 But participants were much more inclined to
say that the executive intentionally
encouraged interracial sex
 Explicit moral judgments cannot explain the
difference in judgments about the intention-
ality of the side-effects

82
 Knobe on Norms and
Intentional Action

 However, (following Pizarro & Bloom) Knobe has

recently proposed that perhaps participants were
making non-conscious normative judgments that
the behavior in question violates a norm that is
made salient by the question or situation, even if it is a
norm that they explicitly reject

83
 Knobe on Norms and
Intentional Action
 The picture Knobe now proposes looks like this:

“In reaching a conscious moral judgment,

judgment we can
consider a variety of different moral norms, weigh
these norms against each other, perhaps even
determine that some of the norms are themselves
unjustified.”

84
 Knobe on Norms and
Intentional Action
 Non-conscious moral judgments are formed
through a much simpler (system-1 style) process

 They are formed extremely quickly and therefore

involve very shallow processing

 In generating a non-conscious moral judgment, the

only norms we consider are the ones that first come
to mind.
mind We do not search for additional norms; we do
not weigh norms against each other; we do not ask
whether any of the norms might themselves be
unjustified.
unjustified
85
 Knobe on Norms and
Intentional Action

 Instead, we simply determine whether the behavior in

question violates any of the norms in the very limited
set we are considering

 If it does, we classify it as a transgression.

transgression It is this
judgment as to whether or not the behavior is a
transgression that then influences our intuitions
about intentional action.

86
 Knobe on Norms and
Intentional Action

 The theory predicts that the most salient norms

evoked by a given case will be the ones used to in
making intentionality judgments, even if subsequent
reflection leads the agent to think that there is nothing
wrong with violating the norm – or that doing so would
be a very good thing.

 Here is a vignette that Knobe has recently used to test

this idea

87
 Knobe on Norms and
Intentional Action
In Nazi Germany, there was a law called the ‘racial identification
law.’ The purpose of the law was to help identify people of
certain races so that they could be rounded up and sent to
concentration camps. Shortly after this law was passed, the
CEO of a small corporation decided to make certain
organizational changes. The Vice-President of the corporation
said: “By making those changes, you’ll definitely be increasing
our profits. But you’ll also be violating [fulfilling]
fulfilling the
requirements of the racial identification law.” The CEO said:
“Look, I know that I’ll be violating [fulfilling]
fulfilling the requirements of
the law, but I don’t care one bit about that. All I care about is
making as much profit as I can. Let’s make those organizational
changes!” As soon as the CEO gave this order, the corporation
began making the organizational changes.
 81% of subjects in the violate condition said that he violated
the requirements intentionally; 30% of subjects in the fulfill
condition said that he fulfilled the requirements intentionally.

88
 Knobe on Norms and
Intentional Action

 Knobe’s theory is certainly not the last word on how

intentionality judgments are generated
 His work has inspired dozens of other researchers
 there are many studies I have not mentioned

 and many others are underway

89
 Knobe on Norms and
Intentional Action
 However, IF Knobe’s theory is on the right track, then
intentionality judgments are a product of a kludgy
architecture which can be influenced by norms and
judgments which the agent
 is not aware of,
of and
 does not endorse

 This raises serious questions about the use of those

judgments in further moral deliberation,
deliberation or in the law

90
 From Kludginess to Skepticism
 Both Kelly’s & Knobe’s work support the hypothesis that
motivates this talk

The psychological mechanism underlying moral

intuition is

A Hodgepodge of Multipurpose Kludges

91
 From Kludginess to Skepticism
 Suppose that’s right. What should we conclude about
moral intuition?

 The answer is NOT that all moral intuition should be

rejected
 nor even that intuitions that are closely tied to kludgy

features of the mind should be rejected

 For, as Shaun Nichols has argued, some of the most

admirable features of the cultural evolution of norms –
including the increased scope and acceptance of
norms prohibiting physical harm – are the products of
kludgy design
92
 From Kludginess to Skepticism

 Rather, I suggest, the right conclusion to draw is that

ALL moral intuitions should be viewed with a
healthy dose of skepticism
 The mechanisms that give rise to them may not have
been well designed to do anything
 So we should be skeptical about moral intuitions for
roughly the same reason that we should be skeptical
of the output of a kludgy piece of computer
software

93
 From Kludginess to Skepticism

 Compare and Contrast

 The friends of intuition (e.g. moral sense theorists) think

the system producing them is well designed for morally
admirable goals
 though it can sometimes misfire when conditions are
unfavorable
 Previous enemies of intuition (e.g. Singer) think the system
producing them has been well designed for morally
problematic goals

 We believe that the system producing them is a kludge –

much of it has not been well designed at all!
94
 From Kludginess to Skepticism

 But if we should be skeptical about all

intuition, how can we go about making
moral decisions?

 That’s a BIG question & a HARD one.

one
 Perhaps I’ll be able to suggest an answer …

95
From Kludginess to Skepticism

…the next time I come to Paris

96