The Symbolic Foundations of Conditioned Behavior 1st

Edition

Visit the link below to download the full version of this book:

https://medipdf.com/product/the-symbolic-foundations-of-conditioned-behavior-1st
-edition/

Click Download Now

Copyright @ 2002 by Lawrence Erlbaum Associates, Inc.

All rights reserved. No part of the book may be reproduced in

any form, by photostat. microform, retrieval system, or any other
means, without the prior written permissionof the publisher.

Lawrence Erlbaum Associates. Inc., Publishers

10 Industrial Avenue
Mahwah, New Jersey 07430

Cover design by Kathryn Houghtaling Lacey

0-8058-2934-2

Books published by Lawrence Erlbaum Associates are printed on acid-free paper,

and their bindings are chosen for strength and durability.
Printed in the United States of America
1 0 9 8 7 6 5 4 3 2 1
Contents

Preface V
Series Preface ix

introduction 1

ResponseTiming 5
The Peak Procedure 9
Scalar Expectancy Theory 11
The FI Scallop 14
The Timing of Aversive CRs 15
Timing the CS: Discrimination 20
Summary22

Acquisition 23
Quantitative Results 26
Rate Estimation Theory 36
Generalizing the Model 46

Cue
Competition
and
inhibitory
Conditioning 57
Experimental Results 57
Two Principles 61
General Solution to the Rate Estimation Problem 62
Intuitive “Derivations” 65
Conclusions 78

iii
iv CONTENTS

4 Extinction 80
Model of Simple Extinction 83
Generalizing the Model 86

5 Backward,
Secondary,
and
Trace
Conditioning 10 7
Delay Conditioning Versus Trace Conditioning 108
Forward Versus Backward Conditioning 109
Secondary Versus Primary Conditioning 111

6 Operant
Choice 124
Opting Versus Allocating 125
Hyperbolic Discounting and Self-Control 141
Harmonic Averaging andthe Preference
for Variabillty 143
The Equivalence of Delayed Rewards
and Probabilistic Rewards 148
lime-Scale Invariance in Free Operant Avoidance 151
Summary153

7 The
Challenge
for
Associative
Theory 156
Different Answers to Basic Questions 156
Contrasting Basic Assumptions 159
The Challenges Posed by Experimental Findings 163
Summary176

References 178

AuthorIndex 189

Subject
Index 193
 Preface

As this book goes to press, there is an ongoing discussion on the Animal Learning
and Behavior List Server about the decline of behaviorism in psychology and the
rise of cognitivism. Most of the participants in the discussion do research and
teaching in animal learning. Most of them remain to varying degrees behaviorists.
They are, by and large, uncomfortable with cognitive theorizing and unhappy
about the decline of behaviorism. In this book, we hope to persuade students of
animal learning that cognitive theorizing is essential for an understanding of the
phenomena revealed by conditioning experiments.We hope also to persuade the
cognitive psychology community that conditioning phenomena offer such a strong
empirical foundation for a rigorous brand of cognitive psychology that the study
of animal learning should reclaim a more central place in the fieldof psychology.
There is, we believe, no way to achieve a coherent uilderstandingof animal
conditioning phenomena without recognizing that computational processingof
information-bearing sensory signals constructs a symbolic representation of
selected aspects of the animal’s experience, which is stored in memory and
subsequently retrieved for use in the decision processes that determine the behavior
we observe. These essentially cognitive notions-information processing,
computation, symbolic representation, memory storage, retrieval from memory,
and decision processes-are in no sense merely metaphors. They are what the
brain is doing to produce the behavior we observe. We cannot understand the
phenomena revealed by conditioning experiments without understanding the
structure of the underlying information-processing operations in the brain, any
more than we can understand the phenomena of chemistry without understanding
the structure of the underlying atoms.
We believe our analysis of the work we review also merits the attention of
the cognitive science and philosophy communities because it speaks directly to a
central question that separates many forms of connectionist modeling from the
artificial intelligence(AI) approach to the mind. This is also a question that figures
prominently in the philosophy of mind, particularly those aspects of the philosophy
of mind that have been influenced by connectionist modeling and by
neurobiological considerations, that is,by the argument that the explanation for
how the mind works is to be sought in our current understanding of how the brain
works. Here the question is whether learning is primarily a matter of learning
that or primarily a matter of learning to.
In traditional AI and traditional philosophy of mind, computers and minds
learn that something is true about the world. That is, they acquire beliefs. What
they then do follows from what they believe together with the computer’s goals

V
vi PREFACE

or the mind’s intentions. Connectionist modeling-and the philosophy of mind

allied to it-rejects this account in favor ofan account in which symbolic
representation of properties of the world (beliefs) play no role. Learning in
connectionist computers (and in the brains imaginedby philosophers who base
their arguments of presumptions about how the brain works) is assumed to be a
matter of learning to do something-that is, to respond in a certainway to certain
inputs-by means of a rewiring process. The rewiring is drivenbyeither feedback
from the consequences of previous actions (back propagation) or by statistical
patterns of co-occurrence in the inputs (Hebbian or unsupervised learning
mechanisms).
It is often taken for granted by both schools of thought that animals in
conditioning experiments are learning to respond rather than learning that certain
things are true about the environment in which they find themselves during the
experiment. The question is often taken to be whether anything other than the
learning to that we supposedly see in animal conditioning experiments is required
to explain human learning. Our argument is that the behavior we see in conditio
experiments is the consequence of learning that certain things are true, rather than
learning to respondin certain ways.In other words, the symbol processing nature
of mental and neural activity is evident in simple conditioning, where the behavio
that emerges must be seen as driven by what the animal has come to believe is
true. The rabbit blinksin response to a conditioned tone, because it has come to
believe that a shock to the orbit follows the tone at a half-second latency with
great regularity.
One might suppose, therefore, that we are at risk of “leaving the rat lost in
thought” as Guthrie is said to have remarked about Tolman’s account of spatial
learning. On the contrary, however, our models are built around decision processes,
which are formally specified models of how what the animal knows gets translat
into whatit does. Our models are explicit about how what has been learned becom
manifest in what is done, whereas associative models (learning to models) are
famously vague on this very point-how the rewiring caused by experience gets
translated into observable behavior.
We believe that our arguments also merit the attention of the neurobiological
community. What onelooks for in the nervous system is strongly determined by
what a behaviorally based conceptual scheme leads one to to expect
find there.If
you think that talk about the memory for a variable like the duration of an inte
or the magnitude of a reward, or the distance between two landmarks is just a
metaphor, then you are not going to waste your time looking for its neurobiolog
realization, any more than the many biochemists who thought genes were just
metaphors wasted their time looking for the molecular realization of the gene.
Neuroscientists interested in the neurobiology of learning and memory are
not, for the most part, looking for the mechanisms of information processing.
Rather they are looking for associative bonds in the nervous system, that is, changes
PREFACE vii

in synaptic conductance produced by the temporal pairing of synaptic signals.

This is evidentin the following quotation from the abstract of an articlein Nature,
which attracted widespread attention from the press (e.g., Wade, 1999), as a
possible neuroscience breakthrough:

Hebb’s rule (1949) statesthat learning and memory are based on modifications
of synaptic strength among neurons that are simultaneously active. This implies
that enhanced synaptic coincidence detection would lead to better learning and
memory. If the NMDA (N-methyl-D-aspartate) receptor, a synaptic coincidence
a graded switchfor memory formation, enhanced signal detection
detector, acts as
by NMDA receptors, should enhance learning and memory. . . . (Tang et al.,
1999, p. 63)

There is, we argue, little empirical foundation for the claim that coincidence
detection on a time scale of a few hundred milliseconds plays any role in learning
or memory, at least notin the learning and memory processes that mediate basic
conditioning. Aswe show at some length, the behavioral data indicate that what
matters in conditioning are the relative durations of the intervalsin the protocol,
not their absolute durations. The importance of relative durations as opposed to
absolute durations is central to our suggestion that conditioning processes are
time-scale invariant.
We argue that what neuroscientists ought to be looking for are not mechanisms
of synaptic plasticity activated bynarrowtemporal coincidences but rather
mechanisms by which variables may be stored and retrieved. Mechanisms for the
storage and retrieval of variables, together with mechanisms for doing
computations with those variables, are the heart and soul of a conventional
computer, so there can be no question about the physical realizability of such
mechanisms. How they are realized i n neural tissue is anothermatter.
Neuroscientists will not get an answer to this profoundly important question until
they begin to actively look for the mechanisms of information processing.
This book is based closely on a paper in the Psychological Review entitled
“Time, Rate, and Conditioning” (Gallistel& Gibbon, 2000). Most of the figures
and much of the text first appeared there. That paper was being written at the time
one of us (CRG) was asked to give the MacEachran Lectures at the University of
Alberta. The lectures, of which this book is a product, were given October6-8,
1997. CRG is grateful to our colleagues at the University of Alberta for the
opportunity they provided to put this material in lecture and book form and for
the many fruitful discussions during his visit. He is also grateful for their patience
with the long delay in publication. We are both indebted to many colleagues for
critical readings of parts or all of what appears here. Wewish particularly to
thank Ralph Miller for detailed and meticulous critical readings of the
Psychological Review manuscript.
viii PREFACE

ACKNOWLEDGMENTS

We gratefully acknowledge support from the following grants during the period
when these works were being written: SBR-9720410, entitled “Learning and
Intelligent Systems: Learning in Complex Environments by Natural and Artificial
Systems,” from the National Science Foundation (Roche1 Gelman, Orville
Chapman, Charles R. Gallistel, Edward P. Stabler, Charles E. Taylor, Phillip J.
Kellman, John R. Merriam, James W. Stigler, and Joseph A. Wise, CoPIs) and
MH14649 from the National Institutes of Health to John Gibbon.
During the copyediting process, John Gibbon died. He was a scientist of the
as a theorist, andas an experimentalist. He pioneered the application
first rank, both
of the information-processing framework to the analysis of timing behavior. He
will be sorely missed by the field, to which he contributedso much. To those who
knew him as a friend and collaborator, his loss is beyond the power of words to
express.

“ c . R. Gallistel
 John M. MacEachran
Memorial Lecture Series

TheDepartmentofPsychologyattheUniversityofAlbertainauguratedthe
MacEachranMemorialLectureSeriesin1975inhonor of thelateJohn M.
MacEachran.ProfessorMacEachranwasborninOntarioin1877.In 1906 he
received a PhD in Philosophy from Queen’s University in 1905. In 1906 he left
for Germany to begin more formal study in psychology, first spending just less
thanayearinBerlinwithStumpf,andthenmoving to Leipzig, where he
completedasecondPhDin1908withWundtashissupervisor.Duringthis
period he also spent timein Paris studying under Durkheim and Henri Bergson.
With these impressive qualifications the University of Alberta was particularly
fortunate in attracting him to its faculty in 1909.
ProfessorMacEachran’simpacthasbeensignificantattheuniversity,
provincial, and national levels. At the University of Alberta he offered the first
courses in psychologyandsubsequently served as Head oftheDepartmentof
Philosophy and Psychology and Provost of the University until his retirementin
1945. It was largely owing to his activities and example that several areas of
academicstudywereestablishedonafirmandenduringbasis.Inaddition to
playing a major role in establishing the Faculties of Medicine, Education, and
Law in theProvince,ProfessorMacEachranwasalsoinstrumental in the
formative stages of the Mental Health Movement in Alberta. At a national level,
he was one of the founders of the Canadian Psychological Association and also
became itsfirstHonoraryPresident in 1939.JohnM.MacEachranwas indeed
one of the pioneers in the development of psychologyin Canada.
Perhaps the most significant aspect of theMacEachranMemorialLecture
Series has been the continuing agreement that the Department of Psychology at
the University ofAlbertahaswithLawrenceErlbaumAssociates,Publishers,
Inc., for the publication of each lecture series. The following is a list of the
Invited Speakers and the titles of their published lectures:

ix
X SERIES PREFACE

1975 Frank, A. Geldard (Princeton University)

Sensory Saltation: Metastability in the Perceptual World

1976 Benton J. Underwood (Northwestern University)

Temporal Codes for Memories: Issues and Problems

1977 David Elkind (Rochester University)

The Child’s Reality: Three Developmental Themes

1978 Harold Kelly (University of California, Los Angeles)

Personal Relationships: Their Structures and processes

1979 Robert Rescorla (Yale University)

Pavlovian Second-Order Conditioning: Studies in Associative Learning

1980 Mortimer Mishkin (NIMH-Bethesda)

Cognitive Circuits (unpublished)

1981 James Greeno (University of Pittsburgh)

Current Cognitive Theory in Problem Solving (Unpublished)

1982 William Uttal (University of Michigan)

Visual Form Detection in 3-Dimensional Space

1983 Jean Mandler (University of California, San Diego)

Stories, Scripts, and Scenes: Aspects of Schema Theory

1984 George Collier and Carolyn Rovee-Collier (Rutgers University

Learning and Motivation: Function and Mechanisms (unpublished)

1985 Alice Eagly (Purdue University)

Sex Differences in Social Behavior: A Social-Role Interpretation

1986 Karl Pribram (Stanford University)

Brain and Perception: Holonomy and Structure in Figural Processing

1987 Abram Amsel (University of Texas at Austin)

Behaviorism, Neobehaviorism, and Cognitivism in Learning Theory:
Historical and Contemporary Perspectives

1988 Robert S . Siegler and Eric Jenkins (Camegie Mellon University)

How Children Discover New Strategies
SERIES PREFACE xi
1989 Robert Efron (University of California, Martinez)
The Decline and Fall of Hemispheric Specialization

1999 Philip N. Johnson-Laird (Princeton University)

Human and Machine Thinking

1991 Timothy A. Salthouse (Georgia Institute of Technology)

Mechanisms of Age-Cognition Relations in Adulthood

I992 Scott Paris (University of Michigan)

Authentic Assessment of Children’s Literacy and Learning

1993 Bryan Kolb (University of Lethbridge)

Brain Plasticity and Behavior

1994 Max Coltheart (Maquarie University)

Our Mental Lexicon: Empirical Evidence of the Modularity of Mind
(unpublished)

1995 Norbert Schwarz (University of Michigan)

Cognition and Communication: Judgmental Biases, Research Methods,
and the Logic of Conversation

1996 Gilbert Gottlieb (University of North Carolina at Chapel Hill)

Synthesizing Nature-Nurture: Prenatal Roots of Instinctive Behavior

1997 Charles R. Gallistel (Rutgers University) and John Gibbon (New York
State Psychiatric Institute and Columbia University)
The Symbolic Foundations of Conditioned Behavior

Eugene C. Lechelt, Coordinator

M a c h c h r a n Memorial Lecture Series

Sponsored by theDepartment of Psychology,TheUniversity of Alberta,in

memory of John M. MacEachran, pioneer in Canadian psychology.
This Page Intentionally Left Blank
 Introduction

In this book, we present anew conceptual framework for the understanding

of the learning that occurs in the Pavlovian and operant conditioning para-
digms. Many of the experiments whose results we seek to explain are famil-
iar to anyone who has taken a course in basic learning, and even to most
students who have had only an introductory course in experimental psy-
chology. We show that many of the best known results from the vast condi-
tioning literature-particularly the quantitative results-can be more readily
explained if one starts from the assumption that what happens in the
course of conditioning is not the formation of associations but rather the
learning of the temporal intervals in the experimental protocol. What ani-
mals acquire are not associations, but symbolic knowledge of quantifiable
properties of their experience. In the final chapter, we argue that this con-
clusion has broad implications for cognitive science, for neurobiology, and
for all those disciplines concerned with the nature of mind.
Conditioning paradigms were created to test and elaborate associative
conceptions of the learning process. In these paradigms, the subject is pre-
sented with simple, unstructured, or very simply structured stimuli-tones,
lights, noises, clickers, buzzers-whose temporal relations to each other
and to one ormore reinforcing stimuli are manipulated. The best known ex-
ample comes from the work of Pavlov (1928), who repeatedly sounded a
tone or noise, followed by the presentation of food to hungry dogs. He ob-
served that in time the dogs salivated in response to the tone or noise. He
originated the study of what is now called Pavlovian conditioning, and he
was such an astute observer and recorder of the phenomena to be o b
2 INTRODUCTION

served when animals are being conditioned that students ofanimal learning
continue even now to read his lectures with profit.
Stimuli like food, water, puffs of air delivered to the sclera, or mildly
painful shocks to the feet-stimuli that reliably motivate observable behav-
ior-are called reinforcers.The terminology reflects the conceptual frame-
work that Pavlov and almost all students of conditioning after him have a p
plied to the understanding of this phenomenon. Pavlov thought that the
food strengthened(reinforced) a connection between elements in the nerv-
ous system.The connection served as a conducting pathway over which ex-
citation propagated from the tone-sensitive elements to the food-sensitive
elements. The development of this pathway-the conditioned reflex path-
way-explained how it was that the tone came in time to elicit a response
similar to the responseelicited by the food itself.This conception of the un-
derlying process-that it involves the strengthening of a connection-still
dominates thinking about basic learning.
Pavlov also called reinforcers unconditioned stimuli (USs). We will use
the terms reinforcer and US more or less interchangeably. Following Pav-
lov, we will call the originally neutral stimuli (the tones, lights, etc.) condi-
tioned stimuli (CSs for short) and the responses that develop to them con-
ditioned responses (CRs).
Clearly, Pavlov’saccount of learning that occurs during conditioning is a
“learning to” account; the dog learns to salivate, rather than learning that
the tone predicts food. In associative models of the conditioning process,
symbolic knowledge of the world is not acquired. The altered conductive
connections (the associations) may mediate an adaptive response-for ex-
ample, a blink that shields the eye from an impending puffof air-but they
do not encode what it is about the experienced world that makes an appro-
priately timed blink adaptive. The connection forged by repeated experi-
ence of a tone and an air puff ora tone and food does not encode the tem-
poral relation between CS and the US.
In contemporary discussions of associative conditioning, properties of
the stimuli used are commonly assumed to be encoded in stimulus traces
left behind in the nervous system by the transient activity that the CSs and
USs evoke (Balleine, Garner, Ganzalez, & Dickinson, 1995; Bouton, 1993;
Colwill & Rescorla, 1990; Dickinson, 1989; Dickinson & Balleine, 1994;Res-
coria, 1991, 1993;Rescorla & Colwill, 1989).However, associative theories do
not specify the principles governing stimulus encoding, so it is a moot ques-
tion whether stimulus properties (e.g., amount, intensity, color, flavor, size,
duration, tonal composition) may themselves be represented by associa-
tive strengths, and, if so, how. In associative theories, as currently elabo-
rated, the strengthof the associative bond does not specify any objectively
describable property of the CS, the US, or the relation between them. That
is why the associations produced by conditioning do not have symbolic
INTRODUCTION 3

content. Their strengths do not specify objective facts about the animal’s
conditioning experience.
The subjects in conditioning experiments do, however, learn the tempo-
ral intervals in the protocols. This conclusion, once controversial, is now
widely accepted, on the basis of the kinds of experimental evidence re-
viewed at length in the chapters that follow. This temporal learning has
been modeled quantitatively by so-called timing models (Church, Broad-
bent, & Gibbon, 1992; Gibbon, 1977,1992; Gibbon, Church, & Meck,1984;
Killeen & Fetterman, 1988).
The ability of timing models to explain the timing of conditioned re-
sponses is widely recognized. It is not widely appreciated, however, how
fundamentally the discovery of an interval timing capacity may alter our
conception of the conditioning process itself. Timingmodels give us models
of conditioning in which symbolic knowledge is the foundation of the o b
served behavior. They are models of how this knowledge is acquired and
used. In this new conceptual framework, almost every aspect of basic con-
ditioning appears in a different light. Our purpose in this book is to make
clear salient features of that conceptual framework.
One feature of this conceptual framework is that the learning mecha-
nisms that mediate conditioned behavior should not be thought of as basic
to higher learning of all kinds. What is primarily manipulated in the great
majority of experiments commonly discussed under the heading of classi-
cal or operant conditioning is the temporal relations among stimuli. The
models we discuss are specific to this kind of learning. Our models operate
in the domain of nonstationary multivariate time series analysis. They do
not purport to be general theories of learning. On the contrary, they are
predicated on the assumption that there can be no such thing as a general
theory of learning, because learning mechanisms, like other biological
mechanisms,have problem-specific structures (Gallistel, 1992b, 1999b).
Mechanisms with problem-specificstructure aremore or less inherentin an
information processing approach to thebrain, becausedifferent kinds of in-
formation must be processed in different ways.
Within the account we propose, there is no important distinction-at the
level of process-between instrumentaland classical conditioning. The
learning that occurs in both kinds of protocols depends on mechanisms
for learning temporal intervals and rates and using those intervals and
rates tomeasure contingency. On the other hand,in our framework,the ac-
quisition of a conditioned response, the extinction of that response and the
timing of the response are distinct problems, requiring distinct decisions
for their solution.
In our view, different learning mechanisms may make use of a common
set of elementary neurocomputational operations, such as the storageand
retrieval of the values of variables (distances, intervals, intensities, etc.),
4 INTRODUCTION

and the adding, subtracting, multiplying, dividing, and ordering of these

variables. However, at the level of the learning processes themselves, the
processes that compute and utilize symbolic representations of the condi-
tioning experience, different problems necessitate different computations.
Thus, learning is inherently modular in this framework, and thebasis of the
modularity is computational: Different representations must be computed
in different ways.
A modularity of processing rooted in differing computational require-
ments is an unusual assumption within the studyof learning, but it is the or-
dinary assumption within sensory psychophysics. There, different deci-
sions that the subject makes about the properties of a stimulus-whether it
is red or green, moving to theleft or moving to theright, and so on-depend
on different decision variables, which are assumed to be computed in dif-
ferent cortical modules. The modularity of stimulus processing is taken for
granted in contemporary sensory psychophysics. Our models make the
same modularity assumption for the stimulus processing that mediates
conditioned behavior.
Our focus is on the quantitative facts. This, too, is unusual. Most learning
models have been content to predict only the direction of the effects of var-
ious manipulations, not the magnitude of the effects.
In the final chapter, we discuss the radical challenge that this new infor-
mation processing, cognitive framework poses for the traditional associa-
tive framework. This challenge should be of broad interest to contempe
rary psychologists because it is one of the few areas in experimental
psychology where the associative framework and the information process-
ing framework meet head on, offering alternative ways of thinking about an
extensive body of experimentally established facts. One canthen ask,
which wayof thinking about these facts gives a clearer morerigorously for-
mulated and more broadly applicable account? And why? What is it about
one framework that makes it more powerful and more successful than the
other? These are questions of enduring import for the field of psychology.