Imagining the Cognitive Life of Things

Edwin Hutchins ([email protected])

Department of Cognitive Science
University of California San Diego

Abstract morpheme level, I try to help the students see the

morphological, lexical, syntactic, prosodic, and pragmatic
Human interaction is complex. An embodied perspective on organization of talk. Then I have students identify the
interaction shows that it is richly multimodal. The complexity cultural models that account for the discourse level
of the interaction system allows for a surprising variety of
organization of the talk. What underlying assumptions hold
emergent cognitive effects. This paper attempts to place the
cognitive life of things in the context of rich multimodal
the parts of a discourse together? By the time they have
interactions. done this, it is apparent that verbal behavior is richly
multimodal in the sense that it integrates many layers of
Part I: The cognitive life of things is manifest structure, each consisting of a distinct set of patterns, and all
coordinated with one another by a very complex set of
in the ways people jointly engage things with relations.
their bodies and their words
In the last chapter of Cognition in the Wild (Hutchins, The richness of embodied interaction
1995), I argue that cognitive science made a fundamental For the final project in the course, I have the students
category error when it mistook the properties of a person in collect, transcribe, and analyze video recording of real-
interaction with a social and material world for the cognitive world interaction (often in a work setting). Students are
properties of whatever is inside the person. One enduring often astonished to discover how much more complex video
problem with this claim is that it demands a description of records of interaction are than audio records of spoken
how cognitive properties arise from the interaction of person narratives. In addition to talk, video recordings include the
with social and material world. Cognition in the Wild bodies of the participants, and the world that the interactants
provides a profoundly incomplete answer to this question. co-inhabit. I show my students a diagram (figure 1) to
In the years since it’s publication, Cognition in the Wild has illustrate the increased richness of the video data compared
been criticized for saying so little about the people in the to audio.
navigation setting. It describes the tools of the trade, and the
historical development of the tools. It describes social
processes and the cognitive properties of those social Gesture
processes, but it says almost nothing about the embodied Body Talk
practices of the navigators as flesh-and-blood people. For
the most part, the cognitive processes described in
Cognition in the Wild, and in other treatments of distributed
cognition, are presented without reference to the role of the
body in thinking. That is, in spite of the fact that distributed Action Reference
cognition claims that the interaction of people with things is
a central phenomenon of cognition, the approach has
remained oddly disembodied.

The multimodality of verbal behavior World

For the past three months, I have been teaching my
undergraduate course on the methods of cognitive Figure 1: A multi-modal interaction system
ethnography. Late in the quarter we transcribe and analyze
spoken discourse. When dealing with language materials, Just as there are multiple complex patterns in verbal
the students learn that everyday language is complex. The behavior, there are also multiple complex patterns in the
transcription methods we use (borrowed from Gail Jefferson deployment of the body (hand gestures, of course, but also
(1984)) capture the words that are spoken, but also focus other limbs, body orientation, head position and motion,
attention on super-segmental features of timing, prosody, eye-gaze, facial expression, and more) and often very
stress patterns; the aspects of language that are most complex patterns (arrays of things) in the world that the
affected by the fact that speech is produced by a body in interactants share. Furthermore, with the additional
action. We seem to forget sometimes that speaking is an elements come new relationships among the elements.
embodied activity. After transcribing at the word and Some of these relationships have already spawned entire
fields of inquiry: co-speech gesture, environmentally- coordination. We are already familiar with the B/T/W facet
coupled gesture, and reference for example. This growing for an individual. We have now added three facets: B/B/W,
appreciation for the cognitive importance of the details of T/T/W, and a facet that would be called the “table” on a
body-world interactions owes a lot to the availability of gem, B/B/T/T. A typical conversation between two people
inexpensive digital video. Frame-by-frame examination of who are not handling artifacts might be thought to take
interactions of people with one another and with their place at the table.
environments for thinking makes it possible to see
previously hidden detail. The T/T/W facet is the proper location of discourse analysis,
although we have already begun our discussion of it above.
The diagram and this discussion are gross The coordination processes that give rise to turn taking,
oversimplifications, of course. The uses of the body are sequential contingency of meaning, projection, and
also richly multimodal (as we will see below), as are intersubjective reference and anaphora belong here. B/B/W
interactions with the world. Thus each mode depicted in the comprises relations of coordination (entrainment of bodily
multi-modal system diagram is itself a multi-modal system. motion as in synchronous applause), cooperation (passing
on the street, e.g.) and collaboration (lifting something
Situating embodied interaction in the social world heavy together). Also the establishment and maintenance of
The diagram in figure 1 is also an oversimplification shared attention, as well as the recipient design of gestures
because it leaves out the embodied nature of social are forms of coordination on this facet. Talk1/Body2 can be
interactions. For the most part, when people engage things seen in conventional greetings. Person 2 waves, or nods,
cognitively, they do so in social contexts. Consider the person 1 says “Hey!” . Also in the microstructure of on-
added complexity that comes when we consider interaction going interaction in which a speaker shapes what is said
systems in which multiple actors are engaged in joint based on the on-going non-verbal reactions of the
activity. interlocutor (Hindmarsh and Heath, 2000).
Addressing the embodied nature of social interaction
requires a change to the diagram. Rather than a single body The most interesting interactions take place in a system that
and a single stream of talk, we now have two bodies, and involves the entire gem of interaction. Figure 3 shows two
two streams of talk, and a single shared world as shown in pilots working together in a complex setting: the flight deck
figure 2. I call this diagram the “square-cut gem” because of a Boeing 747-400. They are jointly planning an approach
of its inverted tetrahedron shape. at their destination airport. A quick look here highlights
some of kinds of coordination that arise when embodied
interaction is situated in the social realm.
Body Body

Talk Talk

World
Figure 2: A square-cut gem of interaction Figure 3: Pilots jointly imagine the future

Now, instead of adding new elements, we are only adding The system of body/world coordination is rich. In addition
new relations. The new relations hold between body and to the pilot on the left locating and opening to the relevant
body, between talk and talk and between the body of one page in the route manual, the two pilots are engaged in
and the talk of another. The many cognitive properties of mutual recipient design in talk and in gesture. Each is
systems of socially distributed cognition also enter at this shaping his own verbal and non-verbal behavior to suit the
level. needs of the other. Relations of talk to the world
demonstrate constraints in both directions. The spatial
There are six kinds of edges in the square-cut gem of organization of a thing (airport description page) is being
interaction. The edges are loci of coordination and each is used to create sequential temporal structure in the activity
characterized by a distinct set of coordination types. In by adopting a spatial “flow” of attention across the page.
addition to edges, there are facets on the gem, which Reciprocally, talk constrains the engagement of the thing by
correspond to more complex systems of multimodal making deictic reference to the items on the page. Talk and
body are coordinated as co-speech gesture is used to the thumb to the segments. It is clear that the motor
highlight items referred to and to guide own and other’s memory for the touch pattern is learned before the visual
attention. Reciprocally, co-gesture speech provides clues on representation of the array of month names that can be
how to identify the meaningful segments of a complex fluid produced by running the touch pattern in coordination with
gestural movement. Both pilots use their body orientations the standard list of month names. In this case, the body
and postures (construed within the constraints of the work plays a key role in the construction of a complex material
stations) to demonstrate shared attention and interest. The anchor. The hand is opportunistically appropriated as a
relations among the two speaking roles are complexly “thing” possessing a set of regions and a learned motor
organized by social and professional roles and an routine controls the assignment of conceptual content to the
intersubjectively shared understanding of the nature of the regions. In interaction with things, embodied processes can
task. Using these means and others, the pilots jointly become “somatic” anchors for conceptual blends; they are
imagine the future of their airplane from their current internal proxies for abstract relations among the things with
location, 150 miles from their destination, all the way to the which one interacts. I will return to this point in an
arrival gate. The human interaction system is composed of extended example below.
a very complex network of coordination among myriad
dynamic behavioral patterns. I propose to use this vision of Let us return now to the rich systems of interaction that are
interaction to further explore the cognitive life of things. visible in video recordings. An emerging field of inquiry
can be seen in the work of a number of researchers who
The power of embodied interaction with culturally focus on the details of the creation and manipulation of
constructed settings. material representations. In domains as varied as scientific
research (Goodwin, 1994; Alač, 2003), architectural practice
The power of cognitive artifacts to transform human thought (Murphy, 2004), student learning (Singer and Goldin-
is widely recognized. But of what importance are the Meadow, 2005, Alač and Hutchins, 2005), airline piloting
details of the bodily interaction with a cognitive artifact in (Hutchins and Palen, 1997), telecommunications
producing its cognitive power? The study of cognitive troubleshooting (Hindmarsh and Heath, 2000), and surgery
artifacts has not attended closely to the nature of the on- (Goodwin, n.d.), researchers are showing how interactions
going activities in which people think in interactions with between the body and cultural artifacts constitute an
things. This important topic has dropped into the interstices important form of thinking. These interactions are not taken
between academic disciplines. Of course, ergonomics as “indications” of invisible mental processes, rather they
considers the fit between the properties of bodies and the are taken as the thinking processes themselves. In this brief
manipulation of tools. But, we do not yet have a tradition of paper I will try to add to this project. I will illustrate how
thought that appreciates the profoundly situated character of processes of imagination, inference, error detection, and
embodied thinking in complex culturally organized settings. even “Aha!” insight can arise in cultural practices that bring
the body into interaction with the cultural environment in
In my work on material anchors for conceptual blends particular ways.
(Hutchins, 2005), I tried to show how in some kinds of
activities, physical relations become proxies for conceptual Using the body to imagine the dynamics of things
relations. In the linguistic phenomenon known as fictive
The notion of somatic anchors for conceptual blends adds a
motion, a conceptual relationship is mapped onto a real or
new dimension to the now familiar phenomena of
imagined space to produce a trajectory to guide the
environmentally-coupled gestures (Goodwin, Hutchins and
allocation of attention. In the well-known memory
Palen, Becvar, Alac many more). Let me give just two brief
technique called the “method of loci,” and in the techniques examples here in which scientists engage static graphical
that Kirsh (1995) groups under the heading “the intelligent
representations with their moving bodies in order to imagine
use of space,” an imagined trajectory is mapped onto real
the complex dynamics of entities that cannot be observed
things in space, thus producing sequential relations among
directly. The first example comes from the work of my
the things. My analysis of material anchors for conceptual student, Amaya Becvar (Becvar, Hollan, Hutchins (in press
blends was, however, like most other work on cognitive
a, and in press b)). Figure 4 shows the principal investigator
artifacts, disembodied.
of a molecular biology laboratory. The problem at hand is
to imagine how the shape of the thrombin molecule changes
The body did come into play at one point in a way that I can
when another molecule, thrombomodulin, attaches to it. She
now see as suggestive. Some Japanese school kids learn a
has just aligned her left hand with a ribbon diagram of a
technique for mapping the names of months and days of the
thrombin molecule that is depicted on an overhead
week onto their left hand in a way that allows the hand to be
transparency lying on the platen of an overhead projector.
used as a date calculator1. The mapping of names of
She lifts her hand from the transparency as if, by virtue of
months onto the segments of the first three fingers of the left
touching her fingers to the loops in the ribbon diagram, her
hand is learned as a sequential pattern of touching the tip of
hand has become the molecule. She then says, “Our theory
1 is that the molecule moves either like this or like this.”
See Nakahara (1996) for details.
Coincident with each of the two deictic utterances of “this” as-yet-unbuilt building. He gives these gestures the label,
she produces a motion with her hand. In one the finger tips “action in the subjunctive mood.” One important insight
come together like the mouth of a purse. In the other, the here is that these representations are tentative in a way that
fingers rotate around an imaginary axis emerging from the the lines that appear on the building drawings are not. They
palm of the hand. In this way, she imagines two dynamic suggest that representations can have different degrees of
modes for the loops of the thrombin molecule. commitment to the entities that are represented. Some
representations make strong commitments, while others are
more tentative. Some are tangible and permanent (lines in
ink on a building drawing) while others are less permanent
(pencil markings), while still others are ephemeral
(gestures). I will argue below that understanding the
cognitive life of things requires that we understand the full
range of commitment of our embodied engagements with
things.

Relations among representational contents in

Figure 4: The thrombin hand multimodal representations
We need to consider one more recently described
The second example comes from the work of another of my phenomenon before taking on an extended example. Susan
students, Morana Alač (Alač, 2003). In figure 5, we see a Goldin-Meadow (Singer and Goldin-Meadow, 2005)
brain imaging researcher positioning his hands in front of a describes a “mismatch” between talk and environmentally
computer display showing a functional image of a brain. coupled gesture in interaction as an index of readiness to
His hands are cupped and they enclose a volume that is learn. Mismatch may be the wrong word here, but the
about that of a three-dimensional human brain. He says, insight that different media in a multimodal interaction may
“You can see it’s sort of …” During the pause following carry different information is key. It is possible to
the word “of” he moves his left hand up and his right hand distinguish cases in which gesture and talk carry
down. Subsequent talk and action show that he is referring overlapping or congruent representations from those in
to the shearing of the image – an artifact introduced by the which gesture and talk carry different, but complementary,
software that computes the image. In this way, he uses his representations. The latter cases are labeled “mismatch” by
body to imagine a process, “shearing,” that is held Goldin-Meadow, and they are taken to indicate that a child
responsible for the observed asymmetry of the image. This is on the verge of conceptual breakthrough. Goldin-
is an interesting bit of scientific reasoning because the Meadow is describing emergent properties of a multimodal
imagined process is both very useful and completely system of representations (although she does not use the
fictional. term “emergent”). In the example below, I will argue that
the embodied imagining that is made possible by
complementary representations in the multi-modal
body/world system can give rise to an “Aha!” insight. The
additional twist here is that the complementary
representations are not carried in talk and gesture, but are
both are contained in the multiple modalities of the relations
of body to the world of things. A new insight arises as an
emergent property of the juxtaposition of complementary
content in distinct modalities.

Figure 5: The sheared brain

In these two examples, the researchers use the motion of Part II: An example of thinking with things in
their own bodies in coordination with static representations ship navigation
to imagine dynamic properties of entities that cannot be Until recently, ship navigation was performed on paper
seen or sensed directly. These bodily motions are the charts using manual plotting tools. The data on which this
medium in which the researchers reason about the entities analysis is based were originally collected more than 20
that are depicted in the static representations. years ago on the bridge of a US Navy ship when these
practices were still common. In order to fix the position of
Imagining virtual worlds a ship, navigators measure the bearing from the ship to at
Similar work by Murphy (2004) shows how architects least three landmarks. When plotted on a chart, the bearing
imagine the dynamics of people moving in buildings that do of a landmark from the ship becomes a line of position
not yet exist. He describes environmentally-coupled (LOP), that is, it is a line on which the ship must be located.
gestures that represent imagined motion along paths in the Plotting a LOP involves setting the measured bearing on a
protractor scale on a plotting tool (called the hoey) and then navigators choose sets of landmarks to use for
placing the hoey on the chart so that the protractor arm future fixes2.
passes through the depiction of the landmark on the chart 4. Positioning the straight edge of the plotting
and the base of the protractor scale is aligned with the protractor (hoey) in the approximate location of
directional frame of the chart. Once the plotting tool is LOPs and manipulating the position of the straight
correctly placed, the navigator uses a pencil to draw a line edge to imagine variations on previously plotted
on the chart in the vicinity of the projected position of the LOPs. In this case, the navigator uses the plotting
ship. Two intersecting lines of position determine, or “fix”, tool’s straight edge to temporarily
the position of the ship. Navigators usually try to plot three imagine/create/locate approximate LOPs.
lines of position, because the intersection of three LOPs Manipulation of the hoey integrates motor, visual,
forms a triangle. A small fix triangle indicates that the proprioceptive and tactile representations.
position fixing information is good. A large triangle Examining the placed tool adds stable visual
indicates problems somewhere in the chain of representations to the system.
representations that lead to the fix triangle. In general, the 5. Plotting LOPs with the hoey. In these cases, the
navigator’s level of anxiety is proportional to the size of the hoey is exactly aligned with the landmark and the
fix triangle. directional frame of the chart, and a pencil line is
drawn on the chart. This integrates visual, motor,
In order to produce a useful fix triangle, the spatial proprioceptive, and tactile representations and also
relationships among the ship and the three landmarks must produces a durable representation of the LOP on
be such that the LOPs intersect at steep angles. This means the surface of the chart.
that navigators must anticipate where the ship will be when
the next set of position fixing observations is made and must Across these five cases, one sees increasing tangibility and
choose a set of three landmarks with appropriate relations accuracy of the LOP representations, increasing
for the fix. involvement of the body and the culturally organized
setting, and increasing cognitive and physical cost. It is
A gradient of tangibility in the representations of LOPs. worth noting that the fifth practice is the only one that is
formally part of the position plotting “task.” As such, it is
The video data contain records of activities that create many the only one that would be likely to be examined in a study
sorts of representations of LOPs. Each system is an of the ergonomics of position plotting. But the others are
integrated whole that brings representations in multiple essential to understanding how navigators actually
modalities into coordination. Each has distinct cognitive accomplish their work. I will argue below that it is
properties. The observed systems include the following: necessary to understand the entire range of practices
described here in order to understand how embodied
1. Situated seeing of the chart. Navigators look at the interactions with cultural artifacts can support abstraction.
location of the estimated next position of the ship
and name sets of landmarks that may produce the In the middle region of this gradient (cases 2-4) are actions
appropriate angles between LOPs. In this case, the that Murphy (2004) has called “action in the subjunctive
representations of the LOPs are produced in visual mood.” These are “as-if” actions or “may it be thus”
imagination superimposed on the chart surface. actions. These actions produce ephemeral representations
2. Enacting LOPs with finger motions in the air above of potential, but not yet realized, states of affairs or
the chart. In these cases, the visual imagination of processes. Navigators reason about the relationships among
the LOPs is augmented by gestures made the virtual LOPs that are created by these gestures. These
approximately 0.5 meters above the chart surface. are thus clear examples of thinking with the body in
This practice adds gesture to the system and interaction with a cultural artifact. Notice that the
integrates motor, visual, and proprioceptive boundaries of gesture and instrumental action become
representations. blurred in this continuum.
3. Tracing LOPs with finger motions on or very near
the surface of the chart. In these cases, the visual I will now focus briefly on the fourth class, positioning the
imagination of the LOPs is augmented by gestures hoey arm and manipulating the position of the arm to
made in close coordination with the visual structure imagine LOPs. This class of representational system was
of the chart. Making contact with the chart surface observed in an event that was a crisis for a navigation team.
while gesturing integrates tactile representations While entering a narrow navigation channel, a ship suffered
into the system. This practice is especially the failure of its main gyrocompass. Upon losing the
important in episodes of joint reasoning in which gyrocompass, the navigation crew could no longer simply
read the true bearing of a given landmark and plot that

2
A more detailed discussion of this system appears in Hutchins (in
press).
bearing. Rather, they were then required to compute the true picked up the hoey and moved the arm in the
bearing by adding the corrected magnetic ship’s head to the direction of the 120 degree scale position. It is not
relative bearing of the landmark. The correct equation is: possible to establish exactly where the hoey arm
true bearing equals compass heading plus deviation plus was on the scale when he stopped moving it, nor
magnetic variation, plus the relative bearing of the can one determine whether or which scale values
landmark. (TB = C + D + V + RB). This perturbation he had read at that point. It is clear that he was
disrupted the ability of the crew to plot accurate positions moving the hoey arm across the scale and attending
for the ship. The crew explored various computational to the scale values.
variations of TB = C +V +RB for 38 lines of position. Then 3. While moving the hoey arm toward alignment with
they discovered that a key term, deviation, was missing the scale at the120 degree mark, the plotter was
from their computations. After reconfiguring their work to also imagining the location with respect to the fix
include the deviation term, the team gradually regained the triangle of the LOP that was plotted using 120
functional ability to plot accurate positions. degrees. With his body he imagines a slight
clockwise rotation of the LOP that would make the
The discovery of a missing term triangle smaller. Several representations are
integrated into a single meaning complex at this
How can the discovery that this term was missing be moment. They include the hoey arm and protractor
explained? The discovery appeared as an “Aha!” insight. scale and the imagined clockwise rotation of the
Taken in the context of the computations that the crew was hoey arm on the chart. This combination of
doing, this discovery is, like most creative insights, representations creates the image of a small
mysterious. There is nothing in the pattern of computational clockwise motion of the arm with respect to the
efforts leading up to the discovery that indicates they are hoey base that would decrease the size of the
nearing this development. The processes that underlie the triangle (see figure 6). In this composite image, it
“Aha!” insight remain invisible to a computational can also be seen that a clockwise motion
perspective in part because that perspective represents corresponds to motion across the scale toward
everything in a single mono-modal (or even a-modal) larger numbers (see figure 7). This implies a
system3. A careful examination of the way the body slightly larger numeric value for the bearing to that
engages the tools in the setting, however, helps to solve the landmark.
mystery of how the discovery was made, and why it
happened when it did. The insight was achieved in and
emerged out of the navigator’s bodily engagement with the
setting.

Here is a very brief account of the course of events:

Lines of position had been plotted to each of the three
landmarks, but the fix triangle that was produced was
unacceptably large. Such a large triangle was clear evidence
of the presence of an error somewhere in the process that
created it. The LOPs were then checked, and at least one
possible source of error was tested with respect to each one.
Then the plotter then did the following:
1. He aligned the hoey approximately for one
landmark, them moved it slightly clockwise with
respect to the previously plotted LOP. He then
quickly shifted the hoey on the surface of the chart Figure 6: The superimposition of imagined clockwise
and aligned it approximately with the second rotation onto the hoey plotting tool.
landmark. He also adjusted this LOP slightly
clockwise. In these two moves, the plotter was
exploring the lines of position and using his body
and the tools (chart and hoey) to imagine LOPs that
would make the fix triangle smaller.
2. He spoke (self-regulatory speech) the remembered
bearing to the third landmark, 120 degrees, while
the hoey was still lying on the chart. He then

3
In Hutchins (1995), I provide a disembodied analysis of this
event that fails to explain how the discovery of the missing term
was made.
 may be a resource for adaptive processes when routine
activity is disrupted.

Part III: Understanding the cognitive life of

things is essential to understanding cognitive
life in general
The processes described above can be characterized in terms
of some general implications of the embodied enacted view
of cognition.
Figure 7. Seeing that clockwise rotation produces larger
numbers. • In certain culturally constructed settings, bodily
motion acquires meaning by virtue of its relation to
4. The plotter then seems to remember that the the spatial structure of things. Goodwin calls this
explored actions to improve the other two LOPs phenomenon “environmentally-coupled gesture.”
was also clockwise rotation (the actions that In some circumstances, the body itself becomes a
established those relations occurred three and eight cognitive artifact, upon which meaningful
second earlier, respectively). This implies that all environmentally-coupled gestures can be
three of the bearings should be slightly larger performed (Enfield, 2005; Hutchins, 2005). In
numbers. such settings, motion in space acquires conceptual
5. The final step in the process is noticing that the meaning and reasoning can be performed by
need to add a small amount to each bearing moving the body. Courses of action become trains
suggests that a term may be missing from the of thought. For example, when working on the
correction calculation. Unfortunately, the chart, movement away from the body is
processes that produce this last step are hidden conceptually northward, toward the body is south,
from our view. However, very conventional clockwise rotation is increasing measure of
processes of semantic association could be invoked degrees. When actions are performed by experts in
to explain this final step because the elements that these domains, the integrations of bodily sensation
are created by the complex of embodied with directional frame produces embodied
representations closely match the elements of the reasoning. Navigators sometimes speak of their
schema for deviation. It is no longer a mysterious reasoning skills in as “thinking like a compass.” I
process. believe this could be better described as “having
bodily sensation like a compass.”
In a traditional cognitive explanation of creative insight, one • Practitioners who live and work in complex
would postulate the entire discovery process in terms of culturally constituted settings such as ships’
interactions among unobservable internal representations. bridges, airline flight decks, and science
What makes such accounts mysterious is that we know so laboratories habitually integrate representations in
little about these internal representations. By construing the multiple sense modalities. It must be assumed that
engagement of the body with the things in the working these multimodal representations are involved in
environment as a form of thinking, we can directly observe the construction of memories for past events, the
much of the setup for the insightful discovery. We can now experience of the present, and the anticipation of
ask: what opportunities for noticing conceptual relations are the future.
facilitated by the means of representation that are created in • Complex integrated multimodal representations are
the engagement of the body with the culturally constituted likely to be more stable than single-mode
world of action? The bodily anticipation of clockwise representations. One way to accomplish this is to
rotation becomes a somatic anchor for the concept of embed the representations in durable material
increasing bearing number value. The serendipitous media; what I have elsewhere called, “material
combination of tool-based representations and the anchors for conceptual blends.” Another way to do
imagination of future bodily motions produces an image of this is to embody the representations in bodily
adding a small amount to each of three bearings. motions. These become “somatic anchors for
conceptual blends.” Stabilization of complex
From the perspective of a formal representation of the task, conceptual representations by either means
the means by which the tools are manipulated by the body facilitates their manipulation.
appear as mere implementational details. They are a source • Integrating representations in a single embodied
of variability in performance, but are formally irrelevant to process facilitates sharing the reasoning process
the accomplishment of the task. It is worth pointing out, with others and the establishment and maintenance
however, that this variability in task irrelevant dimensions
 of intersubjective understandings (the so-called interaction during the development of interaction
common ground). practices?” The possible mechanisms of change are
• Culturally-embedded embodied thinking and acting multiplied by the complexity of the relations among the
benefit from adaptive possibilities via both the components of the system. Possible explanations of the
variability in interactions with material development of more powerful systems of reference, for
representations and the variability inherent in example, could be constructed by imagining new systems of
social interaction. We know least about this aspect embodied coordination such as those documented in the first
of these systems. sections of the paper. Making this move would free the
field from the constraint of locating mechanisms of change
Imagining the pre-history of the cognitive life of things inside individuals.
The square-cut gem diagram describes a cognitive ecology.
By cognitive ecology I mean that all of the elements and Speaking of what is inside individuals, how should one
relations potentially interact with one another and that each imagine the role of the brain? Before projecting the brain
is part of the environment for all of the others. At this point back in time, let’s consider its role in the interaction system.
we must also keep in mind that this diagram is dramatically In a seminal paper, Rumelhart et al. (1986) argued that
oversimplified. Most of the elements are hidden in the people are naturally good at just a few things: matching
summary terms. This rich cognitive ecology gives rise to patterns, manipulating objects in the world, and imagining
some powerful cognitive processes. The embodied simple dynamics. We can take these to be properties of the
interaction with things creates mechanisms for reasoning, operation of the brain. It is easy to see that these are likely
imagination, “Aha!” insight, and abstraction. Cultural properties of ancient as well as modern brains. Rumelhart
things provide the mediational means to domesticate the et al. go on to say that these abilities account for many of
embodied imagination. the accomplishments of human cognition, including
symbolic behavior. By interacting with particular kinds of
This ecology has been in place – in different forms - for a cultural things, we can produce complex cognitive
very long time. The sorts of accounts we create about the accomplishments while employing simple cognitive
development of cognition should be informed by and processes. Once we have learned to interact with these
sensitive to this rich system of interactions. If we were to things, we may learn to imagine both the things and our
imagine projecting this interaction system back in time, the interaction with them. Then we can organize our thinking
elements of the system and the relations among them would using internal resources in ways that previously required
surely be changed. Some elements and relations would be interaction with external cultural things. In this perspective,
changed more than others. If projected back to an ancestor interaction with the material and social world come first,
at the level of development of contemporary chimpanzees, and imagination of those interactions come later. This is a
for example, the entire square-cut gem of interaction would thoroughly Vygotskian point of view. But can we give it an
be present. One big change would be that talk is replaced embodied interpretation? The bodily interactions with
by other sorts of vocalizations. It is interesting that the things give rise to somatic anchors for conceptual blends.
super-segmental aspects of vocalization are already present Imagining bodily interaction with things can become a form
and playing similar roles then as they play now in the of conceptual thinking. As imagined actions become more
expression of emotion. Complex body/world relations are stylized (remember the gradient of tangibility of
present in tool use and exploration (imagination?) of ways representations of LOPs) the somatic anchors for conceptual
to use tools. I defer to other with more expertise, but I blends become increasingly detached from the conditions of
believe that some forms of environmentally-coupled gesture bodily-engagement-with-the-world that gave rise to them.
are also present in activities such as threatening with a stick. This could be described as the imagination of embodied
On the B/B/W facet Chris Johnson has documented learning abstraction.
by socially-prompted exploration of objects such as a stick
or a rope (Johnson, n.d.). Body/Body relations are present Put the Rumelhart et al. perspective together with the
of course, for this is where animals enact various forms of square-cut gem model of interaction and project it into the
coordination (greeting, fighting), cooperation (grooming, past. As we bring it forward in time, we might now ask,
carrying, mating), and collaboration (group hunting). “What changes in brains are required to account for the
Body/Vocalization relations exhibit many forms of development of modern interaction practices?” Since there
coordination, although some maintain that chimps do not are so many possible mechanisms of change in this complex
point (Tomasello, in press). As far as I know, system, I suspect that the required changes in the brain are
Vocal/Vocal/World relations would be weak. One might less dramatic than has been assumed commonly.
observe hooting at the approach of a predator, but reference
would presumably be limited. Of course, not all thinking and imagination are carried out
in the interaction of body, talk, and cultural world.
Now, to move this system forward through time, we should However, it is a reasonable hypothesis that all thinking and
ask, “What happened in the square-cut gem system of imagination have their origins in this interaction system. It
is for this reason that imagining the cognitive life of things tools, and reasoning: Essays on situated cognition.
will be essential to understanding cognitive life in general. Heidelberg, Germany: Springer-Verlag. Pp. 23-40.
Jefferson, G. 1984. Transcription Notation, in J. Atkinson
Acknowledgments and J. Heritage (eds), Structures of Social Interaction,
The research described here was funded by grants from the New York: Cambridge University Press, 1984.
Santa Fe Institute and from the Boeing Commercial
Airplane Group. Access to field sites was made possible by Johnson, C. n.d., Presentation to the UCSD DCOG-HCI lab,
the US Navy and the Boeing Commercial Airplane Group. March, 2006.
Kirsh, David, 1995. The intelligent use of space. Artificial
Intelligence 73 (1–2), 31–68.
Murphy, Keith M. 2004. “Imagination as Joint Activity: The
References Case of Architectural Interaction”, Mind, Culture, &
Activity, 11 (4), 270-281.
Alač, M. 2003 "Squashing, Rotating, Seeing, and Going: On
Visual Knowledge in fMRI Research", Proceedings of Nakahara, Hiroyuki, 1996. Japanese Hand Calculator.
the 25th Annual Meeting of the Cognitive Science Unpublished manuscript, University of Tokyo.
Society, Richard Alterman and David Kirsh (eds.),
Rumelhart, D., Smolensky, P., McClelland, J., and Hinton,
2003, 73-78.
G. 1986. Schemata and sequential thought processes in
Alač, M & E. Hutchins 2004 "I See What You are Saying: PDP models. IN J. McClelland, D. Rumelhart, and the
Action as Cognition in fMRI Brain Mapping Practice", PDP research group (eds.), Parallel Distributed
Journal of Cognition and Culture, 4:3, 2004, 629-661. Processing: Explorations in the microstructure of
cognition, volume 2. MIT Press.
Becvar, L.A., Hollan, J., Hutchins, E. (in press a)
"Representational Gestures as Cognitive Artifacts for Singer, M. A., & Goldin-Meadow, S.A. Children learn
Developing Theory in a Scientific Laboratory." when their teachers’ gestures and speech differ.
Artifacts in Workplace Practice. Kluwer Academic Psychological Science, 2005, 16, 85-89.
Publishers.
Tomasello, M. (in press). Why don’t apes point? In N.
Becvar, L.A., Hollan, J., Hutchins, E. (in press b) "Hands as Enfield & S. Levinson (Eds.), Roots of Human
Molecules: Representational Gestures as Cognitive Sociality. Wenner-Gren. Elsevier
Artifacts for Developing Theory in a Scientific
Laboratory." Semiotica.
Enfield, N. J.. (2005). The body as cognitive artifact in
kinship representations. Current anthropology, 46(1),
51-81.
Goodwin, C. 1994 Professional Vision. American
Anthropologist 96(3): 606-33.
Goodwin, C. (n.d.) Presentation given at UCSD.
Hindmarsh, J.and Heath, C. Embodied reference: A study of
deixis in workplace interaction. Journal of Pragmatics
32 (2000) 1855-1878.
Hutchins, E. 1995 Cognition in the Wild. Cambridge, MA:
MIT Press
Hutchins, E. 2005 Material anchors for conceptual blends.
Journal of Pragmatics 37 (2005) 1555–1577
Hutchins, E. (in press) The distributed cognition
perspective on human interaction, In N. Enfield & S.
Levinson (Eds.), Roots of Human Sociality. Wenner-
Gren. Elsevier.
Hutchins, E. & Palen, L. (1997) Constructing Meaning
from Space, Gesture, and Speech. In L. B. Resnick, R.
Saljo, C. Pontecorvo, and B. Burge (Eds) Discourse,