See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/256763181

Embodied Comprehension of Stories: Interactions between Language Regions

and Modality-specific Neural Systems

Article in Journal of Cognitive Neuroscience · September 2013

DOI: 10.1162/jocn_a_00487 · Source: PubMed

CITATIONS READS

55 296

6 authors, including:

Yisheng Xu Siyuan Liu

27 PUBLICATIONS 1,952 CITATIONS

U.S. Department of Health and Human Services
65 PUBLICATIONS 1,892 CITATIONS
SEE PROFILE
SEE PROFILE

Allen Braun
Walter Reed Army Institute of Research
164 PUBLICATIONS 12,218 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Evolution of brain language areas View project

Functional MRI - Passive rhythm perception in musicians View project

All content following this page was uploaded by Yisheng Xu on 07 August 2014.

The user has requested enhancement of the downloaded file.

 Embodied Comprehension of Stories: Interactions
between Language Regions and Modality-specific
Neural Systems

Ho Ming Chow1, Raymond A. Mar2, Yisheng Xu1, Siyuan Liu1,

Suraji Wagage1, and Allen R. Braun1

Abstract
■ The embodied view of language processing proposes that visually vivid, action-based, and emotionally charged content.
comprehension involves multimodal simulations, a process that Activity of neural systems associated with visual-spatial, motor,
retrieves a comprehenderʼs perceptual, motor, and affective and affective processing were selectively modulated by the rele-
knowledge through reactivation of the neural systems responsi- vant story content. Importantly, when functional connectivity
ble for perception, action, and emotion. Although evidence in patterns associated with the left inferior frontal gyrus (LIFG),
support of this idea is growing, the contemporary neuroanatom- the left posterior middle temporal gyrus (pMTG), and the bilateral
ical model of language suggests that comprehension largely anterior temporal lobes (aTL) were compared, both LIFG and
emerges as a result of interactions between frontotemporal lan- pMTG, but not the aTL, showed enhanced connectivity with
guage areas in the left hemisphere. If modality-specific neural the three modality-specific systems relevant to the story content.
systems are involved in comprehension, they are not likely to Taken together, our results suggest that language regions are
operate in isolation but should interact with the brain regions engaged in perceptual, motor, and affective simulations of the
critical to language processing. However, little is known about described situation, which manifest through their interactions
the ways in which language and modality-specific neural systems with modality-specific systems. On the basis of our results and
interact. To investigate this issue, we conducted a functional past research, we propose that the LIFG and pMTG play unique
MRI study in which participants listened to stories that contained roles in multimodal simulations during story comprehension. ■

INTRODUCTION
Boulenger et al., 2008; Neininger & Pulvermuller, 2003).
According to the embodied view of language, comprehen- Nevertheless, the precise mechanism by which these sys-
sion involves multimodal simulations in which the com- tems facilitate semantic processing is unclear (Meteyard,
prehenderʼs contextually relevant perceptual, motor, and Cuadrado, Bahrami, & Vigliocco, 2012; Mahon & Caramazza,
emotional knowledge is retrieved through partial reactiva- 2008).
tion of modality-specific brain systems (Barsalou, 2003). At the same time, decades of research on the neuro-
These systems are associated with distinct neural substrates science of language have shown that there exists a network
and are thought to be specialized for perceptual, motor, of brain regions consistently activated and functionally
and affective processing. Brain imaging studies have shown connected during language comprehension, regardless of
that regions within these neural systems are selectively content (Binder & Desai, 2011; Lau, Phillips, & Poeppel,
activated during sentence processing (e.g., Desai, Binder, 2008; Tyler & Marslen-Wilson, 2008; Vigneau et al., 2006;
Conant, & Seidenberg, 2010) and story comprehen- Price, 2000). Furthermore, there is substantial evidence
sion (Speer, Reynolds, Swallow, & Zacks, 2009; Ferstl & demonstrating that activation, manipulation, and integra-
von Cramon, 2007; Ferstl, Rinck, & von Cramon, 2005) tion of semantic information is associated with a set of
in response to different types of content, which suggests regions within this network. These include classical core
their involvement in processing the meaning of language. language regions, the left inferior frontal gyrus (LIFG; Lau
Neuropsychological research has further demonstrated et al., 2008; Badre, Poldrack, Pare-Blagoev, Insler, &
that modality-specific systems are not simply associated Wagner, 2005; Hagoort, 2005) and the left posterior mid-
with, but may be directly involved in, semantic processing dle temporal gyrus (pMTG; Turken & Dronkers, 2011;
( Willems, Labruna, DʼEsposito, Ivry, & Casasanto, 2011; Hickok & Poeppel, 2007). In addition to these two core
language regions, recent studies have shown that the
bilateral anterior temporal lobes (aTL) play an important
1
National Institutes of Health, Bethesda, MD, 2York University, role in language processing, particularly semantic in-
Toronto, Canada tegration (Lambon Ralph, Sage, Jones, & Mayberry, 2010;
No rights reserved. This work was authored as part of the Contributorʼs official Journal of Cognitive Neuroscience 26:2, pp. 279–295
duties as an Employee of the United States Government and is therefore a work
of the United States Government. In accordance with 17 U.S.C. 105, no copyright doi:10.1162/jocn_a_00487
protection is available for such works under U.S. law.
Patterson, Nestor, & Rogers, 2007; Jung-Beeman, 2005). that their interactions with the modality-specific systems
Together, the LIFG, pMTG, and aTL are not only engaged would be differentially enhanced in a content-dependent
in processing the meaning of words and sentences but are fashion. In other words, interactions between the lan-
also consistently activated during discourse comprehen- guage regions and visual-spatial, motor, and affective sys-
sion (Mar, 2011; Ferstl, Neumann, Bogler, & von Cramon, tems should be selectively modulated by the visually vivid,
2008). If modality-specific neural systems play an integral action-based, or emotionally charged content. More im-
role in processing the meaning of language, they are portantly, we expected that the language regions might
not likely to operate in isolation and should operate in interact with the modality-specific systems in different
concert with these language regions. ways, perhaps reflecting their distinct functional roles in
Recent theories have proposed that an additional set simulating sensorimotor or emotional information. Observ-
of heteromodal areas may play an important role in dis- ing and identifying these differences may make it possible
course comprehension. These include the medial pre- to incorporate the functions of modality-specific neural sys-
frontal cortex (MPFC), posterior cingulate cortex (PCC), tems into the traditional account of language processing.
and bilateral parietal lobules (IPL; Binder & Desai, 2011;
Ferstl et al., 2008; Mason & Just, 2006). However, unlike
the language regions we defined in the previous paragraph, METHODS
it is unclear whether these areas are directly involved in
building up modality-specific representations or whether Participants
they serve other functions such as monitoring coherence. Participants were 24 native speakers of English (11 women
Moreover, this network has also been associated with a and 13 men; mean age = 25.5 years, SD = 2.7 years,
wide variety of other cognitive functions such as theory range = 21–33 years) with no history of neurological
of mind and prospection (e.g., Buckner, Andrews-Hanna, or psychiatric disease. All were right-handed according
& Schacter, 2008). Considering the functional uncertainty to the Edinburgh Handedness Inventory (Oldfield, 1971).
that surrounds this network of regions and the fact that we Informed written consent was obtained from each par-
know little about how language and modality-specific neu- ticipant in accordance with the protocol approved by
ral systems relate during narrative comprehension, we the NIH CNS Institutional Review Board. Participants
focused our investigation on the LIFG, pMTG, and aTL were compensated for participating in the study. The
in this study. data from two other participants were excluded because
To examine the interactions between the language re- of poor performance on the comprehension task ad-
gions and modality-specific systems, we used fMRI to eval- ministrated after each fMRI run, indicating that they
uate participantsʼ responses while they listened to stories. might not have paid attention to the stories (response
These stories included paragraphs containing visually accuracies were below 75%; three SDs lower than the
vivid, action-based, or emotionally charged content. The mean accuracy of the remaining participants; 88.5%, SD =
three content types were manipulated in the first, second, 4.3%). Their data were excluded from further analyses as
and third paragraphs of a story, respectively. We chose to a result.
map the content types onto the paragraph structure in
this way because it fits with the canonical narrative struc-
Story Stimuli
ture, which begins with a setting, describes actions, and
ends with an emotional resolution. To rule out the pos- Our stimuli consisted of 18 stories written by the authors.
sibility that any effect of story content could be driven by Each story consisted of three paragraphs, conforming to
paragraph order, we included control paragraphs for each a canonical narrative structure: (1) the introduction and
of the three paragraph positions. These control paragraphs description of the setting and the main protagonist, (2)
contained minimal description of perceptual, action, emo- the unfolding of narrative events, and (3) the outcomes
tional content. Comparing the vivid perception, action, or and reactions of the protagonist. Because the three con-
emotional paragraphs to the sequence-matched control tent types of interest fit naturally with this structure, story
paragraphs makes it possible to remove the potential content was manipulated as follows. Twelve stories
effect of sequence (see Methods for details). Given these contained vivid descriptions of a scene in the first para-
controls, we expected that visually vivid, action-based, or graph (Perception condition), 12 stories contained rich
emotionally charged content would selectively engage neu- descriptions of a protagonistʼs actions in the second par-
ral systems devoted to visual-spatial, motor, and affective agraph (Action condition), and 12 stories vividly de-
processing, respectively. Crucially, with these activations scribed a happy or sad conclusion (Emotion condition)
identified, we employed connectivity methods to examine in the third paragraph. In 12 stories, one or two control
the relationships between language regions and these paragraphs were inserted into each of the three para-
modality-specific regions. graph positions to serve as Control conditions. These
More specifically, we hypothesized that the language control paragraphs described settings, portrayed typical
ROIs (i.e., LIFG, pMTG, and aTL) would be activated sequences of everyday events, or presented a coherent
independent of the type of content being processed, but resolution of the actions, but contained muted descrip-

280 Journal of Cognitive Neuroscience Volume 26, Number 2

tions of vivid visual, motor, or emotional content. The Mary Had a Little Lamb, Jack Be Nimble, and Humpty
control paragraphs were therefore coherent with the Dumpty) to which all participants had been exposed ear-
other parts of a story, conformed to a canonical narrative lier in life were presented to the participants. Like the
structure, and ensured that an accumulating discourse- stories, these contain phonological, lexical, and syntactic
level context took place, but without vivid description structures, making it possible to control for low-level lan-
of the three content types of interest. The control para- guage processing. It is important to point out, however,
graphs also served as fillers that limited participantsʼ ex- that there are some shortcomings of this approach to a
pectation of content type. The experimental conditions baseline condition. For one, nursery rhymes might be
for each story and the associated topics for each story considered a type of short narrative, so there may have
are listed in Table 1. Examples of story are shown in Ta- been excessive control for the creation of a discourse
ble 2, and the full set of stimuli is available on-line at model (i.e., both the experimental and control condi-
www.nidcd.nih.gov/research/scientists/pages/ brauna. tions have narrative elements). Moreover, the nursery
aspx. Additionally, three traditional nursery rhymes (i.e., rhymes differed from our experimental narratives in

Table 1. The Experimental Conditions and Topic of the Stories

Condition Topic Condition Topic

Story 1 Perception Theatre hall Story 10 Control N/A

Action Playing the piano Action Fixing TV and cables
Emotion Winning a prize on stage Emotion Getting compliments from your boss
Story 2 Perception New York City Story 11 Control N/A
Action Renovating apartment Action Dancing
Emotion Confrontation with neighbors Emotion Getting injured in a car accident
Story 3 Perception Paris Story 12 Control N/A
Action Playing with a toddler Action Assembling furniture
Control N/A Control N/A
Story 4 Perception Basketball stadium Story 13 Perception Disco
Action Playing basketball Control N/A
Emotion Winning a game Emotion A close relative dying in a hospital
Story 5 Perception Mountain ranges Story 14 Perception 3-D movie
Action Jogging Control N/A
Emotion Being robbed Emotion Meeting personal idol
Story 6 Perception Decorative church Story 15 Perception Biochemical lab
Action Conducting music Control N/A
Control N/A Control N/A
Story 7 Control N/A Story 16 Perception London
Action Baking Control N/A
Emotion Getting compliments from others Emotion Being cheated on by your romantic
partner
Story 8 Control N/A Story 17 Perception Disneyland
Action Watching a football game Control N/A
Emotion Being dumped by your romantic partner Emotion Marriage proposal
Story 9 Control N/A Story 18 Perception Los Angeles
Action Painting walls Control N/A
Control N/A Control N/A

Chow et al. 281

282
Journal of Cognitive Neuroscience

Table 2. Example Story Stimuli

Condition Story 1 Condition Story 9

Perception From her spot backstage, Hannah caught a glance of the spacious theater Control Two weeks of vacation time were almost gone before Alice got around to
hall where the music competition was being held. The hallʼs ornamented her home renovations. The walls of the apartment where she had been
balconies hung over the heads of those sitting on the lower level. The living for several years were in disrepair. Still, it was so easy to busy
buzz of their talking was loud, and yet they seemed small, almost miniature, herself with other things—vacation time was always too short to do
in their seats. She heard orchestra players tuning their instruments everything that needed to be done. But with only a few days left of her
and felt butterflies in her stomach. It was the first time she would be vacation, she was finally going to get to work fixing up her place.
competing while accompanied by an entire orchestra.
Action Hannah slowly walked out onto the stage. The theater-goers abruptly fell Action Walking through the local hardware store, Alice grabbed supplies and
silent, before clapping their hands enthusiastically. She sat at the piano dropped them in her basket. Then, at home, she pushed the furniture
stool, placing her hands on the piano keys, ready to begin. The conductor into the center of each room and threw large plastic sheets over each
waved both his hands, vigorously, and the orchestra started playing. piece. She then laid down tape along the edges of the walls to prevent
Her fingers moved swiftly across the keys and her feet lightly pressed smudges on the floorboards. Finally, she started painting—first slowly
the piano pedals. Her body swayed gently back and forth, as if the very dabbing the corners with special brushes, then more rapidly rolling
sound of the music was moving her. paint on large parts of the wall with her rollers.
Emotion Only when she was in the backstage room did Hannah let herself feel the Control In the evening, Alice was about half-way done with painting. She left all the
joy. With the thunderous applause still in her ears, she felt a wave of windows open to air the place out. She took a shower, got dressed, and
pleasure overcome her. Before long, she was being called back to the went out to find some food. She ate at one of the local restaurants, only
stage by the judges and she walked out unsteadily, her face flushed with 5 min away from her home. When she got back home, the apartment
excitement. With tears of joy in her eyes, she accepted a large bouquet still smelled of paint and was chilly too. Alice called up her best friend
of flowers and a check that was awarded to her. She felt the thrill of victory. Jane and asked to crash at her place.

Stories contained perceptual content in the first paragraph, action content in the second, and emotional content in the third. As a form of control, some stories had control paragraphs with minimal
perception, action, or emotion content (e.g., Story 9, Paragraphs 1 and 3). The full set of stimuli is available on-line at www.nidcd.nih.gov/research/scientists/pages/ brauna.aspx.
Volume 26, Number 2
 Average Word Imageability

Readability index indicates comprehension difficulty of a text (developed by Kincaid et al., 1975). Higher values indicate that a text is easier to read. An index of 60–70 is approximately eighth grade level.
terms of their familiarity, length, and the fact that they are

Ratingc per Paragraph

Occurrence in a corpus of 1.014 million words provided by Kucera and Francis (1967). Only open-class words available from the MRC Psycholinguistic Database (Coltheart, 1981) were considered.
presented in verse. However, our reasoning was that the

455.7 (33.9)e
familiar and overlearned nature of these nursery rhymes,

Ratings of likelihood of a word evoking mental images (ranged from 100 to 700). Only open-class words available from the MRC Psycholinguistic Database (Coltheart, 1981) were considered.
506.8 (18.2)
473.4 (33.0)
490.5 (22.1)
which were repeated multiple times in the experiment to
match the duration of story stimuli, would make it unlikely
that rich discourse models be created for these rhymes.
The use of the nursery rhyme baseline, in this case, should
allow us to identify language regions that participate in the
creation of such models. In light of the aforementioned
concerns, however, interpreting this contrast between

Average Word Frequencyb

the experimental stories and nursery rhymes should be
undertaken with some caution.

per Paragraph

e
Stories were constructed so that various linguistic features

124.9 (30.7)

209.8 (67.9)
109.5 (20.0)

133.0 (30.4)
would be closely matched across all conditions. These fea-
tures included the number of syllables per paragraph,
number of words per paragraph, number of words per
sentence, Flesch-Kincaid readability score per paragraph
(Kincaid, Fishburne, Rogers, & Chissom, 1975), average
word frequency (occurrence per million words) of content

Table 3. Mean and Standard Deviation (in Parentheses) of the Linguistic Measures of the Paragraphs in Each Condition
words per paragraph, and average imageability rating (like-

Readability Indexa
lihood of a word evoking mental images) of content words

69.3 (7.5)d
per paragraph. For evaluating word frequency and image-

71.4 (7.1)
76.3 (5.0)
70.9 (8.6)
ability, only open-class words available from the MRC Psy-
cholinguistic Database (Coltheart, 1981) were considered
to exclude the extreme values associated with closed-class
words that often bias such measures. The summary statistics
of the paragraphs in each condition are listed in Table 3.
Number of Words per Sentence
For each linguistic feature, the difference between the
Perception, Action, and Emotion conditions is less than

Indicates the Control condition is significantly different ( p < .05) from all the other conditions.
10%, and there is no significant difference between the

Indicates the Control condition is significantly different ( p < .05) from the Emotion condition.
three conditions (t test, p < .05 Bonferroni corrected),
16.7 (2.0)
16.9 (2.4)
15.6 (2.1)
16.2 (3.0)
except in readability between the Perception and Emotion
condition (the Emotional material being somewhat easier
to read). This measure was included in the general linear
model (GLM) analysis as a covariate to remove its poten-
tial effects from our primary interest in story content (see
Methods for details).
The stories and nursery rhymes were spoken by a male
native English speaker (aged 35) in a neutral tone at
Number of Words

speech rate of around 155 words per minute. All stimuli

77.1 (2.7)
78.3 (2.0)
77.9 (3.0)
76.7 (2.1)

were recorded digitally in a sound-isolated booth at a

sampling rate of 44.1 kHz. For each story paragraph and
nursery rhyme, the duration was normalized to exactly
30 sec and the loudness was equalized using audio editing
software (Audition, Adobe Systems, Inc.). Because each
story contained three paragraphs, the total duration of a
Number of Syllables

story is 90 sec.
101.0 (8.1)
109.4 (6.6)
105.6 (5.4)
108.3 (6.9)

Stimulus Norming
To confirm that the content of stories had been mani-
pulated effectively as intended, story paragraphs were
normed in a prestudy with 19 participants (3 men and
16 women, aged 22–34 years). None of these participants
Perception
Condition

Emotion

were scanned in the subsequent fMRI study. Individuals

Control
Action

were asked to rate the content of each story segment

with respect to the three dimensions we manipulated:
d
b

e
a

Chow et al. 283

(1) vividness of the described scene, (2) vividness of the Data Acquisition
described bodily action, and (3) the intensity of the pro-
Whole-brain gradient-echo EPI data were acquired on a
tagonistsʼ emotion. Ratings were provided on a scale of
3-T GE Signa scanner with an eight-channel head coil
1 (not vivid/intense) to 5 (highly vivid/intense). The
(repetition time = 1500 msec, echo time = 30 msec, flip
responses were analyzed using a repeated-measure
angle = 90°, 64 × 64 matrix, field of view = 224 mm,
ANOVA model with Content Type and Rating forming
30 sagittal slices acquired in interleaved order, slice thick-
two factors. Both Content Type, F(2, 36) = 22.0, p <
ness = 5 mm, acceleration factor = 2). High-resolution
.05, and Rating, F(2, 36) = 32.0, p < .05, were statistically
anatomical images were acquired using MP-RAGE (voxel
significant. Importantly, their interaction was also signifi-
size = 0.81 × 0.81 × 1.0 mm).
cant, F(4, 144) = 74.2, p < .05. To gain further information
about these effects, t tests were performed to compare
ratings between pairs of content types, with p values cor-
rected for multiple comparisons using the Bonferroni pro- Data Analysis
cedure. The results are listed in Table 4. Importantly, the fMRI Data Preprocessing
comparisons of ratings between Perception, Action, and
Using AFNI (Cox, 1996), functional images of each par-
Emotion story segments showed that only the rating of
ticipant were aligned to the first images of each run on
the designated dimension for each condition was signifi-
a slice-by-slice basis, timing differences in acquiring each
cantly more vivid than that in other two conditions ( p <
slice were corrected, and each volume was aligned to the
.05 in all cases). The participants of the fMRI experiment
first volume of the first run. To remove the physiological
also rated the stories using the same system after scanning.
and motion-related artifacts, spatial independent com-
These results were congruent with the norming study,
ponent analysis (McKeown et al., 1998) was applied to
showing that only the rating of the designated dimension
the motion and slice-time corrected functional data for
for each condition was rated significantly more vivid
each participant using the GIFT toolbox (mialab.mrn.org/
than that in both of the other two conditions ( p < .05
software/gift/index.html). In spatial independent com-
in all cases).
ponent analysis, each functional image was treated as a
mixture of multiple spatially independent signal and noise
components. The number of components in each data
Experimental Design
set was estimated by minimum description length criterion
All 18 stories were presented in random order without (Li, Adali, & Calhoun, 2007). Classification of artifactual
repetition in six runs of an fMRI session. In each run, and neuronal ICA components was based on their degree
three stories and one nursery rhyme were presented of spatial clustering, location of major positively weighted
binaurally via a SilentScan 3100 pneumatic headphone clusters, and neighborhood connectedness between posi-
system (Avotec, Stuart, FL). To allow the hemodynamic tively and negatively weighted clusters. Using these criteria,
response to return to baseline, paragraphs within a story the noise components were identified by human experts
were separated by a 12-sec interval; between stories or and their variances were then subtracted from the original
between a story and a nursery rhyme block, the interval data set. The functional and anatomical images of each
was 16 sec. Participants were instructed to listen to the participant were coregistered and normalized to the ste-
stories carefully and to do nothing during ISIs. To measure reotaxic space of the Montreal Neurological Institute
whether or not participants paid attention to the stimuli, (MNI) using SPM8 (www.fil.ion.ucl.ac.uk/spm/). All
they were asked to answer three yes–no comprehension coordinates are reported in MNI space. The functional
questions after each run, responding via button press. images were resampled to a voxel size equal to 3 × 3 ×

Table 4. Mean and Standard Error (in Parentheses) of Vividness Ratings in Each Condition

Condition Vividness of the Described Scene Vividness of the Described Bodily Action Intensity of Emotion
Perception 4.2 (0.1)a 3.1 (0.1) 2.7 (0.1)b
Action 3.6 (0.2) 4.1 (0.1)a 2.2 (0.1)
c
Emotion 3.5 (0.2) 3.5 (0.1) 4.4 (0.1)a
Control 2.9 (0.2)a 2.7 (0.1)a 2.1 (0.1)a

Boldface indicates the ratings that were expected to be significantly higher than the other conditions.
a
Indicates significant differences from all the other conditions.
b
Indicates a significant difference ( p < .05) from the Action condition.
c
Indicates a significant difference ( p < .05) from the Perception condition.

284 Journal of Cognitive Neuroscience Volume 26, Number 2

3 mm and smoothed with an isotropic 6-mm FWHM graphs using the same conjunction method described
Gaussian kernel. above (i.e., first–second ∩ first–third, second–first ∩
second–third, and third–first ∩ third–second). Doing so
allows us to examine the effect of paragraph order across
Activation Analysis
the stories, independent of story content.
The activation analysis was conducted using the frame- To identify the regions activated in all story conditions,
work of the GLM implemented in SPM8. For the subject- we compared each story condition (including the Control
level analysis, separate regressors were constructed by conditions) to the nursery rhyme condition separately.
convolving the box-car function of each condition Conjunctions between these contrasts were identified
with the canonical hemodynamic function. The model using the same threshold described above.
also included regressors of no interest to account for Because the size of the amygdala is small compared
variance because of low-frequency (<1/128 Hz) scanner with the spatial extent threshold used for family-wise
drift and global signal fluctuations. Additionally, we error correction, an anatomical ROI approach was used
removed potential effects of no interest associated with to detect increased hemodynamic response for this re-
our auditory presentations to focus on our primary in- gion. For each condition, the average responses of all
terest in the content types. Specifically, the means and voxels located in left and right amygdalae, defined accord-
standard deviations of the fundamental frequency and ing to a standard atlas (Tzourio-Mazoyer et al., 2002), was
intensity of the auditory stimuli, along with the readabil- calculated. The average response between conditions was
ity index for each paragraph, were included as individual compared using paired t tests, and the threshold of sig-
regressors. The estimated hemodynamic responses for nificance was set at p < .05.
each condition and participant were entered into a re-
peated-measures ANOVA model, implemented in SPM8,
Functional Connectivity Analysis
to make statistical inferences at the population level (i.e.,
a random effects analysis). Contrasts between conditions Additional preprocessing steps were carried out for the
were computed using paired t tests. functional connectivity analyses. We removed variance
To identify the regions that responded selectively to explained by the task regressors and those associated
each content type, we used a conjunction analysis similar with effects of no interest from the preprocessed data,
to that described in a previous study (Simmons, Reddish, for each participantʼs GLM. The residual time series of
Bellgowan, & Martin, 2010). Selective activation was de- each voxel was then band-pass filtered with cutoff fre-
fined as a cluster exhibiting reliably greater activity for a quencies at 0.03 and 0.3 Hz, which ensured that the
particular content type than the other two. To control for connectivity between regions was not affected by high
the potential corresponding effect of sequence, for each frequency physiological noise or low-frequency scanner
comparison, the difference between two conditions had drift patterns. The epochs of each condition were con-
to be significantly stronger than the difference between catenated according to their onsets and offsets with a
the corresponding sequence-matched control paragraphs 6-sec delay accounting for the sluggishness of the hemo-
(i.e., an interaction between story content and paragraph dynamic response, generating a data set for each con-
order). For example, to qualify as a selective activation dition (Bokde, Tagamets, Friedman, & Horwitz, 2001).
for perceptual content, the activity of a region had to The seed locations were defined as the peak activations
be significantly greater in two separate statistical tests in the LIFG (−51, 24, 18; t = 6.1), left pMTG (−51, −42, 0;
of interaction: (Perception > Action) > (first > second con- t = 8.9), left aTL (−51, −9, −18; t = 13.1), and right
trol paragraphs) and (Perception > Emotion) > (first > aTL (54, −6, −18; t = 11.6). These peaks were determined
third control paragraphs). The alpha threshold for each by comparing the averaged response across all story con-
of the individual tests of interaction was set at p < .0075 ditions (i.e., Perceptual, Action, Emotion, and Control
one-tailed with minimum cluster size equal to 58 voxels, conditions) with the Nursery Rhymes baseline ( p < .05,
corresponding to p < .05 (family-wise error corrected) corrected). For each condition, the eigenvector of all
based on Monte Carlo simulations (computed by 3dClust- voxels within a sphere (radius of 5 mm) centered at each
Sim in AFNI). Because the individual tests were cor- seedʼs coordinate was computed. A correlation map
rected for multiple comparisons, the regions surviving was generated by calculating a Pearsonʼs correlation
in the conjunction analysis are regarded as reliable. How- coefficient (r) between a given seedʼs time series and each
ever, it is possible that small intersections between voxelʼs time series in the brain. Fisherʼs transformation
clusters from different statistical tests could arise from was then applied to convert these correlation coefficients
spatial smoothing and resampling (Simmons et al., to Z values. For each seed, the participantsʼ Fisherʼs
2010). To rule out this possibility, a small cluster size Z maps were subjected to a repeated-measures ANOVA
threshold of at least 10 voxels was applied in the con- model to make statistical inferences at the population
joined areas. level. Contrasts between two conditions were computed
In addition to comparing the Perception, Action, and using paired t tests. To identify connectivity changes spe-
Emotion conditions, we also compared the control para- cific to the Perception, Action, and Emotion conditions,

Chow et al. 285

we employed the same conjunction analysis method used accuracy of the 24 participants included in the analyses
in the activation analyses. The alpha threshold for each of was 88.5 ± 4.3% (SD), indicating that they paid attention
the individual tests of interaction was set at p < .015 one- to the story stimuli.
tailed with minimum cluster size equal to 79 voxels,
corresponding to p < .05 (family-wise error corrected)
based on Monte Carlo simulations (computed by 3dClust- Activations Modulated by Story Content
Sim in AFNI). These analyses revealed clusters that in- Content-specific activations were defined as clusters of
creased in correlation with a seed region for a particular voxels exhibiting reliably greater activity for a particular
type of content, to a degree greater than that observed content type than for the other two types of content
for both of the other two types of story content and the while controlling for the paragraph sequence effect,
sequence-matched control paragraphs in three separate when tested individually (see Methods). A composite
statistical tests. map of content-specific activations is shown in Figure 1
(see also Table 5). As predicted, perceptual, action-based,
RESULTS and emotionally charged story content modulated re-
sponses in the related modality-specific brain regions.
Behavioral Results In the Perception condition, we observed robust activa-
In general, participants accurately answered the com- tions in regions associated with visual-spatial processing,
prehension questions at the end of each run. The mean including the left occiptoparietal cortex, the precuneus,

Figure 1. Brain regions

selectively activated in
processing visually vivid
(pink), action-based (red),
and emotionally charged
(blue) story content.
A region was considered
reliably activated for a condition
only if the activation in that
condition was significantly
stronger than the other two
in two separate comparisons
controlling for the paragraph-
sequence effect (see Methods).
All individual comparisons were
corrected for family-wise error
( p < .05), and the results are
rendered on a single-subject
anatomical image.

286 Journal of Cognitive Neuroscience Volume 26, Number 2

Table 5. Regions Activated Specifically for Visually Vivid, Action-based, and Emotionally Charged Story Content
Peak MNI Coordinates

Cluster Hemisphere x y z Min. t a Volume (cm3)

Perception
Parahippocampal/fusiform gyri L −27 −33 −18 5.4 3.8
Parahippocampal/fusiform gyri R 30 −33 −18 5.1 2.2
Retrosplenial cortex/precuneus R 12 −51 12 5.2 12.1
Retrosplenial cortex/precuneus L −12 −54 12 5.0 3.5
Middle cingulate cortex L −9 −36 27 5.5 1.5
Middle occipital lobe L −30 −84 33 4.6 4.3

Action
Anterior intraparietal area/ L −54 −33 36 4.0 2.1
somatosensory cortex
Left dorsal premotor cortex L −18 6 57 3.2 1.8

Emotion

g
Superior temporal sulcus L −54 3 −24 8.7
Temporal parietal junction L −51 −60 24 8.2
41.6
Middle temporal gyrus L −54 −42 0 5.9
Orbital frontal gyrus L −54 21 9 5.6

g
Superior temporal sulcus R 48 9 −30 9.6
Superior temporal sulcus R 60 −30 −6 7.2
16.6
Temporal parietal junction R 57 −60 30 6.1
Orbital frontal gyrus R 54 21 9 5.6
Cerebellum L −24 −78 −36 8.6 6.9
Cerebellum R 30 −78 −36 8.1 8.0
Cerebellar vermis L −6 −60 −48 7.1 4.0
Medial prefrontal cortex L −6 54 30 8.3 6.3
Medial prefrontal cortex L/R 0 60 −15 5.2 1.6
Posterior cingulate cortex/ L/R 0 −54 33 5.7 7.2
precuneus
Medial orbital frontal L/R 0 60 −15 5.2 1.6
Middle frontal gyrus L −48 6 51 5.2 3.3
a
Min. t is the minimal t statistics of the individual comparisons between a condition and each of the other two conditions.

the retrosplenial cortex, the parahippocampal gyrus, and the TPJ, and the orbital frontal gyri bilaterally. Emotionally
the fusiform gyrus. The Action condition was associated charged content also elicited significantly greater re-
with selective activation of the anterior intraparietal (AIP) sponses than the other two content types in bilateral
area including the left somatosensory cortex and the amygdalae, according to our ROI analysis (Figure 2).
left dorsal premotor cortex (PMd). Emotionally charged To examine the effect of paragraph sequence, inde-
story content elicited responses in paralimbic and pre- pendent of story content, we compared the first, second,
frontal areas frequently associated with affective pro- and third control paragraphs using the conjunction analy-
cessing, social concept processing, and mentalizing. These sis method and threshold. When the first paragraph was
included the ventral and dorsal MPFC, the PCC, the STS, compared with the second and the third paragraphs in

Chow et al. 287

Figure 2. Activity in the left
and right amygdalae in response
to visually vivid, action-based,
and emotionally charged story
content. This ROI analysis
showed that the responses to
emotionally charged content
were significantly stronger than
the responses to the other two
content types (indicated by
bracketed asterisks; p < .05).
Error bars in both panels
indicate the standard error
in each condition.

two separate tests, only the pre-SMA and the left anterior ent types of story content. The LIFG showed selective
insula were significantly activated. But, the visual-spatial increases in functional connectivity with the modality-
processing regions that were activated in the Perception specific regions activated for perceptual, action-based,
condition, including the occipital lobe, retrosplenial cor- and emotionally charged content in the Perception, Action,
tex, parahippocampal gyrus, were not detected in this and Emotion conditions, respectively (Figure 4 and
comparison. There were no significant activations at all Table 6). In other words, when perceptual content was
detected when the second paragraph was compared with presented in the story, there was greater connectivity be-
the first and the third paragraphs, nor were any significant tween the LIFG and areas associated with perception,
activations found when we compared the third with the and so forth. Comparing Perception with the Action, Emo-
first and the second paragraphs. These analyses provide tion, and Control conditions, the LIFG was more strongly
further evidence that the modality-specific activations connected with the left and right parahippocampal gyrus.
associated with the three content types are not driven by These regions overlapped with the clusters that exhibited
the paragraph sequence in a story. increased responses selective to perceptual content, as
reported in the previous section. The Action condition
selectively modulated connectivity between the LIFG and
Activations Common to All Story Conditions
a cluster in the left AIP area, the same region that showed
To identify regions activated by story comprehension increased activity for action-based content. However,
per se, independent of content type, we compared each neither the LIFG, nor other regions tested (i.e., pMTG
story condition individually to the nursery rhymes condi- and bilateral aTL), showed enhanced connectivity with
tion to control for low-level language processing. Figure 3 the left PMd in the Action condition, a region that had
shows regions that were commonly activated across all been selectively activated for action-based content. Lastly,
individual comparisons. As expected, language areas LIFG activity was more strongly correlated with the right
were activated, including the LIFG, the posterior superior IPL (extending to the TPJ) by the Emotion condition,
and middle temporal gyri, and the bilateral aTL. Impor- compared with the Perception, Action, and Control con-
tantly, the seed regions representing the language regions ditions. The same anatomical regions were strongly acti-
selected for the connectivity analyses overlapped with vated when emotionally charged story segments were
the activated clusters in this analysis (Figure 3). In other contrasted with the other two types of content.
words, the seed regions were activated in every story con- A parallel analysis focusing on the pMTG revealed a
dition when they were each compared with listening to similar pattern to that observed with the LIFG in the sense
nursery rhymes. Moreover, additional regions that have that all three types of content selectively modulated con-
been associated with narrative comprehension were also nectivity with the modality-specific regions that had been
active, including the left dorsal lateral pFC, the MPFC, identified in the previous activation analysis (Figure 4).
the PCC, the parahippocampal gyrus, and the fusiform gy- However, the specific associations observed were slightly
rus (Mar, 2011; Ferstl et al., 2008; Yarkoni, Speer, & Zacks, different. During the Perception condition, the pMTG
2008; Xu, Kemeny, Park, Frattali, & Braun, 2005). Exten- was more strongly connected to the left parahippocampal
sive activations were also observed in the lingual gyrus gyrus. During the Action condition, there was a significant
and the cerebellum. increase in connectivity between the pMTG and both the
left posterior inferior temporal sulcus and the AIP area.
During the Emotion condition, the pMTG was more
Modulation of Functional Connectivity with the
strongly connected to the PCC, but not the IPL/TPJ.
LIFG, pMTG, and aTL With respect to aTL connectivity, the pattern of con-
We next examined how functional connectivity with the nectivity enhancement was markedly different, in that it
language areas was influenced by presentation of differ- was confined to the Emotion condition (Figure 4). While

288 Journal of Cognitive Neuroscience Volume 26, Number 2

listening to emotionally charged content, the right aTL Lastly, we evaluated the connectivity patterns asso-
showed strong connectivity enhancement with the MPFC, ciated with the first, second, and third control paragraphs
PCC, bilateral IPL/TPJ, and right orbital frontal gyrus. A using the LIFG, pMTG, and aTL seed regions. However,
similar connectivity pattern was observed for the left aTL, none of these comparisons yielded any significant clus-
which was more connected to the bilateral IPL/TPJ and ters at the specified threshold.
PCC. Neither the left nor right aTL showed significant
increase in connectivity with any regions associated with
DISCUSSION
the Perception and Action conditions.
Our goal was to understand how the language regions
interact with modality-specific systems during naturalistic,
spoken language comprehension. Consistent with the
embodied view of language comprehension and previous
studies on story content (Speer et al., 2009; Ferstl &
von Cramon, 2007; Ferstl et al., 2005), we demonstrated
that modality-specific activations are selectively modulated
by the content of a story. To investigate the interactions
between language and modality-specific regions, we ex-
amined the modulatory effects of content on the connec-
tions between language regions and modality-specific
regions. As predicted, the functional connectivity analyses
illustrated that the language regions interacted selectively
with modality-specific systems during the comprehension
of different types of story content, but the connectivity
patterns associated with each of the language ROIs were
markedly different. Although the LIFG, pMTG, and bilaterial
aTL were strongly activated for all stories, regardless of
content, only the LIFG and pMTG showed increased func-
tional connectivity with all three of the modality-specific
neural systems in response to the corresponding story
content.
Our results suggest that sensorimotor and affective
simulations during narrative comprehension involve not
only the modality-specific system relevant to the content
but that these multimodal simulations likely manifest
through collaboration between the modality-specific neu-
ral systems and language areas. Since both the LIFG and
pMTG interacted with modality-specific regions in a con-
tent-dependent fashion, the LIFG and pMTG may subserve
a general mechanism to activate the comprehenderʼs per-
ceptual, motor, and affective knowledge and integrate it
into a coherent mental representation of the situation
described, known as a situation model (Zwaan & Radvansky,
1998; Kintsch, 1988).

Neural Responses to Different Forms of

Story Content
Visually vivid descriptions of scenes evoked strong re-
sponses in major components of the parietomedial
temporal pathway for visual-spatial processing (Kravitz,
Saleem, Baker, & Mishkin, 2011), including regions critical
Figure 3. Activations common to all story conditions when each for processing visual scenes, construction of mental scenes,
was compared individually to the nursery rhyme condition and the and spatial navigation (Vann, Aggleton, & Maguire, 2009;
approximate locations of the seed regions selected for the functional
Epstein, 2008). This implies that generating a mental
connectivity analyses (indicated by green dots). The minimum t
statistics for the individual comparisons are presented using yellow– scene of the described situation may involve a reactivation
orange color scale and rendered on a single-subject anatomical image. of past personal perceptual experience triggered by the
Each comparison was corrected for family-wise error ( p < .05). text (Mar & Oatley, 2008).

Chow et al. 289

Figure 4. Regions that exhibited selective increases in connectivity with the LIFG seed (A), the left pMTG seed (B), the left aTL seed (C),
and the right aTL (D) for visually vivid (pink), action-based (red), and emotionally charged (blue) story content. The green dots indicate
the approximated locations of the seed regions. An increase in connectivity was considered reliable for a condition only if the increase in that
condition was significantly larger than the other two in two separate comparisons controlling for the paragraph-sequence effect (see Methods).
All individual comparisons were corrected for family-wise error ( p < .05), and the results are rendered on a single-subject anatomical image.

Descriptions of actions elicited robust responses in the MPFC, PCC, STS, and IPL). Although the role of the
a different set of brain regions, those that play an im- amygdala during mentalizing is not clear, a previous
portant role in the planning and control of complex meta-analysis demonstrated that this structure is reliably
movements (Gazzola & Keysers, 2009; Filimon, Nelson, activated across studies of mentalizing (Mar, 2011). In
Hagler, & Sereno, 2007; Buccino et al., 2001). Increased addition, it has been shown that patients who acquired a
activation in these motor areas during the processing of lesion in the amygdala early in life were impaired in the
action-based content suggest that motor and somato- ability to reason about the mental states of others (Shaw
sensory representations of the described actions are et al., 2004). Although we cannot rule out the possibility
simulated during narrative comprehension, consistent that activations in the amygdala merely reflect the partici-
with past work on single-word and sentence processing pantsʼ own arousal triggered by the storiesʼ emotional
(Tettamanti et al., 2005; Hauk, Johnsrude, & Pulvermuller, content, in light of the evidence mentioned above, the
2004). amygdala may play a role in evaluating the protagonistʼs
With respect to emotionally charged content, we found emotional state (Thom et al., 2012; Dapretto et al., 2006;
that these story segments elicited selective responses in Keysers & Gazzola, 2006; Ruby & Decety, 2004) and may
the left and right amygdalae. Activation of the amygdala also act in concert with the mentalizing network to inter-
has been associated with the processing of affective stim- pret the intentions underlying the protagonistsʼ actions
uli (Phelps, 2006), including emotional words (Herbert (Olsson & Ochsner, 2008).
et al., 2009; Isenberg et al., 1999). However, understand-
ing a protagonistʼs emotional reactions during story com-
Neural Responses to Narrative Sequence
prehension is more complicated than processing simple
Independent of Content
emotional stimuli because it requires understanding the
complex social situations narrated in the story. This re- In our analyses, the control paragraphs were used to
quires the use of social concepts (Frith, 2007; Dalgleish, rule out the possibility that condition-dependent effects
2004) and inferring the mental states of the protagonist might be because of the sequence in which the experimen-
(e.g., beliefs and intentions; Olsson & Ochsner, 2008). tal conditions were presented. These control paragraphs
Consistent with this view, we observed increased activa- can also be used to examine the effect of accumulating
tions in a set of heteromodal regions during the Emotion context independent of story content. Xu and colleagues
condition, the so-called mentalizing network (including (2005) showed that the endings of Aesopʼs fables were

290 Journal of Cognitive Neuroscience Volume 26, Number 2

 Table 6. Regions that Exhibited Selective Increase in Connectivity with LIFG, Left pMTG, Left aTL, and Right aTL for Visually Vivid, Action-based, and Emotionally Charged
Story Content

LIFG Left pMTG Left aTL Right aTL

Peak MNI Peak MNI Peak MNI Peak MNI
Coordinates Coordinates Coordinates Coordinates
Volume Volume Volume Volume
Hemisphere x y z Min. t a (cm3) x y z Min. t (cm3) x y z Min. t (cm3) x y z Min. t (cm3)

Perception
Parahippocampal gyrus R 24 −33 −15 3.8 1.5
Parahippocampal gyrus L −30 −18 −18 3.7 2.8 −21 −36 −15 2.6 0.5

Action
Inferior temporal sulcus L −48 −60 −6 4.1 1.4
Anterior intraparietal area L −45 −42 45 4.0 4.3 −57 −33 39 3.0 0.6

Emotion
Inferior parietal lobule L −57 −60 36 3.3 0.9 −45 −57 33 3.7 2.4
Inferior parietal lobule R 48 −36 36 3.2 1.6 54 −60 36 3.0 1.3 51 −60 33 3.6 2.4
Posterior cingulate cortex/ R 6 −45 27 3.6 5.9
Precuneus
Posterior cingulate cortex/ L −6 −60 30 3.5 4.3 −3 −51 30 3.0 2.2
Precuneus
Medial prefrontal cortex R 9 48 39 2.8 2.8
a
Chow et al.

Min. t is the minimal t statistics of the individual comparisons between a condition and each of the other two conditions.
291
associated with increased hemodynamic responses in not independent of the language regions. On the contrary,
the MPFC, PCC, and bilateral IPL. However, when we they are functionally coupled in a systematic way, with
compared the third control paragraph with the first and multimodal simulations possibly emerging from the inter-
second, none these regions were significantly activated. actions between language and modality-specific systems.
An important difference in story materials might help to The connectivity results for the aTL differed markedly
explain these diverging results, as Xu and colleagues did from those observed for the IFG and pMTG, in that the
not take story content into consideration. It is plausible modulation of connectivity for the aTL was only observed
that the fables used in that study were emotional at the during the Emotion condition. During the presentation of
end of the fable compared with the beginning or involved emotional content, the correlations between the aTL and
moral teachings and social concepts that were only pre- elements of the mentalizing system were significantly en-
sented at the end of the fables. If this is the case, the in- hanced relative to the presentation of the other two types
creased responses in the MPFC, PCC, and bilateral IPL of content. This pattern of results can inform the current
associated with at the end of the fables previously reported debate on the role of the aTL in semantic processing. Past
by Xu and colleagues (2005) would be consistent with the research on semantic dementia patients suggests that the
results of this study; we observed activity in this same set aTL is involved in a wide variety of semantic tasks (Lambon
of regions in response to emotional content. Ralph et al., 2010; Davies, Halliday, Xuereb, Kril, & Hodges,
2009; Rogers et al., 2006; Williams, Nestor, & Hodges,
2005). This has led a number of researchers to propose
Interactions between Language and
that the aTL is a “semantic hub,” binding multimodal in-
Modality-specific Systems
formation into amodal, domain-general representations
In addition to identifying activations modulated by the (Lambon Ralph et al., 2010; Ferstl et al., 2008; Patterson
three content types, we interrogated the data with respect et al., 2007; Jung-Beeman, 2005). Evidence for the semantic
to functional connectivity. The foregoing activation analy- hub hypothesis primarily comes from tasks that rely on
sis identified the extent to which activity in each voxel, single words or pictures for stimuli, so it is unclear whether
reflected in the hemodynamic response, was sustained the aTL behaves similarly when people are presented with
throughout the onset and offset of paragraphs for each discourse. It is possible that in this situation the aTL also
condition. On the other hand, the functional connectivity acts as a semantic hub, integrating modality-specific infor-
analyses quantified the degree of synchronization between mation into a coherent discourse representation. If this is
transient activity in spatially segregated regions. Although the case, then we would expect the aTL to interact with
connectivity analyses cannot determine how this transient all three modality-specific systems to integrate information
activity is triggered within a task condition, it reflects a and form these representations. In contrast, in our data
direct or indirect functional relationship between two connectivity with the aTL was only modulated by emo-
regions. Examining functional connectivity is particularly tional content. This indicates that the functional role of
important for the understanding of higher-level cognition the aTL in processing words or pictures may be different
such as narrative comprehension, because these complex than its role during discourse comprehension. In narrative
functions are generally assumed to emerge from inter- comprehension, the aTL may play a more specialized role
actions between regions with specialized functions (Friston, in comprehending protagonistsʼ emotions and retrieving
2002). Because the language regions examined in our social concepts during story comprehension, consistent
study are all robustly activated during discourse compre- with past research that has associated the aTL with social
hension (in our data and in previous studies; Binder et al., semantic-processing (Binder & Desai, 2011; Simmons
2011; Mar, 2011; Ferstl et al., 2008), the activation analysis et al., 2010; Simmons & Martin, 2009). It is also possible
is not ideally suited to differentiate their roles in processing that different regions in the aTL are devoted to different
different types of story content. Using a connectivity analy- processes. The ventral portion of the aTL may be respon-
sis, we revealed that the language regions demonstrate sible for “semantic hub” functions (Binney, Embleton,
different degrees of functional connectivity with modality- Jefferies, Parker, & Ralph, 2010), in contrast to the superior
specific regions and that these are selectively enhanced in portion of the aTL where our seeds were located. Lastly, it
response to the different types of story content. is also possible that the anterior temporal regions may not
While listening to visually vivid content, both the LIFG have been adequately sampled during our data acquisi-
and pMTG were more functionally connected with regions tion because these areas of the brain are quite vulnerable
responsible for visual-spatial processing. Similar patterns to susceptibility artifacts (Visser, Jefferies, & Lambon Ralph,
were observed during the comprehension of action-based 2010). Further studies are needed to properly clarify these
and emotionally charged content. Both areas demon- issues.
strated significantly greater connectivity to regions respon- Both the LIFG and pMTG showed enhanced connectiv-
sible for motor processing and mentalizing during action ity with elements of all three modality-specific systems in
and emotion paragraphs, respectively. This pattern of a content-specific manner. This indicates that the LIFG
content-specific connectivity enhancement suggests that and pMTG might contribute to a general mechanism for
simulations instantiated in modality-specific systems are multimodal simulations through which perceptual, motor,

292 Journal of Cognitive Neuroscience Volume 26, Number 2

and affective representations of the described situation Barsalou, L. W. (2003). Situated simulation in the human
are constructed. Incorporating our connectivity results into conceptual system. Language and Cognitive Processes,
18, 513–562.
contemporary models of language processing (Lau et al., Binder, J. R., & Desai, R. H. (2011). The neurobiology of
2008; Tyler & Marslen-Wilson, 2008; Hagoort, 2005), we semantic memory. Trends in Cognitive Sciences, 15,
postulate that the pMTG and LIFG may play distinct roles 527–536.
in activating comprehenderʼs text-relevant perceptual, Binder, J. R., Gross, W. L., Allendorfer, J. B., Bonilha, L.,
motor, and affective knowledge and using this knowledge Chapin, J., Edwards, J. C., et al. (2011). Mapping anterior
temporal lobe language areas with fMRI: A multicenter
to form coherent, multimodal mental representations. normative study. Neuroimage, 54, 1465–1475.
The pMTG has been shown to play an important role in Binney, R. J., Embleton, K. V., Jefferies, E., Parker, G. J., &
accessing conceptual knowledge based on brain imaging Ralph, M. A. (2010). The ventral and inferolateral aspects
and lesion studies (Hickok & Poeppel, 2007; Dronkers, of the anterior temporal lobe are crucial in semantic
Wilkins, Van Valin, Redfern, & Jaeger, 2004). Extending memory: Evidence from a novel direct comparison of
distortion-corrected fMRI, rTMS, and semantic dementia.
the language processing models proposed by Lau and Cerebral Cortex, 20, 2728–2738.
colleagues (2008) and Hickok and Poeppel (2007), we Bokde, A. L., Tagamets, M. A., Friedman, R. B., & Horwitz, B.
propose that the pMTG serve as an interface between (2001). Functional interactions of the inferior frontal
lexical and modality-specific representations. Through this cortex during the processing of words and word-like
interface, the comprehenderʼs text-related knowledge is stimuli. Neuron, 30, 609–617.
Boulenger, V., Mechtouff, L., Thobois, S., Broussolle, E.,
activated and represented in perceptual, motor, and affec- Jeannerod, M., & Nazir, T. A. (2008). Word processing in
tive neural systems. This would account for the modulation Parkinsonʼs disease is impaired for action verbs but not
of connectivity between the pMTG and the modality- for concrete nouns. Neuropsychologia, 46, 743–756.
specific regions observed in our study. In contrast, the Buccino, G., Binkofski, F., Fink, G. R., Fadiga, L., Fogassi, L.,
interactions between the LIFG and modality-specific re- Gallese, V., et al. (2001). Action observation activates
premotor and parietal areas in a somatotopic manner:
gions may reflect the selection of activated information An fMRI study. European Journal of Neuroscience, 13,
based on the context established by preceding portions 400–404.
of the text (Thompson-Schill, DʼEsposito, Aguirre, & Farah, Buckner, R. L., Andrews-Hanna, J. R., & Schacter, D. L.
1997), along with integration of this information into (2008). The brainʼs default network: Anatomy, function,
modality-specific representations (Hagoort, 2005). As the and relevance to disease. Annals of the New York
Academy of Sciences, 1124, 1–38.
story unfolds, perceptual, motor, and affective represen- Coltheart, M. (1981). The MRC psycholinguistic database.
tations are updated through activation and integration Quarterly Journal of Experimental Psychology:
processes (Zwaan, 2003), forming a multimodal situation Section A-Human Experimental Psychology, 33, 497–505.
model of the narrative. Cox, R. W. (1996). AFNI: Software for analysis and visualization
Taken together, our results provide a more complete of functional magnetic resonance neuroimages. Computers
in Biomedical Research, 29, 162–173.
understanding of the neural mechanism underlying lan- Dalgleish, T. (2004). The emotional brain. Nature Reviews
guage comprehension. Comprehension emerges not only Neuroscience, 5, 583–589.
from the interactions between frontal and temporal brain Dapretto, M., Davies, M. S., Pfeifer, J. H., Scott, A. A.,
systems, as proposed by contemporary models of language Sigman, M., Bookheimer, S. Y., et al. (2006). Understanding
processing, but also from content-dependent interactions emotions in others: Mirror neuron dysfunction in children
with autism spectrum disorders. Nature Neuroscience, 9,
between these language regions and modality-specific 28–30.
brain systems. Davies, R. R., Halliday, G. M., Xuereb, J. H., Kril, J. J., &
Hodges, J. R. (2009). The neural basis of semantic memory:
Evidence from semantic dementia. Neurobiology of Aging,
Acknowledgments 30, 2043–2052.
Desai, R. H., Binder, J. R., Conant, L. L., & Seidenberg, M. S.
The authors would like to thank Maja Djikic for her assistance
(2010). Activation of sensory-motor areas in sentence
in story composition. We also thank Alex Martin for helpful dis-
comprehension. Cerebral Cortex, 20, 468–478.
cussions and the NIH Fellows Editorial Board for editing an
Dronkers, N. F., Wilkins, D. P., Van Valin, R. D., Jr., Redfern,
earlier version of this paper. This work was supported by the
B. B., & Jaeger, J. J. (2004). Lesion analysis of the brain
Intramural Research Program of the NIDCD.
areas involved in language comprehension. Cognition,
Reprint requests should be sent to Ho Ming Chow, National Insti- 92, 145–177.
tutes of Health, 10/5C410, 10 Center Dr., Bethesda, MD 20892, or Epstein, R. A. (2008). Parahippocampal and retrosplenial
via e-mail: [email protected]. contributions to human spatial navigation. Trends in
Cognitive Sciences, 12, 388–396.
Ferstl, E. C., Neumann, J., Bogler, C., & von Cramon, D. Y.
(2008). The extended language network: A meta-analysis
REFERENCES of neuroimaging studies on text comprehension. Human
Brain Mapping, 29, 581–593.
Badre, D., Poldrack, R. A., Pare-Blagoev, E. J., Insler, R. Z., Ferstl, E. C., Rinck, M., & von Cramon, D. Y. (2005). Emotional
& Wagner, A. D. (2005). Dissociable controlled retrieval and temporal aspects of situation model processing during
and generalized selection mechanisms in ventrolateral text comprehension: An event-related fMRI study. Journal
prefrontal cortex. Neuron, 47, 907–918. of Cognitive Neuroscience, 17, 724–739.

Chow et al. 293

Ferstl, E. C., & von Cramon, D. Y. (2007). Time, space grounding conceptual content. Journal of Physiology
and emotion: fMRI reveals content-specific activation Paris, 102, 59–70.
during text comprehension. Neuroscience Letters, 427, Mar, R. A. (2011). The neural bases of social cognition and story
159–164. comprehension. Annual Review of Psychology, 62, 103–134.
Filimon, F., Nelson, J. D., Hagler, D. J., & Sereno, M. I. Mar, R. A., & Oatley, K. (2008). The function of fiction is the
(2007). Human cortical representations for reaching: abstraction and simulation of social experience. Perspectives
Mirror neurons for execution, observation, and imagery. on Psychological Science, 3, 173–192.
Neuroimage, 37, 1315–1328. Mason, J., & Just, M. A. (2006). Neuroimaging contributions
Friston, K. (2002). Functional integration and inference in to the understanding of discourse processes.
the brain. Progress in Neurobiology, 68, 113–143. In M. T. M. A. Gernsbacher (Ed.), Handbook of
Frith, C. D. (2007). The social brain? Philosophical psycholinguistics (pp. 765–799). Amsterdam: Elsevier.
Transactions of the Royal Society of London, Series B, McKeown, M. J., Makeig, S., Brown, G. G., Jung, T. P.,
Biological Sciences, 362, 671–678. Kindermann, S. S., Bell, A. J., et al. (1998). Analysis of
Gazzola, V., & Keysers, C. (2009). The observation and fMRI data by blind separation into independent spatial
execution of actions share motor and somatosensory components. Human Brain Mapping, 6, 160–188.
voxels in all tested subjects: Single-subject analyses of Meteyard, L., Cuadrado, S. R., Bahrami, B., & Vigliocco, G.
unsmoothed fMRI data. Cerebral Cortex, 19, 1239–1255. (2012). Coming of age: A review of embodiment and
Hagoort, P. (2005). On Broca, brain, and binding: A new the neuroscience of semantics. Cortex, 48, 788–804.
framework. Trends in Cognitive Sciences, 9, 416–423. Neininger, B., & Pulvermuller, F. (2003). Word-category
Hauk, O., Johnsrude, I., & Pulvermuller, F. (2004). Somatotopic specific deficits after lesions in the right hemisphere.
representation of action words in human motor and Neuropsychologia, 41, 53–70.
premotor cortex. Neuron, 41, 301–307. Oldfield, R. C. (1971). The assessment and analysis of handedness:
Herbert, C., Ethofer, T., Anders, S., Junghofer, M., The Edinburgh inventory. Neuropsychologia, 9, 97–113.
Wildgruber, D., Grodd, W., et al. (2009). Amygdala Olsson, A., & Ochsner, K. N. (2008). The role of social cognition
activation during reading of emotional adjectives-An in emotion. Trends in Cognitive Sciences, 12, 65–71.
advantage for pleasant content. Social Cognitive and Patterson, K., Nestor, P. J., & Rogers, T. T. (2007). Where do
Affective Neuroscience, 4, 35–49. you know what you know? The representation of semantic
Hickok, G., & Poeppel, D. (2007). The cortical organization knowledge in the human brain. Nature Reviews
of speech processing. Nature Reviews Neuroscience, Neuroscience, 8, 976–987.
8, 393–402. Phelps, E. A. (2006). Emotion and cognition: Insights from
Isenberg, N., Silbersweig, D., Engelien, A., Emmerich, S., studies of the human amygdala. Annual Review of
Malavade, K., Beattie, B., et al. (1999). Linguistic threat Psychology, 57, 27–53.
activates the human amygdala. Proceedings of the Price, C. J. (2000). The anatomy of language: Contributions
National Academy of Sciences, U.S.A., 96, 10456–10459. from functional neuroimaging. Journal of Anatomy, 197,
Jung-Beeman, M. (2005). Bilateral brain processes for 335–359.
comprehending natural language. Trends in Cognitive Rogers, T. T., Hocking, J., Noppeney, U., Mechelli, A.,
Sciences, 9, 512–518. Gorno-Tempini, M. L., Patterson, K., et al. (2006).
Keysers, C., & Gazzola, V. (2006). Towards a unifying neural Anterior temporal cortex and semantic memory:
theory of social cognition. Progress in Brain Research, Reconciling findings from neuropsychology and functional
156, 379–401. imaging. Cognitive, Affective & Behavioral Neuroscience,
Kincaid, J. P., Fishburne, R. P., Rogers, R. L., & Chissom, 6, 201–213.
B. S. (1975). Derivation of new readability formulas Ruby, P., & Decety, J. (2004). How would you feel versus
(automated readability index, fog count, and Flesch how do you think she would feel? A neuroimaging study
reading ease formula) for navy enlisted personnel. of perspective-taking with social emotions. Journal of
Research Branch Report 8-75. Chief of Naval Technical Cognitive Neuroscience, 16, 988–999.
Training. Memphis, TN: Naval Air Station. Shaw, P., Lawrence, E. J., Radbourne, C., Bramham, J.,
Kintsch, W. (1988). The role of knowledge in discourse Polkey, C. E., & David, A. S. (2004). The impact of early
comprehension: A construction-integration model. and late damage to the human amygdala on “theory of
Psychological Review, 95, 163–182. mind” reasoning. Brain, 127, 1535–1548.
Kravitz, D. J., Saleem, K. S., Baker, C. I., & Mishkin, M. (2011). Simmons, W. K., & Martin, A. (2009). The anterior temporal
A new neural framework for visuospatial processing. lobes and the functional architecture of semantic memory.
Nature Reviews Neuroscience, 12, 217–230. Journal of the International Neuropsychological Society,
Kucera, H., & Francis, W. N. (1967). Computational Analysis 15, 645–649.
of Present-Day American English. Providence: Brown Simmons, W. K., Reddish, M., Bellgowan, P. S., & Martin, A.
University Press. (2010). The selectivity and functional connectivity of the
Lambon Ralph, M. A., Sage, K., Jones, R. W., & Mayberry, anterior temporal lobes. Cerebral Cortex, 20, 813–825.
E. J. (2010). Coherent concepts are computed in the Speer, N. K., Reynolds, J. R., Swallow, K. M., & Zacks, J. M.
anterior temporal lobes. Proceedings of the National (2009). Reading stories activates neural representations of
Academy of Sciences, U.S.A., 107, 2717–2722. visual and motor experiences. Psychological Science, 20,
Lau, E. F., Phillips, C., & Poeppel, D. (2008). A cortical 989–999.
network for semantics: (De)constructing the N400. Tettamanti, M., Buccino, G., Saccuman, M. C., Gallese, V.,
Nature Reviews Neuroscience, 9, 920–933. Danna, M., Scifo, P., et al. (2005). Listening to action-related
Li, Y. O., Adali, T., & Calhoun, V. D. (2007). Estimating sentences activates frontoparietal motor circuits. Journal
the number of independent components for functional of Cognitive Neuroscience, 17, 273–281.
magnetic resonance imaging data. Human Brain Thom, N. J., Johnson, D. C., Flagan, T., Simmons, A. N.,
Mapping, 28, 1251–1266. Kotturi, S. A., Van Orden, K. F., et al. (2012). Detecting
Mahon, B. Z., & Caramazza, A. (2008). A critical look at the emotion in others: Increased insula and decreased medial
embodied cognition hypothesis and a new proposal for prefrontal cortex activation during emotion processing

294 Journal of Cognitive Neuroscience Volume 26, Number 2

 in elite adventure racers. Social Cognitive and Affective Visser, M., Jefferies, E., & Lambon Ralph, M. A. (2010). Semantic
Neuroscience. doi: 10.1093/scan/nst029. processing in the anterior temporal lobes: A meta-analysis
Thompson-Schill, S. L., DʼEsposito, M., Aguirre, G. K., & of the functional neuroimaging literature. Journal of
Farah, M. J. (1997). Role of left inferior prefrontal cortex Cognitive Neuroscience, 22, 1083–1094.
in retrieval of semantic knowledge: A reevaluation. Willems, R. M., Labruna, L., DʼEsposito, M., Ivry, R., &
Proceedings of the National Academy of Sciences, U.S.A., Casasanto, D. (2011). A functional role for the motor
94, 14792–14797. system in language understanding: Evidence from
Turken, A. U., & Dronkers, N. F. (2011). The neural architecture of theta-burst transcranial magnetic stimulation.
the language comprehension network: Converging evidence Psychological Science, 22, 849–854.
from lesion and connectivity analyses. Frontiers in Systems Williams, G. B., Nestor, P. J., & Hodges, J. R. (2005).
Neuroscience, 5, 1. Neural correlates of semantic and behavioural deficits
Tyler, L. K., & Marslen-Wilson, W. (2008). Fronto-temporal in frontotemporal dementia. Neuroimage, 24,
brain systems supporting spoken language comprehension. 1042–1051.
Philosophical Transactions of the Royal Society of Xu, J., Kemeny, S., Park, G., Frattali, C., & Braun, A.
London, Series B, Biological Sciences, 363, 1037–1054. (2005). Language in context: Emergent features of word,
Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., sentence, and narrative comprehension. Neuroimage,
Crivello, F., Etard, O., Delcroix, N., et al. (2002). Automated 25, 1002–1015.
anatomical labeling of activations in SPM using a macroscopic Yarkoni, T., Speer, N. K., & Zacks, J. M. (2008). Neural
anatomical parcellation of the MNI MRI single-subject substrates of narrative comprehension and memory.
brain. Neuroimage, 15, 273–289. Neuroimage, 41, 1408–1425.
Vann, S. D., Aggleton, J. P., & Maguire, E. A. (2009). What Zwaan, R. A. (2003). The immersed experiencer: Toward
does the retrosplenial cortex do? Nature Reviews an embodied theory of language comprehension.
Neuroscience, 10, 792–802. The Psychology of Learning and Motivation, 44,
Vigneau, M., Beaucousin, V., Herve, P. Y., Duffau, H., Crivello, 35–62.
F., Houde, O., et al. (2006). Meta-analyzing left hemisphere Zwaan, R. A., & Radvansky, G. A. (1998). Situation models
language areas: Phonology, semantics, and sentence in language comprehension and memory. Psychological
processing. Neuroimage, 30, 1414–1432. Bulletin, 123, 162–185.

Chow et al. 295

This article has been cited by:

1. Rolf A. Zwaan. 2014. Embodiment and language comprehension: reframing the discussion. Trends in Cognitive Sciences 18:5,
229-234. [CrossRef]

View publication stats