Brain and Language 101 (2007) 181184 www.elsevier.

com/locate/b&l

Editorial

Gesture, brain, and language

People gesture when they speak. For the past 20 years, researchers have investigated the behavioral roles that these co-speech gestures play in language processing. This special issue represents the next phase of research on this topic by presenting eight papers that explore the neural links between gesture and speech during language production and comprehension. Since the publication in 1992 of David McNeills book Hand and Minda work that clearly situated the study of co-speech gesture into the domain of cognition and languagethere has been growing interest in gesture across several disciplines, from cognitive science to psycholinguistics and from developmental psychology to evolutionary anthropology. One of the consistent messages from this varied research is that speech and gesture are deeply connected systems of communication. Specically, researchers have demonstrated that gestures: (1) are semantically and temporally linked to the content of the ongoing speech stream, (2) have similar communicative functions as speech, and (3) develop closely with language acquisition in children. Furthermore, in terms of comprehension, listeners/viewers seem to pick up meaning of gesture, and do so in ways that are integrated with the speech signal (see Goldin-Meadow, 2003; Kendon, 2004; McNeill, 1992, 2005). This interest in discovering links between speech and gesture gained new momentum with Rizzolatti and Arbibs seminal discovery of mirror neurons (di Pellegrino, Fadiga, Fogassi, Gallese, & Rizzolatti, 1992; Gallese, Fadiga, Fogassi, & Rizzolatti, 1996; Rizzolatti & Arbib, 1998). These neurons discharge both when a monkey executes a specic manual action and when he observes another primate executing the same action. Since their discovery, several papers have investigated whether the human brain, specically Brocas area, also has similar mirror properties and furthermore whether this remarkable example of evolutionary conservation might reect a neural relationship between language and action systems in humans (for recent reviews, see Corballis, 2003; Nishitani, Schu rmann, Amunts, & Hari, 2005). The possible evolutionary link between action and language has fueled recent research on the neural processing

of speech and gesture in humans. Indeed, gestures are a special type of actionthey naturally and ubiquitously accompany spoken language, and certain types of gesture never occur in isolation from speech (McNeill, 1992). Although separate lines of research in the domains of language and action suggest that the two systems share overlapping brain areas/functions (e.g., Fiebach & Schubotz, 2006; Pulvermuller, 2005; Rizzolatti & Craighero, 2004), there is a surprising paucity of research directly and empirically investigating language and action together. If researchers want to better understand this neural relationship in the human brain, a prime place to look is co-speech gesture. Previous studies investigating the neural links between speech and gestures have focused on clinical populations, such as aphasics (see Rose, 2006, for a review) and splitbrain patients (Kita & Lausberg, in press; Lausberg, Davis, & Rothenha user, 2000; McNeill & Pedelty, 1995). However, there has been conicting evidence from these studies, possibly because of the variability in brain damage and case histories. In this issue, we attempted to bring together the state of the art in research using fresh approaches and improved techniques to shed new light on the neural relationship between speech and co-speech gestures. The papers cover a wide range of methodologies and topics that vary from production to comprehension, in pre-verbal infants, children, and adults, and in typical as well as special populations (i.e., Down syndrome). Some of the studies in this special issue use observational techniques (Iverson, Hall, Nickel, & Wozniak, 2007; Kita, de Condappa, & Mohr, 2007; Stefanini, Caselli, & Volterra, 2007), whereas others use online cognitive neuroscience techniques such as event-related potentials (ERPs) (Kelly, Ward, Creigh, & Bartolotti, 2007; Sheehan, Namy, & Mills, 2007; Wu & Coulson, 2007) and functional magnetic resonance imaging (fMRI) (Skipper, Goldin-Meadow, Nusbaum, & Small, 2007). All papers, however, address the issue of how speech and gestures are related in the brain during language processing and all explore what this can tell us about whether speech and gesture constitute a unied and integrated

0093-934X/$ - see front matter 2007 Elsevier Inc. All rights reserved. doi:10.1016/j.bandl.2007.03.006

182

Editorial / Brain and Language 101 (2007) 181184

system (Bernardis & Gentilucci, 2006; McNeill, 1992, 2005). Here we give brief summaries of each paper in the issue, raise methodological and conceptual issues about the current research, and make suggestions for further investigations. The rst paper, by Kita et al. (2007) investigates whether the dierential activation of the two cerebral hemispheres during dierent linguistic processes (i.e., in metaphor versus literal processing) inuences the handedness of iconic gestures. They show that during metaphoric languagewhich, according to the authors, dominantly invokes processing in the right hemispherespeakers are less likely to prefer their right hands over their left hands when they produce iconic gestures that imitate actions. This nding is in contrast to previous research that has found a right-hand preference when people produce cospeech gestures. The authors interpret this nding to mean that dierent types of gestures reect dierent cognitive processes during language production that map onto to not just the traditional left hemisphere, but the right hemisphere as well. Iverson et al. (2007) investigate the precursors to gesture production by linking handedness of rhythmic motor activity to the onset of babbling in pre-verbal infants. In contrast to previous work, the authors did not uncover a specic relationship between right-handed (left hemispheric) motor activity and onset of babbling. They interpret this to mean that specic linguistic mechanisms may not drive babbling onset, but rather that babbling may reect more general perceptual and motor developments. From there, the authors speculate that the coupling of rhythmic motor activity and babbling may serve as the foundation upon which speech and gesture systems develop during initial stages of language production. This suggests that the link between speech and gesture can be traced back to foundational neural systems that are generally specialized for action, rather than specically specialized for language. In the paper by Stefanini et al. (2007) the authors investigate the relationship between iconic gestures and speech in children with Down syndrome. Although the children all had delays in language development, the authors found that the DS children compensated for their delays by using more iconic gestures than typically developing children. Thus, according to the authors, even though DS children might suer from lexical production problems, their semantically appropriate gestures reveal a deeper conceptual knowledge of language than their verbal abilities would suggest. In this way, they claim that semantic features of words are encoded in sensory motor form, thus providing evidence for a link between language and action systems even in atypically developing populations. The next set of papers use event-related potentials (ERPs) to investigate online processing of gesture and speech during language comprehension. These studies build on previous behavioral ndings that have shown that listeners/viewers pick up information from both speech and

gesture during comprehension. The advantage of the ERP technique over previous behavioral methodologies is that it provides a window into how and when listeners/viewers comprehend gesture online and can reveal neural correlates of gesture and speech comprehension. In the rst paper, Kelly et al. (2007) replicate previous research (Kelly, Kravitz, & Hopkins, 2004) demonstrating that there is a larger N400 eectreecting semantic integration processes (Kutas & Van Petten, 1994)fo r speech that is preceded by incongruent versus congruent iconic gestures. Furthermore, they demonstrate that the size and scalp distribution of the N400 eect are modulated by whether the listener/viewer believed the speech and gesture belonged to the same or dierent person. That is, when participants heard an utterance produced by one person but actually watched another person perform the accompanying hand gestures, the size and scalp distribution of the N400 eect were smaller than when participant saw that the speech and gesture came from the same person. The authors conclude that the semantic integration of speech and gesture is not exclusively automatic and may be under some degree of neurocognitive control. In the second ERP study, Wu and Coulson (2007) investigate whether gestures that clarify the meaning of speech enable a listener to better conceptualize visuo-spatial aspects of a speakers meaning. Participants viewed gesturespeech utterances followed by pictures that were either related (or unrelated) to the gesture and speech or to just the speech alone. In the gesture and speech condition, there was a large N400 dierence between the unrelated and related pictures, whereas for the speech condition, the N400 dierence between the two types of pictures was less pronounced. The authors interpret this nding to mean that visuo-spatial aspects of gestures combine with speech to build stronger and more vivid expectations of the pictures than just speech alone. In the next paper, Sheehan et al. (2007) report the rst developmental ERP study on gesture. Their study builds on previous behavioral research by exploring neural correlates of processing gesture and speech in 18- and 24-monthold infants (Namy & Waxman, 1998). The researchers presented infants with videos of either a gesture or a word, followed by a picture that conveyed the same or dierent information as the gesture or word. They found an increase in the N400 component when 18-month-old infants saw incongruent pictures presented after words alone and gestures alone. However, at 24 months this eect was found with words only. The authors interpret these ndings as providing neurological evidence for a developmental change in semantic processing of gestures compared to speech. To complement the ERP research investigating the timing of gesturespeech processing, we have one paper that uses functional magnetic resonance imaging (fMRI) to explore neural networks involved in gesture and speech comprehension. This contribution by Skipper et al. (2007) is important because although we know that language

Editorial / Brain and Language 101 (2007) 181184

183

comprehension activates specic neural structures (e.g., Brocas area), it is not clear how these structures process gestures that accompany spoken language. In this study, participants watched videos of dierent stories composed of speech and gesture. The main nding was that when participants viewed stories in which speech was accompanied by congruent iconic gestures, Brocas area exerted the least inuence on other brain areas compared to when stories were accompanied by speech-irrelevant self-grooming movements or no gestures at all. The authors claim that the information provided by co-speech gestures helped listeners disambiguate the accompanying speech, and this in turn o-loaded phonological demands and lexical selection/retrieval processes from Brocas area. This study is one of the rst pieces of evidence showing that semantic processing of co-speech gestures modulates Brocas area zyu (see also Willems, O rek, & Hagoort, in press). We conclude with a paper by Willems and Hagoort (2007) that reviews the current research on neural correlates of co-speech gesture comprehension (including very new work not covered in this issue) by situating it in the cognitive neuroscience literature on neural links between language and action in general. Specically, they relate research on co-speech gesture to three separate but related bodies of work: motor bases for speech perception, cortical processing of action words/sentences, and brain mechanisms for sign language. Their review reveals that the neural relationship between speech and gesture may reect a fundamental relationship between language and action more generally. Moreover, it shows, according to the authors, that the brain is highly exible and openminded in how it uses context to connect these dierent neurocognitive domains. Overall, the papers in this special issue provide evidence that there are neural links between speech and co-speech gestures during production and comprehension across multiple ages and dierent populations. However, it is still an open theoretical question whether speech and gesture constitute a unied and integrated system (Bernardis & Gentilucci, 2006; McNeill, 1992, 2005) or are independent but highly interactive systems (e.g., de Ruiter, 1998). There are several avenues of research that could address this overarching theoretical issue, as well as other important questions in the study of gesture, brain, and language. One imperative goal is to discover creative ways to use brain imaging and electrophysiological techniques to better understand the role of co-speech gestures in language production. Another important goal is to determine whether the neural processing of co-speech gestures is unique compared to other actions, such as signs, pantomimes, and noncommunicative goal-directed actions. Research should also extend what we know about how the brain processes iconic gestures to understanding other types of gesture, such as points, metaphorics, and beats. Finally, there are exciting possibilities for comparative studies to explore the neural relationship between speech and gesture across ages, cultures, languages, clinical

populations (e.g., autism), and linguistic competencies (e.g., multilingual processing). Although these directions for future research make it clear that the eld of gesture, language, and brain is young and has much to accomplish, we believe that the studies presented in this special issue represent some of the important rst steps toward addressing these issues and ultimately understanding how gesture and speech are linked in the human brain. Acknowledgments zyu A. O rek was supported by Grant 051.02.040 from the Dutch Organisation for Scientic Research (NWO) during the preparation of the special issue. References
Bernardis, P., & Gentilucci, M. (2006). Speech and gesture share the same communication system. Neuropsychologia, 44, 178190. Corballis, M. C. (2003). From mouth to hand: Gesture, speech, and the evolution of right-handedness. Behavioral and Brain Sciences, 26, 198208. de Ruiter, J. P. (1998). Gesture and speech production. Doctoral dissertation, Radboud University, Nijmegen. di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., & Rizzolatti, G. (1992). Understanding motor events: A neurophysiological study. Experimental Brain Research, 91(1), 176180. Fiebach, C. J., & Schubotz, R. I. (2006). Dynamic anticipatory processing of hierarchical sequential events: A common role for Brocas area and ventral premotor cortex across domains? Cortex, 42(4), 499502. Gallese, V., Fadiga, L., Fogassi, L., & Rizzolatti, G. (1996). Action recognition in the premotor cortex. Brain, 119, 593609. Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge, MA: Harvard University Press. Iverson, J. M., Hall, A. J., Nickel, L., & Wozniak, R. H. (2007). The relationship between reduplicated babble onset and laterality biases in infant rhythmic arm movements. Brain and Language, 101, 198207. Kelly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89, 253260. Kelly, S. D., Ward, S., Creigh, P., & Bartolotti, J. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language, 101, 222233. Kendon, A. (2004). Gesture. Cambridge University Press. Kita, S., & Lausberg, H. (in press). Generation of co-speech gestures based on spatial imagery from the right hemisphere: Evidence from splitbrain patients. Cortex. Kita, S., de Condappa, O., & Mohr, M. (2007). Metaphor explanation attenuates the right-hand preference for depictive co-speech gestures that imitate actions. Brain and Language, 101, 185197. Kutas, M., & Van Petten, C. K. (1994). Psycholinguistics electried: Eventrelated brain potential investigations. In M. A. Gernsbacher (Ed.), Handbook of psycholinguistics (pp. 83143). San Diego, CA: Academic Press. Lausberg, H., Davis, M., & Rothenha user, A. (2000). Hemispheric specialization in spontaneous gesticulation in a patient with callosal disconnection. Neuropsychologia, 38, 16541663. McNeill, D. (1992). Hand and mind. University of Chicago Press. McNeill, D. (2005). Gesture and thought. University of Chicago Press. McNeill, D., & Pedelty, L. (1995). Right brain and gesture. In K. Emmorey & J. Reilly (Eds.), Language, gesture, and space (pp. 6385). Hillsdale, NJ: Erlbaum.

184

Editorial / Brain and Language 101 (2007) 181184 zyu Willems, R. M., O rek, A., & Hagoort, P. (in press). When language meets action: The neural integration of gesture and speech. Cerebral Cortex. Wu, Y. C., & Coulson, S. (2007). How iconic gestures enhance communication: An ERP Study. Brain and Language, 101, 234245.

Namy, L. L., & Waxman, S. R. (1998). Words and gestures: Infants interpretations of dierent forms of symbolic reference. Child Development, 69, 295308. Nishitani, N., Schu rmann, M., Amunts, K., & Hari, R. (2005). Brocas region: From action to language. Physiology, 20, 6069. Pulvermuller, F. (2005). Brain mechanisms linking language and action. Nature Reviews Neuroscience, 6(7), 576582. Rizzolatti, G., & Arbib, M. A. (1998). Language within our grasp. TINS, 21, 188194. Rizzolatti, G., & Craighero, L. (2004). The mirrorneuron system. Annual Review of Neuroscience, 27, 169192. Rose, M. L. (2006). The utility of hand and arm movements in the treatment of aphasia. Advances in Speech-Language Pathology, 8, 92109. Sheehan, E. A., Namy, L. L., & Mills, D. L. (2007). Developmental changes in neural activity to familiar words and gestures. Brain and Language, 101, 246259. Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2007). Speech-associated gestures, Brocas area, and the human mirror system. Brain and Language, 101, 260277. Stefanini, S., Caselli, M. C., & Volterra, V. (2007). Spoken and gestural production in a naming task by young children with Down syndrome. Brain and Language, 101, 208221. Willems, R. W., & Hagoort, P. (2007). Neural evidence for the interplay between language, gesture, and action: A review. Brain and Language, 101, 278289.

zyu Asl O rek Donders Center for Cognitive Neuroimaging, Radboud University, and Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands E-mail address: [email protected] Spencer D. Kelly Department of Psychology, Neuroscience Program, Colgate University, Hamilton, NY, USA E-mail address: [email protected] Available online 12 April 2007