REVIEWARTICLE

Multimodal Discourse Analysis:

Media, modes and technologies
Odysseas Constantinou
Cardiff University
PHILIP LEVINE and RON SCOLLON (eds.). Discourse & Technology: Multimodal
DiscourseAnalysis.Washington, D.C.: GeorgetownUniversity Press. 2004. 240 pp.
Pb (1589011015) $49.95.
SIGRID NORRIS. Analyzing Multimodal Interaction: A Methodological Framework.
NewYork and London: Routledge. 2004. 177 pp. Pb (041532856X) 18.99.
KAY L. OHALLORAN (ed.). Multimodal Discourse Analysis. London and NewYork:
Continuum. 2004. 252 pp. Hb (0826472567) 75.00.
The commonalities among the fields of (especially, critical) discourse
analysis and multimodality ^ such as the interest in the motivation of speaker
or sign-maker ^ are obvious on even the most cursory engagement with both
fields (cf. Iedema 2003). Predictably, the marriage of the terms multimodality
and discourse, within the four years since the publication of Kress and van
Leeuwens (2001) Multimodal Discourse, has become established within the
discourse analytic and social semiotic communities. The recent appearance of
the three books on review serves to further reify the bonds that exist between
the fields of Discourse Analysis and Social Semiotics, the latter of which,
having evolved from Hallidayan Systemic Functional Linguistics (SFL) and
extended by the work of Kress and van Leeuwen (1996), is arguably the parent
discipline of multimodality as a contemporary analytic concern.
In that same four-year span, very few books have, at least explicitly and
centrally, focussed on multimodalityas a keyconcern. Some have tackled related
issues: for example, Scollon and Scollons (2003) Discourses in Place: Language in
the Material World considered the material, worldliness of contemporary
communication (what they term geosemiotics) as manifested in various
face-to-face interactions such as coffee shop encounters, essentially fusing the
studyof non-verbal communicationwitha social-semiotic orientationgrounded
for the most part in Kress and Van Leeuwens (1996) Reading Images. There has,
Journal of Sociolinguistics 9/4, 2005: 602^618
# Published by Blackwell Publishing Ltd. 2005
9600 Garsington Road, Oxford OX4 2DQ, UKand 350 Main Street, Malden MA 02148, USA.
however, been much postgraduate activity concerned with multimodality, and
many theses completed which remain (as yet) unpublished. But inthe sense that
prominent academic publications guide and orient the field from and for which
theyare produced far more influentially than doctoral theses do, the publication
of the three books on reviewcanbe seenas heraldinga newphase in multimodal
research and ushering ina newgeneration of researchers. Theycanalso be seen
as the culminationof the wide varietyof researchthat has beenundertakensince
the publication of Kress and Van Leeuwens Multimodal Discourse. It is in this
capacity ^ as a public forumfor diverse and oftencompeting theoretical notions ^
that the three books can be considered valuable extensions of, and contributions
to, the academic fieldof multimodality.
TWOAPPROACHES
The books ^ two edited volumes, one monograph ^ are in large part the results of
the workof either current doctoral candidates or recent graduates. As such, they
can be seen as the research of the first generation of students committed to
further exploring territory already opened up by the most prominent advocates
of multimodal discourse analysis (MDA): Van Leeuwen, Kress, Lemke,Thibault,
Scollon, Baldry, OHalloran, and so on. Represented in these papers are two
approaches to multimodality that appear to be most prevalent currently. One
approach remains faithful to the origins of multimodality in SFL, and to its
systemic goals. This approach is well represented in OHallorans volume, the
researchwithinit practicallyloyal tothe traditionof systemic-functional theory
(p. 5). The other approach is less bound to the systemic origins of multimodality
and appears to derive its motivation, initially, froma critical inclination towards
mediated representations of (often current/topical) discourses of interest. As
such, it is closer in methodological orientation to both Discourse Analysis (DA)
and, its younger, tougher sibling Critical DiscourseAnalysis. This approachoften
deploys the same or similar analytic tools that can be found in SFL-oriented
research, but utilises them mainly as a means for examining the truth effects
(Chouliaraki in LeVine and Scollon: 155) of the representationunder scrutiny. In
Chouliarakis words, and indirect relationtotelevisionas a semiotic site:
Multimodality provides a discourse analytic point of entry into the procedures by
which televisual texts articulate language and visuality, orality, and writing; and the
procedures by which meaning is inseparably inscribed onto these distinct media:
verbal/aural, visual/pictorial, visual/graphic. (p. 154)
In short, the SFL-inspired work begins with an interest in the generation and
development of multimodal systemic theory (whether it be of print media, film,
music and so on), whereas the work that is continued in the spirit of DA begins
with an object of analysis, and deploys and constructs analytic categories
concerning multimodal semiosis that are intended to illuminate some aspect(s) of
the object under scrutiny. These approaches are not in contest. Rather, they
REVIEWARTICLE 603
# Blackwell Publishing Ltd. 2005
complement each other since, most obviously, each aims towards the same broad
objective of studying the phenomenon of multi-semiotic meaning making; and,
perhaps less obviously, it is particularly useful to have active in an academic
community, at anyone moment, boththose researchers whose strengths lie inthe
development of analytic frameworks and tools, and those whose critical investiga-
tions benefit from the use of those tools. Furthermore, each methodological
orientationcanleadtoresults that arefed back intothe discipline whichit is closer
alliedto. So, for instance, the chapters whichconstitute OHallorans edited volume
contribute as much in the way of theoretical insight and development towards
HallidayanSFLtheoryas theydotowards thegeneral project of multimodality.
In this review article, I want to focus on the overlap between the two
approaches ^ that is, on the conceptual agreement between both the DA and
SFL traditions in multimodal analysis, and that within eachapproach. I therefore
treat the books under reviewhere as instances of the issues that I identify in the
work on multimodality in general. The concepts crucial to multimodality, from
both DAand SFL perspectives, can be summarised as: medium and mode. Even
though these same terms are often deployed in a range of work, the concepts
they refer to are understood differently. Terms such as these are, of course,
operationalised individually inthe methodologies of the various works, rendering
it a mistake to take the relevant authors to task for their particular,
idiosyncratic treatment of keyconcepts (as long as their particular uses of those
concepts are made explicit and kept consistent). That said, I believe that it
would be a serious error to simply brush aside the deeper implications of the
various and competing deployments of some key concepts (as I see them).
Identifying and questioning the sometimes very subtle differences betweenthe
uses of these concepts that form the foundation upon which the research and
findings (interpretations) proceed can help us to understand the research
agendas of the individuals involved, and thus to situate historically and
critically evaluate the multimodal research agenda as a whole. The possible
terminological and conceptual differences are often noted but they are almost
always treated as a peripheral issue ^ commonlyas a literal footnote. Moreover,
terminological and conceptual agreement between the above-mentioned
complementary approaches to multimodality would further aid their
complementarity or their working relationship. The aim here is to address these
terminological issues directly.
In what follows, I provide a brief outline of the content of the three books,
then move onto the more detailed discussion. I conclude with some related
issues concerning the field of MDA.
CONTENT OVERVIEW
LeVine and Scollons Discourse and Technology: Multimodal Discourse Analysis
offers a selection of papers presented at the fifty-third Georgetown University
Round Table 2002 conference (GURT). The explicit aim of the volume (and
604 CONSTANTINOU
# Blackwell Publishing Ltd. 2005
conference) is to explore the relationship of discourse and technology within
the domain of multimodality. This relationship is twofold, and the book divides
its interest between the role of technology in (multimodal) discourse analytic
research and the impact of communication technologies on meaning making.
The impact of online communication practices and technologies on DA
research is a prevalent issue in this volume, more so than that of other
technologies. Other themes include (as stated in LeVine and Scollons
introduction): the study of social (inter)actions as a multimodal event, taken
up with great skill by Sigrid Norris, Ingrid de Saint-Georges, Elisa Everts and
Alexandra Johnson; the utility of multimodal discourse analysis in workplace
settings, covered by Whalen et al; and in educational social interactions,
discussed in depth by Rodney Jones, Angela Goddard, ElaineYakura, and Boyd
Davis and Peyton Mason.
OHallorans volumeentitled, Multimodal DiscourseAnalysis: Systemic-Functional
Perspectives, as the subtitle suggests, is far more specific in its methodological
orientation than Discourse and Technology. The chapters that constitute this
collection have much the same motivation and aims in common: that is, both to
extend the areas of interest, the sites of analysis, that SFL-informed multimodal
analysis covers; and to extend the theory itself. While these are not necessarily
separate concerns, there does appear to be a definite tendency in the chapters in
this volume topursue either one or the other aim.
A methodologically consistent selection of chapters are included here,
perhaps not surprising given that the majority of them are authored by
OHallorans own doctoral students. The book is divided into three sections
which represent three sites of study: Three-dimensional objects in space,
Electronic media and film, and Print media. The two most significant
contributions the book makes are in those chapters that deal with methods
(broadly speaking) for researching multimodal phenomena, and those that are
concerned with the interaction between two or more semiotic modes (termed
intersemiosis for the most part). Anthony Baldrys and Kay OHallorans
chapters come under the first category; and both are concerned with computer-
based techniques for analysing film and television product. Specifically,
Baldry is concerned with expounding a methodological model for multimodal
concordancing, and outlining the utility of a software tool he and others
haven developed ^ the Multimodal Corpus Authoring system (MCA); and
OHalloran explores the use of Adobe Premiere 6 for the multimodal analysis
of film. OHallorans chapter also comes under the second category in that it
aims to explore the ways in which the different semiotic modes deployed are
integrated and configured in Roman Polanskis film, Chinatown. Other works
of note in this category includeVictor Lim Feis chapter, in which intersemiosis
is discussed in terms of reinforcement of meaning between two or more
modes; and Pangs discussion on intersemiosis in museum displays. Also
worth a mention is the inclusion of Michael OTooles chapter, in which he
REVIEWARTICLE 605
# Blackwell Publishing Ltd. 2005
deploys a SFL-inspired model of architectural meaning to the Sydney Opera
House.
Finally, Sigrid Norris monograph Analyzing Multimodal Interaction: A
Methodological Framework differs from the other two books in terms of
function. Aside from its status as a monograph, and not an anthology, this
book is essentially Norris doctoral thesis reworked for publication as a text-
book. In its transmutation from thesis to textbook, a few thesis-like aspects
have survived, though these might also be considered usual textbook content.
For one, Norris lingers at length on a few issues of method: the first undertaking
is the definition, delimitationandoperationalisationof concepts andterminology
deployed. The second is a comprehensive survey and outline of transcription
issues in multimodal interaction analysis.What is valuable here is not only the
pedagogic aspect of this undertaking, but what it reveals about Norris world
view in her own research and in others. Speculations on what survived from
Norris thesis aside, the book also fulfils expectations about what a textbook
should strive to accomplish. To this end, each chapter includes at least one
assignment: bounded boxes of text that serve to define key concepts; and a
chapter summary.
Upon first encounter, Norriss project seems remarkably similar to that of the
non-verbal communication schools, prevalent inthe 70s and 80s particularly
in the field of anthropology. And, indeed, Norriss work is closely allied with
that body of ideas, and draws extensively on its literature; for example, the
section on Gesture (in Chapter 2, Communicative modes) extensively deploys
concepts of Kendon and McNeill. Yet it differs drastically from conventional
non-verbal approaches to interaction because of its explicit social semiotic
orientation (especially Kress and Van Leeuwen 1996). Norriss project, in
short, is to theorise the multimodality at play in everyday human-human
interactions.
MEDIA ANDMODES: FUNDAMENTAL CONCEPTS IN MULTIMODALITY
In this section I will explore the various uses, across the many chapters of the
three books under discussion, of two key concepts: mediumand mode. As men-
tioned above, this discussion is not intended to serve as an impassioned call
for terminological consistency. Rather, it is to identify the divergences among
the books and their chapters of their varied deployments of (what I believe to
be) fundamental concepts.
Modes and modeness
1
A frequent concern in criticism of work in multimodality is the reification of
the term and concept of mode, and of communicative modes themselves.
That criticismwould be justified to small degree in a reviewof the three works,
606 CONSTANTINOU
# Blackwell Publishing Ltd. 2005
for the modeness of the modes that are deployed as analytic objects is
sometimes taken as given in the chapters on review. For example, in the
introduction to Discourse and Technology, soon after discussing the usage of
mode as a term and concept, Scollon and LeVine state that [a] monomodal
concept of discourse is distorting (p. 3). On a first reading, this statement
seems true enough. Yet it also invites the question: is a multimodal concept of
discourse not distorting? Surely we should not so unreservedly comprehend a
multimodal viewas any more true or correct than a monomodal one? If we do
make such a judgment, then it follows that we are also seeing modes (or
modality) as things that are out there, that can either be captured or missed
by a given theoretical and analytical lens. Instead, it would be more
appropriate to consider a multimodal perspective as an analytic construct
built to enable those analysts that are so inclined to explore a wider range of
phenomena than just the linguistic and verbal semiotic systems. And, so it
follows, it would also be more appropriate to see the variety of identified
semiotic modes as heuristically-defined analytic constructs.
Among the authors who do not adopt the view that modes are easily
distinguishable and discrete meaning-making units is Norris, who actually
takes a firm stance against that view. For Norris, the apparent reification of
modes is the result of a lack of attention to the very concept of mode: once
researchers attend directly and explicitly to the concept, it is likely they would
concede that mode, by definition, is a heuristic concept, an analytic tool, and
merely a way of seeing the functioning of a text in terms of components.
Another example of a reflective viewon modeness in the work under review is
Angela Goddards chapter inDiscourse andTechnology, inwhich she problematises
the distinctions among certain terms, such as speech and writing; spoken
and written; oral and literate. These, for Goddard, are anything but
unproblematic distinctions, especially in the context of internet chat systems.
She writes:
One of the problems with using speech and writing as apparently transparent
categories is that, historically, there have been many changes in how notions of
speech and writing have beenviewed, resulting in a complex picture. (p. 36)
To exemplify this point, Goddard challenges Crystals (2001) outline of the
distinctions between speech and writing:
Speech is typically time-bound, spontaneous, face-to-face, socially interactive,
loosely structured, immediately revisable, and prosodically rich. Writing is typically
space-bound, contrived, visually decontextualised, factually communicative,
elaborately structured, repeatedly revisable, and graphically rich. (Crystal 2001: 28)
This, for Goddard, is true only of particular genres of speech and writing.
Invoking Tannen (1982), she argues that contrasts such as factuality and
interactivity are very much predicated on using casual conversation as the
prototype for speech and essayist literacy as the prototype for writing (p. 36).
REVIEWARTICLE 607
# Blackwell Publishing Ltd. 2005
The actual use of language of interactive written discourse (the internet chat
system prototype) subverts this distinction. To put this in context, Goddards
(for me, welcome) problematising of the transparency of, and distinction
between, the categories speechandwriting is pertinent to the assumed defin-
ability of any semiotic mode, in the three books under review and elsewhere,
in the sense that there is a historical basis to all semiotic systems, and the
boundedness of any system (as mode) does not escape this dimension.
Although, as Goddard argues, language as both writing and speech is
extremely difficult to define absolutely, this difficulty has not precluded the
widespread deployment of the term as a concrete and easily defined concept,
in both lay and professional contexts. Furthermore, it is possible that this
prevalent,naturalised viewof the assumed boundedness of language (as speech/
writing) has contributed to how we understand other semiotic modes, since it
seems that we have tended to apply this boundedness unproblematically to
other communicative means such as sound, image, gesture, and so on.
Further contributing to the conception of non-linguistic modes as bounded
is the application of analytic concepts originally developed in the context of
linguistic research (and proved successful in that context) to non-linguistic
modes of meaning making. Perhaps the most obvious example would be Kress
and van Leeuwens (1996) and van Leeuwens (1999) exporting of the concept
of modality (as relating to the representation of truth and reality) from
Hallidayan functional grammar into the semiotic systems of image and of
sound, respectively. But the application of linguistic concepts to non-linguistic
and non-verbal semiotic modes, as Bateman et al. (2002: 4) have argued, has
occurred without the same empirical basis that could be found in linguistic
analyses. Theories tested against data in one field (linguistics) do not permit
the wholesale transfer of the theory to other fields (image, sound, etc.).
Bateman et al. (2002: 4) call the analyses and theories of visual semiotics (in
particular, Kress and van Leeuwen 1996) examples of impressionistic or
interpretive analyses: analytic and explanatory procedures whose claims
perhaps cannot be supported. They continue:
The analytic procedures for establishing to what extent [the analytic principle of
Given and New] could be a reliable property of layout rather than an occasionally
plausible account are unclear. Nevertheless, following on the initial presentation of
the analytic scheme in van Leeuwen and Kress (1995), it has been presented again in
Kress and van Leeuwen (1996, 1998) and is now itself being adopted as unproblematic,
or given, in some systemically-based research on multimodality (see, for example,
Royce (1998), Martin (2002)). (Bateman et al. 2002: 6)
It is preciselythisgivennessof thefoundationof, say, contemporaryvisual semio-
tics that contributes to the objectifying viewof modes inother proposed semiotic
systems. AsVictor LimFei notes inOHallorans Multimodal DiscourseAnalysis:
While the direct adoption of a linguistic theory for other semiotic resources has been
criticised (for example, Saint-Martin, 1990), Sonesson (1993: 343) cautions that the
608 CONSTANTINOU
# Blackwell Publishing Ltd. 2005
outright rejection of the linguistics model must be at least na|ve, and as
epistemologically unsound as its unqualified acceptance. As such, a delicate balance
between the adoption of and rejection of linguistics theories to visual analysis and
intersemiotic processes must be maintained. (p. 221)
Modeness, as van Leeuwen (1999) in his work on the semiotics of sound and
music, and Kress and van Leeuwen (2001)argue, derives from the evolution of
media according to the socio-cultural interest in using a particular medium to
perform, fulfil or enable a particular social function. This conception of
modeness would entail that a mode is never absolutely defined or bounded,
only in a state of constant evolution ^ a point that needs to be heeded by
researchers at every level of their research. Sigrid Norris shares this view ^ and
in fact, has the most considered and therefore convincing account of mode of
the selection of work on review (though, in fairness to the other contributions,
she has much more scope inwhich to explore these issues). In her words:
Modes are unavoidably construed as distinct entities. [Yet] modes are not bounded
units. A mode is a loose concept of a grouping of signs that have acquired meaning in
our historical development. (p. 11)
This position would no doubt be shared elsewhere by Carey Jewitt, but it
remains implicit in her chapter in D&T. The explicit aim of her contribution is
to untangle some of the complex connections between the two concepts,
media and mode, which have, as she argues, become woven together like two
threads in a cloth (p. 184). Jewitt makes her position clear up front: modes are
means of representing, and media, means of disseminating. Yet, although Jewitt
attempts to untangle some of the complex connections (p. 184) between
media and modes, she sometimes contributes to their conflation. For instance,
in contrasting the media of book and CD-ROM, Jewitt argues that, in reading a
book, the reader is given a clear reading path ^ from the top left corner of a
page to the bottom right, and so on, from page one to the end (p. 187) and
suggests that this is a characteristic of the book as medium. Yet, reading path
is as determined equally by the mode of written language (in this case, written
English), not only by the mediumof book.
Elsewhere in the reviewed works, mode is not always so explicitly and
purposively defined. Its use in some of the contributors chapters implies that
those researchers think of a mode as more of achannel of communication (for
instance, Lim Feis chapter in Multimodal Discourse Analysis) ^ therefore
naturalising a modes discreteness by its relation to a particular sense organ
(the recognition of which in turn completes the circuit and creates the
channel). Most chapters in OHallorans volume bypass the issue of what a
mode or asemiotic actually is (if we follow the logic of many of the chapters
use of the termmultisemiotic instead of multimodal). It might be unfair to
take the writers to task for this, as the express purpose of those chapters is to
further some other aspect of a systemic-functional theory of multimodal
REVIEWARTICLE 609
# Blackwell Publishing Ltd. 2005
semiosis. In this sense, the lack of philosophical orientation to these writers
analytic practice is testament to the distinction made earlier in this review:
that each of the two approaches to multimodality has a different but comple-
mentary job to do. However, in contrasting the three books that represent the
two approaches, it would appear that what is usually seen as the moreapplied
discipline ^ DA ^ is actually the more theoretically reflexive in this case, in
that discussions of mode and modeness figure in manyof the chapters included
in those books. For most of the chapters in Multimodal Discourse Analysis,
modeness, or difference within the semiotic as Chouliaraki calls it, is a fact of
semiosis, and is rarely opened up to question.
Media
As outlined above, Norris in Analyzing Multimodal Interaction and Kress and
van Leeuwen (2001) believe that a mode is an abstract resource for semiosis,
not a physical or material one. This definition is posited in contrast to that of
media, which is defined by Kress and van Leeuwen as [the] material
resources used in the production of meaningful products or communicative
events. Media can be both materials and tools (e.g. airwaves and radio)
(Kress and van Leeuwen 2001: 21). In Analyzing Multimodal Interaction,
Norris calls for our conception of mode to remain open, yet the same needs to
be argued for the concept of media. Media certainly can be defined as the
material means with which meanings are made; but another dimension of
understandingis made possible byopeningthe concept uptoother interpretations.
As Jonathan Sterne (2003) argues in his book The Audible Past, most studies
that engage in the process of mediation and its products are let down by a
rather murky and tangled conception of medium, which for him is
a recurring set of contingent social relations and social practices, and contingency is
the keyhere. As the larger fields of economic and cultural relations aroundatechnology
or technique extend, repeat and mutate, they become recognisable to users as a med-
ium. A medium is therefore the social basis that allows a set of technologies to stand
out as a unified thing with clearly defined functions. (Sterne 2003: 182)
For Sterne, then, a medium is, in Luka' cs (1971) terms, a relationship between
people that has acquired a phantom objectivity, a certainthingness (Sterne
1999: 504). It is not a physical or material basis with which meaning is made
and modes evolve ^ such as a technology or set of technologies. Rather, a given
medium ^ television, in Sternes (1999) research ^ stands for the plethora of
relations between people, and the phantom objectivity that those relations
attain imbues that relationship with an autonomy that seems so strictly
rational and all-embracing as to conceal every trace of its fundamental nature:
the relation between people (Luka cs1971: 83, quoted in Sterne1999: 504).
Nicholas Cook (1998), in Analysing Musical Multimedia, meditates in his last
chapter on the question: What is a medium? He finds value in a definition
610 CONSTANTINOU
# Blackwell Publishing Ltd. 2005
offered by Jerrold Levinson, who denies that the term is equivalent to material
or physical dimensions (1984: 7; original emphasis).Rather, he continues,by a
medium I mean a developed way of using given materials or dimensions, with
certain entrenched properties, practices, and possibilities (Levinson 1984: 7,
quoted in Cook1998: 262).
Almost all of the chapters included in the books on review, and certainly in
Norrismonograph, see mediaas thephysical stuff of communication. For instance,
Jewitt (2004: 184), in Discourse and Technology, argues that media refers to how
texts are disseminated (p. 184, my emphasis), which would certainly be a
sufficiently open definition to include not only the tools and technologies of
dissemination, but its practices and infrastructure too. This would be in keeping
with Sternes and Levinsons notions of media quoted above.Yet inthe remainder of
the sentence, she gives the examples of printed book, CD-ROM, or computer
application (loc. cit). Jewitt is not wrong to use these examples, but she has missed
an entire other dimension of media (and mediation) and let her definition
perpetuate the notion that media are entirely physical means. In light of the
complexityof the termmode inthe workonmultimodality, this conceptionof media
is hardly surprising, as any alternative definition to a physical/material one would
likely overlap with the definitionof mode. In fact, in Levinsons definitionabove, his
treatment of mediumappears morelike the prevalent conceptionof mode.
CONCLUSION
The commonthread running throughthe three books is, of course, a multimodal
perspective on meaning making. There are other commonalities but, essentially,
the function of each book is different: NorrisAnalyzing Multimodal Interaction
fulfils a practical and pedagogical function, in that it aims to promote a
methodological and analytic framework for understanding the multimodality
at play in our real-time, face-to-face interactions, and deploys illustrations
and post-chapter exercises to that end; OHallorans volume presents much
recent work froma particular sub-discipline, keeping the reader up-to-date on
the state of play in SFL/multimodality research; LeVine and Scollons volume,
a wide-ranging collection, intends to tackle two perspectives on the same
phenomenon: the impact of technology on discourse, and the use of technology
in DA-oriented multimodal research ^ thus neatly ushering the concept of
technology into multimodal discourse analysis. On this basis it would seem
that the books are only comparable in a very general sense. Yet, as three
offerings made available at roughly the same time, they serve to define the
field of MDA in its current status; to offer abirds-eye view of this field; and to
determine what presently counts as an interesting and relevant line of enquiry
in MDA research.
Giventhe different functionof each book, it is difficult to compare theminan
evaluative sense, though the merits and shortfalls of each can be discussed.
REVIEWARTICLE 611
# Blackwell Publishing Ltd. 2005
I found the selection of topics and research styles in LeVine and Scollons
Discourse andTechnologya little too broad to be cohesive, and a little too one-sided
in its approach to the relationship of discourse and technology ^ but this is
understandable given the origins of the book and, by extension, the selection
of chapters included. That said, I would have liked to have seen more of the
chapters cover the first perspective on discourse and technology: the impact
of technology on meaning making. In terms of page count, Discourse and
Technology gives more space to examining the place of technology in MDA
research than technologys bearing on meaning making. When it does engage
with the latter, it attends to the impact of particular kinds of technology (online
communications technologies)on semiosis ^ a missed opportunity to explore
the array of technologies in media production and reception, and, more
fundamentally, the definition and delimitation of the termtechnology itself, a
subject which ^ alarmingly, given its title ^ is not raised in the volume. If we
are to understand and theorise the relationship of technology and semiosis, a
broader conception of technology is desirable here, as is the scope for the
inclusion of media production as well as reception technologies.
Understandably, a small percentage of the work included in the book attends to
the impact of technology on semiosis directly (though, as mentioned above,
many papers do focus on the place of technology in multimodal discourse
analytic research). Furthermore, and more generally, the theme of the relation-
ship of semiosis and technology should also encompass the notion of
constraint; that is, howsemiosis and the semiotic resources withina particular
context of practice are shaped by (and shape) the concerns, methods and
resources of the producer(s) in a specific setting. This notion includes a
historical dimension, so that investigation of technological constraints on a
particular site of meaning making involves looking at the technological
developments of a practice, and the coincident changes in the meaning-
making possibilities of that site. While some papers included in Discourse and
Technology (and indeed in the other works under review) do attend to this
notion, most do not; and while the idea of constraint may not be at all relevant
to many papers, to others it would contribute a welcome (in my opinion
necessary) dimension of analysis. Meaning making, as Fairclough et al.
(2001)noted recently,
cannot be reduced to the play of differences among networks of signs [and] cannot be
understood without identifying and exploring the extra-semiotic conditions that
make semiosis possible and secure its effectivity. (Fairclough et al. 2001: 5)
On a positive note, the wide-range of methodologies, topics and analytic
frameworks represented in Discourse and Technology is stimulating, and
consequently gives a sense of the GURT conference from which the chapters
originate.
In contrast to the broadness of Discourse and Technology, OHallorans
Multimodal Discourse Analysis is perhaps a little too narrow in methodological
612 CONSTANTINOU
# Blackwell Publishing Ltd. 2005
scope to be of extended interest to readers coming fromoutside the SFL sphere.
Many of the analyses and frameworks are extremely technical, with little to
help the SFL novice along the way. I admit that appeal to a wider readership
than that already extant in the SFL field might not be the aim, but there are
other clues that indicate that it is, at least, a partial aim: the title for instance,
Multimodal Discourse Analysis, suggests a DA dimension to the common SFL
approach to semiosis, and therefore a wider audience ^ not to mention how it
might be shelved in bookshops, actual or virtual (see below for further
discussion on this point). That said, Multimodal Discourse Analysis is an
outstanding volume that simply takes time to get acquainted ^ even, I imagine,
for the DA-oriented researcher who has already deployed SFL and SFL-inspired
theories (e.g. Kress and van Leeuwen 1996 would be a typical example) in
their work. The level of technicality referred to above, though difficult, is also
an asset to Multimodal Discourse Analysis: theories and frameworks are
outlined in great depth and with a high level of skill. Without its technicality,
perhaps, there would be little to commend this strand of research into
multimodality, as connections with theories outside the SFL sphere are few.
But there is much to commend in OHallorans book, particularly for the
seasoned SFL-inspired researcher.
The under-representationof some modes of semiosis in Multimodal Discourse
Analysis is my final criticism: most strikingly, sound ^ in film, TV, websites and
inreal spaces such as museums and domestic settings. Although this lack is
also true of the other books under review, it is Multimodal DiscourseAnalysis in
particular that aims to represent new sites of semiotic analysis. Presently, in
the area of the semiotics of sound design and music in multimodal contexts we
have only van Leeuwen (1999), Iedema (2001), and my own work on sound in
TV and film (Constantinou 2002, forthcoming). My call for sound to be
explored in a critically engaged and systematic way should not indicate that I
believe the study of sound, or audiovisuality, more important than other
potential semiotic modes ^ smell, for instance ^ but rather it means that I
believe sound to be as central to todays mass media as the visual, though
greatly understudied. Sound and image constitute the hardcore of multimodal
discourse, as hearing and sight are, what bothJohn Cage and Bertrand Russell
have called, the public senses: those most engaged by the mass media (Sterne
2003; Kahn1988; Russell 1921).
Norris Analyzing Multimodal Interaction, though a rigorous and rich
exposition of a methodological framework, would benefit from some pages
spent dealing with some fundamental issues concerning the proposed
methodology. As mentioned above, the definition and command of the concept
of mode throughout Analyzing Multimodal Interaction is exemplary ^ Norris
continually reiterates, where necessary, her point about the boundedness of
modes ^ but her general epistemological stance is not questioned. It would be
unfair to evaluate this book solely on this basis, given its pedagogical and
practical function, yet it remains a bugbear of mine that reflexivity (in this
REVIEWARTICLE 613
# Blackwell Publishing Ltd. 2005
case, a researcher questioning her own epistemology) is not considered neces-
sary in a pedagogical sense. In this sense, it could be argued that Analyzing
Multimodal Interaction should also aim to be used to teach researchers, via
demonstration, how to construct their own frameworks, not only how to use
the present one. Without more information and elaboration regarding the
construction of Norris methodological framework, the reader is left to assume
that ^ aside from the heuristic orientation to mode, and other concepts in
operation ^ Norris is ascribing her framework with a higher epistemological
status than could be justified.
To her credit, Norriss focus on multimodality is not made merely on
methodological grounds. Rather, she aligns her own project with the strongly
political aspects of multimodality (cf. Lemke 2002; Kress and Van Leeuwen
2001): that language, in both written and spoken forms, need not be considered
central in any explanation of human interaction, and that the plethora of other
communicative modes are always mobilised in every communicative and
material moment. It is this orientation to interaction that differentiates Norriss
work from the enormous body of research on non-verbal communication, in
which, for example, gesture, proxemics, smell, voice quality, and so on, are
considered ancillary modes of communication, with language as their master
(cf. the notion of media primacy in Cook1998).
Alsoof merit here is Norrisexplicit inclusionof mindandthe role of consciousness
in interaction as part of the necessary theorisation of a multimodal interactional
framework. In doing so, she marks out sharply the difference between theorising
real-time,transient multimodal interactionandtheorising multimodal represen-
tationinthe ever-increasing varietyof (mass-)mediated forms. Essentially, this is a
questionof materiality: different communicative events have different materialities
associated withthem, some enduring, like print media, others fleeting, suchas live
music.What is at issue in her considerationof consciousness (absent frommuchof
the previous work on multimodality) is the intentionality behind the communica-
tive events of humans. In the more durable materialities such as those involved in
the productionof prevalent media forms (e.g. televisionnews programmes), it is lit-
tle wonder the humanmind has not beenconsidered necessary for inclusioninthe
theory: we assumethat inmore durable forms of communicationtheverypresence
of a particular communicative action is enough to regard it as intentional, i.e. the
result of a conscious prior decision. In stark contrast, in a spontaneous and
unscripted interaction ^ a coffee shop encounter, for instance ^ we need to know
which of the many gestures and other non-linguistic, non-verbal actions that we
witness inour observations are to be considered a part of a particular aspect of an
interaction, and which are to be considered marginal. Only by appeal to the
humanmindcanthis be achieved.
Finally, the book is extremely well-organised and the argument, for the most
part, clear and concise. Its application in the classroom would be simple to
implement, especially as a complement to other, more contemplative and
reflexive works on multimodality, such as Multimodal Discourse (Kress andVan
614 CONSTANTINOU
# Blackwell Publishing Ltd. 2005
Leeuwen 2001). It deploys clear illustrations and exercises that are well
thought out and relevant.
A final note on Discourse and Multimodality
As noted in the introduction, the marriage of multimodality and discourse
analysis can be traced back to Kress and van Leeuwens Multimodal Discourse
(2001). The title of that book, however, suggests far more than even a surplus
of theoretical connections. It situates the concerns of discourse analysis as
inherently multimodal ^ which, of course, theyare (cf. Lemke1998 and various
others). But it narrows the scope of multimodality, and invites the question:
Are the concerns of multimodality necessarily oriented towards discourse, or
to the interests of discourse analysis? While the book attempts to sketch out a
comprehensive theory of multimodality, which includes discourseas only one
of four strata, it would seem that the title suggests a far more tapered
perspective.
In fact, the original title of Kress and van Leeuwen (2001) was
Multimodality. It was the publisher, Arnold, who recommended the books title
be changed ^ and narrowed in scope ^ from Multimodality to Multimodal
Discourse
2
. Discourse was then considered a marketable buzzword, and a
sensible market to pursue, in that the conceptual termmultimodality would
lack an anchor for otherwise prospective readers. Consequently, the book
found its home among other discourse-oriented works, and its title suggests it
is aimed at the DA community. It is without doubt that the title neatly
packages the content for a number of readers interested in the frontiers of
discourse analysis, while simultaneously isolating a number of other, less
field-specific readers who might otherwise be interested. Yet this is only a mild
criticismof the way the academic publishing industry works. Of course, where
there are funds to recoup, it can be expected that there are professionals
whose task it is to reify (or perhaps create) academic markets. My intention
here is not to reprimand those who facilitate the (non-electronic) distribution
of original and interesting theory. Academic marketing is a necessary evil and
the ins and outs of the marketing machine are not the issue here. What is at
issue, however, is the impact such branding might have on subsequent
research and publication (cf. Bateman et al. 2002: 4).
Which leads me to the future of the field of MDA. At present, there is a large
number of researchers who are pursuing what might be called traditional or
conventional multimodal discourse analysis; that is, the study of multimodal
texts (or the multimodal study of texts, to be more precise) from a social
semiotic orientation to discourse and meaning making. Those who do pursue
this line of inquiry will most likely derive their fundamental methodologies
from Kress and van Leeuwen (1996, 2001). There are fewer researchers,
however, concerned to widen their analytic and methodological lenses to
include analysis of the extra-semiotic, constraining features of a production
REVIEWARTICLE 615
# Blackwell Publishing Ltd. 2005
or interactional environment. This strand is extremely important to pursue,
and is recognised as such not only outside of the MDA field (e.g. Cottle 2003;
Fairclough et al. 2001) but also within it (especially Iedema 2003; Bateman
et al. 2003). Specifically, those interested in this pursuit should consider how
the dynamics of various media production and interactional processes, and
the practical and technological constraints of the media involved, play a part
in establishing the parameters for meaning making. This does not indicate
that I think what I have called above conventional multimodal analysis is of
less value thanthe proposed research strand. Rather, I believe it to be a comple-
ment to text-based analyses for the reasonthat, in SimonCottles (2003) words:
When the analyst infers, as is often the case from a textual interpretation alone, the
possible explanations, motivations or reasons that have informed their production,
angels are often to be found dancing on a pinhead of textual analysis [. . .] If we want
to understand why media representations assume the forms that they do [. . .] we can-
not rely upon readings of media texts alone, no matter how analytically refined and
methodologically sophisticated these may be. Studies of media organisation and
production powerfully address the problem of inference head-on by attending
empirically to the often complex articulation of differing influences and constraints,
both material and discursive, intended and unintended, structurally determined and
culturally mediated, embedded within the moment of production ^ and which can be
recovered by research. (Cottle 2003: 5)
NOTES
1. I use the term modeness instead of modality because the latter has been used,
already confusingly, in two other senses. One is synonymous withmode. The other
concerns the representation of truth through multiple semiotic means (see Kress
and van Leeuwen1996 for discussion).
2. Van Leeuwen, personal communication.
REFERENCES
Bateman, John, Judy Delin and Renate Henschel. 2002. Multimodality and empiricism:
Methodological issues in the study of multimodal meaning-making. GeM Project
Report 2002/01. University of Bremen/University of Stirling.
Delin, Judy, John Bateman and Patrick Allen. 2003. A model of genre in document
layout. Information DesignJournal 11(1): 54^66
Constantinou, Odysseas. 2002. Sound constructs. Unpublished MAthesis, University of
Wales, Cardiff.
Constantinou, Odysseas. forthcoming. Audiovisual organisation: Synchronicity and
continuity in broadcast news.
Cook, Nicholas. 1998. Analysing Musical Multimedia. NewYork: Oxford University Press.
616 CONSTANTINOU
# Blackwell Publishing Ltd. 2005
Cottle, Simon (ed.). (2003) Media Organization and Production. London: Sage.
Crystal, David. 2001Language and the Internet. Cambridge: Cambridge University Press.
Fairclough, Norman, R. Jessop and A. Sayer. 2001. Critical realism and semiosis. Paper
presented at the International Association for Critical Realism Annual Conference,
Roskilde, Denmark, 17^19 August 2001.
Iedema, Rick. 2001. Analysing film and television: A social semiotic account of
Hospital: An Unhealthy Business. In Theo van Leeuwen and Carey Jewitt (eds)
Handbook of Visual Analysis. London: Sage. 183^204.
Iedema, Rick. 2003. Multimodality, resemiotization: Extending the analysis of
discourse as multi-semiotic practice.Visual Communication 2: 29^57.
Kahn, Douglas. 1988. Cage and phonography. Paper presented at the Cage at Wesleyen
Symposium, Middletown, Connecticut, 27 February 1988. Online at: http://www.
vasulka.org/archive/Artists3/Kahn,Douglas/CagePhotography.rtf
Kress, Gunther and Theo van Leeuwen. 1996. Reading Images ^ A Grammar of Visual
Design. London: Routledge.
Kress, Gunther and Theo van Leeuwen. 1998. Front pages: The (critical) analysis of
newspaper layout. In Allan Bell and Peter Garrett (eds.) Approaches to Media
Discourse. Oxford: Blackwell. 186^219.
Kress, Gunther and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and
media of Contemporary Communication. London: Arnold.
Royce, Terry. 1998. Synergy on the page: Exploring intersemiotic complementarity in
page-based multimodal text. JASFL Occasional Papers1: 25^49.
Lemke, Jay. 2002. Travels in hypermodality.Visual Communication1: 299^325.
Levinson, Jerrold. 1984. Hybrid art forms. Journal of Aesthetic Education. 18: 5^13.
Luka cs, Georg. 1971. History and Class Consciousness: Studies in Marxist Dialectics.
Cambridge, Massachusetts: MIT Press.
Martin, JimR. 2002. Fairtrade: Negotiatingmeaninginmultimodal texts. InPatrickCoppock
(ed.) The Semiotics of Writing: Transdisciplinary Perspectives on the Technology of Writing.
Turnhout BrepolsPublishersandBloomington: IndianaUniversityPress.311^338.
Russell, Bertrand. 1921. The Analysis of Mind. Penn States Electronics Classics Series.
Available at: http://www2.hn.psu.edu/faculty/jmanis/b-russell.htm
Saint-Martin, Fernande. 1990. Semiotics of Visual Language. Bloomington: Indiana
University Press.
Scollon, Ronand SuzieWong Scollon. 2003. Discourses in Place: Language in the Material
World. London: Routledge.
Sterne, Jonathan. 1999. Television under construction: American television and the
problemof distribution, 1926^62. Media, Culture &Society 21: 503^530.
Sterne, Jonathan. 2003. The Audible Past: Cultural Origins of Sound Reproduction.
Durham, North Carolina: Duke University Press.
Sonesson, Go ran. 1993. Pictorial semiotics, perceptual ecology, and Gestalt theory.
Semiotica 99: 319^399.
Tannen, Deborah. 1982. Oral and literate strategies in spoken and written narrative.
Language 58: 1^21.
Van Leeuwen, Theo and Gunther Kress. 1995. Critical layout analysis. Internationale
Schulbuchforschung17: 25^43.
Van Leeuwen,Theo. 1999 Speech, Music, Sound. London: Macmillan.
REVIEWARTICLE 617
# Blackwell Publishing Ltd. 2005
Address correspondence to:
Odysseas Constantinou
Centre for Language and Communication Research
ENCAP
Cardiff University
Humanities Building
Colum Drive
Cardiff CF103EU
Wales, U.K.
[email protected]
618 CONSTANTINOU
# Blackwell Publishing Ltd. 2005